Mixed Reality

With every new project that involves XR in some way, designers face the problem that the terminology of this new branch has not yet fully manifested itself. A common vocabulary for these technologies is therefore of utmost importance to keep communication within the team clear and direct. For this reason, we would also like to share our XR compendium with you.

Basics

Augmented Reality

Projection of virtual objects onto the real world in real time. Compatible with modern smartphones or AR headsets like Microsoft Hololens.

Augmented Virtuality

Projection of real objects onto a virtual world. Requires VR-compatible headsets such as HTC Vive or Oculus Rift.

Virtual Reality

Simulation and display of virtual objects in stereoscopic images. Creates spatial effect by recording the user’s eyes separately.

Umbrella term for all technologies that are located between the unadulterated, real world and completely virtual reality (e.g. augmented reality or augmented virtuality).

Reality-Virtuality Continuum

The reality-virtuality continuum shows the different stages in the transition from real space to a completely virtual environment.

Hardware & Technology

Eye Tracking

Capture and record the eye movements of a personality.

Head Tracking

Head tracking refers to a method of detecting the position, attitude, and movements of the head to provide a display corresponding to the viewing angle or to enable other head-based control. (wikipedia)

HMD — Head-Mounted Display

A head-mounted display is a visual output device worn on the head that displays images generated on a computer on a screen near the eye or projects images directly onto the retina through virtual retina display (e.g., HTC Vive, Oculus Rift, Microsoft Hololens).(wikipedia)

Holophonic Sound

Spatial audio playback method with the aim of creating virtual, acoustic environments. The generated sound can thus serve for spatial orientation in a scenery.

HUD — Head-up Display

Display system in which the user can maintain his head position or viewing direction because the information is projected into the field of view. Known navigation element from the controls of computer games or modern airplanes.

Imaging

Photogrammatic design

A 3D object or environment is created automatically on the basis of a large number of photographs from different perspectives (3d scan) in a 3D program without modeling.

Stereoscopy

Generation of spatial images using two conventional images showing a scene from slightly different viewing angles.

Feedback & User Guidance

Haptical Feedback / Rattle

Haptic feedback of a device, e.g. vibration on a controller.

User Experience and User Interface

Contentlayer / Screen Space

Areas decoupled from the scenery, which can hold additional information.

Diegetic UI

Positioning of the user interface both in the spatial environment and in relation to the context and setting of the application.

Fuse Button

UI elements that trigger a visual countdown when the user has focused on them for a defined time. Replaces direct feedback when the user has no way to trigger an action on a button or hotspot using a controller or physical buttons on the viewer.

Gaze Selection

Selecting UI elements using eye tracking, or head movement.

Hotspot/Hotspot with label

UI element that is linked to an object in the scenery.

Hotspot-Menu

Additional navigation layer in a hotspot.

Hotspot-Area

Sensitive area in which a hotspot is activated.

Hover

Visual/auditory/haptic feedback when the reticle activates a UI element.

Reticle

Corresponds to the mouse pointer in a conventional interface. Controlled either by head movement, gestures or controller.

Spatial UI

Positioning of the user interface in the virtual or real spatial environment (e.g. as a projection on walls).

Interactions

360° Image/Video

Correspond to spherical panoramas. Spherical panoramas or spherical panoramas are panoramic images in which all angles visible from one point of view are reproduced. On such photographs, an image angle of 360° in width and 180° in height is reproduced. (wikipedia)

Immersion

Describes the effect that makes the user of a virtual environment feel real. If the degree of immersion is particularly high, it is also referred to as ‘presence’.

Position Tracking

Detection of the position, attitude and movements of a body in space.

Transporter/Teleport

Hotspot that leads to a change of location within the application.

VR Video

Stereoscopically captured 180° or 360° video that allows the user to look around.

Design

Context Design

Design in relation to the user’s current context (e.g., location, activity, time) to provide more user-friendly results.

Equirectangular

Back translation of a spherical panorama into a two-dimensional image. Equirectangular two-dimensional images represent 180° image angles in the vertical image axis. The image of a virtual sphere surrounding the viewpoint is projected onto a surface.

First-person-design

The principle of not guiding the user as the designer of the application. The user should be free to test the possibilities himself and make decisions independently.

Non-narrative-design

The intentional absence of a story within an application. Instead, the user should be able to develop his own story.

Skeuomorphism

The leaning of visual, auditory, or haptic elements against real objects/processes.

Environment

Field-of-view

The field of view, is the area of a scene that the user can see at any given time.

Five years after the arrival of VR technologies in our technology-centric society, extended realities are still a red-hot topic. The possibilities still seem limitless.

As an experienced digital brand agency, we are always expanding our in-house realities and can assure you that it is far from too late to jump in. Therefore, we would like to share with you some learnings as orientation, inspiration and good practices. Here are our golden rules for designing user interfaces for XR applications.

Learn from conventional media

As a pioneer in an extremely young and at least equally unexplored terrain, one naturally likes to rely on the familiar and one’s own experience from less immersive and more conventional media.

In Dead Space 3, the glowing spine indexes the damage taken, while the weapon displays the remaining ammo via hologram. Image source: dreamdawn.com

Game design is the ideal source of inspiration, as three-dimensional user interfaces have been explored there for decades. Since the days of the first consoles, game designers have been looking for ways to embed their user interfaces into the setting and narrative of their creations to maximize immersion.

While this is not true for all games, role-playing games in particular are based on the principle of immersing the user as much as possible in the virtual world.

Star Citizen not only presents the UI in a believable way, but also makes it visible to other players. Image source: vrnerds.de

Unbelievable interfaces create a disturbing dissonance here, which can quickly lead to the loss of the player’s attention. A central aspect that must also be avoided in XR applications in order to achieve the greatest possible immersion for the user.

Dirt Rally offers only authentic speedometer and cockpit displays for orientation. Image source: uploadvr.com

The solution is the diegetic UI, i.e. a user interface that exploits the spatiality of the application as well as the narrative and the setting in order to provide the user with credible information. This can be, for example, the clock on the protagonist’s arm, to which attention must be paid at regular intervals so as not to miss important events, or the dashboard in the cockpit of a vehicle, which can be used to read off speeds and other parameters.

More about UI in video games in this artikel of Anthony Stonehouse.

Choose the right interface concept

Probably the most important aspect for all design in XR is to always be aware of the aim and nature of the project. For example, a minimalist AR companion for concertgoers does not need to be absolutely immersive. Authenticity may not be the primary focus in such meta-interfaces. In this example, Diegetic UI is not superior to a regular, static interface, but only complicates the user experience unnecessarily.

Overall, user interfaces can be divided into three categories. Static UI, places information as a static overlay over the user’s field of view (this method is most popular with AR apps). In comparison, Spatial UI uses the three-dimensional environment to position the interface, such as a floating menu that curves around the user. Finally, there is Diegetic UI, based on which one does not break with the narrative of the application and the authenticity of the setting, and renders user-relevant information through believable shapes (a working clock on the wall, the display of a virtual smartphone, etc.). Before starting any project, it is therefore particularly relevant to weigh the pros and cons of the different platforms, as well as the different UI methods, and find the most effective solution for this project.

Design cross-platform and in patterns

The biggest challenge of Extended Realities – and also the biggest difference to conventional media – is the diversity of interaction design. The various disciplines could not be more different: While one application is controlled via motion tracking through gestures and hand movements, the next uses exclusively controllers and buttons, whereas another works only via gaze control.

This rich fauna of interaction possibilities requires strong design patterns and pattern libraries, especially for projects across multiple XR platforms, so that the user journey and user experience remain as consistent and user-friendly as possible.

Guide the user – gently and cautiously

Good UI: Clean, crisp and comprehensive. Image source: aixlab.com.

In the same vein, interface density is something to consider. Popups and crowded interfaces are already confusing on two-dimensional media and cause some headaches, but in XR – especially when you are positioned in the middle of the UI – chaotic and crowded UI has a claustrophobic effect.

Therefore, it is recommended to keep your distance first, let the user approach the virtual environment, and only then interact.

Final Thoughts

In any sense, one should not be put off by XR and its associated complexity. As a trailblazer, new aspects are explored every day and the opportunity to pioneer new niches is hardly so close in any other field. Thanks to open source software such as Unity and Blender, the entry threshold is extremely low and even laymen can build prototypes within a very short time. Go explore!