Originally published on UXmatters, [August 21, 2023]
Imagine for a moment that you’re transported to a world in which digital and physical realities blend, a world where you can interact with holograms as naturally as you do with tangible objects. Welcome to the realm of mixed reality (MR)—a fascinating convergence of the real and virtual worlds that has the potential to revolutionize our experience of life.
Ever since I first saw Apple Vision Pro, I’ve been spellbound by the amazing experiences that this technology offers. It’s like an entirely new universe has just come to existence, and we’re all explorers in these unexplored territories. However, this doesn’t mean we should wander aimlessly.
As UX designers, we have a unique opportunity—and responsibility—to shape this landscape in a way that enhances user experiences. Technology is just a tool, although a powerful one. But the real magic happens when we use this tool to solve real-world problems, to inspire, or to bring joy. That’s where UX design comes into play. With the rise of MR, UX designers are at the forefront of an exciting evolution. The canvas on which we’re used to working is expanding and becoming more dynamic. Now, more than ever, we need a comprehensive set of UX principles to guide us in crafting MR experiences that are not just impressive, but user friendly, accessible, and meaningful.
Exploring the Intersection of Technology and User Experience
When during WWDC 2023, Apple unveiled their Vision Pro mixed-reality glasses, or spatial computer, it was like opening a door to a new dimension—a dimension in which reality is not confined just to what people can touch or see with their naked eye. It offers a universe where we can amplify our perceptions through layers of digital interactivity.
Even as a seasoned UX designer, I found myself wide eyed with awe. Just the thought of creating digital experiences that seamlessly integrate with the real world was enough to send a thrill of anticipation coursing through me. However, once the implications of this technology began to sink in, I realized that this isn’t just another UX design playground. MR technology has taken a quantum leap that calls for a reevaluation of our existing UX design principles.
Why? People experience the digital user interfaces that we’re used to designing through screens—whether for mobile, desktop, or even augmented reality (AR) apps. With mixed-reality technologies such as Vision Pro, the screen is no longer a separate entity; instead, it’s intertwined with the user’s environment. The line between user and device fades and, suddenly, designing a good experience isn’t just about simplicity, efficiency, and aesthetics, but about how well you can blur the edges between the real and digital worlds. This is where the complexity lies. These are the questions with which I found myself struggling as I imagined the potential of Apple Vision Pro:
- How do you design a user interface that isn’t restricted to a flat surface, but exists in 3D space?
- How can you ensure the usability of a user interface that relies not just on taps and clicks, but on gestures, the user’s gaze, and voice commands?
- How can you guarantee that the digital dimension does not invade the user’s physical space or that inconsistencies between the real and virtual worlds don’t break the user’s sense of immersion?
In other words, how can we redefine UX principles to account for the unique, immersive nature of mixed reality?
Imagining this future led me down a rabbit hole of online, secondary research; deep discussions with colleagues and friends, and lots of thought experiments. But most importantly, it led me to think about UX principles for mixed reality, principles that would ensure that we, as UX designers, are ready to embrace this new reality and design experiences that are immersive, intuitive, and inclusive.
So, without further ado, let’s explore ten UX principles for mixed reality and consider how they can guide us in designing mixed-reality experiences.
10 Essential UX Principles for Mixed Reality
After immersing myself in the challenging, yet exciting arena of mixed reality, I’ve drawn up a set of principles that I believe should guide our designs. Let’s explore each one in turn.
1. Spatial Consistency: In our mixed-reality designs, the virtual environment should be consistent with our real-world spatial understanding. Objects in mixed reality should respect real-world rules of physics such as gravity and solidity and should remain consistent in size and location with real-world objects—unless the UX designer deliberately designs them to deviate from the real world for a specific purpose. Abiding by this principle would ensure that the user could intuitively understand and interact with the MR environment. Following real-world physics also makes the environment feel more familiar and believable.
💡 Example: Imagine that the user places a virtual toolbox on a real table. The toolbox should stay on that table, not float away or go through the table. Even if the user leaves the room, then comes back to that room, the user still expects the toolbox to be there. Maintaining spatial consistency means the toolbox would remain where the user has placed it, respecting the laws of real-world physics.
2. Contextual Relevance: An MR experience should be relevant to the user’s current environment, needs, and tasks. Thus, UX designers should provide contextual cues that relate to the user’s physical environment or the task the user is performing. These contexts should provide whatever information or features are most likely to be useful to the user at any given moment. Depending on the current context, this could mean highlighting certain features or information within the user’s field of view or adapting the user interface to match the user’s physical environment.
💡 Example: Imagine that the user is using a cooking app. As the user goes through the steps for a recipe, the app shows the next step or tool immediately within the user’s field of view. If the user were cutting vegetables, the app would highlight the knife and cutting board in the virtual environment, enhancing the cooking experience.
3. Balance: Mixed reality can quickly lead to cognitive overload. Avoiding cognitive overload and promoting a healthy balance between the virtual world and the real world are important. Users should be able to easily comprehend the information that the user interface presents and also know when to take a break.
💡 Example: In a shopping experience, showing every product detail for all items that are currently within the user’s view could overwhelm the user. Instead, detailed product information should appear only when the user is focusing on a particular product. Thus, the user interface would provide a properly balanced amount of information.
4. Seamless Transition: Transitions between the real world and the virtual world should be as frictionless as possible. Smooth transitions between real and virtual worlds can enhance the user experience by avoiding abrupt changes that could disorient the user or disrupt the user’s workflow.
💡 Example:
In a virtual-reality tour of a historical site, moving from exterior views to interiors should involve passing through doors or entrances, giving the user smooth transitions between different environments. Also, as the user moves from a historic monument in the real world to historical images from the past in a virtual context, these transitions should feel seamless and natural, as if users are changing their focus in the real world.
5. Authenticity(Reality-Virtual Boundaries): With mixed reality, it’s easy for users to lose touch with their actual surroundings. Therefore, UX designers must design clear boundaries between the virtual and real worlds, using visual cues, reminders, or distinct visual styles that set the virtual environment apart. This can keep users grounded in reality while they navigate the virtual world..
💡 Example: Imagine a home-decorating app. As users move virtual furniture around, the app should visually distinguish real-world objects and virtual ones, perhaps using different outlines or shading to show that the furniture is moving. This would prevent users from losing touch with their actual surroundings.
6. Sensory Harmonization: To provide an immersive experience in mixed reality, all sensory inputs should work in harmony with one another. Sight, sound, touch, and potentially even smell and taste should align seamlessly. Any inconsistencies or mismatches between different types of sensory feedback could disrupt the user’s sense of immersion and cause confusion.
💡 Example: If the user were playing a virtual piano in the MR environment, the sound of each key should synchronize with the press of that virtual key. Plus, any haptic feedback should sync with these actions, providing a coherent multisensory experience.
7. Clear Feedback: Just as for any UX design, clear feedback is critical. Users need to understand their interactions and the results that they produce. This becomes even more crucial in a 3D space in which users can interact using gestures, gaze, or voice commands.
💡 Example: If the user were operating a virtual MR device using voice commands, the system should provide clear feedback such a visual tick, a sound, or a haptic buzz to acknowledge the successful execution of a command.
8. Accessibility: It is essential to remember that mixed-reality experiences should be accessible to everyone. Our designs must consider people with varying physical abilities, different types of sensory processing, and neurodiversity. Inclusivity should be at the forefront of our design-thinking process.
💡 Example: A reading app could support voice commands and read the text aloud for users with visual impairments and incorporate visual sign language for users with hearing impairments.
9. Intuitive Gestural Language: As UX designers, we need to define a set of gestures that feel natural and intuitive to users. Overloading the user with complex gestures could lead to frustration, negatively impacting the overall user experience. As mixed-reality technology evolves, we should establish a universal language of gestures, enhancing the accessibility and usability of our designs.
💡 Example: In an app for learning a new language, users could shake their head to dismiss a word they already know and nod to save a word for further review, mimicking our real-world gestures of agreement and denial.
10. Personal Space: Respecting the user’s personal space is fundamental to the MR experience. While virtual environments offer a vast playground, UX designers must ensure that virtual objects or avatars do not intrude into the user’s personal space or cause discomfort. This includes understanding social norms about proximity and personal boundaries in various cultural contexts.
💡 Example: On social-interaction platforms, when avatars approach each other, they should respect the cultural norms of personal space. The platform should prevent avatars from getting too close, similar to the way people maintain a comfortable distance from others during real-world conversations.
The Future of UX Principles for Mixed Reality
As we stand at the brink of an exciting new era of mixed-reality experiences, the UX principles that we hold today must continue to evolve with the technology. This is a certainty. In the same way that our design principles have adapted with the introduction of mobile technology and the rise of artificial intelligence, they must once again shift to accommodate the immersive, boundless world of mixed reality.
One of the most significant areas of development that I foresee is in the domain of haptic feedback. Currently, our ability to touch and feel things in a virtual environment is relatively limited. However, ongoing research in haptics—such as through the use of wearable devices—could significantly enhance our ability to interact with and perceive the virtual world. Such advancements would not only enhance the authenticity and immersion of MR experiences but also introduce an entirely new dimension to consider during UX design.
Increasingly, the incorporation of artificial intelligence (AI) and machine learning (ML) into mixed reality will have profound implications for UX design. As AIs become more adept at understanding and predicting user behaviors, the potential for personalized and adaptable user interfaces in MR becomes enormous. These advances will likely lead to the emergence of new UX principles centering around personalization, privacy, and data usage in MR.
Plus, since MR is an ideal platform for social interactions and shared experiences, the need for social UX principles will become more important. As users interact with each other in MR environments in more intuitive and immersive ways, UX designers will need to consider such factors as group dynamics, shared spaces, and social norms when designing these experiences.
Finally, our focus will shift from designing for individual sensory modalities such as visual, auditory, or haptic toward multisensory experience design. In a mixed-reality environment, users should perceive rather than just see or hear. As UX designers, we’ll need to better understand holistic perception, integrating inputs from multiple human senses, and our UX design strategies will need to address this.
However, these exciting new developments come with challenges. As UX designers, we’ll need to navigate complex issues such as motion sickness in virtual reality (VR), the ethics of immersive technologies, and the need for universal design standards in MR to name just a few areas of concern. However, despite these challenges, the future of MR is bright, and as UX designers, we’ll have a crucial role to play in shaping that future. So we must not shy away from these challenges, but embrace them and look upon them as opportunities to innovate and redefine what is possible.
Conclusion: Embracing the Mixed-Reality Revolution
As we find ourselves on the cusp of this fascinating mixed-reality revolution, it is clear that, as UX designers, our journey is poised to transcend conventional boundaries. Our canvas is no longer confined to screens or two-dimensional planes. Instead, it extends into the vibrant, infinite realm of MR and promises an era of unrivaled user experiences.
The ten MR principles that I’ve outlined in this article are just a starting point for our journey into the exciting world of mixed-reality design. Spatial consistency, contextual relevance, balance, seamless transition, authenticity, sensory harmonization, clear feedback, accessibility, intuitive gestural language, and respect people’s personal space will be crucial as we venture into the world of mixed-reality design. But, as with all design principles, these MR design principles must evolve as we learn, experiment, and our understanding of MR technologies grows.
The path ahead, while challenging, is full of potential. We are no longer just designing user interfaces; we’re creating immersive experiences that blend the virtual and the real, the tangible and the intangible. MR design will shape how people perceive, interact with, and understand the world around them. As UX designers, reaching the frontiers of mixed-reality experiences is our responsibility, but more importantly, it is our opportunity.
I want to invite you to think about the potential impact of these MR principles on your design philosophy. How will you embrace the mixed-reality revolution? What changes will you make to your design approach? As UX designers, the future of mixed reality rests in our hands, so it is up to us to mold immersive, intuitive, inclusive MR experiences for all.
The dawn of mixed reality is here, and it’s exhilarating. Are you ready to embrace MR and redefine the user-experience paradigm?
