Beyond Screens - Mixed Reality User Experience Design

In the framework of my master’s degree thesis I created a set of design guidelines, based on the concept and principles of universal design, for the development of multidimensional interfaces in mixed reality environments. It was then possible to put into practice those guidelines in a field that has always welcomed favourably this type of technology, the surgical one.

Following a user-centered design process and the proposed ux design guidelines, it was possible to create a proof of concept interface, with an interaction design study on natural gestures , that could solve some of the gaps within the intra-operative process due to the contextual disabilities that are affecting the surgical operators.

The Challenge

It seems quite clear in recent years that the direction we are taking regards a progressive departure from those that a while ago were an essential prerequisite for the interaction with digital devices: the screens.

Traditional interfaces are increasingly evolving, becoming more and more invisible and integrated within a multidimensional environment, towards a condition ‘beyond screens’. In this regard, mixed reality offers a field of experimentation also supporting the human need to embody their own technologies.

Starting from these premises, through an intensive research, analysis and a user-centered design approach, I developed my master’s degree thesis project called Beyond Screens - Designing universal experiences within mixed reality environments.

Preliminary Research and Analysis

Historical-Anthropological Analysis

Starting from a historical-anthropological analysis on the relationship between the man and his extensions, through concepts linked to the world of cybernetics and transhumanism, I have defined how this connection has strongly evolved towards a progressive incorporation.


Mixed reality, through the creation of multidimensional environments, is able to bring to extremes concepts such as internet of things and ubiquitous computing, making traditional interfaces more and more evanescent.


Because the perception of the frame that separates the organic and the virtual reality is becoming more and more thin (reality-virtuality continuum), the issue of accessibility assumes an increasingly important value. I decided to deal with this issue through universal design principles that takes accessibility to an extreme.

UX Design Guidelines for Mixed Reality Environments

After tracing the initial premises through an intensive preliminary research and analysis, I presented 8 design guidelines for the development of interfaces for multi-dimensional environments. Those best practice were developed taking into account the current evolutionary trend of humans in relation to digital technologies as well as the idea of universal design and its principles.

These guidelines introduce the idea of a holistic design that requires to consider various elements in synergy, in this case: user, environment and technology. And to understand how those three elements are interacting with each other.

Proof of Concept

I then decided to develop a proof of concept interface in order to validate and iterate the proposed guidelines. The P.O.C concerns the development of a Microsoft Hololens interactive application to be introduced in a field were contextual disability is high: the surgical field.

The whole project was developed through a user-centered design approach, following the design thinking process.

Empathize - User Research and Anslysis

Preliminary Research

To choose the primary audience to focus on it was necessary to identify the roles within the surgical team. To do this I carried out a desktop research and read some introduction to surgery books.

There are two main roles that need to always have their eyes focused on the operating table and therefore limited opportunities to observe and interact with external monitors: surgeons and nurses. I decided at this early stage to focus on the surgeon role.

Netnographic Research

To better investigate the role of the surgeon and to be able to prepare the interviews effectively I conducted a netnographic research on some discussion forums dedicated to surgeons.


I then proceeded to interview two surgeons and also two nurses to get an external view on that role. This is what emerged:

  • Once the surgeon is sterile, he can no longer come into contact with anything that is not sterile itself. If during an operation, the surgeon needs to view some plates, he needs to ask for help to a room assistant and give instructions on how to navigate the plate.
  • The tools that surgeons use most are their 5 senses
  • When they can’t use their senses or they are not enough they need to use some technological extensions in some cases controlled indirectly due to the sterility factor

  • Contextual Inquiry and Task Mapping

    Due to logistical problems, it was not possible to carry out a real contextual analysis. But online I was able to view some audio-visual documents that allowed me to understand how surgical operators work in the field. The contextual analysis confirms the things that emerged from the interviews.

    Through task mapping I was able to define user tasks in the form of a storyboard. I also added the variables emotions and thoughts, typical of the user journey map, to get a complete picture.

    Segmentation and User Modeling

    I then could proceed with the segmentation of the 'surgeon' target into three clusters, through the variable defined by Kotler P. 'lifecycle', in this context renamed more properly ‘careercycle'. Below there are the three clusters that I identified. For each cluster I developed a personas model and an empathy map in order to explore needs and wants:

  • Specializing and new surgeons
  • Expert surgeon
  • Senior surgeon

  • User Stories and User Needs

    After defining some user stories I was able to put together all the insights collected previously and to define the needs of the target:

  • Surgeons need to have the possibility to consult any vital parameters and documents directly, maintaining a condition of sterility and the use of their hands
  • Absolute concentration and avoid any distractions
  • Have different types of auxiliary assistance based on their level of preparation
  • Speed up the entire intraoperative process

  • Goals and Experience Strategy

    Make surgical work more efficient and provide the surgeon with a "sixth sense” that allows him to overcome the limits of his 5 senses during surgery


    By developing a system that allows surgeons to consult medical documents, real time images and 3D scans autonomously

    Let the surgeon iteraci with the system while keeping their hands free and sterile


    By leveraging the potential of mixed reality and by designing interactions that reflects human natural gestures and the ones related to the surgical field


    Logical Design

    Within the logical design I defined the sections, the architecture/flow, the labeling and the general semiosis (how the system should talk to the user).

    In this phase I defined the system metaphor which would lead the entire project. Since this is a project that aims to allow consultation and navigation, the metaphor which is the most familiar to this type of context is the archive.The elements that characterize the real clinical archiving environment must therefore be translated into the idea of a digital archive:

  • Medical records
  • Cataloguing system
  • Shelves
  • Auxiliary aids
  • Possibility of consultation

  • At this stage I also started thinking on how the interface would be. The general structure has been designed to be modular in order to allow total freedom of interaction and management by the user.

    Interaction Design

    Starting from the work done by Wobbrock J. O., Morris M. R. and Wilson A. D., I then decided to conduct a simulation session with some users, in order to identify recurring patterns of interaction to be applied into mixed reality context as natural gestures.

    I have provided users with a scenario and some tasks. They were asked to manipulate and perform some operations against physical objects considered as virtually generated elements in order to identify some interaction patterns.

    Visual Design

    Within the visual design phase colours, typography, iconography and shapes have been designed to fit the user, the technology and the environment.

    Colours: needs to fit an operating room without getting mixed up in the background or faded by the strong lights. Therefore the colours needs to be in contrast with the surrounding environment.

    Typography: the use of sans serif fonts is preferable in order to ease the reading within an environment that is characterised by movement. The size of the typography depends on the distance and on the physical ability. I conducted a flash test in order to determine a safe font size based on surgery working-distance which is approx. 50 cm.

    Iconography: the use of solid icon is preferred in order to ease the reading and reduce cognitive load. After selecting all the icons I tested them out to evaluate: findability, recognition, information scent, attractiveness.

    Shapes: in order to be consistent with the idea of modularity, I favoured the use of geometric shapes with clear cuts, which also helps defining the areas of content more clearly.

    High Fidelity Mockup