Project Helios

Project Helios

Project Helios

Designing AR Spatial Smart Glasses

Designing AR Spatial Smart Glasses

Designing AR Spatial Smart Glasses

End to end product design for a new to market product: Using an iterative, user-centric process of user research, product strategy, stakeholder alignment, interaction design, UX, UI design and usability testing, I designed an AR Smart Glasses UX for all day use.

End to end product design for a new to market product: Using an iterative, user-centric process of user research, product strategy, stakeholder alignment, interaction design, UX, UI design and usability testing, I designed an AR Smart Glasses UX for all day use.

Crafting a familiar but new, world first user experience for a new to market product

Crafting a familiar but new, world first user experience for a new to market product

Crafting a familiar but new, world first user experience for a new to market product

AR Smart Glasses and Smart Frames enable users to interact with UI at any time throughout their day through their lenses. Current interaction offerings such as touchpads on the frames and controllers create friction, there was an opportunity for Ultraleap to make an impact on the industry by bringing spatial interaction to always on Smart Glasses and Smart Frames so users could simply interact with using their hand, with no peripheral devices or effort.

The final interface and interaction set

The final interface and interaction set

World first AI on event sensors enabling spatial gestures

Event sensors are small cameras which sense movement and output a single image of the movement. Event sensors run at super low power, unlocking potential use of hand tracking in small form factor wearables.

Research Achievements

Research Achievements

Research Achievements

  • Understand the current market and industry roadmap

  • Discover pain points with current solutions

  • Discover common use cases

  • Create personas

  • Understand the key values that our technology must offer to succeed with OEMs

  • Co-ordinate with product and ML engineers on the approach to the project

Design Achievements

Design Achievements

Design Achievements

  • Co-ordinate with ML engineers on perceived technology constraints

  • Conceptualise and deliver interaction design

  • Work with engineering to create prototype interactions

  • Facilitate extensive usability testing

  • UX/UI design of AR smart glasses demo

  • UX design of smart frames demo

Creating an Industry Roadmap

Creating an Industry Roadmap

Putting together a roadmap from multiple sources to understand the industry landscape in the short, medium and long term. This industry roadmap was used to anticipate where the industry is going so we could match our solution to expectations from OEMs with the aim of integrating our solution into their product roadmap.

Putting together a roadmap from multiple sources to understand the industry landscape in the short, medium and long term. This industry roadmap was used to anticipate where the industry is going so we could match our solution to expectations from OEMs with the aim of integrating our solution into their product roadmap.

Hands-on experience with Smart Frames

Hands-on experience with Smart Frames

I spent a week, hands-on with the RayBan Meta smart frames, using them in my daily life to understand use cases, benefits and pain points of the current product offering. This was key to understanding the product category, usability of current smart frames and opportunities for Ultraleap. I wrote up my experience with them for project stakeholders and encouraged others to follow the same process in wearing them in their daily lives and writing up their experience.

I spent a week, hands-on with the RayBan Meta smart frames, using them in my daily life to understand use cases, benefits and pain points of the current product offering. This was key to understanding the product category, usability of current smart frames and opportunities for Ultraleap. I wrote up my experience with them for project stakeholders and encouraged others to follow the same process in wearing them in their daily lives and writing up their experience.

Understanding End Users

Understanding End Users

We needed to understand who we were building for in order to shape the product throughout development. Using publicly available data in Facebook and Reddit communities on specific products, I collected around 300 real user comments that were relevant to use of the products. These comments were then card sorted into themes and used in persona creation.

We needed to understand who we were building for in order to shape the product throughout development. Using publicly available data in Facebook and Reddit communities on specific products, I collected around 300 real user comments that were relevant to use of the products. These comments were then card sorted into themes and used in persona creation.

Persona Creation

Persona Creation

From the data collected, I created 5 personas spanning the breadth of the use cases, pain points and positives discovered in the data collection. These personas spanned the 5 most common use cases of activities, travel, capturing memories, communication, virtual assistant and included the issues and benefits observed in each use case.

From the data collected, I created 5 personas spanning the breadth of the use cases, pain points and positives discovered in the data collection. These personas spanned the 5 most common use cases of activities, travel, capturing memories, communication, virtual assistant and included the issues and benefits observed in each use case.

Defining features for AR Smart Glasses Demo

Defining features for AR Smart Glasses Demo

Aligning closely with with other designers, VPs and product owners, I sought to define the feature set which would be showcased in an AR smart glasses product. Using the personas, individual comments and industry roadmap, we created a map of features which currently exist in AR smart glass products and are promoted in future glasses. Gaining stakeholder alignment with the recommendations, I refined down to a small feature set including the most common use cases that OEMs would look for when integrating interaction solutions in their product roadmap.

Aligning closely with with other designers, VPs and product owners, I sought to define the feature set which would be showcased in an AR smart glasses product. Using the personas, individual comments and industry roadmap, we created a map of features which currently exist in AR smart glass products and are promoted in future glasses. Gaining stakeholder alignment with the recommendations, I refined down to a small feature set including the most common use cases that OEMs would look for when integrating interaction solutions in their product roadmap.

Creation of Design Principles

Creation of Design Principles

As a guiding light to use while workshopping, based on the personas, I created a number of principles which we knew were important to end users. These are the principles which any interaction solutions had to follow if we were to be successful in the market with both OEM integrations and end users.

As a guiding light to use while workshopping, based on the personas, I created a number of principles which we knew were important to end users. These are the principles which any interaction solutions had to follow if we were to be successful in the market with both OEM integrations and end users.

Considering Ergonomics

Considering Ergonomics

When beginning to work on interaction concepts, it was vital to consider ergonomics in the real use cases which the glasses are used for. In general end users are likely to use the product in frequent but short sessions, much like using a smartwatch so interactions had to be easily accessible and not cause any strain over time.

When beginning to work on interaction concepts, it was vital to consider ergonomics in the real use cases which the glasses are used for. In general end users are likely to use the product in frequent but short sessions, much like using a smartwatch so interactions had to be easily accessible and not cause any strain over time.

Creating varied interaction concepts

Creating varied interaction concepts

As a team, involving product designers and engineers, we formulated a number of possible interaction methods and matched those against the personas and design principles. We used the personas rigorously through the process, always considering how each solution would work in different use cases, if the user was trail running or if they were seated in a cramped bus carriage for example. It was important to create varied solutions to refine down further to make sure we hadn't missed a potential solution and design all solutions based on the personas.

As a team, involving product designers and engineers, we formulated a number of possible interaction methods and matched those against the personas and design principles. We used the personas rigorously through the process, always considering how each solution would work in different use cases, if the user was trail running or if they were seated in a cramped bus carriage for example. It was important to create varied solutions to refine down further to make sure we hadn't missed a potential solution and design all solutions based on the personas.

Usability Testing Prototypes

Usability Testing Prototypes

Usability testing was key to understanding the direction which we should take. Working closely with the engineering team and providing clear storyboards of how each concept functioned we quickly built out each concept. Testing quickly, collecting both qualitative and quantitative data, I understood the direction to take and the next steps to improving the promising concepts. Aligning the results with the VP and product owners we set about a continual improvement loop of design iteration and testing which lead us to a final recommended prototype.

Usability testing was key to understanding the direction which we should take. Working closely with the engineering team and providing clear storyboards of how each concept functioned we quickly built out each concept. Testing quickly, collecting both qualitative and quantitative data, I understood the direction to take and the next steps to improving the promising concepts. Aligning the results with the VP and product owners we set about a continual improvement loop of design iteration and testing which lead us to a final recommended prototype.

Milestone

Recommending a final interaction set to key stakeholders

Recommending a final interaction set to key stakeholders

The chosen interaction set was designed to be familiar but new. Drawing upon year of phone use and remote controls, users would be able to simply swipe their thumb across their index finger to navigate, tapping on it to select. These small movements, physical feedback, familiarity, ease and robustness to different environments were the reasons for choosing the interaction set. Presenting to the VP and other key stakeholders, I received buy-in for the tracking research team to begin work in training new ML models to output and enable the interaction set using event sensors.

The chosen interaction set was designed to be familiar but new. Drawing upon year of phone use and remote controls, users would be able to simply swipe their thumb across their index finger to navigate, tapping on it to select. These small movements, physical feedback, familiarity, ease and robustness to different environments were the reasons for choosing the interaction set. Presenting to the VP and other key stakeholders, I received buy-in for the tracking research team to begin work in training new ML models to output and enable the interaction set using event sensors.

Designing AR Smart Glasses Application

Designing AR Smart Glasses Application

Now that the feature and interaction sets were defined, I was able to design the Figma prototype for the AR smart glasses application. Much like the interaction set, it was important for the UI to feel familiar but new if it was to match user expectation when performing the micro-interactions. Starting with low fidelity wireframes, varied concepts were experimented throughout the process, then working upwards into higher fidelity ready for stakeholder alignment and engineering handover.

Now that the feature and interaction sets were defined, I was able to design the Figma prototype for the AR smart glasses application. Much like the interaction set, it was important for the UI to feel familiar but new if it was to match user expectation when performing the micro-interactions. Starting with low fidelity wireframes, varied concepts were experimented throughout the process, then working upwards into higher fidelity ready for stakeholder alignment and engineering handover.

Designing User Flow

Designing User Flow

Critical to the familiarity of the interface was the user flow. Designing in principles of simplicity and consistency, the IA supports users expectations by following familiar patterns.

Critical to the familiarity of the interface was the user flow. Designing in principles of simplicity and consistency, the IA supports users expectations by following familiar patterns.

Milestone

Smart Glasses Demo

Smart Glasses Demo

Working closely with the engineering and tracking research teams we delivered an AR smart glasses demo on the Avegant glasses with fully functioning hand tracked micro gestures using event sensors. Users can double tap to wake, swipe left and right to navigate the UI and double tap to select. Use cases demonstrated were home screen navigation, media, notifications, calendar and weather.

Also created was an integration to Spotify, enabling users to swipe left and right to change tracks and double pinch to play/pause on audio only smart glasses.

Working closely with the engineering and tracking research teams we delivered an AR smart glasses demo on the Avegant glasses with fully functioning hand tracked micro gestures using event sensors. Users can double tap to wake, swipe left and right to navigate the UI and double tap to select. Use cases demonstrated were home screen navigation, media, notifications, calendar and weather.

Also created was an integration to Spotify, enabling users to swipe left and right to change tracks and double pinch to play/pause on audio only smart glasses.

Usability testing and robustness validation

Usability testing and robustness validation

Once the interaction set was delivered by tracking research, we tested them on the smart frames demo, and I facilitated usability testing of the Avegant AR smart glasses demo. Testing led to a number of design changes which were key to forming the final product and understanding improvements which would need to be made to the gesture set which was fed back and aligned with the tracking research team for their future roadmap.

Once the interaction set was delivered by tracking research, we tested them on the smart frames demo, and I facilitated usability testing of the Avegant AR smart glasses demo. Testing led to a number of design changes which were key to forming the final product and understanding improvements which would need to be made to the gesture set which was fed back and aligned with the tracking research team for their future roadmap.

Research Roadmap

Research Roadmap

Following Project Helios' debut at AWE US, a research roadmap needed to be drawn detailing activities to be undertaken to enable the company to continually improve the product and shorten the iteration loop.

Following Project Helios' debut at AWE US, a research roadmap needed to be drawn detailing activities to be undertaken to enable the company to continually improve the product and shorten the iteration loop.