Presentation: Multi-Modal Input Design for Magic Leap

Track: Future of Human Computer Interaction

Location: Pacific BC

Duration: 10:35am - 11:25am

Day of week: Wednesday

Level: Intermediate

Persona: Architect, Developer, Front-end Developer, General Software, Mobile Developer

Share this on:

What You’ll Learn

  1. Learn special design considerations for developing for Magic Leap One.

  2. Understand how Magic Leap is furthering and combining input modalities to improve the user experience in mixed reality.

  3. Learn about some of the use cases mixed reality can be used to address.


As wearable spatial computing devices become more of a reality, new opportunities for interaction between humans and computers arise. Magic Leap embraces a wide range of inputs including hand and eye tracking, speech, a wireless 6DoF controller, and support for external peripherals. Learn what types of new input modalities are coming online and how they can be used and combined in different ways to surpass existing approaches in terms of throughput, discoverability, accessibility, and prediction with stories and examples from Magic Leap's Interaction Lab.


What’s the focus of the work you do today?


I’m the design lead for the Interaction Lab at Magic Leap.

We are part of the Product Org and formerly a content-focused wing of Systems. Our original charter included a mix of doing first turn-ons and system tests for new platform features. Over time we have shifted more toward being an incubation team, rapid prototyping experiences and looking for new ways of interacting with our platform using the many input modalities we support. One of our customers is developers, so we build tools, iterate on our SDK, and publish sample projects with documentation as part of “Magic Kit.”

We build a lot of proof of concepts to explore the possibilities of the medium. You will find us focused on questions like: what kinds of experiences and new products can we enable with our tools and features, and are our features and KPIs good enough to be able to build the kind of products we think are valuable?

We're a hybrid team composed primarily of former game developers. We’re customers of our own tech and, in the process, also try to make sure it’s usable for external developers. We’re very design and product focused and have a mix of engineers, artists, and designers who work very closely to discover the potential of our product and feed our findings to other teams and partners. Our motto is “Fail Fast!”




What's the motivation for your talk?


This year we finally pulled the curtain back on Magic Leap with Magic Leap One, a dev kit and our first public offering. We want to spread the word and get people aware of our platform and see how it's different from other mixed reality and augmented reality systems.

We want to show that as we move further into the spatial computing era, we can democratize computing even further, making it more accessible by embracing natural inputs and user expectations, including for people with disabilities. For instance, we can show how when you start combining input modalities in different ways, similar to how people use a keyboard and mouse together, you can unlock a lot of new potential use cases, particularly when done in a wearable untethered mobile form factor. We can combine your gaze with gestures and speech or other modalities to accomplish things not possible with existing mediums today.


What topics are you going to cover specific to multimodal input design for Magic Leap?


Over time the trajectory of HCI has been to make computers conform more to humans. Early computers were very hard to use. You had to be an expert. One topic we’ll address is how we can improve existing interactions with computers through increasing intuitiveness and breadth of inputs, making functionality more natural and thus discoverable. Another is, because Magic Leap is wearable and mobile, it opens up new possibilities for tasks focused around spatial design, either in context of real physical spaces or involving simulating real objects at scale. There’s a lot of use there for a variety of industries including interior or industrial design, but also training, education, and more.


What is a technology problem that keeps you up at night?


So much of our existing technology is used for escapism. I’m particularly aware of this, having spent the majority of my career making video games prior to joining Magic Leap. When you force people to focus on a screen, I think that’s what naturally happens. Escapism certainly has its uses, but too much can cause harm as well. I think that we’re seeing humanity having an extinction psychology moment right now where things have gotten really bad, so we’re doubling down on our escapism. I came to Magic Leap, because I think there’s great value in creating a new type of computing platform that keeps us grounded in the real world and lets us use it as the foundation for our digital experiences rather than taking humans even further out of their surroundings. Now, maybe more than ever before, we need to find ways to get people back grounded in reality. I’m excited for the potential we have to do that with this new medium.

Speaker: Colman Bryant

Mixed Reality Game and Product Designer @MagicLeap

Colman Bryant is currently Lead Designer for the Interaction Lab at Magic Leap, an internal incubation team that explores interaction paradigms and prototypes new products and use cases for Magic Leap's Mixed Reality platform. In a past life, he worked in the game industry as Design Director at Hi-Rez Studios, where he lead design for hit service-based Free-to-Play games including Smite and Jetpack Fighter.

Find Colman Bryant at

Similar Talks

Software Engineer @PayPal
CTO at Propel Inc, building @FreshEBT
Co-Founder & Chief Operating Officer @npmjs
Software Developer and Cloud Specialist @Checkfront
Software Engineer @coinbase
Senior Software Engineer @stitchfix


Monday, 5 November

Tuesday, 6 November

Wednesday, 7 November

The all-new QCon app!

Available on iOS and Android

The new QCon app helps you make the most of your conference experience. Easily browse and follow the conference schedule, star the talks you want to attend, and keep tabs on your personal itinerary. Download the app now for free on iOS and Android.