Presentation: Multi-Modal Input Design for Magic Leap
Share this on:
What You’ll Learn
-
Learn special design considerations for developing for Magic Leap One.
-
Understand how Magic Leap is furthering and combining input modalities to improve the user experience in mixed reality.
-
Learn about some of the use cases mixed reality can be used to address.
Abstract
As wearable spatial computing devices become more of a reality, new opportunities for interaction between humans and computers arise. Magic Leap embraces a wide range of inputs including hand and eye tracking, speech, a wireless 6DoF controller, and support for external peripherals. Learn what types of new input modalities are coming online and how they can be used and combined in different ways to surpass existing approaches in terms of throughput, discoverability, accessibility, and prediction with stories and examples from Magic Leap's Interaction Lab.
What’s the focus of the work you do today?
I’m the design lead for the Interaction Lab at Magic Leap.
We are part of the Product Org and formerly a content-focused wing of Systems. Our original charter included a mix of doing first turn-ons and system tests for new platform features. Over time we have shifted more toward being an incubation team, rapid prototyping experiences and looking for new ways of interacting with our platform using the many input modalities we support. One of our customers is developers, so we build tools, iterate on our SDK, and publish sample projects with documentation as part of “Magic Kit.”
We build a lot of proof of concepts to explore the possibilities of the medium. You will find us focused on questions like: what kinds of experiences and new products can we enable with our tools and features, and are our features and KPIs good enough to be able to build the kind of products we think are valuable?
We're a hybrid team composed primarily of former game developers. We’re customers of our own tech and, in the process, also try to make sure it’s usable for external developers. We’re very design and product focused and have a mix of engineers, artists, and designers who work very closely to discover the potential of our product and feed our findings to other teams and partners. Our motto is “Fail Fast!”
What's the motivation for your talk?
This year we finally pulled the curtain back on Magic Leap with Magic Leap One, a dev kit and our first public offering. We want to spread the word and get people aware of our platform and see how it's different from other mixed reality and augmented reality systems.
We want to show that as we move further into the spatial computing era, we can democratize computing even further, making it more accessible by embracing natural inputs and user expectations, including for people with disabilities. For instance, we can show how when you start combining input modalities in different ways, similar to how people use a keyboard and mouse together, you can unlock a lot of new potential use cases, particularly when done in a wearable untethered mobile form factor. We can combine your gaze with gestures and speech or other modalities to accomplish things not possible with existing mediums today.
What topics are you going to cover specific to multimodal input design for Magic Leap?
Over time the trajectory of HCI has been to make computers conform more to humans. Early computers were very hard to use. You had to be an expert. One topic we’ll address is how we can improve existing interactions with computers through increasing intuitiveness and breadth of inputs, making functionality more natural and thus discoverable. Another is, because Magic Leap is wearable and mobile, it opens up new possibilities for tasks focused around spatial design, either in context of real physical spaces or involving simulating real objects at scale. There’s a lot of use there for a variety of industries including interior or industrial design, but also training, education, and more.
What is a technology problem that keeps you up at night?
So much of our existing technology is used for escapism. I’m particularly aware of this, having spent the majority of my career making video games prior to joining Magic Leap. When you force people to focus on a screen, I think that’s what naturally happens. Escapism certainly has its uses, but too much can cause harm as well. I think that we’re seeing humanity having an extinction psychology moment right now where things have gotten really bad, so we’re doubling down on our escapism. I came to Magic Leap, because I think there’s great value in creating a new type of computing platform that keeps us grounded in the real world and lets us use it as the foundation for our digital experiences rather than taking humans even further out of their surroundings. Now, maybe more than ever before, we need to find ways to get people back grounded in reality. I’m excited for the potential we have to do that with this new medium.
Similar Talks
Tracks
Monday, 5 November
-
Microservices / Serverless Patterns & Practices
Evolving, observing, persisting, and building modern microservices
-
Practices of DevOps & Lean Thinking
Practical approaches using DevOps & Lean Thinking
-
JavaScript & Web Tech
Beyond JavaScript in the Browser. Exploring WebAssembly, Electron, & Modern Frameworks
-
Modern CS in the Real World
Thoughts pushing software forward, including consensus, CRDT's, formal methods, & probabilistic programming
-
Modern Operating Systems
Applied, practical, & real-world deep-dive into industry adoption of OS, containers and virtualization, including Linux on Windows, LinuxKit, and Unikernels
-
Optimizing You: Human Skills for Individuals
Better teams start with a better self. Learn practical skills for IC
Tuesday, 6 November
-
Architectures You've Always Wondered About
Next-gen architectures from the most admired companies in software, such as Netflix, Google, Facebook, Twitter, & more
-
21st Century Languages
Lessons learned from languages like Rust, Go-lang, Swift, Kotlin, and more.
-
Emerging Trends in Data Engineering
Showcasing DataEng tech and highlighting the strengths of each in real-world applications.
-
Bare Knuckle Performance
Killing latency and getting the most out of your hardware
-
Socially Conscious Software
Building socially responsible software that protects users privacy & safety
-
Delivering on the Promise of Containers
Runtime containers, libraries, and services that power microservices
Wednesday, 7 November
-
Applied AI & Machine Learning
Applied machine learning lessons for SWEs, including tech around TensorFlow, TPUs, Keras, PyTorch, & more
-
Production Readiness: Building Resilient Systems
More than just building software, building deployable production ready software
-
Developer Experience: Level up your Engineering Effectiveness
Improving the end to end developer experience - design, dev, test, deploy, operate/understand.
-
Security: Lessons Attacking & Defending
Security from the defender's AND the attacker's point of view
-
Future of Human Computer Interaction
IoT, voice, mobile: Interfaces pushing the boundary of what we consider to be the interface
-
Enterprise Languages
Workhorse languages found in modern enterprises. Expect Java, .NET, & Node in this track
