On Augmented Reality

Half a decade ago I tried on a first generation AR headset, an early Hololens prototype. A low-resolution screen (the size of a deck of cards at arms length) floated in my view, revealing a holographic supercar. Leaning in and out towards the engine block you could block out everything but the tiny screen and believe for a second there was an object there that no one else could see… but the technology was too limited to really sell the illusion for more than a moment.

This year there are several competitive AR/VR headsets in the gaming/entertainment space, and Apple released the much-vaunted Vision Pro. While it is still early, we can be sure that there are lower-priced and better-featured Vision devices maturing within the unmarked rooms in Cupertino. We've been promised virtual reality for decades now. It remains to be seen whether this the moment.

I've been thinking about what this technology means as a designer, user, and as a father. Here are some random thoughts.

Just Add Magic

I'm excited for the design challenges that come with dimensionality. Since before the mother of all demos designers have been emulating three dimensional reality with visual tricks: shadows, hues, layers.. We mock and mimic physics in order to help the user understand the purpose and affordances of the untouchable object hiding behind the sheet of glass.

Designers struggle to make things simple. They take an experience or object and attempt to add magic.

As an example, a designer making a computer solitaire game spends immense time crafting a 2d emulation of interacting with cards. He aims to convince the user that giving up the physical interactivity and engagement is worth computer system handling shuffling and the rules. Screens and mice instead of tables and laughs… but with the convenience of digital magic.

A virtual reality system simply hands you the cards. No persuasion needed. The designer then focuses on adding magic to the recognizable physical card system. We see something recognizable, but magical. Virtual objects can be familiar to the user yet surprise them by transcending normal physics and limitations:

  • A rolodex that sorts by recency and transforms the avatar of person as you start the call.
  • Recipe books that filter to your guest's dietary restrictions.
  • Roads that emphasize your destination and preferred fast food joints.
  • Sticky notes on a wall accessible by teams across continents and timezones
  • Tools that are simple and easy to understand and use and yet take no weight to carry.1

Imagine: instead of an amorphous screenslab your working tools appear to yourself and to others as what they are: a camera, a writing tablet, an actual stack of todo stickies, a deck of contact cards.

Spatial Computing/VR/AR pushes digital product design up out of two dimensions and into the existing design domains of Service Design or Environmental Design… but with magic.

It's exciting.

Where There is Illusion, People Change.

That same day I tried a (then) brand new Vive headset. After playing some games for a few minutes I took off the headset to hand to another person. The jarring "teleportation" sensation of being ripped from one world into the real was profound. It has to be experienced, not explained. It seems like newer headsets provide even more immersive experiences.

Location is a powerful force on our behavioral systems. Having a device that can teleport us to another location is powerful tool for perhaps creating creative-only spaces… I easily see myself creating a "location" in which I only write. (I presently use sound and music to try and trick my brain, but I'd pay money for a virtual writing cabin.)

We aren't the first to realize this power.

The Diamond Age by Neal Stephenson imagines a magic book given to young Nell. It can teach and guide her to become her full princess potential… and nudges her towards a particular interesting path of life. Ender's Game by Orson Scott Card imagines a similar device: a VR game so real that it teaches character and empathy. Both stories are hopeful. The tools are written by well-meaning people who want the best for the kids exposed to the technology.

I gotta wonder though… who gets to write the magic tools to teach my children? Who decides what is "interesting" in the Primer? Who decides what is good character and healthy empathy?

The same tools that we can use to create positive self-directed behavioral change can be wielded against us.

I have a hard time believing that the same corporations that make their money off of measuring what we click, what causes our scrolling to linger, what words we respond to with action, and measure their employees engagement, lines of code typed, will ignore this new pool of data. The whole apparatus of surveillance capitalism will benefit more than the headset user. We are placing a device on our heads that records our subconscious eye saccades and facial ticks… not metrics associated with our attention, but our actual attention.

We will likely be Pavlov'd in even more subtle and darker UX ways than ever before. Having a strong code of ethics as a designer is going to be very important… we will have more power than ever before.

Outsourcing Personality

In Charlie Stross's novel Accelerando2, a novel telling the story of the singularity, the tech-forward Manfred has his goggles ripped by a street thief named Jack. Manfred suddenly has no clue what he was about to do:

"I'm Manfred – Manfred. My memory. What's happened to my memory?" Elderly Malaysian tourists point at him from the open top deck of a passing bus. He burns with a sense of horrified urgency. I was going somewhere, he recalls. What was I doing? It was amazingly important, he thinks, but he can't remember what exactly it was. He was going to see someone about – it's on the tip of his tongue –

Manfred has outsourced so much of his thinking and memory to the computer on his face that his youthful assailant Jack is actually able to conduct his meeting for Manfred… transferring Manfred's goggles with their tasks and agents is basically transferring Manfred's goals and context to a new body.

In a very real sense, the glasses are Manfred, regardless of the identity of the soft machine with its eyeballs behind the lenses. And it is a very puzzled Manfred who picks himself up and, with a curious vacancy in his head – except for a hesitant request for information about accessories for Russian army boots – dusts himself off and heads for his meeting on the other side of town.

As someone who regularly writes my thoughts out in a digital format, who uses sophisticated calendars and task management to be a good steward… this is a little terrifying. The futuristic ability to carry and see through our technology rather than just consult3 it is powerful and scary.

I'll be clutching my bullet journal a little more tightly.

I believe that virtual environments provide an opportunity for product designers and UX designers to stop designing screens and to focus on service design experiences. What is this person trying to do? Where are they standing? Who are they with? How could we enrich a physical tool with raw magic to make a person more focused, creative, and joyful in their work? It is a very exciting time to be interested in building tools to serve other people.

I believe that wearable technology with high fidelity of data on the human experience will enable profit-minded corporations to read and interpret emotion and subconscious action, combined with emotionally resonant and realistic experiences and memories4. Unprecedented power to enrich and enslave. The ability to actually change what a person sees in the world.

It's a weird time to be in technology. Buckle up.


  1. The magical item of Bag of Holding in real world use. 

  2. Accelerando is a brilliant book that requires some strong content warnings… so you are warned. 

  3. Consulting technology as an advisor feels like a relatively healthy mental model. I'm afraid I do much more than consult at this point, but maybe that's a north start to drive towards. 

  4. There's a reason that in D&D illusion magic is both under-appreciated and arguably the most powerful of all. Evil creatures in countless stories can shape-shift and change your perception of reality… Steven King's IT, JRR Tolkien's Anatar, Vampires, Demons, superheroes, The Matrix… 


🔖