Altered Reality, The Technology Behind Virtual, Augmented and Mixed Realities



Sandy Ressler (NIST)

USPTO Patent Examiner Training

May 2024

DISCLAIMER: Any mention of commercial products is for informational purposes only and does not imply endorsement by NIST.



Just your typical Patent Examiner room ;-)


give me 5 funny attributed quotes about virtual or augmented reality


  1. "Virtual Reality: Where you can experience the thrill of skydiving without the inconvenient possibility of actual death." - Unknown

  2. "Augmented Reality: Because reality itself is just not enough anymore. Now, you can add dragons!" - Unknown

  3. "Virtual Reality: The only place where you can wake up from a nightmare and realize you're still in one." - Unknown

  4. "In Virtual Reality, you can be anyone you want, which is great news for anyone who's really tired of being themselves." - Unknown

  5. "Augmented Reality: Making it socially acceptable to chase imaginary creatures across the street since the 21st century." - Unknown






  • Virtual Reality
    • technology
    • applications
  • Augmented (Mixed) Reality
    • technology
    • applications
  • Standards
  • That Metaverse thing
    • digital twins
  • An Integration Case Study (Apple Vision Pro)

First a little history

Ivan Sutherland

(WAYYYYYYY ahead of his time)



Jaron Lanier coined "VR"


NASA Jet Propulsion Labs early pioneering (training work) 1990

What is Virtual Reality?

strap a screen to your face

Virtual Reality


  • Immersive
  • Compelling
  • Isolating
  • Issues with "simulator sickness"
  • time limited (only wear that screen for so long)
  • high resolution screen
  • head tracking
  • hand tracking (sometime with extra controller devices) for interaction
  • spatial audio
  • eye tracking, also for interaction (gaze&pinch)

Integrated UI


  • OLD: WIMP (Windows, Icons, Menu, Pointers)
    • "drag n drop"
    • desktop metaphor
  • NEW: 
    • eye tracking
    • gesture input
    • 3D immersive display
    • "gaze n pinch"

Virtual Reality


Virtual Reality


  • training
  • therapy
  • data visualization
  • virtual real estate
  • exercise

Virtual Reality


Virtual Reality

Physical Therapy

Virtual Reality

Treatment of Phobias

Virtual Reality

Virtual Real Estate

Virtual Reality


Virtual Reality

Exercise - Beat Saber

Virtual Reality

Fire Research: 360 degree videos

Virtual Reality

Fire Research: 360 degree videos

Virtual Reality

(significant problems)

  • simulator sickness
  • duration issues
  • comfort - fit/weight
  • form factor

Augmented Reality

"Augmented reality: because reality wasn't confusing enough on its own." - Unknown

Augmented (mixed) Reality

  • Keeps you in the "real world"
  • Mobile phone is intro device
    • ubiquity of phones
    • many compelling use cases
      • navigation
      • language translation
      • already highly popular (SnapChat lens)
  • Interact with real objects
    • tactile doesn't need to be simulated (it's real)

Augmented (mixed) Reality

  • When using an HMD (Head Mounted Display) "video pass through" used as substitute for a see-through display
    • Meta Quest 3
    • Apple Vision Pro
    • Varjo
  • Virtual objects seamlessly integration with the "real world".

Augmented (mixed) Reality

Icons floating in real world space

Augmented Reality


Augmented Reality

Navigation Aid

Augmented Reality


Augmented Reality


Augmented Reality

Language Translation

Augmented Reality


Augmented Reality

  • Visual fidelity
  • Lag/virtual shaking-swimming
  • Heat dissipation - power requirements
  • Eye/Gaze tracking UI 
    • when I get tired gaze is off
  • Eye glass form factor - holy grail (coming!)

significant remaining problems

Augmented Reality

eye glasses are coming

Networking and Connectivity (Multiuser)

  • Shared office

  • Concert

  • Sports Game

  • Parties

Metaverse Adjacent


Shared Office.  Meta Horizon Workrooms



Google Project Starline






Serious Games

NPSNET demo (1995)

"Standards are like toothbrushes: everybody agrees we need them, but nobody wants to use someone else's." ... Grace Hopper



make the world interoperable

Internet - Networking Standards

that turned into something big



  • TCP/IP layered protocol
  •  RFC (request for comment)(
  • was not obvious..remember Xerox ring network or proprietary networks on SGI for multiuser interaction (flight game)

3D File Formats


Device Standards

  • OpenXR
  • WebXR
  • write application once and deliver on multiple hardware platforms

Interaction Standards



  • Secion 508
  • W3C has a group WAI (Web Accessibility Initiative)

USA Standards Strategy


U.S. Government National Standards Strategy for Critical and Emerging Technology

That "Metaverse" thing

“The story so far:
In the beginning the Universe was created.
This has made a lot of people very angry and been widely regarded as a bad move.”

― Douglas Adams, The Restaurant at the End of the Universe


don't worry nobody really knows what it is

Metaverse Standards Forum: www.metaverse


ITU-T Metaverse Focus Group:

Spatial version of the Internet

Meta (Facebook)




Mike Zyda (USC)

proposed Metaverse University


Use Cases: digital twins


virtual real estate




Integrating and Interoperable (star trek rodenburry archive)

Web Based: aframe.js


NeRF - NVidia Instant NeRF

SnapChat Lens Creation

Minecraft: 141 million active monthly users 2021

Fortnite: 250 million monthly active users 2023


User Generated Content

big business

virtually unlimited changes (mods)

constrained but extensive customization

  • text to image generators
    • DallE
    • midjourney
  • text to 3D objects
    • Point-e (openai)
  • text to VR environment
  • text to video

this will be out of date by the time you read this..the field is changing at explosive speed!!

AI image generator tools


"NFTs have single-handedly turned the internet into the world's most expensive flea market." - Seth Meyers

Cryptocurrency - Web3



Virtual Economies


Cryptocurrency has only existed since approximately 2009/2010 with the creation of Bitcoin. The explosive growth of the value of a "coin" fueled surge of interest and investments. The key innovation of bitcoin is the "blockchain". A blockchain is a digital ledger. It keeps track of all transactions on the "chain"

Cryptocurrency - Web3

blockchain = distributed ledger


decentralized applications

Create keys (a pair is generated, one public, one private.


Public Key Cryptography

foundational technology

Virtual Economies

  • Second Life (linden dollar)
  • Game gold

"Virtual currencies: because why not trade your hard-earned real money for some pixels and in-game perks?" - Anonymous

digital assets


smart contracts

NIST blockchain report:


is a distributed ledger

a "chain" of blocks

Fungible versus Nonfungible

NFTs exist and are associate with a particular "chain". Most common is ETH (Ethereum). 

You mine it you own it (or you buy it).

Mining an NFT is really the act of digitally signing some item. 

March 2021

NFT: Non Fungible Token

provenance and scarecity

Integration Case Study

Apple Vision Pro

Apple Vision Pro

  • Eye Tracking
  • Hand/Gesture Tracking
  • Immersive Display
  • Immersive Audio
  • Head Tracking

new paradigm: gaze & pinch

Integration Example (AR)

Apple Vision Pro

  • eye fatigue: gaze is off
  • arm fatigue "I pinched why don't you make the selection"
  • Immersive Display - the passthrough gets fuzzy

new paradigm: gaze & pinch

Problems - It's not all rainbows and unicorns


  • Technology is evolving FAST!
  • Virtual Reality ..applications
  • Augmented Reality .. useful applications
  • Standards
  • Metaverse
  • Integration is getting there
  • This talk is obsolete already 

Reference URLs




  • Unity
  • Unreal
  • Omniverse

That's All!

Note the spherical projection surface


Copy of ARVRXR

By Sandy Ressler