Altered Reality, The Technology Behind Virtual, Augmented and Mixed Realities

 

 

Sandy Ressler (NIST)

USPTO Patent Examiner Training

May 2024

DISCLAIMER: Any mention of commercial products is for informational purposes only and does not imply endorsement by NIST.

PTO

  • https://threejs.org/examples/?q=webgpu#webgpu_cubemap_adjustments
  • https://engagevr.io/
  • https://culturalheritageimaging.org/

Just your typical Patent Examiner room ;-)

You

give me 5 funny attributed quotes about virtual or augmented reality

ChatGPT

  1. "Virtual Reality: Where you can experience the thrill of skydiving without the inconvenient possibility of actual death." - Unknown

  2. "Augmented Reality: Because reality itself is just not enough anymore. Now, you can add dragons!" - Unknown

  3. "Virtual Reality: The only place where you can wake up from a nightmare and realize you're still in one." - Unknown

  4. "In Virtual Reality, you can be anyone you want, which is great news for anyone who's really tired of being themselves." - Unknown

  5. "Augmented Reality: Making it socially acceptable to chase imaginary creatures across the street since the 21st century." - Unknown

 

 

 

 

Topics

  • Virtual Reality
    • technology
    • applications
  • Augmented (Mixed) Reality
    • technology
    • applications
  • Standards
  • That Metaverse thing
    • digital twins
  • An Integration Case Study (Apple Vision Pro)

First a little history

Ivan Sutherland

(WAYYYYYYY ahead of his time)

1963

60 FREAKING YEARS AGO!!!

Jaron Lanier coined "VR"

1987

NASA Jet Propulsion Labs early pioneering (training work) 1990

What is Virtual Reality?

strap a screen to your face

Virtual Reality

(characteristics)

  • Immersive
  • Compelling
  • Isolating
  • Issues with "simulator sickness"
  • time limited (only wear that screen for so long)
  • high resolution screen
  • head tracking
  • hand tracking (sometime with extra controller devices) for interaction
  • spatial audio
  • eye tracking, also for interaction (gaze&pinch)

Integrated UI

 

  • OLD: WIMP (Windows, Icons, Menu, Pointers)
    • "drag n drop"
    • desktop metaphor
  • NEW: 
    • eye tracking
    • gesture input
    • 3D immersive display
    • "gaze n pinch"

Virtual Reality

Applications

Virtual Reality

Applications

  • training
  • therapy
  • data visualization
  • virtual real estate
  • exercise

Virtual Reality

Training

Virtual Reality

Physical Therapy

Virtual Reality

Treatment of Phobias

Virtual Reality

Virtual Real Estate

Virtual Reality

Exercise

Virtual Reality

Exercise - Beat Saber

Virtual Reality

Fire Research: 360 degree videos

Virtual Reality

Fire Research: 360 degree videos

Virtual Reality

(significant problems)

  • simulator sickness
  • duration issues
  • comfort - fit/weight
  • form factor

Augmented Reality

"Augmented reality: because reality wasn't confusing enough on its own." - Unknown

Augmented (mixed) Reality

  • Keeps you in the "real world"
  • Mobile phone is intro device
    • ubiquity of phones
    • many compelling use cases
      • navigation
      • language translation
      • already highly popular (SnapChat lens)
  • Interact with real objects
    • tactile doesn't need to be simulated (it's real)

Augmented (mixed) Reality

  • When using an HMD (Head Mounted Display) "video pass through" used as substitute for a see-through display
    • Meta Quest 3
    • Apple Vision Pro
    • Varjo
  • Virtual objects seamlessly integration with the "real world".

Augmented (mixed) Reality

Icons floating in real world space

Augmented Reality

Applications

https://medium.com/@sressler/googles-live-view-the-first-useful-augmented-reality-application-91525209743f

Augmented Reality

Navigation Aid

Augmented Reality

Training

Augmented Reality

Training

Augmented Reality

Language Translation

Augmented Reality

Shopping

Augmented Reality

  • Visual fidelity
  • Lag/virtual shaking-swimming
  • Heat dissipation - power requirements
  • Eye/Gaze tracking UI 
    • when I get tired gaze is off
  • Eye glass form factor - holy grail (coming!)

significant remaining problems

Augmented Reality

eye glasses are coming

Networking and Connectivity (Multiuser)

  • Shared office

  • Concert

  • Sports Game

  • Parties

Metaverse Adjacent

Networking

Shared Office.  Meta Horizon Workrooms

Networking

Teleprescence

Google Project Starline

Networking

Concert

Networking

Sports

Multiuser

Serious Games

NPSNET demo (1995)

"Standards are like toothbrushes: everybody agrees we need them, but nobody wants to use someone else's." ... Grace Hopper

 

Standards

make the world interoperable

Internet - Networking Standards

that turned into something big

 

 

  • TCP/IP layered protocol
  •  RFC (request for comment)(https://www.ietf.org/standards/rfcs/)
  • was not obvious..remember Xerox ring network or proprietary networks on SGI for multiuser interaction (flight game)

3D File Formats

 

Device Standards

  • OpenXR
  • WebXR
  • write application once and deliver on multiple hardware platforms

Interaction Standards

Accessibility

 

  • Secion 508 section508.gov
  • W3C has a group WAI (Web Accessibility Initiative) https://www.w3.org/WAI/

USA Standards Strategy

 

U.S. Government National Standards Strategy for Critical and Emerging Technology

That "Metaverse" thing

“The story so far:
In the beginning the Universe was created.
This has made a lot of people very angry and been widely regarded as a bad move.”

― Douglas Adams, The Restaurant at the End of the Universe

Metaverse

don't worry nobody really knows what it is

Metaverse Standards Forum: https://metaverse-standards.org/ www.metaverse

IEEE: https://metaverse.ieee.org/

ITU-T Metaverse Focus Group: https://www.itu.int/en/ITU-T/focusgroups/mv/Pages/default.aspx

Spatial version of the Internet

Meta (Facebook)

Google

Apple

Microsoft

Mike Zyda (USC)

proposed Metaverse University

https://www.youtube.com/watch?v=3zkZOBU5Pcs

Metaverse

Use Cases: digital twins

https://www.nationalacademies.org/event/42596_05-2024_ethical-needs-and-questions-for-digital-twins

Metaverse 

virtual real estate

Metaverse

Podcasts

Metaverse

Integrating and Interoperable (star trek rodenburry archive)

Web Based: aframe.js

Photogrammetry

NeRF - NVidia Instant NeRF

SnapChat Lens Creation

Minecraft: 141 million active monthly users 2021

Fortnite: 250 million monthly active users 2023

 

User Generated Content

big business

virtually unlimited changes (mods)

constrained but extensive customization

  • text to image generators
    • DallE
    • midjourney
  • text to 3D objects
    • Point-e (openai)
  • text to VR environment
    • blockadelabs.com
  • text to video
    • runwayml.com

this will be out of date by the time you read this..the field is changing at explosive speed!!

AI image generator tools

 

"NFTs have single-handedly turned the internet into the world's most expensive flea market." - Seth Meyers

Cryptocurrency - Web3

 

Blockchain

Virtual Economies

NFTs

Cryptocurrency has only existed since approximately 2009/2010 with the creation of Bitcoin. The explosive growth of the value of a "coin" fueled surge of interest and investments. The key innovation of bitcoin is the "blockchain". A blockchain is a digital ledger. It keeps track of all transactions on the "chain"

Cryptocurrency - Web3

blockchain = distributed ledger

DAPPs

decentralized applications

Create keys (a pair is generated, one public, one private.

 

Public Key Cryptography

foundational technology

Virtual Economies

  • Second Life (linden dollar)
    • https://en.wikipedia.org/wiki/Economy_of_Second_Life
  • Game gold
  •  

"Virtual currencies: because why not trade your hard-earned real money for some pixels and in-game perks?" - Anonymous

digital assets

NFTs

smart contracts

https://www.rendertoken.com/

NIST blockchain report: https://www.nist.gov/blockchain

Blockchain

is a distributed ledger

a "chain" of blocks

Fungible versus Nonfungible

NFTs exist and are associate with a particular "chain". Most common is ETH (Ethereum). 

You mine it you own it (or you buy it).

Mining an NFT is really the act of digitally signing some item. 

March 2021

NFT: Non Fungible Token

provenance and scarecity

Integration Case Study

Apple Vision Pro

Apple Vision Pro

  • Eye Tracking
  • Hand/Gesture Tracking
  • Immersive Display
  • Immersive Audio
  • Head Tracking

new paradigm: gaze & pinch

Integration Example (AR)

Apple Vision Pro

  • eye fatigue: gaze is off
  • arm fatigue "I pinched why don't you make the selection"
  • Immersive Display - the passthrough gets fuzzy

new paradigm: gaze & pinch

Problems - It's not all rainbows and unicorns

Summary

  • Technology is evolving FAST!
  • Virtual Reality ..applications
  • Augmented Reality .. useful applications
  • Standards
  • Metaverse
  • Integration is getting there
  • This talk is obsolete already 

Reference URLs

Template

Teleprescence

Something

  • Unity
  • Unreal
  • Omniverse

That's All!

Note the spherical projection surface

Link:

Copy of ARVRXR

By Sandy Ressler