Altered Reality, The Technology Behind Virtual, Augmented and Mixed Realities
Sandy Ressler (NIST)
USPTO Patent Examiner Training
May 2024
DISCLAIMER: Any mention of commercial products is for informational purposes only and does not imply endorsement by NIST.
This slide deck: https://slides.com/sressler/arvrxr
PTO
- https://threejs.org/examples/?q=webgpu#webgpu_cubemap_adjustments
- https://engagevr.io/
- https://culturalheritageimaging.org/
Just your typical Patent Examiner room ;-)
You
give me 5 funny attributed quotes about virtual or augmented reality
ChatGPT
-
"Virtual Reality: Where you can experience the thrill of skydiving without the inconvenient possibility of actual death." - Unknown
-
"Augmented Reality: Because reality itself is just not enough anymore. Now, you can add dragons!" - Unknown
-
"Virtual Reality: The only place where you can wake up from a nightmare and realize you're still in one." - Unknown
-
"In Virtual Reality, you can be anyone you want, which is great news for anyone who's really tired of being themselves." - Unknown
-
"Augmented Reality: Making it socially acceptable to chase imaginary creatures across the street since the 21st century." - Unknown
Topics
- Virtual Reality
- technology
- applications
- Augmented (Mixed) Reality
- technology
- applications
- Standards
- That Metaverse thing
-
- digital twins
- An Integration Case Study (Apple Vision Pro)
First a little history
Ivan Sutherland
(WAYYYYYYY ahead of his time)
1963
60 FREAKING YEARS AGO!!!
Jaron Lanier coined "VR"
1987
NASA Jet Propulsion Labs early pioneering (training work) 1990
What is Virtual Reality?
strap a screen to your face
Virtual Reality
(characteristics)
- Immersive
- Compelling
- Isolating
- Issues with "simulator sickness"
- time limited (only wear that screen for so long)
- high resolution screen
- head tracking
- hand tracking (sometime with extra controller devices) for interaction
- spatial audio
- eye tracking, also for interaction (gaze&pinch)
Integrated UI
- OLD: WIMP (Windows, Icons, Menu, Pointers)
- "drag n drop"
- desktop metaphor
- NEW:
- eye tracking
- gesture input
- 3D immersive display
- "gaze n pinch"
Virtual Reality
Applications
Virtual Reality
Applications
- training
- therapy
- data visualization
- virtual real estate
- exercise
Virtual Reality
Training
Virtual Reality
Physical Therapy
Virtual Reality
Treatment of Phobias
Virtual Reality
Virtual Real Estate
Virtual Reality
Exercise
Virtual Reality
Exercise - Beat Saber
Virtual Reality
Fire Research: 360 degree videos
Virtual Reality
Fire Research: 360 degree videos
Virtual Reality
(significant problems)
- simulator sickness
- duration issues
- comfort - fit/weight
- form factor
Augmented Reality
"Augmented reality: because reality wasn't confusing enough on its own." - Unknown
Augmented (mixed) Reality
- Keeps you in the "real world"
- Mobile phone is intro device
- ubiquity of phones
- many compelling use cases
- navigation
- language translation
- already highly popular (SnapChat lens)
- Interact with real objects
- tactile doesn't need to be simulated (it's real)
Augmented (mixed) Reality
- When using an HMD (Head Mounted Display) "video pass through" used as substitute for a see-through display
- Meta Quest 3
- Apple Vision Pro
- Varjo
- Virtual objects seamlessly integration with the "real world".
Augmented (mixed) Reality
Icons floating in real world space
Augmented Reality
Applications
https://medium.com/@sressler/googles-live-view-the-first-useful-augmented-reality-application-91525209743f
Augmented Reality
Navigation Aid
Augmented Reality
Training
Augmented Reality
Training
Augmented Reality
Language Translation
Augmented Reality
Shopping
Augmented Reality
- Visual fidelity
- Lag/virtual shaking-swimming
- Heat dissipation - power requirements
- Eye/Gaze tracking UI
- when I get tired gaze is off
- Eye glass form factor - holy grail (coming!)
significant remaining problems
Augmented Reality
eye glasses are coming
Networking and Connectivity (Multiuser)
-
Shared office
-
Concert
-
Sports Game
-
Parties
Metaverse Adjacent
Networking
Shared Office. Meta Horizon Workrooms
Networking
Teleprescence
Google Project Starline
Networking
Concert
Networking
Sports
Multiuser
Serious Games
NPSNET demo (1995)
"Standards are like toothbrushes: everybody agrees we need them, but nobody wants to use someone else's." ... Grace Hopper
Standards
make the world interoperable
Internet - Networking Standards
that turned into something big
- TCP/IP layered protocol
- RFC (request for comment)(https://www.ietf.org/standards/rfcs/)
- was not obvious..remember Xerox ring network or proprietary networks on SGI for multiuser interaction (flight game)
3D File Formats
- glTF www.khronos.org
- fast optimized for real time, lots of use in wide range of companies
- USD https://openusd.org/release/index.html
- selected by Apple and Nvidia as "metaverse" format..from Pixar
- X3D https://www.web3d.org/x3d/what-x3d
- successor to VRML, ISO standard, used a lot in academia
Device Standards
- OpenXR
- WebXR
- write application once and deliver on multiple hardware platforms
Interaction Standards
Accessibility
- Secion 508 section508.gov
- W3C has a group WAI (Web Accessibility Initiative) https://www.w3.org/WAI/
USA Standards Strategy
U.S. Government National Standards Strategy for Critical and Emerging Technology
That "Metaverse" thing
“The story so far:
In the beginning the Universe was created.
This has made a lot of people very angry and been widely regarded as a bad move.”
― Douglas Adams, The Restaurant at the End of the Universe
Metaverse
don't worry nobody really knows what it is
Metaverse Standards Forum: https://metaverse-standards.org/ www.metaverse
IEEE: https://metaverse.ieee.org/
ITU-T Metaverse Focus Group: https://www.itu.int/en/ITU-T/focusgroups/mv/Pages/default.aspx
Spatial version of the Internet
Meta (Facebook)
Apple
Microsoft
Mike Zyda (USC)
proposed Metaverse University
https://www.youtube.com/watch?v=3zkZOBU5Pcs
Metaverse
Use Cases: digital twins
https://www.nationalacademies.org/event/42596_05-2024_ethical-needs-and-questions-for-digital-twins
Metaverse
virtual real estate
Metaverse
Podcasts
Metaverse
Integrating and Interoperable (star trek rodenburry archive)
Web Based: aframe.js
Photogrammetry
NeRF - NVidia Instant NeRF
SnapChat Lens Creation
Minecraft: 141 million active monthly users 2021
Fortnite: 250 million monthly active users 2023
User Generated Content
big business
virtually unlimited changes (mods)
constrained but extensive customization
- text to image generators
- DallE
- midjourney
- text to 3D objects
- Point-e (openai)
- text to VR environment
- blockadelabs.com
- text to video
- runwayml.com
this will be out of date by the time you read this..the field is changing at explosive speed!!
AI image generator tools
"NFTs have single-handedly turned the internet into the world's most expensive flea market." - Seth Meyers
Cryptocurrency - Web3
Blockchain
Virtual Economies
NFTs
Cryptocurrency has only existed since approximately 2009/2010 with the creation of Bitcoin. The explosive growth of the value of a "coin" fueled surge of interest and investments. The key innovation of bitcoin is the "blockchain". A blockchain is a digital ledger. It keeps track of all transactions on the "chain"
Cryptocurrency - Web3
blockchain = distributed ledger
DAPPs
decentralized applications
Create keys (a pair is generated, one public, one private.
Public Key Cryptography
foundational technology
Virtual Economies
- Second Life (linden dollar)
- https://en.wikipedia.org/wiki/Economy_of_Second_Life
- Game gold
"Virtual currencies: because why not trade your hard-earned real money for some pixels and in-game perks?" - Anonymous
digital assets
NFTs
smart contracts
https://www.rendertoken.com/
NIST blockchain report: https://www.nist.gov/blockchain
Blockchain
is a distributed ledger
a "chain" of blocks
Fungible versus Nonfungible
NFTs exist and are associate with a particular "chain". Most common is ETH (Ethereum).
You mine it you own it (or you buy it).
Mining an NFT is really the act of digitally signing some item.
March 2021
NFT: Non Fungible Token
provenance and scarecity
Integration Case Study
Apple Vision Pro
Apple Vision Pro
- Eye Tracking
- Hand/Gesture Tracking
- Immersive Display
- Immersive Audio
- Head Tracking
new paradigm: gaze & pinch
Integration Example (AR)
Apple Vision Pro
- eye fatigue: gaze is off
- arm fatigue "I pinched why don't you make the selection"
- Immersive Display - the passthrough gets fuzzy
new paradigm: gaze & pinch
Problems - It's not all rainbows and unicorns
Summary
- Technology is evolving FAST!
- Virtual Reality ..applications
- Augmented Reality .. useful applications
- Standards
- Metaverse
- Integration is getting there
- This talk is obsolete already
Reference URLs
- Udio.com (AI tool to make songs "Fortress of Rights")
- Open.ai (chatGPT) illustration of room of patent examiners
- https://www.forbes.com/sites/forbesfinancecouncil/2023/04/11/convergence-of-web3-ai-and-metaverse-navigating-the-great-reset-for-investors/?sh=6cf4b0fe4e7e
- https://www.indigo9digital.com/blog/how-six-leading-retailers-use-augmented-reality-apps-to-disrupt-the-shopping-experience
- https://medium.com/@julesurbach/what-is-the-metaverse-really-57c30562c237
Template
Teleprescence
Something
- Unity
- Unreal
- Omniverse
That's All!
Note the spherical projection surface
