Altered Reality, The Technology Behind Virtual, Augmented and Mixed Realities
Sandy Ressler (NIST)
USPTO Patent Examiner Training
May 2024
DISCLAIMER: Any mention of commercial products is for informational purposes only and does not imply endorsement by NIST.
This slide deck: https://slides.com/sressler/arvrxr
Just your typical Patent Examiner room ;-)
Topics
- Virtual Reality
- technology
- applications
- Augmented (Mixed) Reality
- technology
- applications
- Standards
- That Metaverse thing
-
- digital twins
- An Integration Case Study (Apple Vision Pro)
First a little history
Ivan Sutherland
(WAYYYYYYY ahead of his time)
1963
60 FREAKING YEARS AGO!!!
Jaron Lanier coined "VR"
1987
NASA Jet Propulsion Labs early pioneering (training work) 1990
What is Virtual Reality?
strap a screen to your face
Virtual Reality
(characteristics)
- Immersive
- Compelling
- Isolating
- Issues with "simulator sickness"
- time limited (only wear that screen for so long)
- high resolution screen
- head tracking
- hand tracking (sometime with extra controller devices) for interaction
- spatial audio
- eye tracking, also for interaction (gaze&pinch)
Integrated UI
- OLD: WIMP (Windows, Icons, Menu, Pointers)
- "drag n drop"
- desktop metaphor
- NEW:
- eye tracking
- gesture input
- 3D immersive display
- "gaze n pinch"
Virtual Reality
Applications
Virtual Reality
Training
Virtual Reality
Physical Therapy
Virtual Reality
Exercise
Virtual Reality
Treatment of Phobias
Virtual Reality
Exercise - Beat Saber
Virtual Reality
Virtual Real Estate
Virtual Reality
(significant problems)
- simulator sickness
- duration issues
- comfort - fit/weight
- form factor
Augmented Reality
"Augmented reality: because reality wasn't confusing enough on its own." - Unknown
Augmented (mixed) Reality
- Keeps you in the "real world"
- Mobile phone is intro device
- ubiquity of phones
- many compelling use cases
- navigation
- language translation
- already highly popular (SnapChat lens)
- Interact with real objects
- tactile doesn't need to be simulated (it's real)
Augmented (mixed) Reality
- When using an HMD (Head Mounted Display) "video pass through" used as substitute for a see-through display
- Meta Quest 3
- Apple Vision Pro
- Varjo
- Virtual objects seamlessly integration with the "real world".
Augmented (mixed) Reality
Icons floating in real world space
Augmented Reality
Applications
https://medium.com/@sressler/googles-live-view-the-first-useful-augmented-reality-application-91525209743f
Augmented Reality
Navigation Aid
Augmented Reality
Training
Augmented Reality
Language Translation
Augmented Reality
Shopping
Augmented Reality
- Visual fidelity
- Lag/virtual shaking-swimming
- Heat dissipation - power requirements
- Eye/Gaze tracking UI
- when I get tired gaze is off
- Eye glass form factor - holy grail (coming!)
significant remaining problems
Augmented Reality
eye glasses are coming
Networking
Concert
Networking
Sports
NPSNET demo (1995)
"Standards are like toothbrushes: everybody agrees we need them, but nobody wants to use someone else's." ... Grace Hopper
Standards
make the world interoperable
Internet - Networking Standards
that turned into something big
- TCP/IP layered protocol
- RFC (request for comment)(https://www.ietf.org/standards/rfcs/)
- was not obvious..remember Xerox ring network or proprietary networks on SGI for multiuser interaction (flight game)
3D File Formats
- glTF www.khronos.org
- fast optimized for real time, lots of use in wide range of companies
- USD https://openusd.org/release/index.html
- selected by Apple and Nvidia as "metaverse" format..from Pixar
- X3D https://www.web3d.org/x3d/what-x3d
- successor to VRML, ISO standard, used a lot in academia
Device Standards
- OpenXR
- WebXR
- write application once and deliver on multiple hardware platforms
Interaction Standards
Accessibility
- Secion 508 section508.gov
- W3C has a group WAI (Web Accessibility Initiative) https://www.w3.org/WAI/
USA Standards Strategy
U.S. Government National Standards Strategy for Critical and Emerging Technology
That "Metaverse" thing
“The story so far:
In the beginning the Universe was created.
This has made a lot of people very angry and been widely regarded as a bad move.”
― Douglas Adams, The Restaurant at the End of the Universe
Metaverse
don't worry nobody really knows what it is
Metaverse Standards Forum: https://metaverse-standards.org/ www.metaverse
IEEE: https://metaverse.ieee.org/
ITU-T Metaverse Focus Group: https://www.itu.int/en/ITU-T/focusgroups/mv/Pages/default.aspx
Spatial version of the Internet
Meta (Facebook)
Apple
Microsoft
Mike Zyda (USC)
proposed Metaverse University
https://www.youtube.com/watch?v=3zkZOBU5Pcs
Metaverse
Use Cases: digital twins
https://www.nationalacademies.org/event/42596_05-2024_ethical-needs-and-questions-for-digital-twins
Metaverse
virtual real estate
Metaverse
Podcasts
Integration Case Study
Apple Vision Pro
Apple Vision Pro
- Eye Tracking
- Hand/Gesture Tracking
- Immersive Display
- Immersive Audio
- Head Tracking
new paradigm: gaze & pinch
Integration Example (AR)
Apple Vision Pro
- eye fatigue: gaze is off
- arm fatigue "I pinched why don't you make the selection"
- Immersive Display - the passthrough gets fuzzy
new paradigm: gaze & pinch
Problems - It's not all rainbows and unicorns
Summary
- Technology is evolving FAST!
- Virtual Reality ..applications
- Augmented Reality .. useful applications
- Standards
- Metaverse
- Integration is getting there
- This talk is obsolete already
Reference URLs
- Udio.com (AI tool to make songs "Fortress of Rights")
- Open.ai (chatGPT) illustration of room of patent examiners
- https://www.forbes.com/sites/forbesfinancecouncil/2023/04/11/convergence-of-web3-ai-and-metaverse-navigating-the-great-reset-for-investors/?sh=6cf4b0fe4e7e
- https://www.indigo9digital.com/blog/how-six-leading-retailers-use-augmented-reality-apps-to-disrupt-the-shopping-experience
- https://medium.com/@julesurbach/what-is-the-metaverse-really-57c30562c237
That's All!
Note the spherical projection surface
