Altered Reality, The Technology Behind Virtual, Augmented and Mixed Realities
Sandy Ressler (NIST)
USPTO Patent Examiner Training
May 2024
DISCLAIMER: Any mention of commercial products is for informational purposes only and does not imply endorsement by NIST.
This slide deck: https://slides.com/sressler/arvrxr
Just your typical Patent Examiner room ;-)
Topics
- Virtual Reality
- technology
- applications
- Augmented (Mixed) Reality
- technology
- applications
- Standards
- That Metaverse thing
-
- digital twins
- An Integration Case Study (Apple Vision Pro)
First a little history
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/1591921/image2.jpg)
Ivan Sutherland
(WAYYYYYYY ahead of his time)
1963
60 FREAKING YEARS AGO!!!
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/1603471/head.jpg)
Jaron Lanier coined "VR"
1987
NASA Jet Propulsion Labs early pioneering (training work) 1990
![](https://www.nasa.gov/sites/default/files/1990_a_new_continent.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/11189688/NAsa__into_Vr_from_Nasa.gov_-1168x778.webp)
What is Virtual Reality?
strap a screen to your face
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/11216195/597D3769-AB3C-44E3-94C4-770A451C5C56_1_102_o.jpeg)
Virtual Reality
(characteristics)
- Immersive
- Compelling
- Isolating
- Issues with "simulator sickness"
- time limited (only wear that screen for so long)
- high resolution screen
- head tracking
- hand tracking (sometime with extra controller devices) for interaction
- spatial audio
- eye tracking, also for interaction (gaze&pinch)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10263259/SandyMan_an_icon_for_head_position_tracking_0_4ee4c46f-be4b-4d01-9147-a0ce3cf8d8ab.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10263192/SandyMan_an_icon_for_eye_tracking_0_dce23853-b2c3-4f9e-9d13-f0e67eeefefd.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10263161/SandyMan_an_icon_for_hand_tracking_0_36f6ae79-cdbb-4735-95bb-edfea2c68e5b.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10264808/SandyMan_an_icon_for_sound_all_around_the_ear_of_a_head_0_f6cafc31-3f90-46dc-a540-5782acb1be50.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10265071/SandyMan_icon_for_head_mounted_display_not_on_a_head_as_cartoon_c8051e86-94a7-4792-8c9e-0c4fcb2718cf.png)
Integrated UI
- OLD: WIMP (Windows, Icons, Menu, Pointers)
- "drag n drop"
- desktop metaphor
- NEW:
- eye tracking
- gesture input
- 3D immersive display
- "gaze n pinch"
Virtual Reality
Applications
Virtual Reality
Training
Virtual Reality
Physical Therapy
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10390081/IMG_7194-rotated.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10390125/vr-exercise-treadmill.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10390122/maxresdefault-518915939.jpg)
Virtual Reality
Exercise
Virtual Reality
Treatment of Phobias
Virtual Reality
Exercise - Beat Saber
Virtual Reality
Virtual Real Estate
Virtual Reality
(significant problems)
- simulator sickness
- duration issues
- comfort - fit/weight
- form factor
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10370739/SandyMan_extreme_closeup_old_woman_head_focused_on_right_eye_we_749e1635-754b-4361-914e-23bdd0ca8be8.png)
Augmented Reality
"Augmented reality: because reality wasn't confusing enough on its own." - Unknown
Augmented (mixed) Reality
- Keeps you in the "real world"
- Mobile phone is intro device
- ubiquity of phones
- many compelling use cases
- navigation
- language translation
- already highly popular (SnapChat lens)
- Interact with real objects
- tactile doesn't need to be simulated (it's real)
Augmented (mixed) Reality
- When using an HMD (Head Mounted Display) "video pass through" used as substitute for a see-through display
- Meta Quest 3
- Apple Vision Pro
- Varjo
- Virtual objects seamlessly integration with the "real world".
Augmented (mixed) Reality
Icons floating in real world space
Augmented Reality
Applications
https://medium.com/@sressler/googles-live-view-the-first-useful-augmented-reality-application-91525209743f
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10265083/google-live-view-google-maps.jpg)
Augmented Reality
Navigation Aid
Augmented Reality
Training
Augmented Reality
Language Translation
Augmented Reality
Shopping
Augmented Reality
- Visual fidelity
- Lag/virtual shaking-swimming
- Heat dissipation - power requirements
- Eye/Gaze tracking UI
- when I get tired gaze is off
- Eye glass form factor - holy grail (coming!)
significant remaining problems
Augmented Reality
eye glasses are coming
Networking
Concert
Networking
Sports
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10456698/screenshot_35.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10456741/screenshot_36.png)
NPSNET demo (1995)
"Standards are like toothbrushes: everybody agrees we need them, but nobody wants to use someone else's." ... Grace Hopper
Standards
make the world interoperable
Internet - Networking Standards
that turned into something big
- TCP/IP layered protocol
- RFC (request for comment)(https://www.ietf.org/standards/rfcs/)
- was not obvious..remember Xerox ring network or proprietary networks on SGI for multiuser interaction (flight game)
3D File Formats
- glTF www.khronos.org
- fast optimized for real time, lots of use in wide range of companies
- USD https://openusd.org/release/index.html
- selected by Apple and Nvidia as "metaverse" format..from Pixar
- X3D https://www.web3d.org/x3d/what-x3d
- successor to VRML, ISO standard, used a lot in academia
Device Standards
- OpenXR
- WebXR
- write application once and deliver on multiple hardware platforms
Interaction Standards
Accessibility
- Secion 508 section508.gov
- W3C has a group WAI (Web Accessibility Initiative) https://www.w3.org/WAI/
USA Standards Strategy
U.S. Government National Standards Strategy for Critical and Emerging Technology
That "Metaverse" thing
“The story so far:
In the beginning the Universe was created.
This has made a lot of people very angry and been widely regarded as a bad move.”
― Douglas Adams, The Restaurant at the End of the Universe
Metaverse
don't worry nobody really knows what it is
Metaverse Standards Forum: https://metaverse-standards.org/ www.metaverse
IEEE: https://metaverse.ieee.org/
ITU-T Metaverse Focus Group: https://www.itu.int/en/ITU-T/focusgroups/mv/Pages/default.aspx
Spatial version of the Internet
Meta (Facebook)
Apple
Microsoft
Mike Zyda (USC)
proposed Metaverse University
https://www.youtube.com/watch?v=3zkZOBU5Pcs
Metaverse
Use Cases: digital twins
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/11272149/screenshot_67.png)
https://www.nationalacademies.org/event/42596_05-2024_ethical-needs-and-questions-for-digital-twins
Metaverse
virtual real estate
Metaverse
Podcasts
Integration Case Study
Apple Vision Pro
Apple Vision Pro
- Eye Tracking
- Hand/Gesture Tracking
- Immersive Display
- Immersive Audio
- Head Tracking
new paradigm: gaze & pinch
Integration Example (AR)
Apple Vision Pro
- eye fatigue: gaze is off
- arm fatigue "I pinched why don't you make the selection"
- Immersive Display - the passthrough gets fuzzy
new paradigm: gaze & pinch
Problems - It's not all rainbows and unicorns
Summary
- Technology is evolving FAST!
- Virtual Reality ..applications
- Augmented Reality .. useful applications
- Standards
- Metaverse
- Integration is getting there
- This talk is obsolete already
Reference URLs
- Udio.com (AI tool to make songs "Fortress of Rights")
- Open.ai (chatGPT) illustration of room of patent examiners
- https://www.forbes.com/sites/forbesfinancecouncil/2023/04/11/convergence-of-web3-ai-and-metaverse-navigating-the-great-reset-for-investors/?sh=6cf4b0fe4e7e
- https://www.indigo9digital.com/blog/how-six-leading-retailers-use-augmented-reality-apps-to-disrupt-the-shopping-experience
- https://medium.com/@julesurbach/what-is-the-metaverse-really-57c30562c237
That's All!
Note the spherical projection surface
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/11272636/CC95BAA7-C203-42F9-A551-56329AEC1F2A_1_105_c.jpeg)
Link:
ARVRXR
By Sandy Ressler
ARVRXR
- 160