Altered Reality, The Technology Behind Virtual, Augmented and Mixed Realities
Sandy Ressler (NIST)
USPTO Patent Examiner Training
May 2024
DISCLAIMER: Any mention of commercial products is for informational purposes only and does not imply endorsement by NIST.
This slide deck: https://slides.com/sressler/arvrxr
PTO
- https://threejs.org/examples/?q=webgpu#webgpu_cubemap_adjustments
- https://engagevr.io/
- https://culturalheritageimaging.org/
Just your typical Patent Examiner room ;-)
You
give me 5 funny attributed quotes about virtual or augmented reality
ChatGPT
-
"Virtual Reality: Where you can experience the thrill of skydiving without the inconvenient possibility of actual death." - Unknown
-
"Augmented Reality: Because reality itself is just not enough anymore. Now, you can add dragons!" - Unknown
-
"Virtual Reality: The only place where you can wake up from a nightmare and realize you're still in one." - Unknown
-
"In Virtual Reality, you can be anyone you want, which is great news for anyone who's really tired of being themselves." - Unknown
-
"Augmented Reality: Making it socially acceptable to chase imaginary creatures across the street since the 21st century." - Unknown
Topics
- Virtual Reality
- technology
- applications
- Augmented (Mixed) Reality
- technology
- applications
- Standards
- That Metaverse thing
-
- digital twins
- An Integration Case Study (Apple Vision Pro)
First a little history
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/1591921/image2.jpg)
Ivan Sutherland
(WAYYYYYYY ahead of his time)
1963
60 FREAKING YEARS AGO!!!
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/1603471/head.jpg)
Jaron Lanier coined "VR"
1987
NASA Jet Propulsion Labs early pioneering (training work) 1990
![](https://www.nasa.gov/sites/default/files/1990_a_new_continent.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/11189688/NAsa__into_Vr_from_Nasa.gov_-1168x778.webp)
What is Virtual Reality?
strap a screen to your face
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/11216195/597D3769-AB3C-44E3-94C4-770A451C5C56_1_102_o.jpeg)
Virtual Reality
(characteristics)
- Immersive
- Compelling
- Isolating
- Issues with "simulator sickness"
- time limited (only wear that screen for so long)
- high resolution screen
- head tracking
- hand tracking (sometime with extra controller devices) for interaction
- spatial audio
- eye tracking, also for interaction (gaze&pinch)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10263259/SandyMan_an_icon_for_head_position_tracking_0_4ee4c46f-be4b-4d01-9147-a0ce3cf8d8ab.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10263192/SandyMan_an_icon_for_eye_tracking_0_dce23853-b2c3-4f9e-9d13-f0e67eeefefd.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10263161/SandyMan_an_icon_for_hand_tracking_0_36f6ae79-cdbb-4735-95bb-edfea2c68e5b.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10264808/SandyMan_an_icon_for_sound_all_around_the_ear_of_a_head_0_f6cafc31-3f90-46dc-a540-5782acb1be50.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10265071/SandyMan_icon_for_head_mounted_display_not_on_a_head_as_cartoon_c8051e86-94a7-4792-8c9e-0c4fcb2718cf.png)
Integrated UI
- OLD: WIMP (Windows, Icons, Menu, Pointers)
- "drag n drop"
- desktop metaphor
- NEW:
- eye tracking
- gesture input
- 3D immersive display
- "gaze n pinch"
Virtual Reality
Applications
Virtual Reality
Applications
- training
- therapy
- data visualization
- virtual real estate
- exercise
Virtual Reality
Training
Virtual Reality
Physical Therapy
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10390081/IMG_7194-rotated.jpg)
Virtual Reality
Treatment of Phobias
Virtual Reality
Virtual Real Estate
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10390125/vr-exercise-treadmill.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10390122/maxresdefault-518915939.jpg)
Virtual Reality
Exercise
Virtual Reality
Exercise - Beat Saber
Virtual Reality
Fire Research: 360 degree videos
Virtual Reality
Fire Research: 360 degree videos
Virtual Reality
(significant problems)
- simulator sickness
- duration issues
- comfort - fit/weight
- form factor
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10370739/SandyMan_extreme_closeup_old_woman_head_focused_on_right_eye_we_749e1635-754b-4361-914e-23bdd0ca8be8.png)
Augmented Reality
"Augmented reality: because reality wasn't confusing enough on its own." - Unknown
Augmented (mixed) Reality
- Keeps you in the "real world"
- Mobile phone is intro device
- ubiquity of phones
- many compelling use cases
- navigation
- language translation
- already highly popular (SnapChat lens)
- Interact with real objects
- tactile doesn't need to be simulated (it's real)
Augmented (mixed) Reality
- When using an HMD (Head Mounted Display) "video pass through" used as substitute for a see-through display
- Meta Quest 3
- Apple Vision Pro
- Varjo
- Virtual objects seamlessly integration with the "real world".
Augmented (mixed) Reality
Icons floating in real world space
Augmented Reality
Applications
https://medium.com/@sressler/googles-live-view-the-first-useful-augmented-reality-application-91525209743f
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10265083/google-live-view-google-maps.jpg)
Augmented Reality
Navigation Aid
Augmented Reality
Training
Augmented Reality
Training
Augmented Reality
Language Translation
Augmented Reality
Shopping
Augmented Reality
- Visual fidelity
- Lag/virtual shaking-swimming
- Heat dissipation - power requirements
- Eye/Gaze tracking UI
- when I get tired gaze is off
- Eye glass form factor - holy grail (coming!)
significant remaining problems
Augmented Reality
eye glasses are coming
Networking and Connectivity (Multiuser)
-
Shared office
-
Concert
-
Sports Game
-
Parties
Metaverse Adjacent
Networking
Shared Office. Meta Horizon Workrooms
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10391742/screenshot_29.png)
Networking
Teleprescence
Google Project Starline
Networking
Concert
Networking
Sports
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10456698/screenshot_35.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10456741/screenshot_36.png)
Multiuser
Serious Games
NPSNET demo (1995)
"Standards are like toothbrushes: everybody agrees we need them, but nobody wants to use someone else's." ... Grace Hopper
Standards
make the world interoperable
Internet - Networking Standards
that turned into something big
- TCP/IP layered protocol
- RFC (request for comment)(https://www.ietf.org/standards/rfcs/)
- was not obvious..remember Xerox ring network or proprietary networks on SGI for multiuser interaction (flight game)
3D File Formats
- glTF www.khronos.org
- fast optimized for real time, lots of use in wide range of companies
- USD https://openusd.org/release/index.html
- selected by Apple and Nvidia as "metaverse" format..from Pixar
- X3D https://www.web3d.org/x3d/what-x3d
- successor to VRML, ISO standard, used a lot in academia
Device Standards
- OpenXR
- WebXR
- write application once and deliver on multiple hardware platforms
Interaction Standards
Accessibility
- Secion 508 section508.gov
- W3C has a group WAI (Web Accessibility Initiative) https://www.w3.org/WAI/
USA Standards Strategy
U.S. Government National Standards Strategy for Critical and Emerging Technology
That "Metaverse" thing
“The story so far:
In the beginning the Universe was created.
This has made a lot of people very angry and been widely regarded as a bad move.”
― Douglas Adams, The Restaurant at the End of the Universe
Metaverse
don't worry nobody really knows what it is
Metaverse Standards Forum: https://metaverse-standards.org/ www.metaverse
IEEE: https://metaverse.ieee.org/
ITU-T Metaverse Focus Group: https://www.itu.int/en/ITU-T/focusgroups/mv/Pages/default.aspx
Spatial version of the Internet
Meta (Facebook)
Apple
Microsoft
Mike Zyda (USC)
proposed Metaverse University
https://www.youtube.com/watch?v=3zkZOBU5Pcs
Metaverse
Use Cases: digital twins
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/11272149/screenshot_67.png)
https://www.nationalacademies.org/event/42596_05-2024_ethical-needs-and-questions-for-digital-twins
Metaverse
virtual real estate
Metaverse
Podcasts
Metaverse
Integrating and Interoperable (star trek rodenburry archive)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10414090/screenshot_30.png)
Web Based: aframe.js
Photogrammetry
NeRF - NVidia Instant NeRF
SnapChat Lens Creation
Minecraft: 141 million active monthly users 2021
Fortnite: 250 million monthly active users 2023
User Generated Content
big business
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10450717/screenshot_31.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10450720/th-3349638942.jpg)
virtually unlimited changes (mods)
constrained but extensive customization
- text to image generators
- DallE
- midjourney
- text to 3D objects
- Point-e (openai)
- text to VR environment
- blockadelabs.com
- text to video
- runwayml.com
this will be out of date by the time you read this..the field is changing at explosive speed!!
AI image generator tools
"NFTs have single-handedly turned the internet into the world's most expensive flea market." - Seth Meyers
Cryptocurrency - Web3
Blockchain
Virtual Economies
NFTs
Cryptocurrency has only existed since approximately 2009/2010 with the creation of Bitcoin. The explosive growth of the value of a "coin" fueled surge of interest and investments. The key innovation of bitcoin is the "blockchain". A blockchain is a digital ledger. It keeps track of all transactions on the "chain"
Cryptocurrency - Web3
blockchain = distributed ledger
DAPPs
decentralized applications
Create keys (a pair is generated, one public, one private.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10422066/screenshot_34.png)
Public Key Cryptography
foundational technology
Virtual Economies
- Second Life (linden dollar)
- https://en.wikipedia.org/wiki/Economy_of_Second_Life
- Game gold
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10359048/second-life-business-week.jpg)
"Virtual currencies: because why not trade your hard-earned real money for some pixels and in-game perks?" - Anonymous
digital assets
NFTs
smart contracts
https://www.rendertoken.com/
NIST blockchain report: https://www.nist.gov/blockchain
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10450961/blockchain.png)
Blockchain
is a distributed ledger
a "chain" of blocks
Fungible versus Nonfungible
NFTs exist and are associate with a particular "chain". Most common is ETH (Ethereum).
You mine it you own it (or you buy it).
Mining an NFT is really the act of digitally signing some item.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10450944/screenshot_33.png)
March 2021
NFT: Non Fungible Token
provenance and scarecity
Integration Case Study
Apple Vision Pro
Apple Vision Pro
- Eye Tracking
- Hand/Gesture Tracking
- Immersive Display
- Immersive Audio
- Head Tracking
new paradigm: gaze & pinch
Integration Example (AR)
Apple Vision Pro
- eye fatigue: gaze is off
- arm fatigue "I pinched why don't you make the selection"
- Immersive Display - the passthrough gets fuzzy
new paradigm: gaze & pinch
Problems - It's not all rainbows and unicorns
Summary
- Technology is evolving FAST!
- Virtual Reality ..applications
- Augmented Reality .. useful applications
- Standards
- Metaverse
- Integration is getting there
- This talk is obsolete already
Reference URLs
- Udio.com (AI tool to make songs "Fortress of Rights")
- Open.ai (chatGPT) illustration of room of patent examiners
- https://www.forbes.com/sites/forbesfinancecouncil/2023/04/11/convergence-of-web3-ai-and-metaverse-navigating-the-great-reset-for-investors/?sh=6cf4b0fe4e7e
- https://www.indigo9digital.com/blog/how-six-leading-retailers-use-augmented-reality-apps-to-disrupt-the-shopping-experience
- https://medium.com/@julesurbach/what-is-the-metaverse-really-57c30562c237
Template
Teleprescence
Something
- Unity
- Unreal
- Omniverse
That's All!
Note the spherical projection surface
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/11272636/CC95BAA7-C203-42F9-A551-56329AEC1F2A_1_105_c.jpeg)
Link:
Copy of ARVRXR
By Sandy Ressler
Copy of ARVRXR
- 132