Virtual Reality the Metaverse and All that Reality Stuff
Sandy Ressler (NIST)
USPTO Patent Examiner Training
May 2023
DISCLAIMER: Any mention of commercial products is for informational purposes only and does not imply endorsement by NIST.
This slide deck: https://slides.com/sressler/deck-1557bb
Just your typical Patent Examiner room ;-)
"Just when I discovered the meaning of life, they changed it." ― George Carlin, on constantly evolving technology
“I've come up with a set of rules that describe our reactions to technologies:
1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.
2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.
3. Anything invented after you're thirty-five is against the natural order of things.”
― Douglas Adams, The Salmon of Doubt
Topics (human created)
- Virtual Reality
- applications
- Augmented Reality
- mixed reality
- applications
- Standards
- Tools
- That Metaverse thing
- digital twins
- Blockchain/NFTs
Topics (ChatGPT)
-
Virtual Reality (VR) and Augmented Reality (AR
-
3D Graphics and Spatial Computing
-
Networking and Connectivity
-
Blockchain and Cryptocurrencies
-
Virtual Economies
-
Social Platforms and Interoperability
-
Gaming and Entertainment
- User Generated Content (UGC) and Customization
- Artificial Intelligence and Machine Learning
Not bad but it missed standards and AI and ML can be applied to anything, I'll rate it a B.
prompt:
give me a bulleted list of the main components of the metaverse
First a little history
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/1591921/image2.jpg)
Ivan Sutherland
(WAYYYYYYY ahead of his time)
1963
60 FREAKING YEARS AGO!!!
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/1603471/head.jpg)
Jaron Lanier coined "VR"
1987
NASA Jet Propulsion Labs early pioneering (training work) 1990
![](https://www.nasa.gov/sites/default/files/1990_a_new_continent.jpg)
What is Virtual Reality?
strap a screen to your face
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10263004/SandyMan_a_few_people_in_office_working_with_VR_headsets_on_car_e6693efe-71e2-47b8-b46e-fad397ebc911.png)
Virtual Reality
(characteristics)
- Immersive
- Compelling
- Isolating
- Issues with "simulator sickness"
- time limited (only wear that screen for so long)
- high resolution screen
- head tracking
- hand tracking (sometime with extra controller devices) for interaction
- spatial audio
- eye tracking, also for interaction
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10263259/SandyMan_an_icon_for_head_position_tracking_0_4ee4c46f-be4b-4d01-9147-a0ce3cf8d8ab.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10263192/SandyMan_an_icon_for_eye_tracking_0_dce23853-b2c3-4f9e-9d13-f0e67eeefefd.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10263161/SandyMan_an_icon_for_hand_tracking_0_36f6ae79-cdbb-4735-95bb-edfea2c68e5b.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10264808/SandyMan_an_icon_for_sound_all_around_the_ear_of_a_head_0_f6cafc31-3f90-46dc-a540-5782acb1be50.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10265071/SandyMan_icon_for_head_mounted_display_not_on_a_head_as_cartoon_c8051e86-94a7-4792-8c9e-0c4fcb2718cf.png)
As an aside
- spatial audio
- head tracked airpod
- eye tracking
- contact lens screens (mojo)
Virtual Reality
Applications
- training
- therapy
- data visualization
- virtual real estate
- exercise
Virtual Reality
Training
Virtual Reality
Physical Therapy
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10390081/IMG_7194-rotated.jpg)
Virtual Reality
Treatment of Phobias
Virtual Reality
Virtual Real Estate
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10390125/vr-exercise-treadmill.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10390122/maxresdefault-518915939.jpg)
Virtual Reality
Exercise
Virtual Reality
Exercise - Beat Saber
Virtual Reality
Fire Research: 360 degree videos
Virtual Reality
Fire Research: 360 degree videos
Virtual Reality
(significant problems)
- simulator sickness
- duration issues
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10370739/SandyMan_extreme_closeup_old_woman_head_focused_on_right_eye_we_749e1635-754b-4361-914e-23bdd0ca8be8.png)
Augmented Reality
"Augmented reality: because reality wasn't confusing enough on its own." - Unknown
Augmented (mixed) Reality
- Keeps you in the "real world"
- Mobile phone is intro device
- ubiquity of phones
- many compelling use cases
- navigation
- language translation
- already highly popular (SnapChat lens)
- Interact with real objects
- tactile doesn't need to be simulated (it's real)
https://medium.com/@sressler/googles-live-view-the-first-useful-augmented-reality-application-91525209743f
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10265083/google-live-view-google-maps.jpg)
Augmented Reality
Navigation Aid
Augmented Reality
Training
Augmented Reality
Language Translation
Augmented Reality
Shopping
Immersive Reality
Data Visualization
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10390268/BillDiagram2.png)
Augmented Reality
- Visual fidelity
- Lag/virtual shaking-swimming
- Heat dissipation - power requirements
- Eye/Gaze tracking UI
- Eye glass form factor - holy grail (coming!)
significant remaining problems
Networking and Connectivity (Multiuser)
"The Internet: where men are men, women are men, and children are undercover FBI agents." - Anonymous
Networking and Connectivity (Multiuser)
-
Shared office
-
Concert
-
Sports Game
-
Parties
Networking
Shared Office. Meta Horizon Workrooms
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10391742/screenshot_29.png)
Networking
Teleprescence
Google Project Starline
Networking
Concert
Networking
Sports
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10456698/screenshot_35.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10456741/screenshot_36.png)
Multiuser
Serious Games
NPSNET demo (1995)
Tools
"Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live."
- John Woods
Tools
- General Purpose Authoring: Unreal, Unity, Omniverse
- Web based: aframe.js
- Specialized: LensStudio, 360 degree video
- Photogrammetry: 3D scenes from 2D photographs
NeRF - Neural Radiance Field
In summary, NeRF is a method that uses machine learning to represent and render 3D scenes from 2D images. By training a neural network on a set of input images, NeRF can generate realistic, high-quality views of the scene from new perspectives. - ChatGPT
"Unity is the virtual reality tool that lets you create your own universe – just make sure you don't get lost in it while debugging." - Unknown
General Purpose Creation Tools
- Unity: https://unity.com/
- Unreal: https://www.unrealengine.com/en-US
- Omniverse: https://www.nvidia.com/en-us/omniverse/
All are highly capable and robust tools - as always the choice is "it depends" on your application requirements.
Web Based: aframe.js
Photogrammetry
NeRF - NVidia Instant NeRF
SnapChat Lens Creation
- text to image generators
- DallE
- midjourney
- text to 3D objects
- Point-e (openai)
- text to VR environment
- blockadelabs.com
- text to video
- runwayml.com
this will be out of date by the time you read this..the field is changing at explosive speed!!
AI image generator tools
Minecraft: 141 million active monthly users 2021
Fortnite: 250 million monthly active users 2023
User Generated Content
big business
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10450717/screenshot_31.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10450720/th-3349638942.jpg)
virtually unlimited changes (mods)
constrained but extensive customization
"The nice thing about standards is that there are so many to choose from." ... Andrew S. Tanenbaum
3D objects - file formats
communications - Internet TCP/IP
interaction - UX
Standards
make the world interoperable
3D File Formats
- glTF www.khronos.org
- fast optimized for real time, lots of use in wide range of companies
- USD https://openusd.org/release/index.html
- selected by Apple and Nvidia as "metaverse" format..from Pixar
- X3D https://www.web3d.org/x3d/what-x3d
- successor to VRML, ISO standard, used a lot in academia
Internet - Networking Standards
that turned into something big
- TCP/IP layered protocol
- RFC (request for comment)(https://www.ietf.org/standards/rfcs/)
- was not obvious..remember Xerox ring network or proprietary networks on SGI for multiuser interaction (flight game)
Interaction Standards
Accessibility
- Secion 508 section508.gov
- W3C has a group WAI (Web Accessibility Initiative) https://www.w3.org/WAI/
USA Standards Strategy
brand spanking new (may 2023)
U.S. Government National Standards Strategy for Critical and Emerging Technology
"NFTs have single-handedly turned the internet into the world's most expensive flea market." - Seth Meyers
Cryptocurrency - Web3
Blockchain
Virtual Economies
NFTs
Cryptocurrency has only existed since approximately 2009/2010 with the creation of Bitcoin. The explosive growth of the value of a "coin" fueled surge of interest and investments. The key innovation of bitcoin is the "blockchain". A blockchain is a digital ledger. It keeps track of all transactions on the "chain"
Cryptocurrency - Web3
blockchain = distributed ledger
DAPPs
decentralized applications
Create keys (a pair is generated, one public, one private.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10422066/screenshot_34.png)
Public Key Cryptography
foundational technology
Virtual Economies
- Second Life (linden dollar)
- https://en.wikipedia.org/wiki/Economy_of_Second_Life
- Game gold
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10359048/second-life-business-week.jpg)
"Virtual currencies: because why not trade your hard-earned real money for some pixels and in-game perks?" - Anonymous
digital assets
NFTs
smart contracts
https://www.rendertoken.com/
NIST blockchain report: https://www.nist.gov/blockchain
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10450961/blockchain.png)
Blockchain
is a distributed ledger
a "chain" of blocks
Fungible versus Nonfungible
NFTs exist and are associate with a particular "chain". Most common is ETH (Ethereum).
You mine it you own it (or you buy it).
Mining an NFT is really the act of digitally signing some item.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10450944/screenshot_33.png)
March 2021
NFT: Non Fungible Token
provenance and scarecity
That "Metaverse" thing
“The story so far:
In the beginning the Universe was created.
This has made a lot of people very angry and been widely regarded as a bad move.”
― Douglas Adams, The Restaurant at the End of the Universe
Mike Zyda (USC)
proposed Metaverse University
https://www.youtube.com/watch?v=3zkZOBU5Pcs
That "Metaverse" thing
don't worry nobody really knows what it is
Metaverse Standards Forum: https://metaverse-standards.org/ www.metaverse
IEEE: https://metaverse.ieee.org/
ITU-T Metaverse Focus Group: https://www.itu.int/en/ITU-T/focusgroups/mv/Pages/default.aspx
Spatial version of the Internet
Meta (Facebook)
Apple
Microsoft
Metaverse
virtual real estate
Metaverse
Use Cases: digital twins
Metaverse
Podcasts
Metaverse
Integrating and Interoperable (star trek rodenburry archive)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/2854/images/10414090/screenshot_30.png)
Summary
- Technology is evolving FAST!
- Virtual Reality ..applications
- Augmented Reality .. useful applications
- Multiuser
- Standards
- Tools
- Crypto - Blockchain - Distributed Ledger
- Metaverse
- This talk is obsolete already
Notes
- https://www.forbes.com/sites/forbesfinancecouncil/2023/04/11/convergence-of-web3-ai-and-metaverse-navigating-the-great-reset-for-investors/?sh=6cf4b0fe4e7e
- Bullet Two
- Bullet Three
- https://medium.com/@julesurbach/what-is-the-metaverse-really-57c30562c237
Template
Teleprescence
Something
- Unity
- Unreal
- Omniverse
Link:
VR the Metaverse and More
By Sandy Ressler
VR the Metaverse and More
- 452