Virtual Reality the Metaverse and All that Reality Stuff
Sandy Ressler (NIST)
USPTO Patent Examiner Training
May 2023
DISCLAIMER: Any mention of commercial products is for informational purposes only and does not imply endorsement by NIST.
This slide deck: https://slides.com/sressler/deck-1557bb
Just your typical Patent Examiner room ;-)
"Just when I discovered the meaning of life, they changed it." ― George Carlin, on constantly evolving technology
“I've come up with a set of rules that describe our reactions to technologies:
1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.
2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.
3. Anything invented after you're thirty-five is against the natural order of things.”
― Douglas Adams, The Salmon of Doubt
Topics (human created)
- Virtual Reality
- applications
- Augmented Reality
- mixed reality
- applications
- Standards
- Tools
- That Metaverse thing
- digital twins
- Blockchain/NFTs
Topics (ChatGPT)
-
Virtual Reality (VR) and Augmented Reality (AR
-
3D Graphics and Spatial Computing
-
Networking and Connectivity
-
Blockchain and Cryptocurrencies
-
Virtual Economies
-
Social Platforms and Interoperability
-
Gaming and Entertainment
- User Generated Content (UGC) and Customization
- Artificial Intelligence and Machine Learning
Not bad but it missed standards and AI and ML can be applied to anything, I'll rate it a B.
prompt:
give me a bulleted list of the main components of the metaverse
First a little history
Ivan Sutherland
(WAYYYYYYY ahead of his time)
1963
60 FREAKING YEARS AGO!!!
Jaron Lanier coined "VR"
1987
NASA Jet Propulsion Labs early pioneering (training work) 1990
What is Virtual Reality?
strap a screen to your face
Virtual Reality
(characteristics)
- Immersive
- Compelling
- Isolating
- Issues with "simulator sickness"
- time limited (only wear that screen for so long)
- high resolution screen
- head tracking
- hand tracking (sometime with extra controller devices) for interaction
- spatial audio
- eye tracking, also for interaction
As an aside
- spatial audio
- head tracked airpod
- eye tracking
- contact lens screens (mojo)
Virtual Reality
Applications
- training
- therapy
- data visualization
- virtual real estate
- exercise
Virtual Reality
Training
Virtual Reality
Physical Therapy
Virtual Reality
Treatment of Phobias
Virtual Reality
Virtual Real Estate
Virtual Reality
Exercise
Virtual Reality
Exercise - Beat Saber
Virtual Reality
Fire Research: 360 degree videos
Virtual Reality
Fire Research: 360 degree videos
Virtual Reality
(significant problems)
- simulator sickness
- duration issues
Augmented Reality
"Augmented reality: because reality wasn't confusing enough on its own." - Unknown
Augmented (mixed) Reality
- Keeps you in the "real world"
- Mobile phone is intro device
- ubiquity of phones
- many compelling use cases
- navigation
- language translation
- already highly popular (SnapChat lens)
- Interact with real objects
- tactile doesn't need to be simulated (it's real)
https://medium.com/@sressler/googles-live-view-the-first-useful-augmented-reality-application-91525209743f
Augmented Reality
Navigation Aid
Augmented Reality
Training
Augmented Reality
Language Translation
Augmented Reality
Shopping
Immersive Reality
Data Visualization
Augmented Reality
- Visual fidelity
- Lag/virtual shaking-swimming
- Heat dissipation - power requirements
- Eye/Gaze tracking UI
- Eye glass form factor - holy grail (coming!)
significant remaining problems
Networking and Connectivity (Multiuser)
"The Internet: where men are men, women are men, and children are undercover FBI agents." - Anonymous
Networking and Connectivity (Multiuser)
-
Shared office
-
Concert
-
Sports Game
-
Parties
Networking
Shared Office. Meta Horizon Workrooms
Networking
Teleprescence
Google Project Starline
Networking
Concert
Networking
Sports
Multiuser
Serious Games
NPSNET demo (1995)
Tools
"Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live."
- John Woods
Tools
- General Purpose Authoring: Unreal, Unity, Omniverse
- Web based: aframe.js
- Specialized: LensStudio, 360 degree video
- Photogrammetry: 3D scenes from 2D photographs
NeRF - Neural Radiance Field
In summary, NeRF is a method that uses machine learning to represent and render 3D scenes from 2D images. By training a neural network on a set of input images, NeRF can generate realistic, high-quality views of the scene from new perspectives. - ChatGPT
"Unity is the virtual reality tool that lets you create your own universe – just make sure you don't get lost in it while debugging." - Unknown
General Purpose Creation Tools
- Unity: https://unity.com/
- Unreal: https://www.unrealengine.com/en-US
- Omniverse: https://www.nvidia.com/en-us/omniverse/
All are highly capable and robust tools - as always the choice is "it depends" on your application requirements.
Web Based: aframe.js
Photogrammetry
NeRF - NVidia Instant NeRF
SnapChat Lens Creation
- text to image generators
- DallE
- midjourney
- text to 3D objects
- Point-e (openai)
- text to VR environment
- blockadelabs.com
- text to video
- runwayml.com
this will be out of date by the time you read this..the field is changing at explosive speed!!
AI image generator tools
Minecraft: 141 million active monthly users 2021
Fortnite: 250 million monthly active users 2023
User Generated Content
big business
virtually unlimited changes (mods)
constrained but extensive customization
"The nice thing about standards is that there are so many to choose from." ... Andrew S. Tanenbaum
3D objects - file formats
communications - Internet TCP/IP
interaction - UX
Standards
make the world interoperable
3D File Formats
- glTF www.khronos.org
- fast optimized for real time, lots of use in wide range of companies
- USD https://openusd.org/release/index.html
- selected by Apple and Nvidia as "metaverse" format..from Pixar
- X3D https://www.web3d.org/x3d/what-x3d
- successor to VRML, ISO standard, used a lot in academia
Internet - Networking Standards
that turned into something big
- TCP/IP layered protocol
- RFC (request for comment)(https://www.ietf.org/standards/rfcs/)
- was not obvious..remember Xerox ring network or proprietary networks on SGI for multiuser interaction (flight game)
Interaction Standards
Accessibility
- Secion 508 section508.gov
- W3C has a group WAI (Web Accessibility Initiative) https://www.w3.org/WAI/
USA Standards Strategy
brand spanking new (may 2023)
U.S. Government National Standards Strategy for Critical and Emerging Technology
"NFTs have single-handedly turned the internet into the world's most expensive flea market." - Seth Meyers
Cryptocurrency - Web3
Blockchain
Virtual Economies
NFTs
Cryptocurrency has only existed since approximately 2009/2010 with the creation of Bitcoin. The explosive growth of the value of a "coin" fueled surge of interest and investments. The key innovation of bitcoin is the "blockchain". A blockchain is a digital ledger. It keeps track of all transactions on the "chain"
Cryptocurrency - Web3
blockchain = distributed ledger
DAPPs
decentralized applications
Create keys (a pair is generated, one public, one private.
Public Key Cryptography
foundational technology
Virtual Economies
- Second Life (linden dollar)
- https://en.wikipedia.org/wiki/Economy_of_Second_Life
- Game gold
"Virtual currencies: because why not trade your hard-earned real money for some pixels and in-game perks?" - Anonymous
digital assets
NFTs
smart contracts
https://www.rendertoken.com/
NIST blockchain report: https://www.nist.gov/blockchain
Blockchain
is a distributed ledger
a "chain" of blocks
Fungible versus Nonfungible
NFTs exist and are associate with a particular "chain". Most common is ETH (Ethereum).
You mine it you own it (or you buy it).
Mining an NFT is really the act of digitally signing some item.
March 2021
NFT: Non Fungible Token
provenance and scarecity
That "Metaverse" thing
“The story so far:
In the beginning the Universe was created.
This has made a lot of people very angry and been widely regarded as a bad move.”
― Douglas Adams, The Restaurant at the End of the Universe
Mike Zyda (USC)
proposed Metaverse University
https://www.youtube.com/watch?v=3zkZOBU5Pcs
That "Metaverse" thing
don't worry nobody really knows what it is
Metaverse Standards Forum: https://metaverse-standards.org/ www.metaverse
IEEE: https://metaverse.ieee.org/
ITU-T Metaverse Focus Group: https://www.itu.int/en/ITU-T/focusgroups/mv/Pages/default.aspx
Spatial version of the Internet
Meta (Facebook)
Apple
Microsoft
Metaverse
virtual real estate
Metaverse
Use Cases: digital twins
Metaverse
Podcasts
Metaverse
Integrating and Interoperable (star trek rodenburry archive)
Summary
- Technology is evolving FAST!
- Virtual Reality ..applications
- Augmented Reality .. useful applications
- Multiuser
- Standards
- Tools
- Crypto - Blockchain - Distributed Ledger
- Metaverse
- This talk is obsolete already
Notes
- https://www.forbes.com/sites/forbesfinancecouncil/2023/04/11/convergence-of-web3-ai-and-metaverse-navigating-the-great-reset-for-investors/?sh=6cf4b0fe4e7e
- Bullet Two
- Bullet Three
- https://medium.com/@julesurbach/what-is-the-metaverse-really-57c30562c237
Template
Teleprescence
Something
- Unity
- Unreal
- Omniverse