Virtual Reality the Metaverse and All that Reality Stuff

Sandy Ressler (NIST)

USPTO Patent Examiner Training

May 2023

DISCLAIMER: Any mention of commercial products is for informational purposes only and does not imply endorsement by NIST.

Just your typical Patent Examiner room ;-)

 

"Just when I discovered the meaning of life, they changed it." ― George Carlin, on constantly evolving technology

 

“I've come up with a set of rules that describe our reactions to technologies:


1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.


2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.


3. Anything invented after you're thirty-five is against the natural order of things.”
― Douglas Adams, The Salmon of Doubt

Topics (human created)

  • Virtual Reality
    • applications
  • Augmented Reality
    • mixed reality
    • applications
  • Standards
  • Tools
  • That Metaverse thing
    • digital twins
    • Blockchain/NFTs

Topics (ChatGPT)

  • Virtual Reality (VR) and Augmented Reality (AR

  • 3D Graphics and Spatial Computing

  • Networking and Connectivity

  • Blockchain and Cryptocurrencies

  • Virtual Economies

  • Social Platforms and Interoperability

  • Gaming and Entertainment

  • User Generated Content (UGC) and Customization
  • Artificial Intelligence and Machine Learning

Not bad but it missed standards and AI and ML can be applied to anything, I'll rate it a B.

prompt:

give me a bulleted list of the main components of the metaverse

 

 

First a little history

Ivan Sutherland

(WAYYYYYYY ahead of his time)

1963

60 FREAKING YEARS AGO!!!

Jaron Lanier coined "VR"

1987

NASA Jet Propulsion Labs early pioneering (training work) 1990

What is Virtual Reality?

strap a screen to your face

Virtual Reality

(characteristics)

  • Immersive
  • Compelling
  • Isolating
  • Issues with "simulator sickness"
  • time limited (only wear that screen for so long)
  • high resolution screen
  • head tracking
  • hand tracking (sometime with extra controller devices) for interaction
  • spatial audio
  • eye tracking, also for interaction
  •  

As an aside

  • spatial audio
    • head tracked airpod
  • eye tracking
    • contact lens screens (mojo)

Virtual Reality

Applications

  • training
  • therapy
  • data visualization
  • virtual real estate
  • exercise

Virtual Reality

Training

Virtual Reality

Physical Therapy

Virtual Reality

Treatment of Phobias

Virtual Reality

Virtual Real Estate

Virtual Reality

Exercise

Virtual Reality

Exercise - Beat Saber

Virtual Reality

Fire Research: 360 degree videos

Virtual Reality

Fire Research: 360 degree videos

Virtual Reality

(significant problems)

  • simulator sickness
  • duration issues

Augmented Reality

"Augmented reality: because reality wasn't confusing enough on its own." - Unknown

Augmented (mixed) Reality

  • Keeps you in the "real world"
  • Mobile phone is intro device
    • ubiquity of phones
    • many compelling use cases
      • navigation
      • language translation
      • already highly popular (SnapChat lens)
  • Interact with real objects
    • tactile doesn't need to be simulated (it's real)

https://medium.com/@sressler/googles-live-view-the-first-useful-augmented-reality-application-91525209743f

Augmented Reality

Navigation Aid

Augmented Reality

Training

Augmented Reality

Language Translation

Augmented Reality

Shopping

Immersive Reality

Data Visualization

Augmented Reality

  • Visual fidelity
  • Lag/virtual shaking-swimming
  • Heat dissipation - power requirements
  • Eye/Gaze tracking UI 
  • Eye glass form factor - holy grail (coming!)

significant remaining problems

Networking and Connectivity (Multiuser)

"The Internet: where men are men, women are men, and children are undercover FBI agents." - Anonymous

Networking and Connectivity (Multiuser)

  • Shared office

  • Concert

  • Sports Game

  • Parties

Networking

Shared Office.  Meta Horizon Workrooms

Networking

Teleprescence

Google Project Starline

Networking

Concert

Networking

Sports

Multiuser

Serious Games

NPSNET demo (1995)

Tools

"Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live."

- John Woods

Tools

  • General Purpose Authoring: Unreal, Unity, Omniverse
  • Web based: aframe.js
  • Specialized: LensStudio, 360 degree video
  • Photogrammetry: 3D scenes from 2D photographs

NeRF - Neural Radiance Field

In summary, NeRF is a method that uses machine learning to represent and render 3D scenes from 2D images. By training a neural network on a set of input images, NeRF can generate realistic, high-quality views of the scene from new perspectives. - ChatGPT

"Unity is the virtual reality tool that lets you create your own universe – just make sure you don't get lost in it while debugging." - Unknown

General Purpose Creation Tools

  • Unity: https://unity.com/
  • Unreal: https://www.unrealengine.com/en-US
  • Omniverse: https://www.nvidia.com/en-us/omniverse/

All are highly capable and robust tools - as always the choice is "it depends" on your application requirements.

Web Based: aframe.js

Photogrammetry

NeRF - NVidia Instant NeRF

SnapChat Lens Creation

  • text to image generators
    • DallE
    • midjourney
  • text to 3D objects
    • Point-e (openai)
  • text to VR environment
    • blockadelabs.com
  • text to video
    • runwayml.com

this will be out of date by the time you read this..the field is changing at explosive speed!!

AI image generator tools

 

Minecraft: 141 million active monthly users 2021

Fortnite: 250 million monthly active users 2023

 

User Generated Content

big business

virtually unlimited changes (mods)

constrained but extensive customization

"The nice thing about standards is that there are so many to choose from." ... Andrew S. Tanenbaum

 

3D objects - file formats

communications - Internet TCP/IP

interaction - UX

Standards

make the world interoperable

3D File Formats

 

Internet - Networking Standards

that turned into something big

 

 

  • TCP/IP layered protocol
  •  RFC (request for comment)(https://www.ietf.org/standards/rfcs/)
  • was not obvious..remember Xerox ring network or proprietary networks on SGI for multiuser interaction (flight game)

Interaction Standards

Accessibility

 

  • Secion 508 section508.gov
  • W3C has a group WAI (Web Accessibility Initiative) https://www.w3.org/WAI/

USA Standards Strategy

brand spanking new (may 2023)

U.S. Government National Standards Strategy for Critical and Emerging Technology

"NFTs have single-handedly turned the internet into the world's most expensive flea market." - Seth Meyers

Cryptocurrency - Web3

 

Blockchain

Virtual Economies

NFTs

Cryptocurrency has only existed since approximately 2009/2010 with the creation of Bitcoin. The explosive growth of the value of a "coin" fueled surge of interest and investments. The key innovation of bitcoin is the "blockchain". A blockchain is a digital ledger. It keeps track of all transactions on the "chain"

Cryptocurrency - Web3

blockchain = distributed ledger

DAPPs

decentralized applications

Create keys (a pair is generated, one public, one private.

 

Public Key Cryptography

foundational technology

Virtual Economies

  • Second Life (linden dollar)
    • https://en.wikipedia.org/wiki/Economy_of_Second_Life
  • Game gold
  •  

"Virtual currencies: because why not trade your hard-earned real money for some pixels and in-game perks?" - Anonymous

digital assets

NFTs

smart contracts

https://www.rendertoken.com/

NIST blockchain report: https://www.nist.gov/blockchain

Blockchain

is a distributed ledger

a "chain" of blocks

Fungible versus Nonfungible

NFTs exist and are associate with a particular "chain". Most common is ETH (Ethereum). 

You mine it you own it (or you buy it).

Mining an NFT is really the act of digitally signing some item. 

March 2021

NFT: Non Fungible Token

provenance and scarecity

That "Metaverse" thing

“The story so far:
In the beginning the Universe was created.
This has made a lot of people very angry and been widely regarded as a bad move.”

― Douglas Adams, The Restaurant at the End of the Universe

Mike Zyda (USC)

proposed Metaverse University

https://www.youtube.com/watch?v=3zkZOBU5Pcs

That "Metaverse" thing

don't worry nobody really knows what it is

Metaverse Standards Forum: https://metaverse-standards.org/ www.metaverse

IEEE: https://metaverse.ieee.org/

ITU-T Metaverse Focus Group: https://www.itu.int/en/ITU-T/focusgroups/mv/Pages/default.aspx

Spatial version of the Internet

Meta (Facebook)

Google

Apple

Microsoft

Metaverse 

virtual real estate

Metaverse

Use Cases: digital twins

Metaverse

Podcasts

Metaverse

Integrating and Interoperable (star trek rodenburry archive)

Summary

  • Technology is evolving FAST!
  • Virtual Reality ..applications
  • Augmented Reality .. useful applications
  • Multiuser 
  • Standards
  • Tools
  • Crypto - Blockchain - Distributed Ledger
  • Metaverse
  • This talk is obsolete already

Notes

  • https://www.forbes.com/sites/forbesfinancecouncil/2023/04/11/convergence-of-web3-ai-and-metaverse-navigating-the-great-reset-for-investors/?sh=6cf4b0fe4e7e
  • Bullet Two
  • Bullet Three
  • https://medium.com/@julesurbach/what-is-the-metaverse-really-57c30562c237

Template

Teleprescence

Something

  • Unity
  • Unreal
  • Omniverse

Link:

VR the Metaverse and More

By Sandy Ressler

VR the Metaverse and More

  • 145