The state of Augmented Reality and React Native in 2023
Software Architect & Consultant
Web / Mobile / XR / IoT / Web3 / AI
Augmented Reality
1901 First recorded AR reference (L. Frank Baum - "The Master Key")
1952 Sensorama
Morton Heilig
1968 Sword of Damocles
Ivan Sutherland
1982 AR for weather broadcasters Dan Reitan
1992 Virtual Fixtures Louis Rosenberg
1996 First AR markers CyberCode
1998 – NASA develops AR dashboard for X38
1999 Eye tap - wearable AR glasses Steve Mann
2008 AR used by BMW for commercial purposes
2010 – Kinnect
2012 – Google Glass
90's - 2000's
2010-2020
2013 – AR is used in car manufacturing
2016 AR/VR investment reaches 1.1 billion
2016 Microsoft Hololens/Meta 2
2016 – Pokemon Go
2017 – Apple anounced ARKit, Google - ARCore
2018 Magic Leap (light field technology)
ARKit / ARCore
AR with React Native
What we can do with it?
Anatomy of the scene
AR Scene
props passed and are accessible within scene component
Top level component
AR Scene
Lighting
Plane selector
AR Component
So is AR just a gimmick?
Let's see what it is capable of!
Features for mobile
https://unity.com/features/mobile
React Native and Unity
JS
Native
Unity as a library
Demo
GameObject in Unity
Method name to call
Payload
In Unity
Native iOS Plugin
Unity Native Plugins (iOS)
NativeCallProxy.h
In Unity Project under Assets/Plugins/iOS create two files
Unity Native Plugins (iOS)
NativeCallProxy.mm
Sending message to Native in Unity
ButtonBehavior.cs
public static extern void sendMessageToMobileApp(string message);
NativeAPI.sendMessageToMobileApp("The button has been tapped!");
Assign script to gameObject in Unity and click handler to button in Unity
Receiving message from RN in Unity
We need to parse JSON we receive from RN
RNInterop.cs
RNInterop.cs
RN Integration steps (iOS)
Build Unity Project for iOS into folder within your RN project
unity/builds/ios
Open Xcode workspace of your React Native project
ios/rnunitynative.xcworkspace
right click and add Files to workspace. Select Unity Project
Add UnityFramework to Embedded Frameworks,
Remove UnityFramework from Build Phases Link Binary with Libraries
Drag and drop Embed Frameworks before Compile sources
Add Privacy - Camera Usage Description key to Info.plist
1. Change Data membership to be UnityFramework
2.Change build products path for UnityFramework to be your RN ios build folder
Do it every time after you build Unity Project
3.Change Native Plugins Target membership to UnityFramework & Public
4.Build UnityFramework separately
Integrate with React Native
Create
ReactNativeUnityViewManager.mm
Expose module to JS
Export view property to be callback
Expose postMessage method to JS
Send events to JS
Make sure we run on main thread
UnityFramework API to call method by nameReactNativeUnityView.mm
Framework wrapper to handle loading and options
Getting back to JS side
Let's get to the cool news.
There is a library for that! 😉
No need to write Native code
Still need to do all the setup and build steps
Now you know how it actually works 🎉
Helpful links
Demo code
Learn unity
Unity Asset store
Unity AR Samples
Thank You