ARCore - A big step towards universal mobile Mixed Reality
Codecamp Iași, Spring 2019
Dragoș Silion
What AR(MR) mostly meant until one year ago
Smart glasses had their go
- Vuzix Blade
- Epson Moverio
- Google Glass
- ODG
Most SDKs where Marker Based
- ARToolkit
- AR.js
- Vuforia
- Wikitude
Google Tango
- Actual Mixed Reality
- Dense scans
- Anchor persistence
MR Headsets
- Hololens
- Daqri
- Meta
MR Headsets
- Spatial Mapping
- Spatial Understanding
- Persistent Anchors
- Hand Gestures
- Voice Assistant
- Not cheap
- Not so comfortable
- Small FoV
- Volatile SDKs with low regard to backwards compatibility
- No support for mobile SDKs / Web
ARKit & ARCore
(Q1 of 2018)
Common Features
- Spatial Mapping - Feature Points
- Spatial Understanding - Vertical/Horizontal Planes
- Shared Anchors
- Light Estimation
- Facial Mapping
- Image detection and Tracking
ARCore
- Object Tracking
- Anchors Persistence
ARCore
- Cross-platform (iOS)
Code example &
live demo
Also works on Android, iOS, React Native and (soon?) on web
Feature Points

IMU
Camera Feed
Feature Points
Camera Pose
Env. Mapping
SLAM - Simulatenously Location and Mapping
AR Session
AR Frames
Trackables
Camera feed
Camera Pose
Light Estimation
etc.
Dynamic Data
Planar Tracking

TrackableHit hit;
TrackableHitFlags raycastFilter = TrackableHitFlags.PlaneWithinPolygon |
TrackableHitFlags.FeaturePointWithSurfaceNormal;
if (Frame.Raycast(touch.position.x, touch.position.y, raycastFilter, out hit))
{
if ((hit.Trackable is DetectedPlane) &&
Vector3.Dot(FirstPersonCamera.transform.position - hit.Pose.position,
hit.Pose.rotation * Vector3.up) < 0)
{
Debug.Log("Hit at back of the current DetectedPlane");
}
else
{
GameObject prefab;
if (hit.Trackable is FeaturePoint)
{
prefab = AndyPointPrefab;
}
else
{
prefab = AndyPlanePrefab;
}
var andyObject = Instantiate(prefab, hit.Pose.position, hit.Pose.rotation);
andyObject.transform.Rotate(0, k_ModelRotation, 0, Space.Self);
var anchor = hit.Trackable.CreateAnchor(hit.Pose);
andyObject.transform.parent = anchor.transform;
}
}TrackableHit hit;
TrackableHitFlags raycastFilter = TrackableHitFlags.PlaneWithinPolygon |
TrackableHitFlags.FeaturePointWithSurfaceNormal;
if (Frame.Raycast(touch.position.x, touch.position.y, raycastFilter, out hit))
{
if ((hit.Trackable is DetectedPlane) &&
Vector3.Dot(FirstPersonCamera.transform.position - hit.Pose.position,
hit.Pose.rotation * Vector3.up) < 0)
{
Debug.Log("Hit at back of the current DetectedPlane");
}
else
{
GameObject prefab;
if (hit.Trackable is FeaturePoint)
{
prefab = AndyPointPrefab;
}
else
{
prefab = AndyPlanePrefab;
}
var andyObject = Instantiate(prefab, hit.Pose.position, hit.Pose.rotation);
andyObject.transform.Rotate(0, k_ModelRotation, 0, Space.Self);
var anchor = hit.Trackable.CreateAnchor(hit.Pose);
andyObject.transform.parent = anchor.transform;
}
}TrackableHit hit;
TrackableHitFlags raycastFilter = TrackableHitFlags.PlaneWithinPolygon |
TrackableHitFlags.FeaturePointWithSurfaceNormal;
if (Frame.Raycast(touch.position.x, touch.position.y, raycastFilter, out hit))
{
if ((hit.Trackable is DetectedPlane) &&
Vector3.Dot(FirstPersonCamera.transform.position - hit.Pose.position,
hit.Pose.rotation * Vector3.up) < 0)
{
Debug.Log("Hit at back of the current DetectedPlane");
}
else
{
GameObject prefab;
if (hit.Trackable is FeaturePoint)
{
prefab = AndyPointPrefab;
}
else
{
prefab = AndyPlanePrefab;
}
var andyObject = Instantiate(prefab, hit.Pose.position, hit.Pose.rotation);
andyObject.transform.Rotate(0, k_ModelRotation, 0, Space.Self);
var anchor = hit.Trackable.CreateAnchor(hit.Pose);
andyObject.transform.parent = anchor.transform;
}
}Image Targets
Image Targets
// Get updated augmented images for this frame.
Session.GetTrackables<AugmentedImage>(
m_TempAugmentedImages, TrackableQueryFilter.Updated);
// Create visualizers and anchors for updated augmented images that are tracking and do
// not previously have a visualizer. Remove visualizers for stopped images.
foreach (var image in m_TempAugmentedImages)
{
AugmentedImageVisualizer visualizer = null;
m_Visualizers.TryGetValue(image.DatabaseIndex, out visualizer);
if (image.TrackingState == TrackingState.Tracking && visualizer == null)
{
// Create an anchor to ensure that ARCore keeps tracking this augmented image.
Anchor anchor = image.CreateAnchor(image.CenterPose);
visualizer = (AugmentedImageVisualizer)Instantiate(
AugmentedImageVisualizerPrefab, anchor.transform);
visualizer.Image = image;
m_Visualizers.Add(image.DatabaseIndex, visualizer);
}
else if (image.TrackingState == TrackingState.Stopped && visualizer != null)
{
m_Visualizers.Remove(image.DatabaseIndex);
GameObject.Destroy(visualizer.gameObject);
}
}Image Targets
if (Image == null || Image.TrackingState != TrackingState.Tracking)
{
FrameLowerLeft.SetActive(false);
FrameLowerRight.SetActive(false);
FrameUpperLeft.SetActive(false);
FrameUpperRight.SetActive(false);
return;
}
float halfWidth = Image.ExtentX / 2;
float halfHeight = Image.ExtentZ / 2;
FrameLowerLeft.transform.localPosition =
(halfWidth * Vector3.left) + (halfHeight * Vector3.back);
FrameLowerRight.transform.localPosition =
(halfWidth * Vector3.right) + (halfHeight * Vector3.back);
FrameUpperLeft.transform.localPosition =
(halfWidth * Vector3.left) + (halfHeight * Vector3.forward);
FrameUpperRight.transform.localPosition =
(halfWidth * Vector3.right) + (halfHeight * Vector3.forward);
FrameLowerLeft.SetActive(true);
FrameLowerRight.SetActive(true);
FrameUpperLeft.SetActive(true);
FrameUpperRight.SetActive(true);Shared Anchors
Facial Mapping
- Left forehead (LEFT_FOREHEAD)
- Right forehead (RIGHT_FOREHEAD)
- Tip of the nose (NOSE_TIP)

So what's the current state of AR?
Smart Glasses are still nothing wow
Headsets are still going strong
...very strong
We might be in mobile SDKs era
(over ARCore/ARKit)
Vuforia - Same feature, more stable over ARCore + New Features

We might be in mobile SDKs era
(over ARCore/ARKit)
Azure cloud anchors

We might be in mobile SDKs era
(over ARCore/ARKit)
LENS STUDIO


We might be in mobile SDKs era
(over ARCore/ARKit)
LENS STUDIO
We might be in mobile SDKs era
(over ARCore/ARKit)
6d.ai
We might be in mobile SDKs era
(over ARCore/ARKit)
And other crazy stuff (Posemoji)
And WebAR
(TBD)
If you are curious about AR, get started.
Get Coding!
Start Creating!
It's not about 💲big budgets💲 anymore.
Thank you!
Dragoș Silion
twitter - @silidragos
dragos@colorfulcoding.com

Please help me with some honest feedback 👍👎:
Codecamp Iasi Spring 2019
By silidragos
Codecamp Iasi Spring 2019
- 145