Human vs. AI: Build a Mobile App with Vue.js,

ML Kit, and NativeScript

@jenlooper

Jen Looper

Progress

Senior Developer Advocate

Founder and CEO: Vue Vixens 

Who am I?

"friend of bots"

@jenlooper

🤖

@jenlooper

Agenda

What is Machine Learning?

Why is Machine Learning so important?

How can Machine Learning help make more interesting mobile apps?

How can I build an app using two new technologies: ML Kit and NativeScript-Vue?

WHAT

WHY

HOW

What even is machine learning?

@jenlooper

ML is easy

not

@jenlooper

a way to give “computers the ability to learn without being explicitly programmed.”

Machine Learning is:

@jenlooper

"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.” (Tom Mitchell, 1997).

@jenlooper

@jenlooper

How to make a machine learn*

Gather data

*"supervised learning"

@jenlooper

 

🍎🍊🥕
training set

 

 

🍑🍐🍍
test set

 

fruits: [🍎,🍊]

veggies: [🥕,🌽]

no categories

🤖💭

🍎: "this is an apple!"

🍎: "right!"

🐕: "this is an apple!"

🐕: "uhh..."

80% accurate... try again!

How to make a machine learn*

*"unsupervised learning"

@jenlooper

Gather data

@jenlooper

[🍲,🥨,🌭,🍣]

no categories!

 

🤖🎓💭

a self-teaching pattern-recognizing algorithm

🥨: "this is a pretzel!"

🥨: "right!"

🌭: "this is a pretzel!"

🌭: "uhh..."

80% accurate... try again!

How to make a machine learn*

*"reinforcement learning"

Gather data

[🍲,🥨,🌭,🍣]

no categories!

 

🤖🕶🎓💭

a 'curious' self-teaching pattern-recognizing algorithm*

🥨: "this is a pretzel!"

🥨: "right!"

🌭: "this is a chili dog!"

🌭: "actually, yeah!"

80% accurate... try again!

*give it harder and harder problems, reward its curiosity, like a child

Deep Learning

a subfield of machine learning

uses ML algorithms in layers to create an artificial neural network to make independent decisions

Tools for local machine learning

@jenlooper

TensorFlow, Google's library for machine learning

Learn about TensorFlow by doing code labs

@jenlooper

bit.ly/tf-4-poets

A JavaScript library for training and deploying ML models in the browser and on Node.js

https://js.tensorflow.org/

🔖Bookmark TensorFlow for now!

ML in the wild

@jenlooper

Good uses of ML

StitchFix combines ML + human curation

Runs input through algorithm

connects shopper to stylist

🤖

👱‍♀️

👩‍💻

indicates style preferences

builds recommendations - factors in location, trends, season

curates and ships box

gives feedback:

👍buys

or

👎returns items

@jenlooper

Scary uses of ML

when you install a ton of surveillance cameras, you can...

get really good at facial recognition algorithms

match faces to IDs

monitor group emotions

track location

@jenlooper

🤖🎥

@jenlooper

good and bad?

🚓 New York and Miami:  ML for crime forecasting

@jenlooper

🚑 Northshore University Health System: ML to predict cardiac arrest using electronic medical records.

👩‍🎓Tulsa school systems: ML for early intervention to prevent students from dropping out of school or graduating late

with great power

comes great responsibility!

@jenlooper

ML + Mobile = ❤️

mini collectors-of-data

packed with sensors

supercharged with mobile apps

App Developers have a new arsenal!

make your apps 'smarter' - more helpful

build a whole app around ML (SeeingAI, Shazam)

Makeup apps

built using neural networks

Facial skin beautification via sparse representation over learned layer dictionary

make your apps 'smarter' - more helpful

...or add little ML touches

@jenlooper

Do-it-yourself Machine Learning is hard

you need a lot of firepower & skills

@jenlooper

Try a third-party service!

Tools:

@jenlooper

+

=

Thank you Igor

did I leave the oven on?

Let's talk about NativeScript

@jenlooper

NativeScript is…

an open source framework for building truly native mobile apps with JavaScript. Use web skills, like TypeScript, Angular, Vue, and CSS, and get native UI and performance on iOS and Android.

@jenlooper

Rich, animated, “no compromise” native UI

(with shared UI code)

@jenlooper

You know JavaScript? You know NativeScript! 

@jenlooper

Why Vue for NativeScript?

📱Vue's 2.0 adoption of the virtual DOM enables native mobile rendering (Angular 2+ is similar)

💻 Vue offers a great way for web developers to embrace mobile platforms via NativeScript or Weex

☁️ Vue is lightweight so highly appropriate for mobile

🤝 NativeScript and Vue have great code-sharing potential!

my first NativeScript-Vue App

@jenlooper

https://play.nativescript.org/

Making a Machine Learning - powered mobile app

@jenlooper

+

DIY machine learning

made a little easier!

@jenlooper

Machine learning on device with a plugin

💸What if you don't want to make a bunch of costly REST API calls?

🔌What if you need offline capability?

🎓What if you need to train something really custom?

@jenlooper

Machine learning on device

TensorFlow Lite

Designed for low-end Androids, works for iOS and Android

@jenlooper

New! Hot! ML Kit

@jenlooper

@jenlooper

Image labeling

Identify objects, locations, activities, animal species, products++

Text recognition (OCR)

Recognize and extract text from images

Face detection

Detect faces and facial landmarks

Barcode scanning

Scan and process barcodes

Landmark detection

Identify popular landmarks in an image

How does it work?

@jenlooper

  • Leverages Firebase to access your app's data
  • Bundles together Google's ML technologies, including Google Cloud Vision API, TensorFlow Lite, and Android Neural Networks API in a single SDK
  • WORKS ON DEVICE

Putting ML Kit to the test with NativeScript-Vue

@jenlooper

Picme: A game testing ML Kit accuracy and speed

The Ultimate Challenge

image collections by Karen Zack @teenybiscuit

The Ultimate Challenge

image collections by Karen Zack @teenybiscuit

How did I build it?

  • Used Vue-CLI template
  • Used 3 plugins
  • Vuex

scaffold an app!

use the Vue CLI with a template

vue init nativescript-vue/vue-cli-template picme

install some useful plugins

  • nativescript-swipe-layout
  • nativescript-card-view
  • nativescript-plugin-firebase
import firebase from 'nativescript-plugin-firebase';

firebase.init({
  // Optionally pass in properties for db, auth & cloud messaging
}).then(
  instance => {
    console.log("firebase.init done");
  },
  error => {
    console.log(`firebase.init error: ${error}`);
  }
);
...

Vue.prototype.$firebase = firebase;

Build the UI

use the Vue.js template block

<SwipeLayout row="2" col="0" v-for="item in count" :key="item" 
@swipeLeft="swipeCallback($event,item)" @swipeRight="swipeCallback($event,item)">                             
              
      <GridLayout rows="2*,2*" columns="*">

           <CardView row="0" elevation="20" margin="10" radius="10" 
                backgroundColor="white" shadowOpacity=".2" shadowRadius="5">
               
                <Image :src="'~/images/'+imageFolder+'/'+item+'.jpg'"/>        
           
            </CardView> 

          ...

     </GridLayout>
            
</SwipeLayout>

Using the Firebase plugin for ML Kit

queryMLKit(id){
        this.response = ["thinking..."];
        ...
        let image = '/images/'+this.imageFolder+'/'+this.currImage+'.jpg';

        ...
        source.fromFile(path)
          .then((imageSource) => {
            this.$firebase.mlkit.imagelabeling.labelImageOnDevice({
              image: { imageSource: source },
              confidenceThreshold: 0.6
          })
          .then((result) => {
            ...
            for (let i = 0; i < result.labels.length; ++i) {
              tempArr.push(
                result.labels[i].text + ' : ' + Math.floor((result.labels[i].confidence) * 100) + '%'
              );
              ...
            }
            this.calculateBotScore(this.imageFolder,this.currImage,labelsArr)

           ...      
          }
        )
        ...
        })
      },

query ML Kit

Test ML Kit's result against possible answers

export const Images = [
    {
      type: "barnowl",
      ids: [1,4,5,7,10,12,13,15],
      acceptedAnswers: ["bird","owl"]
    },
    {
      type: "apple",
      ids: [2,3,6,8,9,11,14,16],
      acceptedAnswers: ["food","apple"]
    },
...
]

Use Vuex to keep score

  
const state = {
  humanScore: 0,
  botScore: 0
};

const mutations = {
  clearScore (state){
    state.humanScore = 0;
    state.botScore = 0;
  },
  setHumanScore (state) {
    state.humanScore++;
  },
  setBotScore (state) {
    state.botScore++;
  }
};

const actions = {
  clearScore: ({commit}) => commit('clearScore'),    
  setHumanScore: ({commit}) => commit('setHumanScore'),
  setBotScore: ({commit}) => commit('setBotScore'),
  
};

...

The result

Can you beat the AI?

Let's try!

🤖 vs. 👩

@jenlooper

nativescript-vue.org

Made with Slides.com