3 strategies to make your apps feel all the feels!

Boost Your Apps' Emotional Intelligence

@jenlooper

Jen Looper

Progress

Senior Developer Advocate

Who am I?

@jenlooper

Let's talk about mobile apps

NativeScript

@jenlooper

NativeScript is…

an open source framework for building truly native mobile apps with JavaScript. Use web skills, like TypeScript, Angular and CSS, and get native UI and performance on iOS and Android.

@jenlooper

NativeScript is the best tool for cross-platform native app development 🎉

@jenlooper

Rich, animated, “no compromise” native UI

(with shared UI code)

@jenlooper

You know JavaScript? You know NativeScript! 

@jenlooper

Write once...

@jenlooper

Craft the UI with XML

@jenlooper

Built plugins with native libraries

@jenlooper

...or use the Marketplace for plugins

@jenlooper

NativeScript community Slack channel

@jenlooper

@jenlooper

Help!

My apps are stupid and boring

@jenlooper

Try an IoT integration!

Try two machine learning APIs

Let's fix that!

Talk a little about what's possible next

@jenlooper

Make your apps "smarter"

"smart" = more human

@jenlooper

Let's build something

@jenlooper

QuickNoms: a smart recipe app

Powered by Firebase & NativeScript

Submit your recipes on the web!

QuickNoms.com

@jenlooper

Mobile App Features:

Algolia search

Firebase Remote Config marquee

goal: empathetic apps

@jenlooper

Move from a simple master/detail app to...

@jenlooper

Make your app 'sensitive'

Build an IoT integration to craft a recipe recommender based on room temperature

@jenlooper

Add a sensor

@jenlooper

Build the device

wifi-connected Particle Photon + temperature sensor - about $25 total

@jenlooper

Flash code to the Photon

Photon reads temp every 10 secs, writes data to Particle Cloud

Build webhook

webhook lives in Particle Cloud, watches for data written by Photon to cloud

Webhook writes to Firebase

@jenlooper

app consumes data and reacts

Select recipes tagged as 'hot' or 'cold' - atmosphere type recipes

@jenlooper

Observable subscribes to

temperature saved to Firebase

ngOnInit(): void {
        this.recipesService.getTemperatures(AuthService.deviceId).subscribe((temperature) => {           
            this.temperature$ = temperature[0].temperature;            
            this.getRecommendation(this.mode)                                                  
        })
    }

    getRecommendation(mode){
       
        if (mode == 'F'){
            if (Number(this.temperature$) > 70) {
                this.gradient = this.hotGradient;
                this.recommendation = this.hotRecommendation;            
            } 
            else {
                this.gradient = this.coolGradient;
                this.recommendation = this.coolRecommendation;                           
            }
        }
...

Scale the idea

@jenlooper

demo

@jenlooper

Add some Machine Learning

@jenlooper

Machine Learning + Mobile = ❤️

think of the possibilities for photos, video, audio

@jenlooper

ML is easy

not

@jenlooper

What even is machine learning?

@jenlooper

a way to give “computers the ability to learn without being explicitly programmed.”

Machine Learning is:

@jenlooper

"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.” (Tom Mitchell, 1997).

@jenlooper

@jenlooper

How to make a machine learn*

Gather a lot of data (images, sounds)

Divide that data into a training set and a test set

Use an algorithm to train a model with the training set by pairing input with expected output

  • The training set is categorized (sorted by hand or by machine)
  • The test set is uncategorized

*"supervised learning"

Use the test set to test the accuracy of the training

rinse & repeat

@jenlooper

ML in the wild

Good uses of ML

StitchFix combines ML + human curation

Formulas to pick out clothes based on customer input

Formulas to pair a shopper with a stylist

Formulas to calculate distance of warehouse to customer

Algorithms to search and classify clothing trends to recommend

@jenlooper

@jenlooper

Scary uses of ML

install a ton of surveillance cameras

get really good at ml-powered facial recognition

match faces to IDs

monitor emotions...and manipulate them

push ads at people based on age/gender

invisibly track location

@jenlooper

@jenlooper

good and bad?

MIT students used an algorithm to optimize school bus routes

50 superfluous routes eliminated

$3-5 million saved

50 union bus drivers out of work

@jenlooper

with great power

comes great responsibility!

@jenlooper

Microsoft 👏

"snow leopard or not" - partnership with the Snow Leopard Trust

Microsoft 👏

"AI For Earth"

https://www.microsoft.com/en-us/aiforearth

DIY Machine Learning is hard

you need a lot of firepower & skillz

@jenlooper

Use a third party with pretrained models

@jenlooper

Specialists in image analysis

Took top 5 awards in 2013 ImageNet challenge

Innovative techniques in training models to analyze images

Offer useful pre-trained models like "Food" "Wedding" "NSFW"

Or, train your own model!

@jenlooper

"Does this dish qualify as a QuickNom?"

Use Clarif.ai's pretrained Food model to analyze images of plates of food for inspiration

probably not!

might be!

@jenlooper

Take a picture

takePhoto() {
        
        const options: camera.CameraOptions = {
            width: 300,
            height: 300,
            keepAspectRatio: true,
            saveToGallery: false
        };

        camera.takePicture(options)
        .then((imageAsset: ImageAsset) => {
            this.processRecipePic(imageAsset);
        }).catch(err => {
            console.log(err.message);
        });
    
    }

@jenlooper

Send it to Clarif.ai via

REST API call

    public queryClarifaiAPI(imageAsBase64):Promise<any>{
        return http.request({
            url: AuthService.clarifaiUrl,
            method: "POST",
            headers: {
                "Content-Type": "application/json",
                 "Authorization": "Key " + AuthService.clarifaiKey,
            },
            content: JSON.stringify({
                "inputs": [{
                    "data": {
                        "image": {
                            "base64": imageAsBase64
                        }
                    }
                }]
            })
        })
      .then(function (response) {
          return response
        }
      )}

@jenlooper

Analyze returned tags

.then(res => {
   this.loader.hide();
      try {
          let result = res.content.toJSON();
          let tags = result.outputs[0].data.concepts.map( mc => mc.name + '|' + mc.value );
          let ingredients = [];
          tags.forEach(function(entry) {
              let prob = entry.split('|');
              prob = prob[1];
              let ingred = entry.split('|');
                 if(prob > 0.899){
                    ingredients.push(ingred[0])
                 }
                                    
              });
            //there should be between four and eight discernable ingredients                     
            if (ingredients.length >= 4 && ingredients.length <= 8) {
               alert("Yes! This dish might qualify as a QuickNom! It contains "+ingredients)
            }
            else {
               alert("Hmm. This recipe doesn't have the qualifications of a QuickNom. 
                    Try again!")
            }
       }

if between 4 & 8 ingredients are listed with over .899 certainty,

it's a QuickNom!

QuickNom dishes have a few easy-to-see, simple ingredients

demo

@jenlooper

"What can I make with an avocado?"

Use Google's Vision API to match images with recipes

@jenlooper

Do it all with Google!

Leverage its consumption of millions of photos via Google Photos with Cloud Vision API

  • Label Detection
  • Explicit Content Detection
  • Logo Detection
  • Landmark Detection
  • Face Detection
  • Web Detection (search for similar)

@jenlooper

takePhoto() {
        
        const options: camera.CameraOptions = {
            width: 300,
            height: 300,
            keepAspectRatio: true,
            saveToGallery: false
        };

        camera.takePicture(options)
        .then((imageAsset: ImageAsset) => {
            this.processItemPic(imageAsset);
        }).catch(err => {
            console.log(err.message);
        });
    
    }

Take a picture

@jenlooper

public queryGoogleVisionAPI(imageAsBase64: string):Promise<any>{
        return http.request({
            url: "https://vision.googleapis.com/v1/images:annotate?key="+AuthService.googleKey,
            method: "POST",
            headers: {
                "Content-Type": "application/json",
                "Content-Length": imageAsBase64.length,
            },
            content: JSON.stringify({
                "requests": [{
                "image": {
                "content": imageAsBase64 
                },
                "features" : [
                    {
                        "type":"LABEL_DETECTION",
                        "maxResults":1
                    }
                ]                      
            }]
        })
      })
      .then(function (response) {
          return response
        }
      )}

Send it to Google

this.mlService.queryGoogleVisionAPI(imageAsBase64)
    .then(res => {
        let result = res.content.toJSON();
        this.ingredient = result.responses[0].labelAnnotations.map( mc => mc.description );                                                               
        this.ngZone.run(() => {
            this.searchRecipes(this.ingredient)
        })
    });

Grab the first label returned and send to Algolia search

@jenlooper

demo

@jenlooper

Looking forward

@jenlooper

DIY machine learning

made a little easier!

@jenlooper

Machine learning on device

What if you don't want to make a bunch of expensive $$ REST API calls?

What if you need offline capability?

What if you want to keep your data on device?

What if you need to train something really custom?

@jenlooper

Machine learning on device

Now landed in iOS 11: Core ML

Train a model externally, let Core ML process it for your app on device

@jenlooper

Machine learning on device

TensorFlow Mobile (v1)

Designed for low-end Androids, works for iOS and Android

@jenlooper

New! Hot! TensorFlow Lite!

next-gen version of TensorFlow for mobile

"on-device machine learning inference with low latency and a small binary size."

TensorFlow Lite for iOS

you have the option to convert to CoreML!

Featuring:

  • a new model file format, based on "FlatBuffers" - smaller/faster/more memory efficient than ProtocolBuffers
  • new mobile-optimized interpreter
  • an interface to leverage hardware acceleration (Android)
  • small footprint! 75 - 400 kb!

 

Watch this project! Coming soon: train ON DEVICE

Machine learning on device

TensorFlow powers Google Translate!

@jenlooper

Google Translate (realtime text recognition using TensorFlow models on device)

demo:

@jenlooper

TensorFlow Mobile on iOS

demo:

@jenlooper

@jenlooper

Boost Your Apps' Emotional Intelligence with Machine Learning

By Telerik DevRel

Boost Your Apps' Emotional Intelligence with Machine Learning

For ATO, DevReach, JazzConf

  • 1,945