3 strategies to make your apps feel all the feels!

Boost Your Apps' Emotional Intelligence

Jen Looper

Progress

Senior Developer Advocate

Let's talk about mobile apps

NativeScript

NativeScript is…

an open source framework for building truly native mobile apps with JavaScript. Use web skills, like TypeScript, Angular and CSS, and get native UI and performance on iOS and Android.

NativeScript is the best tool for cross-platform native app development 🎉

Rich, animated, “no compromise” native UI

(with shared UI code)

You know JavaScript? You know NativeScript! 

Write once...

Craft the UI with XML

Built plugins with native libraries

...or use the Marketplace for plugins

NativeScript community Slack channel

Help!

My apps are stupid and boring

Make your apps "smarter"

Let's build something

Powered by Firebase & NativeScript

QuickNoms.com

Algolia search

Firebase Remote Config marquee

Make your app 'sensitive'

Build an IoT integration to craft a recipe recommender based on room temperature

Add a sensor

Build the device

wifi-connected Particle Photon + temperature sensor - about $25 total

Flash code to the Photon

Photon reads temp every 10 secs, writes data to Particle Cloud

Build webhook

webhook lives in Particle Cloud, watches for data written by Photon to cloud

Webhook writes to Firebase

app consumes data and reacts

Select recipes tagged as 'hot' or 'cold' - atmosphere type recipes

ngOnInit(): void {
        this.recipesService.getTemperatures(AuthService.deviceId).subscribe((temperature) => {           
            this.temperature$ = temperature[0].temperature;            
            this.getRecommendation(this.mode)                                                  
        })
    }

    getRecommendation(mode){
       
        if (mode == 'F'){
            if (Number(this.temperature$) > 70) {
                this.gradient = this.hotGradient;
                this.recommendation = this.hotRecommendation;            
            } 
            else {
                this.gradient = this.coolGradient;
                this.recommendation = this.coolRecommendation;                           
            }
        }
...

Scale the idea

Machine Learning + Mobile = ❤️

think of the possibilities for photos, video, audio

not

What even is machine learning?

a way to give “computers the ability to learn without being explicitly programmed.”

Machine Learning is:

"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.” (Tom Mitchell, 1997).

How to make a machine learn*

• The training set is categorized (sorted by hand or by machine)
• The test set is uncategorized

*"supervised learning"

ML in the wild

Good uses of ML

Scary uses of ML

good and bad?

MIT students used an algorithm to optimize school bus routes

Microsoft 👏

"snow leopard or not" - partnership with the Snow Leopard Trust

Microsoft 👏

"AI For Earth"

https://www.microsoft.com/en-us/aiforearth

DIY Machine Learning is hard

you need a lot of firepower & skillz

probably not!

might be!

takePhoto() {
        
        const options: camera.CameraOptions = {
            width: 300,
            height: 300,
            keepAspectRatio: true,
            saveToGallery: false
        };

        camera.takePicture(options)
        .then((imageAsset: ImageAsset) => {
            this.processRecipePic(imageAsset);
        }).catch(err => {
            console.log(err.message);
        });
    
    }

    public queryClarifaiAPI(imageAsBase64):Promise<any>{
        return http.request({
            url: AuthService.clarifaiUrl,
            method: "POST",
            headers: {
                "Content-Type": "application/json",
                 "Authorization": "Key " + AuthService.clarifaiKey,
            },
            content: JSON.stringify({
                "inputs": [{
                    "data": {
                        "image": {
                            "base64": imageAsBase64
                        }
                    }
                }]
            })
        })
      .then(function (response) {
          return response
        }
      )}

.then(res => {
   this.loader.hide();
      try {
          let result = res.content.toJSON();
          let tags = result.outputs[0].data.concepts.map( mc => mc.name + '|' + mc.value );
          let ingredients = [];
          tags.forEach(function(entry) {
              let prob = entry.split('|');
              prob = prob[1];
              let ingred = entry.split('|');
                 if(prob > 0.899){
                    ingredients.push(ingred[0])
                 }
                                    
              });
            //there should be between four and eight discernable ingredients                     
            if (ingredients.length >= 4 && ingredients.length <= 8) {
               alert("Yes! This dish might qualify as a QuickNom! It contains "+ingredients)
            }
            else {
               alert("Hmm. This recipe doesn't have the qualifications of a QuickNom. 
                    Try again!")
            }
       }

if between 4 & 8 ingredients are listed with over .899 certainty,

it's a QuickNom!

QuickNom dishes have a few easy-to-see, simple ingredients

Use Google's Vision API to match images with recipes

• Label Detection
• Explicit Content Detection
• Logo Detection
• Landmark Detection
• Face Detection
• Web Detection (search for similar)

takePhoto() {
        
        const options: camera.CameraOptions = {
            width: 300,
            height: 300,
            keepAspectRatio: true,
            saveToGallery: false
        };

        camera.takePicture(options)
        .then((imageAsset: ImageAsset) => {
            this.processItemPic(imageAsset);
        }).catch(err => {
            console.log(err.message);
        });
    
    }

public queryGoogleVisionAPI(imageAsBase64: string):Promise<any>{
        return http.request({
            url: "https://vision.googleapis.com/v1/images:annotate?key="+AuthService.googleKey,
            method: "POST",
            headers: {
                "Content-Type": "application/json",
                "Content-Length": imageAsBase64.length,
            },
            content: JSON.stringify({
                "requests": [{
                "image": {
                "content": imageAsBase64 
                },
                "features" : [
                    {
                        "type":"LABEL_DETECTION",
                        "maxResults":1
                    }
                ]                      
            }]
        })
      })
      .then(function (response) {
          return response
        }
      )}

this.mlService.queryGoogleVisionAPI(imageAsBase64)
    .then(res => {
        let result = res.content.toJSON();
        this.ingredient = result.responses[0].labelAnnotations.map( mc => mc.description );                                                               
        this.ngZone.run(() => {
            this.searchRecipes(this.ingredient)
        })
    });

DIY machine learning

made a little easier!

Now landed in iOS 11: Core ML

Train a model externally, let Core ML process it for your app on device

TensorFlow Mobile (v1)

Designed for low-end Androids, works for iOS and Android

New! Hot! TensorFlow Lite!

next-gen version of TensorFlow for mobile

"on-device machine learning inference with low latency and a small binary size."

TensorFlow Lite for iOS

you have the option to convert to CoreML!

Featuring:

• a new model file format, based on "FlatBuffers" - smaller/faster/more memory efficient than ProtocolBuffers
• new mobile-optimized interpreter
• an interface to leverage hardware acceleration (Android)
• small footprint! 75 - 400 kb!

Watch this project! Coming soon: train ON DEVICE

TensorFlow powers Google Translate!

TensorFlow Mobile on iOS