The Great

Minibar Challenge

Supervised Drinking with Jen

Hello!

Jen Looper

Cloud Developer Advocate Lead
Microsoft

Founder/CEO: Vue Vixens 🦊

👑

@jenlooper

inventor of the Vuetini

"Queen of the apps"

(according to Wellesley/Weston

Magazine,

so it must be true)

I travel a lot

I stay in a lot of hotels

I

❤️
mobile

apps

put it all together...

What if we could create a mobile app...🤔

that would be able to ease my life in a hotel...
...by making sense of the minibar by suggesting cocktails to make with available ingredients
...do it in NativeScript-Vue
...make it an 'intelligent' app
...have some fun while you're at it!

Let's make an AI!

👊

Step 1: gather data

Let's try Kaggle - The Data Scientist's Heaven

500 crowdsourced cocktails scraped from thecocktaildb.com

not enough cocktailing going on here

a clue from Beth Skwarecki, creator of @botcocktails

Old Mr. Boston's Official Bartending Guide, 1935

Zomg there's a web site

...with a scrapable API

All ur cocktails belong to me

back to Kaggle

Step 2: Time to build a

mobile app

Use the NativeScript CLI: tns create mixology

Scaffold an app

run it in emulators

fix up the UI

Presenting...MixoLogy

icon

logo

home

Step 3: Leverage that data

"I have Tequila, what can I make?"

(hint, this does not need machine learning)

a job for...

let's use Firebase

❤️

upload the cocktails to Firebase

From Excel to JSON

Integrate Firebase

import Vue from 'nativescript-vue'

import firebase from 'nativescript-plugin-firebase'

firebase
	.init({
		
	})
	.then(
	    instance => {
		console.log('firebase.init done')
	    },
	    error => {
	        console.log(`firebase.init error: ${error}`)
	    }
	);

Vue.prototype.$firebase = firebase

...

new Vue({
	render: h => h('frame', [h(Home)]),
}).$start()

main.js

firebase plugin

demo

select an ingredient, show appropriate recipes from Firebase

we can do better

💪

💡

Let's run the bottles through an image-recognition service!

Let's try Google Cloud Vision API

pre-trained generic model

How about Clarifai?

Microsoft?

This is a job for

a custom-trained model!

(what is this model of which you speak?)

a mathematical model of sample data built using a machine learning algorithm in order to make predictions without being explicitly programmed

Step 4: Train your model

no reason to reinvent the wheel:

Google Codelab "TensorFlow for Poets"

You need:

Local installation of TensorFlow

Python

A bunch of images saved in folders

Python scripts from the Codelab

🔥Hot tip

Finding enough images is TIME CONSUMING

use ffmpeg to convert movies to images: get 100+ images easily

Using MobileNet

retrain on your custom categories

MobileNet

Test your new model

Step 5: Use that model

😢 this is the worst part

convert the files

TensorFlow produces a 'float-trained' .pb model file - convert it to quantized* .tflite for use on mobile

*quantized format - you'll lose accuracy 😢

Convert the model

IMAGE_SIZE=224
toco \
--graph_def_file=tf_files/retrained_graph.pb \
--output_file=tf_files/retrained_graph.tflite \
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TFLITE \
--inference_type=QUANTIZED_UINT8 \
--input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3 \
--input_array=input \
--output_array=final_result \
--mean_value=128 \
--std_dev_values=128 \
--default_ranges_min=0 \
--default_ranges_max=6

now you have a TensorFlow lite file + a list of labels

verify that the .tflite file is quantized

interpreter = tf.contrib.lite.Interpreter(model_path="./tf_files/retrained_graph.tflite")
interpreter.allocate_tensors()

# Print input shape and type
print(interpreter.get_input_details()[0]['shape'])  
print(interpreter.get_input_details()[0]['dtype'])  

# Print output shape and type
print(interpreter.get_output_details()[0]['shape']) 
print(interpreter.get_output_details()[0]['dtype'])

[  1 224 224 3 ]
<class 'numpy.uint8'>
[ 1 17 ]
<class 'numpy.uint8'>

NOW use it in your app!

takePicture() {
      this.ingredients = []
      this.pictureFromCamera = null
      camera
        .takePicture({ width: 224, height: 224, keepAspectRatio: true })
        .then(imageAsset => {
          new ImageSource().fromAsset(imageAsset).then(imageSource => {
            this.pictureFromCamera = imageSource;
            setTimeout(() => this.queryMLKit(imageSource), 500)
          });
        });
    },

query ML Kit

  queryMLKit(imageSrc) {
      this.$firebase.mlkit.custommodel
        .useCustomModel({
          image: imageSrc,
          localModelFile: "~/assets/models/retrained_graph.tflite",
          labelsFile: "~/assets/models/retrained_labels.txt",
          maxResults: 5,
          modelInput: [
            {
              shape: [1, 224, 224, 3],
              type: "QUANT"
            }
          ]
        }) 
        .then(result => {
          
            for (var i=0; i<result.result.length; i++){
              this.ingredients.push(result.result[i])
            }
            
        })
        ...
    },

then query Firebase for matching recipes

demo

Yay! Custom models

but we can do more!

🔥

AIs should be "weird"

aiweirdness.com by Janelle Shane

🤖 textgenrnn

a text-generating neural network that generates words that seem to make sense, sort of, in context

The AI iterates through your set of words and phrases until it finds patterns and then generates weird stuff

generated ice cream flavors

generated cookies

generated knitting patterns

let's do this

run our cocktail recipes through textgenrnn

https://github.com/minimaxir/textgenrnn

Clean the data (one more time)

each recipe in one block in a text file:

Gauguin,2 oz Light Rum,1 oz Passion Fruit Syrup,1 oz Lemon Juice,1 oz Lime Juice,Combine ingredients with a cup of crushed ice in blender and blend at low speed. Serve in old-fashioned glass. Top with a cherry.

Fort Lauderdale,1 1/2 oz Light Rum,1/2 oz Sweet Vermouth,1/4 oz Juice of Orange,1/4 oz Juice of a Lime,Shake with ice and strain into old-fashioned glass over ice cubes. Add a slice of orange.

Apple Pie,3 oz Apple schnapps,1 oz Cinnamon schnapps, Apple slice,Pour into ice-filled old-fashioned glass. Garnish with apple and top with cinnamon.

Cuban Cocktail No. 1,1/2 oz Juice of a Lime,1/2 oz Powdered Sugar,2 oz Light Rum,Shake with ice and  strain into cocktail glass.

feed it to textgenrnn locally

LOOPERMAC:textgenrnn Looper$ python3
>>> from textgenrnn import textgenrnn
Using TensorFlow backend.
>>> textgen = textgenrnn()
>>> textgen.train_from_file('datasets/cocktails_simpler_text.txt',num_epochs=1)
1,971 texts collected.
Training on 184,914 character sequences.
Epoch 1/1
1444/1444 [==============================] - 282s 196ms/step - loss: 0.8350
####################
Temperature: 1.0
####################
Rumed Cftsuismenther,1 oz Light Rum,1/2 oz Citrus-Brandy,1 oz Capbain,Shake with ice and strain into chilled old-fashioned glass.

Anderry Gcal Gin,1/2 oz Orange Bitters, Lemon twist,Shake with ice and strain into cocktail glass.

🍹👍

Generate 500 fake cocktails & import them to the app

parseCocktail(cocktailObject) {
      let newCocktail = JSON.stringify(cocktailObject.cocktail)
      this.selectedCocktail = []
      let recipe = newCocktail.split(",")
      for (var i = 0; i < recipe.length; i++) {
        this.selectedCocktail.push(recipe[i].replace('"', ""))
      }
    }

enable your accelerometer and shake to mix yourself a drink!

demo

Thank you!

@jenlooper

github.com/jlooper/mixology-mobile

🙏 gratitude to community members Igor Randjelovic, Eddy Verbruggen, and Jona Rodrigues

The Great

Minibar Challenge

Hello!

Jen Looper

Cloud Developer Advocate Lead Microsoft

Founder/CEO: Vue Vixens 🦊

inventor of the Vuetini

"Queen of the apps"

I travel a lot

I stay in a lot of hotels

I

❤️mobile

apps

put it all together...

What if we could create a mobile app...🤔

Let's make an AI!

👊

Step 1: gather data

not enough cocktailing going on here

a clue from Beth Skwarecki, creator of @botcocktails

Old Mr. Boston's Official Bartending Guide, 1935

Zomg there's a web site

...with a scrapable API

All ur cocktails belong to me

Step 2: Time to build a

mobile app

Scaffold an app

run it in emulators

fix up the UI

Step 3: Leverage that data

let's use Firebase

upload the cocktails to Firebase

Integrate Firebase

demo

select an ingredient, show appropriate recipes from Firebase

we can do better

💪

💡

Let's run the bottles through an image-recognition service!

Let's try Google Cloud Vision API

How about Clarifai?

Microsoft?

This is a job for

a custom-trained model!

(what is this model of which you speak?)

Step 4: Train your model

You need:

Local installation of TensorFlow

Python

A bunch of images saved in folders

Python scripts from the Codelab

🔥Hot tip

Using MobileNet

retrain on your custom categories

MobileNet

Test your new model

Step 5: Use that model

convert the files

TensorFlow produces a 'float-trained' .pb model file - convert it to quantized* .tflite for use on mobile

Convert the model

now you have a TensorFlow lite file + a list of labels

NOW use it in your app!

query ML Kit

demo

Yay! Custom models

but we can do more!

🔥

AIs should be "weird"

🤖 textgenrnn

a text-generating neural network that generates words that seem to make sense, sort of, in context

generated ice cream flavors

generated cookies

generated knitting patterns

let's do this

run our cocktail recipes through textgenrnn

Clean the data (one more time)

each recipe in one block in a text file:

feed it to textgenrnn locally

🍹👍

Generate 500 fake cocktails & import them to the app

Cloud Developer Advocate Lead
Microsoft

❤️
mobile