The Great

Minibar Challenge

Supervised Drinking with Jen


Jen Looper

Cloud Developer Advocate Lead

Founder/CEO: Vue Vixens 🦊



inventor of the Vuetini

"Queen of the apps"

(according to Wellesley/Weston


so it must be true)

I travel a lot

I stay in a lot of hotels




put it all together...

What if we could create a mobile app...🤔

  • that would be able to ease my life in a hotel...
  • making sense of the minibar by suggesting cocktails to make with available ingredients
  • it in NativeScript-Vue
  • ...make it an 'intelligent' app
  • ...have some fun while you're at it!

Let's make an AI!


Step 1: gather data

Let's try Kaggle - The Data Scientist's Heaven

500 crowdsourced cocktails scraped from

not enough cocktailing going on here

a clue from Beth Skwarecki, creator of @botcocktails

Old Mr. Boston's Official Bartending Guide, 1935

Zomg there's a web site

...with a scrapable API

All ur cocktails belong to me

back to Kaggle

Step 2: Time to build a

mobile app

Use the NativeScript CLI: tns create mixology

Scaffold an app

run it in emulators

fix up the UI





Step 3: Leverage that data

"I have Tequila, what can I make?"

(hint, this does not need machine learning)

a job for...

let's use Firebase


upload the cocktails to Firebase

From Excel to JSON

Integrate Firebase

import Vue from 'nativescript-vue'

import firebase from 'nativescript-plugin-firebase'

	    instance => {
		console.log('firebase.init done')
	    error => {
	        console.log(`firebase.init error: ${error}`)

Vue.prototype.$firebase = firebase


new Vue({
	render: h => h('frame', [h(Home)]),


firebase plugin


select an ingredient, show appropriate recipes from Firebase

we can do better



Let's run the bottles through an image-recognition service!

Let's try Google Cloud Vision API

pre-trained generic model

How about Clarifai?


This is a job for

a custom-trained model!

(what is this model of which you speak?)

 a mathematical model of sample data built using a machine learning algorithm in order to make predictions without being explicitly programmed

Step 4: Train your model

no reason to reinvent the wheel:

Google Codelab "TensorFlow for Poets"

You  need:

Local installation of TensorFlow


A bunch of images saved in folders

Python scripts from the Codelab

🔥Hot tip

Finding enough images is TIME CONSUMING

use ffmpeg to convert movies to images: get 100+ images easily

Using MobileNet

retrain on your custom categories


Test your new model

Step 5: Use that model

😢 this is the worst part

convert the files

TensorFlow produces  a 'float-trained' .pb model file - convert it to quantized* .tflite for use on mobile

*quantized format - you'll lose accuracy 😢

Convert the model


toco \
--graph_def_file=tf_files/retrained_graph.pb \
--output_file=tf_files/retrained_graph.tflite \
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TFLITE \
--inference_type=QUANTIZED_UINT8 \
--input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3 \
--input_array=input \
--output_array=final_result \
--mean_value=128 \
--std_dev_values=128 \
--default_ranges_min=0 \

now you have a TensorFlow lite file + a list of labels

verify that the .tflite file is quantized

interpreter = tf.contrib.lite.Interpreter(model_path="./tf_files/retrained_graph.tflite")

# Print input shape and type

# Print output shape and type
[  1 224 224 3 ]
<class 'numpy.uint8'>
[ 1 17 ]
<class 'numpy.uint8'>

NOW use it in your app!

takePicture() {
      this.ingredients = []
      this.pictureFromCamera = null
        .takePicture({ width: 224, height: 224, keepAspectRatio: true })
        .then(imageAsset => {
          new ImageSource().fromAsset(imageAsset).then(imageSource => {
            this.pictureFromCamera = imageSource;
            setTimeout(() => this.queryMLKit(imageSource), 500)

query ML Kit

  queryMLKit(imageSrc) {
          image: imageSrc,
          localModelFile: "~/assets/models/retrained_graph.tflite",
          labelsFile: "~/assets/models/retrained_labels.txt",
          maxResults: 5,
          modelInput: [
              shape: [1, 224, 224, 3],
              type: "QUANT"
        .then(result => {
            for (var i=0; i<result.result.length; i++){

then query Firebase for matching recipes


Yay! Custom models

but we can do more!


AIs should be "weird" by Janelle Shane

🤖 textgenrnn

a text-generating neural network that generates words that seem to make sense, sort of, in context

The AI iterates through your set of words and phrases until it finds patterns and then generates weird stuff

generated ice cream flavors

generated cookies

generated knitting patterns

let's do this

run our cocktail recipes through textgenrnn

Clean the data (one more time)

each recipe in one block in a text file:

Gauguin,2 oz Light Rum,1 oz Passion Fruit Syrup,1 oz Lemon Juice,1 oz Lime Juice,Combine ingredients with a cup of crushed ice in blender and blend at low speed. Serve in old-fashioned glass. Top with a cherry.

Fort Lauderdale,1 1/2 oz Light Rum,1/2 oz Sweet Vermouth,1/4 oz Juice of Orange,1/4 oz Juice of a Lime,Shake with ice and strain into old-fashioned glass over ice cubes. Add a slice of orange.

Apple Pie,3 oz Apple schnapps,1 oz Cinnamon schnapps, Apple slice,Pour into ice-filled old-fashioned glass. Garnish with apple and top with cinnamon.

Cuban Cocktail No. 1,1/2 oz Juice of a Lime,1/2 oz Powdered Sugar,2 oz Light Rum,Shake with ice and  strain into cocktail glass.

feed it to textgenrnn locally

LOOPERMAC:textgenrnn Looper$ python3
>>> from textgenrnn import textgenrnn
Using TensorFlow backend.
>>> textgen = textgenrnn()
>>> textgen.train_from_file('datasets/cocktails_simpler_text.txt',num_epochs=1)
1,971 texts collected.
Training on 184,914 character sequences.
Epoch 1/1
1444/1444 [==============================] - 282s 196ms/step - loss: 0.8350
Temperature: 1.0
Rumed Cftsuismenther,1 oz Light Rum,1/2 oz Citrus-Brandy,1 oz Capbain,Shake with ice and strain into chilled old-fashioned glass.

Anderry Gcal Gin,1/2 oz Orange Bitters, Lemon twist,Shake with ice and strain into cocktail glass.


Generate 500 fake cocktails & import them to the app 

parseCocktail(cocktailObject) {
      let newCocktail = JSON.stringify(cocktailObject.cocktail)
      this.selectedCocktail = []
      let recipe = newCocktail.split(",")
      for (var i = 0; i < recipe.length; i++) {
        this.selectedCocktail.push(recipe[i].replace('"', ""))

enable your accelerometer and shake to mix yourself a drink!


Thank you!



🙏 gratitude to community members Igor Randjelovic, Eddy Verbruggen, and Jona Rodrigues

Made with