@jenlooper
Jen Looper
Progress
Senior Developer Advocate
Founder and CEO: Vue Vixens
@jenlooper
🤖
@jenlooper
What is Machine Learning?
Why is Machine Learning so important?
How can Machine Learning help make more interesting mobile apps?
How can I build an app using two new technologies: ML Kit and NativeScript-Vue?
@jenlooper
not
@jenlooper
a way to give “computers the ability to learn without being explicitly programmed.”
@jenlooper
"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.” (Tom Mitchell, 1997).
@jenlooper
@jenlooper
Gather data
*"supervised learning"
@jenlooper
🍎🍊🥕
training set
🍑🍐🍍
test set
fruits: [🍎,🍊]
veggies: [🥕,🌽]
no categories
🤖💭
🍎: "this is an apple!"
🍎: "right!"
🐕: "this is an apple!"
🐕: "uhh..."
80% accurate... try again!
*"unsupervised learning"
@jenlooper
Gather data
@jenlooper
[🍲,🥨,🌭,🍣]
no categories!
🤖🎓💭
a self-teaching pattern-recognizing algorithm
🥨: "this is a pretzel!"
🥨: "right!"
🌭: "this is a pretzel!"
🌭: "uhh..."
80% accurate... try again!
*"reinforcement learning"
Gather data
[🍲,🥨,🌭,🍣]
no categories!
🤖🕶🎓💭
a 'curious' self-teaching pattern-recognizing algorithm*
🥨: "this is a pretzel!"
🥨: "right!"
🌭: "this is a chili dog!"
🌭: "actually, yeah!"
80% accurate... try again!
*give it harder and harder problems, reward its curiosity, like a child
a subfield of machine learning
uses ML algorithms in layers to create an artificial neural network to make independent decisions
@jenlooper
TensorFlow, Google's library for machine learning
@jenlooper
bit.ly/tf-4-poets
A JavaScript library for training and deploying ML models in the browser and on Node.js
https://js.tensorflow.org/
🔖Bookmark TensorFlow for now!
@jenlooper
StitchFix combines ML + human curation
Runs input through algorithm
connects shopper to stylist
🤖
👱♀️
👩💻
indicates style preferences
builds recommendations - factors in location, trends, season
curates and ships box
gives feedback:
👍buys
or
👎returns items
@jenlooper
when you install a ton of surveillance cameras, you can...
get really good at facial recognition algorithms
match faces to IDs
monitor group emotions
track location
@jenlooper
🤖🎥
@jenlooper
🚓 New York and Miami: ML for crime forecasting
@jenlooper
🚑 Northshore University Health System: ML to predict cardiac arrest using electronic medical records.
👩🎓Tulsa school systems: ML for early intervention to prevent students from dropping out of school or graduating late
@jenlooper
mini collectors-of-data
packed with sensors
supercharged with mobile apps
make your apps 'smarter' - more helpful
build a whole app around ML (SeeingAI, Shazam)
Facial skin beautification via sparse representation over learned layer dictionary
...or add little ML touches
@jenlooper
@jenlooper
@jenlooper
did I leave the oven on?
@jenlooper
an open source framework for building truly native mobile apps with JavaScript. Use web skills, like TypeScript, Angular, Vue, and CSS, and get native UI and performance on iOS and Android.
@jenlooper
Rich, animated, “no compromise” native UI
(with shared UI code)
@jenlooper
You know JavaScript? You know NativeScript!
@jenlooper
📱Vue's 2.0 adoption of the virtual DOM enables native mobile rendering (Angular 2+ is similar)
💻 Vue offers a great way for web developers to embrace mobile platforms via NativeScript or Weex
☁️ Vue is lightweight so highly appropriate for mobile
🤝 NativeScript and Vue have great code-sharing potential!
@jenlooper
https://play.nativescript.org/
@jenlooper
+
@jenlooper
💸What if you don't want to make a bunch of costly REST API calls?
🔌What if you need offline capability?
🎓What if you need to train something really custom?
@jenlooper
TensorFlow Lite
Designed for low-end Androids, works for iOS and Android
@jenlooper
@jenlooper
@jenlooper
Identify objects, locations, activities, animal species, products++
Recognize and extract text from images
Detect faces and facial landmarks
Scan and process barcodes
Identify popular landmarks in an image
@jenlooper
@jenlooper
Picme: A game testing ML Kit accuracy and speed
image collections by Karen Zack @teenybiscuit
image collections by Karen Zack @teenybiscuit
use the Vue CLI with a template
vue init nativescript-vue/vue-cli-template picme
import firebase from 'nativescript-plugin-firebase';
firebase.init({
// Optionally pass in properties for db, auth & cloud messaging
}).then(
instance => {
console.log("firebase.init done");
},
error => {
console.log(`firebase.init error: ${error}`);
}
);
...
Vue.prototype.$firebase = firebase;
<SwipeLayout row="2" col="0" v-for="item in count" :key="item"
@swipeLeft="swipeCallback($event,item)" @swipeRight="swipeCallback($event,item)">
<GridLayout rows="2*,2*" columns="*">
<CardView row="0" elevation="20" margin="10" radius="10"
backgroundColor="white" shadowOpacity=".2" shadowRadius="5">
<Image :src="'~/images/'+imageFolder+'/'+item+'.jpg'"/>
</CardView>
...
</GridLayout>
</SwipeLayout>
queryMLKit(id){
this.response = ["thinking..."];
...
let image = '/images/'+this.imageFolder+'/'+this.currImage+'.jpg';
...
source.fromFile(path)
.then((imageSource) => {
this.$firebase.mlkit.imagelabeling.labelImageOnDevice({
image: { imageSource: source },
confidenceThreshold: 0.6
})
.then((result) => {
...
for (let i = 0; i < result.labels.length; ++i) {
tempArr.push(
result.labels[i].text + ' : ' + Math.floor((result.labels[i].confidence) * 100) + '%'
);
...
}
this.calculateBotScore(this.imageFolder,this.currImage,labelsArr)
...
}
)
...
})
},
query ML Kit
export const Images = [
{
type: "barnowl",
ids: [1,4,5,7,10,12,13,15],
acceptedAnswers: ["bird","owl"]
},
{
type: "apple",
ids: [2,3,6,8,9,11,14,16],
acceptedAnswers: ["food","apple"]
},
...
]
const state = {
humanScore: 0,
botScore: 0
};
const mutations = {
clearScore (state){
state.humanScore = 0;
state.botScore = 0;
},
setHumanScore (state) {
state.humanScore++;
},
setBotScore (state) {
state.botScore++;
}
};
const actions = {
clearScore: ({commit}) => commit('clearScore'),
setHumanScore: ({commit}) => commit('setHumanScore'),
setBotScore: ({commit}) => commit('setBotScore'),
};
...
🤖 vs. 👩
@jenlooper
nativescript-vue.org