Introduction to Actions on Google
& Assistant Raspbian
Baskin Tapkan
February 10, 2018
Google DevFestMN
About me...
Sr. Software Engineer at ReachLocal (part of USA Today)
Interests in:
- Code, Technology, Le Web, Mentoring, pairing
- Wind-powered & water/snow sports (Windsurfing, Kite Boarding & Snow kiting)
- Biking, Hiking, Camping, Traveling
- Senior IEEE member
Building Blocks
-
A developer project of type Actions - tracks and manages through the approval process and lets administers actions with support for analytics and Actions Simulator (test w/o an actual device)
-
Action Package - defines metadata about the actions in the app such as invocations and fulfillment service.
-
Fulfillment - defines the actual functionality of the app, receiving requests and sending responses back via the Google Assistant brokered communication
Fulfillment
Process Flow
Conversational UI
-
Taking turns in a dialog
-
Tracking the conversation flow via thread
-
Language Efficiency - reading between the lines
- Guessing user behavior and earning trust
Six Steps of Conversation
-
Open a channel to set up common ground (A => B)
-
Commit to engage (B commits to conv. w/ A)
-
Construct meaning ( A & B = {set of ideas, context })
-
Evolve ( A & B learned/gained something)
-
Reach an agreement, or move to repair
-
Act or interact (turn things on/off etc)
Cooperative Conversation
-
Quality - Only say things that are true
-
Quantity - Don't be more or less informative than needed
-
Relevance - Only say things that are relevant
- Manner - Be brief, get to the point and avoid ambiguity
Designing an Action
-
Picking the right use cases
-
Creating a persona
-
Writing dialog
- Testing and iterate
Persona Profile
Assistant App
// Assistant App is a base class of:
// 1. Actions SDK
const app = new ActionsSdkApp({request: request, response: response});
app.userStorage.someProperty = 'someValue';
// 2. Dialogflow
const app = new DialogflowApp({request: request, response: response});
app.userStorage.someProperty = 'someValue';- Asking user for delivery address
- Asking for date/time
- Asking for confirmation
- Asking for new surface (eg Android, G. Home type device)
....
Dialogflow
- Agents - NLU modules transform user requests to actionable data
- Intents - represent a mapping between user utterance and the action to be taken
- Entities - used to extract parameter values from language inputs
Dialogflow
Actions SDK
Actions SDK
- "Ok Google, talk to My Favorite Shoe Store"
- "Ok Google, talk to My Favorite Shoe Store to buy some shoes"
- "Ok Google, talk to My Favorite Shoe Store to check on my order"
- "Ok Google, talk to My Favorite Shoe Store to show me today's deals"
Create actions in a package by mapping an intent to fulfillment.
An action defines an entry point into app's functionality and maps an intent to process.
gactions CLI
Command Line Interface to test and update the app for the Assistant
$ gactions help
$ gactions help ARGUMENT
$ gactions help deploy
$ gactions init
$ gactions init --force
$ gactions list --project PROJECT_ID
$ gactions test --action_package PACKAGE_NAME --project PROJECT_ID
$ gactions update --action_package PACKAGE_NAME --project PROJECT_ID
$ gactions selfupdateFulfillment
functions
hosting
database
storage
Firebase
firebase CLI
// make sure nodejs and npm are installed and up-to-date
$ npm install firebase-functions@latest firebase-admin@latest --save
$npm install -g firebase-tools
// creates a .firebaserc file
$ firebase init functions
// deploys and provides you a URL to copy & paste in Fulfillment
$ firebase deploy --only functions
DEMO
Google on Assistant on Raspberry PI
USB Console Cable
USB Microphone
Speaker (USB or 3.5mm-jack)
Serial Console & Drivers
Enable serial console on Raspberry PI
// add this line at the very end of /boot/config.txt
...
enable_uart=1Install drivers for Prolific & SiLabs for your flavor OS (Mac / Windows / Linux) - better to reboot after install
The driver on the website didn't work for Win10 and also need putty.exe
Audio & Speakers
Configure & Test the Audio
$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 1: Device [USB PnP Sound Device], device 0: USB Audio [USB Audio]
Subdevices: 1/1
Subdevice #0: subdevice #0Locate the Speaker in the list
$aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: ALSA [bcm2835 ALSA], device 0: bcm2835 ALSA [bcm2835 ALSA]
Subdevices: 8/8
...
card 0: ALSA [bcm2835 ALSA], device 1: bcm2835 ALSA [bcm2835 IEC958/HDMI]
Subdevices: 1/1
Subdevice #0: subdevice #0Project Configuration
Python Env & Package
# Python 3
$ sudo apt-get update
$ sudo apt-get install python3-dev python3.4-venv
$ python3 -m venv env
$ env/bin/python -m pip install --upgrade pip setuptools wheel
$ source env/bin/activate
# get the package
(env) $ sudo apt-get install portaudio19-dev libffi-dev libssl-dev
# pip install the latest
(env) $ python -m pip install --upgrade google-assistant-library
(env) $ python -m pip install --upgrade google-assistant-sdk[samples]Configure Python env and get the package
Auth Tool & Creds
# Install the authorization tool
(env) $ python -m pip install --upgrade google-auth-oauthlib[tool]
# Generate credentials
(env) $google-oauthlib-tool --scope https://www.googleapis.com/auth/assistant-sdk-prototype \
--save --headless --client-secrets /path/to/client_secret_client-id.json
# Please visit this URL to authorize this application: https://...
# Copy and paste the auth code from above
Enter the authorization code:
# SUCCESS
credentials saved: /path/to/.config/google-oauthlib-tool/credentials.json
# DO NOT RENAME credentials**.json fileRegister & Run Code
Use the Registration Tool
(env) $ googlesamples-assistant-devicetool register-model \
--manufacturer raspberry-pi-devfestmn \
--product-name whitaker-home \
--description home-automation \
--type LIGHT \
--model whitaker-home-m1List Device Models
(env) $ googlesamples-assistant-devicetool list --model(env) $ googlesamples-assistant-hotword \
--project_id whitaker-home \
--device_model_id whitaker-home-m1Run the code
PI -DEMO
If-this-then-that
Observations
- Catching up to Alexa (skills) in development, but query tooling is already better than Alexa. Kudos to Google!
- Development experience is pretty good, but there are a few gotchas and head scratchers
- Test Simulator is behaving unexpectedly. Saving `Test Draft` and when the test is active cannot be predicted
- Sample apps are quite decent and more coming! Stay up-to-date. Check back frequently.
Thank you
In Loving Memory of Mr. Baxter...
[April, 2005 - January 2018]
Departing Thoughts
- Future is not just CRUD applications anymore (jk - still is but on new ~surfaces~ )
- Needs more context and personalization - such as a picture or video with the user request
- Collaborate, share your projects and learn from others. Join the communities and still check
- Don't Panic & carry a towel