Introduction to Actions on Google
& Assistant Raspbian

Baskin Tapkan

February 10, 2018

Google DevFestMN

@baskint

About me...

Interests in:

• Code, Technology, Le Web, Mentoring, pairing
• Wind-powered &  water/snow sports (Windsurfing, Kite Boarding & Snow kiting)
• Biking, Hiking, Camping, Traveling
• Senior IEEE member 

Conversational UI

• Taking turns in a dialog
   
• Tracking the conversation flow via thread
   
• Language Efficiency - reading between the lines
   
• Guessing user behavior and earning trust

https://developers.google.com/actions/design/

Six Steps of Conversation

1. Open a channel to set up common ground (A => B)
    
2. Commit to engage (B commits to conv. w/ A)
    
3. Construct meaning  ( A & B = {set of ideas, context })
    
4. Evolve ( A & B learned/gained something)
    
5. Reach an agreement, or move to repair
    
6. Act or interact (turn things on/off etc)
    

https://developers.google.com/actions/design/how-conversations-work

Cooperative Conversation

• Quality - Only say things that are true
   
• Quantity - Don't be more or less informative than needed
   
• Relevance - Only say things that are relevant
   
• Manner - Be brief, get to the point and avoid ambiguity​

https://developers.google.com/actions/design/be-cooperative

Designing an Action

• Picking the right use cases
   
• Creating a persona
   
• Writing dialog
   
• Testing and iterate

https://developers.google.com/actions/design/principles

https://www.nngroup.com/articles/why-personas-fail/

Persona Profile

Dialogflow

• Agents - NLU modules transform user requests to actionable data
   
• Intents - represent a mapping between user utterance and the action to be taken
   
• Entities - used to extract parameter values from language inputs

https://dialogflow.com/docs/getting-started/basics

Dialogflow

Actions SDK

Actions SDK

• "Ok Google, talk to My Favorite Shoe Store"
• "Ok Google, talk to My Favorite Shoe Store to buy some shoes"
• "Ok Google, talk to My Favorite Shoe Store to check on my order"
• "Ok Google, talk to My Favorite Shoe Store to show me today's deals"

Create actions in a package by mapping an intent to fulfillment.
An action defines an entry point into app's functionality and maps an intent to process.

gactions CLI

Command Line Interface to test and update the app for the Assistant

https://developers.google.com/actions/tools/gactions-cli

$ gactions help
$ gactions help ARGUMENT
$ gactions help deploy
$ gactions init
$ gactions init --force
$ gactions list --project PROJECT_ID
$ gactions test --action_package PACKAGE_NAME --project PROJECT_ID
$ gactions update --action_package PACKAGE_NAME --project PROJECT_ID
$ gactions selfupdate

Fulfillment

functions

hosting

database

storage

Firebase

firebase CLI

// make sure nodejs and npm are installed and up-to-date
$ npm install firebase-functions@latest firebase-admin@latest --save

$npm install -g firebase-tools

// creates a .firebaserc file
$ firebase init functions 

// deploys and provides you a URL to copy & paste in Fulfillment
$ firebase deploy --only functions

Google on Assistant on Raspberry PI

USB Console Cable

https://learn.adafruit.com/adafruits-raspberry-pi-lesson-5-using-a-console-cable/connect-the-lead

USB Microphone

Speaker (USB or 3.5mm-jack)

https://www.amazon.com/AYL-Guarantee-Rechargeable-Expandable-Smartphones/dp/B007OYAVLI

https://www.amazon.com/gp/product/B00IR8R7WQ/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

Serial Console & Drivers 

https://developers.google.com/assistant/sdk/guides/library/python/embed/setup

Enable serial console on Raspberry PI

// add this line at the very end of /boot/config.txt

...
enable_uart=1

Install drivers for Prolific & SiLabs for your flavor OS (Mac / Windows / Linux) - better to reboot after install

The driver on the website didn't work for Win10 and also need putty.exe

https://learn.adafruit.com/adafruits-raspberry-pi-lesson-5-using-a-console-cable/test-and-configure

Audio & Speakers

Configure & Test the Audio

Locate the Speaker in the list

$aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: ALSA [bcm2835 ALSA], device 0: bcm2835 ALSA [bcm2835 ALSA]
  Subdevices: 8/8
...
card 0: ALSA [bcm2835 ALSA], device 1: bcm2835 ALSA [bcm2835 IEC958/HDMI]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

Project Configuration

https://developers.google.com/assistant/sdk/guides/library/python/embed/config-dev-project-and-account

Python Env & Package

https://developers.google.com/assistant/sdk/guides/library/python/embed/install-sample

# Python 3
$ sudo apt-get update
$ sudo apt-get install python3-dev python3.4-venv 
$ python3 -m venv env
$ env/bin/python -m pip install --upgrade pip setuptools wheel
$ source env/bin/activate

# get the package
(env) $ sudo apt-get install portaudio19-dev libffi-dev libssl-dev

# pip install the latest
(env) $ python -m pip install --upgrade google-assistant-library
(env) $ python -m pip install --upgrade google-assistant-sdk[samples]

Configure Python env and get the package

Auth Tool & Creds

https://developers.google.com/assistant/sdk/guides/library/python/embed/install-sample

# Install the authorization tool
(env) $ python -m pip install --upgrade google-auth-oauthlib[tool]

# Generate credentials
(env) $google-oauthlib-tool --scope https://www.googleapis.com/auth/assistant-sdk-prototype \
          --save --headless --client-secrets /path/to/client_secret_client-id.json

# Please visit this URL to authorize this application: https://...

# Copy and paste the auth code from above
Enter the authorization code:

# SUCCESS
credentials saved: /path/to/.config/google-oauthlib-tool/credentials.json

# DO NOT RENAME credentials**.json file

Register & Run Code

https://developers.google.com/assistant/sdk/guides/library/python/embed/install-sample

Use the Registration Tool

(env) $ googlesamples-assistant-devicetool register-model \
--manufacturer raspberry-pi-devfestmn \
--product-name whitaker-home \
--description home-automation \
--type LIGHT \
--model whitaker-home-m1

List Device Models

(env) $ googlesamples-assistant-devicetool list --model

(env) $ googlesamples-assistant-hotword \
--project_id whitaker-home \
--device_model_id whitaker-home-m1

Run the code

Observations 

https://github.com/baskint/actions_on_google_devfestMN

• Catching up to Alexa (skills) in development, but query tooling is already better than Alexa. Kudos to Google!
   
• Development experience is pretty good, but there are a few gotchas and head scratchers
   
• Test Simulator is behaving unexpectedly. Saving `Test Draft` and when the test is active cannot be predicted
   
• Sample apps are quite decent and more coming! Stay up-to-date. Check back frequently.

Thank you

In Loving Memory of Mr. Baxter...
[April, 2005 - January 2018]

Departing Thoughts

https://github.com/baskint/actions_on_google_devfestMN

• Future is not just CRUD applications anymore (jk - still is but on new ~surfaces~ )
   
• Needs more context and personalization - such as a picture or video with the user request
   
• Collaborate, share your projects and learn from others. Join the communities and still check
   
• Don't Panic & carry a towel