Hi...

A girl named Kajal walks during mobility training at the Bihar Natraheen Parishad, or Bihar School for Blind Girls, in Patna, India, in 2009.

Credit: J P Davidson/Flickr Creative Commons

"Let Me See"

The perfect companion for sightless people

Features

Listen/respond to

commands

Capture images & analyse them

Speak back the results

Demo

How?

Listening

my name is [name]
what should I say?
(describe) what do you see
how do I look?
what color is this?

(can you) read this (for me)

if('annyang' in window) {

  const action = () => describeWhatISee;

  [
   {'show me': action},
   {'let me see': action},
   {'(describe) what do you see': action}
  ].forEach(annyang.addCommands);

  annyang.start({ continuous: true });

}

Adding commands

Capturing images

Using the Web Platform

navigator.mediaDevices

<canvas></canvas>

<video></video>

if ('MediaDevices' in window || navigator.getUserMedia) {

  navigator.mediaDevices.enumerateDevices()

    .then((devices) => devices
        .filter( (device) => device.kind === 'videoinput' )

    .then( (devices) => {

      navigator.getUserMedia(
        { video: { /* filter devices here */ } },
        (stream) => videoRef.src = window.URL.createObjectURL(stream),
        (e) => console.error('failed', e)
      );

    });
}

Grabbing the stream

let draw = () => {
  
    context.drawImage(videoRef, 0, 0, videoWidth, videoHeight);
    window.requestAnimationFrame(draw);

};

videoRef.addEventListener('play', draw, false);

//...

getImageAsBase64() {
    return canvasRef.toDataURL();
}

Extracting the image

Analyse the footage

Using Google Cloud Vision API

Pre-trained ML models.

REST based API

post(content) {

    const API = `https://vision.googleapis.com/v1/images:annotate?key=${KEY}`;

    const request = {
      requests: [{
        image: { content },
        features: [{
          type: 'LABEL_DETECTION',
          maxResults: 10
        }]
      }]
    };

    return http.post(API, request).map(processMetadata);

}

Request the API

processMetadata(data) {

    if (data.responses.labelAnnotations) {
      let labels = [];
      (data.responses.labelAnnotations || []).forEach((lbl) => {
        labels.push(lbl.description);
      });
      return { labels };
    }

    // data.responses.faceAnnotations
    // data.responses.imagePropertiesAnnotation
    // data.responses.textAnnotations

}

Process the response

Speak back the results

Using the Web Platform

Text-to-Speech API

if ('responsiveVoice' in window) {

    responsiveVoice.OnVoiceReady = () => {
        responsiveVoice.setDefaultVoice('US English Female');

        this.say(`Hello. How can I help you?`, { delay: 1000 });

    };

}

Text-to-Speech

Deploying to production

Beez

Neez.

> npm run build-prod
> npm run bump-version
> npm run print-version
> npm run git-release

> firebase deploy

One

More Thing

Things to Improve

NLP (Text Sentiment)

Assistant Companion Bot  w/ API.ai

Better offline Support (!)

I18n

Improve Lighthouse Score (82)

Analytics

Async / Await 😎

Thank you

using software to improve lives

By Wassim Chegham

using software to improve lives

  • 255
Loading comments...

More from Wassim Chegham