Preoccupy the user's tab

Artsiom Mezin

@iketari

About me

A Senior front-end dev at Behavox.

 

Have been doing web for over 10 years.

Always curious.

 

 

What is it about?

It is about a way of implementing a remote access

Why do we need it?

  1. To support

  2. To collaborate

  3. To report bugs and issues

  4. To teach and to learn (theoretically)

  5. … could  be anything else what you can do toghter

Nothing new. It exists.

Yes, but...

  1. Access is granted not just to an app but the entire system.

  2. Users have to install a client part of the solution.

  3. An operator has to install a host program on his side.

The Web can help

  1. Browsers sandbox is isolated.

  2. Web apps do not need to be installed.

  3. The host also could be implemented as a web app.

Technological questions

  1. How to share the user’s screen?

  2. How to perform an  event on the user's side and how to transfer it to there?

  3. How to install the client part so that it would work within any web application?

How to share the user’s screen?

  1. HTML2Canvas -> canvas.captureStream(frameRate)

  2. Do not share a screen, share the DOM

  3. navigator.getDisplayMedia() + WebRTC

HTML to Canvas

console.time('HTMLToCanvas');

html2canvas(document.querySelector('body')).then(canvas => {
    let img = canvas.toDataURL("image/png");
    document.body.append('<img id="result" src="'+img+'"/>');
    console.timeEnd('HTMLToCanvas');
});

// HTMLToCanvas: 26687.60693359375ms / 0.0374 FPS

TOGETHERJS

Title Text

MediaDevices.getDisplayMedia()

let promise = navigator.mediaDevices.getDisplayMedia(constraints);

Title Text

MediaDevices.getDisplayMedia()

Title Text

WebRTC

1. No plugins

2. P2P

3. Audio, Video and Data all encrypted in transit

Title Text

WebRTC. Signalling.

Title Text

Signalling. AWS Lambda

Title Text

Lambda. OnConnect

// Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: MIT-0

var AWS = require("aws-sdk");
AWS.config.update({ region: process.env.AWS_REGION });
var DDB = new AWS.DynamoDB({ apiVersion: "2012-10-08" });

exports.handler = function (event, context, callback) {
  var putParams = {
    TableName: process.env.TABLE_NAME,
    Item: {
      connectionId: { S: event.requestContext.connectionId }
    }
  };

  DDB.putItem(putParams, function (err) {
    callback(null, {
      statusCode: err ? 500 : 200,
      body: err ? "Failed to connect: " + JSON.stringify(err) : "Connected."
    });
  });
};

Title Text

Lambda. OnMessage

exports.handler = async (event, context) => {
    const apigwManagementApi = new AWS.ApiGatewayManagementApi({
      apiVersion: '2018-11-29',
      endpoint: event.requestContext.domainName + '/' + event.requestContext.stage
    });

    const { data } = JSON.parse(event.body);

    switch (data.action) {
      case 'register':
        await registerController(data, event, ddb, apigwManagementApi);
        break;
      case 'communicate':
        await communicateController(data, event, ddb, apigwManagementApi);
        break;
      
      default:
        break;
    }
}

Title Text

All together

Technological questions

  1. How to share the user’s screen?

  2. How to perform an event on the user's side and how to transfer it to there?

  3. How to install the client part so that it would work within any web application?

How to perform a desirable event on the user's side and how to transfer it there?

Preoccupy

What happens when a user clicks on some element on the page?

  1. The element is being located.

  2. A JavaScript event is being fired.

  3. A standard action is happening.

PreoccupyJS is trying to reproduce browser's behaviour

  1. Find an element

  2. Fire an event

  3. Make changes in DOM if necessary

let el = document.elementFromPoint(x, y); 
// https://developer.mozilla.org/en-US/docs/Web/API/DocumentOrShadowRoot/elementFromPoint

el.dispatchEvent(new MouseEvent('click', opts));

if (document.activeElement !== null) {
  this.fireEvent('blur', document.activeElement as HTMLElement);
}

if (el.focus) {
  el.focus();
}

PreoccupyJS contains a number of replaceable Actions each one for a dedicated browser event

export class ClickToAction implements PreoccupyAction {
  static type: string = ActionsName.CLICK_TO;
  static eventName: string = 'click';
  constructor(
    public payload: {
      x: number;
      y: number;
    }
  ) {
    this.type = ClickToAction.type;
  }

  performEvent(dom: DomController, stack: PreoccupyAction[]) {
    dom.clickTo(this.payload);
  }

  static handleEvent(host: Host, event: Event): PreoccupyAction {
    const payload = host.getRelativeCoordinate(event as MouseEvent);
    return new ClickToAction(payload);
  }
}

PreoccupyJS handles events on the host side, serialize and passes them to a transport

import { Host, RxjsTransport } from 'preoccupyjs';

// fill free to style and modify this element as you wish, but keep it in the DOM!
const hostEl = document.createElement('div'); 
hostEl.tabIndex = 0; // make it focusable

const transport = new RxjsTransport(options as {
    subject: new Subject<object>; //e.g. WebSocketSubject
});


const host = new Host(transport, hostEl);
host.start(); // run the communication whenever your app is ready!

Transport is responsible for delivering a serialized action to the client's side, where PreoccupyJS will perform it

import { Host, RxjsTransport, DomController } from 'preoccupyjs';

//Set up the transport
const transport = new RxjsTransport(options as {
    subject: new Subject() // e.g. WebSocketSubject
});

// Set up and run the client
// It's possible to specify the controlled scope of the page by passing any DOM element
const client = new Client(transport, new DomController(document.body)); 

// run the communication whenever your app is ready!
client.start();

Technological questions

  1. How to share the user’s screen?

  2. How to perform an event on the user's side and how to transfer it to there?

  3. How to install the client part so that it would work within any web application?

How to install the client part so that it would work within any web application?

Bookmarklets

(in theory)

javascript:(function(){
  //Statements returning a non-undefined type, e.g. assignments
})();
<a href="{bookmarkletText}"> Bookmark me </a>
import bookmarklet from './assets/bookmarklet.bundle';

// via JSX
return (
  <a href={bookmarklet}>
    Click me, I am a bookmarklet.
  </a>
);
 
// via vanilla JS
let link = document.createElement('a');
    link.href = bookmarklet;
    link.innerHTML = 'Click me, I am a bookmaklet';
return link;

Bookmarklets

(in practice)

Demo

Thank you! Any questions?

Please tell me what to improve or what to dig into next time!

Twitter: @iketari

Title Text

PreoccupyJS

By iketari

PreoccupyJS

  • 192