Preoccupy the user's tab
Artsiom Mezin
@iketari

About me
A Senior front-end dev at Behavox.
Have been doing web for over 10 years.
Always curious.

What is it about?
It is about a way of implementing a remote access

Why do we need it?
-
To support
-
To collaborate
-
To report bugs and issues
-
To teach and to learn (theoretically)
-
… could be anything else what you can do toghter
Nothing new. It exists.

Yes, but...
-
Access is granted not just to an app but the entire system.
-
Users have to install a client part of the solution.
-
An operator has to install a host program on his side.

The Web can help
-
Browsers sandbox is isolated.
-
Web apps do not need to be installed.
-
The host also could be implemented as a web app.

Technological questions
-
How to share the user’s screen?
-
How to perform an event on the user's side and how to transfer it to there?
-
How to install the client part so that it would work within any web application?

How to share the user’s screen?
-
HTML2Canvas -> canvas.captureStream(frameRate)
-
Do not share a screen, share the DOM
-
navigator.getDisplayMedia() + WebRTC
HTML to Canvas
console.time('HTMLToCanvas');
html2canvas(document.querySelector('body')).then(canvas => {
let img = canvas.toDataURL("image/png");
document.body.append('<img id="result" src="'+img+'"/>');
console.timeEnd('HTMLToCanvas');
});
// HTMLToCanvas: 26687.60693359375ms / 0.0374 FPS

TOGETHERJS


Title Text
MediaDevices.getDisplayMedia()
let promise = navigator.mediaDevices.getDisplayMedia(constraints);

Title Text
MediaDevices.getDisplayMedia()

Title Text
WebRTC


1. No plugins
2. P2P
3. Audio, Video and Data all encrypted in transit
Title Text
WebRTC. Signalling.


Title Text
Signalling. AWS Lambda


Title Text
Lambda. OnConnect

// Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: MIT-0
var AWS = require("aws-sdk");
AWS.config.update({ region: process.env.AWS_REGION });
var DDB = new AWS.DynamoDB({ apiVersion: "2012-10-08" });
exports.handler = function (event, context, callback) {
var putParams = {
TableName: process.env.TABLE_NAME,
Item: {
connectionId: { S: event.requestContext.connectionId }
}
};
DDB.putItem(putParams, function (err) {
callback(null, {
statusCode: err ? 500 : 200,
body: err ? "Failed to connect: " + JSON.stringify(err) : "Connected."
});
});
};
Title Text
Lambda. OnMessage

exports.handler = async (event, context) => {
const apigwManagementApi = new AWS.ApiGatewayManagementApi({
apiVersion: '2018-11-29',
endpoint: event.requestContext.domainName + '/' + event.requestContext.stage
});
const { data } = JSON.parse(event.body);
switch (data.action) {
case 'register':
await registerController(data, event, ddb, apigwManagementApi);
break;
case 'communicate':
await communicateController(data, event, ddb, apigwManagementApi);
break;
default:
break;
}
}
Title Text
All together


Technological questions
-
How to share the user’s screen? -
How to perform an event on the user's side and how to transfer it to there?
-
How to install the client part so that it would work within any web application?

How to perform a desirable event on the user's side and how to transfer it there?


Preoccupy
What happens when a user clicks on some element on the page?
-
The element is being located.
-
A JavaScript event is being fired.
-
A standard action is happening.
PreoccupyJS is trying to reproduce browser's behaviour
-
Find an element
-
Fire an event
-
Make changes in DOM if necessary
let el = document.elementFromPoint(x, y);
// https://developer.mozilla.org/en-US/docs/Web/API/DocumentOrShadowRoot/elementFromPoint
el.dispatchEvent(new MouseEvent('click', opts));
if (document.activeElement !== null) {
this.fireEvent('blur', document.activeElement as HTMLElement);
}
if (el.focus) {
el.focus();
}
PreoccupyJS contains a number of replaceable Actions each one for a dedicated browser event
export class ClickToAction implements PreoccupyAction {
static type: string = ActionsName.CLICK_TO;
static eventName: string = 'click';
constructor(
public payload: {
x: number;
y: number;
}
) {
this.type = ClickToAction.type;
}
performEvent(dom: DomController, stack: PreoccupyAction[]) {
dom.clickTo(this.payload);
}
static handleEvent(host: Host, event: Event): PreoccupyAction {
const payload = host.getRelativeCoordinate(event as MouseEvent);
return new ClickToAction(payload);
}
}
PreoccupyJS handles events on the host side, serialize and passes them to a transport
import { Host, RxjsTransport } from 'preoccupyjs';
// fill free to style and modify this element as you wish, but keep it in the DOM!
const hostEl = document.createElement('div');
hostEl.tabIndex = 0; // make it focusable
const transport = new RxjsTransport(options as {
subject: new Subject<object>; //e.g. WebSocketSubject
});
const host = new Host(transport, hostEl);
host.start(); // run the communication whenever your app is ready!
Transport is responsible for delivering a serialized action to the client's side, where PreoccupyJS will perform it
import { Host, RxjsTransport, DomController } from 'preoccupyjs';
//Set up the transport
const transport = new RxjsTransport(options as {
subject: new Subject() // e.g. WebSocketSubject
});
// Set up and run the client
// It's possible to specify the controlled scope of the page by passing any DOM element
const client = new Client(transport, new DomController(document.body));
// run the communication whenever your app is ready!
client.start();
Technological questions
-
How to share the user’s screen? -
How to perform an event on the user's side and how to transfer it to there? -
How to install the client part so that it would work within any web application?

How to install the client part so that it would work within any web application?

Bookmarklets
(in theory)
javascript:(function(){
//Statements returning a non-undefined type, e.g. assignments
})();
<a href="{bookmarkletText}"> Bookmark me </a>
import bookmarklet from './assets/bookmarklet.bundle';
// via JSX
return (
<a href={bookmarklet}>
Click me, I am a bookmarklet.
</a>
);
// via vanilla JS
let link = document.createElement('a');
link.href = bookmarklet;
link.innerHTML = 'Click me, I am a bookmaklet';
return link;
Bookmarklets
(in practice)
Demo

Thank you! Any questions?
Please tell me what to improve or what to dig into next time!
Twitter: @iketari

Title Text
PreoccupyJS
By iketari
PreoccupyJS
- 228