The trickiest

of the project

Artem Arkhipov

Lead Web Developer at Techmagic

@ar_arkhipov

artem.arkhipov@techmagic.co

.cloud

What is

.cloud

What is

.cloud

IS NOT about drawing only

.cloud

main technology stack:

And few third party services...

So what about tricky element??

Transform this:

To this:

The way of thinking

1. Do it on client side and send directly to S3

Not reliable

Browser support

Performance is not stable

2. Use jGraph server ( Java )

Support of some weird and old things

We have a lot of custom stylings etc.

Objects content is generated by angular

3. We should find serverless solution (FaaS)

We need scalability to cope with load

We are experienced with AWS Lambda

Serverless concept is so cool

The way of thinking

4. Send diagram HTML and render it to image with PhantomJS or similar tools

Not reliable

Potentially huge requests

5. Load whole Diagram Editor in PhantomJS or Selenium and capture screenshot

Seems to be OK

ACTUALLY NO

But finally I met him...

Headless Chrome

Image is a perfect copy

High performance, it takes about 2-3 sec for 1 image

Reliable

Next steps

Write main handler functions:

Run headless chrome inside AWS Lambda

- Launch chrome as child process

- Open virtual tabs with diagrams

- Wait till diagrams are loaded

- Capture screenshots

- Save screenshots to AWS S3

- Close tabs

- Trigger image-resizing Lambda via SNS

Create lighter Diagram Editor version

Make "delayed lambda invocation" mechanism

const CHROME_URL = 'http://127.0.0.1:9222';

const CHROME_FLAGS = [
    '--headless',
    '--disable-gpu',
    '--remote-debugging-port=9222',
    '--window-size=1200,800',
    '--no-sandbox',
    '--user-data-dir=/tmp/user-data',
    '--hide-scrollbars',
    '--enable-logging',
    '--log-level=0',
    '--v=99',
    '--single-process',
    '--data-path=/tmp/data-path',
    '--ignore-certificate-errors',
    '--homedir=/tmp',
    '--disk-cache-dir=/tmp/cache-dir'
];

Some examples

const spawn = require('child_process').spawn;
let chrome;

function spawnChrome() {
    console.log('Spawning Chrome...');
    chrome = spawn(__dirname+'/headless-chrome/headless_shell', 
        config.CHROME_FLAGS, {
        cwd: '/tmp',
        shell: true,
        detached: true,
        stdio: 'ignore'
    });
    chrome.unref();
    chrome.on('close', (code) => {
        console.log('Chrome closed with code: ', code);
        chrome = null;
        spawnChrome();
    });
}

Run chrome

Open tab & connect

const CDP = require('chrome-remote-interface');

function openTab() {
    return CDP.New().then((tab) => { //create tab
        return CDP({tab}).then((client) => { //create client
            return Promise.all([
                client.Page.enable(),
                client.Runtime.enable()
            ]).then(() => {
                return Promise.resolve({client}); //ready
            })
        })
    })
}

Navigate to page

const Promise = require('bluebird');

function navigate({url}) {
    return new Promise((resolve, reject) => {
        // setup any events or conditions
        client.Page.loadEventFired(() => resolve())
        
        //run navigation to page
        client.Page.navigate({url});

    }).timeout(config.LOAD_TIMEOUT);
}

Capture and close

function capture({client}) {
    return client.Page.captureScreenshot({
        fromSurface: true,
        format: 'png'
    }).then((screenshot) => {
        client.close();
        const image = screenshot.data;
        return Promise.resolve({image});
    })
}

function closeTabs() {
    console.log('Cleaning opened tabs...');
    return CDP.List().then((tabs) => {
        return Promise.all(tabs.map(({id}) => CDP.Close({id})))
    })
}

High level overview

Delayed Invocation

Conclusion

Scalable and reliable

Execution time about 1 - 2.5s

160 000 images cost less than ~10$ (AWS free tier is not included!)

Artem Arkhipov

Lead Web Developer at Techmagic

@ar_arkhipov

ar.arkhipov@gmail.com

Thank You!

artem.arkhipov@techmagic.co