The trickiest

of the project

Artem Arkhipov

Lead Web Developer at Techmagic

@ar_arkhipov

artem.arkhipov@techmagic.co

.cloud

.cloud

What is 

?

.cloud

What is 

?

.cloud

IS NOT about drawing only

.cloud

main technology stack:

And few third party services...

So what about tricky element??

Transform this:

To this:

The way of thinking

1. Do it on client side and send directly to S3

  • Not reliable
  • Browser support
  • Performance is not stable 

2. Use jGraph server ( Java )

  • Support of some weird and old things
  • We have a lot of custom stylings etc.
  • Objects content is generated by angular

3. We should find serverless solution (FaaS)

  • We need scalability to cope with load
  • We are experienced with AWS Lambda
  • Serverless concept is so cool

The way of thinking

4. Send diagram HTML and render it to image with PhantomJS or similar tools

  • Not reliable
  • Potentially huge requests

5. Load whole Diagram Editor in PhantomJS or Selenium and capture screenshot

  • Seems to be OK
  • ACTUALLY NO

But finally I met him...

Headless Chrome

  • Image is a perfect copy 
  • High performance, it takes about 2-3 sec for 1 image
  • Reliable 

Next steps

Write main handler functions:

Run headless chrome inside AWS Lambda

- Launch chrome as child process

- Open virtual tabs with diagrams

- Wait till diagrams are loaded

- Capture screenshots

- Save screenshots to AWS S3

- Close tabs

- Trigger image-resizing Lambda via SNS

Create lighter Diagram Editor version

Make "delayed lambda invocation" mechanism

const CHROME_URL = 'http://127.0.0.1:9222';

const CHROME_FLAGS = [
    '--headless',
    '--disable-gpu',
    '--remote-debugging-port=9222',
    '--window-size=1200,800',
    '--no-sandbox',
    '--user-data-dir=/tmp/user-data',
    '--hide-scrollbars',
    '--enable-logging',
    '--log-level=0',
    '--v=99',
    '--single-process',
    '--data-path=/tmp/data-path',
    '--ignore-certificate-errors',
    '--homedir=/tmp',
    '--disk-cache-dir=/tmp/cache-dir'
];

Some examples

const spawn = require('child_process').spawn;
let chrome;

function spawnChrome() {
    console.log('Spawning Chrome...');
    chrome = spawn(__dirname+'/headless-chrome/headless_shell', 
        config.CHROME_FLAGS, {
        cwd: '/tmp',
        shell: true,
        detached: true,
        stdio: 'ignore'
    });
    chrome.unref();
    chrome.on('close', (code) => {
        console.log('Chrome closed with code: ', code);
        chrome = null;
        spawnChrome();
    });
}

Run chrome

Open tab & connect

const CDP = require('chrome-remote-interface');

function openTab() {
    return CDP.New().then((tab) => { //create tab
        return CDP({tab}).then((client) => { //create client
            return Promise.all([
                client.Page.enable(),
                client.Runtime.enable()
            ]).then(() => {
                return Promise.resolve({client}); //ready
            })
        })
    })
}

Navigate to page

const Promise = require('bluebird');

function navigate({url}) {
    return new Promise((resolve, reject) => {
        // setup any events or conditions
        client.Page.loadEventFired(() => resolve())
        
        //run navigation to page
        client.Page.navigate({url});

    }).timeout(config.LOAD_TIMEOUT);
}

Capture and close 

function capture({client}) {
    return client.Page.captureScreenshot({
        fromSurface: true,
        format: 'png'
    }).then((screenshot) => {
        client.close();
        const image = screenshot.data;
        return Promise.resolve({image});
    })
}
function closeTabs() {
    console.log('Cleaning opened tabs...');
    return CDP.List().then((tabs) => {
        return Promise.all(tabs.map(({id}) => CDP.Close({id})))
    })
}

High level overview

Delayed Invocation

Conclusion

Scalable and reliable

Execution time about 1 - 2.5s

160 000 images cost less than ~10$ (AWS free tier is not included!)

Artem Arkhipov

Lead Web Developer at Techmagic

@ar_arkhipov

ar.arkhipov@gmail.com

Thank You!

artem.arkhipov@techmagic.co

The trickiest element of the project

By Artem Arkhipov

The trickiest element of the project

Scalable and reliable serverless solution. Running headless chrome on AWS Lambda to regullary create screenshots.

  • 487