I'm @holsee 👋

I help run:

 


 

 

 

Talk WIP*

TL;DR Elixir Rocks for Scaling connections and Clustering Resources!

 

I just want to share how we applied this to...

Headless Chrome Browser Instances and

Web Socket connections to Chromes RDP endpoints

in order to scale concurrent test executions :D

 

Yet another elixir 💖 story.

My team runs elixir applications which run at scale with a global user base... 

We run synthetic

tests across the 🌍 to obtain insight into user's experiences in different geographic locations over time...

 

We created a synthetic testing platform tailored to our needs



which is deployed using containers across many AWS Regions

Web Driver

Chrome Driver

Our initial test scenario execution stack =>

 

This worked great :)

...until we
wanted to ramp up the number of concurrently executing test scenarios per server :(

Web Driver

Chrome Driver

💥

💥

💥

Request timeouts

💥

Chrome Process Instability hidden behind ChromeDriver

Instability when running > 5 concurrent tests per server :(

 No self repair... so we had to reboot our servers when crashes took place

Memory Leaks

Wallaby has only Experimental
Chrome Support

 

The instability and multiple layers of leaky abstractions

between test definitions

and what

chrome actually executed

was problematic for us...

So we created...

...in order to scale our Chrome instances and speak Chrome Remote Debug Protocol directly.

..with browser

resource clustering we could add more instances as needed to meet the demand

[Test Scenario Clients]
web socket connections speaking
chrome remote debug protocol

node1@.

nodeN@.


Upstream connections

|

|
|
redirecting communication via TCP Proxy
|

|

|

Downstream
Chrome Instance [Resource Pool]

Transparent Proxy connections to Chrome Servers

Test Scenario Client
web socket connections speaking
chrome remote debug protocol

node1@.

nodeN@.


ws:// client

|

|
|
Proxy built on :gen_tcp

|

|

Chrome's ws:// RDP Server

ChromeServer
Supervisor

ChromeServer

P
O
R
T

Supervising Chrome with Elixir...

ChromeServer
Supervisor

ChromeServer

P
O
R
T

💥

💥

💥

Why?

Architectural Diagram next slide...
ain't got time for that...
SKIP!

$ curl http://localhost:1330/api/v1/connection
ws://127.0.0.1:1331/devtools/page/962A3A359239A007A2C4C7788B0B1050                                                                    

$ curl http://localhost:1330/api/v1/connection
ws://127.0.0.1:1331/devtools/page/B04D067000B6BC31B2C7112AD7F58DB0                                                                      

Client agnostics - any Client that speaks Chrome RDP of ws:// will work:

# Get ws:// to Chrome and Establish Websocket Connection
page_conn = ChroxyClient.page_session!(%{host: "localhost", port: 1330})

Using the Elixir ChroxyClient

14:17:20.417 pid=<0.338.0> module=Plug.Logger 
  [info]  GET /api/v1/connection

14:17:20.417 pid=<0.223.0> module=Chroxy.ChromeManager 
  [info]  Selected chrome server: {:undefined, #PID<0.224.0>, :worker, [Chroxy.ChromeServer]}

14:17:20.471 request_id=2kiujeru0evs11muts000025 pid=<0.338.0> module=Plug.Logger 
  [info]  Sent 200 in 54ms

Chroxy Server handles HTTP request for a Chrome Connection and selects the chrome resource which will fulfil  the request when the client connects....

14:45:41.118 pid=<0.221.0> module=Chroxy.ProxyListener 
  [info]  Connection accepted, spawning proxy server to manage connection

14:45:41.121 pid=<0.341.0> module=Chroxy.ProxyServer 
  [debug] Downstream connection established

When the Web Socket connection is established the  Downstream connection is made to the Chrome Instance:


14:45:41.147 pid=<0.341.0> module=Chroxy.ProxyServer 
  [debug] Up -> Down: "GET /devtools/page/C540B9C7342B201D326121298999110B HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: ZadY5sMpf8BKLXeSlUzlbA==\r\n\r\n"

14:45:41.148 pid=<0.341.0> module=Chroxy.ProxyServer 
  [debug] Up <- Down: "HTTP/1.1 101 WebSocket Protocol Handshake\r\nUpgrade: WebSocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: TTnKGNozGsCuWJwii21T3iUd2RY=\r\n\r\n"

Chroxy acts as a Transparent Proxy...

 

E.g. The client & chrome instance performing WebSocket Upgrade Negotiation after connection:

session = fn() ->
  # Ask Chroxy Server for a ws:// connection to a Chrome page
  ChroxyClient.page_session!(%{host: "localhost", port: 1330})
end



# Using any Client Library that speaks Chrome Debug Protocol...



goto = fn(page, url) ->
  # We can issue command to naviagte to URL
  ChromeRemoteInterface.RPC.Page.navigate(page, %{url: url})
end





eval_js = fn(page, js) ->
  # We can evaluate JS
  case ChromeRemoteInterface.RPC.Runtime.evaluate(page, %{expression: js}) do
    {:ok,  %{"result" => %{"result" => result}}} ->
      {:ok, result}
    error ->
      error
  end	
end

#... etc

Wrapper functions for demo client interaction:

Run JS:

Goto Page:

14:56:11.309 pid=<0.338.0> module=Chroxy.ProxyServer 
  [debug] Up -> Down: <<129, 201, 175, 4, 98, 27, 212, 38, 18, 122, 221, 101, 15, 104, 141, 62, 25, 57, 202, 124, 18, 105, 202, 119, 17, 114, 192, 106, 64, 33, 141, 95, 83, 70, 143, 47, 73, 59, 244, 54, 63, 57, 210, 40, 64, 118, 202, 112, 10, 116, ...>>

14:56:11.311 pid=<0.338.0> module=Chroxy.ProxyServer 
  [debug] Up <- Down: <<129, 126, 2, 29, 123, 34, 105, 100, 34, 58, 52, 44, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 116, 121, 112, 101, 34, 58, 34, 111, 98, 106, 101, 99, 116, 34, 44, 34, 115, ...>>

14:56:16.077 pid=<0.338.0> module=Chroxy.ProxyServer 
  [debug] Up -> Down: <<129, 200, 24, 240, 28, 21, 99, 210, 108, 116, 106, 145, 113, 102, 58, 202, 103, 55, 125, 136, 108, 103, 125, 131, 111, 124, 119, 158, 62, 47, 58, 171, 45, 72, 56, 219, 60, 78, 42, 173, 62, 104, 52, 210, 113, 112, 108, 152, 115, 113, ...>>

14:56:16.078 pid=<0.338.0> module=Chroxy.ProxyServer 
  [debug] Up <- Down: <<129, 59, 123, 34, 105, 100, 34, 58, 53, 44, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 116, 121, 112, 101, 34, 58, 34, 115, 116, 114, 105, 110, 103, 34, 44, 34, 118, 97, 108, ...>>

...

The client / server communicating via the ProxyServer

iex(20)> page = session.()
15:07:23.571 [debug] WebSocket: ws://127.0.0.1:1331/devtools/page/F00A608972260AF140F0E0E666F4D0DA
#PID<0.220.0>

iex(21)> goto.(page, "https://github.com/holsee")
{:ok, _}

iex(24)> eval_js.(page, "1 + 1")
{:ok, %{"description" => "2", "type" => "number", "value" => 2}}

The client / server communicating via the ProxyServer

15:10:15.122 pid=<0.347.0> module=Chroxy.ProxyServer 
  [debug] Upstream socket closed, terminating proxy

15:10:15.122 pid=<0.346.0> module=Chroxy.ChromeProxy 
  [info]  Proxy connection down - closing page

Chroxy cleans up the Chrome Resource when Upstream connection closes:

15:44:25.826 pid=<0.331.0> module=Chroxy.ChromeServer 
  [info]  [CHROME: 1744] "DevTools listening on ws://127.0.0.1:9223/devtools/browser/3a538ecb-b7cf-4f75-8d3d-f931ebf8dc45"

$ kill 1744

15:44:59.272 pid=<0.336.0> module=Chroxy.ChromeServer 
  [info]  [CHROME: 2107] "DevTools listening on ws://127.0.0.1:9223/devtools/browser/d5188fb4-f586-4447-ae0b-5ae29740a6ff"

And restarts chrome instances if they Crash:

  • We have seen 1000s concurrent stable connections to Chrome instances/resources per server (much better than 4)!
  • Managing Chrome in Elixir supervision trees gives us the resiliency we desire.
  • Less abstraction has made finding / fixing issues much easier and faster.

 

Open Source!

  • Source: https://github.com/holsee/chroxy
  • Hex: https://hex.pm/packages/chroxy
  • Docs: https://hexdocs.pm/chroxy/Chroxy.html

 

In conclusion...


Thanks!!! 🙇 

 

P.S. I hope to be giving the full talk soon where I dig into the implementation in the near future!

Chroxy - Scaling Chrome Connections

By Steven Holdsworth

Chroxy - Scaling Chrome Connections

Scaling Chrome Remote Debug Connections via Elixir Proxy Server

  • 1,113