I'm @holsee 👋
I help run:
Talk WIP*
TL;DR Elixir Rocks for Scaling connections and Clustering Resources!
I just want to share how we applied this to...
Headless Chrome Browser Instances and
Web Socket connections to Chromes RDP endpoints
in order to scale concurrent test executions :D
Yet another elixir 💖 story.
My team runs elixir applications which run at scale with a global user base...
We run synthetic
tests across the 🌍 to obtain insight into user's experiences in different geographic locations over time...
We created a synthetic testing platform tailored to our needs
which is deployed using containers across many AWS Regions
Web Driver
Chrome Driver
Our initial test scenario execution stack =>
This worked great :)
...until we
wanted to ramp up the number of concurrently executing test scenarios per server :(
Web Driver
Chrome Driver
💥
💥
💥
Request timeouts
💥
Chrome Process Instability hidden behind ChromeDriver
Instability when running > 5 concurrent tests per server :(
No self repair... so we had to reboot our servers when crashes took place
Memory Leaks
Wallaby has only Experimental
Chrome Support
The instability and multiple layers of leaky abstractions
between test definitions
and what
chrome actually executed
was problematic for us...
So we created...
...in order to scale our Chrome instances and speak Chrome Remote Debug Protocol directly.
..with browser
resource clustering we could add more instances as needed to meet the demand
[Test Scenario Clients]
web socket connections speaking
chrome remote debug protocol
node1@.
nodeN@.
Upstream connections
|
|
|
redirecting communication via TCP Proxy
|
|
|
Downstream
Chrome Instance [Resource Pool]
Transparent Proxy connections to Chrome Servers
Test Scenario Client
web socket connections speaking
chrome remote debug protocol
node1@.
nodeN@.
ws:// client
|
|
|
Proxy built on :gen_tcp
|
|
Chrome's ws:// RDP Server
ChromeServer
Supervisor
ChromeServer
P
O
R
T
Supervising Chrome with Elixir...
ChromeServer
Supervisor
ChromeServer
P
O
R
T
💥
💥
💥
Why?
Architectural Diagram next slide...
ain't got time for that...
SKIP!
$ curl http://localhost:1330/api/v1/connection
ws://127.0.0.1:1331/devtools/page/962A3A359239A007A2C4C7788B0B1050
$ curl http://localhost:1330/api/v1/connection
ws://127.0.0.1:1331/devtools/page/B04D067000B6BC31B2C7112AD7F58DB0
Client agnostics - any Client that speaks Chrome RDP of ws:// will work:
# Get ws:// to Chrome and Establish Websocket Connection
page_conn = ChroxyClient.page_session!(%{host: "localhost", port: 1330})
Using the Elixir ChroxyClient
14:17:20.417 pid=<0.338.0> module=Plug.Logger
[info] GET /api/v1/connection
14:17:20.417 pid=<0.223.0> module=Chroxy.ChromeManager
[info] Selected chrome server: {:undefined, #PID<0.224.0>, :worker, [Chroxy.ChromeServer]}
14:17:20.471 request_id=2kiujeru0evs11muts000025 pid=<0.338.0> module=Plug.Logger
[info] Sent 200 in 54ms
Chroxy Server handles HTTP request for a Chrome Connection and selects the chrome resource which will fulfil the request when the client connects....
14:45:41.118 pid=<0.221.0> module=Chroxy.ProxyListener
[info] Connection accepted, spawning proxy server to manage connection
14:45:41.121 pid=<0.341.0> module=Chroxy.ProxyServer
[debug] Downstream connection established
When the Web Socket connection is established the Downstream connection is made to the Chrome Instance:
14:45:41.147 pid=<0.341.0> module=Chroxy.ProxyServer
[debug] Up -> Down: "GET /devtools/page/C540B9C7342B201D326121298999110B HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: ZadY5sMpf8BKLXeSlUzlbA==\r\n\r\n"
14:45:41.148 pid=<0.341.0> module=Chroxy.ProxyServer
[debug] Up <- Down: "HTTP/1.1 101 WebSocket Protocol Handshake\r\nUpgrade: WebSocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: TTnKGNozGsCuWJwii21T3iUd2RY=\r\n\r\n"
Chroxy acts as a Transparent Proxy...
E.g. The client & chrome instance performing WebSocket Upgrade Negotiation after connection:
session = fn() ->
# Ask Chroxy Server for a ws:// connection to a Chrome page
ChroxyClient.page_session!(%{host: "localhost", port: 1330})
end
# Using any Client Library that speaks Chrome Debug Protocol...
goto = fn(page, url) ->
# We can issue command to naviagte to URL
ChromeRemoteInterface.RPC.Page.navigate(page, %{url: url})
end
eval_js = fn(page, js) ->
# We can evaluate JS
case ChromeRemoteInterface.RPC.Runtime.evaluate(page, %{expression: js}) do
{:ok, %{"result" => %{"result" => result}}} ->
{:ok, result}
error ->
error
end
end
#... etc
Wrapper functions for demo client interaction:
Run JS:
Goto Page:
14:56:11.309 pid=<0.338.0> module=Chroxy.ProxyServer
[debug] Up -> Down: <<129, 201, 175, 4, 98, 27, 212, 38, 18, 122, 221, 101, 15, 104, 141, 62, 25, 57, 202, 124, 18, 105, 202, 119, 17, 114, 192, 106, 64, 33, 141, 95, 83, 70, 143, 47, 73, 59, 244, 54, 63, 57, 210, 40, 64, 118, 202, 112, 10, 116, ...>>
14:56:11.311 pid=<0.338.0> module=Chroxy.ProxyServer
[debug] Up <- Down: <<129, 126, 2, 29, 123, 34, 105, 100, 34, 58, 52, 44, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 116, 121, 112, 101, 34, 58, 34, 111, 98, 106, 101, 99, 116, 34, 44, 34, 115, ...>>
14:56:16.077 pid=<0.338.0> module=Chroxy.ProxyServer
[debug] Up -> Down: <<129, 200, 24, 240, 28, 21, 99, 210, 108, 116, 106, 145, 113, 102, 58, 202, 103, 55, 125, 136, 108, 103, 125, 131, 111, 124, 119, 158, 62, 47, 58, 171, 45, 72, 56, 219, 60, 78, 42, 173, 62, 104, 52, 210, 113, 112, 108, 152, 115, 113, ...>>
14:56:16.078 pid=<0.338.0> module=Chroxy.ProxyServer
[debug] Up <- Down: <<129, 59, 123, 34, 105, 100, 34, 58, 53, 44, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 116, 121, 112, 101, 34, 58, 34, 115, 116, 114, 105, 110, 103, 34, 44, 34, 118, 97, 108, ...>>
...
The client / server communicating via the ProxyServer
iex(20)> page = session.()
15:07:23.571 [debug] WebSocket: ws://127.0.0.1:1331/devtools/page/F00A608972260AF140F0E0E666F4D0DA
#PID<0.220.0>
iex(21)> goto.(page, "https://github.com/holsee")
{:ok, _}
iex(24)> eval_js.(page, "1 + 1")
{:ok, %{"description" => "2", "type" => "number", "value" => 2}}
The client / server communicating via the ProxyServer
15:10:15.122 pid=<0.347.0> module=Chroxy.ProxyServer
[debug] Upstream socket closed, terminating proxy
15:10:15.122 pid=<0.346.0> module=Chroxy.ChromeProxy
[info] Proxy connection down - closing page
Chroxy cleans up the Chrome Resource when Upstream connection closes:
15:44:25.826 pid=<0.331.0> module=Chroxy.ChromeServer
[info] [CHROME: 1744] "DevTools listening on ws://127.0.0.1:9223/devtools/browser/3a538ecb-b7cf-4f75-8d3d-f931ebf8dc45"
$ kill 1744
15:44:59.272 pid=<0.336.0> module=Chroxy.ChromeServer
[info] [CHROME: 2107] "DevTools listening on ws://127.0.0.1:9223/devtools/browser/d5188fb4-f586-4447-ae0b-5ae29740a6ff"
And restarts chrome instances if they Crash:
- We have seen 1000s concurrent stable connections to Chrome instances/resources per server (much better than 4)!
- Managing Chrome in Elixir supervision trees gives us the resiliency we desire.
- Less abstraction has made finding / fixing issues much easier and faster.
Open Source!
- Source: https://github.com/holsee/chroxy
- Hex: https://hex.pm/packages/chroxy
- Docs: https://hexdocs.pm/chroxy/Chroxy.html
In conclusion...
Thanks!!! 🙇
P.S. I hope to be giving the full talk soon where I dig into the implementation in the near future!
Chroxy - Scaling Chrome Connections
By Steven Holdsworth
Chroxy - Scaling Chrome Connections
Scaling Chrome Remote Debug Connections via Elixir Proxy Server
- 1,113