I'm @holsee 👋
I help run:
Talk WIP*
I just want to share how we applied this to...
Headless Chrome Browser Instances and
Web Socket connections to Chromes RDP endpoints
in order to scale concurrent test executions :D
Yet another elixir 💖 story.
Web Driver
Chrome Driver
Web Driver
Chrome Driver
Request timeouts
Chrome Process Instability hidden behind ChromeDriver
Instability when running > 5 concurrent tests per server :(
No self repair... so we had to reboot our servers when crashes took place
Memory Leaks
Wallaby has only Experimental
Chrome Support
So we created...
...in order to scale our Chrome instances and speak Chrome Remote Debug Protocol directly.
[Test Scenario Clients]
web socket connections speaking
chrome remote debug protocol
node1@.
nodeN@.
Upstream connections
|
|
|
redirecting communication via TCP Proxy
|
|
|
Downstream
Chrome Instance [Resource Pool]
Transparent Proxy connections to Chrome Servers
Test Scenario Client
web socket connections speaking
chrome remote debug protocol
node1@.
nodeN@.
ws:// client
|
|
|
Proxy built on :gen_tcp
|
|
Chrome's ws:// RDP Server
ChromeServer
Supervisor
ChromeServer
P
O
R
T
Supervising Chrome with Elixir...
ChromeServer
Supervisor
ChromeServer
P
O
R
T
Why?
Architectural Diagram next slide...
ain't got time for that...
SKIP!
$ curl http://localhost:1330/api/v1/connection
ws://127.0.0.1:1331/devtools/page/962A3A359239A007A2C4C7788B0B1050
$ curl http://localhost:1330/api/v1/connection
ws://127.0.0.1:1331/devtools/page/B04D067000B6BC31B2C7112AD7F58DB0
Client agnostics - any Client that speaks Chrome RDP of ws:// will work:
# Get ws:// to Chrome and Establish Websocket Connection
page_conn = ChroxyClient.page_session!(%{host: "localhost", port: 1330})
Using the Elixir ChroxyClient
14:17:20.417 pid=<0.338.0> module=Plug.Logger
[info] GET /api/v1/connection
14:17:20.417 pid=<0.223.0> module=Chroxy.ChromeManager
[info] Selected chrome server: {:undefined, #PID<0.224.0>, :worker, [Chroxy.ChromeServer]}
14:17:20.471 request_id=2kiujeru0evs11muts000025 pid=<0.338.0> module=Plug.Logger
[info] Sent 200 in 54ms
Chroxy Server handles HTTP request for a Chrome Connection and selects the chrome resource which will fulfil the request when the client connects....
14:45:41.118 pid=<0.221.0> module=Chroxy.ProxyListener
[info] Connection accepted, spawning proxy server to manage connection
14:45:41.121 pid=<0.341.0> module=Chroxy.ProxyServer
[debug] Downstream connection established
When the Web Socket connection is established the Downstream connection is made to the Chrome Instance:
14:45:41.147 pid=<0.341.0> module=Chroxy.ProxyServer
[debug] Up -> Down: "GET /devtools/page/C540B9C7342B201D326121298999110B HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: ZadY5sMpf8BKLXeSlUzlbA==\r\n\r\n"
14:45:41.148 pid=<0.341.0> module=Chroxy.ProxyServer
[debug] Up <- Down: "HTTP/1.1 101 WebSocket Protocol Handshake\r\nUpgrade: WebSocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: TTnKGNozGsCuWJwii21T3iUd2RY=\r\n\r\n"
Chroxy acts as a Transparent Proxy...
E.g. The client & chrome instance performing WebSocket Upgrade Negotiation after connection:
session = fn() ->
# Ask Chroxy Server for a ws:// connection to a Chrome page
ChroxyClient.page_session!(%{host: "localhost", port: 1330})
end
# Using any Client Library that speaks Chrome Debug Protocol...
goto = fn(page, url) ->
# We can issue command to naviagte to URL
ChromeRemoteInterface.RPC.Page.navigate(page, %{url: url})
end
eval_js = fn(page, js) ->
# We can evaluate JS
case ChromeRemoteInterface.RPC.Runtime.evaluate(page, %{expression: js}) do
{:ok, %{"result" => %{"result" => result}}} ->
{:ok, result}
error ->
error
end
end
#... etc
Wrapper functions for demo client interaction:
Run JS:
Goto Page:
14:56:11.309 pid=<0.338.0> module=Chroxy.ProxyServer
[debug] Up -> Down: <<129, 201, 175, 4, 98, 27, 212, 38, 18, 122, 221, 101, 15, 104, 141, 62, 25, 57, 202, 124, 18, 105, 202, 119, 17, 114, 192, 106, 64, 33, 141, 95, 83, 70, 143, 47, 73, 59, 244, 54, 63, 57, 210, 40, 64, 118, 202, 112, 10, 116, ...>>
14:56:11.311 pid=<0.338.0> module=Chroxy.ProxyServer
[debug] Up <- Down: <<129, 126, 2, 29, 123, 34, 105, 100, 34, 58, 52, 44, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 116, 121, 112, 101, 34, 58, 34, 111, 98, 106, 101, 99, 116, 34, 44, 34, 115, ...>>
14:56:16.077 pid=<0.338.0> module=Chroxy.ProxyServer
[debug] Up -> Down: <<129, 200, 24, 240, 28, 21, 99, 210, 108, 116, 106, 145, 113, 102, 58, 202, 103, 55, 125, 136, 108, 103, 125, 131, 111, 124, 119, 158, 62, 47, 58, 171, 45, 72, 56, 219, 60, 78, 42, 173, 62, 104, 52, 210, 113, 112, 108, 152, 115, 113, ...>>
14:56:16.078 pid=<0.338.0> module=Chroxy.ProxyServer
[debug] Up <- Down: <<129, 59, 123, 34, 105, 100, 34, 58, 53, 44, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 114, 101, 115, 117, 108, 116, 34, 58, 123, 34, 116, 121, 112, 101, 34, 58, 34, 115, 116, 114, 105, 110, 103, 34, 44, 34, 118, 97, 108, ...>>
...
The client / server communicating via the ProxyServer
iex(20)> page = session.()
15:07:23.571 [debug] WebSocket: ws://127.0.0.1:1331/devtools/page/F00A608972260AF140F0E0E666F4D0DA
#PID<0.220.0>
iex(21)> goto.(page, "https://github.com/holsee")
{:ok, _}
iex(24)> eval_js.(page, "1 + 1")
{:ok, %{"description" => "2", "type" => "number", "value" => 2}}
The client / server communicating via the ProxyServer
15:10:15.122 pid=<0.347.0> module=Chroxy.ProxyServer
[debug] Upstream socket closed, terminating proxy
15:10:15.122 pid=<0.346.0> module=Chroxy.ChromeProxy
[info] Proxy connection down - closing page
Chroxy cleans up the Chrome Resource when Upstream connection closes:
15:44:25.826 pid=<0.331.0> module=Chroxy.ChromeServer
[info] [CHROME: 1744] "DevTools listening on ws://127.0.0.1:9223/devtools/browser/3a538ecb-b7cf-4f75-8d3d-f931ebf8dc45"
$ kill 1744
15:44:59.272 pid=<0.336.0> module=Chroxy.ChromeServer
[info] [CHROME: 2107] "DevTools listening on ws://127.0.0.1:9223/devtools/browser/d5188fb4-f586-4447-ae0b-5ae29740a6ff"
And restarts chrome instances if they Crash:
Open Source!
Thanks!!! 🙇
P.S. I hope to be giving the full talk soon where I dig into the implementation in the near future!