Bringing ADB into the Firefox Process


Brandon Kase


8/1/13

outline


  • Motivation
  • Demo
  • What I did + Technologies
  • Architecture


motivation


adb


android debug bridge


       +---------------+           +------------+
       |               +----------->            |
       |  Android Phone|           |Laptop      |
       |               <-----------+            |
       +---------------+           +------------+

adb


Firefoxos debug bridge


       +---------------+           +------------+
       |               +----------->            |
       |FirefoxOS Phone|           |Laptop      |
       |               <-----------+            |
       +---------------+           +------------+

adb


Super high-level overview

      +-----------------+                +------------------+
      |                 | adb is running |                  |
      |                 |                |                  |
      | $ adb devices   +--------------->|full_unagi  device|
      |                 |IPC with sockets|                  |
      |                 |                |                  |
      +--------+--------+                +--------------^---+
               |                                        |
               |                                        |IPC with sockets
               | adb not running                        |               
               |      +-------------------------+       |
               |      | Fork a subprocess       |       |
               +----->| (2 seconds of sleeping) +-------+
                      |                         |
                      +-------------------------+

using ADB and its daemon to  implement an adb Api?


  • Mixture of adb in a subshell and raw sockets
  • Sometimes wait for initialization and subprocess forking
  • Have to kill the forked daemon when you're done


It's not ideal

Who uses adb as an api?



firefoxos simulator

outline


  • Motivation
  • Demo
  • What I did + Technologies
  • Architecture




< Demo >

outline


  • Motivation
  • Demo
  • What I did + Technologies
  • Architecture



so what did I do?

So what did I do?



  • Compiled ADB as a shared library
  • Loaded ADB into the Simulator add-on using js-ctypes
  • Started porting ADB to JavaScript
  • Changed some ADB behavior

Getting the source


Compiling adb as a shared libary


  • Ported the Makefile from Debian's android-tools-adb

  • Linux
    • Easy
  • Mac
    • Easy
  • Windows
    • Nightmare

Compiling adb as a shared library

on windows


  • Ported to Visual Studio's cl.exe compiler
  • cl.exe doesn't support
    • C11
    • C99
    • Only C89
  • ADB is not written against C89
  • Compile most things as C++
  • C++ is not a superset of C
  • Had to go through thousands of compile errors

Running adb in the simulator


  • So it compiled
  • Did it work?
  • Linux
    • Mostly
  • Mac
    • Mostly
  • Windows
    • Of course not

how do you load native code from javascript


js-ctypes

js-ctypes



is super cool

js-ctypes


Foreign-function interface to C from JavaScript


 let { Cu } = require('chrome');
Cu.import("resources://gre/modules/ctypes.jsm");


let lib = ctypes.open("some_lib.so");
let some_function = lib.declare("some_function",
default_abi,
ctypes.int,
ctypes.char.ptr);

let x = some_function("just some string");

js-ctypes


Small wrapper I made for declaring functions

 let lib = ctypes.open("some_lib.so");
I.declare({ name: "some_function",
returns: ctypes.int,
args: [ ctypes.char.ptr ]
}, lib);

... /* later in the code */

let some_function = I.use("some_function");
some_function("just some string");

Avoids globals and has easy-to-remember syntax

JS-ctypes



  • Now you know how to call C code from JS
  • How do you call JS code from C?


js-ctypes

It's complicated (call a function with a C function pointer)

/* C code */
DLL_EXPORT void long_task(void (*callback)(char *)) {
char * the_answer = something_long();
callback(the_answer);
}
/* JS code */
let CallbackType = ctypes.FunctionType(ctypes.default_abi,
ctypes.int,
[ ctypes.char.ptr ]);
let long_task = I.declare({ name: "long_task",
returns: ctypes.void_t,
args: [ CallbackType.ptr ]
}, lib);
let callbackFn = function(str) { console.log("in JS", str.readString()); };
long_task( CallbackType.ptr(callbackFn) );
It's worse in the real code

js-ctypes

Fix? 

/* C code */
void long_task() {
struct a_message {
char * str;
};
struct a_message m = { something_long() }; void * res;
MSG(&res, "achannel", m);
}
/* JS code */
let jsMsgFn = function(channel, args) {
switch(channel.readString()) {
case "achannel":
let [ x ] = JsMessage.unpack(args, ctypes.char.ptr);
return JsMessage.pack(11, Number);
}
};
Use MSG macro to avoid modifying much native code

back to running adb



why didn't adb

"just work"

Why didn't adb "just work'


  • ADB calls exit(1) for minor errors

  • ADB doesn't cleanup before quitting
  • ADB calls exit(1) to quit in every code path
  • exit(1) crashes Firefox

    Why didn't adb just work


    • Needed more control over threads
      • Moved threads to JavaScript Chrome Workers
        • Chrome Worker: Web Worker with access to js-ctypes
      • JavaScript callbacks can only be called from the same thread that they were created on


    • Chrome Workers don't have access to Chrome (ironically)
      • Chrome Workers can't spawn Chrome Workers (have to send a message back to UI thread)
      • Made EventedChromeWorker object to help with this

    Why didn't adb just work


    • Windows
    • If you kill a Chrome Worker while it is executing native code
      • Linux:
        • The runtime finishes executing the native code until it returns from all functions
      • Mac:
        • The runtime finishes executing the native code until it returns from all functions
      • Windows:
        • The runtime crashes Firefox
    • Synchronous unloading of add-on

      it works!


      • + 38,104 additions

      • - 804 deletions



      Before:
       
      After:

      features


      • Supports all commands needed by the FirefoxOS Simulator
        • adb push
        • adb devices
        • raw device tracking
        • adb forward (port forwarding)
        • adb shell
      • First "layer" and-a-half of ADB ported to JavaScript
      • Device tracking ported to JavaScript

      TODO


      • Port all "layers" to JavaScript
      • Make sure all memory leaks are gone
        • For now, there are only leaks if you disable or uninstall the Simulator add-on and then enable or reinstall it without restarting Firefox in between
      • Make Linux and Mac exit code less hacky (more stable)
        • As a side-effect of being so crashy, Windows exit handling is actually pretty robust now

      outline


      • Motivation
      • Demo
      • What I did + Technologies
      • Architecture

      What are the "layers"


      how does adb work? (before and after)

      adb internals

      Before

          +------------+                   +----------------+
          | PHONE      |                   | COMPUTER       |
          |            | Platform-specific | +------------+ |
          | +--------+ | drivers           | | adb daemon | |
          | | adbd   |+--------------------->| process    | |
          | |        |<---------------------++-^+---------+ |
          | +--------+ |     USB           |   ||           |
          +------------+                   |   || SOCKETS   |
                                           |   ||           |
                                           |   ||           |
                                           |   ||           |
                                           | +-+v---------+ |
                                           | | adb client | |
                                           | | process    | |
                                           | |            | |
                                           | |            | |
                                           | |            | |
                                           | +------------+ |
                                           |                |
                                           +----------------+

      ADB INTERNALS

      After

          +------------+                   +----------------+
          | PHONE      |                   | COMPUTER       |
          |            | Platform-specific | +------------+ |
          | +--------+ | drivers           | |            | |
          | | adbd   |+--------------------->|  Simulator | |
          | |        |<---------------------+|   Add-on   | |
          | +--------+ |     USB           | |     in     | |
          +------------+                   | |  Firefox   | |
                                           | |  process   | |
                                           | |            | |
                                           | |            | |
                                           | +------------+ |
                                           |                |
                                           +----------------+

      adb internals, deeper


      • Architecture broken up into "layers"
        • ADB commandline handling
        • ADB client services
        • Transport layer
        • fdevents
        • Platform-specific driver code


      ADB commandline handling


      • Mostly string parsing and other initialization code
      • Completely gone now

      ADB client services


      • A service returns a file descriptor
      • /* C code */
        int fd = _adb_connect("shell:ls");

        /* JS code */
        let output = readFully(fd);
        console.log(output);
      • Pretty cool abstraction
      • In some places unnecessary (should callback to JS directly)

      transport layer



      • Forwards packets from device over USB to sockets
        • Part of backend for the ADB client services
        • Due to limitations of Mac and Linux needs threads and synchronous reads and writes over USB
      • Used to tracks devices
        • Now pure JavaScript

      fdevents


      • Event loop system based off of file descriptors
        • Runs on main server thread
      • Backed by
        • Linux: epoll
        • Mac: select
        • Windows: select
      • Fun Fact:
        • There is a 1000+ line Windows shim for fdevents and exposing an interface to read/write from an int fd  just like on Unix

      platform specific driver code


      • Platform-specific
      • Long
      • Complex
      • Would be hard to port to JavaScript

      Adb internals, deeper, deeper

      Workers                                                           Per phone
            UI              Server           Device Loop           Input          Output
      
      JS:Simulator starts
         ADB    +
                |
                +--------->JS:Set up code
                                +
                                |
                                |      (really C to JS to UI to new worker to C)
                                v         
                           C:server_thread+-->C:Start device
                                +                loop
                                |               (look for devices)
                                v
                           C:fdevent_loop
                                +
                                |          on new device
      JS:device tracker<-------++------------------------------> C:Write to device
                                |
                                +--------------------------------------------->C:Read
      from device


      THanks

      questions?

      Bringing ADB to the Firefox Process

      By bkase

      Bringing ADB to the Firefox Process

      • 2,034