State of the Art

Tools available so far

Ant

. Former build system from Google

. Poor module support

. Very slow

Tools available so far

Maven

. Really ???

Tools available so far

Gradle

. Official build system from Google

. Faster than ANT

. Mediocre module support

. Very slow when many libraries involved

Don't scale :(

Tools available so far

We need a tool which :

. is flexible and easy to configure

. supports modules

. is fast (build & deployment)

. is well integrated with nowadays

  IDE's

Typical Android Build

Typical Android Build

An APK is composed of :

. a resources.arsc (xml layouts, drawables, assets...)

. a classes.dex (java code, native code, libraries...)

. some other stuff (manifest, certificates...)

Dependency graph

res

res

lib

lib

src

lib

res

res

R.java

lib

lib

lib

dx

ANT build steps

APK

(generate classes.dex)

APK

src

resources.arsc

dx

Typical Android Build

Typical Android Build

native libraries

classes.dex

resources.arsc

APK

bootstrap

APK

c

c

c

Phone

c

APK

APK

ADB

(5-6 MB/sc)

Typical Android build

Lack of speed :

. APK Packaging takes time 

  > 7 seconds for large apps

. APK sent through ADB ( 5-6 MB /sc )

. Package Manager does a lot of work

  (depending on size & target env.)

Sources : Michael Bolin & David Reiss - Droidcon NYC 2014

   Buck

Buck in a nutshell

. Inspired by Google Blaze

. Open-Sourced since April 2013

. Provides fast incremental builds

. Makes a heavy use of parallelization

. Used by FB to build its native apps

. Encourages small and reusables modules organized

  around BUCK files

Buck in a nutshell

. A module is defined with a BUCK file

. Declarative/imperative configuration

. Decompose the build process into "Build Rules"

. A build rule is a procedure for producing 1 or 0 output file

  from a set of input file

android_library(
    name='ui',
    srcs=glob(['src/main/java/**/*.java']),
    deps=[
        '//res/com/droidcon/montreal:res',
        '//java/src/com/droidcon/common:common',
        '//third-party/guave:guava',
    ],
    visibility=['PUBLIC'],
)

android_resource

android_resource(
  name = 'res1',
  package = 'com.droidcon.montreal',
  res = 'res/droidcon/montreal',
  deps = [
    '//res/com/droidcon:res',
  ],
)

int anim fade_in_seq 0x7f04000b
int attr actionBarSize 0x7f0100af
int color fbui_bg_dark 0x7f060071

R.txt

res

deps

An xml-change that leaves the index unaffected won't trigger a regeneration of R.txt.

android_library

android_library(
  name = 'lib1',
  srcs = glob(['*.java']),
  deps = [
    '//java/com/droidcon/lib2:src',
    '//res/com/droidcon/montreal:montreal-res',
  ],
)

R.txt

R.txt

R.txt

R.java

deps

src

jar

(without R.java) 

Implementation changes do not trigger dependents rebuild.

(merged)

classpath

android_binary

android_binary(
  name = 'droidcon-montreal',
  manifest = 'AndroidManifest.xml',
  deps = [
    '//java/com/droidcon:montreal',
  ],
)

res

resources.arsc

dex

res

R.java

dex

dex

APK

The dex merging has been speeded-up from O(N^2) to O(N lg N)

jar

jar

dex merge

(generates classes.dex)

Distributed cache

build rule

output file

Cache Key

Continuous Integration

Buck is fast !

. optimized dx

. daemon (buckd)

. heavy use of parallelisation

c

Java code

c

Resources

c

Dexing

c

APK packaging

Source : Michael Bolin & David Reiss - Droidcon NYC 2014

Wait ! There's more !

native libraries

classes.dex

resources.arsc

APK

Why my build is slow ?

bootstrap

APK

c

c

c

Phone

c

APK

APK

- ADB Protocol

- Package Manager

These steps must be minimized to increase efficiency

- APKBuilder

Slow build reasons :

the 64K limit

 Google Play Services
 Guava
 AppCompat V7
 Android Support V4
 Jackson
 Joda-Time
 Apache Commons Lang3
 Apache Commons IO
 Gson

Name  

Version

Overall Method Count

 6.1.11
 18.0
 21.0.0
 21.0.0
 2.4.3
 2.5
 3.3.2
 2.4
 2.3

29,460
14,842
12,324
 8,078
6,731
5,025
3,582
1,571
1,243

Source : https://www.contentful.com/blog/2014/10/30/android-and-the-dex-64k-methods-limit/

native libraries

classes.dex

resources.arsc

Multi-dexed app

Exopackage

bootstrap

APK

c

c

c

Phone

c

APK

APK

- APK Builder only needed for 

- Package Manager limited for

- Reduce deployment time

Pros :

  res. & native code changes

  Java changes only

Thanks to the 64K limit, multidexing is possible

Exopackage

android_binary(
  name = 'antennapod',
  manifest = 'AndroidManifest.xml',
  target = 'Google Inc.:Google APIs:19',
  keystore = ':debug_keystore',
  use_split_dex = True,
  exopackage_modes = ['secondary_dex'],
  primary_dex_patterns = [
    '^de/danoeh/antennapod/AppShell^',
    '^de/danoeh/antennapod/BuildConfig^',
    '^com/droidcon/buck/android/support/exopackage/',
  ],  
  deps = [
    ':application-lib',
    ':main-lib',
  ],
)

The app entry point has to be changed, along with its configuration.

In production, all those tweaks are disabled, and the APK expects to contain everything in order to run properly.

AntennaPod + Exopackage

Source : https://facebook.github.io/buck/article/exopackage.html

Buck in brief

. Fast, modular and stable

. Drastically reduces build and deployment time

. Migration from Gradle to Buck is (very) easy

. Good support and promising

. Some limits however

Sources

Buck repo :

. https://facebook.github.io/buck/

Discussion group:

. https://groups.google.com/forum/#!forum/buck-build

Buck's documentation

. https://github.com/facebook/buck

Extra thanks for David Reiss, Michael Bolin & Simon Stewart

Text

c

Text

I am a Scala developer at 1Science, but also a mobile dev freelancer. 

francistoth@coding-hipster.com

Thank you ! Questions ?

RDD

Partition

Partition

Partition

Cluster

HadoopRDD

FilteredRDD

MappedRDD

HadoopRDD

FilteredRDD

MappedRDD

ShuffledRDD

textfile(...)

.filter(...)

.map(...)

.count()

textfile(...).filter(...).map(...).count()

Worker 1

Worker 2

Worker N

Driver

Worker 1

Worker 2

Worker N

Driver

collect()

}

{

Worker 1

Worker 2

Worker N

Driver

Worker 1

Worker 2

Worker N

Driver

HadoopRDD

MappedRDD

MappedRDD

ShuffledRDD

textfile

flatMap

map

reduceByKey

filter

MappedRDD

Stage 1

Stage 2

Stage 1

HadoopRDD

flatMap

map

reduceByKey

filter

textfile(hdfs://...)

    .flatMap(_.split(" "))

    .map((_ , 1))

    .reduceByKey(_ + _)

    .filter(_ > 1)

HadoopRDD

flatMap

map

reduceByKey

filter

textfile(hdfs://...)

    .flatMap(_.split(" "))

    .map((_ , 1))

    .reduceByKey(_ + _)

    .filter(_ > 1)

Stage 1

Stage 2

HadoopRDD

flatMap

map

reduceByKey

filter

textfile(hdfs://...)

    .flatMap(_.split(" "))

    .map((_ , 1))

    .reduceByKey(_ + _)

    .filter(_ > 1)

(a, 1)

(b, 1)

(b, 1)

(a, 1)

(b, 1)

(a, 1)

(b, 1)

(b, 1)

(a, 1)

(b, 1)

(b, 1)

(a, 1)

(b, 2)

(a, 1)

(b, 1)

(a, 1)

(b, 1)

(a, 1)

(b, 1)

(b, 1)

(a, 3)

Shuffle

}

{

(a, 1)

(b, 2)

(b, 5)

(a, 1)

(b, 1)

(b, 1)

(a, 1)

(b, 1)

(a, 1)

(b, 1)

(b, 1)

(a, 1)

(b, 2)

(c, 1)

(b, 1)

(c, 1)

(c, 4)

(a, 1)

(b, 3)

(c, 6)

Provinces / Territories

People in Canada

Quebec

New Brunswick

British Colombia

Provinces / Territories

People in Canada

People in Canada

People in Canada

People in Canada

People in World

All data from both RDDs

Final joined output

People in Canada

People in World

Final joined output

{ Filter people from Canada }

Driver

Executor

Cache

Executor

Cache

Executor

Cache

Cluster Manager

Driver

Executor

Cache

Executor

Cache

Executor

Cache

Driver

S3

Dataframe

{ JSON }

and more...

Dataframe

{ JSON }

Dataframe

Dataframe

Dataframe

{ Schema }

Dataframe

Rows

+

SQL

Confoo 2016 - Francis Toth

Spark

batches of input data

Spark Streaming

Receivers

input of data streams

results published to external systems

Source : Learning Spark by Andy Konwinski, Holden Karau, and Patrick Wendell

Streaming