Making Python
work for big web data

web data

Each crawl is billions of pages...

Released

totally free

What good is data if it's not used?

 

Accessing & using it needs to be easy

Sexy & simple science

iPython Notebook

Numpy & SciPy

Pandas

Scikit Learn

...but it's less simple when handling
big data

public static void main(String[] args)

Creating a starter kit for Python

Simple to use framework for experiments
+

A Python library for accessing files on S3
=

mrjob to the rescue (thx Yelp!)

mrjob = trivial Hadoop streaming with Python

mrjob conveniences

python mrcc.py --conf-path mrjob.conf <in>

python mrcc.py -r emr --conf-path mrjob.conf <in>

mrjob annoyances

(✖ from collections import Counter) 

Luckily that can be fixed! Python 2.7 installation...

    # Many Bothans died to bring us this information
    ami_version: 3.0.4
    interpreter: python2.7
    bootstrap:
    - sudo yum install -y python27 python27-devel gcc-c++
    - sudo python2.7 get-pip.py#
    - sudo pip2.7 install boto mrjob simplejson

Big issue: grabbing the data from S3?

Python has no streaming decompression of gzip

Pulling to local storage and then processing
= terrible performance

gzipstream = streaming gzip decompression

Anything that has read()
(S3, HTTP, Gopher, IPoAC, ...)

gzipstream handles multi-part gzip files
(our web data is stored like this for random access)

gzip + gzip + ... + gzip + gzip

Beautiful but slow :'(

Saving Python's performance

Heavy lifting with faster languages

<haiku>
If I/O is slow,
Python will make you happy,
else Java or C*
</haiku>

Saving Python's performance

Process once, store forever

Processing all the raw data each time is really inefficient
Create derived datasets that have a single focus

For example, Common Crawl releases processed datasets:

• metadata (30% of the original)
  
• extracted text (15% of the original)
  

Friends of Common Crawl make derived datasets

• hyperlink graph (64 billion edges)
• extracted table text, microdata, and more..!

Common Crawl mrjob kit

pip install -r requirements.txt
# Add your AWS credentials
python mrcc.py -r emr --conf-path mrjob.conf input/test-100.txt

https://github.com/commoncrawl/cc-mrjob

Help us make it trivial to use Python to access the world's greatest collection of human knowledge!

Check out and use:
https://github.com/commoncrawl/cc-mrjob

Stephen Merity

smerity.com / @smerity

@commoncrawl

Attributions

Universities should teach big data courses with ... y'know ...