Data Science

Drew Conway, IQT Quarterly, 2011

Courtesy: Oracle 

Source: http://columbiadatascience.com/

Exploration

Engineering

Machine Learning, EDA etc.,

"Big Data"

Data Plumbing

Quick turnaround?

Automation

Framework

Working with data is messy

Be language and tool agnostic

Get familiar with the command line

In the beginning was the command line

UNIX metaphors - pipe, filter, composable programs

The data landscape

Data sources

• Server logs
• Sensors
• Human generated (twitter, geolocation)
• APIs
• Databases
  • RDBMS
  • No/NotOnly/New SQL databases

Data Formats

• API
  • JSON
  • XML
• HTML - web scraping
• Delimited files
• Binary
  • XLS
  • PDF
• Legacy
• "unstructured" data

Tools of the trade

Skill sets

• Basic programming ability
  • procedural
  • object based
  • scripting
• Text processing
• Network / HTTP 
• Databases (and some SQL)

What's in your toolkit?

• Command line tools
• Programming language
  • Python. Duh!
• Data analysis Tools
  • PyData, R etc., 
• Language Libraries 
• Database query tools
• Spreadsheet (like) programs

Command line tools

• wget
• curl
• head/tail
• grep
• sort
• cut
• sed
• awk
• uniq

textproc

Unix text proc

Of the UNIX heritage

Third party tools: jq, CSVKit etc.,

The CSV Challenge

Why Python

• Community
• Availability
• Approachability
• Multi-faceted
• Well understood (limitations and strengths)
• A fair balance of expressiveness, power, permissibility
• Team player

Batteries included

• String
• Process
• OS
• CSV
• Glob
• Arg/Opt parse
• ConfigParse
• Json
• XML/ElementTree
• Collections

Third party Libraries

• Requests - HTTP client
• BeautifulSoup - HTML scraping
• Arrow - date and time
• SQLAlchemy - database access

# Installing third party packages:

pip install arrow
pip install requests

# PROTIP: look up virtualenv and virtualenvwrapper
# Alternate: conda [part of Continuum.io / anaconda package]

Text Encoding

Time

strptime

PROTIP: strftime.org

# From string to time
from time import strptime

print strptime('2015-05-01', '%y-%m-%d')
time.struct_time(tm_year=2015, tm_mon=5, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=4, tm_yday=121, tm_isdst=-1)


#From time to string
from time import strftime
print strftime('%Y-%m-%dT%H:%M:%S', datetime.timetuple(datetime.now()))

Time: Arrow

>>> import arrow
>>> utc = arrow.utcnow()
>>> utc
<Arrow [2013-05-11T21:23:58.970460+00:00]>

>>> utc = utc.replace(hours=-1)
>>> utc
<Arrow [2013-05-11T20:23:58.970460+00:00]>

>>> local = utc.to('US/Pacific')
>>> local
<Arrow [2013-05-11T13:23:58.970460-07:00]>

>>> arrow.get('2013-05-11T21:23:58.970460+00:00')
<Arrow [2013-05-11T21:23:58.970460+00:00]>

>>> local.timestamp
1368303838

>>> local.format('YYYY-MM-DD HH:mm:ss ZZ')
'2013-05-11 13:23:58 -07:00'

>>> local.humanize()
'an hour ago'

Fetching Data off the web

import requests
import json

r = requests.get('http://httpbin.org/get')
if r.status_code == 200:
    data = json.loads(r.text)
    print "Your IP is: ", data['origin']
    print "FYI, the text response: \n", r.text
    print "prettified json: \n",  json.dumps(data, sort_keys=True,
                  indent=4, separators=(',', ': '))

Requests/GET

Your IP is:  108.223.55.55
FYI, the text response: 
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.5.1 CPython/2.7.9 Darwin/14.3.0"
  }, 
  "origin": "108.223.55.55", 
  "url": "http://httpbin.org/get"
}

prettified json: 
{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "python-requests/2.5.1 CPython/2.7.9 Darwin/14.3.0"
    },
    "origin": "108.223.55.55",
    "url": "http://httpbin.org/get"
}

import requests
data = {'name': 'Who?', 'age': 55}
r = requests.get('http://example.com', data=data)
r.text

Requests/POST

Interacting with APIs

httpie

comes in handy for  automating

• testing / learning new APIs
• data downloads
• automating periodic data fetches

http://httpbin.org/

# You can install httpbin as a library from PyPI and run it as a WSGI app.

$ pip install httpbin
$ gunicorn httpbin:app

Databases

SQL Skills 

• SELECT statement
• WHERE clause
• JOINs
• Basics of indexing 

Command line tools  in Python

Python libraries for building 

Command line tools

Inbuilt Libraries

• fileinput
• argparse

Thirdparty Libraries

• docopt

fileinput

#!/usr/bin/env python

# Program to print words that start with an uppercase letter
# example usage: cat test.txt | ./filterr

import fileinput
import string


for line in fileinput.input():
    words = line.split()
    for word in words:
        first_letter = ord(word[0])
        if first_letter >=65 and first_letter <=90:
            print word

cat ~/tmp/habits.txt| ./filterr 
Habits
John
Doe
March

argparse

import argparse

parser = argparse.ArgumentParser()

parser.add_argument('-s', action='store', dest='simple_value',
                    help='Store a simple value')

parser.add_argument('-c', action='store_const', dest='constant_value',
                    const='value-to-store',
                    help='Store a constant value')

parser.add_argument('-t', action='store_true', default=False,
                    dest='boolean_switch',
                    help='Set a switch to true')
parser.add_argument('-f', action='store_false', default=False,
                    dest='boolean_switch',
                    help='Set a switch to false')

parser.add_argument('-a', action='append', dest='collection',
                    default=[],
                    help='Add repeated values to a list',
                    )

parser.add_argument('-A', action='append_const', dest='const_collection',
                    const='value-1-to-append',
                    default=[],
                    help='Add different values to list')
parser.add_argument('-B', action='append_const', dest='const_collection',
                    const='value-2-to-append',
                    help='Add different values to list')

parser.add_argument('--version', action='version', version='%(prog)s 1.0')
results = parser.parse_args()
print 'simple_value     =', results.simple_value
print 'constant_value   =', results.constant_value
print 'boolean_switch   =', results.boolean_switch
print 'collection       =', results.collection
print 'const_collection =', results.const_collection

argparse ..

$ python argparse_action.py -h

usage: argparse_action.py [-h] [-s SIMPLE_VALUE] [-c] [-t] [-f]
                          [-a COLLECTION] [-A] [-B] [--version]

optional arguments:
  -h, --help       show this help message and exit
  -s SIMPLE_VALUE  Store a simple value
  -c               Store a constant value
  -t               Set a switch to true
  -f               Set a switch to false
  -a COLLECTION    Add repeated values to a list
  -A               Add different values to list
  -B               Add different values to list
  --version        show program's version number and exit

docopt

"""Naval Fate.

Usage:
  naval_fate.py ship new <name>...
  naval_fate.py ship <name> move <x> <y> [--speed=<kn>]
  naval_fate.py ship shoot <x> <y>
  naval_fate.py mine (set|remove) <x> <y> [--moored | --drifting]
  naval_fate.py (-h | --help)
  naval_fate.py --version

Options:
  -h --help     Show this screen.
  --version     Show version.
  --speed=<kn>  Speed in knots [default: 10].
  --moored      Moored (anchored) mine.
  --drifting    Drifting mine.

"""
from docopt import docopt


if __name__ == '__main__':
    arguments = docopt(__doc__, version='Naval Fate 2.0')
    print(arguments)

Summary

• It is still programming
• Embrace the chaos
• Plumbing vs Architecting vs Building
• Document your steps ("Repeatable")
• Python is an expressive language. Learn it. Use it.

Thanks!

Contact:

@btbytes

http://github.com/btbytes

pradeep@btbytes.com

Slides will be on indypy's github repo