Pycon and Python

Allen & Chris

April, 2015

Part 1: What we retained from Pycon

Part 2: Allen's "wish I knew before"

Part 3: Chris' Pythonic stuff

Part 1

What we retained from Pycon

We're doing it right!

  • Different companies, different implementations
  • HAProxy, Nginx, uWSGI, StatsD, Graphite, Logstash
  • Amazing tools and services out there

New Relic

Code Climate

Sentry

Elastic's ELK stack

Runscope

You know we aren't testing enough

  • Unit test (mock, py.test)
  • Integration test (Betamax)

Profiling

Authentication / Security

  • Cross-site scripting
  • Cross-site request forgery
  • Cross-site SQL injection

MongoDB configuration

  • when optimized, 30-50k inserts Per Sec
  • 70k with Pymongo 3

Reveal.js

or slides.com

Part 2

Allen's "wish I knew before"

8 useful Python things I would have liked someone to show me when I was new

1. Installing Python
2. Using virtualenv and the requirements.txt file
3. Useful Python libraries
4. Know the difference between mutable and immutable data types
5. Positional argruments vs. keyword argruments
6. What is the __init__ file?
7. Functions return none implicitly
8. Which IDE should I use?

- You don't want to override Centos' system python.  Bad things will happen.
- follow the instructions here:
https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4
- tldr: use 'make altinstall' instead of 'make install'

1.1 Installing Python and pip

- Pip is a package management system used to install and manage software packages within Python.
- Most packages will be found off PyPI (Python Package Index)
- The same link shows you how to install pip

1.2 Installing Python and pip

virtualenv builds python sandboxes where you don't have to jeopordize your global environment.
You can safely:
 - install any python packages
 - switch between python versions
 - delete whole virtual environments

2.1 Using virtualenv and the requirements.txt file

2.2 virtuelenv

#Create a new virtual environment:
<shell>
$virtualenv /home/main/spotly27env
</shell>

#Activate it:
<shell>
$source /home/main/spotly27env/bin/activate
</shell>

<spotly27env>$
<spotly27env>$ which pip
/home/main/spotly27env/bin/pip

<spotly27env>$ which python
/home/main/spotly27env/bin/python2.7

</shell>

2.3 virtuelenv / pip

Install stuff:
<shell>
<spotly27env>$pip install requests
</shell>
Exit your virtualenv:
<shell>
<spotly27env>$deactivate
$
</shell>

2.4 requirements.txt

Flask==0.9
requests>=1.2.3

$pip install -r /path/to/requirements.txt 

2.5 virtualenv

-You can create as many different virtual environments as you want
-You can switch between each on the fly
-Python 3 has this functionality built in
-Using a virtualenv wrapper can make this even easier for you

3.1 Useful Python libraries

Standard Python Library
-It does things very well. Get familiar with it.
https://docs.python.org/2/library/

json - a compact json encoder and decoder

encoding example (serializing into json):
 

import json
data = [ { 'a':'A', 'b':(2, 4), 'c':3.0 } ]
data_string = json.dumps(data)
print 'JSON:', data_string
>>JSON: [{"a": "A", "c": 3.0, "b": [2, 4]}]

3.2 Useful Python libraries

xml.etree.ElementTree - easily parse xml

using the sample xml file:

<?xml version="1.0"?>
<data>
    <country name="Liechtenstein">
        <rank>1</rank>
        <year>2008</year>
        <gdppc>141100</gdppc>
        <neighbor name="Austria" direction="E"/>
        <neighbor name="Switzerland" direction="W"/>
    </country>
    <country name="Singapore">
        <rank>4</rank>
        <year>2011</year>
        <gdppc>59900</gdppc>
        <neighbor name="Malaysia" direction="N"/>
    </country>
    <country name="Panama">
        <rank>68</rank>
        <year>2011</year>
        <gdppc>13600</gdppc>
        <neighbor name="Costa Rica" direction="W"/>
        <neighbor name="Colombia" direction="E"/>
    </country>
</data>

3.2 Useful Python libraries

xml.etree.ElementTree - easily parse xml

using the sample xml file:

import xml.etree.ElementTree as ET
tree = ET.parse('my_xml_data.xml')
root = tree.getroot()
for child in root:
   print child.tag, child.attrib

country {'name': 'Liechtenstein'}
country {'name': 'Singapore'}
country {'name': 'Panama'}

3.3 datetime.timedelta

from datetime import datetime, timedelta
...
hours_to_grant = int(price_point['product']['hours_to_grant'])
access = datetime.utcnow().replace(tzinfo=pytz.utc) + timedelta(hours=hours_to_grant)

adding/subtracting units of time from a datetime object 

3.4 requests

import requests

r = requests.post("http://bugs.python.org", data={'@number': 12524, '@action': 'show'})
print(r.status_code, r.reason)
>>200, OK

standard way of making http requests with python

example of POST request:

This library makes it very easy to make different types of http requests.

i.e. post, get, delete
You can pass in headers, authentication

4.1 Know the difference between mutable and immutable data types

Quick review of the common data types:
number     # 1, 2.0
string     # "oh hello"
list       # ['abc', 2, 1.0]
dictionary # {'name': 'john','code':6734, 'dept': 'sales'}

 

Objects whose value can change are said to be mutable; objects whose value is unchangeable once they are created are called immutable

numbers and strings are immutable
lists and dictionaries are mutable

4.2 Know the difference between mutable and immutable data types

x = 1
y = 2
x = x + y

A new object is created for x

boring_courses = ['physics', 'chemistry']
boring_courses.append('allens python tutorial')

The variable boring_courses still points to the same list object

How does this help me?

4.3 Know the difference between mutable and immutable data types

def add_ketchup(ingredients):
    ingredients.append('ketchup')
    return ingredients


hotdog = ['bun', 'weiner']
hotdog_with_ketchup = add_ketchup(hotdog)


print hotdog
>> ['bun','weiner','ketchup']

print hotdog_with_ketchup
>> ['bun','weiner','ketchup']

WTH?
ingredients within the function is still referencing the same object as the original hotdog
You will probably run into this bug and hate yourself

5.1 Positional argruments vs. keyword argruments

def my_function(a, b, c = 1):
	return a + b + c

# Positional:
print my_function(1, 2, 3)

# Use the default:
print my_function(1, 2)

# Keyword (Named):
print my_function(b=1, a=2, c=3)

You can mix positional and keyword but the positional ones must come first in your definition.

5.2 Positional argruments vs. keyword argruments

# accept an arbritary number of positional argruments
def all_the_things(*args):
    for arg in args:
        print arg

self.all_the_things(1, 2, 3, 4, 5)

>> 1
>> 2
>> 3
>> 4
>> 5

# accept an arbritary number of keyword argruments
def all_the_things(**kwargs):
    for key, value in kwargs.iteritems():
         print "%s = %s" % (key, value)

self.all_the_things(first_name='Tom', last_name='Hanks')
>>first_name = Tom
>>last_name = Hanks

Now that you know the difference between positional and keyword argruments we can do this:

6. What is the __init__ file?

Blank __init__.py files will be put in folders to make Python treat the directory as a package to be available for import.

myproject/foo/my_module.py
myproject/foo/__init__.py


you can import the code in my_module.py as
 


If you remove the __init__.py file, Python will no longer look for submodules inside that directory, so attempts to import the module will fail.

The init file is usually blank but can be used to join multiple modules together for a single import

import spam.module

7. Functions return none implicitly

In Python every function returns a value and 'None' is what is returned if you dont explicitly return something else.

def make_a_sound(animal):
    if animal =='pig':
	return 'oink'
    elif animal == 'horse':
        return 'hayyyy'
    elif animal == 'cat':
        return 'meow'       

print self.make_a_sound('Horses')

>>None

8.1 Which IDE should I use?

Pycharm.
For Python development use pycharm.

 

Why?
-Code completion
-Automatic error detection and resolution
    - common design mistakes
    - syntax errors
-Code navigation

8.2 Which IDE should I use?

My Favorite Pycharm shortcuts

shift shift - brings up a search dialog to find a specific Python class
hold ctrl + click on a function - brings you to the function declaration
ctrl + shift + f - searches for text in your entire project
ctrl + alt + left/right arrow - navigates to the previous/next cursor position

Part 3

Chris thinks it's useful to know

Package for humans

If.......Else?

def get_bk(self, user_id):
    Try:
        subscription = self.find_subscription(user_id)
    Except NoSubscriptionError:
        return None

    If subscription.get('billing_key'):
        return subscription['billing_key']
    else:
        return None

If.......Else?

def get_bk(self, user_id):
    Try:
        subscription = self.find_subscription(user_id)
    Except NoSubscriptionError:
        return None

    If subscription.get('billing_key'):
        return subscription['billing_key']
    return None

If.......Else?

def print_bk(self, user_id):
    Try:
        subscription = self.find_subscription(user_id)
    Except NoSubscriptionError:
        raise SomethingsWrongError

    If subscription.get('billing_key'):
        print subscription['billing_key']
    else:
        raise SomethingsWrongError

If.......Else?

def print_bk(self, user_id):
    Try:
        subscription = self.find_subscription(user_id)
    Except NoSubscriptionError:
        raise SomethingsWrongError

    If not subscription.get('billing_key'):
        raise SomethingsWrongError
    print subscription['billing_key']
        

It's not True

def get_charge_count(self, sub_id):
    """
    for this example, let's assume subscription must have a field called 'total_charge_count'
    """
    try:
        subscription = find_subscription(sub_id)
    except NoSubscriptionError:
        raise SomethingsWrongError

    # checking if the field 'total_charge_count' exists
    if subscription.get('total_charge_count'):
        return subscription['total_charge_count']
    raise SomethingsWrongError

It's not True

  • None
  • False
  • 0
  • " ", ( ), { }, [ ]

is is cool, but...

  • is checks identity
  • == checks equality

is is cool, but...

print 4 is 4
True


print [] is []
False


print [] == []
True

Iterator vs Container

List Comprehension

employees = [
    {
        "id": 00158,
        "name": "allen huynh",
        "title" : "software developer"
    },
    {
        "id": 00219,
        "name": "chris sim",
        "title" : "software noob"
    }
]

x = []
for employee in employees:
    if get_title(employee) == 'software developer':
        first_name = get_first_name(employee)
        x.append(first_name)

print x
['Allen']
    

List Comprehension

employees = [
    {
        "id": 00158,
        "name": "allen huynh",
        "title" : "software developer"
    },
    {
        "id": 00219,
        "name": "chris sim",
        "title" : "software noob"
    }
]

x = [get_first_name(employee) for employee in employees if get_title(employee) == 'software developer']

print x
['Allen']
    

get value from Dictionary

price_point = {
    "_id" : ObjectId("540f9b7eee48f2j6f6901687"),
    "carrier_name" : "Claro CO",
    "platform" : "ng",
    "price" : "1234.56"
}


x = price_point['carrier_name']

x = price_point.get('carrier_name', 'Fido CA')

String formatting

%

.format( )

String formatting

name = "Chris Sim"
company = "airG"
anniversary_count = 2

print "%s has been working at %s for %d years" % (name, company, anniversary_count)

print "%(name)s has been working at %(company)s for %(anniv)d years" % \
    {"name": name, "company": company, "anniv": anniversary_count}


print "{} has been working at {} for {} years".format(name, company, anniversary_count)



companies = ['airG', 'Facebook', 'Twitter']

print "{0} has {0} community product and {1} and {2} are jealous of {0}".format(*companies)


airG has airG community product and Facebook and Twitter are jealous of airG

String formatting

pyformat.info

Super()

class Father(object):

    def spank_child(self, child_name):
        if self.spank(self.find_child(child_name)):
            return True, "he won't do it again"
        return False, "he's hiding somewhere"          


class HisSon(Father):
    
    # overriding
    def spank_child(self, child_name):
        spanked, conclusion = super(HisSon, self).spank_child(child_name)
        if not spanked:
            with self.wait_till_he_comes_out(child_name) as child:
                self.spank(child)
        return True, "he won't do it again"

Timezone...

if access and access > datetime.utcnow().replace(tzinfo=pytz.utc):
    subscription['product']['access'] = (access + timedelta(hours=hours_to_grant))

try/except/finally

try:
    speed = calculate_speed(arg1, arg2)
except ValueError:
    print "Speed could not be calculated"
except:
    log("Unexpected error:" + sys.exc_info()[0])
    raise
else:
    print "speed was {}".format(speed)
finally:
    clean_up()

__init__()

every class comes with it,

it's a constructor func

closures

def outer():
    x = 3
    def inner():
        print x
    return inner

foo = outer()

# will this print 3?

decorators

@ notation above function signature

@login_required
@autolock
@send_stats
def create_forum(forum_name, *args, **kwargs):
    pass
create_forum = login_required(autolock(send_stats(create_forum(forum_name, *args, **kwargs))))

Example with decorator

Example without decorator

context manager

'with' notation

with open('output.txt', 'w') as f:
    f.write('Hi there!')
f = open('output.txt', 'w')
f.write('Hi there!')
f.close()

Example with context manager

Example without context manager

context manager

'with' notation

def create(self, price_point, **kwargs):
    with lock_and_unlock(self, price_point, **kwargs) as lock_id:
        if lock_id is None:
            return {"error": "Account locked"}, 517
        user_id = kwargs.get('user_id')
        billing_key = kwargs.get('billing_key')

        status, subscription = self.check_subscription_record(billing_key, price_point, user_id)

Example from Fruitloops Codebase

Exceptions

class MyCustomError(Exception):
    pass


Thanks for joining us!

Alex's Q&A / Django session will follow in a week or 2

pycon2015

By Chris Sim