Groundwork

Tools and resources

Think Python: How to think like a Computer Scientist

IPython

• An advanced shell to develop Python interactively. Very powerful and feature rich.
• Can be integrated with vim (or other editor) so you can jump between editing text and an interactive session
• Allows quick access to documentation for standard and 3rd party libraries.

IPython notebook

PIP

pip is used to install packages for python.

virtualenv

virtualenv is used to "freeze" the requirements for a project. If project A needs foo v1.0 but project B needs foo v1.1 you can have these projects side by side.

pip install ipython

Extending python

Spyder IDe

Simple, designed to work well with IPython and scientific computing.

Pycharm

Fully featured IDE better for larger projects involving many libraries.

Integrated development environment (IDE)

Flask

Make python web apps quickly and easily. Useful for developing python-based RESTful APIs

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

Other cool stuff

• Click - make CLI programs easily
• pygal - make SVG charts and graph for the web
• python anywhere - host python code in the cloud
• c9 - cloud IDE, can pair program with someone else easily
• Hitchhiker's Guide to Python - This opinionated guide exists to provide both novice and expert Python developers a best-practice handbook to the installation, configuration, and usage of Python on a daily basis.
• gunicorn - serve your python app to the world
• sphinx - document your app

Python Basics

in a flash

Data Types

• Strings
• Integers
• Floating point
• List
  • can be a mix of other types
• Dictionaries
• Tuples

In [1]: name = "Racker"

In [2]: type(name)
Out[2]: str

In [3]: age = 15

In [4]: type(age)
Out[4]: int

In [5]: exact_age = 15.5

In [6]: type(exact_age)
Out[6]: float

In [8]: also_known_as = ["RackSpace", 1, 1.0]

In [9]: type(also_known_as)
Out[9]: list

In [10]: latam = {"uno": 1, "dos": 2, "tres": 3}

In [11]: type(latam)
Out[11]: dict

In [12]: lat_lon = (23.5, 16.1)

In [13]: type(lat_lon)
Out[13]: tuple

Slice Notation

We can select parts of strings and lists using a special type of notation.

http://stackoverflow.com/a/509295

In [1]: name = "RackSpace"

In [2]: print name[0:4]
Rack

In [3]: print name[5:]
pace

In [4]: print name[:]
RackSpace

In [5]: print name[::-1]
ecapSkcaR

In [6]: words = ["first", "second", "third"]

In [7]: words[0]
Out[7]: 'first'

In [8]: words[1]
Out[8]: 'second'

In [11]: words[0:2]
Out[11]: ['first', 'second']

Variables

• Assign using '='
• Can use variables to create other variables
• cannot start with a number

In [1]: name = "Racker"

In [3]: print name
Racker

In [4]: age = 14

In [5]: print age
14

In [1]: first = "Grahm"

In [2]: last = "Weston"

In [3]: full_name = first + last

In [4]: print full_name
GrahmWeston

In [5]: 2pac = "alive"
  File "<ipython-input-5-1afacaf60420>", line 1
    2pac = "alive"
       ^
SyntaxError: invalid syntax

Loops

• For
• While

In [1]: for i in range(1, 3):
   ...:     print i
   ...:     
1
2

In [2]: x = 0

In [4]: while x < 3:
   ...:     print x
   ...:     x += 1
   ...:     
0
1
2

Conditions

• if
• elif
• else
• Comparison operators
  • > greater than
  • < less than
  • == equal
  • != not equal
  • >= greater than or equal
  • <= less than or equal

In [5]: x = 0

In [6]: if x > 1:
   ...:     print "greater than one"
   ...: elif x < 0: 
   ...:     print "less than zero"
   ...: else:
   ...:     print "equal to zero"
   ...:     
equal to zero

In [7]: for i in range(1, 10):
   ...:     if i <= 0:
   ...:         print "at least zero"
   ...:     elif i >= 5:
   ...:         print "greater than five"
   ...:     elif i != 9:
   ...:         print "not there yet"
   ...:     else:
   ...:         print i
   ...:         
not there yet
not there yet
not there yet
not there yet
greater than five
greater than five
greater than five
greater than five
greater than five

Remember: range does not include the 2nd number so this is comparing the integers 1 thru 9

Functions

• We can define functions to reuse bits of code over and over again.
• What goes inside the parentheses are the arguments we give to the function to work with, in this case 'name' and 'number_of_times'

In [8]: def say_hello(name, number_of_times):
   ...:     for i in range(number_of_times):
   ...:         print "Hello " + name
   ...:         

In [10]: say_hello("Taylor", 3)
Hello Taylor
Hello Taylor
Hello Taylor


In [11]: say_hello("Rhodes", 2)
Hello Rhodes
Hello Rhodes

Libraries

Libraries are collections of functions that allows us to pick and choose the functionality of our programs.

When we import libraries, we can use the functions within.

Importing the 'math' library gives us constants like the value of pi or functions like Cos(x)

In [1]: import math

In [2]: print math.pi
3.14159265359

In [3]: print math.cos(90)
-0.448073616129

Working with files

We can use the open() function
 to create files.

In [1]: myfile = open('hello.txt', 'w')

In [2]: myfile.write("Hi there!")

In [3]: myfile.close()

In [4]: ls
hello.txt

In [5]: cat hello.txt
Hi there!

Subprocess

the admins best friend

baby steps

#!/usr/bin/env python

import subprocess

subprocess.call("ls")

In  this first example we have a few things to consider.

• We start with a shebang to allow the process to be run by the system
• we use the "import" keyword to bring in functions from a "library"
• we call the function from the library using "dot notation".

[root@python ~]# python one.py
file1	file2	file3	one.py

The results:

one.py

A slight modification

#!/usr/bin/env python

from subprocess import call

call("ls")

[root@python ~]# python one.2.py
file1 file2 file3 one.2.py one.py

The results are the same as before:

We can clean things up a little bit by changing the way that we import.

Here, we only import the "call" function from the "subprocess" library. We can then just use "call" directly.

Use caution, sometimes it's better to know where a function came from.

one.2.py

Something useful

• First we're creating a variable named 'folders' which is a list of strings
• subprocess will change directory to /var
• make a dir called 'www' and cd into that dir
• for each string in 'folders' repeat the same command
• we print the current working directory (cwd)
• finally call ls on the /var/www/ dir

#!/usr/bin/env python

import subprocess

folders = ['admin','ftp','http','https','subdomains']

subprocess.os.chdir("/var")

subprocess.os.mkdir("www")

subprocess.os.chdir("www")

for folder in folders:
   subprocess.os.mkdir(folder)

print subprocess.os.getcwd()

subprocess.call("ls")

[root@python ~]# python one.3.py
/var/www
admin  ftp  http  https  subdomains

one.3.py

Results:

Challenge

Create a script that:

• Installs apache
• Create vhost
  • use best practices
• Opens port 80 in IPtables
• Starts apache
• chkconfig apache on
• Logs information

Requests and beautiful soup

Making sense of The web

Requests: http for humans

Python’s standard urllib2 module provides most of the HTTP capabilities you need, but the API is thoroughly broken. It was built for a different time — and a different web. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks.

Things shouldn’t be this way. Not in Python.

>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}

Start simple

• We use HTTP "verbs" (GET, POST, etc.) to interact with the web
• we "get" the main site @ http://icanhazip.com
• we print the request, by default we are given the response code, in this case 200, ok

#!/usr/bin/env python

import requests

req = requests.get("http://icanhazip.com")
print req

two.py

[root@python ~]# python two.py
<Response [200]>

Results:

The request comes back with a lot of useful information

In [3]: req.
req.apparent_encoding  req.cookies            req.iter_content       req.ok                 req.request
req.close              req.encoding           req.iter_lines         req.raise_for_status   req.status_code
req.connection         req.headers            req.json               req.raw                req.text
req.content            req.history            req.links              req.reason             req.url

In [3]: req.headers
Out[3]:
{'access-control-allow-methods': 'GET',
 'access-control-allow-origin': '*',
 'connection': 'close',
 'content-length': '36',
 'content-type': 'text/plain; charset=UTF-8',
 'date': 'Tue, 23 Sep 2014 16:21:13 GMT',
 'server': 'Apache',
 'x-icanhaznode': 'icanhazip1.rs',
 'x-rtfm': "Learn about this site at http://bit.ly/icanhazip-faq and don't abuse the service",
 'x-you-should-apply-for-a-job': "If you're reading this, apply here: http://rackertalent.com/"}

For example:

Something useful

To make working with HTML easy, let's use BeautifulSoup

pip install beautifulsoup4

#!/usr/bin/env python

import requests
from bs4 import BeautifulSoup

zipcode = raw_input("Which zip code?\n")
req = requests.get('http://www.uszip.com/zip/' + zipcode)

soup = BeautifulSoup(req.content)
print soup.title.string

[root@py ~]# python two.2.py 
Which zip code?
78266
Garden Ridge, TX zip code

two.2.py

Results:

• We use the 'raw_input' function to take user input from the keyboard.
• Make the request
• use bs4 to make the content available for use.

We have some extra text at the end: 'zip code' How can we get rid of it?

Challenge

Use the json and requests libraries to obtain your RackSpace API authentication token

https://identity.api.rackspacecloud.com/v2.0/tokens

Regular Expressions

Now you've got two problems

Regular expressions

We can use regular expressions in python by using the 're' library

It is much more efficient to compile a regular expression for searching

In [1]: import re

In [2]: sentence = "The quick brown fox"

In [3]: regex = re.compile("The (.*) fox")

In [4]: re.findall(regex, sentence)
Out[4]: ['quick brown']

Capturing Stdout

• In this example we can capture the output of the command "ip a" and store it to a variable 'out'
• Notice that we have to pass the command as a list of strings
• Once we have the output we can search using regular expressions

In [1]: import re

In [2]: import subprocess

In [3]: out = subprocess.check_output(["ip", "a"])

In [4]: regex = re.compile("inet (.*)/")

In [5]: re.findall(regex, out)
Out[5]: ['127.0.0.1', '104.130.24.77', '10.176.4.153']

something useful

Ok, maybe not that useful

• We make a request to obtain the text of a project gutenberg book
• We search for all the times "The" appears in the text
• We store the answer in a variable and check it's length which tells us how many times the word appeared

In [1]: import requests

In [2]: import re

In [3]: req = requests.get("http://www.gutenberg.org/cache/epub/98/pg98.txt")

In [4]: regex = re.compile("The")

In [5]: result = re.findall(regex, req.content)

In [6]: print len(result)
985

Challenge

Make a website that dumps system info into its index.html

From a second system use python to get that data and use regex to get useful data about the first system

Script some action based on the information received.

Key Points

• use IPython to explore the functions in a library
• Build a script one step at a time. Write, test, fix, repeat.
• The trick is to mix python core features (loops, if, data types) with library functions (subprocess.call())

Bonus Round

List comprehension

List comprehensions

This is a way of describing what elements a list should contain

In [1]: numbers = [x for x in range(1, 10)]

In [2]: print numbers
[1, 2, 3, 4, 5, 6, 7, 8, 9]

In [3]: evens = [x for x in range(1, 25) if x % 2 == 0]

In [4]: print evens
[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24]

In [5]: squares = [x ** 2 for x in range(1, 10) if x % 2 == 0]

In [6]: print squares
[4, 16, 36, 64]