Python for the web admin
A quick intro for Linux Admins
Groundwork
Tools and resources
- An advanced shell to develop Python interactively. Very powerful and feature rich.
- Can be integrated with vim (or other editor) so you can jump between editing text and an interactive session
- Allows quick access to documentation for standard and 3rd party libraries.
pip is used to install packages for python.
virtualenv is used to "freeze" the requirements for a project. If project A needs foo v1.0 but project B needs foo v1.1 you can have these projects side by side.
pip install ipythonExtending python
Simple, designed to work well with IPython and scientific computing.
Fully featured IDE better for larger projects involving many libraries.
Integrated development environment (IDE)
Make python web apps quickly and easily. Useful for developing python-based RESTful APIs
from flask import Flask
app = Flask(__name__)
@app.route("/")
def hello():
return "Hello World!"
if __name__ == "__main__":
app.run()Other cool stuff
- Click - make CLI programs easily
- pygal - make SVG charts and graph for the web
- python anywhere - host python code in the cloud
- c9 - cloud IDE, can pair program with someone else easily
- Hitchhiker's Guide to Python - This opinionated guide exists to provide both novice and expert Python developers a best-practice handbook to the installation, configuration, and usage of Python on a daily basis.
- gunicorn - serve your python app to the world
- sphinx - document your app
Python Basics
in a flash
Data Types
- Strings
- Integers
- Floating point
- List
- can be a mix of other types
- Dictionaries
- Tuples
In [1]: name = "Racker"
In [2]: type(name)
Out[2]: str
In [3]: age = 15
In [4]: type(age)
Out[4]: int
In [5]: exact_age = 15.5
In [6]: type(exact_age)
Out[6]: float
In [8]: also_known_as = ["RackSpace", 1, 1.0]
In [9]: type(also_known_as)
Out[9]: list
In [10]: latam = {"uno": 1, "dos": 2, "tres": 3}
In [11]: type(latam)
Out[11]: dict
In [12]: lat_lon = (23.5, 16.1)
In [13]: type(lat_lon)
Out[13]: tuple
Slice Notation
We can select parts of strings and lists using a special type of notation.
In [1]: name = "RackSpace"
In [2]: print name[0:4]
Rack
In [3]: print name[5:]
pace
In [4]: print name[:]
RackSpace
In [5]: print name[::-1]
ecapSkcaR
In [6]: words = ["first", "second", "third"]
In [7]: words[0]
Out[7]: 'first'
In [8]: words[1]
Out[8]: 'second'
In [11]: words[0:2]
Out[11]: ['first', 'second']
Variables
- Assign using '='
- Can use variables to create other variables
- cannot start with a number
In [1]: name = "Racker"
In [3]: print name
Racker
In [4]: age = 14
In [5]: print age
14
In [1]: first = "Grahm"
In [2]: last = "Weston"
In [3]: full_name = first + last
In [4]: print full_name
GrahmWeston
In [5]: 2pac = "alive"
File "<ipython-input-5-1afacaf60420>", line 1
2pac = "alive"
^
SyntaxError: invalid syntax
Loops
- For
- While
In [1]: for i in range(1, 3):
...: print i
...:
1
2
In [2]: x = 0
In [4]: while x < 3:
...: print x
...: x += 1
...:
0
1
2
Conditions
- if
- elif
- else
- Comparison operators
- > greater than
- < less than
- == equal
- != not equal
- >= greater than or equal
- <= less than or equal
In [5]: x = 0
In [6]: if x > 1:
...: print "greater than one"
...: elif x < 0:
...: print "less than zero"
...: else:
...: print "equal to zero"
...:
equal to zero
In [7]: for i in range(1, 10):
...: if i <= 0:
...: print "at least zero"
...: elif i >= 5:
...: print "greater than five"
...: elif i != 9:
...: print "not there yet"
...: else:
...: print i
...:
not there yet
not there yet
not there yet
not there yet
greater than five
greater than five
greater than five
greater than five
greater than five
Remember: range does not include the 2nd number so this is comparing the integers 1 thru 9
Functions
- We can define functions to reuse bits of code over and over again.
- What goes inside the parentheses are the arguments we give to the function to work with, in this case 'name' and 'number_of_times'
In [8]: def say_hello(name, number_of_times):
...: for i in range(number_of_times):
...: print "Hello " + name
...:
In [10]: say_hello("Taylor", 3)
Hello Taylor
Hello Taylor
Hello Taylor
In [11]: say_hello("Rhodes", 2)
Hello Rhodes
Hello Rhodes
Libraries
Libraries are collections of functions that allows us to pick and choose the functionality of our programs.
When we import libraries, we can use the functions within.
Importing the 'math' library gives us constants like the value of pi or functions like Cos(x)
In [1]: import math
In [2]: print math.pi
3.14159265359
In [3]: print math.cos(90)
-0.448073616129
Working with files
We can use the open() function
to create files.
In [1]: myfile = open('hello.txt', 'w')
In [2]: myfile.write("Hi there!")
In [3]: myfile.close()
In [4]: ls
hello.txt
In [5]: cat hello.txt
Hi there!Subprocess
the admins best friend
baby steps
#!/usr/bin/env python
import subprocess
subprocess.call("ls")
In this first example we have a few things to consider.
- We start with a shebang to allow the process to be run by the system
- we use the "import" keyword to bring in functions from a "library"
- we call the function from the library using "dot notation".
[root@python ~]# python one.py
file1 file2 file3 one.pyThe results:
one.py
A slight modification
#!/usr/bin/env python
from subprocess import call
call("ls")
[root@python ~]# python one.2.py
file1 file2 file3 one.2.py one.pyThe results are the same as before:
We can clean things up a little bit by changing the way that we import.
Here, we only import the "call" function from the "subprocess" library. We can then just use "call" directly.
Use caution, sometimes it's better to know where a function came from.
one.2.py
Something useful
- First we're creating a variable named 'folders' which is a list of strings
- subprocess will change directory to /var
- make a dir called 'www' and cd into that dir
- for each string in 'folders' repeat the same command
- we print the current working directory (cwd)
- finally call ls on the /var/www/ dir
#!/usr/bin/env python
import subprocess
folders = ['admin','ftp','http','https','subdomains']
subprocess.os.chdir("/var")
subprocess.os.mkdir("www")
subprocess.os.chdir("www")
for folder in folders:
subprocess.os.mkdir(folder)
print subprocess.os.getcwd()
subprocess.call("ls")[root@python ~]# python one.3.py
/var/www
admin ftp http https subdomainsone.3.py
Results:
Challenge
Create a script that:
- Installs apache
- Create vhost
- use best practices
- Opens port 80 in IPtables
- Starts apache
- chkconfig apache on
- Logs information
Requests and beautiful soup
Making sense of The web
Requests: http for humans
Python’s standard urllib2 module provides most of the HTTP capabilities you need, but the API is thoroughly broken. It was built for a different time — and a different web. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks.
Things shouldn’t be this way. Not in Python.
>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}Start simple
- We use HTTP "verbs" (GET, POST, etc.) to interact with the web
- we "get" the main site @ http://icanhazip.com
- we print the request, by default we are given the response code, in this case 200, ok
#!/usr/bin/env python
import requests
req = requests.get("http://icanhazip.com")
print reqtwo.py
[root@python ~]# python two.py
<Response [200]>Results:
The request comes back with a lot of useful information
In [3]: req.
req.apparent_encoding req.cookies req.iter_content req.ok req.request
req.close req.encoding req.iter_lines req.raise_for_status req.status_code
req.connection req.headers req.json req.raw req.text
req.content req.history req.links req.reason req.urlIn [3]: req.headers
Out[3]:
{'access-control-allow-methods': 'GET',
'access-control-allow-origin': '*',
'connection': 'close',
'content-length': '36',
'content-type': 'text/plain; charset=UTF-8',
'date': 'Tue, 23 Sep 2014 16:21:13 GMT',
'server': 'Apache',
'x-icanhaznode': 'icanhazip1.rs',
'x-rtfm': "Learn about this site at http://bit.ly/icanhazip-faq and don't abuse the service",
'x-you-should-apply-for-a-job': "If you're reading this, apply here: http://rackertalent.com/"}For example:
Something useful
To make working with HTML easy, let's use BeautifulSoup
pip install beautifulsoup4#!/usr/bin/env python
import requests
from bs4 import BeautifulSoup
zipcode = raw_input("Which zip code?\n")
req = requests.get('http://www.uszip.com/zip/' + zipcode)
soup = BeautifulSoup(req.content)
print soup.title.string
[root@py ~]# python two.2.py
Which zip code?
78266
Garden Ridge, TX zip code
two.2.py
Results:
- We use the 'raw_input' function to take user input from the keyboard.
- Make the request
- use bs4 to make the content available for use.
We have some extra text at the end: 'zip code' How can we get rid of it?
Challenge
Use the json and requests libraries to obtain your RackSpace API authentication token
https://identity.api.rackspacecloud.com/v2.0/tokens
Regular Expressions
Now you've got two problems
Regular expressions
We can use regular expressions in python by using the 're' library
It is much more efficient to compile a regular expression for searching
In [1]: import re
In [2]: sentence = "The quick brown fox"
In [3]: regex = re.compile("The (.*) fox")
In [4]: re.findall(regex, sentence)
Out[4]: ['quick brown']Capturing Stdout
- In this example we can capture the output of the command "ip a" and store it to a variable 'out'
- Notice that we have to pass the command as a list of strings
- Once we have the output we can search using regular expressions
In [1]: import re
In [2]: import subprocess
In [3]: out = subprocess.check_output(["ip", "a"])
In [4]: regex = re.compile("inet (.*)/")
In [5]: re.findall(regex, out)
Out[5]: ['127.0.0.1', '104.130.24.77', '10.176.4.153']something useful
Ok, maybe not that useful
- We make a request to obtain the text of a project gutenberg book
- We search for all the times "The" appears in the text
- We store the answer in a variable and check it's length which tells us how many times the word appeared
In [1]: import requests
In [2]: import re
In [3]: req = requests.get("http://www.gutenberg.org/cache/epub/98/pg98.txt")
In [4]: regex = re.compile("The")
In [5]: result = re.findall(regex, req.content)
In [6]: print len(result)
985
Challenge
Make a website that dumps system info into its index.html
From a second system use python to get that data and use regex to get useful data about the first system
Script some action based on the information received.
Key Points
- use IPython to explore the functions in a library
- Build a script one step at a time. Write, test, fix, repeat.
- The trick is to mix python core features (loops, if, data types) with library functions (subprocess.call())
Bonus Round
List comprehension
List comprehensions
This is a way of describing what elements a list should contain
In [1]: numbers = [x for x in range(1, 10)]
In [2]: print numbers
[1, 2, 3, 4, 5, 6, 7, 8, 9]
In [3]: evens = [x for x in range(1, 25) if x % 2 == 0]
In [4]: print evens
[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24]
In [5]: squares = [x ** 2 for x in range(1, 10) if x % 2 == 0]
In [6]: print squares
[4, 16, 36, 64]