# optional step
sudo yum install git
# clone the repo
git clone https://github.com/jeremyprice/RU_Python_IV
python --version
#install on CentOS 7
sudo yum install -y python-virtualenv
# create a new virtual env
virtualenv -p python3 <directory_name>
# work in the new virtual env
cd <directory_name>
source bin/activate
# stop working on the virtual env
deactivate
# make sure you are in the virtualenv
pip install ipython
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Line #1 = shebang
# Line #2 = encoding declaration
"""Docstrings"""
# Inline documentation
Import(s)
Statements
and del from not while
as elif global or with
assert else if pass yield
break except import print
class exec in raise
continue finally is return
def for lambda try
tup = (1, "a", 3)
tup = tuple()
tup = (1,) # Comma needed for single element
gen = (x**2 for x in range(1,4)) # Not a tuple!
Lists are sequences that can be changed (“mutable”)
Lists can contain other object types and be sliced
A list is a set of objects separated by commas and enclosed in square brackets
Examples:
lst = [1, "a", 3]
lst = list()
lst = [x for x in xrange(5)]
A dictionary (“dict”) type is a set of expressions that are associated 1 to 1 with other expressions
A dict provides a handy “mapping” between a “key” expression and its associated “value” expression
A dict is not an ordered sequence, it’s a mapping.
Examples:
dic = {"a": 1}
dic = dict(a=1)
dic = {x: chr(x) for x in range(97, 107)}
st = {1,2,2,3}
st = set([1,2,2,3])
st = {x for x in range(1,4)}
yield is similar to return, but suspends execution of the called function instead of ending the function
Saves the state
On the next call to the function:
yield picks up where it left off
with all identifier values still holding the same values
A function with a yield statement is a generator
Example:
def foo(bar=10):
i = 1
while i < bar:
yield i
i = i + 1
The with statement is used to run a suite of other statements under a “context manager”
Special methods __enter__() and __exit__() are called to setup and takedown a “context”
Common for doing i/o (which auto-closes the file handle) and threading (which auto-acquires and releases lock)
with open("foo.txt", "r") as infile:
all_txt = infile.read()
To make a set of identifiers in another module available for use by your module, you must first use import or from
import pulls in identifiers from module(s) but the module name must be used as a prefix
only module name is added to local namespace
"from" pulls in identifiers from modules but avoids need to prefix with the module name
identifier is added to local namespace
All the docs for version 3:
Standard library:
lab_pyreview.py
Python can spawn and control entire processes using the subprocess module
Generally means redirecting the basic file descriptors (stdin, stdout, stderr) to gain programmatic access
Forks a new process and uses pipes for redirection
As usual, beware of invoking a process based on direct or indirect user input
Contains many convenience functions for creating and managing subprocesses
subprocess.Popen class is the most full-featured interface
subprocess.run() is the simplest interface
run() - spawn a subprocess, wait for completion, return an object representing the process
raises CalledProcessError exception if exit code isn't 0
Caveats:
Be careful using pipes for stdout or stderr
Only use shell=True if need shell features
# examples
proc = subprocess.run(['ls', '-al'])
proc = subprocess.run('false')
Swiss army knife of subprocess spawning
Can map stderr to stdout using subprocess.STDOUT
Use subprocess.PIPE in stdin, stdout, stderr
Check out all the args/kwargs in the docs:
https://docs.python.org/3/library/subprocess.html#subprocess.Popen
- poll() sees if child is terminated
- wait() waits for child to terminate
- communicate(input=None) sends input to child stdin. Use instead of stdin.write()
- send_signal(signal) sends given signal to child
- terminate() and kill() send respective signals to child
# example
subproc = subprocess.Popen(['ls', '-al'])
(stdoutdata, stderrdata) = subproc.communicate()
Note: if args is a string, must use shell=True to specify args
Once again, don’t use shell=True with user entered data
shell=False does not call a shell either directly or indirectly, allowing all characters (even metacharacters) to be passed safely
Popen.std* are file objects if std*=PIPE was used in Popen constructor
Popen.pid is child process ID
Popen.returncode is child exit code. Set by poll(), wait(). None if child not terminated. If negative, indicates child was terminated by that signal number.
lab_subprocess.py
Threads are independently scheduled streams of instructions that run within a single OS process
Threads have their own stack, registers, scheduling attributes, priorities, signal mask, and thread local memory
Threads allow logical concurrency of execution (and possibly parallel execution if configured)
Threads introduce the need for synchronization!
Threads share the same instructions (bytecodes) same identifier bindings, same open files, and other resources
Threads cannot exist outside of an OS process
Threads are “lightweight” – the overhead of creating them is much less than creating a real process
Threads are used for capturing a higher % of available cycles on a single CPU, realtime systems, an asynchronous event handling.
Peer
All threads work on the backlog without a “manager” thread
All threads work on a protected/synchronized queue
Pipeline
Each thread does a stage of a work pipeline
Each thread accepts work from “previous” thread, passes to “next” thread
Contains all the necessary primitives to handle threading:
threading.Thread() - create a thread from a callable
threading.Condition() - create a condition to use for waiting
threading.Event() - create an event to use for waiting
threading.Lock()/RLock() - Lock a critical section or variable
threading.Semaphore() - Classic acquire/release semaphore
threading.Timer() - run a callable after timer expires
import threading
import time
def xyz():
for i in range(10):
print("thread")
time.sleep(1.5)
if __name__ == "__main__":
th = threading.Thread(target=xyz)
th.start()
for i in range(10):
print("main")
time.sleep(1)
Calling another threads join() causes the caller to wait for thread to end
Pass arguments to new thread using the args and kwargs keywords
Locks have acquire() and release() methods that can be automatically invoked on the with statement
Condition variables have a wait() to wait for the condition to be reached
Condition variables also have notify() and notify_all() that can be used to indicate to other threads that the condition variable has changed
Semaphore objects also have acquire() and release() for easy use with with
Event objects have set(), clear(), and wait() which indicate if event has happened or not
import queue
import threading
num_worker_threads = 5
num_work_items = 20
def worker():
thread_id = threading.current_thread().name
while True:
item = q.get()
print("Worker %s:" % thread_id, item)
q.task_done()
if __name__ == '__main__':
q = queue.Queue()
for i in range(num_worker_threads):
t = threading.Thread(target=worker)
t.daemon = True
t.start()
work_items = range(num_work_items)
for item in work_items:
q.put(item)
q.join() # block until all tasks are done
import multiprocessing
import time
def xyz():
for i in range(10):
print("thread")
time.sleep(1)
if __name__ == '__main__':
proc = multiprocessing.Process(target=xyz)
proc.start()
for i in range(10):
print("main")
time.sleep(1)
import multiprocessing
import os
def worker(q):
process_id = os.getpid()
while True:
item = q.get()
print("Worker %s:" % process_id, item)
q.task_done()
if __name__ == "__main__":
num_worker_processes = 5
num_work_items = 20
q = multiprocessing.JoinableQueue(num_work_items+1)
for i in range(num_worker_processes):
p = multiprocessing.Process(target=worker, args=[q])
p.daemon = True
p.start()
work_items = range(num_work_items)
for item in work_items:
q.put(item)
q.join() # block until all tasks are done
print("All done")
lab_multi.py
YAML Ain't Markup Language
YAML is a human friendly data serialization
standard for all programming languages.
Easy for parse for humans and machines
Language independent
YAML, like Python, highly values whitespace
Spaces only, no tabs
It aims to capture sequences, scalars, and map data structures
Used a lot for config files
Native simple data types: integer, float, strings
Scalars, maps, sequences
# Sequence of Scalars
- Mark McGwire
- Sammy Sosa
- Ken Griffey
# Mapping Scalars to Sequences
american:
- Boston Red Sox
- Detroit Tigers
- New York Yankees
national:
- New York Mets
- Chicago Cubs
- Atlanta Braves
# Mapping Scalars to Scalars
hr: 65 # Home runs
avg: 0.278 # Batting average
rbi: 147 # Runs Batted In
# Sequence of Mappings
-
name: Mark McGwire
hr: 65
avg: 0.278
-
name: Sammy Sosa
hr: 63
avg: 0.288
pip install pyyaml
# create a python object from yaml txt
yml_txt = file('myfile.yml', 'r').read()
py_obj = yaml.load(yml_txt)
# serialize a python object to yaml txt
yml_txt = yaml.dump(py_obj)
file('myfile.yml', 'w').write(yml_txt)
---
name: Jeremy
age: unknown
height: 75
{'name': 'Jeremy', 'age': 'unknown', 'height': 75}
---
- item1
- item2
- 999
['item1', 'item2', 999]
---
- id: 111
name: bob
shell: "/bin/bash"
- id: 222
name: mary
shell: "/bin/zsh"
[{'id': 111, 'name': 'bob', 'shell': '/bin/bash'},
{'id': 222, 'name': 'mary', 'shell': '/bin/zsh'}]
---
- item1
- item2
- item3
...
---
another:
- 1
- 2
- 3
...
lab_yaml.py
JSON is trivial to generate and parse
at the cost of reduced human readability
JSON uses lowest common denominator information model
any JSON data can be easily processed by every modern programming environment
YAML designed for human readability and support for serializing arbitrary native data structures.
YAML allows for extremely readable files
more complex to generate and parse
Uses more than the lowest common denominator data types
requires more complex processing when crossing between different programming environments.
Javascript Object Notation
Part of Javascript language definition
Easy for parse for humans and machines
Language independent
“lightweight data interchange format”
Name/value pair collection (dict in Python)
Ordered list of values (list in Python)
An “object” is an unordered set of name:value pairs (called “members”) separated by commas and surrounded by curly braces
{ Name1:Value1, Name2:Value1,… }
An “array” is a set of “elements” surrounded by square brackets
[ element1, element2,… ]
Elements are strings, numbers, true, false, null, object, array
Strings are made of:
Unicode chars (\uhhhh)
\,\\,/,\b,\f,\n,\r,\t
Numbers are signed integer, decimal, or exponent (“e”) flavors only
Whitespace is fine
import json
json.dump(python_obj, fo, **kwargs)
json.dumps(python_obj, **kwargs)
Note: JSON is not a ”framed” protocol
i.e. can’t append multiple JSON objects to same file
import json
python_obj = json.load(fo, **kwargs)
python_obj = json.loads(string, **kwargs)
JSON keys in key/value pairs are always strings. Unlike Python.
Default json module encoder only encodes “ASCII-compatible” strings. Use u for other encodings
Same name in name:value pairs uses the last one
Out of range floats are handled correctly (nan, inf, -inf)
dict -> object
list, tuple -> array
str, unicode -> string
int, float -> number
True, False -> true, false
None -> null
lab_json.py
The pickle module does a similar job to JSON, but is Python specific
Not good for machine data interchange
Allows multiple pickled objects to be dumped to the same file
must be loaded in same order on way back
The shelve module essentially provided a persistent dict for pickled objects in a database
Opinion: Only use these for interchange within your own app
Representational State Transfer
A simple, client/server web services API currently in favor
Way of locating and manipulating “resources” on a network
usually represented by XML or JSON documents
Commonly uses HTTP protocol (GET, POST, PUT, DELETE)
Stateless (all state on client or server)
Simple, predictable resource pathing scheme based on URL
ReSTful services generally map a CRUD interface (Create, Read, Update, Delete) by URL mappings that embed data
e.g. GET v2/{tenant_id}/servers
HTTP POST -> Create a server
GET -> Read the list of servers/specific server
PUT -> Update a specific server
DELETE -> Delete a specific server
HTTP GET calls to a ReSTful service should not change state i.e. read only
Client functions to access URL’s
import urllib.request
urllib.request.urlopen(url[,data])
url is the URL of the target resource
data is to be sent to the server
Only HTTP url types use data currently
If data exists
it must be URL encoded
HTTP GET becomes a POST
A “file-like object” is returned from urlopen() which can be accessed with file semantics (read, readlines, etc.)
Raises: URLError (subclass of IOError) HTTPError (subclass of URLError)
Also takes urllib2.Request objects, useful for including HTTP headers in a dict (or use Request add_header method)
Can handle redirections, HTTP error responses, cookies, proxies, HTTP authentication
import urllib.request
response = urllib.request.urlopen("http://www.google.com")
page = response.read()
post_response = urllib.request.urlopen("http://test.com/post_info",
data={'test1':123})
data = post_response.read()
You got to work with urllib in lab_json.py
pip install requests
More info on requests:
import requests
response = requests.get("https://api.github.com")
print response.status_code
print response.headers
print response.text
print response.json()
import requests
data = {'user': 'bob', 'pass': 'password'}
response = requests.post("https://api.myhost.com/login", data=data)
Note: the default content type for a POST is application/x-www-form-urlencoded
import requests
data = {'user': 'bob', 'pass': 'password'}
response = requests.post("https://api.myhost.com/login", json=data)
Note: the content type for a POST with the JSON argument is application/json
import requests
r = requests.put('http://httpbin.org/put', data = {'key':'value'})
r = requests.delete('http://httpbin.org/delete')
r = requests.head('http://httpbin.org/get')
r = requests.options('http://httpbin.org/get')
headers = {'X-Auth-Token': '62c0e8d7305c4fd88d3772cde0d06d38'}
r = requests.get("https://myapi.com/resource", headers=headers)
lab_requests.py
pip install Flask
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello_world():
return 'Hello World!'
if __name__ == '__main__':
app.run()
# to run the app:
$ python my_app.py
* Running on http://127.0.0.1:5000/
@app.route('/user/<username>')
def show_user_profile(username):
# show the user profile for that user
return 'User %s' % username
@app.route('/post/<int:post_id>')
def show_post(post_id):
# show the post with the given id, the id is an integer
return 'Post %d' % post_id
@app.route('/login', methods=['GET', 'POST'])
def login():
if request.method == 'POST':
do_the_login()
else:
show_the_login_form()
from flask import render_template
@app.route('/hello/')
@app.route('/hello/<name>')
def hello(name=None):
return render_template('hello.html', name=name)
<!doctype html>
<title>Hello from Flask</title>
{% if name %}
<h1>Hello {{ name }}!</h1>
{% else %}
<h1>Hello World!</h1>
{% endif %}
app.debug = True
app.run()
# or
app.run(debug=True)
lab_flask.py
def func(arg1, arg2, arg3='abc', arg4='123'):
pass
def argless(*args):
print(args)
argless()
argless(1,2,3)
def twoargs(x, y, *args):
print(x, y, args)
twoargs(1, 2, 3)
twoargs('a', 'b')
def subtract(x, y):
return x - y
subtract(10, 5)
lst = [10, 5]
subtract(*lst)
def kwargless(**kwargs):
print(kwargs)
kwargless()
kwargless(x=1, y=2)
def subtract(x, y):
return x - y
values = {'x': 10, 'y': 5}
subtract(**values)
lab_args.py
class FakeClass:
pass
a = 1
b = 'hello'
c = FakeClass()
print(a, b, c)
class FakeClass:
pass
a = FakeClass()
b = a
print(b.hello)
a.hello = 'world'
print(b.hello)
x = 123
print(x.__add__)
print(x.__add__(321))
print(dir(x))
import datetime
import imp
print(datetime.datetime.now())
print(datetime.datetime.max, datetime.datetime.min)
class PartyTime():
def __call__(self, *args):
imp.reload(datetime)
value = datetime.datetime(*args)
datetime.datetime = self
return value
def __getattr__(self, value):
if value == 'now':
return lambda: print('Party Time!')
else:
imp.reload(datetime)
value = getattr(datetime.datetime, value)
datetime.datetime = self
return value
datetime.datetime = PartyTime()
print(datetime.datetime.now())
print(datetime.datetime.max, datetime.datetime.min)
x = 'abc'
print(id(x))
print(hex(id(x))
x += 'def'
print(id(x))
print(hex(id(x))
class myInt():
def __init__(self):
self.value = 0
def __str__(self):
return str(self.value)
def __repr__(self):
return str(self.value)
x = myInt()
print(x, id(x))
x_t = (x, x)
print(x_t, id(x_t), id(x_t[0]), id(x_t[1]))
x_t[0] = 999 # exception!
x.value = 999
print(x, id(x))
print(x_t, id(x_t), id(x_t[0]), id(x_t[1]))
def list_widget(in_list):
in_list[0] = 10
in_list = list(range(1, 10))
print(in_list)
in_list[0] = 10
print(in_list)
my_list = [9, 9, 9, 9]
list_widget(my_list)
print(my_list)
import dis
def myFunc():
x = 1
y = 2
z = 'abc'
return x + y
print(myFunc.__name__)
print(myFunc.__code__.co_varnames)
print(myFunc.__code__.co_consts)
print(myFunc.__code__.co_code)
dis.disassemble(myFunc.__code__)
# get all the attributes of an object
dir(obj)
# get a specific attribute of an object
# equivalent to: obj.y
x = getattr(obj, 'y')
# check if an object has an attribute named 'y'
hasattr(obj, 'y')
lab_objects.py
def func():
return "hi"
print(func())
info = "hello world"
def localonly():
print(locals())
print(globals())
localonly()
info = "hello world"
def localonly():
print(info)
localonly()
info = "hello world"
def localonly():
info = "inside localonly"
print(info)
localonly()
print(info)
def localonly():
info = "inside localonly"
print(info)
localonly()
print(info)
def hello(arg1):
print(locals())
hello(1)
def hello(arg1, arg2=0):
print(locals())
hello(1)
hello(1, 3)
hello()
hello(arg2=15, arg1=10)
hello(arg2=99)
def outer():
info = 1
def inner():
print(info)
inner()
outer()
def func():
pass
print(type(func))
print(func.__class__)
print(issubclass(func.__class__, object))
def add(x, y):
return x + y
def sub(x, y):
return x - y
def do_math(operation, x, y):
return operation(x, y)
print(do_math(sub, 10, 5))
print(do_math(add, 2, 2))
def outer():
def inner():
print "inside the inner function"
return inner
my_func = outer()
print(my_func)
print(my_func())
def outside():
info = 1
def inside():
print info
return inside
my_func = outside()
my_func.func_closure
my_func()
def outside(arg1):
def inside():
print arg1
return inside
my_func1 = outside(123)
my_func2 = outside('xyz')
my_func1()
my_func2()
def wrapper(passed_in_func):
def inside():
print "before passed_in_func"
retval = passed_in_func()
return retval + 1
return inside
def constant():
return 2
decorated = wrapper(constant)
decorated()
def add_two_numbers(x, y):
return x + y
def ensure_positivity(func):
def inner(x, y):
return func(abs(x), abs(y))
return inner
add_two_numbers(-10, 10)
add_two_numbers = ensure_positivity(add_two_numbers)
add_two_numbers(-10, 10)
def ensure_positivity(func):
def inner(x, y):
return func(abs(x), abs(y))
return inner
@ensure_positivity
def add_two_numbers(x, y):
return x + y
add_two_numbers(-10, 10)
def debug_call(func):
def inner(*args, **kwargs):
print "{} called with {} {}".format(func.func_name, args, kwargs)
return func(*args, **kwargs)
return inner
@debug_call
def add(x, y=0):
return x + y
print add(1, 2)
print add(y=3, x=5)
print add(5)
lab_decorators.py
with Parameters
def debug_call(func):
def inner(*args, **kwargs):
print "{} called with {} {}".format(func.func_name, args, kwargs)
return func(*args, **kwargs)
return inner
@debug_call
def add(x, y=0):
return x + y
print add(1, 2)
print add(y=3, x=5)
print add(5)
What if we wanted to add a parameter to debug_call?
@debug_call
def add(x, y=0):
return x + y
# translates into:
add = debug_call(add)
@decorator_with_args(arg)
def foo(*args, **kwargs):
pass
# translates into:
foo = decorator_with_args(arg)(foo)
def debug_call(enabled=True):
def real_debug_call(func):
def inner(*args, **kwargs):
if enabled:
print "{} called with {} {}".format(func.func_name,
args,
kwargs)
return func(*args, **kwargs)
return inner
return real_debug_call
@debug_call()
def add(x, y=0):
return x + y
@debug_call(enabled=False)
def sub(x, y):
return x - y
print add(1, 2)
print sub(3, 4)
lab_decorators_with_parameters.py
Best early detection of bugs:
Pair programming
Continuous Integration
Test Driven Development
Test Driven Development (TDD) says essentially:
Write a test that for a proposed feature and verify it fails
Write the minimal amount of code to make the test pass
Refactor
Repeat
Module in Python used to implement TDD
or other unit testing needs
Derive a class from unittest.TestCase
For a test runner, unittest.main() is the default, or roll your own
Simple test to see if two values were multiplied correctly:
import calculator.operations.arithmetic as arith
import unittest
class TestCalculations(unittest.TestCase):
def test_multiply(self):
""" test multiply 2 * 2 """
testVal = arith.mult(2,2)
self.assertEqual(testVal, 4)
if __name__ == "__main__":
unittest.main()
If a test fixture is needed to setup/takedown the calculator for each test, we can add methods setUp() and tearDown()
setUpClass() and tearDownClass() for TestCase-level setup and teardown
The naming standard is that each test method name start with test
For unit tests, the TestCase class offers (3) primary verifiers:
assertEqual(): result is equal to the value we expect
assertTrue(): result is a True assertion
assertRaises(): result is the Exception we expect
There are more assert methods to make life easy:
assertNotEqual(): a != b
assertTrue(): a is True
assertFalse(): a is False
assertIs(): a is b
assertIsNot(): a is not b
assertIsNone(): a is None
assertIsNotNone(): a is not None
assertIn(): a in b
assertNotIn(): a not in b
assertIsInstance(): isinstance(a, b) is True
assertIsNotInstance(): isinstance(a, b) is False
assertGreater(a, b): a > b
assertRaises(ex): raises ex
assertRegexMatches(x, re): regex.search(x) is True
lab_unittest.py
# in Python 3
from unittest.mock import MagicMock
thing = ProductionClass()
thing.method = MagicMock(return_value=3)
thing.method(3, 4, 5, key='value')
3
thing.method.assert_called_with(3, 4, 5, key='value')
from mock import Mock
thing = Mock(side_effect=KeyError('bam'))
thing()