Web scraping
with Python

J rogel-salazar

  @quantum_tunel / @DT_science

2020

General Assembly London

Homework

This workshop

these slides

http://bit.ly/pythonscraping

Data on the web

Web Scraping

1. Read page HTML

# Python 3
from urllib.request import urlopen
url = 'http://isitweekendyet.com/'
pageSource = urlopen(url).read()

# Python 2
from urllib import urlopen
url = 'http://isitweekendyet.com/'
pageSource = urlopen(url).read()

3. Extract what we’re looking for

Mini Project

Mini Project

All together Now

'''A simple script that ... '''

# import modules
from urllib.request import urlopen
from bs4 import BeautifulSoup

# open webpage
url = 
pageSource = 

# parse HTML into Beautiful Soup
BitSoup = 

# extract data from soup

# clean up data

# process data

# action based on data

Questions?

Useful Resources

Useful Modules etc

2 vs 3

Python Usage Survey 2014 visualised

http://www.randalolson.com/2015/01/30/python-usage-survey-2014/

Python 2 & 3 Key Differences

 Thank you 

A bitcoin converter

'''
Bitcoin converter that tells us how much our bitcoins are worth in GBP
NB: expects python3
'''

# import modules
from urllib.request import urlopen
from bs4 import BeautifulSoup

def clean_up_rate(rateString):
    ''''Clean up raw rateString to form rate'''
    rateNumber = rateString[1:10]
    return float(rateNumber)

def main():
    '''Our main function that gets called when we run the program'''
    # open webpage
    url = "http://bitcoinexchangerate.org/c/GBP/1"
    webpage = urlopen(url).read()

    # turn html into beautiful soup
    bitcoinSoup = BeautifulSoup(webpage)

    # extract info from soup
    rateString = bitcoinSoup.find('b').string.strip()

    # clean up data
    rate = clean_up_rate(rateString)

    # get user input
    bitcoinsString = input("How many bitcoins have you got?\n>>> ")
    bitcoins = float(bitcoinsString)

    # print output
    GBP = rate * bitcoins
    print("You have", round(GBP,2), "GBP!")

# this kicks off our program & lets us both run and import the program
if __name__ == '__main__':
    main() 

Navigating to the weekend yet answer

>>> from urllib.request import urlopen
>>> from bs4 import BeautifulSoup
>>> url = "http://isitweekendyet.com/"
>>> source = urlopen(url).read()
>>> soup = BeautifulSoup(source, 'lxml')

>>> soup.body.div.string
'\nNO\n'

# an alternative:
>>> list(soup.body.stripped_strings)[0]
'NO' 

Searching the GA Gallery

GAurl = "https://gallery.generalassemb.ly/"
GAsource = urlopen(GAurl).read()
GAsoup = BeautifulSoup(GAsource, 'lxml')

def countLondon():
    londonProjects = GAsoup.find_all('a', href='/?metro=london')
    londonCount = len(londonProjects)
    return londonCount 

def getUniqueLocations():
    metros = GAsoup.find_all('a', class_='metro')
    uniqueLocations = []
    for metro in metros:
        location = metro.string
        if location not in uniqueLocations:
            uniqueLocations.append(location)
    return uniqueLocations 

Bonus: count projects per location

def getLocationCounts():
    metros = GAsoup.find_all('a', class_='metro')
    locationCounts = {}
    for metro in metros:
        location = metro.string
        if location not in locationCounts:
            locationCounts[location] = 1
        else:
            locationCounts[location] = locationCounts[location] + 1
    return locationCounts   

# pro tip: pretty print the result
from pprint import pprint
locationCounts = getLocationCounts()
pprint(locationCounts)