• Language: Python 3.7.4
• Packages:
  • requests
    • https://pypi.org/project/requests/
  • beautifulsoup4 (建議版本 < 4.9.0)
    • https://pypi.org/project/beautifulsoup4/
  • lxml
    • https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser
  • re
  • scrapy (今天不會特別介紹)
    • https://pypi.org/project/Scrapy/

requests

• pip install requests
• 用來發 HTTP request
  • e.g., GET, POST, PUT, DELETE, ...
• synchronous / 同步

beautifulsoup4

• pip install beautifulsoup4==4.8.0
• parse HTML

lxml

• apt-get install python-lxml

• easy_install lxml

• pip install lxml

re

就 regular expression

https://docs.python.org/3/library/re.html

scrapy

• web crawling framework
• asynchronous / 非同步

requests 常用 method

• requests.get(url, ...)
• requests.post(url, data={'key': 'value', ...})

beautifulsoup4 常用 method

• 搜尋
  • find_all(name, attr, ...)
    • https://www.crummy.com/software/BeautifulSoup/bs4/doc/#find-all
  • find(name, attr, ...)
    • https://www.crummy.com/software/BeautifulSoup/bs4/doc/#find
  • select() - https://www.crummy.com/software/BeautifulSoup/bs4/doc/#css-selectors
  • select_one()

beautifulsoup4 常用 method

• 其他
  • prettify()
    • https://www.crummy.com/software/BeautifulSoup/bs4/doc/#pretty-printing
  • decompose()
    • https://www.crummy.com/software/BeautifulSoup/bs4/doc/#decompose