From problem to application
how to automate tasks
Agenda
About Me
Problem
Minimum Viable Solution
Improvements
Interface
Extensibility
Cleaning up
Summary
About Me
Test Automation Engineer
- Python Developer
- Scrum Master
- Project Manager
Problem
System
Automation
Problem
Music
Gathering
1763 items
Manual Flow
Hikaru Utada & Skrillex - Face My Fears [Official Video]
Hikaru Utada & Skrillex - Face My Fears [Official Video]
Genres: EDM; dubstep; electro; trap;
Solution!
LLameDL
Minimum Viable Solution
Working Script
import webbot
def wait_for_and_click(browser, text):
while not browser.exists(text=text):
pass
browser.click(text=text)
w = webbot.Browser(showWindow=False)
w.go_to("https://portalpacjenta.luxmed.pl/PatientPortal/Account/LogOn")
w.type('name.surname', id="Login")
w.type('pass', id="TempPassword")
w.type('pass', id="Password")
w.click(tag='div', classname='submit')
wait_for_and_click(w, text="Zmień termin")
wait_for_and_click(w, text="Szukaj")
while not w.find_elements(xpath=".//div/text", tag='text'):
pass
available = w.find_elements(xpath=".//div/text", tag='text').pop(0).text
print(available)
w.click(text="Wyloguj się")
w.go_to("https://portalpacjenta.luxmed.pl/PatientPortal/Account/LogOut")
w.close_current_tab()
import requests
from lxml import html
import xml.etree.cElementTree as ET
from collections import namedtuple
FeedTuple = namedtuple('FeedTuple', ['title', 'url', 'feed'])
feed_postfix_list = ['/rss.xml', '/atom.xml', '/feed/atom', '/feed.atom', '/feed', '/feed.xml',
'/feed.rss', '/atom/entries/', '/feed_rss.xml', '/rss', '/posts/rss.xml',
'feeds/all_rss.xml']
def get_feed(url):
content = requests.get(url)
print('-' * 64, content)
if content.status_code != 200:
print('RSS is not available for {url}'.format(url=url))
return FeedTuple("", url, "")
tree = html.fromstring(content.content)
title = tree.xpath("//title").pop().text.strip()
for feed_postfix in feed_postfix_list:
rc = requests.get(url + feed_postfix).status_code
# print(feed_postfix, rc)
if rc == 200:
return FeedTuple(title, url, url + feed_postfix)
rss = tree.xpath("//*[@type='application/rss+xml']")
for i in rss:
rss_url = i.attrib.get('href', None)
if '/comments/' not in rss_url and requests.get(rss_url).status_code == 200:
return FeedTuple(title, url, rss_url)
print('RSS is not available for {url}'.format(url=url))
return FeedTuple(title, url, "")
def create_template():
root = ET.Element("opml", version="1.0")
doc = ET.SubElement(root, "body")
return root, doc
def add_subelement(doc, feed):
ET.SubElement(doc, "outline", type="rss", text=feed.title, title=feed.title, xmlUrl=feed.feed, htmlUrl=feed.url)
return doc
def save(root):
tree = ET.ElementTree(root)
tree.write("opml.xml")
if __name__ == '__main__':
RSS_URLS_list = [
'http://switchweekly.com/',
]
root, doc = create_template()
for rss in RSS_URLS_list:
try:
y = get_feed(rss)
add_subelement(doc, y)
except Exception as e:
print(e)
print(rss)
save(root)
"""
Get URLs from evernote note
xml is starting with <en-note id="en-note" class="editable" g_editable="true" contenteditable="true">
"""
import re
import xml.etree.ElementTree as ET
tree = ET.parse('evernote.xml')
root = tree.getroot()
for child in root:
if child.findall('.//span'):
text = "# " + child.findall('.//span')[0].text
print(text)
print("")
elif child.findall('.//a'):
text = child.findall('.//a')[0].text
print(re.sub('(\\n| +)', ' ', text))
print("<{}>".format(child.findall('.//a')[0].attrib['href']))
print("")
else:
print(child.findall('.//'))
"""
Change filenames for Xiaomi 3S files to format 'yyyy-mm-dd hh-mm-ss'
"""
import re
import os
def get_new_filename(filename):
result = re.search("\w*_(\d{4})(\d{2})(\d{2})_(\d{2})(\d{2})(\d{2})(?:_\w*?)\.(\w*)", filename)
if result:
year, month, day, hour, minute, second, f = result.groups()
return "-".join([year, month, day]) + ' ' + ".".join([hour, minute, second, f])
else:
return ""
if __name__ == '__main__':
directory = "/media/jarek/Work [HDD]/290118/New folder/DCIM/Camera"
dir_files = os.listdir(directory)
for file in dir_files:
new_file = get_new_filename(file)
if new_file:
os.rename(os.path.join(directory, file), os.path.join(directory, new_file))
Readme
Improvements
Existing libraries
last.FM scraper musicbrainzngs
Selenium WebBOT
Unit Tests
"Test Pyramid"
Refactoring
https://refactoring.guru/
Static Code Analysis
Logical
Stylistic
Analytical
Logical
Static types, detect various errors, security issues
MyPy, Bandit, PyFlakes, Pyre
Stylistic
Style convention, docstrings
pycodestyle, pydocstyle
Analytical
McCabe complexity, lines of code...
Mccabe, Radon
Continuous Integration
Interface
CLI
Argparse
Python Fire
Click
Docopt
.
.
.
entry_points
Command-Line Options
"The Art of Unix Programming" Chapter 10. Configuration
GUI
Web frameworks
llamedl --bookmark_name music_to_download --directory_path /home/jarek/Music
Extensibility
Design Patterns
Creational
Structural
Behavioral
https://refactoring.guru/
Python Patterns
https://python-patterns.guide
Composition over inheritance
Inheritance is when you design your types around what they are, and composition is when you design types around what they do.
├── downloaders
│ ├── basedownloader.py
│ └── youtubedownloader.py
├── tagsproviders
│ ├── basetags.py
│ ├── filetags.py
│ ├── lastfmtags.py
│ ├── musicbrainzgstags.py
├── urlproviders
│ ├── basebrowser.py
│ ├── baseurl.py
│ ├── chromeurls.py
│ └── userurl.py
│
│
│
│
├── iyoutube.py
│
├── tagger.py
│
├── ichrome
│
│
│
│
Cleaning up
Documentation
Tutorials
How to guides
References
Explanations
Refactoring
Autoformating
autopep8
yapf
What's next?
-
Docker
-
Support for other browsers
-
Support for other sources
Summary
Problem
System
Automation
Problem
Define a problem
Create "Definition of Done"
System
Steps list
Flowchart
Decision tree
Pipeline
Automation
Automate elements of the system
Solution
Interface
Extensibility
Solution
creating a working code
Interface
increasing coverage of use
adaptation to other needs
Extensibility
easy testing
easy to expand
Thanks for your time
Questions?
Od problemu do aplikacji
By Jarek Pi
Od problemu do aplikacji
- 325