From problem to application

how to automate tasks

Agenda

 

About Me

Problem

Minimum Viable Solution

Improvements

Interface

Extensibility

Cleaning up

Summary

About Me

Test Automation Engineer

  • Python Developer
  • Scrum Master
  • Project Manager

Problem
System
Automation

Problem

Music

Gathering

1763 items

Manual Flow

Hikaru Utada & Skrillex - Face My Fears [Official Video]

Hikaru Utada & Skrillex - Face My Fears [Official Video]

Genres: EDM; dubstep; electro; trap;

Solution!

LLameDL

Minimum Viable Solution

Working Script

import webbot

def wait_for_and_click(browser, text):
    while not browser.exists(text=text):
        pass
    browser.click(text=text)

w = webbot.Browser(showWindow=False)
w.go_to("https://portalpacjenta.luxmed.pl/PatientPortal/Account/LogOn")
w.type('name.surname', id="Login")
w.type('pass', id="TempPassword")
w.type('pass', id="Password")
w.click(tag='div', classname='submit')
wait_for_and_click(w, text="Zmień termin")
wait_for_and_click(w, text="Szukaj")
while not w.find_elements(xpath=".//div/text", tag='text'):
    pass
available = w.find_elements(xpath=".//div/text", tag='text').pop(0).text
print(available)

w.click(text="Wyloguj się")
w.go_to("https://portalpacjenta.luxmed.pl/PatientPortal/Account/LogOut")
w.close_current_tab()
import requests
from lxml import html
import xml.etree.cElementTree as ET
from collections import namedtuple

FeedTuple = namedtuple('FeedTuple', ['title', 'url', 'feed'])

feed_postfix_list = ['/rss.xml', '/atom.xml', '/feed/atom', '/feed.atom', '/feed', '/feed.xml',
                     '/feed.rss', '/atom/entries/', '/feed_rss.xml', '/rss', '/posts/rss.xml',
                     'feeds/all_rss.xml']

def get_feed(url):
    content = requests.get(url)
    print('-' * 64, content)
    if content.status_code != 200:
        print('RSS is not available for {url}'.format(url=url))
        return FeedTuple("", url, "")

    tree = html.fromstring(content.content)
    title = tree.xpath("//title").pop().text.strip()

    for feed_postfix in feed_postfix_list:
        rc = requests.get(url + feed_postfix).status_code
        # print(feed_postfix, rc)
        if rc == 200:
            return FeedTuple(title, url, url + feed_postfix)

    rss = tree.xpath("//*[@type='application/rss+xml']")
    for i in rss:
        rss_url = i.attrib.get('href', None)
        if '/comments/' not in rss_url and requests.get(rss_url).status_code == 200:
            return FeedTuple(title, url, rss_url)

    print('RSS is not available for {url}'.format(url=url))
    return FeedTuple(title, url, "")

def create_template():
    root = ET.Element("opml", version="1.0")
    doc = ET.SubElement(root, "body")
    return root, doc

def add_subelement(doc, feed):
    ET.SubElement(doc, "outline", type="rss", text=feed.title, title=feed.title, xmlUrl=feed.feed, htmlUrl=feed.url)
    return doc

def save(root):
    tree = ET.ElementTree(root)
    tree.write("opml.xml")

if __name__ == '__main__':
    RSS_URLS_list = [
        'http://switchweekly.com/',
    ]
    root, doc = create_template()
    for rss in RSS_URLS_list:
        try:
            y = get_feed(rss)
            add_subelement(doc, y)
        except Exception as e:
            print(e)
            print(rss)
    save(root)
"""
Get URLs from evernote note
xml is starting with <en-note id="en-note" class="editable" g_editable="true" contenteditable="true">
"""
import re
import xml.etree.ElementTree as ET

tree = ET.parse('evernote.xml')
root = tree.getroot()
for child in root:
    if child.findall('.//span'):
        text = "# " + child.findall('.//span')[0].text
        print(text)
        print("")
    elif child.findall('.//a'):
        text = child.findall('.//a')[0].text
        print(re.sub('(\\n| +)', ' ', text))
        print("<{}>".format(child.findall('.//a')[0].attrib['href']))
        print("")
    else:
        print(child.findall('.//'))

"""
Change filenames for Xiaomi 3S files to format 'yyyy-mm-dd hh-mm-ss'
"""
import re
import os


def get_new_filename(filename):
    result = re.search("\w*_(\d{4})(\d{2})(\d{2})_(\d{2})(\d{2})(\d{2})(?:_\w*?)\.(\w*)", filename)
    if result:
        year, month, day, hour, minute, second, f = result.groups()
        return "-".join([year, month, day]) + ' ' + ".".join([hour, minute, second, f])
    else:
        return ""


if __name__ == '__main__':
    directory = "/media/jarek/Work [HDD]/290118/New folder/DCIM/Camera"
    dir_files = os.listdir(directory)
    for file in dir_files:
        new_file = get_new_filename(file)
        if new_file:
            os.rename(os.path.join(directory, file), os.path.join(directory, new_file))

Readme

Improvements

Existing libraries

last.FM scraper        musicbrainzngs

Selenium        WebBOT

Unit Tests

"Test Pyramid"

Refactoring

https://refactoring.guru/

Static Code Analysis

Logical
Stylistic

Analytical

Logical
 

Static types, detect various errors, security issues

MyPy, Bandit, PyFlakes, Pyre

Stylistic

Style convention, docstrings

pycodestyle, pydocstyle

Analytical

McCabe complexity, lines of code...

Mccabe, Radon

Continuous Integration

Interface

CLI

Argparse

Python Fire

Click

Docopt

.

.

.

entry_points

Command-Line Options

"The Art of Unix Programming" Chapter 10. Configuration

GUI

Web frameworks

llamedl --bookmark_name music_to_download --directory_path /home/jarek/Music

Extensibility

Design Patterns

Creational

Structural

Behavioral 

https://refactoring.guru/

Python Patterns 

https://python-patterns.guide

Composition over inheritance

Inheritance is when you design your types around what they are, and composition is when you design types around what they do.

├── downloaders
│   ├── basedownloader.py
│   └── youtubedownloader.py
├── tagsproviders
│   ├── basetags.py
│   ├── filetags.py
│   ├── lastfmtags.py
│   ├── musicbrainzgstags.py
├── urlproviders
│   ├── basebrowser.py
│   ├── baseurl.py
│   ├── chromeurls.py
│   └── userurl.py
│  
│  
│  
│  
├── iyoutube.py
│   
├── tagger.py
│   
├── ichrome
│  
│  
│  
│  

Cleaning up

Documentation

Tutorials

How to guides

References

Explanations

Refactoring

Autoformating

autopep8
yapf

What's next?

  • Docker

  • Support for other browsers

  • Support for other sources

Summary

Problem
System
Automation

Problem

Define a problem

Create "Definition of Done"

System

Steps list

Flowchart

Decision tree

Pipeline

Automation

Automate elements of the system

Solution
Interface
Extensibility

Solution

creating a working code

Interface

increasing coverage of use

adaptation to other needs

Extensibility

easy testing

easy to expand

Thanks for your time

Questions?

Od problemu do aplikacji

By Jarek Pi

Od problemu do aplikacji

  • 330