Hogyan írjunk komplex Slack botot
Pythonban?

@kissgyorgy

Gerrit

Google által fejlesztett
(Guido van Rossum)
Code Review tool

A probléma

A terv

GET

POST

A megoldás

Incoming webhooks!

https://api.slack.com/incoming-webhooks

curl -X POST --data-urlencode \
'payload={"text":"This is a line of text.\nAnd this is another one."}' \
https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

#!/usr/bin/env python3
import json
import requests

GERRIT_URL = 'https://review.balabit'
CHANGES_ENDPOINT = GERRIT_URL + '/changes/?q='
ACCOUNTS_ENDPOINT = GERRIT_URL + '/accounts'
QUERY = '(ownerin:scb-beast+OR+ownerin:scb-beauty)+AND+status:open+AND+NOT+label:Code-Review=2'
GERRIT_CHANGES_URL = CHANGES_ENDPOINT + QUERY
SLACK_WEBHOOK_URL = 'https://hooks.slack.com/services/...'


def get(url):
    ...
    return json.loads(fixed_body)

def process_changes(all_changes):
    ...

def post_to_slack(processed_changes):
    lines = []
    for change in processed_changes:
        ...
        lines.append(change_url)
    requests.post(SLACK_WEBHOOK_URL, json={'text': '\n'.join(lines)})

def main():
    all_changes = get(GERRIT_CHANGES_URL)
    processed_changes = process_changes(all_changes)
    post_to_slack(processed_changes)


if __name__ == '__main__':
    main()

[Unit]
Description=Download gerrit changes and post status to Slack channel

[Service]
Type=oneshot

Environment='QUERY=ownerin:scb-best+AND+status:open+AND+NOT+label:Code-Review=2'
Environment=SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
Environment=CHANNEL=#scb-best

ExecStart=/home/alarm/slackbot/gerrit_slack_bot.py

[Unit]
Description=Download gerrit changes and post status to Slack channel

[Timer]
OnCalendar=Mon-Fri *-*-* 9,10,11,12,13,14,15,16,17,18:00

[Install]
WantedBy=timers.target

gerrit-slack-bot.service:

gerrit-slack-bot.timer:

Ütemezés systemd-vel

Kész.

Vagy mégsem?

A probléma (2.)

Követelmények

törölje le az előző üzenetet
bárki tudjon új szabályt felvenni
könnyű legyen új szabályt felvenni
bármennyi szabályt fel lehessen venni
kevés dependencia
ne kelljen hozzá systemd
dockerben is lehessen futtatni
relatíve kevés erőforrást használjon (pl. Raspberry-n is lehessen futtatni)

Slack

1. App létrehozása

https://api.slack.com/apps

2. Beállítások

OAuth Scope-ok

(Jogosultságok)

https://api.slack.com/docs/oauth-scopes

https://api.slack.com/scopes

object:action:perspective

channels:read - channelek listázása
chat:write:user - privát üzenet küldés
users.read - user információk lekérése
incoming-webhook - egyszerű üzenet küldés
commands - "/"-el kezdődő bot parancsok
bot - bot user, bot token

Bot scope

https://api.slack.com/bot-users

tud üzenetet küldeni
(privát, publikus is)
törölni
channeleket listázni
channel info-kat lekérni
fileokat kezelni
reagálni (emoji)
csillagozni
topicot váltani
...

3. OAuth flow

Token típusok

https://api.slack.com/docs/token-types

xoxp-

xoxb-

xoxa-

Slack API

RPC jellegű
>100 metódus
Nem minden endpoint támogat JSON body-t
header:
groups.* metódusok nem működnek

Content-Type: application/json; charset=utf-8

https://api.slack.com/web

Néhány endpoint
(metódus)

chat.Postmessage - üzenet küldése channelre
chat.delete - általunk küldött üzenet törlése
users.info - információk egy userről
conversations.list - channelek listázása
...

https://api.slack.com/methods

Conversations API

https://api.slack.com/docs/conversations-api

API family
bot scope-al lehet használni
"unified interface" for :
public channels, private channels, direct messages, group direct messages and shared channels
groups.* helyett

If it works like a channel, has messages like a channel, and is experienced like a channel... it's a ~~channel~~ conversation!!!

Conversations API
metódusok

https://api.slack.com/docs/conversations-api#methods

conversations.list - channelek listázása
conversations.history
conversations.setPurpose - topic beállítás
...

Slack üzenet formázások

https://api.slack.com/docs/messages

Üzenetek

{
    "text": "This is a line of text.\nAnd this is another one."
}

UTF-8
Markdown
Multi-line

https://api.slack.com/docs/messages

*bold*

_italic_

`code block`

```bigger 
code 
block
```

Escape

def escape(text):
    """Escape Slack special characters."""
    rv = text.replace('<', '&lt;')
    rv = rv.replace('>', '&gt;')
    rv = rv.replace('&', '&amp;')
    return rv

https://api.slack.com/docs/message-formatting#how_to_escape_characters

Attachments

{
    "attachments": [
        {
            "fallback": "Required plain-text summary of the attachment.",
            "color": "#36a64f",
            "pretext": "Optional text that appears above the attachment block",
            "author_name": "Bobby Tables",
            "author_link": "http://flickr.com/bobby/",
            "author_icon": "http://flickr.com/icons/bobby.jpg",
            "title": "Slack API Documentation",
            "title_link": "https://api.slack.com/",
            "text": "Optional text that appears within the attachment",
            "fields": [
                {
                    "title": "Priority",
                    "value": "High",
                    "short": false
                }
            ],
            "image_url": "http://my-website.com/path/to/image.jpg",
            "thumb_url": "http://example.com/path/to/thumb.png",
            "footer": "Slack API",
            "footer_icon": "https://platform.slack-edge.com/img/default_application_icon.png",
            "ts": 123456789
        }
    ]
}

URL

<{url}|{text}>

Sorhossz

def full_message(self):
    text = (f'CR: {self.code_review_icon} '
            f'V: {self.verified_icon} '
            f'{self.username}: {self.subject}')
    icon_lenghts = len(self.code_review_icon) + len(self.verified_icon)
    # Slack wraps lines around this width, so if we cut out here 
    # explicitly, every patch will fit in one line.
    return textwrap.shorten(text, width=76+(icon_lenghts-2), placeholder='…')

Message builder

https://api.slack.com/docs/messages/builder

Végeredmény

Használt technológiák

SQLite
Python
Django (Flask)
uWSGI

SQLite

néhány user
(nem kell skálázni)
read-heavy
nem kell hozzá extra service
könnyű mozgatni
egyszerű backup

Python

Pipfile

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[requires]

python_version = "3.6"


[packages]

django = "*"
croniter = "*"
requests = "*"
django-constance = {extras = ["database"]}
django-crispy-forms = "*"
envparse = "*"
django-widget-tweaks = "*"


[dev-packages]

ipython = "*"

Django

https://www.djangoproject.com/

Croniter

from croniter import croniter
from datetime import datetime

base = datetime(2010, 1, 25, 4, 46)
iter = croniter('*/5 * * * *', base)  # every 5 minutes
print(iter.get_next(datetime))   # 2010-01-25 04:50:00
print(iter.get_next(datetime))   # 2010-01-25 04:55:00
print(iter.get_next(datetime))   # 2010-01-25 05:00:00

# 04:02 on every Wednesday OR on 1st day of month
iter = croniter('2 4 1 * wed', base)
print(iter.get_next(datetime))   # 2010-01-27 04:02:00
print(iter.get_next(datetime))   # 2010-02-01 04:02:00
print(iter.get_next(datetime))   # 2010-02-03 04:02:00

croniter.is_valid('0 0 1 * *')  # True
croniter.is_valid('0 wrong_value 1 * *')  # False

https://github.com/taichino/croniter

uWSGI

mule
static file serving
caching (memory mapped file)

https://uwsgi-docs.readthedocs.io/

uWSGI config

[uwsgi]
plugins = python3
master = true
http-socket = :8000
pythonpath = web
pythonpath = lib
module = web.wsgi
static-map = /static=static_root

# This will run all scheduled jobs in a programmed mule:
# http://uwsgi-docs.readthedocs.io/en/latest/Mules.html#giving-a-brain-to-mules
mule = bot.py
threads = 3

cache2 = name=channels,items=5000,store=channel_cache.mm,blocksize=1000,key_size=12

Demo

https://github.com/kissgyorgy/gerrit-slack-bot

https://slides.com/kissgyorgy/slackbot

https://twitter.com/kissgyorgy

Repo:

Előadás:

Elérhetőség:

Change letöltés

import json
import requests

def get(api_url):
    res = requests.get(api_url, verify=False)
    # There is a )]}' sequence at the start of each response.
    # we can't process it simply as JSON because of that.
    fixed_body = res.text[4:]
    return json.loads(fixed_body)


class Client:
    def __init__(self, gerrit_url, query):
        self._gerrit_url = gerrit_url
        # For +1 and -1 information, LABELS option has to be requested. See:
        # https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#detailed-labels
        # for owner name, DETAILED_ACCOUNTS:
        # https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#detailed-accounts
        self._changes_api_url = f'{gerrit_url}/changes/?o=LABELS&o=DETAILED_ACCOUNTS&q={query}'

        self.query = query
        self.changes_url = make_changes_url(gerrit_url, query)

    def get_changes(self):
        gerrit_change_list = get(self._changes_api_url)
        return [Change(self._gerrit_url, c) for c in gerrit_change_list]

class Change:
    def __init__(self, gerrit_url, json_change):
        self._gerrit_url = gerrit_url
        self._change = json_change

    @property
    def url(self):
        change_number = self._change['_number']
        return f'{self._gerrit_url}/#/c/{change_number}'

    @property
    def username(self):
        return self._change['owner']['username']

    @property
    def subject(self):
        return self._change['subject']

    @property
    def code_review(self):
        cr = self._change['labels']['Code-Review']
        if 'approved' in cr:
            return CodeReview.PLUS_TWO
        elif 'value' not in cr:
            return CodeReview.MISSING
        elif cr['value'] == 1:
            return CodeReview.PLUS_ONE
        elif cr['value'] == -1:
            return CodeReview.MINUS_ONE
        elif cr['value'] == -2:
            return CodeReview.MINUS_TWO

    @property
    def verified(self):
        ver = self._change['labels']['Verified']
        if not ver:
            return Verified.MISSING
        elif 'approved' in ver:
            return Verified.VERIFIED
        else:
            return Verified.FAILED

Change feldolgozás

OAuth flow

from urllib.parse import urlencode

params = {
    'scope': 'commands,bot',
    'client_id': 'slack beállítás oldalon',
    'state': 'random state,ellenőrzésre szolgál',
    'redirect_uri': 'http://redirect_uri',
}
encoded_params = urlencode(params, safe=',')
button_url = 'https://slack.com/oauth/authorize?' + encoded_params

1.

def request_oauth_token(code):
    # documentation: https://api.slack.com/methods/oauth.access
    res = requests.post('https://slack.com/api/oauth.access', {
        'client_id': self._client_id,
        'client_secret': self._client_secret,
        'redirect_uri': self._redirect_uri,
        'code': code,
    })
    # example in slack_messages/oauth.access.json
    return res.json()

3.

5. API requests

requests.get('https://slack.com/api/' + method, payload, 
             headers={'Authorization': 'Bearer ' + token})

headers = {
    'Authorization': 'Bearer ' + token,
    'Content-Type': 'application/json; charset=utf-8',
}
res = requests.post('https://slack.com/api/' + method, headers=headers, 
                    json=payload)

Hogyan írjunk komplex Slack botot Pythonban?

Gerrit

A probléma

A terv

A megoldás

Ütemezés systemd-vel

Kész.

Vagy mégsem?

A probléma (2.)

Követelmények

Slack

1. App létrehozása

2. Beállítások

OAuth Scope-ok

(Jogosultságok)

Bot scope

3. OAuth flow

Token típusok

Slack API

Néhány endpoint (metódus)

Conversations API

Conversations API metódusok

Slack üzenet formázások

Üzenetek

Escape

Attachments

URL

Sorhossz

Message builder

Végeredmény

Használt technológiák

SQLite

Python

Pipfile

Django

Croniter

uWSGI

uWSGI config

Demo

Change letöltés

Change feldolgozás

OAuth flow

1.

3.

5. API requests

Slack bot

More from Kiss György

Hogyan írjunk komplex Slack botot
Pythonban?

Néhány endpoint
(metódus)

Conversations API
metódusok