Creating a Bittorrent Client with Asyncio

Ahmed Abdalla

PyGotham 2017 @ NYC

SimplyAhmazing

SimplyAhmaz1ng

What is Bittorent?

• Open, peer to peer protocol for sharing files
• Robust, distributed protocol

Why build a Bittorrent Client?

Did a retreat @ the Recurse Center

www.recurse.com

Who uses Bittorrent?

• Gaming companies
  • Blizzard, Eve, etc
• Education
• Government
• Entertainment
  • including pirating
• Tech companies
  • Facebook, Twitter

Agenda

1. Explain Bittorrent protocol
2. Define steps for writing our client
3. Why Asyncio
4. Look at code

Let's download a torrent

Our Bittorrent Client

Write a program that can download a file when given file.torrent as input

$ python client.py file.txt.torrent

   file.txt has finished downloading

The plan

1. Read file.txt.torrent & get Tracker info
2. Request list of peers from tracker
3. Connect to each peer and download file
4. For each file piece download, write to disk

Why Asyncio?

Solving for:

• Maintain multiple concurrent connections to peers
• Network I/O bound application - waiting for data

Asyncio

• Single process, single threaded approach for concurrent applications
• Application code yields control at optimal times

Asyncio Building Blocks

• Coroutine
  • a function that can be paused and resumed
  • Can "yield" data
  • Can be "sent" data
• Task - scheduler for coroutines
  • Tells coroutines to make an execution 'step'
• Event loop - A queue of "tasks" to be executed

Sample Event Loop

queue = [
    Task(some_coroutine),
    Task(another_coroutine)
]


while queue:
    task = queue.pop(0)

    task.execute_coroutine()

    if task.is_coroutine_done():
        print('Finished a task')
    else:
        queue.append(task)
        


print('Event loop finished tasks')

Asyncio: Quick API Tour

Asyncio: Quick API Tour Cont'd

Asyncio: Quick API Tour Cont'd

import asyncio


async def sleep_a_lot()
    await asyncio.gather(*[some_coroutine() for i range(5)])
    print('Done')


loop = asyncio.get_event_loop()

loop.run_until_complete(sleep_a_lot)

Run multiple coroutine "simultaneously"

 Task 1

Task 2

active

inactive

Cooperative Multitasking

• Everything runs in one process, under a single thread
• Tasks "yield" control at optimal times

execution time

import asyncio, sys

async def download(torrent_file):
    # Read and parse ".torrent" file
    torrent = read_torrent(torrent_file)

    # Get peers list from tracker in ".torrent" file
    peer_addresses = await get_peers(torrent)

    # Object to track peer communication/state
    peers = [Peer(addr) for addr in peer_addresses]

    # Wait for all download coroutines to finish
    await asyncio.gather(
        *[peer.download() for peer in peers]  # Producer
    )


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(download(sys.argv[1])
    loop.close()

import asyncio, sys


async def download(torrent_file):
    # Read and parse ".torrent" file
    torrent = read_torrent(torrent_file)

    # Object to track peer communication/state
    peer_addresses = await get_peers(torrent)

    # Queue for storing downloaded file pieces
    file_pieces_queue = asyncio.Queue()

    # Object to coordinate writing file to disk
    file_saver = FileSaver(file_pieces_queue)

    # Object to track peer communication/state
    peers = [Peer(addr, file_pieces_queue) for addr in peer_addresses]

    await asyncio.gather(
        *([peer.download() for peer in peers] +  # Producers
          [file_saver.start()])                  # Consumers
    )

Peer 1

Peer 2

Peer 3

File Saver

Active

Inactive

Our Bittorrent client...

execution time

The plan

1. Read file.txt.torrent & get Tracker info
2. Request list of peers from tracker
3. Connect to each peer and download file
4. For each file piece download, write to disk

Let's open the torrent file

BEncoding

• Organizes data in a terse format
• Saves bandwidth

BDecoding the metafile file

File Pieces

The plan

1. Read file.txt.torrent & get tracker info
2. Request list of peers from tracker
3. Connect to each peer and download file
4. For each file piece download, write to disk

The Tracker

The Tracker is an HTTP(S) server that responds to HTTP GET requests.

It keeps track of who in the swarm has a torrent.

HTTP GET: Who has this file?

RESP: List of N Peers

Tracker

Tracker: HTTP GET

URL: http://my-tracker.com/announce?params=params

Aiohttp GET Request

import aiohttp

torrent = get_torrent_file()

async def request_peers(self):
    async with aiohttp.ClientSession() as session:
        resp = await session.get(tracker_url, params=params)
        resp_data = await resp.read()
        peers = bencoder.decode(resp_data)
        return peers

• Use async packages
• If you can't, run synchronous code in a thread

Tracker Response

b'd8:completei1e10:downloadedi0e10:incomplete
i1e8:intervali1921e12:min intervali960e5:peers
12:\xc0\xa8c\x01\xe90\xc0\xa8cd\xc8\xd5e'

{
  b'complete': 1,
  b'downloaded': 0,
  b'incomplete': 1,
  b'interval': 1918,
  b'min interval': 959,
  b'peers': b'\xc0\xa8c\x01\xe90\xc0\xa8cd\xc8\xd5\xc0\xa8c\x01\xe90\xc0\xa8cd\xc8\xd5'
}

['192.168.99.100:51413', '33.67.123.201:69596']

The plan

1. Read file.txt.torrent & get Tracker info
2. Request list of peers from tracker
3. Connect to each peer and download file
4. For each file piece download, write to disk

Working with Peers

• Manage Peer State
  • Choked
  • Interested
• Handle Peer messages
  • Handshake
  • Keep Alive
  • Choke
  • Unchoke
  • Interested
  • Not Interested
  • Have
  • Bitfield
  • Request
  • Piece

• Choked - Not allowed to send requests.
• Interested - Wants to send requests (download)

Peer State

class Peer(object):
    def __init__(self, host, port, file_queue):
        self.host = host
        self.port = port
        self.file_queue = file_queue

        # Denotes if peer is choking us
        self.peer_choking = True

        # Denotes if we've informed our peer we're interested
        self.am_interested = False


    async def download(self):
        pass

Peer Message Exchange

Client

Peer

Peer Message: Handshake

class Peer(object):
    async def download(self):
        reader, writer = await asyncio.open_connection(
            self.host, self.port
        )

        handshake = b''.join([
            chr(19).encode(),
            b'BitTorrent protocol',
            (chr(0) * 8).encode(),
            info_hash,
            PEER_ID.encode()
        ])

        # Send Handshake
        writer.write(handshake)
        await writer.drain()

        # Read and validate response
        peer_handshake = await reader.read(68)
        self.validate(peer_handshake)

        # Start exchanging messages...

Peer Message Formats

All remaining messages:

<message length> <id> <payload>

Peer Message Formats

Peer Message Exchange

class Peer(object):
    async def download(self):
        
        # Start exchanging messages

        buf = b''  # Holds data read from peer

        while True:
            resp = await reader.read(REQUEST_SIZE)
            buf += resp

            while True:
                if len(buf) < 4:
                    break

                msg_message_length = self.get_message_message_length(buf)

                if msg_message_length == 0:
                    # Handle Keep Alive
                    continue

                msg_id = struct.unpack('>b', buf[4:5])

Peer Message Exchange

class Peer(object):
    async def download(self):
                # ... Looping through Peer messages

                if msg_message_length == 0:
                    # Handle Keep Alive
                    continue

                msg_id = struct.unpack('>b', buf[4:5])  # 5th byte is the ID

                if msg_id == 0:
                    # Handle Choke...

                elif msg_id == 1:
                    # Handle Unchoke...
                    await self.send_interested_message()

                elif msg_id == 2:
                    # Handle Interested...

                elif msg_id == 3:
                    # Handle Not interested...

                elif msg_id == 4:
                    # Handle Have...

                elif msg_id == 5:
                    # Handle Bitfield...

                elif msg_id == 7:
                    # Handle Piece...
                    self.file_queue.enqueue(piece_data)

                await self.request_a_piece()

The plan

1. Read file.txt.torrent & get Tracker info
2. Request list of peers from tracker
3. Connect to each peer and download file
4. For each file piece download, write to disk

File Saver

class FileSaver(object):
    def __init__(self, file_queue):
        self.file_queue = file_queue

    async def start(self):
        while True:
            piece = await self.file_queue.get()

            if not piece: # Poison pill
                return

            await self.save(piece)

import asyncio, sys


async def download(torrent_file):
    # Read and parse ".torrent" file
    torrent = read_torrent(torrent_file)

    # Object to track peer communication/state
    peer_addresses = await get_peers(torrent)

    # Queue for storing downloaded file pieces
    file_pieces_queue = asyncio.Queue()

    # Object to coordinate writing file to disk
    file_saver = FileSaver(file_pieces_queue)

    # Object to track peer communication/state
    peers = [Peer(addr, file_pieces_queue) for addr in peer_addresses]

    await asyncio.gather(
        *([peer.download() for peer in peers] +  # Producers
          [file_saver.start()])                  # Consumers
    )

Check it out on Github

Some Bittorrent Projects

• Support file uploading
• NAT traversal
• Support UDP tracker
• Support sharing of multiple files
• Discover Peers without a tracker
  • Peer Exchange Protocol (PEX)
  • Distributed Hash Table (DHT)
• Download Magnet Links
• Improve download strategy
• Play torrent videos as they're downloading

Learn more about asyncio

500 Lines or Less

A Web Crawler With asyncio Coroutines

by: Jesse Jiryu Davis & Guido van Rossum

Home Assistant

github.com/home-assistant/home-assistant

await TalkFinished('bye')

SimplyAhmazing

SimplyAhmaz1ng