Borg Backup

(a fork of Attic)

"I found the Holy Grail of backups."

(Stavros K. about Attic-Backup, 8/2013)

Thomas Waldmann (@home, 202-12)

ThomasWaldmann.__doc__

• Doing Python since 2001,
  Linux since it was on Floppies,
  loves FOSS.
• Projects:
  • MoinMoin Wiki
  • nsupdate.info
  • bepasty
  • vpngw
  • BorgBackup
• Contact:  tw @ waldmann-edv . de
• Python Developer  (freelance & remote)

It's a backup tool

-

one you maybe actually would enjoy using.

• simple
• efficient
• secure
• safe
• free & open

simple

• each backup is a full backup
• FUSE = mount your backups
• easy pruning of old backups
• tooling:  just borg, ssh, sh
• good help, manpages, docs
• single-file binary
• good fs / OS / arch support

efficient

• very fast for unchanged files
• chunk deduplication
• flexible compression
• sparse file support
• not flooding the fs cache
• sped up by a bit of C and Cython
• HW accelerated crypto

safe

• borg uses:
  • checksums
  • transactions
  • fs: syncing, atomic ops
• backup repo = log-like KV store
• checkpoints while backing up
• off-site remote repositories

secure

• authenticated encryption
  • nothing to see in the repo
  • detect tampering / corruption

• ssh transport for remote repos

• append-only mode repos

• size obfuscation (1.2)

• FOSS, you can see the code

Crypto

• client-side, metadata and data
• authenticated encryption (AE, EtM)
  • aes256-ctr
  • hmac-sha256 or blake2b
  • counter management, never repeat
• encrypted key material:
  • on client or in backup repo
  • passphrase protection: pbkdf2 + AES
• OpenSSL 1.1, only libcrypto

Compression

• chunk-based (not full file)
• algorithms: lz4, zstd, zlib, lzma, none
• with lz4 often faster than with none!
• "auto" mode:
  • first use lz4 predictor
  • if compressible: expensive compression
• "borg recreate" can recompress

Deduplication in SpaceTime

• Deduplication "dimensions":
  • inner deduplication of data set
    • copies of files, similar files
    • lots of zeros (sparse or not)
  • historical deduplication in backup repo
    • many files don't change over time
    • they are all in each of your full backups
  • deduplication between machines
    • just moved that big directory from m1 to m2?
    • same OS or data files everywhere?
    • will all dedup if machines share a backup repo.

Borg Deduplication

• Content defined Chunk Deduplication
  • cut a file into variable sized chunks,
    content defines where a cut happens
    (efficiently done using a rolling hash)
  • MAC(chunk) is the key for the KV store
• No problem with:
  • inserted / deleted / shifted file contents
  • renamed files / dirs
  • VM disk images: only few chunks change

 $ borg info ssh://borg@myserver/repos/myrepo

               Original size   Compressed size   Dedup size
 All archives:      22.76 TB          18.22 TB    486.20 GB

                     Unique chunks         Total chunks
 Chunk index:              6305006            272643223

Borg assimilated Data

Real stats from a real backup repository (shortened).

2 machines,  147 backup archives,  2.5 years.

Borg now

• 1.1: current stable release
  • getting fixes primarily
  • getting few new features,
    if uncritical to stability
  • branch 1.1-maint in the repo
• 1.0: oldstable
  • not supported any more
• 1.2: in beta testing now
  • master branch in repo

Borg 1.2

• lots of code cleanups / refactoring
• setup.py cleaned + pkgconfig now
• internal AEAD-style crypto API
• compat layer for msgpack
• FD (not: path) based operation,
  less race conditions!
• separated recursion and processing
• using pyfuse3 (new) and also
  llfuse (deprecated, fuse2, widespread)
• minimal native windows support

Borg 1.2

• optional chunk size obfuscation
• compact_segments improved:
  • separate cli command now
  • more stable segments
  • faster. separated manifest/commits.
• create --paths-from-stdin
• ctrl-c: checkpoint, then abort
• incremental, time-ltd. repo check
• prune: show rule
• fixed blocksize chunker (disks!)

Borg Future

• 1.1 + 1.2 maintenance
• crypto improvements
• multithreading
• ...

Python / Cython / C

• Python (90%):
  • easy, high level logic
• Cython (5%):
  • write pythonic code, get C-ish speed
  • access C data types, functions, easy "nogil"
  • simple interface code for C libs,
    we use that for OpenSSL, lz4 and own C code.
• C (5%):
  • used for the most resource-usage critical parts
    (CPU as well as RAM usage)
  • own C code, bundled C code
  • hard to maintain, debug

pytest  &  tox

• pytest:
  • pretty and simple tests,
    less boilerplate than stdlib "unittest"
  • powerful framework
  • have fun writing tests
  • optionally remote and parallel tests
• tox:
  • automates testing on all python versions
  • each in a freshly built virtual env
  • plus flake8 checker, for pep8 and more

pyenv

• pull and build any python version you want
• easily switch between versions
• test on minimum requirement:
  • older point release == more bugs
  • py 3.[6789].0 to find all the issues
• build / bundle on latest / greatest release:
  • newer point release == less bugs
  • 3.7.latest to get best build

vagrant, vbox, qemu

• automate VMs:
  create, start, provision, ..., shutdown, destroy
• e.g. run tests / builds on:
  • Linux (misc. dists, old / new, 32 / 64bit, ...)
  • BSD (FreeBSD, OpenBSD, NetBSD)
  • macOS
  • OpenIndiana
  • Windows (maybe)
• PowerPC64 qemu VM with Debian to test on non-x86/x64 BE arch (most archs are LE).
• less surprises "oh, it does not work on X?"

pyinstaller

• creates a single-file binary, bundling together:
  • your Python / Cython / C code
  • (C)Python Interpreter of your choice
  • all required Python stdlib libraries
  • other required libraries
  • but not the (g)libc
• We use it to build Linux, FreeBSD, macOS borg binaries.
• Intentionally build on "old" OS:
  • all as-old or newer deployments will usually work
  • preferably not too old: security updates wanted

Secure Releasing with GPG

• creepy:  users execute downloaded blobs, as root.
• give them a chance to make sure it is authentic:
  • release signing key fingerprint widely published
  • public key uploaded to keyserver
  • document how to use GPG to verify the signatures
• git repo:  sign the release tags  (or every commit)
• release files:  sign them, detached sig

• note: just publishing hashes of files is no protection against attacks (just against accidential corruption)

setuptools_scm

• tired of bumping your version numbers?
• setuptools_scm makes versions from git tags:
  • considers latest tag
  • distance to that tag (commits)
  • workdir state (uncommitted changes?)

• 1.2.3 (tagged release code)

• 1.2.4.dev3+gdeadbee (3 commits later)

• 1.2.4.dev3+gdeadbee.d20170709 ("" + unclean)

Sphinx / Docs

• sphinx - generates html/pdf from reST
• reuse your ArgParser help:
  • build_usage:  html cli usage docs
  • build_man:  man pages
  • see archiver.py and setup.py
• reuse your github README:
  • include it docs intro
  • it's your "elevator speech"
• reuse your docs:
  • nice hosted docs can be your "home page"

ReadTheDocs.org

• builds and hosts your docs:
  • url like borgbackup.readthedocs.io
  • automatically built from github tags
  • version selector, stable, latest
  • nice theme (also for mobile devices)
  • offers PDF (and other) downloads
  • uses sphinx

asciinema.org

• create cool demo "videos" of console tools
• embed them on:
  • home page
  • docs intro
  • github README
• selectable static preview screen
• adjustable playback speed
• copy & paste works from the "video"
• recording is json:
  • fix typos by editing it
  • commit it to your repo

GitHub

• "Organisation" == a common ground
  • Main Repo "borg" with good README
  • Repo "community" with links to related stuff
• "Issues" (+ Labels)
  • bugs / todo / planning
  • ideas / feature requests / discussion
  • questions -> docs enhancements
  • bounties $$ via bountysource.com
• "Pull Requests" + Code Review
• "Milestones" for Release Planning
• "Releases" to publish changelog link, src, bin
• "Actions": CI

Communication Channels

• Mailing List and Archive:
  • borgbackup @ python.org
  • slow, async, permanent
• IRC (and also matrix):
  • #borgbackup @ chat.freenode.net
  • quicker, sync/async, transient
• Twitter:
  • @borgbackup on Twitter
• Usages:
  • support
  • discussion
  • (release) announcements

Borg - you can be assimilated!

• test scalability / reliability / security

• find, file and fix bugs

• file and implement feature requests

• improve docs

• contribute or review code

• spread the word

• create dist packages

• care for misc. platforms (windows)

• donate funds via bountysource

For more information:

borgbackup.org

Questions / Feedback?

• tw @ waldmann-edv . de

• Thomas J Waldmann @ twitter