Attic

 

"The holy grail of backup software"

 

 

Thomas Waldmann @ easterhegg 2015 (updated)

Attic Feature Set (1)

  • simple & fast
  • deduplication
  • compression

  • authenticated encryption

  • easy pruning of old backups

  • simple backend (k/v, fs, via ssh)

Attic Feature Set (2)

  • FOSS (BSD license)

  • good docs

  • good platform / arch support

  • xattr / acl support

  • FUSE support ("mount a backup")

Attic Code

  • 91% Python3 + Cython
    (high-level code, glue code)
  • 9% C
    (performance critical stuff)
  • only ~6000 LOC total
  • few dependencies
  • unit tests, CI

Attic Security

  • Signatures / Authentication
    no undetected corruption/tampering
     
  • Encryption / Confidentiality
    only you have access to your data
     
  • FOSS in Python
    review possible, no buffer overflows

Attic Safety

  • Robustness
    (by append-only design, transactions)
     
  • Checkpoints
    every 5 minutes (between files)
     
  • msgpack with "limited" Unpacker
    (no memory DoS)

Attic Crypto Keys

  • client-side meta+data encryption
     
  • separate keys for sep. concerns
     
  • passphrase pbkdf2 100k rounds
     
  • Keys:
    • none
    • passphrase-only
    • passphrase protected keyfile

Attic Crypto Cipher/MAC

  • AEAD, Encrypt-then-MAC
    • AES256-GCM / GHASH
    • AES256-CTR + HMAC-SHAxxx
    • Counter / IV deterministic, never repeats
       
  • uses OpenSSL

     
  • Intel/AMD: AES-NI, PCLMULQDQ

Attic Compression

  • Python stdlib:
    • zlib  (medium fast)
    • lzma  (slow, high compression)
       

  • blosc library:

    • multithreaded, highly optimized

    • "faster than memcpy"
    • lz4  (superfast, reasonable compression)

    • lz4hc  (very fast, "high compression")

    • zlib  (faster than the implementation from stdlib)

       

Attic Deduplication (1)

  • No problem with:
    • VM images (sparse file support)
    • disk images
    • renamed huge directories/trees
    • inner deduplication of data set
    • historical deduplication
    • deduplication between different machines

 

Attic Deduplication (2)

  • Content defined chunking:
    • "buzhash" rolling hash
    • cut data when hash has specific bit pattern,
      yields chunks with 2^nbits target size
    • seeded, to avoid fingerprinting chunk lenghts
       
  • Store chunks under id into store:
    • id = HASH(chunk)
    • id = MAC(mac_key, chunk)

Attic, the present

  • Works, but beta - be careful! Be more careful with code from git.
     

  • attic 0.14 from 2014-12-17
     

  • don't use anything older!

  • esp. no msgpack < 0.4.6!

Attic - Code Repos

  • official repo:   jborg/attic

     

  • unofficial repo:  attic/merge:

    • master == same as official stuff

    • merge == conservative merges

    • merge-all == merge + more

Attic, the future

  • bus-factor++, fork?

  • improve scalability / reliability / security

  • pull backups? backup-only mode?

  • better logging / exception handling

  • more backends? http / ftp / aws / google / ...

  • other platforms / architectures

  • <you name it>

Attic - we need help!

  • test scalability / reliability / security

  • file bugs

  • file feature requests

  • improve docs

  • contribute code

  • spread the word

  • create dist packages

Attic - Links

  • attic-backup.org  (original)

  • github.com/attic/merge  (unofficial)

  • #attic on chat.freenode.net

Questions / Feedback?

  • Just grab me, I am here all days!

  • Thomas J Waldmann @ twitter

Attic - Demo / Workshop

I'll show a developer installation / recent code.

 

If too complicated, just try something like:

 

 

 

Or use the binary packages from:

 

https://attic-backup.org/downloads/releases/0.14/

 

  apt-get install attic

Attic - Installation Preps


# Debian / Ubuntu

# Python 3.x (>= 3.2) + Headers, Py Package Installer
apt-get install python3.4-dev python3.4 python3-pip

# we need OpenSSL + Headers for Crypto
apt-get install libssl-dev openssl

# ACL support Headers + Library
apt-get install libacl1-dev libacl1

# if you do not have gcc / make / etc. yet
apt-get install build-essential

# optional: lowlevel FUSE py binding - to mount backup archives
apt-get install python3-llfuse fuse

# optional: for unit testing
apt-get install fakeroot

Attic - system wide install


# A) system-wide install with pip, latest original release:

sudo pip install attic

# note: maybe you have to use pip3 to get the python3 pip

Attic - dev install from git

# B) isolated install, latest attic/merge git repo code:

mkdir attic ; cd attic
git clone https://github.com/attic/merge.git attic

# note: use branch "merge" for conservative changes
cd attic
git checkout merge-all  # latest / most features

apt-get install python-virtualenv
virtualenv --python=python3 ../attic-env
source ../attic-env/bin/activate   # always before using!

# install attic + dependencies into virtualenv
pip install cython  # compile .pyx -> .c
pip install tox   # optional, for running unit tests
pip install -e .

# check your install
fakeroot -u tox

attic init / create

# initialize a repository:

attic init /tmp/attic


# create a "first" archive inside this repo: 

attic create /tmp/attic::first ~/Desktop


# create a "second" archive, more verbose:

attic create --progress --stats /tmp/attic::second ~/Desktop


# even more verbose:

attic create -v --stats /tmp/attic::third ~/Desktop

attic list / extract / check

# list repo / archive contents:

attic list /tmp/attic
attic list /tmp/attic::first

# extract ("restore") from an archive to cwd:

mkdir test ; cd test
attic extract /tmp/attic::third

# simulate extraction (good test):

attic extract -v --dry-run /tmp/attic::third

# check consistency of repo:

attic check /tmp/attic

attic info / delete / help

# info about repo:

attic info /tmp/attic

# delete archive:

attic delete /tmp/attic::first

# delete repo:

attic delete /tmp/attic

attic - crypto/compression

# options, options, options, ...

attic init --help

# create a encrypted, compressed repo:
# 21 = lz4 level 1 = very fast, for fast repo storage

attic init -e keyfile -c 21 /tmp/attic-enc

# create a encrypted, compressed repo:
# 69 = zlib level 9 = a lot slower, for slow repo storage

attic init -e keyfile -c 69 /tmp/attic-enc


# ... (same as before, but you need to give passphrase)

attic - remote via ssh

# connect to remote attic via ssh:
# remote attic needs to be compatible with local

attic init ssh://user@host:22/mnt/backup/attic

attic create ssh://user@host:22/mnt/backup/attic::first ~


# also possible: using sshfs or other locally mounted
# network filesystems,  but be careful: locks, perf.

Attic - Links

  • attic-backup.org  (original)

  • github.com/attic/merge  (unofficial)

  • #attic on chat.freenode.net

Questions / Feedback?

  • Just grab me, I am here all days at easterhegg!

  • Thomas J Waldmann @ twitter

Made with Slides.com