Attic

"The holy grail of backup software"

Thomas Waldmann @ easterhegg 2015 (updated)

Attic Feature Set (1)

• simple & fast
• deduplication
• compression

• authenticated encryption

• easy pruning of old backups

• simple backend (k/v, fs, via ssh)

Attic Feature Set (2)

• FOSS (BSD license)

• good docs

• good platform / arch support

• xattr / acl support

• FUSE support ("mount a backup")

Attic Code

• 91% Python3 + Cython
  (high-level code, glue code)
• 9% C
  (performance critical stuff)
• only ~6000 LOC total
• few dependencies
• unit tests, CI

Attic Security

• Signatures / Authentication
  no undetected corruption/tampering
   
• Encryption / Confidentiality
  only you have access to your data
   
• FOSS in Python
  review possible, no buffer overflows

Attic Safety

• Robustness
  (by append-only design, transactions)
   
• Checkpoints
  every 5 minutes (between files)
   
• msgpack with "limited" Unpacker
  (no memory DoS)

Attic Crypto Keys

• client-side meta+data encryption
   
• separate keys for sep. concerns
   
• passphrase pbkdf2 100k rounds
   
• Keys:
  • none
  • passphrase-only
  • passphrase protected keyfile

Attic Crypto Cipher/MAC

• AEAD, Encrypt-then-MAC
  • AES256-GCM / GHASH
  • AES256-CTR + HMAC-SHAxxx
  • Counter / IV deterministic, never repeats
     
• uses OpenSSL
  
   
• Intel/AMD: AES-NI, PCLMULQDQ

Attic Compression

• Python stdlib:
  • zlib  (medium fast)
  • lzma  (slow, high compression)
     

• blosc library:

  • multithreaded, highly optimized

  • "faster than memcpy"
  • lz4  (superfast, reasonable compression)

  • lz4hc  (very fast, "high compression")

  • zlib  (faster than the implementation from stdlib)

     

Attic Deduplication (1)

• No problem with:
  • VM images (sparse file support)
  • disk images
  • renamed huge directories/trees
  • inner deduplication of data set
  • historical deduplication
  • deduplication between different machines

Attic Deduplication (2)

• Content defined chunking:
  • "buzhash" rolling hash
  • cut data when hash has specific bit pattern,
    yields chunks with 2^nbits target size
  • seeded, to avoid fingerprinting chunk lenghts
     
• Store chunks under id into store:
  • id = HASH(chunk)
  • id = MAC(mac_key, chunk)

Attic, the present

• Works, but beta - be careful! Be more careful with code from git.
   

• attic 0.14 from 2014-12-17
   

• don't use anything older!

• esp. no msgpack < 0.4.6!

Attic - Code Repos

• official repo:   jborg/attic
  
   

• unofficial repo:  attic/merge:

  • master == same as official stuff

  • merge == conservative merges

  • merge-all == merge + more

Attic, the future

• bus-factor++, fork?

• improve scalability / reliability / security

• pull backups? backup-only mode?

• better logging / exception handling

• more backends? http / ftp / aws / google / ...

• other platforms / architectures

• <you name it>

Attic - we need help!

• test scalability / reliability / security

• file bugs

• file feature requests

• improve docs

• contribute code

• spread the word

• create dist packages

Attic - Links

• attic-backup.org  (original)

• github.com/attic/merge  (unofficial)

• #attic on chat.freenode.net

Questions / Feedback?

• Just grab me, I am here all days!

• Thomas J Waldmann @ twitter

Attic - Demo / Workshop

I'll show a developer installation / recent code.

If too complicated, just try something like:

Or use the binary packages from:

https://attic-backup.org/downloads/releases/0.14/

  apt-get install attic  

Attic - Installation Preps

# Debian / Ubuntu

# Python 3.x (>= 3.2) + Headers, Py Package Installer
apt-get install python3.4-dev python3.4 python3-pip

# we need OpenSSL + Headers for Crypto
apt-get install libssl-dev openssl

# ACL support Headers + Library
apt-get install libacl1-dev libacl1

# if you do not have gcc / make / etc. yet
apt-get install build-essential

# optional: lowlevel FUSE py binding - to mount backup archives
apt-get install python3-llfuse fuse

# optional: for unit testing
apt-get install fakeroot

Attic - system wide install

# A) system-wide install with pip, latest original release:

sudo pip install attic

# note: maybe you have to use pip3 to get the python3 pip

Attic - dev install from git

# B) isolated install, latest attic/merge git repo code:

mkdir attic ; cd attic
git clone https://github.com/attic/merge.git attic

# note: use branch "merge" for conservative changes
cd attic
git checkout merge-all  # latest / most features

apt-get install python-virtualenv
virtualenv --python=python3 ../attic-env
source ../attic-env/bin/activate   # always before using!

# install attic + dependencies into virtualenv
pip install cython  # compile .pyx -> .c
pip install tox   # optional, for running unit tests
pip install -e .

# check your install
fakeroot -u tox

attic init / create

# initialize a repository:

attic init /tmp/attic


# create a "first" archive inside this repo: 

attic create /tmp/attic::first ~/Desktop


# create a "second" archive, more verbose:

attic create --progress --stats /tmp/attic::second ~/Desktop


# even more verbose:

attic create -v --stats /tmp/attic::third ~/Desktop

attic list / extract / check

# list repo / archive contents:

attic list /tmp/attic
attic list /tmp/attic::first

# extract ("restore") from an archive to cwd:

mkdir test ; cd test
attic extract /tmp/attic::third

# simulate extraction (good test):

attic extract -v --dry-run /tmp/attic::third

# check consistency of repo:

attic check /tmp/attic

attic info / delete / help

# info about repo:

attic info /tmp/attic

# delete archive:

attic delete /tmp/attic::first

# delete repo:

attic delete /tmp/attic

attic - crypto/compression

# options, options, options, ...

attic init --help

# create a encrypted, compressed repo:
# 21 = lz4 level 1 = very fast, for fast repo storage

attic init -e keyfile -c 21 /tmp/attic-enc

# create a encrypted, compressed repo:
# 69 = zlib level 9 = a lot slower, for slow repo storage

attic init -e keyfile -c 69 /tmp/attic-enc


# ... (same as before, but you need to give passphrase)

attic - remote via ssh

# connect to remote attic via ssh:
# remote attic needs to be compatible with local

attic init ssh://user@host:22/mnt/backup/attic

attic create ssh://user@host:22/mnt/backup/attic::first ~


# also possible: using sshfs or other locally mounted
# network filesystems,  but be careful: locks, perf.

Attic - Links

• attic-backup.org  (original)

• github.com/attic/merge  (unofficial)

• #attic on chat.freenode.net

Questions / Feedback?

• Just grab me, I am here all days at easterhegg!

• Thomas J Waldmann @ twitter