Attic
"The holy grail of backup software"
Thomas Waldmann @ easterhegg 2015 (updated)
Attic Feature Set (1)
- simple & fast
- deduplication
- compression
-
authenticated encryption
-
easy pruning of old backups
-
simple backend (k/v, fs, via ssh)
Attic Feature Set (2)
-
FOSS (BSD license)
-
good docs
-
good platform / arch support
-
xattr / acl support
-
FUSE support ("mount a backup")
Attic Code
- 91% Python3 + Cython
(high-level code, glue code) - 9% C
(performance critical stuff) - only ~6000 LOC total
- few dependencies
- unit tests, CI
Attic Security
-
Signatures / Authentication
no undetected corruption/tampering
-
Encryption / Confidentiality
only you have access to your data
-
FOSS in Python
review possible, no buffer overflows
Attic Safety
-
Robustness
(by append-only design, transactions)
-
Checkpoints
every 5 minutes (between files)
-
msgpack with "limited" Unpacker
(no memory DoS)
Attic Crypto Keys
-
client-side meta+data encryption
-
separate keys for sep. concerns
-
passphrase pbkdf2 100k rounds
-
Keys:
- none
- passphrase-only
- passphrase protected keyfile
Attic Crypto Cipher/MAC
-
AEAD, Encrypt-then-MAC
- AES256-GCM / GHASH
- AES256-CTR + HMAC-SHAxxx
-
Counter / IV deterministic, never repeats
-
uses OpenSSL
- Intel/AMD: AES-NI, PCLMULQDQ
Attic Compression
-
Python stdlib:
- zlib (medium fast)
-
lzma (slow, high compression)
-
blosc library:
-
multithreaded, highly optimized
- "faster than memcpy"
- lz4 (superfast, reasonable compression)
-
lz4hc (very fast, "high compression")
-
zlib (faster than the implementation from stdlib)
-
Attic Deduplication (1)
-
No problem with:
- VM images (sparse file support)
- disk images
- renamed huge directories/trees
- inner deduplication of data set
- historical deduplication
- deduplication between different machines
Attic Deduplication (2)
-
Content defined chunking:
- "buzhash" rolling hash
- cut data when hash has specific bit pattern,
yields chunks with 2^nbits target size - seeded, to avoid fingerprinting chunk lenghts
-
Store chunks under id into store:
- id = HASH(chunk)
- id = MAC(mac_key, chunk)
Attic, the present
-
Works, but beta - be careful! Be more careful with code from git.
-
attic 0.14 from 2014-12-17
-
don't use anything older!
-
esp. no msgpack < 0.4.6!
Attic - Code Repos
-
official repo: jborg/attic
-
unofficial repo: attic/merge:
-
master == same as official stuff
-
merge == conservative merges
-
merge-all == merge + more
-
Attic, the future
-
bus-factor++, fork?
-
improve scalability / reliability / security
-
pull backups? backup-only mode?
-
better logging / exception handling
-
more backends? http / ftp / aws / google / ...
-
other platforms / architectures
-
<you name it>
Attic - we need help!
-
test scalability / reliability / security
-
file bugs
-
file feature requests
-
improve docs
-
contribute code
-
spread the word
-
create dist packages
Attic - Links
-
attic-backup.org (original)
-
github.com/attic/merge (unofficial)
-
#attic on chat.freenode.net
Questions / Feedback?
-
Just grab me, I am here all days!
-
Thomas J Waldmann @ twitter
Attic - Demo / Workshop
I'll show a developer installation / recent code.
If too complicated, just try something like:
Or use the binary packages from:
https://attic-backup.org/downloads/releases/0.14/
apt-get install attic
Attic - Installation Preps
# Debian / Ubuntu
# Python 3.x (>= 3.2) + Headers, Py Package Installer
apt-get install python3.4-dev python3.4 python3-pip
# we need OpenSSL + Headers for Crypto
apt-get install libssl-dev openssl
# ACL support Headers + Library
apt-get install libacl1-dev libacl1
# if you do not have gcc / make / etc. yet
apt-get install build-essential
# optional: lowlevel FUSE py binding - to mount backup archives
apt-get install python3-llfuse fuse
# optional: for unit testing
apt-get install fakeroot
Attic - system wide install
# A) system-wide install with pip, latest original release:
sudo pip install attic
# note: maybe you have to use pip3 to get the python3 pip
Attic - dev install from git
# B) isolated install, latest attic/merge git repo code:
mkdir attic ; cd attic
git clone https://github.com/attic/merge.git attic
# note: use branch "merge" for conservative changes
cd attic
git checkout merge-all # latest / most features
apt-get install python-virtualenv
virtualenv --python=python3 ../attic-env
source ../attic-env/bin/activate # always before using!
# install attic + dependencies into virtualenv
pip install cython # compile .pyx -> .c
pip install tox # optional, for running unit tests
pip install -e .
# check your install
fakeroot -u tox
attic init / create
# initialize a repository:
attic init /tmp/attic
# create a "first" archive inside this repo:
attic create /tmp/attic::first ~/Desktop
# create a "second" archive, more verbose:
attic create --progress --stats /tmp/attic::second ~/Desktop
# even more verbose:
attic create -v --stats /tmp/attic::third ~/Desktop
attic list / extract / check
# list repo / archive contents:
attic list /tmp/attic
attic list /tmp/attic::first
# extract ("restore") from an archive to cwd:
mkdir test ; cd test
attic extract /tmp/attic::third
# simulate extraction (good test):
attic extract -v --dry-run /tmp/attic::third
# check consistency of repo:
attic check /tmp/attic
attic info / delete / help
# info about repo:
attic info /tmp/attic
# delete archive:
attic delete /tmp/attic::first
# delete repo:
attic delete /tmp/attic
attic - crypto/compression
# options, options, options, ...
attic init --help
# create a encrypted, compressed repo:
# 21 = lz4 level 1 = very fast, for fast repo storage
attic init -e keyfile -c 21 /tmp/attic-enc
# create a encrypted, compressed repo:
# 69 = zlib level 9 = a lot slower, for slow repo storage
attic init -e keyfile -c 69 /tmp/attic-enc
# ... (same as before, but you need to give passphrase)
attic - remote via ssh
# connect to remote attic via ssh:
# remote attic needs to be compatible with local
attic init ssh://user@host:22/mnt/backup/attic
attic create ssh://user@host:22/mnt/backup/attic::first ~
# also possible: using sshfs or other locally mounted
# network filesystems, but be careful: locks, perf.
Attic - Links
-
attic-backup.org (original)
-
github.com/attic/merge (unofficial)
-
#attic on chat.freenode.net
Questions / Feedback?
-
Just grab me, I am here all days at easterhegg!
-
Thomas J Waldmann @ twitter
Attic
By Thomas Waldmann
Attic
- 3,490