Borg Backup

 

 

(2.0 alpha, updates & plans)

 

 

 

 

 

Thomas Waldmann (@home, 2022-07)

Borg Versions

  • Borg 1.1:
    • supported and very stable.
    • final release 1.1.18, after that only critical fixes.
       
  • Borg 1.2:
    • first releases done: 1.2.0, 1.2.1
       
  • Borg 2.0:
    • borg2 branch, will eventually get into master
    • 2.0.0a2 alpha release is out.
    • breaking changes, bleeding edge for testers!

2.0 = breaking!

  • break compatibility,  issue #6602
  • do all breaking changes in one release
  • all is new: new repos, new server, new clients
  • cli syntax cleanup
  • get rid of all legacy
  • simplify code base
  • solve some fundamental issues (pending since 7y)
  • get rid of troublemakers, stuff that blocks progress
  • no in-place repo upgrade:
    • saves developer time
    • less complex, less potential bugs, clean repos
  • but instead: new borg archive transfer command

CLI: repos + archives

  • no scp style repos any more:  user@host:path
    • no port possible
    • parser sometimes confused this with local path
  • ssh URL is better:  ssh://user@host:port/path
    • a port is possible here
    • easy to parse / disambiguate
  • archive name separate from repo, no "::" any more
    • borg -r REPO diff ARCH1 ARCH2
      • fixed amount of archs: positional params
    • borg -r REPO delete -a ARCH_GLOB --first 3
      • ​​some?:  -a 'crap-*'
      • want just one specific?:  -a only_this_one

repo vs. arch cmds

  • commands work either on repo or on archive(s)
    • borg rcreate = "repo create"  (was: borg init)
    • borg create = "archive create"
    • borg rlist = "repo list"
    • borg list = "archive list"
    • borg delete = "archive delete"
    • borg rdelete = "repo delete"
  • exception: borg check syntax is unchanged
    • works on repo and archives by default
    • --repository-only
    • --archives-only

borg repo concepts

A borg repo is a LOG.

(== stuff only gets appended at the end,
old stuff is never modified [only deleted])

A borg repo is a key/value store.

(key: chunk id = MAC(plaintext), value = ciphertext)

Low-level repo operations:

  • PUT (append a new key/value pair)
  • DELETE (register a delete for a previous put by key)
  • COMMIT (finish a transaction, state is valid now)

Segment files

  • contain a sequence of log entries created by repo ops
  • a non-compact segment contains deleted PUT entries

borg < 2.0:  PUT

PUT log entry structure:

  • crc32 = CRC32(header + content)
  • header: size of entry
  • header: tag (== PUT)
  • header: 256bit key (chunk id)
  • content: value (data)

 

Big design issue

One can not check the header only.

To check the crc32, one must read ALL: header+content

Slow if one is only interested in header values.

Correct size is important to seek to next entry.

borg2:  PUT2

PUT2 log entry structure:

  • crc32 = CRC32(header + digest)
  • header: size of entry
  • header: tag
  • header: 256bit key (chunk id)
  • digest: xxh64 = XXH64(header + content)
  • content: value (data)

 

Notable

Can check the header without reading the content.

Better error detection by stronger and super fast xxh64.

crc32 covers header+digest, digest also covers header.

Sometimes slow CRC32 impl. only used for few bytes.

Borg < 2.0 crypto

old crypto issues

  • potential nonce reuse and counter measures:
    • AES-CTR mode with 1 AES key per repo
    • counter values (IV / nonce) must never be re-used
    • complex counter management needed
    • limited trust in repository
    • local counter knowledge can be lost (e.g. disk defect)
    • multiple clients need to trust repo
    • to avoid counter issues, repos must use different keys
    • no easy replication of encrypted chunks to other repo
  • self-made layering of
    • AES256-CTR + HMAC-SHA256
    • AES256-CTR + BLAKE2b
  • there are faster ready-to-use AEAD ciphers now

borg2:  new crypto

new crypto features

  • Fixes potential nonce reuse issue:
    • random session id generated at start of a borg run
    • session key derived from session id and master key
    • counter (IV / Nonce) starts from 0 for each session
    • no counter management needed, no risk of reuse
  • OpenSSL >= 1.1.1 (including on OpenBSD), providing:
    • super fast AES256-OCB (with AES hw acceleration).
      patents first licensed to FOSS, now abandoned.
    • very fast CHACHA20-POLY1305 (pure sw implementation)
  • use AAD of AEAD cipher to protect header / chunkid
  • Argon2 KDF used for the borg key (was: pbkdf2)
  • undecided: maybe adopt BLAKE3 for the chunk id hash

hardlinks

  • borg < 2.0 approach:
    • first hardlink archived as regular file, with chunkid list
    • second hardlink archived:
      • refers back to first one by-name
      • does not have own chunkid list
    • problematic partial extraction, messy code, special cases
  • borg 2.0 approach:
    • hardlinks archived like a normal item
    • regular files / HLs always have chunkid list (1st, 2nd, ...)
    • if st_nlinks > 1:  item.hlid = H(st_dev, st_ino)
    • rule: hlid is same -> items point to same inode (are HLs).
    • symmetric:  1st HL archived the same way as 2nd, 3rd...

msgpack

  • msgpack old spec - type confusion:
    • did not differentiate between text and binary data
    • text could be encoded, but comes back as binary
    • if you get binary, it could have been binary or text
  • msgpack new spec - roundtripping done right:
    • text (str):
      • comes back as str
      • borg uses utf-8 with surrogate-escape handler
      • gets encoded/decoded automatically
    • binary (bytes):
      • comes back as bytes
      • gets stored "as is"
  • ​borg2 uses the new spec, borg < 2.0 uses old spec.

new tar formats

These were initially intended for data migration.

--tar-format=PAX

ctime and atime support, all ts in ns resolution

could support more metadata, like xattrs, ACLs, ...,
but a lot of work to implement and test

--tar-format=BORG

Like PAX plus custom BORG.* PAX headers
for perfect round-tripping
of all borg supported fs item metadata.

Copy archive from repo1 to repo2:

borg export-tar ... repo1::A | borg import-tar ... repo2::A

 

Problem: no dedup, huge amount of data

borg2 transfer

Create a related new repo:

borg --repo NEWREPO rcreate --other-repo OLDREPO --encryption CIDH

Transfer archives:

borg --repo NEWREPO transfer --other-repo OLDREPO [--dry-run]

Efficiency

  • deduplication: transfer each chunk only once
  • no expensive re-compression / re-chunking
  • but: re-encryption is required (but fast!)
  • old chunks deduplicate with future chunks, requires: related repo (key material), CIDH = compat. ID hash
  • some (cheap) data conversions done on the fly:
    cleanups, type conversions, msgpack changes

release N+1 plans

  • in 2.0 we needed to keep some of the old stuff:
    • borg transfer needs to read old repos / archives
    • users need to transfer their archives to new repos
  • in N+1 (2.1?) we can remove:
    • AES-CTR mode, counter management code
    • old style keys (pbkdf2, Encrypt and MAC)
    • code for old repo index, old chunks index
    • PUT(1) repo code
    • zlib type bytes hack
    • bigint stuff (replaced by msgpack Timestamp)
    • msgpack-related "good that we know the type"
    • hardlink_master processing
    • old RPC protocols
    • borg transfer / item code that converts old to new

Support the Borg

Contributions are welcome!

 

Code, documentation, review, testing, funding, ...

 

Just join us on GitHub and LiberaChat IRC #borgbackup.

 

 

Donations please via LiberaPay or BountySource:

 

https://www.borgbackup.org/support/fund.html

 

For more information:

borgbackup.org

Questions / Feedback?

  • tw @ waldmann-edv . de

  • Thomas J Waldmann @ twitter

Made with Slides.com