Backup and Restore

in Cassandra and OpsCenter

Overview

• Snapshot Operations
• Restore Operations
• Commit Log Archiving/Point in Time Restore
• Remote backup
• From both Cassandra and Opscenter perspectives

Snapshots

Nodetool Snapshot Basics

Performs a flush, then hard links sstables to 

More at 

http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsSnapShot.html

org.apache.cassandra.db
  ->StorageService
    ->takeSnapshot

<data_file_directories>/<ks>/<table>/snapshots/<snapshot-name>/

Under the hood, mbeans

Snapshots in Opscenter

• Under Services -> Backup
• Displays backup history, allows backup and restore.
• Advanced settings we'll cover later
• Backup Service is an Enterprise Feature

More at

http://docs.datastax.com/en/opscenter/5.2/opsc/online_help/services/opscBackupService.html

Snapshots in Opscenter

• Schedule repeated backups or create ad hoc backup
• Select keyspaces
• Set location (on server vs s3)
• Uses the mbean to perform the snapshot rather than shelling out.
• Coordinates the snapshot on all nodes.
• Backs up the schema to schema.json
• Keeps a log for audit

Auditable Records

Remote Snapshots

• Opscenter can also backup to s3
• Specify s3 bucket name, aws credentials
• Optional transfer throttle and compression
• Not all SSTables need to be backed up, because they are immutable only part of the data may require it.

• SSTables need to be stored per node to avoid name collisions.
• However dropping and recreating a table can lead to a naming collision as well, OPSC can attach a timestamp.
• If your data is encrypted, make sure that the encryption key is also put somewhere safe.
• Opsc backs up schemas
• Topologies change over time (more on this in restore).

Restore Operations

SSTableloader Basics

• Expects the schema to already exist for the sstables.
• Expects a directory structure different from that created by the snapshot, specifically <Keyspace>/<Table>/<files>
• Can stream data to other nodes, doesn't just move files into place
• Leaves files in place as they are restored, possible disk penalty.

More at

http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsBulkloader_t.html

Restore Operations

• Select a backup from a list of available snapshots.
• Point in Time restores (more on this later)
• Restore from other location

Restore Operations

• Attempts to recreate the schema or do a schema comparison.  The latter is extremely difficult with thrift.
• Creates symbolic links in a temporary directory to match what SSTableloader expects.
• Logs/audit trail to follow.
• Uses SSTableloader

Remote Restore

• Topologies change over time.
• When topologies shrink multiple nodes worth of data will have to be sent to a single node (sstable naming collisions).

Remote Restore

• When topologies grow some nodes may be idle during a restore.
• Replacement nodes will have a different host ID and will need to be matched to host ID of the snapshot.
• Opscenter handles all of these cases.

Commit Log Archiving

• Cassandra an execute a script when writing commit log segments
• set in commitlog_archiving.properties

http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configLogArchive_t.html

Commit Log Archiving

• Opscenter can enable that also under services->backups service->settings
• Opscenter can also send these to s3 as well.

http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configLogArchive_t.html

Point in Time Restore

• 2 step operation, restore snapshot, then replay commit logs.
• Find the nearest snapshot that happens prior to the point in time desired, perform a restore.
• Update commitlog_archiving.properties with the location of the commit logs as well as the point in time to restore.
• Restart cassandra.

More At

http://docs.datastax.com/en//cassandra/2.0/cassandra/configuration/configLogArchive_t.html

PiT in Opscenter

• OpsCenter can automate the PiT restore process
• Set time (in UTC) OpsCenter will verify that it is capable of restoring to that point in time.
• Commit logs or Snapshots can be local or on S3

PiT Restore Challenges

• Commit log replays don't stream data around the ring, this makes topology changes difficult to handle.
• Comparing schemas can be tricky if the reply contains schema changes.

Questions?

Feel free to reach out:

https://www.linkedin.com/in/philipsdoctor