I published my backup scripts here; this post is a bit more about how I use these scripts to manage my backups and my rationale for why I do backups the way I do. In particular, I’ve setup my backup servers to provide at least some protection against intentional deletion by a hostile party (e.g., ransomware operator, etc.) by enforcing append-only backups with Borg and Restic + Rclone.

Hopefully some of these ideas will be useful to others.

Threat Model and Requirements

My backup strategy is meant to guard against data loss or disclosure due to:

1. Accidental or incidental deletion (e.g., I delete a file by mistake or an OS or software bug renders a file inaccessible.)
2. Intentional deletion by an malicious actor (ransomware, vandalism, etc.)
3. Accidental or intentional disclosure of data on the backup media
4. Loss of access to backups due to backup data being stored in a proprietary format
5. Loss of access to backups due to a bug in the backup software
6. Fire or natural disaster

To protect – as best as possible – against these risks, I have the following requirements for my backup system:

1. Backups have to occur automatically, on an hourly basis. Manual backups are backups that don’t get done. Hourly has proven to be a good interval between backups constantly running and missing newly created files.
2. Backup maintenance and verification must occur automatically, for the same reasons.
3. All data must be stored encrypted-at-risk with client-side encryption. This is, primarily, to prevent disclosure of my data to someone who has or gains access to the backup media (i.e., someone steals one of my backup disks or someone gains access at the cloud provider I use for my offsite backups.) Encrypting all data at rest also makes lifecycling backup disks (i.e., selling, disposing, and RMAing them) a lot easier. It can take a few days to do a secure-write cycle on a 10TB+ disk, especially one that is being sent in for warranty replacement due to write errors. In that case, you can’t really be sure that your data was overwritten securely. Client side encryption eliminates that worry.
4. All data must be backed up to a local backup server to provide a complete backup that I have full control over, as well as a remote backup server to protect against fire or natural disaster. My backup drives are rotated every three months, with the offline drives being stored in a good fire- and burglary-resistant safe.
5. Since I’m making two backups anyway, I use two tools - Borg and Restic - to protect against the risk of a bug in my backup software rendering my backups unreadable or unrecoverable. Both tools are open source, which also eliminates any concern about the data format being proprietary.
6. It must not be possible for the system being backed up to be used to delete (or alter) the backups, at least not without manual, physical intervention on my part. Practically speaking, this means: a) no local-USB-disk backups, b) append-only backup mode enforced on the remote servers, c) any higher-privilege keys that could be use to delete or alter snapshots have to be offline or require physical interaction (e.g., a Yubikey, or similar device.)

The Backup Scheme

Taking the above into account, my backups look like this:

1. A complete backup made using Restic and written to an external drive attached to a dedicated local backup server. The disk is external so it can be rotated easily and regularly. The server only runs SSH on the network, and only supports public key-based authentication. The authorized_keys file strictly limits what the remote systems can do: they can only append to their specific Restic repository. More on that in a bit.

2. A mostly complete backup made using Borg and written to a remote cloud backup service. I use rsync.net but many other services would work just as well. This backup is also enforced server-side to be append-only, and meets my requirement of a separate physical location. This is a “mostly complete” backup: I do exclude some files from my NAS that I have separate physical copies of, like videos, and a few things I mirror from the Internet. This is purely for cost reasons - it would be prohibitively expensive for me to pay for cloud backups for these data sets.

Enforcing Append-only Backups

The heart of making this work and providing some resistance against Ransomware-style backup deletion is the append-only backup. I’ve largely based my approach on Peter Ruderich’s work, described here, but adapted to also work with Borg (in addition to Restic.)

Rather than use REST over HTTPS, for simplicity, I’m using SSH for transport. This has the added benefit of being very easy for me to audit to ensure my setup is doing what I think it is doing; it also works well with my remote backup provider.

All of the append-only enforcement is done server side, with SSH key restrictions. There is a slight difference in that borg has an append-only feature that we can use natively, whereas Restic requires the use of Rclone to enforce append-only mode. The SSH key restrictions are enforced via the key declaration in the authorized_keys file on the server. The entries look like this:

# Append-only key for Borg
restrict,command="borg serve --append-only --restrict-to-repository /path/to/backup/$HOST' ssh-ed25519 AAAA..... user@host
# Append only key for Restic
restrict,command="rclone serve restic --stdio --append-only /path/to/backup/$HOST" ssh-ed25519 AAAA.... user@host

For Restic, on the client side, you need to specify an extra flag to work with this restricted key:

restic -o rclone.program='ssh sftp://backupuser@backupserver/path/to/backup/repo' -r rclone: backup ...

Note that, despite setting the rclone.program flag, you do not need rclone installed on your client machines.

Borg does not require any special arguments or additional configuration on the client side for backups to work via the restricted key.

With these restrictions in place, you won’t be able to use the keys you’ve restricted to delete or rotate snapshots from the backup repository, compact the repository, etc. So, theoretically, if there are no other keys present on the systems being backed up, an attacker shouldn’t be able to use them to destroy your backups.

You’ll need to be able to do periodic maintenance on your backup sets - delete old snapshots, verify the backups, etc. You could use an offline key, like one stored in a Yubikey, to do this. Yubikeys also work well as keys for adhoc restoration. The approach I’ve taken, and which is documented in more detail on the Backuptools page is to have a dedicated host with an unrestricted key that can automatically age out old backups, do backup verification, and verify that the systems I think are backing up actually are uploading snapshots. Note that this is a different server than the local backup server which hosts the backup disk.

Encryption At Rest

To achieve automated encrypted backups, I have to store the encryption key as well as credentials to connect to my backup servers on each protected system. Thus, I have to assume that an attacker will have access to those credentials and the backup encryption key.

I’m not worried about that, however. The purpose of encryption at rest in my threat model is to prevent unauthorized access by someone who gains access to my backup media (either the local hard drives or someone working for or who gained access to the cloud backup service I use.) If an attacker is in a position to obtain the backup encryption passphrase from one of my systems, I have to assume they could just read the data directly off of that system without resorting to accessing the backups.

A Note About Rsync.net

If you are trying to do append-only backups with rsync.net, there are afew things to note:

1. You can’t do append-only backups with Restic on rsync.net, at least not as of the time this post was written (July 2024). This is despite what they claim on their website. I understand from this post that they are working on it. I sent an email asking them to clarify if this should work, if they had an estimate on when it would work, etc., but I haven’t heard back from them yet. In any case, if you try to run rclone serve via your rsync.net account, you’ll see it is stubbed out:

marcusb@mirth ~ % ssh [email protected] rclone serve --help
marcusb@mirth ~ %

Whereas, on a normal rclone build you’d get a bunch of output with that command.

2. You can do append-only backups with Borg on rsync.net, but you need to invoke ‘borg1’ instead of the normal ‘borg’ command. On their systems, ‘borg’ is version 0.29.x. ‘borg1’ is version 1.2.x. So, the authorized_keys entry looks like this instead:

restrict,command="borg1 serve --append-only --restrict-to-repository ./path/to/repo" ....

This doesn’t require any special invocations for the borg backup command to work, but any other commands, like repo verification or aging done via your admin keys will require one additional change: you’ll need to set the BORG_REMOTE_PATH environment variable to “borg1” before running your borg commands.

The Local Backup Server

In order to meet the requirement that an attacker not be able to delete my backups from a backed-up system, I can’t just attach a disk to, say, my NAS and expose it via SSH – that would still present the risk that someone who compromised my NAS, which has quite a bit of attack surface, could delete the backup disk. What I’ve done is setup a dedicated backup host using a small form factor desktop PC. Why one of these? They are cheaper and faster than Raspberry Pis.

While nothing is perfectly secure, I’ve hardened this system as much as possible. The only network service it runs is SSH, it only allows public-key based SSH authentication, and is only accessible over my LAN and VPN. I also rotate my local backup disks every three months, so as a last-ditch protection against an attacker destroying both my current local backup disk and my cloud backup, at least I would have an offline copy that is no more than three months old. I consider that an acceptable risk for my personal data.