There are lots of hard things to get in IT. Self-hosting email is one of them. But backups are definitely in my top 3, too. Embark with me on my quest for an adequate backup system.

Backups are something you hear a lot about. There’s the tao of backup. There’s all the blog posts explaining why you should backup your tool using this or that tool. There the myriad of “you can always restore from backups and if you don’t have any, that’s your fault” comments posted by techbros whenever someone loose their data and ask for help.

But when it come to actually implementing a backup system, at scale, from scratch, you’ll find surprisingly little info. This blog entry is an attempt to help with this situation and hopefully give you a place to start, and, as much as possible, a drop-in solution.

Expect a long post, because I’ll go over my previous setup and my needs, to give you some context. The second part of this post will include code and instructions to deploy a decent backup system from scratch.

A note on my needs

I’m hosting dozens of services, scattered accross a dozen of hosts and vms. I may have different needs than you do, but let’s try to explicit these:

I need my backup solution to be simple. And by that, I mean simple to understand. Cognitively simple. I don’t want to pipe 10s of tools manually to have encryption, compression, remote upload and pruning. I need my backup solution to handle all the backup lifecycle, from creating the backups to restoring one or pruning old entries.

I need my backup solution to be reliable. I don’t mind writing a few scripts, but the core solution should be maintained, tested and trustable.

I need my backup solution to be automatic. And by that, I don’t only mean “run it through a cronjob”. I want to be able to setup backups on existing or new hosts in a few commands (ideally one).

I need my backup solution to be easily configurable. Adding or removing new paths to backup on a host, pruning policies, remote storage… All of this should be easily configurable. I want to be able to backup this configuration too.

I need my backup solution to support multiple storage backends. I need to have multiple copies of my backups, on separate hosts or storage.

There are also a few “nice to have” that are not strict requirements, but definitely helpful:

  • Encrypted backups are better, because it means I can store these on untrusted storage boxes
  • Incremental backups are better, because they consume less space
  • Compressed backups are better, because they consume less space

The journey begins…

…in 2012, with crontabs and ad-hoc shell scripts. FTP storage. It was a dark time. It kind of worked for a while because I managed considerably less services and hosts than I currently do. However, I sometimes forgot to write the backup scripts when hosting a new service. I never tried a restore. I sometimes had full disk issues, because I didn’t have any pruning in place.

A new challenger appears

In 2013, I started working on a small python utility, named Savior, to do it better. It addressed a lot of my needs, and it remained by backup solution for many years. I also add a lot of fun working on it.

However, I was the only one maintaining and using it, and it wasn’t as much as reliable as I wanted it to be.

An unexpected ally

One day, I stumbled upon backupninja, which quickly replaced Savior on all my new services. It wasn’t perfect, but it was more stable, and vastly superior to Savior in terms of feature and configuration. It supported incremental and encrypted backups through duplicity.

But the pruning part wasn’t great, and it wasn’t really maintained. It worked, but for how long?

The silent road

For many years, I never found a decent replacement to backupninja. Let’s be honest, I didn’t try that hard to find one, but whenever I tried something new, like Borg, even if the solution was infinitely better, it was always missing something that would discourage me from doing the switch.

It has always been in the back of my mind though, and everytime I hosted a new service, I felt guilty of not doing backups properly. I was afraid of the day where all of this would fail apart and I wouldn’t be able to restore after a data loss.

It remained that way until last week.

The restic tavern

A few days ago, I discovered autorestic. This is a small utility working on top of restic, a backup tool.

Basically, restic does the heavy lifting of backups: creating, uploading to remote storages, pruning, restoring, etc. Restic does incremental, encrypted backups through a command-line interface.

On top of that, autorestic adds easy to write configuration files that are translated internally to restic calls.

When I discovered it, everything clicked immediatly and the pieces of the puzzle felt in-place. Over the past two days, I’ve been working on my new backup system based on restic and autorestic, and so far, it’s working great. Since I feel much more confident regarding my backups and it checks all my requirements, including a few nice-to-have, I think it would be worth it to share in details how this new system works.

At this point, I feel like my journey for backups is finally over!

Setting-up a backup system with restic, autorestic and S3

For the purpose of this guide, I’ll go through all the required steps to setup an automated backup system using restic, autorestic and S3, by hand on a single server. I’ll also include instructions to do the same with ansible, on multiple servers.

This guide assumes you have a basic knowledge of linux, command-line, and that you have:

  • one or more servers, accessed by SSH, with root access
  • one or more S3-compatible storage spaces (e.g on AWS, Wasabi, or using a self-hosted Minio server). You can use other types of storage with restic, such as SFTP or local, but this is not covered in this guide.
  • you have wget and bzip2 installed (apt-get install wget bzip2)

Installing restic

First, we need to install restic:

# download restic 0.9.6
wget -O /tmp/restic.bz2 https://github.com/restic/restic/releases/download/v0.9.6/restic_0.9.6_linux_amd64.bz2
# unzip the binary
bzip2 -dc /tmp/restic.bz2 > /usr/bin/restic
# make the binary executable
chmod +x /usr/bin/restic
# check restic is installed successfully
restic --help

Installing autorestic

Then, we install autorestic. I’m using my own release, because it includes some of my contributions that are not released yet (like support for pruning). Replace with the latest release URL when there is one:

# download autorestic
wget -O /usr/bin/autorestic https://github.com/EliotBerriot/autorestic/releases/download/Test/autorestic-linux
# make the binary executable
chmod +x /usr/bin/autorestic
# check restic is installed successfully
autorestic --help

Our first backup

Autorestic relies on YAML files for configuration. We’ll create our very first backup configuration, at /etc/.autorestic.yml:

# this is the part of the configuration that describes the paths
# you want to backup. You can have as many locations as you want
locations:
  # this tells autorestic we have a location named `etc`
  # This is mainly used for display purpose during the backup process
  # And when calling the CLI
  etc:
    # this tells autorestic what is the path to backup
    from: /etc
    # this tells autorestic that this location is backuped
    # on the `local-storage` backend, described below
    to: local-storage

# this is the part of the configuration that describes backup backends
# each backend is a place where backups will be stored, after being encrypted
backends:
  # this tells autorestic our backend is named `local-storage`
  # this is mainly used for display purpose during the backup process
  # when calling the CLI and in `locations:`
  local-storage:
    # the type of backend. Common one are local, s3 and sftp
    # we use local for this example because it's easy to setup
    type: local
    # the directory where backups will be stored, locally
    path: /var/backups/restic

The file is heavily commented, and you can also refer to the official documentation if you want to know more about the various options.

Because this file will contain encryption keys, it’s recommended to harden the permissions with chmod 600 /etc/.autorestic.yml.

Now, let’s launch our first backup:

autorestic -c /etc/.autorestic.yml backup --all --verbose

Depending of the amount of data available in /etc, this may takes up to a few minutes (it only takes a few seconds on my host), and should give you the following output:

Configuring Backends
local-storage : Configuring... ⏳created restic repository 2b1cd8995a at /var/backups/restic

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
local-storage : Done ✓

Backing Up
etc ▶ local-storage : Backing up... ⏳created new cache in /root/.cache/restic

Files:         641 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 2.219 MiB

processed 641 files, 2.240 MiB in 0:00
snapshot de83f50a saved 

On the first run, autorestic will initialize any new backend and update the config file with encryption keys. You should keep a backup of these keys somewhere (typically in a password manager), or you won’t be able to access your backup if you loose the config file.

At this point you can verify that restic was called successfully, by listing available snapshots:

autorestic -c /etc/.autorestic.yml exec --backend local-storage snapshots

Restoring

We can also restore the snapshot we made into a directory on the host to check the restore logic:

autorestic -c /etc/.autorestic.yml restore --location etc -- --target /tmp/etc-restore

This will create a /tmp/etc-restore directory with all the contents of the /etc directory according to the last backup snapshots.

Other useful commands

You can call any restic command through autorestic:

# listing shapshots
autorestic -c /etc/.autorestic.yml exec --backend local-storage snapshots

# print changes between two snapshots
autorestic -c /etc/.autorestic.yml exec --backend local-storage diff <snapshot_id1> <snapshot_id2>

# getting the content of a backuped file
# use latest or a snapshot id
autorestic -c /etc/.autorestic.yml exec --backend local-storage dump latest  /etc/shadow

# mount a backend locally to explore snapshots and files
autorestic -c /etc/.autorestic.yml exec --backend local-storage mount /tmp/test

# print statisticts about backup sizes and file counts
autorestic -c /etc/.autorestic.yml exec --backend local-storage stats

More complex configuration with multiple remote storages and pruning

You may rembember I described a few needs that aren’t covered in the previous exemples. In particular:

  • Pruning old backups
  • Copy backups to multiple remote rocations

Luckily, this is supported by restic and autorestic. We’re going to tweak our initial /etc/.autorestic.yml file to support this. I’ve removed previous comments for clarity:


locations:
  etc:
    from: /etc
    # we pass a list of backends here to upload the backups
    # in multiple places
    to:
      - remote-storage-1
      - remote-storage-2
    keep:
      # keep one backup per day for the last 7 days, one backup per week for
      # the last 4 weeks, and one backup per month for the last 2 months
      # all other snapshots will be pruned
      daily: 7
      weekly: 4
      monthly: 2

backends:
  remote-storage-1:
    # the S3 storage type allow uploading backups to any S3-compatible
    # storage
    type: s3
    path: s3.wasabisys.com/mybucket
    AWS_ACCESS_KEY_ID: accesskey
    AWS_SECRET_ACCESS_KEY: secretaccesskey
  remote-storage-2:
    type: s3
    path: minio.mydomain.com/mybucket
    AWS_ACCESS_KEY_ID: accesskey
    AWS_SECRET_ACCESS_KEY: secretaccesskey

Using the config file above, restic would backup /etc as before, but instead of keeping the backup locally at /var/backups/restic, it will now upload the backup to two separate S3 storages. This means that even if my server goes down, the backups are not lost as long as I have access to one of this storages and a copy of the encryption keys.

Nothing has changed on the CLI side, and all the previous commands are working as before:

# run the backup command
autorestic -c /etc/.autorestic.yml backup --all --verbose

Additionally, we’ve introduced pruning through the keep: option. You can refer to the restic documentation on the topic, but to summarize, this option tells restic how many past snapshots we want to keep. If you don’t specify it, no snapshots would ever be pruned, meaning you could run out of storage space (or get more expensive bills).

Whenever you want to prune backups according to your policies, just run:

autorestic -c /etc/.autorestic.yml forget -a --verbose

You can also add the --dry-run option if you want to check the planned operation without touching the data.

Automating

The last piece of the puzzle is to automate the backup process. You can do that using a cronjob:

# edit your crontab
crontab -e
# add the following entry
0 4 * * * /usr/bin/autorestic -c /etc/.autorestic.yml backup -a --verbose && /usr/bin/autorestic -c /etc/.autorestic.yml forget -a --verbose

This cronjob entry will run autorestic backup and autorestic forget every day at 4:00 AM.

Scaling to multiple hosts

Using the guide above, you can now easily setup backups on any of your hosts. However, doing this by hand on multiple hosts is error prone. Ideally, we’d be able to set this up automatically on multiple servers.

Fortunately, some IT automation tools exist exactly for that purpose, and I’m going to show you how to use Ansible to easily install and configure your backups on as many hosts as you need.

A note about Ansible

I won’t dive too much in the details regarding Ansible, because it’s a complex and rich technology. If you want to know more, please have a look at their documentation.

To sum it up, Ansible is a set of tools and programs that let you describe how you want one or more hosts to be configured, and applied the desired configuration.

Let’s say you have 5 hosts, and you want to install a bunch of packages on each of them. The easy way would be to connect to each of them through SSH, and run apt-get install package1 package2. However, if you add a 6th server to this pool a few months later, you’d have to remember what packages you installed exactly, and run the command again on the new server. Now, imagine handling that when you have dozens or even hundreds of commands needed to initialize a host: firewall, backups, webservers, etc.

Ansible takes a different approach. Instead of running commands on server directly, you write configuration, in YAML format. This configuration can be shared with colleagues, saved into a Git repository, and applied to any host you have access to.

There are a few terms that are common in the Ansible terminology:

  • Playbooks: playbooks are list of tasks that you want to apply to one or more hosts
  • Tasks: tasks are units of configuration you want to apply to an host. For example, “install this apt package” and “create a backup user” are two distinct tasks
  • Inventories: inventories are logical collections of hosts. They contain all the information required by Ansible to connect to your hosts. You could have a production inventory, for your production servers, and a test inventory to quickly test your playbooks without impacting production.
  • Roles: roles are reusable playbooks that can be packaged and imported in your own playbooks. Instead of writing your own playbook to install and configure nginx, you could for example use the jdauphant/nginx role.

Installing ansible

Ansible is installed on your local machine, not on the hosts you want to manage. All the commands from now on are meant to run on your own personal computer.

Follow the official installation documentation then come back here.

Creating ansible configuration files

We’ll need to create a few configuration files. I’m putting these under ~/projects/infra, but you can use a different path if you want:

mkdir -p ~/projects/infra
cd ~/projects/infra
mkdir inventory/
touch inventory/prod.yml
touch ansible.cfg
touch requirements.yml
touch playbook.yml

Put the following in ansible.cfg:

[defaults]
inventory = ./inventory/prod.yml

Put the following in inventory/prod.yml:

all:
  hosts:
    1.2.3.4:
    1.2.3.5:

Replace 1.2.3.4 and 1.2.3.5 by the actual ip or domain name of your hosts.

At this point, you can check that your inventory is written correctly by running ansible all -a "cat /etc/hostname" -u root. This simply echo the contents of the /etc/hostname file on each host in your inventory. Replace the value of -u root by the username used for your SSH login, and add --ask-become-pass and --become to allow Ansible to use sudo on the host.

With my config, the previous config echoes something like that:

ansible all -a "cat /etc/hostname" -u root
1.2.3.4| CHANGED | rc=0 >>
host1
1.2.3.5| CHANGED | rc=0 >>
host2

Installing the autorestic-backup role

To make it easier for you, I’ve written and published an Ansible role that manage the installation and configuration of restic/autorestic.

Put the following in requirements.yml:

- src: git+https://code.eliotberriot.com/eliotberriot/autorestic-backup-ansible-role
  name: autorestic-backup
  version: master

Then run ansible-galaxy install -r requirements.yml --force to install the role.

Add the required configuration to your inventory

To leverage the autorestic-backup role, you need to specify a few variables in your inventory, in particular:

  • your restic backends
  • the locations you want to backup for each host

Here is an example inventory with the relevant configuration:

# inventory/prod.yml
all:
  hosts:
    1.2.3.4:
      restic_backup_host_prefix: backup-host1
      restic_backup_locations:
        - from: /etc
        - from: /home
    1.2.3.5:
      restic_backup_host_prefix: backup-host2
      restic_backup_locations:
        - from: /etc
        - from: /var/log

  vars:
    restic_all_backends:
      wasabi:
        type: s3
        path: "s3.wasabisys.com/"
        AWS_ACCESS_KEY_ID: "accesskey"
        AWS_SECRET_ACCESS_KEY: "secretkey"

    restic_enabled_backends: [wasabi]

Write your first playbook

Edit the playbook.yml and add the following:

- hosts: all
  roles:
    - { role: autorestic-backup }

Run your playbook

At this point, you’re ready to test the playbook.

First, we’re going to run in check mode. That means Ansible will tell you about what it plan to do, without applying any change:

ansible-playbook playbook.yml --check --diff -u root

The --diff flag will print more verbose information about what’s going on. If the command runs without issue, rerun it without the --check flag:

ansible-playbook playbook.yml --diff -u root

This should take longer, but at the end, you’ll have restic and autorestic installed and configured on each of your hosts, with a cronjob setup!

If you ever need to change the configuration (e.g change the AWS access key or add a new backup location), edit your inventory file, and rerun ansible-playbook playbook.yml --check --diff -u root. This will automatically apply the required changes on your hosts.

Going further

The autorestic-backup role supports additional variables for configuring pruning, cronjob schedule, restic version, etc. Please refer to the documentation for more information.

In the example I shared, the AWS keys are included in cleartext in the invertory/prod.yml file. This isn’t a good practice if you need to commit your inventory in a Git repository. Have a look at Ansible Vault for guidance about encrypting sensitive variables in your inventories.

Conclusion

Of course, this backup system isn’t perfect (it doesn’t support compression, for instance, eventhough it’s planned on restic’s side), but it’s much better than what I previously have, while fitting all my requirements.

I’ve successfully backed up ~700GB on 11 different hosts using this approach (and the ansible playbook). The initial backups took a while, but subsequent runs where much, much faster (thanks incremental backups :D).

Thank you for reading, I hope this post was useful and you learned a few things!