mirror of
https://github.com/funkypenguin/geek-cookbook/
synced 2025-12-13 01:36:23 +00:00
Add Restic, swiss-army-knife of backup tools! (#121)
This commit is contained in:
David Young
211
manuscript/recipes/restic.md
Normal file
211
manuscript/recipes/restic.md
Normal file
@@ -0,0 +1,211 @@
|
||||
# Restic
|
||||
|
||||
Don't be like [Cameron](http://haltandcatchfire.wikia.com/wiki/Cameron_Howe). Backup your stuff.
|
||||
|
||||
<iframe width="560" height="315" src="https://www.youtube.com/embed/1UtFeMoqVHQ" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
|
||||
[Restic](https://restic.net/) is a backup program intended to be easy, fast, verifiable, secure, efficient, and free. Restic supports a range of backup targets, including local disk, [SFTP](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#sftp), [S3](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#amazon-s3) (*or compatible APIs like [Minio](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#minio-server)*), [Backblaze B2](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#backblaze-b2), [Azure](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#microsoft-azure-blob-storage), [Google Cloud Storage](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#google-cloud-storage), and zillions of others via [rclone](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone).
|
||||
|
||||
Restic is one of the more popular open-source backup solutions, and is often [compared favorable](https://www.reddit.com/r/golang/comments/6mfe4q/a_performance_comparison_of_duplicacy_restic/dk2pkoj/?context=8&depth=9) to "freemium" products by virtue of its [licence](https://github.com/restic/restic/blob/master/LICENSE).
|
||||
|
||||
## Details
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
* [X] [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
|
||||
* [X] [Traefik](/ha-docker-swarm/traefik_public) and [Traefik-Forward-Auth](/ha-docker-swarm/traefik-forward-auth) configured per design
|
||||
* [X] Credentials for one of Restic's [supported repositories](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html)
|
||||
|
||||
## Preparation
|
||||
|
||||
### Setup data locations
|
||||
|
||||
We'll need a data location to bind-mount persistent config (*an exclusion list*) into our container, so create them as below:
|
||||
|
||||
```
|
||||
mkdir -p /var/data/restic/
|
||||
mkdir -p /var/data/config/restic
|
||||
echo /var/data/runtime >> /var/data/restic/restic.exclude
|
||||
```
|
||||
|
||||
!!! note
|
||||
`/var/data/restic/restic.exclude` details which files / directories to **exclude** from the backup. Per our [data layout](/reference/data_layout/), runtime data such as database files are stored in `/var/data/runtime/[recipe]`, and excluded from backups, since we can't safely backup/restore data-in-use. Databases should be backed up by taking dumps/snapshots, and backing up _these_ dumps/snapshots instead.
|
||||
|
||||
### Prepare environment
|
||||
|
||||
Create `/var/data/config/restic/restic-backup.env`, and populate with the following variables:
|
||||
|
||||
```
|
||||
# run on startup, otherwise just on cron
|
||||
RUN_ON_STARTUP=true
|
||||
|
||||
# when to run (TZ ensures it runs when you expect it!)
|
||||
BACKUP_CRON=0 0 1 * * *
|
||||
TZ=Pacific/Auckland
|
||||
|
||||
# restic backend/storage credentials
|
||||
# see https://restic.readthedocs.io/en/stable/040_backup.html#environment-variables
|
||||
#AWS_ACCESS_KEY_ID=xxxxxxxx
|
||||
#AWS_SECRET_ACCESS_KEY=yyyyyyyyy
|
||||
#B2_ACCOUNT_ID=xxxxxxxx
|
||||
#B2_ACCOUNT_KEY=yyyyyyyyy
|
||||
|
||||
# will initialise the repo on startup the first time (if not already initialised)
|
||||
# don't lose this password otherwise you WON'T be able to decrypt your backups!
|
||||
RESTIC_REPOSITORY=<repo_name>
|
||||
RESTIC_PASSWORD=<repo_password>
|
||||
|
||||
# what to backup (excluding anything in restic.exclude)
|
||||
RESTIC_BACKUP_SOURCES=/data
|
||||
|
||||
# define any args to pass to the backup operation (e.g. the exclude file)
|
||||
# see https://restic.readthedocs.io/en/stable/040_backup.html
|
||||
RESTIC_BACKUP_ARGS=--exclude-file /restic.exclude
|
||||
|
||||
# define any args to pass to the forget operation (e.g. what snapshots to keep)
|
||||
# see https://restic.readthedocs.io/en/stable/060_forget.html
|
||||
RESTIC_FORGET_ARGS=--keep-daily 7 --keep-monthly 12
|
||||
```
|
||||
|
||||
Create `/var/data/config/restic/restic-prune.env`, and populate with the following variables:
|
||||
|
||||
```
|
||||
# run on startup, otherwise just on cron
|
||||
RUN_ON_STARTUP=false
|
||||
|
||||
# when to run (TZ ensures it runs when you expect it!)
|
||||
PRUNE_CRON=0 0 4 * * *
|
||||
TZ=Pacific/Auckland
|
||||
|
||||
# restic backend/storage credentials
|
||||
# see https://restic.readthedocs.io/en/stable/040_backup.html#environment-variables
|
||||
#AWS_ACCESS_KEY_ID=xxxxxxxx
|
||||
#AWS_SECRET_ACCESS_KEY=yyyyyyyyy
|
||||
#B2_ACCOUNT_ID=xxxxxxxx
|
||||
#B2_ACCOUNT_KEY=yyyyyyyyy
|
||||
|
||||
# will initialise the repo on startup the first time (if not already initialised)
|
||||
# don't lose this password otherwise you WON'T be able to decrypt your backups!
|
||||
RESTIC_REPOSITORY=<repo_name>
|
||||
RESTIC_PASSWORD=<repo_password>
|
||||
|
||||
# prune will remove any *forgotten* snapshots, if there are some args you want
|
||||
# to pass to the prune operation define them here
|
||||
#RESTIC_PRUNE_ARGS=
|
||||
```
|
||||
|
||||
!!! question "Why create two separate .env files?"
|
||||
Although there's duplication involved, maintaining 2 files for the two services within the stack keeps it clean, and allows you to potentially alter the behaviour of one service without impacting the other in future
|
||||
|
||||
|
||||
### Setup Docker Swarm
|
||||
|
||||
Create a docker swarm config file in docker-compose syntax (v3) in `/var/data/restic/restic.yml` , something like this:
|
||||
|
||||
!!! tip
|
||||
I share (_with my [sponsors](https://github.com/sponsors/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that sponsors can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
|
||||
|
||||
```
|
||||
version: "3.2"
|
||||
|
||||
services:
|
||||
backup:
|
||||
image: mazzolino/restic
|
||||
env_file: /var/data/config/restic/restic-backup.env
|
||||
hostname: docker
|
||||
volumes:
|
||||
- /var/data/restic/restic.exclude:/restic.exclude
|
||||
- /var/data:/data:ro
|
||||
deploy:
|
||||
labels:
|
||||
- "traefik.enabled=false"
|
||||
|
||||
prune:
|
||||
image: mazzolino/restic
|
||||
env_file: /var/data/config/restic/restic-prune.env
|
||||
hostname: docker
|
||||
deploy:
|
||||
labels:
|
||||
- "traefik.enabled=false"
|
||||
|
||||
networks:
|
||||
internal:
|
||||
driver: overlay
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.16.56.0/24
|
||||
```
|
||||
|
||||
!!! note
|
||||
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
|
||||
|
||||
|
||||
|
||||
## Serving
|
||||
|
||||
### Launch Restic stack
|
||||
|
||||
Launch the Restic stack by running `docker stack deploy restic -c <path -to-docker-compose.yml>`, and watch the logs by running `docker service logs restic_backup` - you should see something like this:
|
||||
|
||||
```
|
||||
root@raphael:~# docker service logs restic_backup -f
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | Checking configured repository '<repo_name>' ...
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | Fatal: unable to open config file: Stat: stat <repo_name>/config: no such file or directory
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | Is there a repository at the following location?
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | <repo_name>
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | Could not access the configured repository. Trying to initialize (in case it has not been initialized yet) ...
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | created restic repository 66ffec75f9 at <repo_name>
|
||||
restic_backup.1.9sii77j9jf0x@leonardo |
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | Please note that knowledge of your password is required to access
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | the repository. Losing your password means that your data is
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | irrecoverably lost.
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | Repository successfully initialized.
|
||||
restic_backup.1.9sii77j9jf0x@leonardo |
|
||||
restic_backup.1.9sii77j9jf0x@leonardo |
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | Scheduling backup job according to cron expression.
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | new cron: 0 0 1 * * *
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | (0x50fac0,0xc0000cc000)
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | Stopping
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | Waiting
|
||||
restic_backup.1.9sii77j9jf0x@leonardo | Exiting
|
||||
```
|
||||
|
||||
Of note above is =="Repository successfully initialized"== - this indicates that the repository credentials passed to Restic are correct, and Restic has the necessary access to create repositories.
|
||||
|
||||
### Restoring data
|
||||
|
||||
Repeat after me : "**It's not a backup unless you've tested a restore**"
|
||||
|
||||
The simplest way to test your restore is to run the container once, using the variables you're already prepared, with custom arguments, as follows:
|
||||
|
||||
```
|
||||
docker run --rm -it --name restic-restore --env-file /var/data/config/restic/restic-backup.env \
|
||||
-v /tmp/restore:/restore mazzolino/restic restore latest --target /restore
|
||||
```
|
||||
|
||||
In my example:
|
||||
|
||||
```
|
||||
root@raphael:~# docker run --rm -it --name restic-restore --env-file /var/data/config/restic/restic-backup.env \
|
||||
> -v /tmp/restore:/restore mazzolino/restic restore latest --target /restore
|
||||
Unable to find image 'mazzolino/restic:latest' locally
|
||||
latest: Pulling from mazzolino/restic
|
||||
Digest: sha256:cb827c4c5e63952f8d114c87432ff12d3409a0ba4bcb52f53885dca889b1cb6b
|
||||
Status: Downloaded newer image for mazzolino/restic:latest
|
||||
Checking configured repository 's3:s3.amazonaws.com/restic-geek-cookbook-premix.elpenguino.be' ...
|
||||
Repository found.
|
||||
repository c50738d1 opened successfully, password is correct
|
||||
restoring <Snapshot b5c50b19 of [/data] at 2020-06-24 23:54:27.92318041 +0000 UTC by root@docker> to /restore
|
||||
root@raphael:~#
|
||||
```
|
||||
|
||||
!!! tip "Restoring a subset of data"
|
||||
The example above restores the **entire** `/var/data` folder (*minus any exclusions*). To restore just a subset of data, add the `-i <regex>` argument, i.e. `-i plex`
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
|
||||
1. The `/var/data/restic/restic.exclude` exists to provide you with a way to exclude data you don't care to backup.
|
||||
2. A recent benchmark of various backup tools, including Restic, can be found [here](https://forum.duplicati.com/t/big-comparison-borg-vs-restic-vs-arq-5-vs-duplicacy-vs-duplicati/9952).
|
||||
3. A paid-for UI for Restic can be found [here](https://forum.restic.net/t/web-ui-for-restic/667/26).
|
||||
Reference in New Issue
Block a user