diff --git a/manuscript/CHANGELOG.md b/manuscript/CHANGELOG.md index 76c672c..e70de6d 100644 --- a/manuscript/CHANGELOG.md +++ b/manuscript/CHANGELOG.md @@ -13,14 +13,13 @@ Also available via: ## Recently added recipes +* Added recipe for [Restic](https://geek-cookbook.funkypenguin.co.nz/recipes/restic/), simple and secure backup solution with **huge** range of target platforms via rclone (_25 June 2020_) * Added recipe for [Duplicati](https://geek-cookbook.funkypenguin.co.nz/recipes/duplicati/), all-in-one backup solution with a handy GUI (_18 June 2020_) * Overhauled [Ceph (Shared Storage)](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph/) recipe for Ceph Octopus (v15) (_25 May 2020_) -* Added recipe for making your own [DIY Kubernetes Cluster](/kubernetes/diycluster/) (_14 December 2019_) -* Added recipe for [authenticating Traefik Forward Auth against KeyCloak](/ha-docker-swarm/traefik-forward-auth/keycloak/) (_16 May 2019_) + ## Recent improvements * Updated [Traefik Forward Auth](/ha-docker-swarm/traefik-forward-auth/) for latest @thomseddon image (_11 Jun 2020_) * Added recipe for [automated snapshots of Kubernetes Persistent Volumes](/kubernetes/snapshots/), instructions for using [Helm](/kubernetes/helm/), and recipe for deploying [Traefik](/kubernetes/traefik/), which completes the Kubernetes cluster design! (_9 Feb 2019_) -* Added detailed description (_and diagram_) of our [Kubernetes design](/kubernetes/design/), plus a [simple load-balancer design](kubernetes/loadbalancer/) to avoid the complexities/costs of permitting ingress access to a cluster (_7 Feb 2019_) -* Added an [introductory/explanatory page, including a children's story, on Kubernetes](/kubernetes/start/) (_29 Jan 2019_) +* Added detailed description (_and diagram_) of our [Kubernetes design](/kubernetes/design/), plus a [simple load-balancer design](kubernetes/loadbalancer/) to avoid the complexities/costs of permitting ingress access to a cluster (_7 Feb 2019_) \ No newline at end of file diff --git a/manuscript/community/contribute.md b/manuscript/community/contribute.md index 1f038dc..029d405 100644 --- a/manuscript/community/contribute.md +++ b/manuscript/community/contribute.md @@ -6,7 +6,7 @@ Got nothing to contribute, but want to give back to the community? Here are some 1. Star :star: the [repo](https://github.com/geek-cookbook/geek-cookbook/) 2. [Sponsor](https://github.com/sponsors/funkypenguin) :heart: the chef -3. Tweet 🐦 the [meat](https://ctt.ac/Vl6mc)! +3. Tweet :bird: the [meat](https://ctt.ac/Vl6mc)! ## Contributing bugfixes 🐛 diff --git a/manuscript/recipes/restic.md b/manuscript/recipes/restic.md new file mode 100644 index 0000000..ab164b9 --- /dev/null +++ b/manuscript/recipes/restic.md @@ -0,0 +1,211 @@ +# Restic + +Don't be like [Cameron](http://haltandcatchfire.wikia.com/wiki/Cameron_Howe). Backup your stuff. + + + +[Restic](https://restic.net/) is a backup program intended to be easy, fast, verifiable, secure, efficient, and free. Restic supports a range of backup targets, including local disk, [SFTP](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#sftp), [S3](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#amazon-s3) (*or compatible APIs like [Minio](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#minio-server)*), [Backblaze B2](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#backblaze-b2), [Azure](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#microsoft-azure-blob-storage), [Google Cloud Storage](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#google-cloud-storage), and zillions of others via [rclone](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#other-services-via-rclone). + +Restic is one of the more popular open-source backup solutions, and is often [compared favorable](https://www.reddit.com/r/golang/comments/6mfe4q/a_performance_comparison_of_duplicacy_restic/dk2pkoj/?context=8&depth=9) to "freemium" products by virtue of its [licence](https://github.com/restic/restic/blob/master/LICENSE). + +## Details + +## Ingredients + +!!! summary "Ingredients" + * [X] [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md) + * [X] [Traefik](/ha-docker-swarm/traefik_public) and [Traefik-Forward-Auth](/ha-docker-swarm/traefik-forward-auth) configured per design + * [X] Credentials for one of Restic's [supported repositories](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html) + +## Preparation + +### Setup data locations + +We'll need a data location to bind-mount persistent config (*an exclusion list*) into our container, so create them as below: + +``` +mkdir -p /var/data/restic/ +mkdir -p /var/data/config/restic +echo /var/data/runtime >> /var/data/restic/restic.exclude +``` + +!!! note + `/var/data/restic/restic.exclude` details which files / directories to **exclude** from the backup. Per our [data layout](/reference/data_layout/), runtime data such as database files are stored in `/var/data/runtime/[recipe]`, and excluded from backups, since we can't safely backup/restore data-in-use. Databases should be backed up by taking dumps/snapshots, and backing up _these_ dumps/snapshots instead. + +### Prepare environment + +Create `/var/data/config/restic/restic-backup.env`, and populate with the following variables: + +``` +# run on startup, otherwise just on cron +RUN_ON_STARTUP=true + +# when to run (TZ ensures it runs when you expect it!) +BACKUP_CRON=0 0 1 * * * +TZ=Pacific/Auckland + +# restic backend/storage credentials +# see https://restic.readthedocs.io/en/stable/040_backup.html#environment-variables +#AWS_ACCESS_KEY_ID=xxxxxxxx +#AWS_SECRET_ACCESS_KEY=yyyyyyyyy +#B2_ACCOUNT_ID=xxxxxxxx +#B2_ACCOUNT_KEY=yyyyyyyyy + +# will initialise the repo on startup the first time (if not already initialised) +# don't lose this password otherwise you WON'T be able to decrypt your backups! +RESTIC_REPOSITORY= +RESTIC_PASSWORD= + +# what to backup (excluding anything in restic.exclude) +RESTIC_BACKUP_SOURCES=/data + +# define any args to pass to the backup operation (e.g. the exclude file) +# see https://restic.readthedocs.io/en/stable/040_backup.html +RESTIC_BACKUP_ARGS=--exclude-file /restic.exclude + +# define any args to pass to the forget operation (e.g. what snapshots to keep) +# see https://restic.readthedocs.io/en/stable/060_forget.html +RESTIC_FORGET_ARGS=--keep-daily 7 --keep-monthly 12 +``` + +Create `/var/data/config/restic/restic-prune.env`, and populate with the following variables: + +``` +# run on startup, otherwise just on cron +RUN_ON_STARTUP=false + +# when to run (TZ ensures it runs when you expect it!) +PRUNE_CRON=0 0 4 * * * +TZ=Pacific/Auckland + +# restic backend/storage credentials +# see https://restic.readthedocs.io/en/stable/040_backup.html#environment-variables +#AWS_ACCESS_KEY_ID=xxxxxxxx +#AWS_SECRET_ACCESS_KEY=yyyyyyyyy +#B2_ACCOUNT_ID=xxxxxxxx +#B2_ACCOUNT_KEY=yyyyyyyyy + +# will initialise the repo on startup the first time (if not already initialised) +# don't lose this password otherwise you WON'T be able to decrypt your backups! +RESTIC_REPOSITORY= +RESTIC_PASSWORD= + +# prune will remove any *forgotten* snapshots, if there are some args you want +# to pass to the prune operation define them here +#RESTIC_PRUNE_ARGS= +``` + +!!! question "Why create two separate .env files?" + Although there's duplication involved, maintaining 2 files for the two services within the stack keeps it clean, and allows you to potentially alter the behaviour of one service without impacting the other in future + + +### Setup Docker Swarm + +Create a docker swarm config file in docker-compose syntax (v3) in `/var/data/restic/restic.yml` , something like this: + +!!! tip + I share (_with my [sponsors](https://github.com/sponsors/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that sponsors can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍 + +``` +version: "3.2" + +services: + backup: + image: mazzolino/restic + env_file: /var/data/config/restic/restic-backup.env + hostname: docker + volumes: + - /var/data/restic/restic.exclude:/restic.exclude + - /var/data:/data:ro + deploy: + labels: + - "traefik.enabled=false" + + prune: + image: mazzolino/restic + env_file: /var/data/config/restic/restic-prune.env + hostname: docker + deploy: + labels: + - "traefik.enabled=false" + +networks: + internal: + driver: overlay + ipam: + config: + - subnet: 172.16.56.0/24 +``` + +!!! note + Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here. + + + +## Serving + +### Launch Restic stack + +Launch the Restic stack by running `docker stack deploy restic -c `, and watch the logs by running `docker service logs restic_backup` - you should see something like this: + +``` +root@raphael:~# docker service logs restic_backup -f +restic_backup.1.9sii77j9jf0x@leonardo | Checking configured repository '' ... +restic_backup.1.9sii77j9jf0x@leonardo | Fatal: unable to open config file: Stat: stat /config: no such file or directory +restic_backup.1.9sii77j9jf0x@leonardo | Is there a repository at the following location? +restic_backup.1.9sii77j9jf0x@leonardo | +restic_backup.1.9sii77j9jf0x@leonardo | Could not access the configured repository. Trying to initialize (in case it has not been initialized yet) ... +restic_backup.1.9sii77j9jf0x@leonardo | created restic repository 66ffec75f9 at +restic_backup.1.9sii77j9jf0x@leonardo | +restic_backup.1.9sii77j9jf0x@leonardo | Please note that knowledge of your password is required to access +restic_backup.1.9sii77j9jf0x@leonardo | the repository. Losing your password means that your data is +restic_backup.1.9sii77j9jf0x@leonardo | irrecoverably lost. +restic_backup.1.9sii77j9jf0x@leonardo | Repository successfully initialized. +restic_backup.1.9sii77j9jf0x@leonardo | +restic_backup.1.9sii77j9jf0x@leonardo | +restic_backup.1.9sii77j9jf0x@leonardo | Scheduling backup job according to cron expression. +restic_backup.1.9sii77j9jf0x@leonardo | new cron: 0 0 1 * * * +restic_backup.1.9sii77j9jf0x@leonardo | (0x50fac0,0xc0000cc000) +restic_backup.1.9sii77j9jf0x@leonardo | Stopping +restic_backup.1.9sii77j9jf0x@leonardo | Waiting +restic_backup.1.9sii77j9jf0x@leonardo | Exiting +``` + +Of note above is =="Repository successfully initialized"== - this indicates that the repository credentials passed to Restic are correct, and Restic has the necessary access to create repositories. + +### Restoring data + +Repeat after me : "**It's not a backup unless you've tested a restore**" + +The simplest way to test your restore is to run the container once, using the variables you're already prepared, with custom arguments, as follows: + +``` +docker run --rm -it --name restic-restore --env-file /var/data/config/restic/restic-backup.env \ + -v /tmp/restore:/restore mazzolino/restic restore latest --target /restore +``` + +In my example: + +``` +root@raphael:~# docker run --rm -it --name restic-restore --env-file /var/data/config/restic/restic-backup.env \ +> -v /tmp/restore:/restore mazzolino/restic restore latest --target /restore +Unable to find image 'mazzolino/restic:latest' locally +latest: Pulling from mazzolino/restic +Digest: sha256:cb827c4c5e63952f8d114c87432ff12d3409a0ba4bcb52f53885dca889b1cb6b +Status: Downloaded newer image for mazzolino/restic:latest +Checking configured repository 's3:s3.amazonaws.com/restic-geek-cookbook-premix.elpenguino.be' ... +Repository found. +repository c50738d1 opened successfully, password is correct +restoring to /restore +root@raphael:~# +``` + +!!! tip "Restoring a subset of data" + The example above restores the **entire** `/var/data` folder (*minus any exclusions*). To restore just a subset of data, add the `-i ` argument, i.e. `-i plex` + + +## Chef's Notes 📓 + +1. The `/var/data/restic/restic.exclude` exists to provide you with a way to exclude data you don't care to backup. +2. A recent benchmark of various backup tools, including Restic, can be found [here](https://forum.duplicati.com/t/big-comparison-borg-vs-restic-vs-arq-5-vs-duplicacy-vs-duplicati/9952). +3. A paid-for UI for Restic can be found [here](https://forum.restic.net/t/web-ui-for-restic/667/26). \ No newline at end of file diff --git a/manuscript/reference/data_layout.md b/manuscript/reference/data_layout.md index f8c6252..7dd0cb2 100644 --- a/manuscript/reference/data_layout.md +++ b/manuscript/reference/data_layout.md @@ -10,7 +10,7 @@ Configuration data goes into `/var/data/config/[recipe name]`, and is typically ## Runtime data -Realtime data (*typically database files or files-in-use*) are stored in `/var/data/runtime/[recipe-name]`, and are **excluded** from backup (_They change constantly, and cannot be safely restored_). +Runtime data (*typically database files or files-in-use*) are stored in `/var/data/runtime/[recipe-name]`, and are **excluded** from backup (_They change constantly, and cannot be safely restored_). ## Static data diff --git a/manuscript/reference/networks.md b/manuscript/reference/networks.md index 7dea08e..1b07782 100644 --- a/manuscript/reference/networks.md +++ b/manuscript/reference/networks.md @@ -52,4 +52,5 @@ In order to avoid IP addressing conflicts as we bring swarm networks up/down, we | [Graylog](https://geek-cookbook.funkypenguin.co.nz/recipes/graylog/) | 172.16.52.0/24 | | [Harbor](https://geek-cookbook.funkypenguin.co.nz/recipes/graylog/) | 172.16.53.0/24 | | [Harbor-Clair](https://geek-cookbook.funkypenguin.co.nz/recipes/graylog/) | 172.16.54.0/24 | -| [Duplicati](https://geek-cookbook.funkypenguin.co.nz/recipes/duplicati/) | 172.16.55.0/24 | \ No newline at end of file +| [Duplicati](https://geek-cookbook.funkypenguin.co.nz/recipes/duplicati/) | 172.16.55.0/24 | +| [Restic](https://geek-cookbook.funkypenguin.co.nz/recipes/restic/) | 172.16.56.0/24 | \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index a35fe32..7d75c17 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -22,10 +22,12 @@ plugins: - search - minify: minify_html: true - # this doesn't work with the current netlify build process, which modifies .md files in the "publish" subfolder before building - git-revision-date-localized: type: date - autolinks + + + # - htmlproofer #theme_dir: mkdocs-material @@ -76,7 +78,6 @@ nav: - Jackett: recipes/autopirate/jackett.md - Heimdall: recipes/autopirate/heimdall.md - End: recipes/autopirate/end.md - - ElkarBackup: recipes/elkarbackup.md - Emby: recipes/emby.md - Home Assistant: - Start: recipes/homeassistant.md @@ -91,10 +92,10 @@ nav: - Miniflux: recipes/miniflux.md - Munin: recipes/munin.md - NextCloud: recipes/nextcloud.md - - OwnTracks: recipes/owntracks.md - phpIPAM: recipes/phpipam.md - Plex: recipes/plex.md - PrivateBin: recipes/privatebin.md + - Restic: recipes/restic.md - Swarmprom: recipes/swarmprom.md - Recipes: - Bitwarden: recipes/bitwarden.md @@ -102,6 +103,7 @@ nav: - Calibre-Web: recipes/calibre-web.md - Collabora Online: recipes/collabora-online.md - Duplicati: recipes/duplicati.md + - ElkarBackup: recipes/elkarbackup.md - Ghost: recipes/ghost.md - GitLab: recipes/gitlab.md - GitLab Runner: recipes/gitlab-runner.md @@ -114,9 +116,10 @@ nav: - OpenLDAP: recipes/keycloak/authenticate-against-openldap.md - Minio: recipes/minio.md - OpenLDAP: recipes/openldap.md - - Piwik: recipes/piwik.md + - OwnTracks: recipes/owntracks.md - Portainer: recipes/portainer.md - Realms: recipes/realms.md + - Restic: recipes/restic.md - Tiny Tiny RSS: recipes/tiny-tiny-rss.md - Traefik: ha-docker-swarm/traefik.md - Traefik Forward Auth: