public/geek-cookbook
1
0
Fork 0
mirror of https://github.com/funkypenguin/geek-cookbook/ synced 2025-12-13 01:36:23 +00:00
Code Issues Releases Wiki Activity

Experiment with PDF generation

Browse Source 
Signed-off-by: David Young <davidy@funkypenguin.co.nz>
This commit is contained in:
David Young
2022-08-19 16:40:53 +12:00
parent c051e0bdad
commit abf9309cb1
317 changed files with 124 additions and 546 deletions
Download Patch File Download Diff File Expand all files Collapse all files

280
docs/docker-swarm/authelia.md Normal file
View File

@@ -0,0 +1,280 @@
---
title: Using Authelia to secure services in Docker
description: Authelia is an open-source authentication and authorization server providing 2-factor authentication and single sign-on (SSO) for your applications via a web portal.
---
# Authelia in Docker Swarm
[Authelia](https://github.com/authelia/authelia) is an open-source authentication and authorization server providing 2-factor authentication and single sign-on (SSO) for your applications via a web portal. Like [Traefik Forward Auth][tfa], Authelia acts as a companion of reverse proxies like Nginx, [Traefik](/docker-swarm/traefik/), or HAProxy to let them know whether queries should pass through. Unauthenticated users are redirected to Authelia Sign-in portal instead. Authelia is a popular alternative to a heavyweight such as [KeyCloak][keycloak].
![Authelia Screenshot](/images/authelia.png){ loading=lazy }
Features include
* Multiple two-factor methods such as
* [Physical Security Key](https://www.authelia.com/docs/features/2fa/security-key) (Yubikey)
* OTP using Google Authenticator
* Mobile Notifications
* Lockout users after too many failed login attempts
* Highly Customizable Access Control using rules to match criteria such as subdomain, username, groups the user is in, and Network
* Authelia [Community](https://discord.authelia.com/) Support
* Full list of features can be viewed [here](https://www.authelia.com/docs/features/)
## Authelia requirements
!!! summary "Ingredients"
Already deployed:
* [X] [Docker swarm cluster](/docker-swarm/design/) with [persistent shared storage](/docker-swarm/shared-storage-ceph/)
* [X] [Traefik](/docker-swarm/traefik/) configured per design
New:
* [ ] DNS entry for your auth host (*"authelia.yourdomain.com" is a good choice*), pointed to your [keepalived](/docker-swarm/keepalived/) IP
### Setup data locations
First, we create a directory to hold the data which authelia will serve:
```bash
mkdir /var/data/config/authelia
```
### Create Authelia config file
Authelia configurations are defined in `/var/data/config/authelia/configuration.yml`. Some are required and some are optional. The following is a variation of the default example config file. Optional configuration settings can be viewed on in [Authelia's documentation](https://www.authelia.com/docs/configuration/)
!!! warning
Your variables may vary significantly from what's illustrated below, and it's best to read up and understand exactly what each option does.
```yaml title="/var/data/config/authelia/configuration.yml"
###############################################################
# Authelia configuration #
###############################################################
server:
host: 0.0.0.0
port: 9091
log:
level: warn
# This secret can also be set using the env variables AUTHELIA_JWT_SECRET_FILE
# I used this site to generate the secret: https://www.grc.com/passwords.htm
jwt_secret: SECRET_GOES_HERE
# https://docs.authelia.com/configuration/miscellaneous.html#default-redirection-url
default_redirection_url: https://authelia.example.com
totp:
issuer: authelia.example.com
period: 30
skew: 1
authentication_backend:
file:
path: /config/users_database.yml
# customize passwords based on https://docs.authelia.com/configuration/authentication/file.html
password:
algorithm: argon2id
iterations: 1
salt_length: 16
parallelism: 8
memory: 1024 # blocks this much of the RAM. Tune this.
# https://docs.authelia.com/configuration/access-control.html
access_control:
default_policy: one_factor
rules:
- domain: "bitwarden.example.com"
policy: two_factor
- domain: "whoami-authelia-2fa.example.com"
policy: two_factor
- domain: "*.example.com" # (1)!
policy: one_factor
session:
name: authelia_session
# This secret can also be set using the env variables AUTHELIA_SESSION_SECRET_FILE
# Used a different secret, but the same site as jwt_secret above.
secret: SECRET_GOES_HERE
expiration: 3600 # 1 hour
inactivity: 300 # 5 minutes
domain: example.com # Should match whatever your root protected domain is
regulation:
max_retries: 3
find_time: 120
ban_time: 300
storage:
encryption_key: SECRET_GOES_HERE_20_CHARACTERS_OR_LONGER
local:
path: /config/db.sqlite3
notifier:
# smtp:
# username: SMTP_USERNAME
# # This secret can also be set using the env variables AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE
# # password: # use docker secret file instead AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE
# host: SMTP_HOST
# port: 587 #465
# sender: batman@example.com # customize for your setup
# For testing purpose, notifications can be sent in a file. Be sure map the volume in docker-compose.
filesystem:
filename: /config/notification.txt
```
1. The wildcard rule must go last, since the first rule to match the request, wins
### Create Authelia user Accounts
Create `/var/data/config/authelia/users_database.yml` this will be where we can create user accounts and give them groups
```yaml title="/var/data/config/authelia/users_database.yml"
# To create a hashed password you can run the following command:
# `docker run authelia/authelia:latest authelia hash-password YOUR_PASSWORD``
users:
batman: # each new user should be defined in a dictionary like this
displayname: "Batman"
# replace this with your hashed password. This one, for the purposes of testing, is "password"
password: "$argon2id$v=19$m=65536,t=3,p=4$cW1adlh3UjhIRE9zSmZyZw$xA4S2X8BjE7LVb4NndJCZnoyHgON5w3FopO4vw5AQxE"
email: batman@example.com
groups:
- admins
- dev
```
To create a hashed password you can run the following command
`docker run authelia/authelia:latest authelia hash-password YOUR_PASSWORD`
### Authelia Docker Swarm config
Create a docker swarm config file in docker-compose syntax (v3), something like this example:
--8<-- "premix-cta.md"
```yaml title="/var/data/config/authelia/authelia.yml"
version: "3.2"
services:
authelia:
image: authelia/authelia
volumes:
- /var/data/config/authelia:/config
networks:
- traefik_public
deploy:
labels:
# traefik common
- traefik.enable=true
- traefik.docker.network=traefik_public
# traefikv1
- traefik.frontend.rule=Host:authelia.example.com
- traefik.port=80
- 'traefik.frontend.auth.forward.address=http://authelia:9091/api/verify?rd=https://authelia.example.com/'
- 'traefik.frontend.auth.forward.trustForwardHeader=true'
- 'traefik.frontend.auth.forward.authResponseHeaders=Remote-User,Remote-Groups,Remote-Name,Remote-Email'
# traefikv2
- "traefik.http.routers.authelia.rule=Host(`authelia.example.com`)"
- "traefik.http.routers.authelia.entrypoints=https"
- "traefik.http.services.authelia.loadbalancer.server.port=9091"
whoami-1fa: # (1)!
image: containous/whoami
networks:
- traefik_public
deploy:
labels:
# traefik
- "traefik.enable=true"
- "traefik.docker.network=traefik_public"
# traefikv1
- "traefik.frontend.rule=Host:whoami-authelia-1fa.example.com"
- traefik.port=80
- 'traefik.frontend.auth.forward.address=http://authelia:9091/api/verify?rd=https://authelia.example.com/'
- 'traefik.frontend.auth.forward.trustForwardHeader=true'
- 'traefik.frontend.auth.forward.authResponseHeaders=Remote-User,Remote-Groups,Remote-Name,Remote-Email'
# traefikv2
- "traefik.http.routers.whoami-authelia-1fa.rule=Host(`whoami-authelia-1fa.example.com`)"
- "traefik.http.routers.whoami-authelia-1fa.entrypoints=https"
- "traefik.http.routers.whoami-authelia-1fa.middlewares=authelia"
- "traefik.http.services.whoami-authelia-1fa.loadbalancer.server.port=80"
whoami-2fa: # (2)!
image: containous/whoami
networks:
- traefik_public
deploy:
labels:
# traefik
- "traefik.enable=true"
- "traefik.docker.network=traefik_public"
# traefikv1
- "traefik.frontend.rule=Host:whoami-authelia-2fa.example.com"
- traefik.port=80
- 'traefik.frontend.auth.forward.address=http://authelia:9091/api/verify?rd=https://authelia.example.com/'
- 'traefik.frontend.auth.forward.trustForwardHeader=true'
- 'traefik.frontend.auth.forward.authResponseHeaders=Remote-User,Remote-Groups,Remote-Name,Remote-Email'
# traefikv2
- "traefik.http.routers.whoami-authelia-2fa.rule=Host(`whoami-authelia-2fa.example.com`)"
- "traefik.http.routers.whoami-authelia-2fa.entrypoints=https"
- "traefik.http.routers.whoami-authelia-2fa.middlewares=authelia"
- "traefik.http.services.whoami-authelia-2fa.loadbalancer.server.port=80"
networks:
traefik_public:
external: true
```
1. Optionally used to test 1FA authentication
2. Optionally used to test 2FA authentication
!!! question "Why not just use Traefik Forward Auth?"
While [Traefik Forward Auth][tfa] is a very lightweight, minimal authentication layer, which provides OIDC-based authentication, Authelia provides more features such as multiple methods of authentication (*Hardware, OTP, Email*), advanced rules, and push notifications.
## Run Authelia
Launch the Authelia stack by running ```docker stack deploy authelia -c <path -to-docker-compose.yml>```
### Test Authelia
To test the service works successfully, try logging into Authelia itself first, as a user whose password you've setup in `/var/data/config/authelia/users_database.yml`.
You'll notice that upon successful login, you're requested to setup 2FA. If (*like me!*) you didn't configure an SMTP server, you can still setup 2FA (*TOTP or webauthn*), and the setup link email instructions should be found in `/var/data/config/authelia/notifications.txt`
Now you're ready to test 1FA and 2FA auth, against the two "whoami" services defined in the docker-compose file.
Try to access each in turn, and confirm that you're _not_ prompted for 2FA on whoami-authelia-1fa, but you _are_ prompted for 2FA on whoami-authelia-2fa! :thumbsup:
## Summary
What have we achieved? By adding a simple label to any service, we can secure any service behind our Authelia, with minimal processing / handling overhead, and benefit from the 1FA/2FA multi-layered features provided by Autheila.
!!! summary "Summary"
Created:
* [X] Authelia configured and available to provide a layer of authentication to other services deployed in the stack
### Authelia vs Keycloak
[KeyCloak][keycloak] is the "big daddy" of self-hosted authentication platforms - it has a beautiful GUI, and a very advanced and mature featureset. Like Authelia, KeyCloak can [use an LDAP server](/recipes/keycloak/authenticate-against-openldap/) as a backend, but _unlike_ Authelia, KeyCloak allows for 2-way sync between that LDAP backend, meaning KeyCloak can be used to _create_ and _update_ the LDAP entries (*Authelia's is just a one-way LDAP lookup - you'll need another tool to actually administer your LDAP database*).
[^1]: The initial inclusion of Authelia was due to the efforts of @bencey in Discord (Thanks Ben!)
--8<-- "recipe-footer.md"

97
docs/docker-swarm/design.md Normal file
View File

@@ -0,0 +1,97 @@
---
title: Design a secure, scalable Docker Swarm
description: Presenting a Docker Swarm design to create your own container-hosting platform, which is highly-available, scalable, portable, secure and automated! 💪
---
# Highly Available Docker Swarm Design
In the design described below, our "private cloud" platform is:
* **Highly-available** (_can tolerate the failure of a single component_)
* **Scalable** (_can add resource or capacity as required_)
* **Portable** (_run it on your garage server today, run it in AWS tomorrow_)
* **Secure** (_access protected with [LetsEncrypt certificates](/docker-swarm/traefik/) and optional [OIDC with 2FA](/docker-swarm/traefik-forward-auth/)_)
* **Automated** (_requires minimal care and feeding_)
## Design Decisions
### Where possible, services will be highly available.**
This means that:
* At least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure.
* [Ceph](/docker-swarm/shared-storage-ceph/) is employed for share storage, because it too can be made tolerant of a single failure.
!!! note
An exception to the 3-nodes decision is running a single-node configuration. If you only **have** one node, then obviously your swarm is only as resilient as that node. It's still a perfectly valid swarm configuration, ideal for starting your self-hosting journey. In fact, under the single-node configuration, you don't need ceph either, and you can simply use the local volume on your host for storage. You'll be able to migrate to ceph/more nodes if/when you expand.
**Where multiple solutions to a requirement exist, preference will be given to the most portable solution.**
This means that:
* Services are defined using docker-compose v3 YAML syntax
* Services are portable, meaning a particular stack could be shut down and moved to a new provider with minimal effort.
## Security
Under this design, the only inbound connections we're permitting to our docker swarm in a **minimal** configuration (*you may add custom services later, like UniFi Controller*) are:
### Network Flows
* **HTTP (TCP 80)** : Redirects to https
* **HTTPS (TCP 443)** : Serves individual docker containers via SSL-encrypted reverse proxy
### Authentication
* Where the hosted application provides a trusted level of authentication (*i.e., [NextCloud](/recipes/nextcloud/)*), or where the application requires public exposure (*i.e. [Privatebin](/recipes/privatebin/)*), no additional layer of authentication will be required.
* Where the hosted application provides inadequate (*i.e. [NZBGet](/recipes/autopirate/nzbget/)*) or no authentication (*i.e. [Gollum](/recipes/gollum/)*), a further authentication against an OAuth provider will be required.
## High availability
### Normal function
Assuming a 3-node configuration, under normal circumstances the following is illustrated:
* All 3 nodes provide shared storage via Ceph, which is provided by a docker container on each node.
* All 3 nodes participate in the Docker Swarm as managers.
* The various containers belonging to the application "stacks" deployed within Docker Swarm are automatically distributed amongst the swarm nodes.
* Persistent storage for the containers is provide via cephfs mount.
* The **traefik** service (*in swarm mode*) receives incoming requests (*on HTTP and HTTPS*), and forwards them to individual containers. Traefik knows the containers names because it's able to read the docker socket.
* All 3 nodes run keepalived, at varying priorities. Since traefik is running as a swarm service and listening on TCP 80/443, requests made to the keepalived VIP and arriving at **any** of the swarm nodes will be forwarded to the traefik container (*no matter which node it's on*), and then onto the target backend.
![HA function](../images/docker-swarm-ha-function.png){ loading=lazy }
### Node failure
In the case of a failure (or scheduled maintenance) of one of the nodes, the following is illustrated:
* The failed node no longer participates in Ceph, but the remaining nodes provide enough fault-tolerance for the cluster to operate.
* The remaining two nodes in Docker Swarm achieve a quorum and agree that the failed node is to be removed.
* The (*possibly new*) leader manager node reschedules the containers known to be running on the failed node, onto other nodes.
* The **traefik** service is either restarted or unaffected, and as the backend containers stop/start and change IP, traefik is aware and updates accordingly.
* The keepalived VIP continues to function on the remaining nodes, and docker swarm continues to forward any traffic received on TCP 80/443 to the appropriate node.
![HA function](../images/docker-swarm-node-failure.png){ loading=lazy }
### Node restore
When the failed (*or upgraded*) host is restored to service, the following is illustrated:
* Ceph regains full redundancy
* Docker Swarm managers become aware of the recovered node, and will use it for scheduling **new** containers
* Existing containers which were migrated off the node are not migrated backend
* Keepalived VIP regains full redundancy
![HA function](../images/docker-swarm-node-restore.png){ loading=lazy }
### Total cluster failure
A day after writing this, my environment suffered a fault whereby all 3 VMs were unexpectedly and simultaneously powered off.
Upon restore, docker failed to start on one of the VMs due to local disk space issue[^1]. However, the other two VMs started, established the swarm, mounted their shared storage, and started up all the containers (services) which were managed by the swarm.
In summary, although I suffered an **unplanned power outage to all of my infrastructure**, followed by a **failure of a third of my hosts**... ==all my platforms are 100% available[^1] with **absolutely no manual intervention**==.
[^1]: Since there's no impact to availability, I can fix (or just reinstall) the failed node whenever convenient.
--8<-- "recipe-footer.md"

183
docs/docker-swarm/docker-swarm-mode.md Normal file
View File

@@ -0,0 +1,183 @@
---
title: Enable Docker Swarm mode
description: For truly highly-available services with Docker containers, Docker Swarm is the simplest way to achieve redundancy, such that a single docker host could be turned off, and none of our services will be interrupted.
---
# Docker Swarm Mode
For truly highly-available services with Docker containers, we need an orchestration system. Docker Swarm (*as defined at 1.13*) is the simplest way to achieve redundancy, such that a single docker host could be turned off, and none of our services will be interrupted.
## Ingredients
!!! summary
Existing
* [X] 3 x nodes (*bare-metal or VMs*), each with:
* A mainstream Linux OS (*tested on either [CentOS](https://www.centos.org) 7+ or [Ubuntu](http://releases.ubuntu.com) 16.04+*)
* At least 2GB RAM
* At least 20GB disk space (_but it'll be tight_)
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
## Preparation
### Bash auto-completion
Add some handy bash auto-completion for docker. Without this, you'll get annoyed that you can't autocomplete ```docker stack deploy <blah> -c <blah.yml>``` commands.
```bash
cd /etc/bash_completion.d/
curl -O https://raw.githubusercontent.com/docker/cli/b75596e1e4d5295ac69b9934d1bd8aff691a0de8/contrib/completion/bash/docker
```
Install some useful bash aliases on each host
```bash
cd ~
curl -O https://raw.githubusercontent.com/funkypenguin/geek-cookbook/master/examples/scripts/gcb-aliases.sh
echo 'source ~/gcb-aliases.sh' >> ~/.bash_profile
```
## Serving
### Release the swarm!
Now, to launch a swarm. Pick a target node, and run `docker swarm init`
Yeah, that was it. Seriously. Now we have a 1-node swarm.
```bash
[root@ds1 ~]# docker swarm init
Swarm initialized: current node (b54vls3wf8xztwfz79nlkivt8) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-bsud7xnvhv4cicwi7l6c9s6l0 \
202.170.164.47:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
[root@ds1 ~]#
```
Run `docker node ls` to confirm that you have a 1-node swarm:
```bash
[root@ds1 ~]# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
b54vls3wf8xztwfz79nlkivt8 * ds1.funkypenguin.co.nz Ready Active Leader
[root@ds1 ~]#
```
Note that when you run `docker swarm init` above, the CLI output gives youe a command to run to join further nodes to my swarm. This command would join the nodes as __workers__ (*as opposed to __managers__*). Workers can easily be promoted to managers (*and demoted again*), but since we know that we want our other two nodes to be managers too, it's simpler just to add them to the swarm as managers immediately.
On the first swarm node, generate the necessary token to join another manager by running ```docker swarm join-token manager```:
```bash
[root@ds1 ~]# docker swarm join-token manager
To add a manager to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-cfm24bq2zvfkcwujwlp5zqxta \
202.170.164.47:2377
[root@ds1 ~]#
```
Run the command provided on your other nodes to join them to the swarm as managers. After addition of a node, the output of ```docker node ls``` (on either host) should reflect all the nodes:
```bash
[root@ds2 davidy]# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
b54vls3wf8xztwfz79nlkivt8 ds1.funkypenguin.co.nz Ready Active Leader
xmw49jt5a1j87a6ihul76gbgy * ds2.funkypenguin.co.nz Ready Active Reachable
[root@ds2 davidy]#
```
### Setup automated cleanup
Docker swarm doesn't do any cleanup of old images, so as you experiment with various stacks, and as updated containers are released upstream, you'll soon find yourself loosing gigabytes of disk space to old, unused images.
To address this, we'll run the "[meltwater/docker-cleanup](https://github.com/meltwater/docker-cleanup)" container on all of our nodes. The container will clean up unused images after 30 minutes.
First, create `docker-cleanup.env` (_mine is under `/var/data/config/docker-cleanup`_), and exclude container images we **know** we want to keep:
```bash
KEEP_IMAGES=traefik,keepalived,docker-mailserver
DEBUG=1
```
Then create a docker-compose.yml as per the following example:
```yaml
version: "3"
services:
docker-cleanup:
image: meltwater/docker-cleanup:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker:/var/lib/docker
networks:
- internal
deploy:
mode: global
env_file: /var/data/config/docker-cleanup/docker-cleanup.env
networks:
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.0.0/24
```
--8<-- "reference-networks.md"
Launch the cleanup stack by running ```docker stack deploy docker-cleanup -c <path-to-docker-compose.yml>```
### Setup automatic updates
If your swarm runs for a long time, you might find yourself running older container images, after newer versions have been released. If you're the sort of geek who wants to live on the edge, configure [shepherd](https://github.com/djmaze/shepherd) to auto-update your container images regularly.
Create `/var/data/config/shepherd/shepherd.env` as per the following example:
```bash
# Don't auto-update Plex or Emby (or Jellyfin), I might be watching a movie! (Customize this for the containers you _don't_ want to auto-update)
BLACKLIST_SERVICES="plex_plex emby_emby jellyfin_jellyfin"
# Run every 24 hours. Note that SLEEP_TIME appears to be in seconds.
SLEEP_TIME=86400
```
Then create /var/data/config/shepherd/shepherd.yml as per the following example:
```yaml
version: "3"
services:
shepherd-app:
image: mazzolino/shepherd
env_file : /var/data/config/shepherd/shepherd.env
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
labels:
- "traefik.enable=false"
deploy:
placement:
constraints: [node.role == manager]
```
Launch shepherd by running ```docker stack deploy shepherd -c /var/data/config/shepherd/shepherd.yml```, and then just forget about it, comfortable in the knowledge that every day, Shepherd will check that your images are the latest available, and if not, will destroy and recreate the container on the latest available image.
## Summary
--8<-- "5-min-install.md"
What have we achieved?
!!! summary "Summary"
Created:
* [X] [Docker swarm cluster](/docker-swarm/design/)
--8<-- "recipe-footer.md"

29
docs/docker-swarm/index.md Normal file
View File

@@ -0,0 +1,29 @@
---
title: Why use Docker Swarm?
description: Using Docker Swarm to build your own container-hosting platform which is highly-available, scalable, portable, secure and automated! 💪
---
# Why Docker Swarm?
Pop quiz, hotshot.. There's a server with containers on it. Once you run enough containers, you start to loose track of compose files / data. If the host fails, all your services are unavailable. What do you do? **WHAT DO YOU DO**?[^1]
<iframe width="560" height="315" src="https://www.youtube.com/embed/Ug2hLQv6WeY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
You too, action-geek, can save the day, by...
1. Enable [Docker Swarm mode](/docker-swarm/docker-swarm-mode/) (*even just on one node*)[^2]
2. Store your swarm configuration and application data in an [orderly and consistent structure](/reference/data_layout/)
3. Expose all your services consistently using [Traefik](/docker-swarm/traefik/) with optional [additional per-service authentication][tfa]
Then you can really level-up your geek-fu, by:
4. Making your Docker Swarm highly with [keepalived](/docker-swarm/keepalived/)
5. Setup [shared storage](/docker-swarm/shared-storage-ceph/) to eliminate SPOFs
6. [Backup](/recipes/duplicity/) your stuff automatically
Ready to enter the matrix? Jump in on one of the links above, or start reading the [design](/docker-swarm/design/)
--8<-- "recipe-footer.md"
[^1]: This was an [iconic movie](https://www.imdb.com/title/tt0111257/). It even won 2 Oscars! (*but not for the acting*)
[^2]: There are significant advantages to using Docker Swarm, even on just a single node.

91
docs/docker-swarm/keepalived.md Normal file
View File

@@ -0,0 +1,91 @@
---
title: Make docker swarm HA with keepalived
description: While having a self-healing, scalable docker swarm is great for availability and scalability, none of that is worth a sausage if nobody can connect to your cluster!
---
# Keepalived
While having a self-healing, scalable docker swarm is great for availability and scalability, none of that is worth a sausage if nobody can connect to your cluster!
In order to provide seamless external access to clustered resources, regardless of which node they're on and tolerant of node failure, you need to present a single IP to the world for external access.
Normally this is done using a HA loadbalancer, but since Docker Swarm aready provides the load-balancing capabilities (*[routing mesh](https://docs.docker.com/engine/swarm/ingress/)*), all we need for seamless HA is a virtual IP which will be provided by more than one docker node.
This is accomplished with the use of keepalived on at least two nodes.
![Ceph Screenshot](../images/keepalived.png){ loading=lazy }
## Ingredients
!!! summary "Ingredients"
Already deployed:
* [X] At least 2 x swarm nodes
* [X] low-latency link (i.e., no WAN links)
New:
* [ ] At least 3 x IPv4 addresses (*one for each node and one for the virtual IP[^1])
## Preparation
### Enable IPVS module
On all nodes which will participate in keepalived, we need the "ip_vs" kernel module, in order to permit services to bind to non-local interface addresses.
Set this up once-off for both the primary and secondary nodes, by running:
```bash
echo "modprobe ip_vs" >> /etc/modules
modprobe ip_vs
```
### Setup nodes
Assuming your IPs are as per the following example:
- 192.168.4.1 : Primary
- 192.168.4.2 : Secondary
- 192.168.4.3 : Virtual
Run the following on the primary
```bash
docker run -d --name keepalived --restart=always \
--cap-add=NET_ADMIN --cap-add=NET_BROADCAST --cap-add=NET_RAW --net=host \
-e KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:['192.168.4.1', '192.168.4.2']" \
-e KEEPALIVED_VIRTUAL_IPS=192.168.4.3 \
-e KEEPALIVED_PRIORITY=200 \
osixia/keepalived:2.0.20
```
And on the secondary[^2]:
```bash
docker run -d --name keepalived --restart=always \
--cap-add=NET_ADMIN --cap-add=NET_BROADCAST --cap-add=NET_RAW --net=host \
-e KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:['192.168.4.1', '192.168.4.2']" \
-e KEEPALIVED_VIRTUAL_IPS=192.168.4.3 \
-e KEEPALIVED_PRIORITY=100 \
osixia/keepalived:2.0.20
```
## Serving
That's it. Each node will talk to the other via unicast (*no need to un-firewall multicast addresses*), and the node with the highest priority gets to be the master. When ingress traffic arrives on the master node via the VIP, docker's routing mesh will deliver it to the appropriate docker node.
## Summary
What have we achieved?
!!! summary "Summary"
Created:
* [X] A Virtual IP to which all cluster traffic can be forwarded externally, making it "*Highly Available*"
--8<-- "5-min-install.md"
[^1]: Some hosting platforms (*OpenStack, for one*) won't allow you to simply "claim" a virtual IP. Each node is only able to receive traffic targetted to its unique IP, unless certain security controls are disabled by the cloud administrator. In this case, keepalived is not the right solution, and a platform-specific load-balancing solution should be used. In OpenStack, this is Neutron's "Load Balancer As A Service" (LBAAS) component. AWS, GCP and Azure would likely include similar protections.
[^2]: More than 2 nodes can participate in keepalived. Simply ensure that each node has the appropriate priority set, and the node with the highest priority will become the master.
--8<-- "recipe-footer.md"

80
docs/docker-swarm/nodes.md Normal file
View File

@@ -0,0 +1,80 @@
---
title: Setup nodes for docker-swarm
description: Let's start building our cluster. You can use either bare-metal machines or virtual machines - the configuration would be the same. To avoid confusion, I'll be referring to these as "nodes" from now on.
---
# Nodes
Let's start building our cluster. You can use either bare-metal machines or virtual machines - the configuration would be the same. To avoid confusion, I'll be referring to these as "nodes" from now on.
!!! note
In 2017, I **initially** chose the "[Atomic](https://www.projectatomic.io/)" CentOS/Fedora image for the swarm hosts, but later found its outdated version of Docker to be problematic with advanced features like GPU transcoding (in [Plex](/recipes/plex/)), [Swarmprom](/recipes/swarmprom/), etc. In the end, I went mainstream and simply preferred a modern Ubuntu installation.
## Ingredients
!!! summary "Ingredients"
New in this recipe:
* [ ] 3 x nodes (*bare-metal or VMs*), each with:
* A mainstream Linux OS (*tested on either [CentOS](https://www.centos.org) 7+ or [Ubuntu](http://releases.ubuntu.com) 16.04+*)
* At least 2GB RAM
* At least 20GB disk space (_but it'll be tight_)
* [ ] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
## Preparation
### Permit connectivity
Most modern Linux distributions include firewall rules which only only permit minimal required incoming connections (like SSH). We'll want to allow all traffic between our nodes. The steps to achieve this in CentOS/Ubuntu are a little different...
#### CentOS
Add something like this to `/etc/sysconfig/iptables`:
```bash
# Allow all inter-node communication
-A INPUT -s 192.168.31.0/24 -j ACCEPT
```
And restart iptables with ```systemctl restart iptables```
#### Ubuntu
Install the (*non-default*) persistent iptables tools, by running `apt-get install iptables-persistent`, establishing some default rules (*dkpg will prompt you to save current ruleset*), and then add something like this to `/etc/iptables/rules.v4`:
```bash
# Allow all inter-node communication
-A INPUT -s 192.168.31.0/24 -j ACCEPT
```
And refresh your running iptables rules with `iptables-restore < /etc/iptables/rules.v4`
### Enable hostname resolution
Depending on your hosting environment, you may have DNS automatically setup for your VMs. If not, it's useful to set up static entries in /etc/hosts for the nodes. For example, I setup the following:
- 192.168.31.11 ds1 ds1.funkypenguin.co.nz
- 192.168.31.12 ds2 ds2.funkypenguin.co.nz
- 192.168.31.13 ds3 ds3.funkypenguin.co.nz
### Set timezone
Set your local timezone, by running:
```bash
ln -sf /usr/share/zoneinfo/<your timezone> /etc/localtime
```
## Serving
After completing the above, you should have:
!!! summary "Summary"
Deployed in this recipe:
* [X] 3 x nodes (*bare-metal or VMs*), each with:
* A mainstream Linux OS (*tested on either [CentOS](https://www.centos.org) 7+ or [Ubuntu](http://releases.ubuntu.com) 16.04+*)
* At least 2GB RAM
* At least 20GB disk space (_but it'll be tight_)
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
--8<-- "recipe-footer.md"

113
docs/docker-swarm/registry.md Normal file
View File

@@ -0,0 +1,113 @@
---
title: Setup pull through Docker registry / cache
description: You may not _want_ your cluster to be pulling multiple copies of images from public registries, especially if rate-limits (hello, Docker Hub!) are a concern. Here's how you setup your own "pull through cache" registry.
---
# Create Docker "pull through" registry cache
Although we now have shared storage for our persistent container data, our docker nodes don't share any other docker data, such as container images. This results in an inefficiency - every node which participates in the swarm will, at some point, need the docker image for every container deployed in the swarm.
When dealing with large container (looking at you, GitLab!), this can result in several gigabytes of wasted bandwidth per-node, and long delays when restarting containers on an alternate node. (_It also wastes disk space on each node, but we'll get to that in the next section_)
The solution is to run an official Docker registry container as a ["pull-through" cache, or "registry mirror"](https://docs.docker.com/registry/recipes/mirror/). By using our persistent storage for the registry cache, we can ensure we have a single copy of all the containers we've pulled at least once. After the first pull, any subsequent pulls from our nodes will use the cached version from our registry mirror. As a result, services are available more quickly when restarting container nodes, and we can be more aggressive about cleaning up unused containers on our nodes (*more later*)
The registry mirror runs as a swarm stack, using a simple docker-compose.yml. Customize **your mirror FQDN** below, so that Traefik will generate the appropriate LetsEncrypt certificates for it, and make it available via HTTPS.
## Requirements
!!! summary "Ingredients"
* [ ] [Docker swarm cluster](/docker-swarm/design/) with [persistent shared storage](/docker-swarm/shared-storage-ceph/)
* [ ] [Traefik](/docker-swarm/traefik/) configured per design
* [ ] DNS entry for the hostname you intend to use, pointed to your [keepalived](/docker-swarm/keepalived/) IP
## Configuration
Create `/var/data/config/registry/registry.yml` as per the following docker-compose example:
```yaml
version: "3"
services:
registry-mirror:
image: registry:2
networks:
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:<your mirror FQDN>
- traefik.docker.network=traefik_public
- traefik.port=5000
ports:
- 5000:5000
volumes:
- /var/data/registry/registry-mirror-data:/var/lib/registry
- /var/data/registry/registry-mirror-config.yml:/etc/docker/registry/config.yml
networks:
traefik_public:
external: true
```
!!! note "Unencrypted registry"
We create this registry without consideration for SSL, which will fail if we attempt to use the registry directly. However, we're going to use the HTTPS-proxied version via [Traefik][traefik], leveraging Traefik to manage the LetsEncrypt certificates required.
Create the configuration for the actual registry in `/var/data/registry/registry-mirror-config.yml` as per the following example:
```yaml
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
delete:
enabled: true
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
proxy:
remoteurl: https://registry-1.docker.io
```
## Running
### Launch Docker registry stack
Launch the registry stack by running `docker stack deploy registry -c <path-to-docker-compose.yml>`
### Enable Docker registry mirror
To tell docker to use the registry mirror, edit `/etc/docker-latest/daemon.json` [^1] on each node, and change from:
```json
{
"log-driver": "journald",
"signature-verification": false
}
```
To:
```json
{
"log-driver": "journald",
"signature-verification": false,
"registry-mirrors": ["https://<your registry mirror FQDN>"]
}
```
Then restart docker itself, by running `systemctl restart docker`
[^1]: Note the extra comma required after "false" above
--8<-- "recipe-footer.md"

230
docs/docker-swarm/shared-storage-ceph.md Normal file
View File

@@ -0,0 +1,230 @@
---
title: Ceph cluster in Docker Swarm
description: Ceph provides persistent storage to your Docker Swarm cluster, supporting either rdb images for host volume mounts, or even fancy cephfs docker volumes.
---
# Shared Storage (Ceph)
While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node.
![Ceph Screenshot](/images/ceph.png){ loading=lazy }
## Ingredients
!!! summary "Ingredients"
3 x Virtual Machines (configured earlier), each with:
* [X] Support for "modern" versions of Python and LVM
* [X] At least 1GB RAM
* [X] At least 20GB disk space (_but it'll be tight_)
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
* [X] A second disk dedicated to the Ceph OSD
* [X] Each node should have the IP of every other participating node hard-coded in /etc/hosts (*including its own IP*)
## Preparation
!!! tip "No more [foolish games](https://www.youtube.com/watch?v=UNoouLa7uxA)"
Earlier iterations of this recipe (*based on [Ceph Jewel](https://docs.ceph.com/docs/master/releases/jewel/)*) required significant manual effort to install Ceph in a Docker environment. In the 2+ years since Jewel was released, significant improvements have been made to the ceph "deploy-in-docker" process, including the [introduction of the cephadm tool](https://ceph.io/ceph-management/introducing-cephadm/). Cephadm is the tool which now does all the heavy lifting, below, for the current version of ceph, codenamed "[Octopus](https://www.youtube.com/watch?v=Gi58pN8W3hY)".
### Pick a master node
One of your nodes will become the cephadm "master" node. Although all nodes will participate in the Ceph cluster, the master node will be the node which we bootstrap ceph on. It's also the node which will run the Ceph dashboard, and on which future upgrades will be processed. It doesn't matter _which_ node you pick, and the cluster itself will operate in the event of a loss of the master node (although you won't see the dashboard)
### Install cephadm on master node
Run the following on the ==master== node:
```bash
MYIP=`ip route get 1.1.1.1 | grep -oP 'src \K\S+'`
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod +x cephadm
mkdir -p /etc/ceph
./cephadm bootstrap --mon-ip $MYIP
```
The process takes about 30 seconds, after which, you'll have a MVC (*Minimum Viable Cluster*)[^1], encompassing a single monitor and mgr instance on your chosen node. Here's the complete output from a fresh install:
[^1]: Minimum Viable Cluster acronym copyright, trademark, and whatever else, to Funky Penguin for 1,000,000 years.
??? "Example output from a fresh cephadm bootstrap"
```
root@raphael:~# MYIP=`ip route get 1.1.1.1 | grep -oP 'src \K\S+'`
root@raphael:~# curl --silent --remote-name --location <https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm>
root@raphael:~# chmod +x cephadm
root@raphael:~# mkdir -p /etc/ceph
root@raphael:~# ./cephadm bootstrap --mon-ip $MYIP
INFO:cephadm:Verifying podman|docker is present...
INFO:cephadm:Verifying lvm2 is present...
INFO:cephadm:Verifying time synchronization is in place...
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
INFO:cephadm:Repeating the final host check...
INFO:cephadm:podman|docker (/usr/bin/docker) is present
INFO:cephadm:systemctl is present
INFO:cephadm:lvcreate is present
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
INFO:cephadm:Host looks OK
INFO:root:Cluster fsid: bf3eff78-9e27-11ea-b40a-525400380101
INFO:cephadm:Verifying IP 192.168.38.101 port 3300 ...
INFO:cephadm:Verifying IP 192.168.38.101 port 6789 ...
INFO:cephadm:Mon IP 192.168.38.101 is in CIDR network 192.168.38.0/24
INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container...
INFO:cephadm:Extracting ceph user uid/gid from container image...
INFO:cephadm:Creating initial keys...
INFO:cephadm:Creating initial monmap...
INFO:cephadm:Creating mon...
INFO:cephadm:Waiting for mon to start...
INFO:cephadm:Waiting for mon...
INFO:cephadm:mon is available
INFO:cephadm:Assimilating anything we can from ceph.conf...
INFO:cephadm:Generating new minimal ceph.conf...
INFO:cephadm:Restarting the monitor...
INFO:cephadm:Setting mon public_network...
INFO:cephadm:Creating mgr...
INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring
INFO:cephadm:Wrote config to /etc/ceph/ceph.conf
INFO:cephadm:Waiting for mgr to start...
INFO:cephadm:Waiting for mgr...
INFO:cephadm:mgr not available, waiting (1/10)...
INFO:cephadm:mgr not available, waiting (2/10)...
INFO:cephadm:mgr not available, waiting (3/10)...
INFO:cephadm:mgr is available
INFO:cephadm:Enabling cephadm module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 5...
INFO:cephadm:Mgr epoch 5 is available
INFO:cephadm:Setting orchestrator backend to cephadm...
INFO:cephadm:Generating ssh key...
INFO:cephadm:Wrote public SSH key to to /etc/ceph/ceph.pub
INFO:cephadm:Adding key to root@localhost's authorized_keys...
INFO:cephadm:Adding host raphael...
INFO:cephadm:Deploying mon service with default placement...
INFO:cephadm:Deploying mgr service with default placement...
INFO:cephadm:Deploying crash service with default placement...
INFO:cephadm:Enabling mgr prometheus module...
INFO:cephadm:Deploying prometheus service with default placement...
INFO:cephadm:Deploying grafana service with default placement...
INFO:cephadm:Deploying node-exporter service with default placement...
INFO:cephadm:Deploying alertmanager service with default placement...
INFO:cephadm:Enabling the dashboard module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 13...
INFO:cephadm:Mgr epoch 13 is available
INFO:cephadm:Generating a dashboard self-signed certificate...
INFO:cephadm:Creating initial admin user...
INFO:cephadm:Fetching dashboard port number...
INFO:cephadm:Ceph Dashboard is now available at:
URL: https://raphael:8443/
User: admin
Password: mid28k0yg5
INFO:cephadm:You can access the Ceph CLI with:
sudo ./cephadm shell --fsid bf3eff78-9e27-11ea-b40a-525400380101 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
INFO:cephadm:Please consider enabling telemetry to help improve Ceph:
ceph telemetry on
For more information see:
https://docs.ceph.com/docs/master/mgr/telemetry/
INFO:cephadm:Bootstrap complete.
root@raphael:~#
```
### Prepare other nodes
It's now necessary to tranfer the following files to your ==other== nodes, so that cephadm can add them to your cluster, and so that they'll be able to mount the cephfs when we're done:
| Path on master | Path on non-master |
|---------------------------------------|------------------------------------------------------------|
| `/etc/ceph/ceph.conf` | `/etc/ceph/ceph.conf` |
| `/etc/ceph/ceph.client.admin.keyring` | `/etc/ceph/ceph.client.admin.keyring` |
| `/etc/ceph/ceph.pub` | `/root/.ssh/authorized_keys` (append to anything existing) |
Back on the ==master== node, run `ceph orch host add <node-name>` once for each other node you want to join to the cluster. You can validate the results by running `ceph orch host ls`
!!! question "Should we be concerned about giving cephadm using root access over SSH?"
Not really. Docker is inherently insecure at the host-level anyway (*think what would happen if you launched a global-mode stack with a malicious container image which mounted `/root/.ssh`*), so worrying about cephadm seems a little barn-door-after-horses-bolted. If you take host-level security seriously, consider switching to [Kubernetes](/kubernetes/) :)
### Add OSDs
Now the best improvement since the days of ceph-deploy and manual disks.. on the ==master== node, run `ceph orch apply osd --all-available-devices`. This will identify any unloved (*unpartitioned, unmounted*) disks attached to each participating node, and configure these disks as OSDs.
### Setup CephFS
On the ==master== node, create a cephfs volume in your cluster, by running `ceph fs volume create data`. Ceph will handle the necessary orchestration itself, creating the necessary pool, mds daemon, etc.
You can watch the progress by running `ceph fs ls` (to see the fs is configured), and `ceph -s` to wait for `HEALTH_OK`
### Mount CephFS volume
On ==every== node, create a mountpoint for the data, by running ```mkdir /var/data```, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually _mounted_ if there's a network / boot delay getting access to the gluster volume:
```bash
mkdir /var/data
MYNODES="<node1>,<node2>,<node3>" # Add your own nodes here, comma-delimited
MYHOST=`ip route get 1.1.1.1 | grep -oP 'src \K\S+'`
echo -e "
# Mount cephfs volume \n
raphael,donatello,leonardo:/ /var/data ceph name=admin,noatime,_netdev 0 0" >> /etc/fstab
mount -a
```
??? note "Additional steps on Debian Buster"
The above configuration worked on Ubuntu 18.04 **without** requiring a secret to be defined in `/etc/fstab`. Other users have [reported different results](https://forum.funkypenguin.co.nz/t/shared-storage-ceph-funky-penguins-geek-cookbook/47/108) on Debian Buster, however, so consider trying this variation if you encounter error 22:
```
apt-get install ceph-common
CEPHKEY=`sudo ceph-authtool -p /etc/ceph/ceph.client.admin.keyring`
echo -e "
# Mount cephfs volume \n
raphael,donatello,leonardo:/ /var/data ceph name=admin,secret=$CEPHKEY,noatime,_netdev 0 0" >> /etc/fstab
mount -a
```
## Serving
### Sprinkle with tools
Although it's possible to use `cephadm shell` to exec into a container with the necessary ceph tools, it's more convenient to use the native CLI tools. To this end, on each node, run the following, which will install the appropriate apt repository, and install the latest ceph CLI tools:
```bash
curl -L https://download.ceph.com/keys/release.asc | sudo apt-key add -
cephadm add-repo --release octopus
cephadm install ceph-common
```
### Drool over dashboard
Ceph now includes a comprehensive dashboard, provided by the mgr daemon. The dashboard will be accessible at `https://[IP of your ceph master node]:8443`, but you'll need to run `ceph dashboard ac-user-create <username> <password> administrator` first, to create an administrator account:
```bash
root@raphael:~# ceph dashboard ac-user-create batman supermansucks administrator
{"username": "batman", "password": "$2b$12$3HkjY85mav.dq3HHAZiWP.KkMiuoV2TURZFH.6WFfo/BPZCT/0gr.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1590372281, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": false}
root@raphael:~#
```
## Summary
What have we achieved?
!!! summary "Summary"
Created:
* [X] Persistent storage available to every node
* [X] Resiliency in the event of the failure of a single node
* [X] Beautiful dashboard
--8<-- "5-min-install.md"
Here's a screencast of the playbook in action. I sped up the boring parts, it actually takes ==5 min== (*you can tell by the timestamps on the prompt*):
![Screencast of ceph install via ansible](https://static.funkypenguin.co.nz/ceph_install_via_ansible_playbook.gif)
[patreon]: <https://www.patreon.com/bePatron?u=6982506>
[github_sponsor]: <https://github.com/sponsors/funkypenguin>
--8<-- "recipe-footer.md"

175
docs/docker-swarm/shared-storage-gluster.md Normal file
View File

@@ -0,0 +1,175 @@
---
title: GlusterFS vs Ceph (the winner)
description: Here's why Ceph was the obvious winner in the ceph vs glusterfs comparison for our docker-swarm cluster.
---
# Shared Storage (GlusterFS)
While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node.
!!! warning
This recipe is deprecated. It didn't work well in 2017, and it's not likely to work any better now. It remains here as a reference. I now recommend the use of [Ceph for shared storage](/docker-swarm/shared-storage-ceph/) instead. - 2019 Chef
## Design
### Why GlusterFS?
This GlusterFS recipe was my original design for shared storage, but I [found it to be flawed](/docker-swarm/shared-storage-ceph/#why-not-glusterfs), and I replaced it with a [design which employs Ceph instead](/docker-swarm/shared-storage-ceph/#why-ceph). This recipe is an alternate to the Ceph design, if you happen to prefer GlusterFS.
## Ingredients
!!! summary "Ingredients"
3 x Virtual Machines (configured earlier), each with:
* [X] CentOS/Fedora Atomic
* [X] At least 1GB RAM
* [X] At least 20GB disk space (_but it'll be tight_)
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
* [ ] A second disk, or adequate space on the primary disk for a dedicated data partition
## Preparation
### Create Gluster "bricks"
To build our Gluster volume, we need 2 out of the 3 VMs to provide one "brick". The bricks will be used to create the replicated volume. Assuming a replica count of 2 (_i.e., 2 copies of the data are kept in gluster_), our total number of bricks must be divisible by our replica count. (_I.e., you can't have 3 bricks if you want 2 replicas. You can have 4 though - We have to have minimum 3 swarm manager nodes for fault-tolerance, but only 2 of those nodes need to run as gluster servers._)
On each host, run a variation following to create your bricks, adjusted for the path to your disk.
!!! note "The example below assumes /dev/vdb is dedicated to the gluster volume"
```bash
(
echo o # Create a new empty DOS partition table
echo n # Add a new partition
echo p # Primary partition
echo 1 # Partition number
echo # First sector (Accept default: 1)
echo # Last sector (Accept default: varies)
echo w # Write changes
) | sudo fdisk /dev/vdb
mkfs.xfs -i size=512 /dev/vdb1
mkdir -p /var/no-direct-write-here/brick1
echo '' >> /etc/fstab >> /etc/fstab
echo '# Mount /dev/vdb1 so that it can be used as a glusterfs volume' >> /etc/fstab
echo '/dev/vdb1 /var/no-direct-write-here/brick1 xfs defaults 1 2' >> /etc/fstab
mount -a && mount
```
!!! warning "Don't provision all your LVM space"
Atomic uses LVM to store docker data, and **automatically grows** Docker's volumes as requried. If you commit all your free LVM space to your brick, you'll quickly find (as I did) that docker will start to fail with error messages about insufficient space. If you're going to slice off a portion of your LVM space in /dev/atomicos, make sure you leave enough space for Docker storage, where "enough" depends on how much you plan to pull images, make volumes, etc. I ate through 20GB very quickly doing development, so I ended up provisioning 50GB for atomic alone, with a separate volume for the brick.
### Create glusterfs container
Atomic doesn't include the Gluster server components. This means we'll have to run glusterd from within a container, with privileged access to the host. Although convoluted, I've come to prefer this design since it once again makes the OS "disposable", moving all the config into containers and code.
Run the following on each host:
````bash
docker run \
-h glusterfs-server \
-v /etc/glusterfs:/etc/glusterfs:z \
-v /var/lib/glusterd:/var/lib/glusterd:z \
-v /var/log/glusterfs:/var/log/glusterfs:z \
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-v /var/no-direct-write-here/brick1:/var/no-direct-write-here/brick1 \
-d --privileged=true --net=host \
--restart=always \
--name="glusterfs-server" \
gluster/gluster-centos
````
### Create trusted pool
On a single node (doesn't matter which), run ```docker exec -it glusterfs-server bash``` to launch a shell inside the container.
From the node, run `gluster peer probe <other host>`.
Example output:
```bash
[root@glusterfs-server /]# gluster peer probe ds1
peer probe: success.
[root@glusterfs-server /]#
```
Run ```gluster peer status``` on both nodes to confirm that they're properly connected to each other:
Example output:
```bash
[root@glusterfs-server /]# gluster peer status
Number of Peers: 1
Hostname: ds3
Uuid: 3e115ba9-6a4f-48dd-87d7-e843170ff499
State: Peer in Cluster (Connected)
[root@glusterfs-server /]#
```
### Create gluster volume
Now we create a *replicated volume* out of our individual "bricks".
Create the gluster volume by running:
```bash
gluster volume create gv0 replica 2 \
server1:/var/no-direct-write-here/brick1 \
server2:/var/no-direct-write-here/brick1
```
Example output:
```bash
[root@glusterfs-server /]# gluster volume create gv0 replica 2 ds1:/var/no-direct-write-here/brick1/gv0 ds3:/var/no-direct-write-here/brick1/gv0
volume create: gv0: success: please start the volume to access data
[root@glusterfs-server /]#
```
Start the volume by running ```gluster volume start gv0```
```bash
[root@glusterfs-server /]# gluster volume start gv0
volume start: gv0: success
[root@glusterfs-server /]#
```
The volume is only present on the host you're shelled into though. To add the other hosts to the volume, run ```gluster peer probe <servername>```. Don't probe host from itself.
From one other host, run ```docker exec -it glusterfs-server bash``` to shell into the gluster-server container, and run ```gluster peer probe <original server name>``` to update the name of the host which started the volume.
### Mount gluster volume
On the host (i.e., outside of the container - type ```exit``` if you're still shelled in), create a mountpoint for the data, by running ```mkdir /var/data```, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually _mounted_ if there's a network / boot delay getting access to the gluster volume:
```bash
mkdir /var/data
MYHOST=`hostname -s`
echo '' >> /etc/fstab >> /etc/fstab
echo '# Mount glusterfs volume' >> /etc/fstab
echo "$MYHOST:/gv0 /var/data glusterfs defaults,_netdev,context="system_u:object_r:svirt_sandbox_file_t:s0" 0 0" >> /etc/fstab
mount -a
```
For some reason, my nodes won't auto-mount this volume on boot. I even tried the trickery below, but they stubbornly refuse to automount:
```bash
echo -e "\n\n# Give GlusterFS 10s to start before \
mounting\nsleep 10s && mount -a" >> /etc/rc.local
systemctl enable rc-local.service
```
For non-gluster nodes, you'll need to replace $MYHOST above with the name of one of the gluster hosts (I haven't worked out how to make this fully HA yet)
## Serving
After completing the above, you should have:
* [X] Persistent storage available to every node
* [X] Resiliency in the event of the failure of a single (gluster) node
[^1]: Future enhancements to this recipe include:
1. Migration of shared storage from GlusterFS to Ceph
2. Correct the fact that volumes don't automount on boot
--8<-- "recipe-footer.md"

206
docs/docker-swarm/traefik-forward-auth/dex-static.md Normal file
View File

@@ -0,0 +1,206 @@
---
title: SSO with traefik forward auth and Dex
description: Traefik forward auth needs an authentication backend, but if you don't want to use a cloud provider, you can setup your own simple OIDC backend, using Dex.
---
# Traefik Forward Auth for SSO with Dex (Static)
[Traefik Forward Auth](/docker-swarm/traefik-forward-auth/) is incredibly useful to secure services with an additional layer of authentication, provided by an OIDC-compatible provider. The simplest possible provider is a self-hosted instance of [CoreOS's Dex](https://github.com/dexidp/dex), configured with a static username and password. This recipe will "get you started" with Traefik Forward Auth, providing a basic authentication layer. In time, you might want to migrate to a "public" provider, like [Google][tfa-google], or GitHub, or to a [Keycloak][keycloak] installation.
--8<-- "recipe-tfa-ingredients.md"
## Preparation
### Setup dex config
Create `/var/data/config/dex/config.yml` something like the following (*this is a bare-bones, [minimal example](https://github.com/dexidp/dex/blob/master/config.dev.yaml)*). At the very least, you want to replace all occurances of `example.com` with your own domain name. (*If you change nothing else, your ID is `foo`, your secret is `bar`, your username is `admin@yourdomain`, and your password is `password`*):
```yaml
# The base path of dex and the external name of the OpenID Connect service.
#
# This is the canonical URL that all clients MUST use to refer to dex. If a
# path is provided, dex's HTTP service will listen at a non-root URL.
issuer: https://dex.example.com
storage:
type: sqlite3
config:
file: var/sqlite/dex.db
web:
http: 0.0.0.0:5556
oauth2:
skipApprovalScreen: true
staticClients:
- id: foo
redirectURIs:
- 'https://auth.example.com/_oauth'
name: 'example.com'
secret: bar
enablePasswordDB: true
staticPasswords:
- email: "admin@example.com"
# bcrypt hash of the string "password"
hash: "$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"
username: "admin"
userID: "08a8684b-db88-4b73-90a9-3cd1661f5466"
```
### Prepare Traefik Forward Auth environment
Create `/var/data/config/traefik-forward-auth/traefik-forward-auth.env` per the following example configuration:
```bash
DEFAULT_PROVIDER: oidc
PROVIDERS_OIDC_CLIENT_ID: foo # This is the staticClients.id value in config.yml above
PROVIDERS_OIDC_CLIENT_SECRET: bar # This is the staticClients.secret value in config.yml above
PROVIDERS_OIDC_ISSUER_URL: https://dex.example.com # This is the issuer value in config.yml above, and it has to be reachable via a browser
SECRET: imtoosexyformyshorts # Make this up. It's not configured anywhere else
AUTH_HOST: auth.example.com # This should match the value of the traefik hosts labels in Traefik Forward Auth
COOKIE_DOMAIN: example.com # This should match your base domain
```
### Setup Docker Stack for Dex
Now create a docker swarm config file in docker-compose syntax (v3), per the following example:
```yaml
version: '3'
services:
dex:
image: dexidp/dex
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/data/config/dex/config.yml:/config.yml:ro
networks:
- traefik_public
command: ['serve','/config.yml']
deploy:
labels:
# traefik
- traefik.enable=true
- traefik.docker.network=traefik_public
# traefikv1
- traefik.frontend.rule=Host:dex.example.com
- traefik.port=5556
- traefik.docker.network=traefik_public
# and for traefikv2:
- "traefik.http.routers.dex.rule=Host(`dex.example.com`)"
- "traefik.http.routers.dex.entrypoints=https"
- "traefik.http.services.dex.loadbalancer.server.port=5556"
networks:
traefik_public:
external: true
```
--8<-- "premix-cta.md"
### Setup Docker Stack for Traefik Forward Auth
Now create a docker swarm config file for traefik-forward-auth, in docker-compose syntax (v3), per the following example:
```yaml
version: "3.2"
services:
traefik-forward-auth:
image: thomseddon/traefik-forward-auth:2.2.0
env_file: /var/data/config/traefik-forward-auth/traefik-forward-auth.env
volumes:
- /var/data/config/traefik-forward-auth/config.ini:/config.ini:ro
networks:
- traefik_public
deploy:
labels:
# traefikv1
- "traefik.port=4181"
- "traefik.frontend.rule=Host:auth.example.com"
- "traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181"
- "traefik.frontend.auth.forward.trustForwardHeader=true"
# traefikv2
- "traefik.docker.network=traefik_public"
- "traefik.http.routers.auth.rule=Host(`auth.example.com`)"
- "traefik.http.routers.auth.entrypoints=https"
- "traefik.http.routers.auth.tls=true"
- "traefik.http.routers.auth.tls.domains[0].main=example.com"
- "traefik.http.routers.auth.tls.domains[0].sans=*.example.com"
- "traefik.http.routers.auth.tls.certresolver=main"
- "traefik.http.routers.auth.service=auth@docker"
- "traefik.http.services.auth.loadbalancer.server.port=4181"
- "traefik.http.middlewares.forward-auth.forwardauth.address=http://traefik-forward-auth:4181"
- "traefik.http.middlewares.forward-auth.forwardauth.trustForwardHeader=true"
- "traefik.http.middlewares.forward-auth.forwardauth.authResponseHeaders=X-Forwarded-User"
- "traefik.http.routers.auth.middlewares=forward-auth"
# This simply validates that traefik forward authentication is working
whoami:
image: containous/whoami
networks:
- traefik_public
deploy:
labels:
# traefik
- "traefik.enable=true"
- "traefik.docker.network=traefik_public"
# traefikv1
- "traefik.frontend.rule=Host:whoami.example.com"
- "traefik.http.services.whoami.loadbalancer.server.port=80"
- "traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181"
- "traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User"
- "traefik.frontend.auth.forward.trustForwardHeader=true"
# traefikv2
- "traefik.http.routers.whoami.rule=Host(`whoami.example.com`)"
- "traefik.http.routers.whoami.entrypoints=https"
- "traefik.http.services.whoami.loadbalancer.server.port=80"
- "traefik.http.routers.whoami.middlewares=forward-auth"
networks:
traefik_public:
external: true
```
## Serving
### Launch
Deploy dex with `docker stack deploy dex -c /var/data/dex/dex.yml`, to launch dex, and then deploy Traefik Forward Auth with `docker stack deploy traefik-forward-auth -c /var/data/traefik-forward-auth/traefik-forward-auth.yml`
Once you redeploy traefik-forward-auth with the above, it **should** use dex as an OIDC provider, authenticating you against the `staticPasswords` username and hashed password described in `config.yml` above.
### Test
Browse to <https://whoami.example.com> (*obviously, customized for your domain and having created a DNS record*), and all going according to plan, you'll be redirected to a CoreOS Dex login. Once successfully logged in, you'll be directed to the basic whoami page :thumbsup:
### Protect services
To protect any other service, ensure the service itself is exposed by Traefik. Add the following label:
```yaml
- "traefik.http.routers.radarr.middlewares=forward-auth"
```
And re-deploy your services :)
## Summary
What have we achieved? By adding an additional label to any service, we can secure any service behind our (static) OIDC provider, with minimal processing / handling overhead.
!!! summary "Summary"
Created:
* [X] Traefik-forward-auth configured to authenticate against Dex (static)
[^1]: You can remove the `whoami` container once you know Traefik Forward Auth is working properly
--8<-- "recipe-footer.md"

134
docs/docker-swarm/traefik-forward-auth/google.md Normal file
View File

@@ -0,0 +1,134 @@
---
title: SSO with traefik forward auth with Google Oauth2
description: Using Traefik Forward Auth, you can selectively apply SSO to your Docker services, using Google Oauth2 / OIDC as your authentication backend!
---
# Traefik Forward Auth using Google Oauth2 for SSO
[Traefik Forward Auth][tfa] is incredibly useful to secure services with an additional layer of authentication, provided by an OIDC-compatible provider. The simplest possible provider is a self-hosted instance of [Dex][tfa-dex-static], configured with a static username and password. This is not much use if you want to provide "normies" access to your services though - a better solution would be to validate their credentials against an existing trusted public source.
This recipe will illustrate how to point Traefik Forward Auth to Google, confirming that the requestor has a valid Google account (*and that said account is permitted to access your services!*)
--8<-- "recipe-tfa-ingredients.md"
## Preparation
### Obtain OAuth credentials
#### TL;DR
Log into <https://console.developers.google.com/>, create a new project then search for and select "**Credentials**" in the search bar.
Fill out the "OAuth Consent Screen" tab, and then click, "**Create Credentials**" > "**OAuth client ID**". Select "**Web Application**", fill in the name of your app, skip "**Authorized JavaScript origins**" and fill "**Authorized redirect URIs**" with either all the domains you will allow authentication from, appended with the url-path (*e.g. <https://radarr.example.com/_oauth>, <https://radarr.example.com/_oauth>, etc*), or if you don't like frustration, use a "auth host" URL instead, like "*<https://auth.example.com/_oauth>*" (*see below for details*)
#### Monkey see, monkey do 🙈
Here's a [screencast I recorded](https://static.funkypenguin.co.nz/2021/screencast_2021-01-29_22-29-33.gif) of the OIDC credentias setup in Google Developer Console
!!! tip
Store your client ID and secret safely - you'll need them for the next step.
### Prepare environment
Create `/var/data/config/traefik-forward-auth/traefik-forward-auth.env` as per the following example:
```bash
PROVIDERS_GOOGLE_CLIENT_ID=<your client id>
PROVIDERS_GOOGLE_CLIENT_SECRET=<your client secret>
SECRET=<a random string, make it up>
# comment out AUTH_HOST if you'd rather use individual redirect_uris (slightly less complicated but more work)
AUTH_HOST=auth.example.com
COOKIE_DOMAINS=example.com
WHITELIST=you@yourdomain.com, me@mydomain.com
```
### Prepare the docker service config
Create `/var/data/config/traefik-forward-auth/traefik-forward-auth.yml` as per the following example:
```yaml
traefik-forward-auth:
image: thomseddon/traefik-forward-auth:2.1.0
env_file: /var/data/config/traefik-forward-auth/traefik-forward-auth.env
networks:
- traefik_public
deploy:
labels # you only need these if you're using an auth host
# traefik
- traefik.enable=true
- traefik.docker.network=traefik_public
# traefikv1
- "traefik.port=4181"
- "traefik.frontend.rule=Host:auth.example.com"
- "traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181"
- "traefik.frontend.auth.forward.trustForwardHeader=true"
# traefikv2
- "traefik.docker.network=traefik_public"
- "traefik.http.routers.auth.rule=Host(`auth.example.com`)"
- "traefik.http.routers.auth.entrypoints=https"
- "traefik.http.routers.auth.tls=true"
- "traefik.http.routers.auth.tls.domains[0].main=example.com"
- "traefik.http.routers.auth.tls.domains[0].sans=*.example.com"
- "traefik.http.routers.auth.tls.certresolver=main"
- "traefik.http.routers.auth.service=auth@docker"
- "traefik.http.services.auth.loadbalancer.server.port=4181"
- "traefik.http.middlewares.forward-auth.forwardauth.address=http://traefik-forward-auth:4181"
- "traefik.http.middlewares.forward-auth.forwardauth.trustForwardHeader=true"
- "traefik.http.middlewares.forward-auth.forwardauth.authResponseHeaders=X-Forwarded-User"
- "traefik.http.routers.auth.middlewares=forward-auth"
```
If you're not confident that forward authentication is working, add a simple "whoami" test container to the above .yml, to help debug traefik forward auth, before attempting to add it to a more complex container.
```yaml
# This simply validates that traefik forward authentication is working
whoami:
image: containous/whoami
networks:
- traefik_public
deploy:
labels:
# traefik
- traefik.enable=true
- traefik.docker.network=traefik_public
# traefikv1
- traefik.frontend.rule=Host:whoami.example.com
- "traefik.http.services.linx.loadbalancer.server.port=80"
- traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181
- traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
- traefik.frontend.auth.forward.trustForwardHeader=true
# traefikv2
- "traefik.http.routers.whoami.rule=Host(`whoami.example.com`)"
- "traefik.http.routers.whoami.entrypoints=https"
- "traefik.http.services.whoami.loadbalancer.server.port=80"
- "traefik.http.routers.whoami.middlewares=forward-auth" # this line enforces traefik-forward-auth
```
--8<-- "premix-cta.md"
## Serving
### Launch
Deploy traefik-forward-auth with ```docker stack deploy traefik-forward-auth -c /var/data/traefik-forward-auth/traefik-forward-auth.yml```
### Test
Browse to <https://whoami.example.com> (*obviously, customized for your domain and having created a DNS record*), and all going according to plan, you should be redirected to a Google login. Once successfully logged in, you'll be directed to the basic whoami page.
## Summary
What have we achieved? By adding an additional three simple labels to any service, we can secure any service behind our choice of OAuth provider, with minimal processing / handling overhead.
!!! summary "Summary"
Created:
* [X] Traefik-forward-auth configured to authenticate against an OIDC provider
[^1]: Be sure to populate `WHITELIST` in `traefik-forward-auth.env`, else you'll happily be granting **any** authenticated Google account access to your services!
--8<-- "recipe-footer.md"

57
docs/docker-swarm/traefik-forward-auth/index.md Normal file
View File

@@ -0,0 +1,57 @@
---
title: Add SSO to Traefik with Forward Auth
description: Traefik Forward Auth protects services running in Docker with an additional layer of authentication, and can be integrated into Keycloak, Google, GitHub, etc using OIDC.
---
# Traefik Forward Auth
Now that we have Traefik deployed, automatically exposing SSL access to our Docker Swarm services using LetsEncrypt wildcard certificates, let's pause to consider that we may not *want* some services exposed directly to the internet...
..Wait, why not? Well, Traefik doesn't provide any form of authentication, it simply secures the **transmission** of the service between Docker Swarm and the end user. If you were to deploy a service with no native security (*[Radarr][radarr] or [Sonarr][sonarr] come to mind*), then anybody would be able to use it! Even services which *may* have a layer of authentication **might** not be safe to expose publically - often open source projects may be maintained by enthusiasts who happily add extra features, but just pay lip service to security, on the basis that "*it's the user's problem to secure it in their own network*".
Some of the platforms we use on our swarm may have strong, proven security to prevent abuse. Techniques such as rate-limiting (*to defeat brute force attacks*) or even support 2-factor authentication (*tiny-tiny-rss or Wallabag support this)*.
Other platforms may provide **no authentication** (*Traefik's own web UI for example*), or minimal, un-proven UI authentication which may have been added as an afterthought.
Still other platforms may hold such sensitive data (*i.e., NextCloud*), that we'll feel more secure by putting an additional authentication layer in front of them.
This is the role of Traefik Forward Auth.
## How does it work?
**Normally**, Traefik proxies web requests directly to individual web apps running in containers. The user talks directly to the webapp, and the webapp is responsible for ensuring appropriate authentication.
When employing Traefik Forward Auth as "[middleware](https://doc.traefik.io/traefik/middlewares/forwardauth/)", the forward-auth process sits in the middle of this transaction - traefik receives the incoming request, "checks in" with the auth server to determine whether or not further authentication is required. If the user is authenticated, the auth server returns a 200 response code, and Traefik is authorized to forward the request to the backend. If not, traefik passes the auth server response back to the user - this process will usually direct the user to an authentication provider (*[Google][tfa-google], [Keycloak][tfa-keycloak], and [Dex][tfa-dex-static] are common examples*), so that they can perform a login.
Illustrated below:
![Traefik Forward Auth](/images/traefik-forward-auth.png){ loading=lazy }
The advantage under this design is additional security. If I'm deploying a web app which I expect only an authenticated user to require access to (*unlike something intended to be accessed publically, like [Linx][linx]*), I'll pass the request through Traefik Forward Auth. The overhead is negligible, and the additional layer of security is well-worth it.
## AuthHost mode
Under normal Oauth2 / OIDC auth, you have to tell your auth provider which URLs it may redirect an authenticated user back to, post-authentication. This is a security feture of the OIDC spec, preventing a malicious landing page from capturing your session and using it to impersonate you. When you're securing many URLs though, explicitly listing them can be a PITA.
[@thomaseddon's traefik-forward-auth](https://github.com/thomseddon/traefik-forward-auth) includes an ingenious mechanism to simulate an "_auth host_" in your OIDC authentication, so that you can protect an unlimited amount of DNS names (_with a common domain suffix_), without having to manually maintain a list.
### How does it work?
Say for example, you're protecting **radarr.example.com**. When you first browse to **<https://radarr.example.com>**, Traefik forwards your session to traefik-forward-auth, to be authenticated. Traefik-forward-auth redirects you to your OIDC provider's login, but instructs the OIDC provider to redirect a successfully authenticated session **back** to **<https://auth.example.com/_oauth>**, rather than to **<https://radarr.example.com/_oauth>**.
When you successfully authenticate against the OIDC provider, you are redirected to the "_redirect_uri_" of <https://auth.example.com>. Again, your request hits Traefik, which forwards the session to traefik-forward-auth, which **knows** that you've just been authenticated (*cookies have a role to play here*). Traefik-forward-auth also knows the URL of your **original** request (*thanks to the X-Forwarded-Host header*). Traefik-forward-auth redirects you to your original destination, and everybody is happy.
This clever workaround only works under 2 conditions:
1. Your "auth host" has the same domain name as the hosts you're protecting (*i.e., auth.example.com protecting radarr.example.com*)
2. You explictly tell traefik-forward-auth to use a cookie authenticating your **whole** domain (*i.e. example.com*)
## Authentication Providers
Traefik Forward Auth needs to authenticate an incoming user against a provider. A provider can be something as simple as a self-hosted [dex][tfa-dex-static] instance with a single static username/password, or as complex as a [Keycloak][keycloak] instance backed by [OpenLDAP][openldap]. Here are some options, in increasing order of complexity...
* [Authenticate Traefik Forward Auth against a self-hosted Dex instance with static usernames and passwords][tfa-dex-static]
* [Authenticate Traefik Forward Auth against a whitelist of Google accounts][tfa-google]
* [Authenticate Traefik Forward Auth against a self-hosted Keycloak instance][tfa-keycloak] with an optional [OpenLDAP backend][openldap]
--8<-- "recipe-footer.md"
[^1]: Authhost mode is specifically handy for Google authentication, since Google doesn't permit wildcard redirect_uris, like [Keycloak][keycloak] does.

104
docs/docker-swarm/traefik-forward-auth/keycloak.md Normal file
View File

@@ -0,0 +1,104 @@
---
title: SSO with traefik forward auth with Keycloak
description: Traefik forward auth can selectively SSO your Docker services against an authentication backend using OIDC, and Keycloak is a perfect, self-hosted match.
---
# Traefik Forward Auth with Keycloak for SSO
While the [Traefik Forward Auth](/docker-swarm/traefik-forward-auth/) recipe demonstrated a quick way to protect a set of explicitly-specified URLs using OIDC credentials from a Google account, this recipe will illustrate how to use your own Keycloak instance to secure **any** URLs within your DNS domain.
!!! tip "Keycloak with Traefik"
Did you land here from a search, looking for information about using Keycloak with Traefik? All this and more is covered in the [Keycloak][keycloak] recipe!
--8<-- "recipe-tfa-ingredients.md"
## Preparation
### Setup environment
Create `/var/data/config/traefik/traefik-forward-auth.env` as per the following example (_change "master" if you created a different realm_):
```bash
CLIENT_ID=<your keycloak client name>
CLIENT_SECRET=<your keycloak client secret>
OIDC_ISSUER=https://<your keycloak URL>/auth/realms/master
SECRET=<a random string to secure your cookie>
AUTH_HOST=<the FQDN to use for your auth host>
COOKIE_DOMAIN=<the root FQDN of your domain>
```
### Prepare the docker service config
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik](/docker-swarm/traefik/) recipe:
```bash
traefik-forward-auth:
image: funkypenguin/traefik-forward-auth
env_file: /var/data/config/traefik/traefik-forward-auth.env
networks:
- traefik_public
deploy:
labels:
- traefik.port=4181
- traefik.frontend.rule=Host:auth.example.com
- traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181
- traefik.frontend.auth.forward.trustForwardHeader=true
```
If you're not confident that forward authentication is working, add a simple "whoami" test container, to help debug traefik forward auth, before attempting to add it to a more complex container.
```bash
# This simply validates that traefik forward authentication is working
whoami:
image: containous/whoami
networks:
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:whoami.example.com
- traefik.port=80
- traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181
- traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
- traefik.frontend.auth.forward.trustForwardHeader=true
```
--8<-- "premix-cta.md"
## Serving
### Launch
Redeploy traefik with `docker stack deploy traefik-app -c /var/data/traefik/traeifk-app.yml`, to launch the traefik-forward-auth container.
### Test
Browse to <https://whoami.example.com> (_obviously, customized for your domain and having created a DNS record_), and all going according to plan, you'll be redirected to a Keycloak login. Once successfully logged in, you'll be directed to the basic whoami page.
### Protect services
To protect any other service, ensure the service itself is exposed by Traefik (_if you were previously using an oauth_proxy for this, you may have to migrate some labels from the oauth_proxy serivce to the service itself_). Add the following 3 labels:
```yaml
- traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181
- traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
- traefik.frontend.auth.forward.trustForwardHeader=true
```
And re-deploy your services :)
## Summary
What have we achieved? By adding an additional three simple labels to any service, we can secure any service behind our Keycloak OIDC provider, with minimal processing / handling overhead.
!!! summary "Summary"
Created:
* [X] Traefik-forward-auth configured to authenticate against Keycloak
[^1]: Keycloak is very powerful. You can add 2FA and all other clever things outside of the scope of this simple recipe ;)
### Keycloak vs Authelia
[KeyCloak][keycloak] is the "big daddy" of self-hosted authentication platforms - it has a beautiful GUI, and a very advanced and mature featureset. Like Authelia, KeyCloak can [use an LDAP server](/recipes/keycloak/authenticate-against-openldap/) as a backend, but _unlike_ Authelia, KeyCloak allows for 2-way sync between that LDAP backend, meaning KeyCloak can be used to _create_ and _update_ the LDAP entries (*Authelia's is just a one-way LDAP lookup - you'll need another tool to actually administer your LDAP database*).
--8<-- "recipe-footer.md"

253
docs/docker-swarm/traefik.md Normal file
View File

@@ -0,0 +1,253 @@
---
title: Traefik exposes Docker services with LetsEncrypt certificates
description: Using Traefik, we can provide secure ingress into our Docker Swarm cluster, which opens up opportunities to provide SSO to multiple services in docker swarm via OIDC / SSO, using traefik-forward-auth.
---
# Traefik
The platforms we plan to run on our cloud are generally web-based, and each listening on their own unique TCP port. When a container in a swarm exposes a port, then connecting to **any** swarm member on that port will result in your request being forwarded to the appropriate host running the container. (_Docker calls this the swarm "[routing mesh](https://docs.docker.com/engine/swarm/ingress/)"_)
So we get a rudimentary load balancer built into swarm. We could stop there, just exposing a series of ports on our hosts, and making them HA using keepalived.
There are some gaps to this approach though:
- No consideration is given to HTTPS. Implementation would have to be done manually, per-container.
- No mechanism is provided for authentication outside of that which the container providers. We may not **want** to expose every interface on every container to the world, especially if we are playing with tools or containers whose quality and origin are unknown.
To deal with these gaps, we need a front-end load-balancer, and in this design, that role is provided by [Traefik](https://traefik.io/).
![Traefik Screenshot](/images/traefik.png){ loading=lazy }
!!! tip
In 2021, this recipe was updated for Traefik v2. There's really no reason to be using Traefikv1 anymore ;)
## Ingredients
!!! summary "Ingredients"
Already deployed:
* [X] [Docker swarm cluster](/docker-swarm/design/) with [persistent shared storage](/docker-swarm/shared-storage-ceph/)
* [X] DNS entry for the hostname you intend to use (*or a wildcard*), pointed to your [keepalived](/docker-swarm/keepalived/) IP
New:
* [ ] Traefik configured per design
* [ ] Access to update your DNS records for manual/automated [LetsEncrypt](https://letsencrypt.org/docs/challenge-types/) DNS-01 validation, or ingress HTTP/HTTPS for HTTP-01 validation
## Preparation
### Prepare traefik.toml
While it's possible to configure traefik via docker command arguments, I prefer to create a config file (`traefik.toml`). This allows me to change traefik's behaviour by simply changing the file, and keeps my docker config simple.
Create `/var/data/traefikv2/traefik.toml` as per the following example:
```bash
[global]
checkNewVersion = true
# Enable the Dashboard
[api]
dashboard = true
# Write out Traefik logs
[log]
level = "INFO"
filePath = "/traefik.log"
[entryPoints.http]
address = ":80"
# Redirect to HTTPS (why wouldn't you?)
[entryPoints.http.http.redirections.entryPoint]
to = "https"
scheme = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.http.tls]
certResolver = "main"
# Let's Encrypt
[certificatesResolvers.main.acme]
email = "batman@example.com"
storage = "acme.json"
# uncomment to use staging CA for testing
# caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
[certificatesResolvers.main.acme.dnsChallenge]
provider = "route53"
# Uncomment to use HTTP validation, like a caveman!
# [certificatesResolvers.main.acme.httpChallenge]
# entryPoint = "http"
# Docker Traefik provider
[providers.docker]
endpoint = "unix:///var/run/docker.sock"
swarmMode = true
watch = true
```
### Prepare the docker service config
!!! tip
"We'll want an overlay network, independent of our traefik stack, so that we can attach/detach all our other stacks (including traefik) to the overlay network. This way, we can undeploy/redepoly the traefik stack without having to bring down every other stack first!" - voice of hard-won experience
Create `/var/data/config/traefik/traefik.yml` as per the following example:
```yaml
version: "3.2"
# What is this?
#
# This stack exists solely to deploy the traefik_public overlay network, so that
# other stacks (including traefik-app) can attach to it
services:
scratch:
image: scratch
deploy:
replicas: 0
networks:
- public
networks:
public:
driver: overlay
attachable: true
ipam:
config:
- subnet: 172.16.200.0/24
```
--8<-- "premix-cta.md"
Create `/var/data/config/traefikv2/traefikv2.env` with the environment variables required by the provider you chose in the LetsEncrypt DNS Challenge section of `traefik.toml`. Full configuration options can be found in the [Traefik documentation](https://doc.traefik.io/traefik/https/acme/#providers). Route53 and CloudFlare examples are below.
```bash
# Route53 example
AWS_ACCESS_KEY_ID=<your-aws-key>
AWS_SECRET_ACCESS_KEY=<your-aws-secret>
# CloudFlare example
# CLOUDFLARE_EMAIL=<your-cloudflare-email>
# CLOUDFLARE_API_KEY=<your-cloudflare-api-key>
```
Create `/var/data/config/traefikv2/traefikv2.yml` as per the following example:
```yaml
version: "3.2"
services:
app:
image: traefik:v2.4
env_file: /var/data/config/traefikv2/traefikv2.env
# Note below that we use host mode to avoid source nat being applied to our ingress HTTP/HTTPS sessions
# Without host mode, all inbound sessions would have the source IP of the swarm nodes, rather than the
# original source IP, which would impact logging. If you don't care about this, you can expose ports the
# "minimal" way instead
ports:
- target: 80
published: 80
protocol: tcp
mode: host
- target: 443
published: 443
protocol: tcp
mode: host
- target: 8080
published: 8080
protocol: tcp
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /var/data/config/traefikv2:/etc/traefik
- /var/data/traefikv2/traefik.log:/traefik.log
- /var/data/traefikv2/acme.json:/acme.json
networks:
- traefik_public
# Global mode makes an instance of traefik listen on _every_ node, so that regardless of which
# node the request arrives on, it'll be forwarded to the correct backend service.
deploy:
mode: global
labels:
- "traefik.docker.network=traefik_public"
- "traefik.http.routers.api.rule=Host(`traefik.example.com`)"
- "traefik.http.routers.api.entrypoints=https"
- "traefik.http.routers.api.tls.domains[0].main=example.com"
- "traefik.http.routers.api.tls.domains[0].sans=*.example.com"
- "traefik.http.routers.api.tls=true"
- "traefik.http.routers.api.tls.certresolver=main"
- "traefik.http.routers.api.service=api@internal"
- "traefik.http.services.dummy.loadbalancer.server.port=9999"
# uncomment this to enable forward authentication on the traefik api/dashboard
#- "traefik.http.routers.api.middlewares=forward-auth"
placement:
constraints: [node.role == manager]
networks:
traefik_public:
external: true
```
Docker won't start a service with a bind-mount to a non-existent file, so prepare an empty acme.json and traefik.log (_with the appropriate permissions_) by running:
```bash
touch /var/data/traefikv2/acme.json
touch /var/data/traefikv2/traefik.log
chmod 600 /var/data/traefikv2/acme.json
chmod 600 /var/data/traefikv2/traefik.log
```
!!! warning
Pay attention above. You **must** set `acme.json`'s permissions to owner-readable-only, else the container will fail to start with an [ID-10T](https://en.wikipedia.org/wiki/User_error#ID-10-T_error) error!
Traefik will populate acme.json itself when it runs, but it needs to exist before the container will start (_Chicken, meet egg._)
Likewise with the log file.
## Serving
### Launch
First, launch the traefik stack, which will do nothing other than create an overlay network by running `docker stack deploy traefik -c /var/data/config/traefik/traefik.yml`
```bash
[root@kvm ~]# docker stack deploy traefik -c /var/data/config/traefik/traefik.yml
Creating network traefik_public
Creating service traefik_scratch
[root@kvm ~]#
```
Now deploy the traefik application itself (*which will attach to the overlay network*) by running `docker stack deploy traefikv2 -c /var/data/config/traefikv2/traefikv2.yml`
```bash
[root@kvm ~]# docker stack deploy traefikv2 -c /var/data/config/traefikv2/traefikv2.yml
Creating service traefikv2_traefikv2
[root@kvm ~]#
```
Confirm traefik is running with `docker stack ps traefikv2`:
```bash
root@raphael:~# docker stack ps traefikv2
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
lmvqcfhap08o traefikv2_app.dz178s1aahv16bapzqcnzc03p traefik:v2.4 donatello Running Running 2 minutes ago *:443->443/tcp,*:80->80/tcp
root@raphael:~#
```
### Check Traefik Dashboard
You should now be able to access[^1] your traefik instance on `https://traefik.<your domain\>` (*if your LetsEncrypt certificate is working*), or `http://<node IP\>:8080` (*if it's not*)- It'll look a little lonely currently (*below*), but we'll populate it as we add recipes :grin:
![Screenshot of Traefik, post-launch](/images/traefik-post-launch.png){ loading=lazy }
### Summary
!!! summary
We've achieved:
* [X] An overlay network to permit traefik to access all future stacks we deploy
* [X] Frontend proxy which will dynamically configure itself for new backend containers
* [X] Automatic SSL support for all proxied resources
[^1]: Did you notice how no authentication was required to view the Traefik dashboard? Eek! We'll tackle that in the next section, regarding [Traefik Forward Authentication](/docker-swarm/traefik-forward-auth/)!
--8<-- "recipe-footer.md"