public/geek-cookbook
1
0
Fork 0
mirror of https://github.com/funkypenguin/geek-cookbook/ synced 2025-12-28 09:01:46 +00:00
Code Issues Releases Wiki Activity

Update for leanpub preview

Browse Source
This commit is contained in:
AutoPenguin
2020-06-03 02:43:21 +00:00
parent 3862d7c2f6
commit bbe30c872c
89 changed files with 636 additions and 648 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
manuscript/ha-docker-swarm/design.md
View File

@@ -5,7 +5,7 @@ In the design described below, our "private cloud" platform is:
* **Highly-available** (_can tolerate the failure of a single component_)
* **Scalable** (_can add resource or capacity as required_)
* **Portable** (_run it on your garage server today, run it in AWS tomorrow_)
* **Secure** (_access protected with [LetsEncrypt certificates]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik/) and optional [OIDC with 2FA]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik-forward-auth/)_)
* **Secure** (_access protected with [LetsEncrypt certificates](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik/) and optional [OIDC with 2FA](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik-forward-auth/)_)
* **Automated** (_requires minimal care and feeding_)
## Design Decisions
@@ -15,7 +15,7 @@ In the design described below, our "private cloud" platform is:
This means that:
* At least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure.
* [Ceph]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph/) is employed for share storage, because it too can be made tolerant of a single failure.
* [Ceph](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph/) is employed for share storage, because it too can be made tolerant of a single failure.
!!! note
An exception to the 3-nodes decision is running a single-node configuration. If you only **have** one node, then obviously your swarm is only as resilient as that node. It's still a perfectly valid swarm configuration, ideal for starting your self-hosting journey. In fact, under the single-node configuration, you don't need ceph either, and you can simply use the local volume on your host for storage. You'll be able to migrate to ceph/more nodes if/when you expand.
@@ -38,8 +38,8 @@ Under this design, the only inbound connections we're permitting to our docker s
### Authentication
* Where the hosted application provides a trusted level of authentication (*i.e., [NextCloud]https://geek-cookbook.funkypenguin.co.nz/recipes/nextcloud/)*), or where the application requires public exposure (*i.e. [Privatebin]https://geek-cookbook.funkypenguin.co.nz/recipes/privatebin/)*), no additional layer of authentication will be required.
* Where the hosted application provides inadequate (*i.e. [NZBGet]https://geek-cookbook.funkypenguin.co.nz/recipes/autopirate/nzbget/)*) or no authentication (*i.e. [Gollum]https://geek-cookbook.funkypenguin.co.nz/recipes/gollum/)*), a further authentication against an OAuth provider will be required.
* Where the hosted application provides a trusted level of authentication (*i.e., [NextCloud](https://geek-cookbook.funkypenguin.co.nz/recipes/nextcloud/)*), or where the application requires public exposure (*i.e. [Privatebin](https://geek-cookbook.funkypenguin.co.nz/recipes/privatebin/)*), no additional layer of authentication will be required.
* Where the hosted application provides inadequate (*i.e. [NZBGet](https://geek-cookbook.funkypenguin.co.nz/recipes/autopirate/nzbget/)*) or no authentication (*i.e. [Gollum](https://geek-cookbook.funkypenguin.co.nz/recipes/gollum/)*), a further authentication against an OAuth provider will be required.
## High availability

4
manuscript/ha-docker-swarm/docker-swarm-mode.md
View File

@@ -128,7 +128,7 @@ networks:
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list]https://geek-cookbook.funkypenguin.co.nz/reference/networks/) here.
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](https://geek-cookbook.funkypenguin.co.nz/reference/networks/) here.
Launch the cleanup stack by running ```docker stack deploy docker-cleanup -c <path-to-docker-compose.yml>```
@@ -169,7 +169,7 @@ Launch shepherd by running ```docker stack deploy shepherd -c /var/data/config/s
After completing the above, you should have:
* [X] [Docker swarm cluster]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/)
* [X] [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/)
## Chef's Notes

2
manuscript/ha-docker-swarm/nodes.md
View File

@@ -3,7 +3,7 @@
Let's start building our cluster. You can use either bare-metal machines or virtual machines - the configuration would be the same. To avoid confusion, I'll be referring to these as "nodes" from now on.
!!! note
In 2017, I **initially** chose the "[Atomic](https://www.projectatomic.io/)" CentOS/Fedora image for the swarm hosts, but later found its outdated version of Docker to be problematic with advanced features like GPU transcoding (in [Plex]https://geek-cookbook.funkypenguin.co.nz/recipes/plex/)), [Swarmprom]https://geek-cookbook.funkypenguin.co.nz/recipes/swarmprom/), etc. In the end, I went mainstream and simply preferred a modern Ubuntu installation.
In 2017, I **initially** chose the "[Atomic](https://www.projectatomic.io/)" CentOS/Fedora image for the swarm hosts, but later found its outdated version of Docker to be problematic with advanced features like GPU transcoding (in [Plex](https://geek-cookbook.funkypenguin.co.nz/recipes/plex/)), [Swarmprom](https://geek-cookbook.funkypenguin.co.nz/recipes/swarmprom/), etc. In the end, I went mainstream and simply preferred a modern Ubuntu installation.
## Ingredients

4
manuscript/ha-docker-swarm/registry.md
View File

@@ -10,8 +10,8 @@ The registry mirror runs as a swarm stack, using a simple docker-compose.yml. Cu
## Ingredients
1. [Docker swarm cluster]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/) with [persistent shared storage]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik) configured per design
1. [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/) with [persistent shared storage](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP

4
manuscript/ha-docker-swarm/shared-storage-ceph.md
View File

@@ -52,7 +52,7 @@ The process takes about 30 seconds, after which, you'll have a MVC (*Minimum Via
INFO:cephadm:Verifying time synchronization is in place...
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
INFO:cephadm:Repeating the final host check...
INFO:cephadm:podman|docker https://geek-cookbook.funkypenguin.co.nz/usr/bin/docker) is present
INFO:cephadm:podman|docker (/usr/bin/docker) is present
INFO:cephadm:systemctl is present
INFO:cephadm:lvcreate is present
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
@@ -143,7 +143,7 @@ Path on master | Path on non-master
Back on the ==master== node, run `ceph orch host add <node-name>` once for each other node you want to join to the cluster. You can validate the results by running `ceph orch host ls`
!!! question "Should we be concerned about giving cephadm using root access over SSH?"
Not really. Docker is inherently insecure at the host-level anyway (*think what would happen if you launched a global-mode stack with a malicious container image which mounted `/root/.ssh`*), so worrying about cephadm seems a little barn-door-after-horses-bolted. If you take host-level security seriously, consider switching to [Kubernetes]https://geek-cookbook.funkypenguin.co.nz/kubernetes/start/) :)
Not really. Docker is inherently insecure at the host-level anyway (*think what would happen if you launched a global-mode stack with a malicious container image which mounted `/root/.ssh`*), so worrying about cephadm seems a little barn-door-after-horses-bolted. If you take host-level security seriously, consider switching to [Kubernetes](https://geek-cookbook.funkypenguin.co.nz/kubernetes/start/) :)
### Add OSDs

2
manuscript/ha-docker-swarm/shared-storage-gluster.md
View File

@@ -3,7 +3,7 @@
While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node.
!!! warning
This recipe is deprecated. It didn't work well in 2017, and it's not likely to work any better now. It remains here as a reference. I now recommend the use of [Ceph for shared storage]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph/) instead. - 2019 Chef
This recipe is deprecated. It didn't work well in 2017, and it's not likely to work any better now. It remains here as a reference. I now recommend the use of [Ceph for shared storage](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph/) instead. - 2019 Chef
## Design

16
manuscript/ha-docker-swarm/traefik-forward-auth.md
View File

@@ -2,28 +2,28 @@
Now that we have Traefik deployed, automatically exposing SSL access to our Docker Swarm services using LetsEncrypt wildcard certificates, let's pause to consider that we may not _want_ some services exposed directly to the internet...
..Wait, why not? Well, Traefik doesn't provide any form of authentication, it simply secures the **transmission** of the service between Docker Swarm and the end user. If you were to deploy a service with no native security (*[Radarr]https://geek-cookbook.funkypenguin.co.nz/recipes/autopirate/radarr/) or [Sonarr]https://geek-cookbook.funkypenguin.co.nz/recipes/autopirate/sonarr/) come to mind*), then anybody would be able to use it! Even services which _may_ have a layer of authentication **might** not be safe to expose publically - often open source projects may be maintained by enthusiasts who happily add extra features, but just pay lip service to security, on the basis that "*it's the user's problem to secure it in their own network*".
..Wait, why not? Well, Traefik doesn't provide any form of authentication, it simply secures the **transmission** of the service between Docker Swarm and the end user. If you were to deploy a service with no native security (*[Radarr](https://geek-cookbook.funkypenguin.co.nz/recipes/autopirate/radarr/) or [Sonarr](https://geek-cookbook.funkypenguin.co.nz/recipes/autopirate/sonarr/) come to mind*), then anybody would be able to use it! Even services which _may_ have a layer of authentication **might** not be safe to expose publically - often open source projects may be maintained by enthusiasts who happily add extra features, but just pay lip service to security, on the basis that "*it's the user's problem to secure it in their own network*".
To give us confidence that **we** can access our services, but BadGuys(tm) cannot, we'll deploy a layer of authentication **in front** of Traefik, using [Forward Authentication](https://docs.traefik.io/configuration/entrypoints/#forward-authentication). You can use your own [KeyCloak]https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/) instance for authentication, but to lower the barrier to entry, this recipe will assume you're authenticating against your own Google account.
To give us confidence that **we** can access our services, but BadGuys(tm) cannot, we'll deploy a layer of authentication **in front** of Traefik, using [Forward Authentication](https://docs.traefik.io/configuration/entrypoints/#forward-authentication). You can use your own [KeyCloak](https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/) instance for authentication, but to lower the barrier to entry, this recipe will assume you're authenticating against your own Google account.
## Ingredients
!!! summary "Ingredients"
Existing:
* [X] [Docker swarm cluster]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/) with [persistent shared storage]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph)
* [X] [Traefik]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik/) configured per design
* [X] [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/) with [persistent shared storage](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph)
* [X] [Traefik](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik/) configured per design
New:
* [ ] Client ID and secret from an OpenID-Connect provider (Google, [KeyCloak]https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/), Microsoft, etc..)
* [ ] Client ID and secret from an OpenID-Connect provider (Google, [KeyCloak](https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/), Microsoft, etc..)
## Preparation
### Obtain OAuth credentials
!!! note
This recipe will demonstrate using Google OAuth for traefik forward authentication, but it's also possible to use a self-hosted KeyCloak instance - see the [KeyCloak OIDC Provider]https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/setup-oidc-provider/) recipe for more details!
This recipe will demonstrate using Google OAuth for traefik forward authentication, but it's also possible to use a self-hosted KeyCloak instance - see the [KeyCloak OIDC Provider](https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/setup-oidc-provider/) recipe for more details!
Log into https://console.developers.google.com/, create a new project then search for and select "Credentials" in the search bar.
@@ -48,7 +48,7 @@ COOKIE_DOMAINS=example.com
### Prepare the docker service config
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik]https://geek-cookbook.funkypenguin.co.nz/recipes/traefik/) recipe:
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik](https://geek-cookbook.funkypenguin.co.nz/recipes/traefik/) recipe:
```
traefik-forward-auth:
@@ -110,7 +110,7 @@ What have we achieved? By adding an additional three simple labels to any servic
## Chef's Notes
1. Traefik forward auth replaces the use of [oauth_proxy containers]https://geek-cookbook.funkypenguin.co.nz/reference/oauth_proxy/) found in some of the existing recipes
1. Traefik forward auth replaces the use of [oauth_proxy containers](https://geek-cookbook.funkypenguin.co.nz/reference/oauth_proxy/) found in some of the existing recipes
2. [@thomaseddon's original version](https://github.com/thomseddon/traefik-forward-auth) of traefik-forward-auth only works with Google currently, but I've created a [fork](https://www.github.com/funkypenguin/traefik-forward-auth) of a [fork](https://github.com/noelcatt/traefik-forward-auth), which implements generic OIDC providers.
3. I reviewed several implementations of forward authenticators for Traefik, but found most to be rather heavy-handed, or specific to a single auth provider. @thomaseddon's go-based docker image is 7MB in size, and with the generic OIDC patch (above), it can be extended to work with any OIDC provider.
4. No, not github natively, but you can ferderate GitHub into KeyCloak, and then use KeyCloak as the OIDC provider.

6
manuscript/ha-docker-swarm/traefik-forward-auth/keycloak.md
View File

@@ -1,13 +1,13 @@
# Using Traefik Forward Auth with KeyCloak
While the [Traefik Forward Auth]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik-forward-auth/) recipe demonstrated a quick way to protect a set of explicitly-specified URLs using OIDC credentials from a Google account, this recipe will illustrate how to use your own KeyCloak instance to secure **any** URLs within your DNS domain.
While the [Traefik Forward Auth](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik-forward-auth/) recipe demonstrated a quick way to protect a set of explicitly-specified URLs using OIDC credentials from a Google account, this recipe will illustrate how to use your own KeyCloak instance to secure **any** URLs within your DNS domain.
## Ingredients
!!! Summary
Existing:
* [X] [KeyCloak]https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/) recipe deployed successfully, with a [local user]https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/create-user/) and an [OIDC client]https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/setup-oidc-provider/)
* [X] [KeyCloak](https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/) recipe deployed successfully, with a [local user](https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/create-user/) and an [OIDC client](https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/setup-oidc-provider/)
New:
@@ -48,7 +48,7 @@ COOKIE_DOMAIN=<the root FQDN of your domain>
### Prepare the docker service config
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik]https://geek-cookbook.funkypenguin.co.nz/recipes/traefik/) recipe:
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik](https://geek-cookbook.funkypenguin.co.nz/recipes/traefik/) recipe:
```
traefik-forward-auth:

6
manuscript/ha-docker-swarm/traefik.md
View File

@@ -18,7 +18,7 @@ To deal with these gaps, we need a front-end load-balancer, and in this design,
!!! summary "You'll need"
Existing
* [X] [Docker swarm cluster]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/) with [persistent shared storage]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph)
* [X] [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/) with [persistent shared storage](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph)
New
@@ -222,7 +222,7 @@ ID NAME IMAGE
You should now be able to access your traefik instance on http://<node IP\>:8080 - It'll look a little lonely currently (*below*), but we'll populate it as we add recipes :)
![Screenshot of Traefik, post-launch]https://geek-cookbook.funkypenguin.co.nz/images/traefik-post-launch.png)
![Screenshot of Traefik, post-launch](https://geek-cookbook.funkypenguin.co.nz/images/traefik-post-launch.png)
### Summary
@@ -236,4 +236,4 @@ You should now be able to access your traefik instance on http://<node IP\>:8080
## Chef's Notes
1. Did you notice how no authentication was required to view the Traefik dashboard? Eek! We'll tackle that in the next section, regarding [Traefik Forward Authentication]https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik-forward-auth/)!
1. Did you notice how no authentication was required to view the Traefik dashboard? Eek! We'll tackle that in the next section, regarding [Traefik Forward Authentication](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik-forward-auth/)!