mirror of
https://github.com/funkypenguin/geek-cookbook/
synced 2026-10-26 10:43:21 +00:00
Update for leanpub preview
This commit is contained in:
AutoPenguin committed
1 parent
bc68bf8c59
commit
6bb9e2f7d4
83 files changed
+12
-296
No files matched your search
@@ -17,7 +17,6 @@ This means that:
|
||||
* At least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure.
|
||||
* [Ceph](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph/) is employed for share storage, because it too can be made tolerant of a single failure.
|
||||
|
||||
!!! note
|
||||
An exception to the 3-nodes decision is running a single-node configuration. If you only **have** one node, then obviously your swarm is only as resilient as that node. It's still a perfectly valid swarm configuration, ideal for starting your self-hosting journey. In fact, under the single-node configuration, you don't need ceph either, and you can simply use the local volume on your host for storage. You'll be able to migrate to ceph/more nodes if/when you expand.
|
||||
|
||||
**Where multiple solutions to a requirement exist, preference will be given to the most portable solution.**
|
||||
|
||||
@@ -4,7 +4,6 @@ For truly highly-available services with Docker containers, we need an orchestra
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary
|
||||
Existing
|
||||
|
||||
* [X] 3 x nodes (*bare-metal or VMs*), each with:
|
||||
@@ -127,7 +126,6 @@ networks:
|
||||
- subnet: 172.16.0.0/24
|
||||
```
|
||||
|
||||
!!! note
|
||||
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](https://geek-cookbook.funkypenguin.co.nz/reference/networks/) here.
|
||||
|
||||
Launch the cleanup stack by running ```docker stack deploy docker-cleanup -c <path-to-docker-compose.yml>```
|
||||
|
||||
@@ -10,7 +10,6 @@ This is accomplished with the use of keepalived on at least two nodes.
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
Already deployed:
|
||||
|
||||
* [X] At least 2 x swarm nodes
|
||||
|
||||
@@ -2,12 +2,10 @@
|
||||
|
||||
Let's start building our cluster. You can use either bare-metal machines or virtual machines - the configuration would be the same. To avoid confusion, I'll be referring to these as "nodes" from now on.
|
||||
|
||||
!!! note
|
||||
In 2017, I **initially** chose the "[Atomic](https://www.projectatomic.io/)" CentOS/Fedora image for the swarm hosts, but later found its outdated version of Docker to be problematic with advanced features like GPU transcoding (in [Plex](https://geek-cookbook.funkypenguin.co.nz/recipes/plex/)), [Swarmprom](https://geek-cookbook.funkypenguin.co.nz/recipes/swarmprom/), etc. In the end, I went mainstream and simply preferred a modern Ubuntu installation.
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
New in this recipe:
|
||||
|
||||
* [ ] 3 x nodes (*bare-metal or VMs*), each with:
|
||||
@@ -67,7 +65,6 @@ ln -sf /usr/share/zoneinfo/<your timezone> /etc/localtime
|
||||
|
||||
After completing the above, you should have:
|
||||
|
||||
!!! summary "Summary"
|
||||
Deployed in this recipe:
|
||||
|
||||
* [X] 3 x nodes (*bare-metal or VMs*), each with:
|
||||
|
||||
@@ -44,7 +44,6 @@ networks:
|
||||
external: true
|
||||
```
|
||||
|
||||
!!! note "Unencrypted registry"
|
||||
We create this registry without consideration for SSL, which will fail if we attempt to use the registry directly. However, we're going to use the HTTPS-proxied version via Traefik, leveraging Traefik to manage the LetsEncrypt certificates required.
|
||||
|
||||
|
||||
@@ -107,7 +106,6 @@ Then restart docker by running:
|
||||
systemctl restart docker-latest
|
||||
```
|
||||
|
||||
!!! tip ""
|
||||
Note the extra comma required after "false" above
|
||||
|
||||
## Chef's notes
|
||||
@@ -6,7 +6,6 @@ While Docker Swarm is great for keeping containers running (_and restarting thos
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
3 x Virtual Machines (configured earlier), each with:
|
||||
|
||||
* [X] Support for "modern" versions of Python and LVM
|
||||
@@ -18,7 +17,6 @@ While Docker Swarm is great for keeping containers running (_and restarting thos
|
||||
|
||||
## Preparation
|
||||
|
||||
!!! tip "No more [foolish games](https://www.youtube.com/watch?v=UNoouLa7uxA)"
|
||||
Earlier iterations of this recipe (*based on [Ceph Jewel](https://docs.ceph.com/docs/master/releases/jewel/)*) required significant manual effort to install Ceph in a Docker environment. In the 2+ years since Jewel was released, significant improvements have been made to the ceph "deploy-in-docker" process, including the [introduction of the cephadm tool](https://ceph.io/ceph-management/introducing-cephadm/). Cephadm is the tool which now does all the heavy lifting, below, for the current version of ceph, codenamed "[Octopus](https://www.youtube.com/watch?v=Gi58pN8W3hY)".
|
||||
|
||||
### Pick a master node
|
||||
@@ -133,16 +131,15 @@ The process takes about 30 seconds, after which, you'll have a MVC (*Minimum Via
|
||||
|
||||
It's now necessary to tranfer the following files to your ==other== nodes, so that cephadm can add them to your cluster, and so that they'll be able to mount the cephfs when we're done:
|
||||
|
||||
Path on master | Path on non-master
|
||||
--------------- | -----
|
||||
`/etc/ceph/ceph.conf` | `/etc/ceph/ceph.conf`
|
||||
`/etc/ceph/ceph.client.admin.keyring` | `/etc/ceph/ceph.client.admin.keyring`
|
||||
`/etc/ceph/ceph.pub` | `/root/.ssh/authorized_keys` (append to anything existing)
|
||||
| Path on master | Path on non-master |
|
||||
|---------------------------------------|------------------------------------------------------------|
|
||||
| `/etc/ceph/ceph.conf` | `/etc/ceph/ceph.conf` |
|
||||
| `/etc/ceph/ceph.client.admin.keyring` | `/etc/ceph/ceph.client.admin.keyring` |
|
||||
| `/etc/ceph/ceph.pub` | `/root/.ssh/authorized_keys` (append to anything existing) |
|
||||
|
||||
|
||||
Back on the ==master== node, run `ceph orch host add <node-name>` once for each other node you want to join to the cluster. You can validate the results by running `ceph orch host ls`
|
||||
|
||||
!!! question "Should we be concerned about giving cephadm using root access over SSH?"
|
||||
Not really. Docker is inherently insecure at the host-level anyway (*think what would happen if you launched a global-mode stack with a malicious container image which mounted `/root/.ssh`*), so worrying about cephadm seems a little barn-door-after-horses-bolted. If you take host-level security seriously, consider switching to [Kubernetes](https://geek-cookbook.funkypenguin.co.nz/kubernetes/start/) :)
|
||||
|
||||
### Add OSDs
|
||||
@@ -196,7 +193,6 @@ root@raphael:~#
|
||||
|
||||
What have we achieved?
|
||||
|
||||
!!! summary "Summary"
|
||||
Created:
|
||||
|
||||
* [X] Persistent storage available to every node
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node.
|
||||
|
||||
!!! warning
|
||||
This recipe is deprecated. It didn't work well in 2017, and it's not likely to work any better now. It remains here as a reference. I now recommend the use of [Ceph for shared storage](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph/) instead. - 2019 Chef
|
||||
|
||||
## Design
|
||||
@@ -13,7 +12,6 @@ This GlusterFS recipe was my original design for shared storage, but I [found it
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
3 x Virtual Machines (configured earlier), each with:
|
||||
|
||||
* [X] CentOS/Fedora Atomic
|
||||
@@ -30,7 +28,7 @@ To build our Gluster volume, we need 2 out of the 3 VMs to provide one "brick".
|
||||
|
||||
On each host, run a variation following to create your bricks, adjusted for the path to your disk.
|
||||
|
||||
!!! note "The example below assumes /dev/vdb is dedicated to the gluster volume"
|
||||
|
||||
```
|
||||
(
|
||||
echo o # Create a new empty DOS partition table
|
||||
@@ -50,7 +48,6 @@ echo '/dev/vdb1 /var/no-direct-write-here/brick1 xfs defaults 1 2' >> /etc/fstab
|
||||
mount -a && mount
|
||||
```
|
||||
|
||||
!!! warning "Don't provision all your LVM space"
|
||||
Atomic uses LVM to store docker data, and **automatically grows** Docker's volumes as requried. If you commit all your free LVM space to your brick, you'll quickly find (as I did) that docker will start to fail with error messages about insufficient space. If you're going to slice off a portion of your LVM space in /dev/atomicos, make sure you leave enough space for Docker storage, where "enough" depends on how much you plan to pull images, make volumes, etc. I ate through 20GB very quickly doing development, so I ended up provisioning 50GB for atomic alone, with a separate volume for the brick.
|
||||
|
||||
### Create glusterfs container
|
||||
@@ -58,6 +55,7 @@ mount -a && mount
|
||||
Atomic doesn't include the Gluster server components. This means we'll have to run glusterd from within a container, with privileged access to the host. Although convoluted, I've come to prefer this design since it once again makes the OS "disposable", moving all the config into containers and code.
|
||||
|
||||
Run the following on each host:
|
||||
|
||||
```
|
||||
docker run \
|
||||
-h glusterfs-server \
|
||||
@@ -71,6 +69,7 @@ docker run \
|
||||
--name="glusterfs-server" \
|
||||
gluster/gluster-centos
|
||||
```
|
||||
|
||||
### Create trusted pool
|
||||
|
||||
On a single node (doesn't matter which), run ```docker exec -it glusterfs-server bash``` to launch a shell inside the container.
|
||||
|
||||
@@ -8,7 +8,6 @@ To give us confidence that **we** can access our services, but BadGuys(tm) canno
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
Existing:
|
||||
|
||||
* [X] [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/) with [persistent shared storage](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph)
|
||||
@@ -22,7 +21,6 @@ To give us confidence that **we** can access our services, but BadGuys(tm) canno
|
||||
|
||||
### Obtain OAuth credentials
|
||||
|
||||
!!! note
|
||||
This recipe will demonstrate using Google OAuth for traefik forward authentication, but it's also possible to use a self-hosted KeyCloak instance - see the [KeyCloak OIDC Provider](https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/setup-oidc-provider/) recipe for more details!
|
||||
|
||||
Log into https://console.developers.google.com/, create a new project then search for and select "Credentials" in the search bar.
|
||||
@@ -82,7 +80,6 @@ If you're not confident that forward authentication is working, add a simple "wh
|
||||
- traefik.frontend.auth.forward.trustForwardHeader=true
|
||||
```
|
||||
|
||||
!!! tip
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy```
|
||||
|
||||
|
||||
@@ -101,7 +98,6 @@ Browse to https://whoami.example.com (*obviously, customized for your domain and
|
||||
|
||||
What have we achieved? By adding an additional three simple labels to any service, we can secure any service behind our choice of OAuth provider, with minimal processing / handling overhead.
|
||||
|
||||
!!! summary "Summary"
|
||||
Created:
|
||||
|
||||
* [X] Traefik-forward-auth configured to authenticate against an OIDC provider
|
||||
|
||||
@@ -4,7 +4,6 @@ While the [Traefik Forward Auth](https://geek-cookbook.funkypenguin.co.nz/ha-doc
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! Summary
|
||||
Existing:
|
||||
|
||||
* [X] [KeyCloak](https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/) recipe deployed successfully, with a [local user](https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/create-user/) and an [OIDC client](https://geek-cookbook.funkypenguin.co.nz/recipes/keycloak/setup-oidc-provider/)
|
||||
@@ -81,7 +80,6 @@ If you're not confident that forward authentication is working, add a simple "wh
|
||||
- traefik.frontend.auth.forward.trustForwardHeader=true
|
||||
```
|
||||
|
||||
!!! tip
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy```
|
||||
|
||||
## Serving
|
||||
@@ -110,7 +108,6 @@ And re-deploy your services :)
|
||||
|
||||
What have we achieved? By adding an additional three simple labels to any service, we can secure any service behind our KeyCloak OIDC provider, with minimal processing / handling overhead.
|
||||
|
||||
!!! summary "Summary"
|
||||
Created:
|
||||
|
||||
* [X] Traefik-forward-auth configured to authenticate against KeyCloak
|
||||
|
||||
@@ -15,7 +15,6 @@ To deal with these gaps, we need a front-end load-balancer, and in this design,
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "You'll need"
|
||||
Existing
|
||||
|
||||
* [X] [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/) with [persistent shared storage](https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph)
|
||||
@@ -30,7 +29,6 @@ To deal with these gaps, we need a front-end load-balancer, and in this design,
|
||||
|
||||
The traefik container is aware of the __other__ docker containers in the swarm, because it has access to the docker socket at **/var/run/docker.sock**. This allows traefik to dynamically configure itself based on the labels found on containers in the swarm, which is hugely useful. To make this functionality work on a SELinux-enabled CentOS7 host, we need to add custom SELinux policy.
|
||||
|
||||
!!! tip
|
||||
The following is only necessary if you're using SELinux!
|
||||
|
||||
Run the following to build and activate policy to permit containers to access docker.sock:
|
||||
@@ -92,7 +90,6 @@ swarmmode = true
|
||||
|
||||
### Prepare the docker service config
|
||||
|
||||
!!! tip
|
||||
"We'll want an overlay network, independent of our traefik stack, so that we can attach/detach all our other stacks (including traefik) to the overlay network. This way, we can undeploy/redepoly the traefik stack without having to bring every other stack first!" - voice of experience
|
||||
|
||||
Create `/var/data/config/traefik/traefik.yml` as follows:
|
||||
@@ -122,7 +119,6 @@ networks:
|
||||
- subnet: 172.16.200.0/24
|
||||
```
|
||||
|
||||
!!! tip
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy```
|
||||
|
||||
|
||||
@@ -181,7 +177,6 @@ touch /var/data/traefik/acme.json
|
||||
chmod 600 /var/data/traefik/acme.json
|
||||
```
|
||||
|
||||
!!! warning
|
||||
Pay attention above. You **must** set `acme.json`'s permissions to owner-readable-only, else the container will fail to start with an [ID-10T](https://en.wikipedia.org/wiki/User_error#ID-10-T_error) error!
|
||||
|
||||
Traefik will populate acme.json itself when it runs, but it needs to exist before the container will start (_Chicken, meet egg._)
|
||||
@@ -226,7 +221,6 @@ You should now be able to access your traefik instance on http://<node IP\>:8080
|
||||
|
||||
### Summary
|
||||
|
||||
!!! summary
|
||||
We've achieved:
|
||||
|
||||
* [X] An overlay network to permit traefik to access all future stacks we deploy
|
||||
|
||||
Reference in new issue
Block a user