mirror of
https://github.com/funkypenguin/geek-cookbook/
synced 2026-10-28 03:33:23 +00:00
Update for leanpub preview
This commit is contained in:
AutoPenguin committed
1 parent
37c2f9a159
commit
65dd34c7ea
120 files changed
+1142
-2639
No files matched your search
@@ -5,7 +5,7 @@ In the design described below, our "private cloud" platform is:
|
||||
* **Highly-available** (_can tolerate the failure of a single component_)
|
||||
* **Scalable** (_can add resource or capacity as required_)
|
||||
* **Portable** (_run it on your garage server today, run it in AWS tomorrow_)
|
||||
* **Secure** (_access protected with [LetsEncrypt certificates](/ha-docker-swarm/traefik/) and optional [OIDC with 2FA](/ha-docker-swarm/traefik-forward-auth/)_)
|
||||
* **Secure** (_access protected with [LetsEncrypt certificates](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/traefik/) and optional [OIDC with 2FA](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/traefik-forward-auth/)_)
|
||||
* **Automated** (_requires minimal care and feeding_)
|
||||
|
||||
## Design Decisions
|
||||
@@ -15,7 +15,7 @@ In the design described below, our "private cloud" platform is:
|
||||
This means that:
|
||||
|
||||
* At least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure.
|
||||
* [Ceph](/ha-docker-swarm/shared-storage-ceph/) is employed for share storage, because it too can be made tolerant of a single failure.
|
||||
* [Ceph](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/shared-storage-ceph/) is employed for share storage, because it too can be made tolerant of a single failure.
|
||||
|
||||
!!! note
|
||||
An exception to the 3-nodes decision is running a single-node configuration. If you only **have** one node, then obviously your swarm is only as resilient as that node. It's still a perfectly valid swarm configuration, ideal for starting your self-hosting journey. In fact, under the single-node configuration, you don't need ceph either, and you can simply use the local volume on your host for storage. You'll be able to migrate to ceph/more nodes if/when you expand.
|
||||
@@ -38,8 +38,8 @@ Under this design, the only inbound connections we're permitting to our docker s
|
||||
|
||||
### Authentication
|
||||
|
||||
* Where the hosted application provides a trusted level of authentication (*i.e., [NextCloud](/recipes/nextcloud/)*), or where the application requires public exposure (*i.e. [Privatebin](/recipes/privatebin/)*), no additional layer of authentication will be required.
|
||||
* Where the hosted application provides inadequate (*i.e. [NZBGet](/recipes/autopirate/nzbget/)*) or no authentication (*i.e. [Gollum](/recipes/gollum/)*), a further authentication against an OAuth provider will be required.
|
||||
* Where the hosted application provides a trusted level of authentication (*i.e., [NextCloud](https://geek-cookbook.funkypenguin.co.nz/)recipes/nextcloud/)*), or where the application requires public exposure (*i.e. [Privatebin](https://geek-cookbook.funkypenguin.co.nz/)recipes/privatebin/)*), no additional layer of authentication will be required.
|
||||
* Where the hosted application provides inadequate (*i.e. [NZBGet](https://geek-cookbook.funkypenguin.co.nz/)recipes/autopirate/nzbget/)*) or no authentication (*i.e. [Gollum](https://geek-cookbook.funkypenguin.co.nz/)recipes/gollum/)*), a further authentication against an OAuth provider will be required.
|
||||
|
||||
|
||||
## High availability
|
||||
@@ -92,4 +92,4 @@ In summary, although I suffered an **unplanned power outage to all of my infrast
|
||||
[^1]: Since there's no impact to availability, I can fix (or just reinstall) the failed node whenever convenient.
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
## Chef's Notes 📓
|
||||
@@ -128,7 +128,7 @@ networks:
|
||||
```
|
||||
|
||||
!!! note
|
||||
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
|
||||
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](https://geek-cookbook.funkypenguin.co.nz/)reference/networks/) here.
|
||||
|
||||
Launch the cleanup stack by running ```docker stack deploy docker-cleanup -c <path-to-docker-compose.yml>```
|
||||
|
||||
@@ -167,10 +167,9 @@ Launch shepherd by running ```docker stack deploy shepherd -c /var/data/config/s
|
||||
|
||||
### Summary
|
||||
|
||||
!!! summary
|
||||
Created
|
||||
After completing the above, you should have:
|
||||
|
||||
* [X] [Docker swarm cluster](/ha-docker-swarm/design/)
|
||||
* [X] [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/design/)
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
## Chef's Notes 📓
|
||||
@@ -68,4 +68,4 @@ That's it. Each node will talk to the other via unicast (no need to un-firewall
|
||||
## Chef's notes 📓
|
||||
|
||||
1. Some hosting platforms (*OpenStack, for one*) won't allow you to simply "claim" a virtual IP. Each node is only able to receive traffic targetted to its unique IP, unless certain security controls are disabled by the cloud administrator. In this case, keepalived is not the right solution, and a platform-specific load-balancing solution should be used. In OpenStack, this is Neutron's "Load Balancer As A Service" (LBAAS) component. AWS, GCP and Azure would likely include similar protections.
|
||||
2. More than 2 nodes can participate in keepalived. Simply ensure that each node has the appropriate priority set, and the node with the highest priority will become the master.
|
||||
2. More than 2 nodes can participate in keepalived. Simply ensure that each node has the appropriate priority set, and the node with the highest priority will become the master.
|
||||
@@ -3,7 +3,7 @@
|
||||
Let's start building our cluster. You can use either bare-metal machines or virtual machines - the configuration would be the same. To avoid confusion, I'll be referring to these as "nodes" from now on.
|
||||
|
||||
!!! note
|
||||
In 2017, I **initially** chose the "[Atomic](https://www.projectatomic.io/)" CentOS/Fedora image for the swarm hosts, but later found its outdated version of Docker to be problematic with advanced features like GPU transcoding (in [Plex](/recipes/plex/)), [Swarmprom](/recipes/swarmprom/), etc. In the end, I went mainstream and simply preferred a modern Ubuntu installation.
|
||||
In 2017, I **initially** chose the "[Atomic](https://www.projectatomic.io/)" CentOS/Fedora image for the swarm hosts, but later found its outdated version of Docker to be problematic with advanced features like GPU transcoding (in [Plex](https://geek-cookbook.funkypenguin.co.nz/)recipes/plex/)), [Swarmprom](https://geek-cookbook.funkypenguin.co.nz/)recipes/swarmprom/), etc. In the end, I went mainstream and simply preferred a modern Ubuntu installation.
|
||||
|
||||
## Ingredients
|
||||
|
||||
|
||||
@@ -10,8 +10,8 @@ The registry mirror runs as a swarm stack, using a simple docker-compose.yml. Cu
|
||||
|
||||
## Ingredients
|
||||
|
||||
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
|
||||
2. [Traefik](/ha-docker-swarm/traefik) configured per design
|
||||
1. [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/design/) with [persistent shared storage](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/shared-storage-ceph.md)
|
||||
2. [Traefik](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/traefik) configured per design
|
||||
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
|
||||
|
||||
|
||||
@@ -110,4 +110,4 @@ systemctl restart docker-latest
|
||||
!!! tip ""
|
||||
Note the extra comma required after "false" above
|
||||
|
||||
## Chef's notes 📓
|
||||
## Chef's notes 📓
|
||||
@@ -2,196 +2,217 @@
|
||||
|
||||
While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node.
|
||||
|
||||
## Design
|
||||
|
||||
### Why not GlusterFS?
|
||||
I originally provided shared storage to my nodes using GlusterFS (see the next recipe for details), but found it difficult to deal with because:
|
||||
|
||||
1. GlusterFS requires (n) "bricks", where (n) **has** to be a multiple of your replica count. I.e., if you want 2 copies of everything on shared storage (the minimum to provide redundancy), you **must** have either 2, 4, 6 (etc..) bricks. The HA swarm design calls for minimum of 3 nodes, and so under GlusterFS, my third node can't participate in shared storage at all, unless I start doubling up on bricks-per-node (which then impacts redundancy)
|
||||
2. GlusterFS turns out to be a giant PITA when you want to restore a failed node. There are at [least 14 steps to follow](https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/sect-Replacing_Hosts.html) to replace a brick.
|
||||
3. I'm pretty sure I messed up the 14-step process above anyway. My replaced brick synced with my "original" brick, but produced errors when querying status via the CLI, and hogged 100% of 1 CPU on the replaced node. Inexperienced with GlusterFS, and unable to diagnose the fault, I switched to a Ceph cluster instead.
|
||||
|
||||
### Why Ceph?
|
||||
|
||||
1. I'm more familiar with Ceph - I use it in the OpenStack designs I manage
|
||||
2. Replacing a failed node is **easy**, provided you can put up with the I/O load of rebalancing OSDs after the replacement.
|
||||
3. CentOS Atomic includes the ceph client in the OS, so while the Ceph OSD/Mon/MSD are running under containers, I can keep an eye (and later, automatically monitor) the status of Ceph from the base OS.
|
||||
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
3 x Virtual Machines (configured earlier), each with:
|
||||
|
||||
* [X] CentOS/Fedora Atomic
|
||||
* [X] Support for "modern" versions of Python and LVM
|
||||
* [X] At least 1GB RAM
|
||||
* [X] At least 20GB disk space (_but it'll be tight_)
|
||||
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
|
||||
* [ ] A second disk dedicated to the Ceph OSD
|
||||
* [X] A second disk dedicated to the Ceph OSD
|
||||
* [X] Each node should have the IP of every other participating node hard-coded in /etc/hosts (*including its own IP*)
|
||||
|
||||
## Preparation
|
||||
|
||||
### SELinux
|
||||
!!! tip "No more [foolish games](https://www.youtube.com/watch?v=UNoouLa7uxA)"
|
||||
Earlier iterations of this recipe (*based on [Ceph Jewel](https://docs.ceph.com/docs/master/releases/jewel/)*) required significant manual effort to install Ceph in a Docker environment. In the 2+ years since Jewel was released, significant improvements have been made to the ceph "deploy-in-docker" process, including the [introduction of the cephadm tool](https://ceph.io/ceph-management/introducing-cephadm/). Cephadm is the tool which now does all the heavy lifting, below, for the current version of ceph, codenamed "[Octopus](https://www.youtube.com/watch?v=Gi58pN8W3hY)".
|
||||
|
||||
Since our Ceph components will be containerized, we need to ensure the SELinux context on the base OS's ceph files is set correctly:
|
||||
### Pick a master node
|
||||
|
||||
One of your nodes will become the cephadm "master" node. Although all nodes will participate in the Ceph cluster, the master node will be the node which we bootstrap ceph on. It's also the node which will run the Ceph dashboard, and on which future upgrades will be processed. It doesn't matter _which_ node you pick, and the cluster itself will operate in the event of a loss of the master node (although you won't see the dashboard)
|
||||
|
||||
### Install cephadm on master node
|
||||
|
||||
Run the following on the ==master== node:
|
||||
|
||||
```
|
||||
mkdir /var/lib/ceph
|
||||
chcon -Rt svirt_sandbox_file_t /etc/ceph
|
||||
chcon -Rt svirt_sandbox_file_t /var/lib/ceph
|
||||
```
|
||||
### Setup Monitors
|
||||
|
||||
Pick a node, and run the following to stand up the first Ceph mon. Be sure to replace the values for **MON_IP** and **CEPH_PUBLIC_NETWORK** to those specific to your deployment:
|
||||
|
||||
```
|
||||
docker run -d --net=host \
|
||||
--restart always \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-e MON_IP=192.168.31.11 \
|
||||
-e CEPH_PUBLIC_NETWORK=192.168.31.0/24 \
|
||||
--name="ceph-mon" \
|
||||
ceph/daemon mon
|
||||
MYIP=`ip route get 1.1.1.1 | grep -oP 'src \K\S+'`
|
||||
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
|
||||
chmod +x cephadm
|
||||
mkdir -p /etc/ceph
|
||||
./cephadm bootstrap --mon-ip $MYIP
|
||||
```
|
||||
|
||||
Now **copy** the contents of /etc/ceph on this first node to the remaining nodes, and **then** run the docker command above (_customizing MON_IP as you go_) on each remaining node. You'll end up with a cluster with 3 monitors (odd number is required for quorum, same as Docker Swarm), and no OSDs (yet)
|
||||
The process takes about 30 seconds, after which, you'll have a MVC (*Minimum Viable Cluster*)[^1], encompassing a single monitor and mgr instance on your chosen node. Here's the complete output from a fresh install:
|
||||
|
||||
### Setup Managers
|
||||
??? "Example output from a fresh cephadm bootstrap"
|
||||
```
|
||||
root@raphael:~# MYIP=`ip route get 1.1.1.1 | grep -oP 'src \K\S+'`
|
||||
root@raphael:~# curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
|
||||
|
||||
Since Ceph v12 ("Luminous"), some of the non-realtime cluster management responsibilities are delegated to a "manager". Run the following on every node - only one node will be __active__, the others will be in standby:
|
||||
root@raphael:~# chmod +x cephadm
|
||||
root@raphael:~# mkdir -p /etc/ceph
|
||||
root@raphael:~# ./cephadm bootstrap --mon-ip $MYIP
|
||||
INFO:cephadm:Verifying podman|docker is present...
|
||||
INFO:cephadm:Verifying lvm2 is present...
|
||||
INFO:cephadm:Verifying time synchronization is in place...
|
||||
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
|
||||
INFO:cephadm:Repeating the final host check...
|
||||
INFO:cephadm:podman|docker (https://geek-cookbook.funkypenguin.co.nz/)usr/bin/docker) is present
|
||||
INFO:cephadm:systemctl is present
|
||||
INFO:cephadm:lvcreate is present
|
||||
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
|
||||
INFO:cephadm:Host looks OK
|
||||
INFO:root:Cluster fsid: bf3eff78-9e27-11ea-b40a-525400380101
|
||||
INFO:cephadm:Verifying IP 192.168.38.101 port 3300 ...
|
||||
INFO:cephadm:Verifying IP 192.168.38.101 port 6789 ...
|
||||
INFO:cephadm:Mon IP 192.168.38.101 is in CIDR network 192.168.38.0/24
|
||||
INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container...
|
||||
INFO:cephadm:Extracting ceph user uid/gid from container image...
|
||||
INFO:cephadm:Creating initial keys...
|
||||
INFO:cephadm:Creating initial monmap...
|
||||
INFO:cephadm:Creating mon...
|
||||
INFO:cephadm:Waiting for mon to start...
|
||||
INFO:cephadm:Waiting for mon...
|
||||
INFO:cephadm:mon is available
|
||||
INFO:cephadm:Assimilating anything we can from ceph.conf...
|
||||
INFO:cephadm:Generating new minimal ceph.conf...
|
||||
INFO:cephadm:Restarting the monitor...
|
||||
INFO:cephadm:Setting mon public_network...
|
||||
INFO:cephadm:Creating mgr...
|
||||
INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring
|
||||
INFO:cephadm:Wrote config to /etc/ceph/ceph.conf
|
||||
INFO:cephadm:Waiting for mgr to start...
|
||||
INFO:cephadm:Waiting for mgr...
|
||||
INFO:cephadm:mgr not available, waiting (1/10)...
|
||||
INFO:cephadm:mgr not available, waiting (2/10)...
|
||||
INFO:cephadm:mgr not available, waiting (3/10)...
|
||||
INFO:cephadm:mgr is available
|
||||
INFO:cephadm:Enabling cephadm module...
|
||||
INFO:cephadm:Waiting for the mgr to restart...
|
||||
INFO:cephadm:Waiting for Mgr epoch 5...
|
||||
INFO:cephadm:Mgr epoch 5 is available
|
||||
INFO:cephadm:Setting orchestrator backend to cephadm...
|
||||
INFO:cephadm:Generating ssh key...
|
||||
INFO:cephadm:Wrote public SSH key to to /etc/ceph/ceph.pub
|
||||
INFO:cephadm:Adding key to root@localhost's authorized_keys...
|
||||
INFO:cephadm:Adding host raphael...
|
||||
INFO:cephadm:Deploying mon service with default placement...
|
||||
INFO:cephadm:Deploying mgr service with default placement...
|
||||
INFO:cephadm:Deploying crash service with default placement...
|
||||
INFO:cephadm:Enabling mgr prometheus module...
|
||||
INFO:cephadm:Deploying prometheus service with default placement...
|
||||
INFO:cephadm:Deploying grafana service with default placement...
|
||||
INFO:cephadm:Deploying node-exporter service with default placement...
|
||||
INFO:cephadm:Deploying alertmanager service with default placement...
|
||||
INFO:cephadm:Enabling the dashboard module...
|
||||
INFO:cephadm:Waiting for the mgr to restart...
|
||||
INFO:cephadm:Waiting for Mgr epoch 13...
|
||||
INFO:cephadm:Mgr epoch 13 is available
|
||||
INFO:cephadm:Generating a dashboard self-signed certificate...
|
||||
INFO:cephadm:Creating initial admin user...
|
||||
INFO:cephadm:Fetching dashboard port number...
|
||||
INFO:cephadm:Ceph Dashboard is now available at:
|
||||
|
||||
```
|
||||
docker run -d --net=host \
|
||||
--privileged=true \
|
||||
--pid=host \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
--name="ceph-mgr" \
|
||||
--restart=always \
|
||||
ceph/daemon mgr
|
||||
```
|
||||
URL: https://raphael:8443/
|
||||
User: admin
|
||||
Password: mid28k0yg5
|
||||
|
||||
### Setup OSDs
|
||||
INFO:cephadm:You can access the Ceph CLI with:
|
||||
|
||||
Since we have a OSD-less mon-only cluster currently, prepare for OSD creation by dumping the auth credentials for the OSDs into the appropriate location on the base OS:
|
||||
sudo ./cephadm shell --fsid bf3eff78-9e27-11ea-b40a-525400380101 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
|
||||
|
||||
```
|
||||
ceph auth get client.bootstrap-osd -o \
|
||||
/var/lib/ceph/bootstrap-osd/ceph.keyring
|
||||
```
|
||||
INFO:cephadm:Please consider enabling telemetry to help improve Ceph:
|
||||
|
||||
On each node, you need a dedicated disk for the OSD. In the example below, I used _/dev/vdd_ (the entire disk, no partitions) for the OSD.
|
||||
ceph telemetry on
|
||||
|
||||
Run the following command on every node:
|
||||
For more information see:
|
||||
|
||||
```
|
||||
docker run -d --net=host \
|
||||
--privileged=true \
|
||||
--pid=host \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /dev/:/dev/ \
|
||||
-e OSD_FORCE_ZAP=1 \
|
||||
-e OSD_DEVICE=/dev/vdd \
|
||||
-e OSD_TYPE=disk \
|
||||
--name="ceph-osd" \
|
||||
--restart=always \
|
||||
ceph/daemon osd_ceph_disk
|
||||
```
|
||||
https://docs.ceph.com/docs/master/mgr/telemetry/
|
||||
|
||||
Watch the output by running ```docker logs ceph-osd -f```, and confirm success.
|
||||
|
||||
!!! warning "Zapping the device"
|
||||
The Ceph OSD container will normally refuse to destroy a partition containing existing data, but above we are instructing ceph to zap (destroy) whatever is on the partition currently. Don't run this against a device you care about, and if you're unsure, omit the "OSD_FORCE_ZAP" variable
|
||||
|
||||
### Setup MDSs
|
||||
|
||||
In order to mount our ceph pools as filesystems, we'll need Ceph MDS(s). Run the following on each node:
|
||||
|
||||
```
|
||||
docker run -d --net=host \
|
||||
--name ceph-mds \
|
||||
--restart always \
|
||||
-v /var/lib/ceph/:/var/lib/ceph/ \
|
||||
-v /etc/ceph:/etc/ceph \
|
||||
-e CEPHFS_CREATE=1 \
|
||||
-e CEPHFS_DATA_POOL_PG=256 \
|
||||
-e CEPHFS_METADATA_POOL_PG=256 \
|
||||
ceph/daemon mds
|
||||
```
|
||||
### Apply tweaks
|
||||
|
||||
The ceph container seems to configure a pool default of 3 replicas (3 copies of each block are retained), which is one too many for our cluster (we are only protecting against the failure of a single node).
|
||||
|
||||
Run the following on any node to reduce the size of the pool to 2 replicas:
|
||||
|
||||
```
|
||||
ceph osd pool set cephfs_data size 2
|
||||
ceph osd pool set cephfs_metadata size 2
|
||||
```
|
||||
|
||||
Disabled "scrubbing" (which can be IO-intensive, and is unnecessary on a VM) with:
|
||||
|
||||
```
|
||||
ceph osd set noscrub
|
||||
ceph osd set nodeep-scrub
|
||||
```
|
||||
INFO:cephadm:Bootstrap complete.
|
||||
root@raphael:~#
|
||||
```
|
||||
|
||||
|
||||
### Create credentials for swarm
|
||||
### Prepare other nodes
|
||||
|
||||
In order to mount the ceph volume onto our base host, we need to provide cephx authentication credentials.
|
||||
It's now necessary to tranfer the following files to your ==other== nodes, so that cephadm can add them to your cluster, and so that they'll be able to mount the cephfs when we're done:
|
||||
|
||||
On **one** node, create a client for the docker swarm:
|
||||
Path on master | Path on non-master
|
||||
--------------- | -----
|
||||
`/etc/ceph/ceph.conf` | `/etc/ceph/ceph.conf`
|
||||
`/etc/ceph/ceph.client.admin.keyring` | `/etc/ceph/ceph.client.admin.keyring`
|
||||
`/etc/ceph/ceph.pub` | `/root/.ssh/authorized_keys` (append to anything existing)
|
||||
|
||||
```
|
||||
ceph auth get-or-create client.dockerswarm osd \
|
||||
'allow rw' mon 'allow r' mds 'allow' > /etc/ceph/keyring.dockerswarm
|
||||
```
|
||||
|
||||
Grab the secret associated with the new user (you'll need this for the /etc/fstab entry below) by running:
|
||||
Back on the ==master== node, run `ceph orch host add <node-name>` once for each other node you want to join to the cluster. You can validate the results by running `ceph orch host ls`
|
||||
|
||||
```
|
||||
ceph-authtool /etc/ceph/keyring.dockerswarm -p -n client.dockerswarm
|
||||
```
|
||||
!!! question "Should we be concerned about giving cephadm using root access over SSH?"
|
||||
Not really. Docker is inherently insecure at the host-level anyway (*think what would happen if you launched a global-mode stack with a malicious container image which mounted `/root/.ssh`*), so worrying about cephadm seems a little barn-door-after-horses-bolted. If you take host-level security seriously, consider switching to [Kubernetes](https://geek-cookbook.funkypenguin.co.nz/)kubernetes/start/) :)
|
||||
|
||||
### Mount MDS volume
|
||||
### Add OSDs
|
||||
|
||||
On each node, create a mountpoint for the data, by running ```mkdir /var/data```, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually _mounted_ if there's a network / boot delay getting access to the gluster volume:
|
||||
Now the best improvement since the days of ceph-deploy and manual disks.. on the ==master== node, run `ceph orch apply osd --all-available-devices`. This will identify any unloved (*unpartitioned, unmounted*) disks attached to each participating node, and configure these disks as OSDs.
|
||||
|
||||
### Setup CephFS
|
||||
|
||||
On the ==master== node, create a cephfs volume in your cluster, by running `ceph fs volume create data`. Ceph will handle the necessary orchestration itself, creating the necessary pool, mds daemon, etc.
|
||||
|
||||
You can watch the progress by running `ceph fs ls` (to see the fs is configured), and `ceph -s` to wait for `HEALTH_OK`
|
||||
|
||||
### Mount CephFS volume
|
||||
|
||||
On ==every== node, create a mountpoint for the data, by running ```mkdir /var/data```, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually _mounted_ if there's a network / boot delay getting access to the gluster volume:
|
||||
|
||||
```
|
||||
mkdir /var/data
|
||||
|
||||
MYHOST=`hostname -s`
|
||||
MYNODES="<node1>,<node2>,<node3>" # Add your own nodes here, comma-delimited
|
||||
MYHOST=`ip route get 1.1.1.1 | grep -oP 'src \K\S+'`
|
||||
echo -e "
|
||||
# Mount cephfs volume \n
|
||||
$MYHOST:6789:/ /var/data/ ceph \
|
||||
name=dockerswarm\
|
||||
,secret=<YOUR SECRET HERE>\
|
||||
,noatime,_netdev,context=system_u:object_r:svirt_sandbox_file_t:s0 \
|
||||
0 2" >> /etc/fstab
|
||||
raphael,donatello,leonardo:/ /var/data ceph name=admin,noatime,_netdev 0 0" >> /etc/fstab
|
||||
mount -a
|
||||
```
|
||||
### Install docker-volume plugin
|
||||
|
||||
Upstream bug for docker-latest reported at https://bugs.centos.org/view.php?id=13609
|
||||
|
||||
And the alpine fault:
|
||||
https://github.com/gliderlabs/docker-alpine/issues/317
|
||||
|
||||
|
||||
## Serving
|
||||
|
||||
After completing the above, you should have:
|
||||
### Sprinkle with tools
|
||||
|
||||
Although it's possible to use `cephadm shell` to exec into a container with the necessary ceph tools, it's more convenient to use the native CLI tools. To this end, on each node, run the following, which will install the appropriate apt repository, and install the latest ceph CLI tools:
|
||||
|
||||
```
|
||||
[X] Persistent storage available to every node
|
||||
[X] Resiliency in the event of the failure of a single node
|
||||
curl -L https://download.ceph.com/keys/release.asc | sudo apt-key add -
|
||||
cephadm add-repo --release octopus
|
||||
cephadm install ceph-common
|
||||
```
|
||||
|
||||
### Drool over dashboard
|
||||
|
||||
Ceph now includes a comprehensive dashboard, provided by the mgr daemon. The dashboard will be accessible at https://[IP of your ceph master node]:8443, but you'll need to run `ceph dashboard ac-user-create <username> <password> administrator` first, to create an administrator account:
|
||||
|
||||
```
|
||||
root@raphael:~# ceph dashboard ac-user-create batman supermansucks administrator
|
||||
{"username": "batman", "password": "$2b$12$3HkjY85mav.dq3HHAZiWP.KkMiuoV2TURZFH.6WFfo/BPZCT/0gr.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1590372281, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": false}
|
||||
root@raphael:~#
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
What have we achieved?
|
||||
|
||||
!!! summary "Summary"
|
||||
Created:
|
||||
|
||||
* [X] Persistent storage available to every node
|
||||
* [X] Resiliency in the event of the failure of a single node
|
||||
* [X] Beautiful dashboard
|
||||
|
||||
## The easy, 5-minute install
|
||||
|
||||
I share (_with [sponsors][github_sponsor] and [patrons][patreon]_) a private "_premix_" GitHub repository, which includes an ansible playbook for deploying the entire Geek's Cookbook stack, automatically. This means that members can create the entire environment with just a ```git pull``` and an ```ansible-playbook deploy.yml```
|
||||
|
||||
Here's a screencast of the playbook in action. I sped up the boring parts, it actually takes ==5 min== (*you can tell by the timestamps on the prompt*):
|
||||
|
||||
|
||||
[patreon]: https://www.patreon.com/bePatron?u=6982506
|
||||
[github_sponsor]: https://github.com/sponsors/funkypenguin
|
||||
|
||||
## Chef's Notes 📓
|
||||
|
||||
Future enhancements to this recipe include:
|
||||
|
||||
1. Rather than pasting a secret key into /etc/fstab (which feels wrong), I'd prefer to be able to set "secretfile" in /etc/fstab (which just points ceph.mount to a file containing the secret), but under the current CentOS Atomic, we're stuck with "secret", per https://bugzilla.redhat.com/show_bug.cgi?id=1030402
|
||||
2. This recipe was written with Ceph v11 "Jewel". Ceph have subsequently releaesd v12 "Kraken". I've updated the recipe for the addition of "Manager" daemons, but it should be noted that the [only reader so far](https://discourse.geek-kitchen.funkypenguin.co.nz/u/ggilley) to attempt a Ceph install using CentOS Atomic and Ceph v12 had issues with OSDs, which lead him to [move to Ubuntu 1604](https://discourse.geek-kitchen.funkypenguin.co.nz/t/shared-storage-ceph-funky-penguins-geek-cookbook/47/24?u=funkypenguin) instead.
|
||||
[^1]: Minimum Viable Cluster acronym copyright, trademark, and whatever else, to Funky Penguin for 1,000,000 years.
|
||||
@@ -3,7 +3,7 @@
|
||||
While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node.
|
||||
|
||||
!!! warning
|
||||
This recipe is deprecated. It didn't work well in 2017, and it's not likely to work any better now. It remains here as a reference. I now recommend the use of [Ceph for shared storage](/ha-docker-swarm/shared-storage-ceph/) instead. - 2019 Chef
|
||||
This recipe is deprecated. It didn't work well in 2017, and it's not likely to work any better now. It remains here as a reference. I now recommend the use of [Ceph for shared storage](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/shared-storage-ceph/) instead. - 2019 Chef
|
||||
|
||||
## Design
|
||||
|
||||
@@ -154,14 +154,12 @@ For non-gluster nodes, you'll need to replace $MYHOST above with the name of one
|
||||
|
||||
After completing the above, you should have:
|
||||
|
||||
```
|
||||
[X] Persistent storage available to every node
|
||||
[X] Resiliency in the event of the failure of a single (gluster) node
|
||||
```
|
||||
* [X] Persistent storage available to every node
|
||||
* [X] Resiliency in the event of the failure of a single (gluster) node
|
||||
|
||||
## Chef's Notes 📓
|
||||
|
||||
Future enhancements to this recipe include:
|
||||
|
||||
1. Migration of shared storage from GlusterFS to Ceph ()[#2](https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook/issues/2))
|
||||
2. Correct the fact that volumes don't automount on boot ([#3](https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook/issues/3))
|
||||
2. Correct the fact that volumes don't automount on boot ([#3](https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook/issues/3))
|
||||
@@ -2,28 +2,28 @@
|
||||
|
||||
Now that we have Traefik deployed, automatically exposing SSL access to our Docker Swarm services using LetsEncrypt wildcard certificates, let's pause to consider that we may not _want_ some services exposed directly to the internet...
|
||||
|
||||
..Wait, why not? Well, Traefik doesn't provide any form of authentication, it simply secures the **transmission** of the service between Docker Swarm and the end user. If you were to deploy a service with no native security (*[Radarr](/recipes/autopirate/radarr/) or [Sonarr](/recipes/autopirate/sonarr/) come to mind*), then anybody would be able to use it! Even services which _may_ have a layer of authentication **might** not be safe to expose publically - often open source projects may be maintained by enthusiasts who happily add extra features, but just pay lip service to security, on the basis that "*it's the user's problem to secure it in their own network*".
|
||||
..Wait, why not? Well, Traefik doesn't provide any form of authentication, it simply secures the **transmission** of the service between Docker Swarm and the end user. If you were to deploy a service with no native security (*[Radarr](https://geek-cookbook.funkypenguin.co.nz/)recipes/autopirate/radarr/) or [Sonarr](https://geek-cookbook.funkypenguin.co.nz/)recipes/autopirate/sonarr/) come to mind*), then anybody would be able to use it! Even services which _may_ have a layer of authentication **might** not be safe to expose publically - often open source projects may be maintained by enthusiasts who happily add extra features, but just pay lip service to security, on the basis that "*it's the user's problem to secure it in their own network*".
|
||||
|
||||
To give us confidence that **we** can access our services, but BadGuys(tm) cannot, we'll deploy a layer of authentication **in front** of Traefik, using [Forward Authentication](https://docs.traefik.io/configuration/entrypoints/#forward-authentication). You can use your own [KeyCloak](/recipes/keycloak/) instance for authentication, but to lower the barrier to entry, this recipe will assume you're authenticating against your own Google account.
|
||||
To give us confidence that **we** can access our services, but BadGuys(tm) cannot, we'll deploy a layer of authentication **in front** of Traefik, using [Forward Authentication](https://docs.traefik.io/configuration/entrypoints/#forward-authentication). You can use your own [KeyCloak](https://geek-cookbook.funkypenguin.co.nz/)recipes/keycloak/) instance for authentication, but to lower the barrier to entry, this recipe will assume you're authenticating against your own Google account.
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
Existing:
|
||||
|
||||
* [X] [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph)
|
||||
* [X] [Traefik](/ha-docker-swarm/traefik/) configured per design
|
||||
* [X] [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/design/) with [persistent shared storage](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/shared-storage-ceph)
|
||||
* [X] [Traefik](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/traefik/) configured per design
|
||||
|
||||
New:
|
||||
|
||||
* [ ] Client ID and secret from an OpenID-Connect provider (Google, [KeyCloak](/recipes/keycloak/), Microsoft, etc..)
|
||||
* [ ] Client ID and secret from an OpenID-Connect provider (Google, [KeyCloak](https://geek-cookbook.funkypenguin.co.nz/)recipes/keycloak/), Microsoft, etc..)
|
||||
|
||||
## Preparation
|
||||
|
||||
### Obtain OAuth credentials
|
||||
|
||||
!!! note
|
||||
This recipe will demonstrate using Google OAuth for traefik forward authentication, but it's also possible to use a self-hosted KeyCloak instance - see the [KeyCloak OIDC Provider](/recipes/keycloak/setup-oidc-provider/) recipe for more details!
|
||||
This recipe will demonstrate using Google OAuth for traefik forward authentication, but it's also possible to use a self-hosted KeyCloak instance - see the [KeyCloak OIDC Provider](https://geek-cookbook.funkypenguin.co.nz/)recipes/keycloak/setup-oidc-provider/) recipe for more details!
|
||||
|
||||
Log into https://console.developers.google.com/, create a new project then search for and select "Credentials" in the search bar.
|
||||
|
||||
@@ -48,7 +48,7 @@ COOKIE_DOMAINS=example.com
|
||||
|
||||
### Prepare the docker service config
|
||||
|
||||
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik](/recipes/traefik/) recipe:
|
||||
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik](https://geek-cookbook.funkypenguin.co.nz/)recipes/traefik/) recipe:
|
||||
|
||||
```
|
||||
traefik-forward-auth:
|
||||
@@ -83,7 +83,7 @@ If you're not confident that forward authentication is working, add a simple "wh
|
||||
```
|
||||
|
||||
!!! tip
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy```
|
||||
|
||||
|
||||
|
||||
@@ -110,7 +110,7 @@ What have we achieved? By adding an additional three simple labels to any servic
|
||||
|
||||
## Chef's Notes 📓
|
||||
|
||||
1. Traefik forward auth replaces the use of [oauth_proxy containers](/reference/oauth_proxy/) found in some of the existing recipes
|
||||
1. Traefik forward auth replaces the use of [oauth_proxy containers](https://geek-cookbook.funkypenguin.co.nz/)reference/oauth_proxy/) found in some of the existing recipes
|
||||
2. [@thomaseddon's original version](https://github.com/thomseddon/traefik-forward-auth) of traefik-forward-auth only works with Google currently, but I've created a [fork](https://www.github.com/funkypenguin/traefik-forward-auth) of a [fork](https://github.com/noelcatt/traefik-forward-auth), which implements generic OIDC providers.
|
||||
3. I reviewed several implementations of forward authenticators for Traefik, but found most to be rather heavy-handed, or specific to a single auth provider. @thomaseddon's go-based docker image is 7MB in size, and with the generic OIDC patch (above), it can be extended to work with any OIDC provider.
|
||||
4. No, not github natively, but you can ferderate GitHub into KeyCloak, and then use KeyCloak as the OIDC provider.
|
||||
@@ -1,13 +1,13 @@
|
||||
# Using Traefik Forward Auth with KeyCloak
|
||||
|
||||
While the [Traefik Forward Auth](/ha-docker-swarm/traefik-forward-auth/) recipe demonstrated a quick way to protect a set of explicitly-specified URLs using OIDC credentials from a Google account, this recipe will illustrate how to use your own KeyCloak instance to secure **any** URLs within your DNS domain.
|
||||
While the [Traefik Forward Auth](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/traefik-forward-auth/) recipe demonstrated a quick way to protect a set of explicitly-specified URLs using OIDC credentials from a Google account, this recipe will illustrate how to use your own KeyCloak instance to secure **any** URLs within your DNS domain.
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! Summary
|
||||
Existing:
|
||||
|
||||
* [X] [KeyCloak](/recipes/keycloak/) recipe deployed successfully, with a [local user](/recipes/keycloak/create-user/) and an [OIDC client](/recipes/keycloak/setup-oidc-provider/)
|
||||
* [X] [KeyCloak](https://geek-cookbook.funkypenguin.co.nz/)recipes/keycloak/) recipe deployed successfully, with a [local user](https://geek-cookbook.funkypenguin.co.nz/)recipes/keycloak/create-user/) and an [OIDC client](https://geek-cookbook.funkypenguin.co.nz/)recipes/keycloak/setup-oidc-provider/)
|
||||
|
||||
New:
|
||||
|
||||
@@ -48,7 +48,7 @@ COOKIE_DOMAIN=<the root FQDN of your domain>
|
||||
|
||||
### Prepare the docker service config
|
||||
|
||||
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik](/recipes/traefik/) recipe:
|
||||
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik](https://geek-cookbook.funkypenguin.co.nz/)recipes/traefik/) recipe:
|
||||
|
||||
```
|
||||
traefik-forward-auth:
|
||||
@@ -82,7 +82,7 @@ If you're not confident that forward authentication is working, add a simple "wh
|
||||
```
|
||||
|
||||
!!! tip
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy```
|
||||
|
||||
## Serving
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ To deal with these gaps, we need a front-end load-balancer, and in this design,
|
||||
!!! summary "You'll need"
|
||||
Existing
|
||||
|
||||
* [X] [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph)
|
||||
* [X] [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/design/) with [persistent shared storage](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/shared-storage-ceph)
|
||||
|
||||
New
|
||||
|
||||
@@ -123,7 +123,7 @@ networks:
|
||||
```
|
||||
|
||||
!!! tip
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy```
|
||||
|
||||
|
||||
Create `/var/data/config/traefik/traefik-app.yml` as follows:
|
||||
@@ -222,7 +222,7 @@ ID NAME IMAGE
|
||||
|
||||
You should now be able to access your traefik instance on http://<node IP\>:8080 - It'll look a little lonely currently (*below*), but we'll populate it as we add recipes :)
|
||||
|
||||
|
||||
images/traefik-post-launch.png)
|
||||
|
||||
### Summary
|
||||
|
||||
@@ -236,4 +236,4 @@ You should now be able to access your traefik instance on http://<node IP\>:8080
|
||||
|
||||
## Chef's Notes 📓
|
||||
|
||||
1. Did you notice how no authentication was required to view the Traefik dashboard? Eek! We'll tackle that in the next section, regarding [Traefik Forward Authentication](/ha-docker-swarm/traefik-forward-auth/)!
|
||||
1. Did you notice how no authentication was required to view the Traefik dashboard? Eek! We'll tackle that in the next section, regarding [Traefik Forward Authentication](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/traefik-forward-auth/)!
|
||||
Reference in new issue
Block a user