mirror of
https://github.com/funkypenguin/geek-cookbook/
synced 2026-11-02 05:48:18 +00:00
Update for leanpub preview
This commit is contained in:
AutoPenguin committed
1 parent
65dd34c7ea
commit
2e8e16157b
193 files changed
+12667
-155
No files matched your search
@@ -92,4 +92,4 @@ In summary, although I suffered an **unplanned power outage to all of my infrast
|
||||
[^1]: Since there's no impact to availability, I can fix (or just reinstall) the failed node whenever convenient.
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
## Chef's Notes
|
||||
@@ -0,0 +1,95 @@
|
||||
# Design
|
||||
|
||||
In the design described below, our "private cloud" platform is:
|
||||
|
||||
* **Highly-available** (_can tolerate the failure of a single component_)
|
||||
* **Scalable** (_can add resource or capacity as required_)
|
||||
* **Portable** (_run it on your garage server today, run it in AWS tomorrow_)
|
||||
* **Secure** (_access protected with [LetsEncrypt certificates](/ha-docker-swarm/traefik/) and optional [OIDC with 2FA](/ha-docker-swarm/traefik-forward-auth/)_)
|
||||
* **Automated** (_requires minimal care and feeding_)
|
||||
|
||||
## Design Decisions
|
||||
|
||||
**Where possible, services will be highly available.**
|
||||
|
||||
This means that:
|
||||
|
||||
* At least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure.
|
||||
* [Ceph](/ha-docker-swarm/shared-storage-ceph/) is employed for share storage, because it too can be made tolerant of a single failure.
|
||||
|
||||
!!! note
|
||||
An exception to the 3-nodes decision is running a single-node configuration. If you only **have** one node, then obviously your swarm is only as resilient as that node. It's still a perfectly valid swarm configuration, ideal for starting your self-hosting journey. In fact, under the single-node configuration, you don't need ceph either, and you can simply use the local volume on your host for storage. You'll be able to migrate to ceph/more nodes if/when you expand.
|
||||
|
||||
**Where multiple solutions to a requirement exist, preference will be given to the most portable solution.**
|
||||
|
||||
This means that:
|
||||
|
||||
* Services are defined using docker-compose v3 YAML syntax
|
||||
* Services are portable, meaning a particular stack could be shut down and moved to a new provider with minimal effort.
|
||||
|
||||
## Security
|
||||
|
||||
Under this design, the only inbound connections we're permitting to our docker swarm in a **minimal** configuration (*you may add custom services later, like UniFi Controller*) are:
|
||||
|
||||
### Network Flows
|
||||
|
||||
* **HTTP (TCP 80)** : Redirects to https
|
||||
* **HTTPS (TCP 443)** : Serves individual docker containers via SSL-encrypted reverse proxy
|
||||
|
||||
### Authentication
|
||||
|
||||
* Where the hosted application provides a trusted level of authentication (*i.e., [NextCloud](/recipes/nextcloud/)*), or where the application requires public exposure (*i.e. [Privatebin](/recipes/privatebin/)*), no additional layer of authentication will be required.
|
||||
* Where the hosted application provides inadequate (*i.e. [NZBGet](/recipes/autopirate/nzbget/)*) or no authentication (*i.e. [Gollum](/recipes/gollum/)*), a further authentication against an OAuth provider will be required.
|
||||
|
||||
|
||||
## High availability
|
||||
|
||||
### Normal function
|
||||
|
||||
Assuming a 3-node configuration, under normal circumstances the following is illustrated:
|
||||
|
||||
* All 3 nodes provide shared storage via Ceph, which is provided by a docker container on each node.
|
||||
* All 3 nodes participate in the Docker Swarm as managers.
|
||||
* The various containers belonging to the application "stacks" deployed within Docker Swarm are automatically distributed amongst the swarm nodes.
|
||||
* Persistent storage for the containers is provide via cephfs mount.
|
||||
* The **traefik** service (*in swarm mode*) receives incoming requests (*on HTTP and HTTPS*), and forwards them to individual containers. Traefik knows the containers names because it's able to read the docker socket.
|
||||
* All 3 nodes run keepalived, at varying priorities. Since traefik is running as a swarm service and listening on TCP 80/443, requests made to the keepalived VIP and arriving at **any** of the swarm nodes will be forwarded to the traefik container (*no matter which node it's on*), and then onto the target backend.
|
||||
|
||||
|
||||
|
||||
### Node failure
|
||||
|
||||
In the case of a failure (or scheduled maintenance) of one of the nodes, the following is illustrated:
|
||||
|
||||
* The failed node no longer participates in Ceph, but the remaining nodes provide enough fault-tolerance for the cluster to operate.
|
||||
* The remaining two nodes in Docker Swarm achieve a quorum and agree that the failed node is to be removed.
|
||||
* The (*possibly new*) leader manager node reschedules the containers known to be running on the failed node, onto other nodes.
|
||||
* The **traefik** service is either restarted or unaffected, and as the backend containers stop/start and change IP, traefik is aware and updates accordingly.
|
||||
* The keepalived VIP continues to function on the remaining nodes, and docker swarm continues to forward any traffic received on TCP 80/443 to the appropriate node.
|
||||
|
||||
|
||||
|
||||
### Node restore
|
||||
|
||||
When the failed (*or upgraded*) host is restored to service, the following is illustrated:
|
||||
|
||||
* Ceph regains full redundancy
|
||||
* Docker Swarm managers become aware of the recovered node, and will use it for scheduling **new** containers
|
||||
* Existing containers which were migrated off the node are not migrated backend
|
||||
* Keepalived VIP regains full redundancy
|
||||
|
||||
|
||||
|
||||
|
||||
### Total cluster failure
|
||||
|
||||
A day after writing this, my environment suffered a fault whereby all 3 VMs were unexpectedly and simultaneously powered off.
|
||||
|
||||
Upon restore, docker failed to start on one of the VMs due to local disk space issue[^1]. However, the other two VMs started, established the swarm, mounted their shared storage, and started up all the containers (services) which were managed by the swarm.
|
||||
|
||||
In summary, although I suffered an **unplanned power outage to all of my infrastructure**, followed by a **failure of a third of my hosts**... ==all my platforms are 100% available with **absolutely no manual intervention**==.
|
||||
|
||||
[^1]: Since there's no impact to availability, I can fix (or just reinstall) the failed node whenever convenient.
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
@@ -172,4 +172,4 @@ After completing the above, you should have:
|
||||
* [X] [Docker swarm cluster](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/design/)
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
## Chef's Notes
|
||||
@@ -0,0 +1,175 @@
|
||||
# Docker Swarm Mode
|
||||
|
||||
For truly highly-available services with Docker containers, we need an orchestration system. Docker Swarm (as defined at 1.13) is the simplest way to achieve redundancy, such that a single docker host could be turned off, and none of our services will be interrupted.
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary
|
||||
Existing
|
||||
|
||||
* [X] 3 x nodes (*bare-metal or VMs*), each with:
|
||||
* A mainstream Linux OS (*tested on either [CentOS](https://www.centos.org) 7+ or [Ubuntu](http://releases.ubuntu.com) 16.04+*)
|
||||
* At least 2GB RAM
|
||||
* At least 20GB disk space (_but it'll be tight_)
|
||||
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
|
||||
|
||||
## Preparation
|
||||
|
||||
### Bash auto-completion
|
||||
|
||||
Add some handy bash auto-completion for docker. Without this, you'll get annoyed that you can't autocomplete ```docker stack deploy <blah> -c <blah.yml>``` commands.
|
||||
|
||||
```
|
||||
cd /etc/bash_completion.d/
|
||||
curl -O https://raw.githubusercontent.com/docker/cli/b75596e1e4d5295ac69b9934d1bd8aff691a0de8/contrib/completion/bash/docker
|
||||
```
|
||||
|
||||
Install some useful bash aliases on each host
|
||||
```
|
||||
cd ~
|
||||
curl -O https://raw.githubusercontent.com/funkypenguin/geek-cookbook/master/examples/scripts/gcb-aliases.sh
|
||||
echo 'source ~/gcb-aliases.sh' >> ~/.bash_profile
|
||||
```
|
||||
|
||||
## Serving
|
||||
|
||||
### Release the swarm!
|
||||
|
||||
Now, to launch a swarm. Pick a target node, and run `docker swarm init`
|
||||
|
||||
Yeah, that was it. Seriously. Now we have a 1-node swarm.
|
||||
|
||||
```
|
||||
[root@ds1 ~]# docker swarm init
|
||||
Swarm initialized: current node (b54vls3wf8xztwfz79nlkivt8) is now a manager.
|
||||
|
||||
To add a worker to this swarm, run the following command:
|
||||
|
||||
docker swarm join \
|
||||
--token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-bsud7xnvhv4cicwi7l6c9s6l0 \
|
||||
202.170.164.47:2377
|
||||
|
||||
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
|
||||
|
||||
[root@ds1 ~]#
|
||||
```
|
||||
|
||||
Run `docker node ls` to confirm that you have a 1-node swarm:
|
||||
|
||||
```
|
||||
[root@ds1 ~]# docker node ls
|
||||
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
|
||||
b54vls3wf8xztwfz79nlkivt8 * ds1.funkypenguin.co.nz Ready Active Leader
|
||||
[root@ds1 ~]#
|
||||
```
|
||||
|
||||
Note that when you run `docker swarm init` above, the CLI output gives youe a command to run to join further nodes to my swarm. This command would join the nodes as __workers__ (*as opposed to __managers__*). Workers can easily be promoted to managers (*and demoted again*), but since we know that we want our other two nodes to be managers too, it's simpler just to add them to the swarm as managers immediately.
|
||||
|
||||
On the first swarm node, generate the necessary token to join another manager by running ```docker swarm join-token manager```:
|
||||
|
||||
```
|
||||
[root@ds1 ~]# docker swarm join-token manager
|
||||
To add a manager to this swarm, run the following command:
|
||||
|
||||
docker swarm join \
|
||||
--token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-cfm24bq2zvfkcwujwlp5zqxta \
|
||||
202.170.164.47:2377
|
||||
|
||||
[root@ds1 ~]#
|
||||
```
|
||||
|
||||
Run the command provided on your other nodes to join them to the swarm as managers. After addition of a node, the output of ```docker node ls``` (on either host) should reflect all the nodes:
|
||||
|
||||
|
||||
```
|
||||
[root@ds2 davidy]# docker node ls
|
||||
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
|
||||
b54vls3wf8xztwfz79nlkivt8 ds1.funkypenguin.co.nz Ready Active Leader
|
||||
xmw49jt5a1j87a6ihul76gbgy * ds2.funkypenguin.co.nz Ready Active Reachable
|
||||
[root@ds2 davidy]#
|
||||
```
|
||||
|
||||
### Setup automated cleanup
|
||||
|
||||
Docker swarm doesn't do any cleanup of old images, so as you experiment with various stacks, and as updated containers are released upstream, you'll soon find yourself loosing gigabytes of disk space to old, unused images.
|
||||
|
||||
To address this, we'll run the "[meltwater/docker-cleanup](https://github.com/meltwater/docker-cleanup)" container on all of our nodes. The container will clean up unused images after 30 minutes.
|
||||
|
||||
First, create docker-cleanup.env (_mine is under /var/data/config/docker-cleanup_), and exclude container images we **know** we want to keep:
|
||||
|
||||
```
|
||||
KEEP_IMAGES=traefik,keepalived,docker-mailserver
|
||||
DEBUG=1
|
||||
```
|
||||
|
||||
Then create a docker-compose.yml as follows:
|
||||
|
||||
```
|
||||
version: "3"
|
||||
|
||||
services:
|
||||
docker-cleanup:
|
||||
image: meltwater/docker-cleanup:latest
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- /var/lib/docker:/var/lib/docker
|
||||
networks:
|
||||
- internal
|
||||
deploy:
|
||||
mode: global
|
||||
env_file: /var/data/config/docker-cleanup/docker-cleanup.env
|
||||
|
||||
networks:
|
||||
internal:
|
||||
driver: overlay
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.16.0.0/24
|
||||
```
|
||||
|
||||
!!! note
|
||||
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
|
||||
|
||||
Launch the cleanup stack by running ```docker stack deploy docker-cleanup -c <path-to-docker-compose.yml>```
|
||||
|
||||
### Setup automatic updates
|
||||
|
||||
If your swarm runs for a long time, you might find yourself running older container images, after newer versions have been released. If you're the sort of geek who wants to live on the edge, configure [shepherd](https://github.com/djmaze/shepherd) to auto-update your container images regularly.
|
||||
|
||||
Create /var/data/config/shepherd/shepherd.env as follows:
|
||||
|
||||
```
|
||||
# Don't auto-update Plex or Emby, I might be watching a movie! (Customize this for the containers you _don't_ want to auto-update)
|
||||
BLACKLIST_SERVICES="plex_plex emby_emby"
|
||||
# Run every 24 hours. Note that SLEEP_TIME appears to be in seconds.
|
||||
SLEEP_TIME=86400
|
||||
```
|
||||
|
||||
Then create /var/data/config/shepherd/shepherd.yml as follows:
|
||||
|
||||
```
|
||||
version: "3"
|
||||
|
||||
services:
|
||||
shepherd-app:
|
||||
image: mazzolino/shepherd
|
||||
env_file : /var/data/config/shepherd/shepherd.env
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
labels:
|
||||
- "traefik.enable=false"
|
||||
deploy:
|
||||
placement:
|
||||
constraints: [node.role == manager]
|
||||
```
|
||||
|
||||
Launch shepherd by running ```docker stack deploy shepherd -c /var/data/config/shepherd/shepherd.yml```, and then just forget about it, comfortable in the knowledge that every day, Shepherd will check that your images are the latest available, and if not, will destroy and recreate the container on the latest available image.
|
||||
|
||||
### Summary
|
||||
|
||||
After completing the above, you should have:
|
||||
|
||||
* [X] [Docker swarm cluster](/ha-docker-swarm/design/)
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
Whitespace-only changes.
@@ -65,7 +65,7 @@ docker run -d --name keepalived --restart=always \
|
||||
|
||||
That's it. Each node will talk to the other via unicast (no need to un-firewall multicast addresses), and the node with the highest priority gets to be the master. When ingress traffic arrives on the master node via the VIP, docker's routing mesh will deliver it to the appropriate docker node.
|
||||
|
||||
## Chef's notes 📓
|
||||
## Chef's notes
|
||||
|
||||
1. Some hosting platforms (*OpenStack, for one*) won't allow you to simply "claim" a virtual IP. Each node is only able to receive traffic targetted to its unique IP, unless certain security controls are disabled by the cloud administrator. In this case, keepalived is not the right solution, and a platform-specific load-balancing solution should be used. In OpenStack, this is Neutron's "Load Balancer As A Service" (LBAAS) component. AWS, GCP and Azure would likely include similar protections.
|
||||
2. More than 2 nodes can participate in keepalived. Simply ensure that each node has the appropriate priority set, and the node with the highest priority will become the master.
|
||||
@@ -0,0 +1,71 @@
|
||||
# Keepalived
|
||||
|
||||
While having a self-healing, scalable docker swarm is great for availability and scalability, none of that is any good if nobody can connect to your cluster.
|
||||
|
||||
In order to provide seamless external access to clustered resources, regardless of which node they're on and tolerant of node failure, you need to present a single IP to the world for external access.
|
||||
|
||||
Normally this is done using a HA loadbalancer, but since Docker Swarm aready provides the load-balancing capabilities (*[routing mesh](https://docs.docker.com/engine/swarm/ingress/)*), all we need for seamless HA is a virtual IP which will be provided by more than one docker node.
|
||||
|
||||
This is accomplished with the use of keepalived on at least two nodes.
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
Already deployed:
|
||||
|
||||
* [X] At least 2 x swarm nodes
|
||||
* [X] low-latency link (i.e., no WAN links)
|
||||
|
||||
New:
|
||||
|
||||
* [ ] At least 3 x IPv4 addresses (one for each node and one for the virtual IP)
|
||||
|
||||
## Preparation
|
||||
|
||||
### Enable IPVS module
|
||||
|
||||
On all nodes which will participate in keepalived, we need the "ip_vs" kernel module, in order to permit serivces to bind to non-local interface addresses.
|
||||
|
||||
Set this up once for both the primary and secondary nodes, by running:
|
||||
|
||||
```
|
||||
echo "modprobe ip_vs" >> /etc/rc.local
|
||||
modprobe ip_vs
|
||||
```
|
||||
|
||||
### Setup nodes
|
||||
|
||||
Assuming your IPs are as follows:
|
||||
|
||||
* 192.168.4.1 : Primary
|
||||
* 192.168.4.2 : Secondary
|
||||
* 192.168.4.3 : Virtual
|
||||
|
||||
Run the following on the primary
|
||||
```
|
||||
docker run -d --name keepalived --restart=always \
|
||||
--cap-add=NET_ADMIN --net=host \
|
||||
-e KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:['192.168.4.1', '192.168.4.2']" \
|
||||
-e KEEPALIVED_VIRTUAL_IPS=192.168.4.3 \
|
||||
-e KEEPALIVED_PRIORITY=200 \
|
||||
osixia/keepalived:1.3.5
|
||||
```
|
||||
|
||||
And on the secondary:
|
||||
```
|
||||
docker run -d --name keepalived --restart=always \
|
||||
--cap-add=NET_ADMIN --net=host \
|
||||
-e KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:['192.168.4.1', '192.168.4.2']" \
|
||||
-e KEEPALIVED_VIRTUAL_IPS=192.168.4.3 \
|
||||
-e KEEPALIVED_PRIORITY=100 \
|
||||
osixia/keepalived:1.3.5
|
||||
```
|
||||
|
||||
## Serving
|
||||
|
||||
That's it. Each node will talk to the other via unicast (no need to un-firewall multicast addresses), and the node with the highest priority gets to be the master. When ingress traffic arrives on the master node via the VIP, docker's routing mesh will deliver it to the appropriate docker node.
|
||||
|
||||
## Chef's notes 📓
|
||||
|
||||
1. Some hosting platforms (*OpenStack, for one*) won't allow you to simply "claim" a virtual IP. Each node is only able to receive traffic targetted to its unique IP, unless certain security controls are disabled by the cloud administrator. In this case, keepalived is not the right solution, and a platform-specific load-balancing solution should be used. In OpenStack, this is Neutron's "Load Balancer As A Service" (LBAAS) component. AWS, GCP and Azure would likely include similar protections.
|
||||
2. More than 2 nodes can participate in keepalived. Simply ensure that each node has the appropriate priority set, and the node with the highest priority will become the master.
|
||||
@@ -0,0 +1,83 @@
|
||||
# Introduction
|
||||
|
||||
## Adding a host
|
||||
|
||||
## Adding storage
|
||||
|
||||
gluster volume add-brick VOLNAME NEW_BRICK
|
||||
|
||||
example
|
||||
|
||||
# gluster volume add-brick test-volume server4:/exp4
|
||||
Add Brick successful
|
||||
|
||||
# Replacing failed host
|
||||
|
||||
Followed https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/sect-Replacing_Hosts.html
|
||||
|
||||
|
||||
[root@glusterfs-server /]# gluster peer status
|
||||
Number of Peers: 1
|
||||
|
||||
Hostname: ds1
|
||||
Uuid: db9c80da-11e4-461d-8ea5-66dd12ca897c
|
||||
State: Peer in Cluster (Disconnected)
|
||||
[root@glusterfs-server /]#
|
||||
|
||||
Grab UUID above
|
||||
|
||||
edit /var/lib/glusterd/glusterd.info
|
||||
change:
|
||||
UUID=aee45c2c-aa19-4d29-bc94-4833f2b22863
|
||||
to
|
||||
UUID=db9c80da-11e4-461d-8ea5-66dd12ca897c
|
||||
|
||||
My peer's id (ds2):
|
||||
[root@glusterfs-server /]# gluster system:: uuid get
|
||||
UUID: 38ca4e8b-8ef5-4165-9f41-5c8b3f0103cc
|
||||
[root@glusterfs-server /]#
|
||||
|
||||
vi /var/lib/glusterd/peers/38ca4e8b-8ef5-4165-9f41-5c8b3f0103cc
|
||||
|
||||
UUID=38ca4e8b-8ef5-4165-9f41-5c8b3f0103cc
|
||||
state=3
|
||||
hostname=ds3
|
||||
|
||||
|
||||
|
||||
Got volume info
|
||||
|
||||
|
||||
[root@glusterfs-server /]# gluster volume info
|
||||
|
||||
Volume Name: gv0
|
||||
Type: Replicate
|
||||
Volume ID: 84e1169c-41dc-467a-9ae1-a474efaf789f
|
||||
Status: Started
|
||||
Snapshot Count: 0
|
||||
Number of Bricks: 1 x 2 = 2
|
||||
Transport-type: tcp
|
||||
Bricks:
|
||||
Brick1: ds1:/var/no-direct-write-here/brick1/gv0
|
||||
Brick2: ds3:/var/no-direct-write-here/brick1/gv0
|
||||
Options Reconfigured:
|
||||
nfs.disable: on
|
||||
transport.address-family: inet
|
||||
[root@glusterfs-server /]#
|
||||
|
||||
|
||||
|
||||
----
|
||||
[root@glusterfs-server /]# getfattr -d -m. -ehex /var/no-direct-write-here/brick1/gv0/
|
||||
getfattr: Removing leading '/' from absolute path names
|
||||
# file: var/no-direct-write-here/brick1/gv0/
|
||||
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
|
||||
trusted.gfid=0x00000000000000000000000000000001
|
||||
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
|
||||
trusted.glusterfs.volume-id=0x84e1169c41dc467a9ae1a474efaf789f
|
||||
|
||||
[root@glusterfs-server /]#
|
||||
|
||||
|
||||
|
||||
setfattr -n trusted.glusterfs.volume-id -v 0x84e1169c41dc467a9ae1a474efaf789f /var/no-direct-write-here/brick1/gv0
|
||||
@@ -76,4 +76,4 @@ After completing the above, you should have:
|
||||
* At least 20GB disk space (_but it'll be tight_)
|
||||
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
|
||||
|
||||
## Chef's Notes 📓
|
||||
## Chef's Notes
|
||||
@@ -0,0 +1,79 @@
|
||||
# Nodes
|
||||
|
||||
Let's start building our cluster. You can use either bare-metal machines or virtual machines - the configuration would be the same. To avoid confusion, I'll be referring to these as "nodes" from now on.
|
||||
|
||||
!!! note
|
||||
In 2017, I **initially** chose the "[Atomic](https://www.projectatomic.io/)" CentOS/Fedora image for the swarm hosts, but later found its outdated version of Docker to be problematic with advanced features like GPU transcoding (in [Plex](/recipes/plex/)), [Swarmprom](/recipes/swarmprom/), etc. In the end, I went mainstream and simply preferred a modern Ubuntu installation.
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
New in this recipe:
|
||||
|
||||
* [ ] 3 x nodes (*bare-metal or VMs*), each with:
|
||||
* A mainstream Linux OS (*tested on either [CentOS](https://www.centos.org) 7+ or [Ubuntu](http://releases.ubuntu.com) 16.04+*)
|
||||
* At least 2GB RAM
|
||||
* At least 20GB disk space (_but it'll be tight_)
|
||||
* [ ] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
|
||||
|
||||
|
||||
## Preparation
|
||||
|
||||
### Permit connectivity
|
||||
|
||||
Most modern Linux distributions include firewall rules which only only permit minimal required incoming connections (like SSH). We'll want to allow all traffic between our nodes. The steps to achieve this in CentOS/Ubuntu are a little different...
|
||||
|
||||
#### CentOS
|
||||
|
||||
Add something like this to `/etc/sysconfig/iptables`:
|
||||
|
||||
```
|
||||
# Allow all inter-node communication
|
||||
-A INPUT -s 192.168.31.0/24 -j ACCEPT
|
||||
```
|
||||
|
||||
And restart iptables with ```systemctl restart iptables```
|
||||
|
||||
#### Ubuntu
|
||||
|
||||
Install the (*non-default*) persistent iptables tools, by running `apt-get install iptables-persistent`, establishing some default rules (*dkpg will prompt you to save current ruleset*), and then add something like this to `/etc/iptables/rules.v4`:
|
||||
|
||||
```
|
||||
# Allow all inter-node communication
|
||||
-A INPUT -s 192.168.31.0/24 -j ACCEPT
|
||||
```
|
||||
|
||||
And refresh your running iptables rules with `iptables-restore < /etc/iptables/rules.v4`
|
||||
|
||||
### Enable hostname resolution
|
||||
|
||||
Depending on your hosting environment, you may have DNS automatically setup for your VMs. If not, it's useful to set up static entries in /etc/hosts for the nodes. For example, I setup the following:
|
||||
|
||||
```
|
||||
192.168.31.11 ds1 ds1.funkypenguin.co.nz
|
||||
192.168.31.12 ds2 ds2.funkypenguin.co.nz
|
||||
192.168.31.13 ds3 ds3.funkypenguin.co.nz
|
||||
```
|
||||
|
||||
### Set timezone
|
||||
|
||||
Set your local timezone, by running:
|
||||
|
||||
```
|
||||
ln -sf /usr/share/zoneinfo/<your timezone> /etc/localtime
|
||||
```
|
||||
|
||||
## Serving
|
||||
|
||||
After completing the above, you should have:
|
||||
|
||||
!!! summary "Summary"
|
||||
Deployed in this recipe:
|
||||
|
||||
* [X] 3 x nodes (*bare-metal or VMs*), each with:
|
||||
* A mainstream Linux OS (*tested on either [CentOS](https://www.centos.org) 7+ or [Ubuntu](http://releases.ubuntu.com) 16.04+*)
|
||||
* At least 2GB RAM
|
||||
* At least 20GB disk space (_but it'll be tight_)
|
||||
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
|
||||
|
||||
## Chef's Notes 📓
|
||||
@@ -110,4 +110,4 @@ systemctl restart docker-latest
|
||||
!!! tip ""
|
||||
Note the extra comma required after "false" above
|
||||
|
||||
## Chef's notes 📓
|
||||
## Chef's notes
|
||||
@@ -0,0 +1,113 @@
|
||||
# Create registry mirror
|
||||
|
||||
Although we now have shared storage for our persistent container data, our docker nodes don't share any other docker data, such as container images. This results in an inefficiency - every node which participates in the swarm will, at some point, need the docker image for every container deployed in the swarm.
|
||||
|
||||
When dealing with large container (looking at you, GitLab!), this can result in several gigabytes of wasted bandwidth per-node, and long delays when restarting containers on an alternate node. (_It also wastes disk space on each node, but we'll get to that in the next section_)
|
||||
|
||||
The solution is to run an official Docker registry container as a ["pull-through" cache, or "registry mirror"](https://docs.docker.com/registry/recipes/mirror/). By using our persistent storage for the registry cache, we can ensure we have a single copy of all the containers we've pulled at least once. After the first pull, any subsequent pulls from our nodes will use the cached version from our registry mirror. As a result, services are available more quickly when restarting container nodes, and we can be more aggressive about cleaning up unused containers on our nodes (more later)
|
||||
|
||||
The registry mirror runs as a swarm stack, using a simple docker-compose.yml. Customize __your mirror FQDN__ below, so that Traefik will generate the appropriate LetsEncrypt certificates for it, and make it available via HTTPS.
|
||||
|
||||
## Ingredients
|
||||
|
||||
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
|
||||
2. [Traefik](/ha-docker-swarm/traefik) configured per design
|
||||
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
|
||||
|
||||
|
||||
## Preparation
|
||||
|
||||
Create /var/data/config/registry/registry.yml as follows:
|
||||
|
||||
```
|
||||
version: "3"
|
||||
|
||||
services:
|
||||
|
||||
registry-mirror:
|
||||
image: registry:2
|
||||
networks:
|
||||
- traefik_public
|
||||
deploy:
|
||||
labels:
|
||||
- traefik.frontend.rule=Host:<your mirror FQDN>
|
||||
- traefik.docker.network=traefik_public
|
||||
- traefik.port=5000
|
||||
ports:
|
||||
- 5000:5000
|
||||
volumes:
|
||||
- /var/data/registry/registry-mirror-data:/var/lib/registry
|
||||
- /var/data/registry/registry-mirror-config.yml:/etc/docker/registry/config.yml
|
||||
|
||||
networks:
|
||||
traefik_public:
|
||||
external: true
|
||||
```
|
||||
|
||||
!!! note "Unencrypted registry"
|
||||
We create this registry without consideration for SSL, which will fail if we attempt to use the registry directly. However, we're going to use the HTTPS-proxied version via Traefik, leveraging Traefik to manage the LetsEncrypt certificates required.
|
||||
|
||||
|
||||
Create /var/data/registry/registry-mirror-config.yml as follows:
|
||||
```
|
||||
version: 0.1
|
||||
log:
|
||||
fields:
|
||||
service: registry
|
||||
storage:
|
||||
cache:
|
||||
blobdescriptor: inmemory
|
||||
filesystem:
|
||||
rootdirectory: /var/lib/registry
|
||||
delete:
|
||||
enabled: true
|
||||
http:
|
||||
addr: :5000
|
||||
headers:
|
||||
X-Content-Type-Options: [nosniff]
|
||||
health:
|
||||
storagedriver:
|
||||
enabled: true
|
||||
interval: 10s
|
||||
threshold: 3
|
||||
proxy:
|
||||
remoteurl: https://registry-1.docker.io
|
||||
```
|
||||
|
||||
## Serving
|
||||
|
||||
### Launch registry stack
|
||||
|
||||
Launch the registry stack by running ```docker stack deploy registry -c <path-to-docker-compose.yml>```
|
||||
|
||||
### Enable registry mirror and experimental features
|
||||
|
||||
To tell docker to use the registry mirror, and (_while we're here_) in order to be able to watch the logs of any service from any manager node (_an experimental feature in the current Atomic docker build_), edit **/etc/docker-latest/daemon.json** on each node, and change from:
|
||||
|
||||
```
|
||||
{
|
||||
"log-driver": "journald",
|
||||
"signature-verification": false
|
||||
}
|
||||
```
|
||||
|
||||
To:
|
||||
|
||||
```
|
||||
{
|
||||
"log-driver": "journald",
|
||||
"signature-verification": false,
|
||||
"experimental": true,
|
||||
"registry-mirrors": ["https://<your registry mirror FQDN>"]
|
||||
}
|
||||
```
|
||||
|
||||
Then restart docker by running:
|
||||
```
|
||||
systemctl restart docker-latest
|
||||
```
|
||||
|
||||
!!! tip ""
|
||||
Note the extra comma required after "false" above
|
||||
|
||||
## Chef's notes 📓
|
||||
@@ -213,6 +213,6 @@ Here's a screencast of the playbook in action. I sped up the boring parts, it ac
|
||||
[patreon]: https://www.patreon.com/bePatron?u=6982506
|
||||
[github_sponsor]: https://github.com/sponsors/funkypenguin
|
||||
|
||||
## Chef's Notes 📓
|
||||
## Chef's Notes
|
||||
|
||||
[^1]: Minimum Viable Cluster acronym copyright, trademark, and whatever else, to Funky Penguin for 1,000,000 years.
|
||||
@@ -0,0 +1,218 @@
|
||||
# Shared Storage (Ceph)
|
||||
|
||||
While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node.
|
||||
|
||||
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
3 x Virtual Machines (configured earlier), each with:
|
||||
|
||||
* [X] Support for "modern" versions of Python and LVM
|
||||
* [X] At least 1GB RAM
|
||||
* [X] At least 20GB disk space (_but it'll be tight_)
|
||||
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
|
||||
* [X] A second disk dedicated to the Ceph OSD
|
||||
* [X] Each node should have the IP of every other participating node hard-coded in /etc/hosts (*including its own IP*)
|
||||
|
||||
## Preparation
|
||||
|
||||
!!! tip "No more [foolish games](https://www.youtube.com/watch?v=UNoouLa7uxA)"
|
||||
Earlier iterations of this recipe (*based on [Ceph Jewel](https://docs.ceph.com/docs/master/releases/jewel/)*) required significant manual effort to install Ceph in a Docker environment. In the 2+ years since Jewel was released, significant improvements have been made to the ceph "deploy-in-docker" process, including the [introduction of the cephadm tool](https://ceph.io/ceph-management/introducing-cephadm/). Cephadm is the tool which now does all the heavy lifting, below, for the current version of ceph, codenamed "[Octopus](https://www.youtube.com/watch?v=Gi58pN8W3hY)".
|
||||
|
||||
### Pick a master node
|
||||
|
||||
One of your nodes will become the cephadm "master" node. Although all nodes will participate in the Ceph cluster, the master node will be the node which we bootstrap ceph on. It's also the node which will run the Ceph dashboard, and on which future upgrades will be processed. It doesn't matter _which_ node you pick, and the cluster itself will operate in the event of a loss of the master node (although you won't see the dashboard)
|
||||
|
||||
### Install cephadm on master node
|
||||
|
||||
Run the following on the ==master== node:
|
||||
|
||||
```
|
||||
MYIP=`ip route get 1.1.1.1 | grep -oP 'src \K\S+'`
|
||||
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
|
||||
chmod +x cephadm
|
||||
mkdir -p /etc/ceph
|
||||
./cephadm bootstrap --mon-ip $MYIP
|
||||
```
|
||||
|
||||
The process takes about 30 seconds, after which, you'll have a MVC (*Minimum Viable Cluster*)[^1], encompassing a single monitor and mgr instance on your chosen node. Here's the complete output from a fresh install:
|
||||
|
||||
??? "Example output from a fresh cephadm bootstrap"
|
||||
```
|
||||
root@raphael:~# MYIP=`ip route get 1.1.1.1 | grep -oP 'src \K\S+'`
|
||||
root@raphael:~# curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
|
||||
|
||||
root@raphael:~# chmod +x cephadm
|
||||
root@raphael:~# mkdir -p /etc/ceph
|
||||
root@raphael:~# ./cephadm bootstrap --mon-ip $MYIP
|
||||
INFO:cephadm:Verifying podman|docker is present...
|
||||
INFO:cephadm:Verifying lvm2 is present...
|
||||
INFO:cephadm:Verifying time synchronization is in place...
|
||||
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
|
||||
INFO:cephadm:Repeating the final host check...
|
||||
INFO:cephadm:podman|docker (/usr/bin/docker) is present
|
||||
INFO:cephadm:systemctl is present
|
||||
INFO:cephadm:lvcreate is present
|
||||
INFO:cephadm:Unit systemd-timesyncd.service is enabled and running
|
||||
INFO:cephadm:Host looks OK
|
||||
INFO:root:Cluster fsid: bf3eff78-9e27-11ea-b40a-525400380101
|
||||
INFO:cephadm:Verifying IP 192.168.38.101 port 3300 ...
|
||||
INFO:cephadm:Verifying IP 192.168.38.101 port 6789 ...
|
||||
INFO:cephadm:Mon IP 192.168.38.101 is in CIDR network 192.168.38.0/24
|
||||
INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container...
|
||||
INFO:cephadm:Extracting ceph user uid/gid from container image...
|
||||
INFO:cephadm:Creating initial keys...
|
||||
INFO:cephadm:Creating initial monmap...
|
||||
INFO:cephadm:Creating mon...
|
||||
INFO:cephadm:Waiting for mon to start...
|
||||
INFO:cephadm:Waiting for mon...
|
||||
INFO:cephadm:mon is available
|
||||
INFO:cephadm:Assimilating anything we can from ceph.conf...
|
||||
INFO:cephadm:Generating new minimal ceph.conf...
|
||||
INFO:cephadm:Restarting the monitor...
|
||||
INFO:cephadm:Setting mon public_network...
|
||||
INFO:cephadm:Creating mgr...
|
||||
INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring
|
||||
INFO:cephadm:Wrote config to /etc/ceph/ceph.conf
|
||||
INFO:cephadm:Waiting for mgr to start...
|
||||
INFO:cephadm:Waiting for mgr...
|
||||
INFO:cephadm:mgr not available, waiting (1/10)...
|
||||
INFO:cephadm:mgr not available, waiting (2/10)...
|
||||
INFO:cephadm:mgr not available, waiting (3/10)...
|
||||
INFO:cephadm:mgr is available
|
||||
INFO:cephadm:Enabling cephadm module...
|
||||
INFO:cephadm:Waiting for the mgr to restart...
|
||||
INFO:cephadm:Waiting for Mgr epoch 5...
|
||||
INFO:cephadm:Mgr epoch 5 is available
|
||||
INFO:cephadm:Setting orchestrator backend to cephadm...
|
||||
INFO:cephadm:Generating ssh key...
|
||||
INFO:cephadm:Wrote public SSH key to to /etc/ceph/ceph.pub
|
||||
INFO:cephadm:Adding key to root@localhost's authorized_keys...
|
||||
INFO:cephadm:Adding host raphael...
|
||||
INFO:cephadm:Deploying mon service with default placement...
|
||||
INFO:cephadm:Deploying mgr service with default placement...
|
||||
INFO:cephadm:Deploying crash service with default placement...
|
||||
INFO:cephadm:Enabling mgr prometheus module...
|
||||
INFO:cephadm:Deploying prometheus service with default placement...
|
||||
INFO:cephadm:Deploying grafana service with default placement...
|
||||
INFO:cephadm:Deploying node-exporter service with default placement...
|
||||
INFO:cephadm:Deploying alertmanager service with default placement...
|
||||
INFO:cephadm:Enabling the dashboard module...
|
||||
INFO:cephadm:Waiting for the mgr to restart...
|
||||
INFO:cephadm:Waiting for Mgr epoch 13...
|
||||
INFO:cephadm:Mgr epoch 13 is available
|
||||
INFO:cephadm:Generating a dashboard self-signed certificate...
|
||||
INFO:cephadm:Creating initial admin user...
|
||||
INFO:cephadm:Fetching dashboard port number...
|
||||
INFO:cephadm:Ceph Dashboard is now available at:
|
||||
|
||||
URL: https://raphael:8443/
|
||||
User: admin
|
||||
Password: mid28k0yg5
|
||||
|
||||
INFO:cephadm:You can access the Ceph CLI with:
|
||||
|
||||
sudo ./cephadm shell --fsid bf3eff78-9e27-11ea-b40a-525400380101 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
|
||||
|
||||
INFO:cephadm:Please consider enabling telemetry to help improve Ceph:
|
||||
|
||||
ceph telemetry on
|
||||
|
||||
For more information see:
|
||||
|
||||
https://docs.ceph.com/docs/master/mgr/telemetry/
|
||||
|
||||
INFO:cephadm:Bootstrap complete.
|
||||
root@raphael:~#
|
||||
```
|
||||
|
||||
|
||||
### Prepare other nodes
|
||||
|
||||
It's now necessary to tranfer the following files to your ==other== nodes, so that cephadm can add them to your cluster, and so that they'll be able to mount the cephfs when we're done:
|
||||
|
||||
Path on master | Path on non-master
|
||||
--------------- | -----
|
||||
`/etc/ceph/ceph.conf` | `/etc/ceph/ceph.conf`
|
||||
`/etc/ceph/ceph.client.admin.keyring` | `/etc/ceph/ceph.client.admin.keyring`
|
||||
`/etc/ceph/ceph.pub` | `/root/.ssh/authorized_keys` (append to anything existing)
|
||||
|
||||
|
||||
Back on the ==master== node, run `ceph orch host add <node-name>` once for each other node you want to join to the cluster. You can validate the results by running `ceph orch host ls`
|
||||
|
||||
!!! question "Should we be concerned about giving cephadm using root access over SSH?"
|
||||
Not really. Docker is inherently insecure at the host-level anyway (*think what would happen if you launched a global-mode stack with a malicious container image which mounted `/root/.ssh`*), so worrying about cephadm seems a little barn-door-after-horses-bolted. If you take host-level security seriously, consider switching to [Kubernetes](/kubernetes/start/) :)
|
||||
|
||||
### Add OSDs
|
||||
|
||||
Now the best improvement since the days of ceph-deploy and manual disks.. on the ==master== node, run `ceph orch apply osd --all-available-devices`. This will identify any unloved (*unpartitioned, unmounted*) disks attached to each participating node, and configure these disks as OSDs.
|
||||
|
||||
### Setup CephFS
|
||||
|
||||
On the ==master== node, create a cephfs volume in your cluster, by running `ceph fs volume create data`. Ceph will handle the necessary orchestration itself, creating the necessary pool, mds daemon, etc.
|
||||
|
||||
You can watch the progress by running `ceph fs ls` (to see the fs is configured), and `ceph -s` to wait for `HEALTH_OK`
|
||||
|
||||
### Mount CephFS volume
|
||||
|
||||
On ==every== node, create a mountpoint for the data, by running ```mkdir /var/data```, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually _mounted_ if there's a network / boot delay getting access to the gluster volume:
|
||||
|
||||
```
|
||||
mkdir /var/data
|
||||
|
||||
MYNODES="<node1>,<node2>,<node3>" # Add your own nodes here, comma-delimited
|
||||
MYHOST=`ip route get 1.1.1.1 | grep -oP 'src \K\S+'`
|
||||
echo -e "
|
||||
# Mount cephfs volume \n
|
||||
raphael,donatello,leonardo:/ /var/data ceph name=admin,noatime,_netdev 0 0" >> /etc/fstab
|
||||
mount -a
|
||||
```
|
||||
|
||||
## Serving
|
||||
|
||||
### Sprinkle with tools
|
||||
|
||||
Although it's possible to use `cephadm shell` to exec into a container with the necessary ceph tools, it's more convenient to use the native CLI tools. To this end, on each node, run the following, which will install the appropriate apt repository, and install the latest ceph CLI tools:
|
||||
|
||||
```
|
||||
curl -L https://download.ceph.com/keys/release.asc | sudo apt-key add -
|
||||
cephadm add-repo --release octopus
|
||||
cephadm install ceph-common
|
||||
```
|
||||
|
||||
### Drool over dashboard
|
||||
|
||||
Ceph now includes a comprehensive dashboard, provided by the mgr daemon. The dashboard will be accessible at https://[IP of your ceph master node]:8443, but you'll need to run `ceph dashboard ac-user-create <username> <password> administrator` first, to create an administrator account:
|
||||
|
||||
```
|
||||
root@raphael:~# ceph dashboard ac-user-create batman supermansucks administrator
|
||||
{"username": "batman", "password": "$2b$12$3HkjY85mav.dq3HHAZiWP.KkMiuoV2TURZFH.6WFfo/BPZCT/0gr.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1590372281, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": false}
|
||||
root@raphael:~#
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
What have we achieved?
|
||||
|
||||
!!! summary "Summary"
|
||||
Created:
|
||||
|
||||
* [X] Persistent storage available to every node
|
||||
* [X] Resiliency in the event of the failure of a single node
|
||||
* [X] Beautiful dashboard
|
||||
|
||||
## The easy, 5-minute install
|
||||
|
||||
I share (_with [sponsors][github_sponsor] and [patrons][patreon]_) a private "_premix_" GitHub repository, which includes an ansible playbook for deploying the entire Geek's Cookbook stack, automatically. This means that members can create the entire environment with just a ```git pull``` and an ```ansible-playbook deploy.yml``` 👍
|
||||
|
||||
Here's a screencast of the playbook in action. I sped up the boring parts, it actually takes ==5 min== (*you can tell by the timestamps on the prompt*):
|
||||
|
||||
|
||||
[patreon]: https://www.patreon.com/bePatron?u=6982506
|
||||
[github_sponsor]: https://github.com/sponsors/funkypenguin
|
||||
|
||||
## Chef's Notes 📓
|
||||
|
||||
[^1]: Minimum Viable Cluster acronym copyright, trademark, and whatever else, to Funky Penguin for 1,000,000 years.
|
||||
@@ -157,7 +157,7 @@ After completing the above, you should have:
|
||||
* [X] Persistent storage available to every node
|
||||
* [X] Resiliency in the event of the failure of a single (gluster) node
|
||||
|
||||
## Chef's Notes 📓
|
||||
## Chef's Notes
|
||||
|
||||
Future enhancements to this recipe include:
|
||||
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
# Shared Storage (GlusterFS)
|
||||
|
||||
While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node.
|
||||
|
||||
!!! warning
|
||||
This recipe is deprecated. It didn't work well in 2017, and it's not likely to work any better now. It remains here as a reference. I now recommend the use of [Ceph for shared storage](/ha-docker-swarm/shared-storage-ceph/) instead. - 2019 Chef
|
||||
|
||||
## Design
|
||||
|
||||
### Why GlusterFS?
|
||||
|
||||
This GlusterFS recipe was my original design for shared storage, but I [found it to be flawed](shared-storage-ceph/#why-not-glusterfs), and I replaced it with a [design which employs Ceph instead](shared-storage-ceph/#why-ceph). This recipe is an alternate to the Ceph design, if you happen to prefer GlusterFS.
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
3 x Virtual Machines (configured earlier), each with:
|
||||
|
||||
* [X] CentOS/Fedora Atomic
|
||||
* [X] At least 1GB RAM
|
||||
* [X] At least 20GB disk space (_but it'll be tight_)
|
||||
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
|
||||
* [ ] A second disk, or adequate space on the primary disk for a dedicated data partition
|
||||
|
||||
## Preparation
|
||||
|
||||
### Create Gluster "bricks"
|
||||
|
||||
To build our Gluster volume, we need 2 out of the 3 VMs to provide one "brick". The bricks will be used to create the replicated volume. Assuming a replica count of 2 (_i.e., 2 copies of the data are kept in gluster_), our total number of bricks must be divisible by our replica count. (_I.e., you can't have 3 bricks if you want 2 replicas. You can have 4 though - We have to have minimum 3 swarm manager nodes for fault-tolerance, but only 2 of those nodes need to run as gluster servers._)
|
||||
|
||||
On each host, run a variation following to create your bricks, adjusted for the path to your disk.
|
||||
|
||||
!!! note "The example below assumes /dev/vdb is dedicated to the gluster volume"
|
||||
```
|
||||
(
|
||||
echo o # Create a new empty DOS partition table
|
||||
echo n # Add a new partition
|
||||
echo p # Primary partition
|
||||
echo 1 # Partition number
|
||||
echo # First sector (Accept default: 1)
|
||||
echo # Last sector (Accept default: varies)
|
||||
echo w # Write changes
|
||||
) | sudo fdisk /dev/vdb
|
||||
|
||||
mkfs.xfs -i size=512 /dev/vdb1
|
||||
mkdir -p /var/no-direct-write-here/brick1
|
||||
echo '' >> /etc/fstab >> /etc/fstab
|
||||
echo '# Mount /dev/vdb1 so that it can be used as a glusterfs volume' >> /etc/fstab
|
||||
echo '/dev/vdb1 /var/no-direct-write-here/brick1 xfs defaults 1 2' >> /etc/fstab
|
||||
mount -a && mount
|
||||
```
|
||||
|
||||
!!! warning "Don't provision all your LVM space"
|
||||
Atomic uses LVM to store docker data, and **automatically grows** Docker's volumes as requried. If you commit all your free LVM space to your brick, you'll quickly find (as I did) that docker will start to fail with error messages about insufficient space. If you're going to slice off a portion of your LVM space in /dev/atomicos, make sure you leave enough space for Docker storage, where "enough" depends on how much you plan to pull images, make volumes, etc. I ate through 20GB very quickly doing development, so I ended up provisioning 50GB for atomic alone, with a separate volume for the brick.
|
||||
|
||||
### Create glusterfs container
|
||||
|
||||
Atomic doesn't include the Gluster server components. This means we'll have to run glusterd from within a container, with privileged access to the host. Although convoluted, I've come to prefer this design since it once again makes the OS "disposable", moving all the config into containers and code.
|
||||
|
||||
Run the following on each host:
|
||||
```
|
||||
docker run \
|
||||
-h glusterfs-server \
|
||||
-v /etc/glusterfs:/etc/glusterfs:z \
|
||||
-v /var/lib/glusterd:/var/lib/glusterd:z \
|
||||
-v /var/log/glusterfs:/var/log/glusterfs:z \
|
||||
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
|
||||
-v /var/no-direct-write-here/brick1:/var/no-direct-write-here/brick1 \
|
||||
-d --privileged=true --net=host \
|
||||
--restart=always \
|
||||
--name="glusterfs-server" \
|
||||
gluster/gluster-centos
|
||||
```
|
||||
### Create trusted pool
|
||||
|
||||
On a single node (doesn't matter which), run ```docker exec -it glusterfs-server bash``` to launch a shell inside the container.
|
||||
|
||||
From the node, run
|
||||
```gluster peer probe <other host>```
|
||||
|
||||
Example output:
|
||||
```
|
||||
[root@glusterfs-server /]# gluster peer probe ds1
|
||||
peer probe: success.
|
||||
[root@glusterfs-server /]#
|
||||
```
|
||||
|
||||
Run ```gluster peer status``` on both nodes to confirm that they're properly connected to each other:
|
||||
|
||||
Example output:
|
||||
```
|
||||
[root@glusterfs-server /]# gluster peer status
|
||||
Number of Peers: 1
|
||||
|
||||
Hostname: ds3
|
||||
Uuid: 3e115ba9-6a4f-48dd-87d7-e843170ff499
|
||||
State: Peer in Cluster (Connected)
|
||||
[root@glusterfs-server /]#
|
||||
```
|
||||
|
||||
### Create gluster volume
|
||||
|
||||
Now we create a *replicated volume* out of our individual "bricks".
|
||||
|
||||
Create the gluster volume by running
|
||||
```
|
||||
gluster volume create gv0 replica 2 \
|
||||
server1:/var/no-direct-write-here/brick1 \
|
||||
server2:/var/no-direct-write-here/brick1
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
[root@glusterfs-server /]# gluster volume create gv0 replica 2 ds1:/var/no-direct-write-here/brick1/gv0 ds3:/var/no-direct-write-here/brick1/gv0
|
||||
volume create: gv0: success: please start the volume to access data
|
||||
[root@glusterfs-server /]#
|
||||
```
|
||||
|
||||
Start the volume by running ```gluster volume start gv0```
|
||||
|
||||
```
|
||||
[root@glusterfs-server /]# gluster volume start gv0
|
||||
volume start: gv0: success
|
||||
[root@glusterfs-server /]#
|
||||
```
|
||||
|
||||
The volume is only present on the host you're shelled into though. To add the other hosts to the volume, run ```gluster peer probe <servername>```. Don't probe host from itself.
|
||||
|
||||
From one other host, run ```docker exec -it glusterfs-server bash``` to shell into the gluster-server container, and run ```gluster peer probe <original server name>``` to update the name of the host which started the volume.
|
||||
|
||||
### Mount gluster volume
|
||||
|
||||
On the host (i.e., outside of the container - type ```exit``` if you're still shelled in), create a mountpoint for the data, by running ```mkdir /var/data```, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually _mounted_ if there's a network / boot delay getting access to the gluster volume:
|
||||
|
||||
```
|
||||
mkdir /var/data
|
||||
MYHOST=`hostname -s`
|
||||
echo '' >> /etc/fstab >> /etc/fstab
|
||||
echo '# Mount glusterfs volume' >> /etc/fstab
|
||||
echo "$MYHOST:/gv0 /var/data glusterfs defaults,_netdev,context="system_u:object_r:svirt_sandbox_file_t:s0" 0 0" >> /etc/fstab
|
||||
mount -a
|
||||
```
|
||||
|
||||
For some reason, my nodes won't auto-mount this volume on boot. I even tried the trickery below, but they stubbornly refuse to automount.
|
||||
```
|
||||
echo -e "\n\n# Give GlusterFS 10s to start before \
|
||||
mounting\nsleep 10s && mount -a" >> /etc/rc.local
|
||||
systemctl enable rc-local.service
|
||||
```
|
||||
|
||||
For non-gluster nodes, you'll need to replace $MYHOST above with the name of one of the gluster hosts (I haven't worked out how to make this fully HA yet)
|
||||
|
||||
## Serving
|
||||
|
||||
After completing the above, you should have:
|
||||
|
||||
* [X] Persistent storage available to every node
|
||||
* [X] Resiliency in the event of the failure of a single (gluster) node
|
||||
|
||||
## Chef's Notes 📓
|
||||
|
||||
Future enhancements to this recipe include:
|
||||
|
||||
1. Migration of shared storage from GlusterFS to Ceph ()[#2](https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook/issues/2))
|
||||
2. Correct the fact that volumes don't automount on boot ([#3](https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook/issues/3))
|
||||
@@ -108,7 +108,7 @@ What have we achieved? By adding an additional three simple labels to any servic
|
||||
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
## Chef's Notes
|
||||
|
||||
1. Traefik forward auth replaces the use of [oauth_proxy containers](https://geek-cookbook.funkypenguin.co.nz/)reference/oauth_proxy/) found in some of the existing recipes
|
||||
2. [@thomaseddon's original version](https://github.com/thomseddon/traefik-forward-auth) of traefik-forward-auth only works with Google currently, but I've created a [fork](https://www.github.com/funkypenguin/traefik-forward-auth) of a [fork](https://github.com/noelcatt/traefik-forward-auth), which implements generic OIDC providers.
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# Traefik Forward Auth
|
||||
|
||||
Now that we have Traefik deployed, automatically exposing SSL access to our Docker Swarm services using LetsEncrypt wildcard certificates, let's pause to consider that we may not _want_ some services exposed directly to the internet...
|
||||
|
||||
..Wait, why not? Well, Traefik doesn't provide any form of authentication, it simply secures the **transmission** of the service between Docker Swarm and the end user. If you were to deploy a service with no native security (*[Radarr](/recipes/autopirate/radarr/) or [Sonarr](/recipes/autopirate/sonarr/) come to mind*), then anybody would be able to use it! Even services which _may_ have a layer of authentication **might** not be safe to expose publically - often open source projects may be maintained by enthusiasts who happily add extra features, but just pay lip service to security, on the basis that "*it's the user's problem to secure it in their own network*".
|
||||
|
||||
To give us confidence that **we** can access our services, but BadGuys(tm) cannot, we'll deploy a layer of authentication **in front** of Traefik, using [Forward Authentication](https://docs.traefik.io/configuration/entrypoints/#forward-authentication). You can use your own [KeyCloak](/recipes/keycloak/) instance for authentication, but to lower the barrier to entry, this recipe will assume you're authenticating against your own Google account.
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "Ingredients"
|
||||
Existing:
|
||||
|
||||
* [X] [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph)
|
||||
* [X] [Traefik](/ha-docker-swarm/traefik/) configured per design
|
||||
|
||||
New:
|
||||
|
||||
* [ ] Client ID and secret from an OpenID-Connect provider (Google, [KeyCloak](/recipes/keycloak/), Microsoft, etc..)
|
||||
|
||||
## Preparation
|
||||
|
||||
### Obtain OAuth credentials
|
||||
|
||||
!!! note
|
||||
This recipe will demonstrate using Google OAuth for traefik forward authentication, but it's also possible to use a self-hosted KeyCloak instance - see the [KeyCloak OIDC Provider](/recipes/keycloak/setup-oidc-provider/) recipe for more details!
|
||||
|
||||
Log into https://console.developers.google.com/, create a new project then search for and select "Credentials" in the search bar.
|
||||
|
||||
Fill out the "OAuth Consent Screen" tab, and then click, "**Create Credentials**" > "**OAuth client ID**". Select "**Web Application**", fill in the name of your app, skip "**Authorized JavaScript origins**" and fill "**Authorized redirect URIs**" with either all the domains you will allow authentication from, appended with the url-path (*e.g. https://radarr.example.com/_oauth, https://radarr.example.com/_oauth, etc*), or if you don't like frustration, use a "auth host" URL instead, like "*https://auth.example.com/_oauth*" (*see below for details*)
|
||||
|
||||
Store your client ID and secret safely - you'll need them for the next step.
|
||||
|
||||
|
||||
### Prepare environment
|
||||
|
||||
Create `/var/data/config/traefik/traefik-forward-auth.env` as follows:
|
||||
|
||||
```
|
||||
CLIENT_ID=<your client id>
|
||||
CLIENT_SECRET=<your client secret>
|
||||
OIDC_ISSUER=https://accounts.google.com
|
||||
SECRET=<a random string, make it up>
|
||||
# uncomment this to use a single auth host instead of individual redirect_uris (recommended but advanced)
|
||||
#AUTH_HOST=auth.example.com
|
||||
COOKIE_DOMAINS=example.com
|
||||
```
|
||||
|
||||
### Prepare the docker service config
|
||||
|
||||
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik](/recipes/traefik/) recipe:
|
||||
|
||||
```
|
||||
traefik-forward-auth:
|
||||
image: funkypenguin/traefik-forward-auth
|
||||
env_file: /var/data/config/traefik/traefik-forward-auth.env
|
||||
networks:
|
||||
- traefik_public
|
||||
# Uncomment these lines if you're using auth host mode
|
||||
#deploy:
|
||||
# labels:
|
||||
# - traefik.port=4181
|
||||
# - traefik.frontend.rule=Host:auth.example.com
|
||||
# - traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181
|
||||
# - traefik.frontend.auth.forward.trustForwardHeader=true
|
||||
```
|
||||
|
||||
If you're not confident that forward authentication is working, add a simple "whoami" test container, to help debug traefik forward auth, before attempting to add it to a more complex container.
|
||||
|
||||
```
|
||||
# This simply validates that traefik forward authentication is working
|
||||
whoami:
|
||||
image: containous/whoami
|
||||
networks:
|
||||
- traefik_public
|
||||
deploy:
|
||||
labels:
|
||||
- traefik.frontend.rule=Host:whoami.example.com
|
||||
- traefik.port=80
|
||||
- traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181
|
||||
- traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
|
||||
- traefik.frontend.auth.forward.trustForwardHeader=true
|
||||
```
|
||||
|
||||
!!! tip
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
|
||||
|
||||
|
||||
|
||||
## Serving
|
||||
|
||||
### Launch
|
||||
|
||||
Redeploy traefik with ```docker stack deploy traefik-app -c /var/data/traefik/traeifk-app.yml```, to launch the traefik-forward-auth container.
|
||||
|
||||
### Test
|
||||
|
||||
Browse to https://whoami.example.com (*obviously, customized for your domain and having created a DNS record*), and all going according to plan, you should be redirected to a Google login. Once successfully logged in, you'll be directed to the basic whoami page.
|
||||
|
||||
## Summary
|
||||
|
||||
What have we achieved? By adding an additional three simple labels to any service, we can secure any service behind our choice of OAuth provider, with minimal processing / handling overhead.
|
||||
|
||||
!!! summary "Summary"
|
||||
Created:
|
||||
|
||||
* [X] Traefik-forward-auth configured to authenticate against an OIDC provider
|
||||
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
|
||||
1. Traefik forward auth replaces the use of [oauth_proxy containers](/reference/oauth_proxy/) found in some of the existing recipes
|
||||
2. [@thomaseddon's original version](https://github.com/thomseddon/traefik-forward-auth) of traefik-forward-auth only works with Google currently, but I've created a [fork](https://www.github.com/funkypenguin/traefik-forward-auth) of a [fork](https://github.com/noelcatt/traefik-forward-auth), which implements generic OIDC providers.
|
||||
3. I reviewed several implementations of forward authenticators for Traefik, but found most to be rather heavy-handed, or specific to a single auth provider. @thomaseddon's go-based docker image is 7MB in size, and with the generic OIDC patch (above), it can be extended to work with any OIDC provider.
|
||||
4. No, not github natively, but you can ferderate GitHub into KeyCloak, and then use KeyCloak as the OIDC provider.
|
||||
@@ -117,6 +117,6 @@ What have we achieved? By adding an additional three simple labels to any servic
|
||||
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
## Chef's Notes
|
||||
|
||||
1. KeyCloak is very powerful. You can add 2FA and all other clever things outside of the scope of this simple recipe ;)
|
||||
@@ -0,0 +1,122 @@
|
||||
# Using Traefik Forward Auth with KeyCloak
|
||||
|
||||
While the [Traefik Forward Auth](/ha-docker-swarm/traefik-forward-auth/) recipe demonstrated a quick way to protect a set of explicitly-specified URLs using OIDC credentials from a Google account, this recipe will illustrate how to use your own KeyCloak instance to secure **any** URLs within your DNS domain.
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! Summary
|
||||
Existing:
|
||||
|
||||
* [X] [KeyCloak](/recipes/keycloak/) recipe deployed successfully, with a [local user](/recipes/keycloak/create-user/) and an [OIDC client](/recipes/keycloak/setup-oidc-provider/)
|
||||
|
||||
New:
|
||||
|
||||
* [ ] DNS entry for your auth host (*"auth.yourdomain.com" is a good choice*), pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
|
||||
|
||||
## Preparation
|
||||
|
||||
### What is AuthHost mode
|
||||
|
||||
Under normal OIDC auth, you have to tell your auth provider which URLs it may redirect an authenticated user back to, post-authentication. This is a security feture of the OIDC spec, preventing a malicious landing page from capturing your session and using it to impersonate you. When you're securing many URLs though, explicitly listing them can be a PITA.
|
||||
|
||||
[@thomaseddon's traefik-forward-auth](https://github.com/thomseddon/traefik-forward-auth) includes an ingenious mechanism to simulate an "*auth host*" in your OIDC authentication, so that you can protect an unlimited amount of DNS names (*with a common domain suffix*), without having to manually maintain a list.
|
||||
|
||||
#### How does it work?
|
||||
|
||||
Say you're protecting **radarr.example.com**. When you first browse to **https://radarr.example.com**, Traefik forwards your session to traefik-forward-auth, to be authenticated. Traefik-forward-auth redirects you to your OIDC provider's login (*KeyCloak, in this case*), but instructs the OIDC provider to redirect a successfully authenticated session **back** to **https://auth.example.com/_oauth**, rather than to **https://radarr.example.com/_oauth**.
|
||||
|
||||
When you successfully authenticate against the OIDC provider, you are redirected to the "*redirect_uri*" of https://auth.example.com. Again, your request hits Traefik, whichforwards the session to traefik-forward-auth, which **knows** that you've just been authenticated (*cookies have a role to play here*). Traefik-forward-auth also knows the URL of your **original** request (*thanks to the X-Forwarded-Whatever header*). Traefik-forward-auth redirects you to your original destination, and everybody is happy.
|
||||
|
||||
This clever workaround only works under 2 conditions:
|
||||
|
||||
|
||||
1. Your "auth host" has the same domain name as the hosts you're protecting (*i.e., auth.example.com protecting radarr.example.com*)
|
||||
2. You explictly tell traefik-forward-auth to use a cookie authenticating your **whole** domain (*i.e. example.com*)
|
||||
|
||||
### Setup environment
|
||||
|
||||
Create `/var/data/config/traefik/traefik-forward-auth.env` as follows (*change "master" if you created a different realm*):
|
||||
|
||||
```
|
||||
CLIENT_ID=<your keycloak client name>
|
||||
CLIENT_SECRET=<your keycloak client secret>
|
||||
OIDC_ISSUER=https://<your keycloak URL>/auth/realms/master
|
||||
SECRET=<a random string to secure your cookie>
|
||||
AUTH_HOST=<the FQDN to use for your auth host>
|
||||
COOKIE_DOMAIN=<the root FQDN of your domain>
|
||||
```
|
||||
|
||||
### Prepare the docker service config
|
||||
|
||||
This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik](/recipes/traefik/) recipe:
|
||||
|
||||
```
|
||||
traefik-forward-auth:
|
||||
image: funkypenguin/traefik-forward-auth
|
||||
env_file: /var/data/config/traefik/traefik-forward-auth.env
|
||||
networks:
|
||||
- traefik_public
|
||||
deploy:
|
||||
labels:
|
||||
- traefik.port=4181
|
||||
- traefik.frontend.rule=Host:auth.example.com
|
||||
- traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181
|
||||
- traefik.frontend.auth.forward.trustForwardHeader=true
|
||||
```
|
||||
|
||||
If you're not confident that forward authentication is working, add a simple "whoami" test container, to help debug traefik forward auth, before attempting to add it to a more complex container.
|
||||
|
||||
```
|
||||
# This simply validates that traefik forward authentication is working
|
||||
whoami:
|
||||
image: containous/whoami
|
||||
networks:
|
||||
- traefik_public
|
||||
deploy:
|
||||
labels:
|
||||
- traefik.frontend.rule=Host:whoami.example.com
|
||||
- traefik.port=80
|
||||
- traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181
|
||||
- traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
|
||||
- traefik.frontend.auth.forward.trustForwardHeader=true
|
||||
```
|
||||
|
||||
!!! tip
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
|
||||
|
||||
## Serving
|
||||
|
||||
### Launch
|
||||
|
||||
Redeploy traefik with ```docker stack deploy traefik-app -c /var/data/traefik/traeifk-app.yml```, to launch the traefik-forward-auth container.
|
||||
|
||||
### Test
|
||||
|
||||
Browse to https://whoami.example.com (*obviously, customized for your domain and having created a DNS record*), and all going according to plan, you'll be redirected to a KeyCloak login. Once successfully logged in, you'll be directed to the basic whoami page.
|
||||
|
||||
### Protect services
|
||||
|
||||
To protect any other service, ensure the service itself is exposed by Traefik (*if you were previously using an oauth_proxy for this, you may have to migrate some labels from the oauth_proxy serivce to the service itself*). Add the following 3 labels:
|
||||
|
||||
```
|
||||
- traefik.frontend.auth.forward.address=http://traefik-forward-auth:4181
|
||||
- traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
|
||||
- traefik.frontend.auth.forward.trustForwardHeader=true
|
||||
```
|
||||
|
||||
And re-deploy your services :)
|
||||
|
||||
## Summary
|
||||
|
||||
What have we achieved? By adding an additional three simple labels to any service, we can secure any service behind our KeyCloak OIDC provider, with minimal processing / handling overhead.
|
||||
|
||||
!!! summary "Summary"
|
||||
Created:
|
||||
|
||||
* [X] Traefik-forward-auth configured to authenticate against KeyCloak
|
||||
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
|
||||
1. KeyCloak is very powerful. You can add 2FA and all other clever things outside of the scope of this simple recipe ;)
|
||||
@@ -234,6 +234,6 @@ You should now be able to access your traefik instance on http://<node IP\>:8080
|
||||
* [X] Automatic SSL support for all proxied resources
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
## Chef's Notes
|
||||
|
||||
1. Did you notice how no authentication was required to view the Traefik dashboard? Eek! We'll tackle that in the next section, regarding [Traefik Forward Authentication](https://geek-cookbook.funkypenguin.co.nz/)ha-docker-swarm/traefik-forward-auth/)!
|
||||
@@ -0,0 +1,239 @@
|
||||
# Traefik
|
||||
|
||||
The platforms we plan to run on our cloud are generally web-based, and each listening on their own unique TCP port. When a container in a swarm exposes a port, then connecting to **any** swarm member on that port will result in your request being forwarded to the appropriate host running the container. (_Docker calls this the swarm "[routing mesh](https://docs.docker.com/engine/swarm/ingress/)"_)
|
||||
|
||||
So we get a rudimentary load balancer built into swarm. We could stop there, just exposing a series of ports on our hosts, and making them HA using keepalived.
|
||||
|
||||
There are some gaps to this approach though:
|
||||
|
||||
- No consideration is given to HTTPS. Implementation would have to be done manually, per-container.
|
||||
- No mechanism is provided for authentication outside of that which the container providers. We may not **want** to expose every interface on every container to the world, especially if we are playing with tools or containers whose quality and origin are unknown.
|
||||
|
||||
To deal with these gaps, we need a front-end load-balancer, and in this design, that role is provided by [Traefik](https://traefik.io/).
|
||||
|
||||
|
||||
|
||||
## Ingredients
|
||||
|
||||
!!! summary "You'll need"
|
||||
Existing
|
||||
|
||||
* [X] [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph)
|
||||
|
||||
New
|
||||
|
||||
* [ ] Access to update your DNS records for manual/automated [LetsEncrypt](https://letsencrypt.org/docs/challenge-types/) DNS-01 validation, or ingress HTTP/HTTPS for HTTP-01 validation
|
||||
|
||||
## Preparation
|
||||
|
||||
### Prepare the host
|
||||
|
||||
The traefik container is aware of the __other__ docker containers in the swarm, because it has access to the docker socket at **/var/run/docker.sock**. This allows traefik to dynamically configure itself based on the labels found on containers in the swarm, which is hugely useful. To make this functionality work on a SELinux-enabled CentOS7 host, we need to add custom SELinux policy.
|
||||
|
||||
!!! tip
|
||||
The following is only necessary if you're using SELinux!
|
||||
|
||||
Run the following to build and activate policy to permit containers to access docker.sock:
|
||||
|
||||
```
|
||||
mkdir ~/dockersock
|
||||
cd ~/dockersock
|
||||
curl -O https://raw.githubusercontent.com/dpw/\
|
||||
selinux-dockersock/master/Makefile
|
||||
curl -O https://raw.githubusercontent.com/dpw/\
|
||||
selinux-dockersock/master/dockersock.te
|
||||
make && semodule -i dockersock.pp
|
||||
```
|
||||
|
||||
### Prepare traefik.toml
|
||||
|
||||
While it's possible to configure traefik via docker command arguments, I prefer to create a config file (`traefik.toml`). This allows me to change traefik's behaviour by simply changing the file, and keeps my docker config simple.
|
||||
|
||||
Create `/var/data/traefik/traefik.toml` as follows:
|
||||
|
||||
```
|
||||
checkNewVersion = true
|
||||
defaultEntryPoints = ["http", "https"]
|
||||
|
||||
# This section enable LetsEncrypt automatic certificate generation / renewal
|
||||
[acme]
|
||||
email = "<your LetsEncrypt email address>"
|
||||
storage = "acme.json" # or "traefik/acme/account" if using KV store
|
||||
entryPoint = "https"
|
||||
acmeLogging = true
|
||||
onDemand = true
|
||||
OnHostRule = true
|
||||
|
||||
# Request wildcard certificates per https://docs.traefik.io/configuration/acme/#wildcard-domains
|
||||
[[acme.domains]]
|
||||
main = "*.example.com"
|
||||
sans = ["example.com"]
|
||||
|
||||
# Redirect all HTTP to HTTPS (why wouldn't you?)
|
||||
[entryPoints]
|
||||
[entryPoints.http]
|
||||
address = ":80"
|
||||
[entryPoints.http.redirect]
|
||||
entryPoint = "https"
|
||||
[entryPoints.https]
|
||||
address = ":443"
|
||||
[entryPoints.https.tls]
|
||||
|
||||
[web]
|
||||
address = ":8080"
|
||||
watch = true
|
||||
|
||||
[docker]
|
||||
endpoint = "tcp://127.0.0.1:2375"
|
||||
domain = "example.com"
|
||||
watch = true
|
||||
swarmmode = true
|
||||
```
|
||||
|
||||
### Prepare the docker service config
|
||||
|
||||
!!! tip
|
||||
"We'll want an overlay network, independent of our traefik stack, so that we can attach/detach all our other stacks (including traefik) to the overlay network. This way, we can undeploy/redepoly the traefik stack without having to bring every other stack first!" - voice of experience
|
||||
|
||||
Create `/var/data/config/traefik/traefik.yml` as follows:
|
||||
|
||||
```
|
||||
version: "3.2"
|
||||
|
||||
# What is this?
|
||||
#
|
||||
# This stack exists solely to deploy the traefik_public overlay network, so that
|
||||
# other stacks (including traefik-app) can attach to it
|
||||
|
||||
services:
|
||||
scratch:
|
||||
image: scratch
|
||||
deploy:
|
||||
replicas: 0
|
||||
networks:
|
||||
- public
|
||||
|
||||
networks:
|
||||
public:
|
||||
driver: overlay
|
||||
attachable: true
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.16.200.0/24
|
||||
```
|
||||
|
||||
!!! tip
|
||||
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
|
||||
|
||||
|
||||
Create `/var/data/config/traefik/traefik-app.yml` as follows:
|
||||
|
||||
```
|
||||
version: "3"
|
||||
|
||||
services:
|
||||
traefik:
|
||||
image: traefik
|
||||
command: --web --docker --docker.swarmmode --docker.watch --docker.domain=example.com --logLevel=DEBUG
|
||||
# Note below that we use host mode to avoid source nat being applied to our ingress HTTP/HTTPS sessions
|
||||
# Without host mode, all inbound sessions would have the source IP of the swarm nodes, rather than the
|
||||
# original source IP, which would impact logging. If you don't care about this, you can expose ports the
|
||||
# "minimal" way instead
|
||||
ports:
|
||||
- target: 80
|
||||
published: 80
|
||||
protocol: tcp
|
||||
mode: host
|
||||
- target: 443
|
||||
published: 443
|
||||
protocol: tcp
|
||||
mode: host
|
||||
- target: 8080
|
||||
published: 8080
|
||||
protocol: tcp
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- /var/data/config/traefik:/etc/traefik
|
||||
- /var/data/traefik/traefik.log:/traefik.log
|
||||
- /var/data/traefik/acme.json:/acme.json
|
||||
networks:
|
||||
- traefik_public
|
||||
# Global mode makes an instance of traefik listen on _every_ node, so that regardless of which
|
||||
# node the request arrives on, it'll be forwarded to the correct backend service.
|
||||
deploy:
|
||||
labels:
|
||||
- "traefik.enable=false"
|
||||
mode: global
|
||||
placement:
|
||||
constraints: [node.role == manager]
|
||||
restart_policy:
|
||||
condition: on-failure
|
||||
|
||||
networks:
|
||||
traefik_public:
|
||||
external: true
|
||||
```
|
||||
|
||||
Docker won't start a service with a bind-mount to a non-existent file, so prepare an empty acme.json (_with the appropriate permissions_) by running:
|
||||
|
||||
```
|
||||
touch /var/data/traefik/acme.json
|
||||
chmod 600 /var/data/traefik/acme.json
|
||||
```
|
||||
|
||||
!!! warning
|
||||
Pay attention above. You **must** set `acme.json`'s permissions to owner-readable-only, else the container will fail to start with an [ID-10T](https://en.wikipedia.org/wiki/User_error#ID-10-T_error) error!
|
||||
|
||||
Traefik will populate acme.json itself when it runs, but it needs to exist before the container will start (_Chicken, meet egg._)
|
||||
|
||||
|
||||
|
||||
## Serving
|
||||
|
||||
### Launch
|
||||
|
||||
First, launch the traefik stack, which will do nothing other than create an overlay network by running `docker stack deploy traefik -c /var/data/traefik/traefik.yml`
|
||||
|
||||
```
|
||||
[root@kvm ~]# docker stack deploy traefik -c traefik.yml
|
||||
Creating network traefik_public
|
||||
Creating service traefik_scratch
|
||||
[root@kvm ~]#
|
||||
```
|
||||
|
||||
Now deploy the traefik appliation itself (*which will attach to the overlay network*) by running `docker stack deploy traefik-app -c /var/data/traefik/traefik-app.yml`
|
||||
|
||||
```
|
||||
[root@kvm ~]# docker stack deploy traefik-app -c traefik-app.yml
|
||||
Creating service traefik-app_app
|
||||
[root@kvm ~]#
|
||||
```
|
||||
|
||||
Confirm traefik is running with `docker stack ps traefik-app`:
|
||||
|
||||
```
|
||||
[root@kvm ~]# docker stack ps traefik-app
|
||||
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
|
||||
74uipz4sgasm traefik-app_app.t4vcm8siwc9s1xj4c2o4orhtx traefik:alpine kvm.funkypenguin.co.nz Running Running 33 seconds ago *:443->443/tcp,*:80->80/tcp
|
||||
[root@kvm ~]#
|
||||
```
|
||||
|
||||
### Check Traefik Dashboard
|
||||
|
||||
You should now be able to access your traefik instance on http://<node IP\>:8080 - It'll look a little lonely currently (*below*), but we'll populate it as we add recipes :)
|
||||
|
||||
|
||||
|
||||
### Summary
|
||||
|
||||
!!! summary
|
||||
We've achieved:
|
||||
|
||||
* [X] An overlay network to permit traefik to access all future stacks we deploy
|
||||
* [X] Frontend proxy which will dynamically configure itself for new backend containers
|
||||
* [X] Automatic SSL support for all proxied resources
|
||||
|
||||
|
||||
## Chef's Notes 📓
|
||||
|
||||
1. Did you notice how no authentication was required to view the Traefik dashboard? Eek! We'll tackle that in the next section, regarding [Traefik Forward Authentication](/ha-docker-swarm/traefik-forward-auth/)!
|
||||
Reference in new issue
Block a user