public/geek-cookbook
1
0
Fork 0
mirror of https://github.com/funkypenguin/geek-cookbook/ synced 2025-12-13 01:36:23 +00:00
Code Issues Releases Wiki Activity

Update for git mirror, separate dev build

Browse Source
This commit is contained in:
David Young
2017-07-25 16:33:35 +12:00
parent 0ab826de05
commit c162535ebd
7 changed files with 423 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
.gitlab-ci.yml
View File

@@ -15,18 +15,29 @@ build site:
expire_in: 1 day
paths:
- public
only:
- master
test site:
stage: test
script:
- echo fake result as a placeholder
deploy site:
deploy dev:
image: garland/docker-s3cmd
stage: deploy
environment: production
except:
- master
script:
- export LC_ALL=C.UTF-8
- export LANG=C.UTF-8
- s3cmd --no-mime-magic --access_key=$ACCESS_KEY --secret_key=$SECRET_KEY --acl-public --delete-removed --delete-after --no-ssl --host=$S3HOST --host-bucket='$S3HOSTBUCKET' sync public s3://geeks-cookbook-dev
deploy prod:
image: garland/docker-s3cmd
stage: deploy
environment: production
only:
- master
script:
- export LC_ALL=C.UTF-8
- export LANG=C.UTF-8

241
docs/ha-docker-swarm/docker-swarm-mode.md
View File

@@ -1,5 +1,207 @@
# Introduction
For truly highly-available services with Docker containers, we need an orchestration system. Docker Swarm (as defined at 1.13) is the simplest way to achieve redundancy, such that a single docker host could be turned off, and none of our services will be interrupted.
## Ingredients
* 3 x CentOS Atomic hosts (bare-metal or VMs). A reasonable minimum would be:
* 1 x vCPU
* 1GB repo_name
* 10GB HDD
* Hosts must be within the same subnet, and connected on a low-latency link (i.e., no WAN links)
## Preparation
### Release the swarm!
Now, to launch my swarm:
```docker swarm init```
Yeah, that was it. Now I have a 1-node swarm.
```
[root@ds1 ~]# docker swarm init
Swarm initialized: current node (b54vls3wf8xztwfz79nlkivt8) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-bsud7xnvhv4cicwi7l6c9s6l0 \
202.170.164.47:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
[root@ds1 ~]#
```
Run ```docker node ls``` to confirm that I have a 1-node swarm:
```
[root@ds1 ~]# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
b54vls3wf8xztwfz79nlkivt8 * ds1.funkypenguin.co.nz Ready Active Leader
[root@ds1 ~]#
```
Note that when I ran ```docker swarm init``` above, the CLI output gave me a command to run to join further nodes to my swarm. This would join the nodes as __workers__ (as opposed to __managers__). Workers can easily be promoted to managers (and demoted again), but since we know that we want our other two nodes to be managers too, it's simpler just to add them to the swarm as managers immediately.
On the first swarm node, generate the necessary token to join another manager by running ```docker swarm join-token manager```:
```
[root@ds1 ~]# docker swarm join-token manager
To add a manager to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-cfm24bq2zvfkcwujwlp5zqxta \
202.170.164.47:2377
[root@ds1 ~]#
```
Run the command provided on your second node to join it to the swarm as a manager. After adding the second node, the output of ```docker node ls``` (on either host) should reflect two nodes:
````
[root@ds2 davidy]# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
b54vls3wf8xztwfz79nlkivt8 ds1.funkypenguin.co.nz Ready Active Leader
xmw49jt5a1j87a6ihul76gbgy * ds2.funkypenguin.co.nz Ready Active Reachable
[root@ds2 davidy]#
````
Repeat the process to add your third node. **You need a new token for the third node, don't re-use the manager token you generated for the second node**.
!!! warning "Seriously. Don't use a token more than once, else it's swarm-rebuilding time."
Finally, ```docker node ls``` should reflect that you have 3 reachable manager nodes, one of whom is the "Leader":
```
[root@ds3 ~]# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
36b4twca7i3hkb7qr77i0pr9i ds1.openstack.dev.safenz.net Ready Active Reachable
l14rfzazbmibh1p9wcoivkv1s * ds3.openstack.dev.safenz.net Ready Active Reachable
tfsgxmu7q23nuo51wwa4ycpsj ds2.openstack.dev.safenz.net Ready Active Leader
[root@ds3 ~]#
```
### Create registry mirror
Although we now have shared storage for our persistent container data, our docker nodes don't share any other docker data, such as container images. This results in an inefficiency - every node which participates in the swarm will, at some point, need the docker image for every container deployed in the swarm.
When dealing with large container (looking at you, GitLab!), this can result in several gigabytes of wasted bandwidth per-node, and long delays when restarting containers on an alternate node. (_It also wastes disk space on each node, but we'll get to that in the next section_)
The solution is to run an official Docker registry container as a ["pull-through" cache, or "registry mirror"](https://docs.docker.com/registry/recipes/mirror/). By using our persistent storage for the registry cache, we can ensure we have a single copy of all the containers we've pulled at least once. After the first pull, any subsequent pulls from our nodes will use the cached version from our registry mirror. As a result, services are available more quickly when restarting container nodes, and we can be more aggressive about cleaning up unused containers on our nodes (more later)
The registry mirror runs as a swarm stack, using a simple docker-compose.yml. Customize __your mirror FQDN__ below, so that Traefik will generate the appropriate LetsEncrypt certificates for it, and make it available via HTTPS.
```
version: "3"
services:
registry-mirror:
image: registry:2
networks:
- traefik
deploy:
labels:
- traefik.frontend.rule=Host:<your mirror FQDN>
- traefik.docker.network=traefik
- traefik.port=5000
ports:
- 5000:5000
volumes:
- /var/data/registry/registry-mirror-data:/var/lib/registry
- /var/data/registry/registry-mirror-config.yml:/etc/docker/registry/config.yml
networks:
traefik:
external: true
```
!!! note "Unencrypted registry"
We create this registry without consideration for SSL, which will fail if we attempt to use the registry directly. However, we're going to use the HTTPS-proxied version via Traefik, leveraging Traefik to manage the LetsEncrypt certificates required.
Create registry/registry-mirror-config.yml as follows:
```
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
delete:
enabled: true
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
proxy:
remoteurl: https://registry-1.docker.io
```
### Enable registry mirror and experimental features
To tell docker to use the registry mirror, and in order to be able to watch the logs of any service from any manager node (_an experimental feature in the current Atomic docker build_), edit **/etc/docker-latest/daemon.json** on each node, and change from:
```
{
"log-driver": "journald",
"signature-verification": false
}
```
To:
```
{
"log-driver": "journald",
"signature-verification": false,
"experimental": true,
"registry-mirrors": ["https://<your registry mirror FQDN>"]
}
```
!!! tip ""
Note the extra comma required after "false" above
### Setup automated cleanup
This needs to be a docker-compose.yml file, excluding trusted images (like glusterfs, traefik, etc)
```
docker run -d \
-v /var/run/docker.sock:/var/run/docker.sock:rw \
-v /var/lib/docker:/var/lib/docker:rw \
meltwater/docker-cleanup:latest
```
### Tweaks
Add some handy bash auto-completion for docker. Without this, you'll get annoyed that you can't autocomplete ```docker stack deploy <blah> -c <blah.yml>``` commands.
```
cd /etc/bash_completion.d/
curl -O https://raw.githubusercontent.com/docker/cli/b75596e1e4d5295ac69b9934d1bd8aff691a0de8/contrib/completion/bash/docker
```
Install some useful bash aliases on each host
```
cd ~
curl -O https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook-recipies/raw/master/bash/gcb-aliases.sh
echo 'source ~/gcb-aliases.sh' >> ~/.bash_profile
```
````
@@ -9,3 +211,42 @@ curl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/Makefile
curl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/dockersock.te
make && semodule -i dockersock.pp
````
## Setup registry
docker run -d \
-p 5000:5000 \
--restart=always \
--name registry \
-v /mnt/registry:/var/lib/registry \
registry:2
{
"log-driver": "journald",
"signature-verification": false,
"experimental": true,
"registry-mirrors": ["https://registry-mirror.funkypenguin.co.nz"]
}
registry-mirror:
image: registry:2
ports:
- 5000:5000
environment:
volumes:
- /var/data/registry:/var/lib/registry
[root@ds1 dockersock]# docker swarm join-token manager
To add a manager to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-09c94wv0opw0y6xg67uzjl13pnv8lxxn586hrg5f47spso9l6j-6zn3dxk7c4zkb19r61owasi15 \
192.168.31.11:2377
[root@ds1 dockersock]#

27
docs/ha-docker-swarm/maintenance.md
View File

@@ -1,3 +1,30 @@
# Introduction
## Adding a host
## Adding storage
gluster volume add-brick VOLNAME NEW_BRICK
example
# gluster volume add-brick test-volume server4:/exp4
Add Brick successful
# Replacing failed host
[root@glusterfs-server /]# gluster peer status
Number of Peers: 1
Hostname: ds1
Uuid: db9c80da-11e4-461d-8ea5-66dd12ca897c
State: Peer in Cluster (Disconnected)
[root@glusterfs-server /]#
Grab UUID above
edit /var/lib/glusterd/glusterd.info
change:
UUID=aee45c2c-aa19-4d29-bc94-4833f2b22863
to
UUID=db9c80da-11e4-461d-8ea5-66dd12ca897c

12
docs/ha-docker-swarm/shared-storage.md
View File

@@ -130,6 +130,10 @@ echo '' >> /etc/fstab >> /etc/fstab
echo '# Mount glusterfs volume' >> /etc/fstab
echo "$MYHOST:/gv0 /var/data glusterfs defaults,_netdev,context="system_u:object_r:svirt_sandbox_file_t:s0" 0 0" >> /etc/fstab
mount -a
```
For some reason, my nodes won't auto-mount this volume on boot. I even tried the trickery below, but they stubbornly refuse to automount.
```
echo -e "\n\n# Give GlusterFS 10s to start before \
mounting\nsleep 10s && mount -a" >> /etc/rc.local
systemctl enable rc-local.service
@@ -137,9 +141,17 @@ systemctl enable rc-local.service
For non-gluster nodes, you'll need to replace $MYHOST above with the name of one of the gluster hosts (I haven't worked out how to make this fully HA yet)
## Serving
!!! summary "Ready to serve..."
After completing the above, you should have:
* [X] Persistent storage available to every node
* [X] Resiliency in the event of the failure of a single (gluster) node
## Sides
Future enhancements to this recipe inculde:
1. Migration of shared storage from GlusterFS to Ceph ()[#2](https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook/issues/2))
2. Correct the fact that volumes don't automount on boot ([#3](https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook/issues/3))

128
docs/ha-docker-swarm/traefik.md
View File

@@ -11,14 +11,136 @@ There are some gaps to this approach though:
To deal with these gaps, we need a front-end load-balancer, and in this design, that role is provided by [Traefik](https://traefik.io/).
## Prepare the host
## Ingredients
## Preparation
### Prepare the host
The traefik container is aware of the __other__ docker containers in the swarm, because it has access to the docker socket at **/var/run/docker.sock**. This allows traefik to dynamically configure itself based on the labels found on containers in the swarm, which is hugely useful. To make this functionality work on our SELinux-enabled Atomic hosts, we need to add custom SELinux policy.
Run the following to build and activate policy to permit containers to access docker.sock:
````
mkdir ~/dockersock
cd ~/dockersock
curl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/Makefile
curl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/dockersock.te
curl -O https://raw.githubusercontent.com/dpw/\
selinux-dockersock/master/Makefile
curl -O https://raw.githubusercontent.com/dpw/\
selinux-dockersock/master/dockersock.te
make && semodule -i dockersock.pp
````
### Prepare traefik.toml
While it's possible to configure traefik via docker command arguments, I prefer to create a config file (traefik.toml). This allows me to change traefik's behaviour by simply changing the file, and keeps my docker config simple.
Create /var/data/traefik/traefik.toml as follows:
```
checkNewVersion = true
defaultEntryPoints = ["http", "https"]
# This section enable LetsEncrypt automatic certificate generation / renewal
[acme]
email = "<your LetsEncrypt email address>"
storage = "acme.json" # or "traefik/acme/account" if using KV store
entryPoint = "https"
acmeLogging = true
onDemand = true
OnHostRule = true
[[acme.domains]]
main = "<your primary domain>"
# Redirect all HTTP to HTTPS (why wouldn't you?)
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[web]
address = ":8080"
watch = true
[docker]
endpoint = "tcp://127.0.0.1:2375"
domain = "<your primary domain>"
watch = true
swarmmode = true
```
### Prepare the docker service config
Create /var/data/traefik/docker-compose.yml as follows:
```
version: "3.2"
services:
traefik:
image: traefik
command: --web --docker --docker.swarmmode --docker.watch --docker.domain=funkypenguin.co.nz --logLevel=DEBUG
ports:
- target: 80
published: 80
protocol: tcp
mode: host
- target: 443
published: 443
protocol: tcp
mode: host
- target: 8080
published: 8080
protocol: tcp
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /var/data/traefik/traefik.toml:/traefik.toml:ro
- /var/data/traefik/acme.json:/acme.json
labels:
- "traefik.enable=false"
networks:
- public
deploy:
mode: global
placement:
constraints: [node.role == manager]
restart_policy:
condition: on-failure
networks:
public:
driver: overlay
ipam:
driver: default
config:
- subnet: 10.1.0.0/24
```
Docker won't start an image with a bind-mount to a non-existent file, so prepare acme.json by running ```touch /var/data/traefik/acme.json```.
### Launch
Deploy traefik with ```docker stack deploy traefik -c /var/data/traefik/docker-compose.yml```
Confirm traefik is running with ```docker stack ps traefik```
## Serving
You now have:
1. Frontend proxy which will dynamically configure itself for new backend containers
2. Automatic SSL support for all proxied resources
## Extra Toppings
Additional features I'd like to see in this recipe are:
1. Include documentation of oauth2_proxy container for protecting individual backends
2. Traefik webUI is available via HTTPS, protected with oauth_proxy
3. Pending a feature in docker-swarm to avoid NAT on routing-mesh-delivered traffic, update the design

35
docs/ha-docker-swarm/vms.md
View File

@@ -32,6 +32,7 @@ I chose the "[Atomic](https://www.projectatomic.io/)" CentOS/Fedora image for th
!!! tip
If you're not using a platform with cloud-init support (i.e., you're building a VM manually, not provisioning it through a cloud provider), you'll need to refer to [trick #1][atomic-trick1] and [#2][atomic-trick2] for a means to override the automated setup, apply a manual password to the CentOS account, and enable SSH password logins.
### Change to latest docker
Run the following on each node to replace the default docker 1.12 with docker 1.13 (_which we need for swarm mode_):
@@ -41,44 +42,12 @@ systemctl enable docker-latest --now
sed -i '/DOCKERBINARY/s/^#//g' /etc/sysconfig/docker
```
### Enable docker experimental features
In order to be able to watch the logs of any service from any manager node, we need to enable "experimental features" in docker. (It's no longer experimental in mainstream, but under the current Atomic).
To effect this, on each node, edit **/etc/docker-latest/daemon.json**, and change from:
```
{
"log-driver": "journald",
"signature-verification": false
}
```
To:
```
{
"log-driver": "journald",
"signature-verification": false,
"experimental": true
}
```
!!! tip ""
Note the extra comma required after "false" above
Add some handy bash auto-completion for docker. Without this, you'll get annoyed that you can't autocomplete ```docker stack deploy <blah> -c <blah.yml>``` commands.
```
cd /etc/bash_completion.d/
curl -O https://raw.githubusercontent.com/docker/cli/b75596e1e4d5295ac69b9934d1bd8aff691a0de8/contrib/completion/bash/docker
```
### Upgrade Atomic
Finally, apply any Atomic host updates, and reboot, by running: ```atomic host upgrade && systemctl reboot```.
### Permit connectivity between VMs
By default, Atomic only permits incoming SSH. We'll want to allow all traffic between our nodes, so add something like this to /etc/sysconfig/iptables:

3
examples/cloud-config.txt
View File

@@ -5,7 +5,8 @@ write_files:
{
"log-driver": "journald",
"signature-verification": false,
"experimental": true
"experimental": true,
"registry-mirrors": ["https://registry-mirror.funkypenguin.co.nz"]
}
# Add users to the system. Users are added after groups are added.
users: