diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index b058ce6..7806376 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -24,7 +24,7 @@ test site: deploy dev: image: garland/docker-s3cmd stage: deploy - environment: production + environment: development except: - master script: @@ -36,9 +36,10 @@ deploy prod: image: garland/docker-s3cmd stage: deploy environment: production - only: + except: - master script: + - apk add rsync openssh --no-cache - export LC_ALL=C.UTF-8 - export LANG=C.UTF-8 - s3cmd --no-mime-magic --access_key=$ACCESS_KEY --secret_key=$SECRET_KEY --acl-public --delete-removed --delete-after --no-ssl --host=$S3HOST --host-bucket='$S3HOSTBUCKET' sync public s3://geeks-cookbook diff --git a/docs/README.md b/docs/README.md index 7a06b4e..2d12d8f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,6 +2,10 @@ ## Structure -1. "Recipies" are sorted by degree of geekiness required to complete them. Relatively straightforward projects are "beginner", more complex projects are "intermediate", and the really fun ones are "advanced". -2. Each recipe contains enough detail in a single page to take a project to completion. +1. "Recipies" generally follow on from each other. I.e., if a particular recipe requires a mail server, that mail server would have been described in an earlier recipe. +2. Each recipe contains enough detail in a single page to take a project from start to completion. 3. When there are optional add-ons/integrations possible to a project (i.e., the addition of "smart LED bulbs" to Home Assistant), this will be reflected either as a brief "Chef's note" after the recipe, or if they're substantial enough, as a sub-page of the main project + +## Conventions + +1. When creating swarm networks, we always explicitly set the subnet in the overlay network, to avoid potential conflicts (which docker won't prevent, but which will generate errors) (https://github.com/moby/moby/issues/26912) diff --git a/docs/ha-docker-swarm/design.md b/docs/ha-docker-swarm/design.md index 101373e..c47f0e3 100644 --- a/docs/ha-docker-swarm/design.md +++ b/docs/ha-docker-swarm/design.md @@ -1,4 +1,4 @@ -# Introduction +# Design In the design described below, the "private cloud" platform is: @@ -24,6 +24,20 @@ This means that: * Services are defined using docker-compose v3 YAML syntax * Services are portable, meaning a particular stack could be shut down and moved to a new provider with minimal effort. +## Security + +Under this design, the only inbound connections we're permitting to our docker swarm are: + +### Network Flows + +* HTTP (TCP 80) : Redirects to https +* HTTPS (TCP 443) : Serves individual docker containers via SSL-encrypted reverse proxy + +### Authentication + +* Where the proxied application provides a trusted level of authentication, or where the application requires public exposure, + + ## High availability ### Normal function diff --git a/docs/ha-docker-swarm/docker-swarm-mode.md b/docs/ha-docker-swarm/docker-swarm-mode.md index 095e484..14d5ca9 100644 --- a/docs/ha-docker-swarm/docker-swarm-mode.md +++ b/docs/ha-docker-swarm/docker-swarm-mode.md @@ -1,4 +1,4 @@ -# Introduction +# Docker Swarm Mode For truly highly-available services with Docker containers, we need an orchestration system. Docker Swarm (as defined at 1.13) is the simplest way to achieve redundancy, such that a single docker host could be turned off, and none of our services will be interrupted. diff --git a/docs/ha-docker-swarm/keepalived.md b/docs/ha-docker-swarm/keepalived.md index b6c3108..2f2cfcf 100644 --- a/docs/ha-docker-swarm/keepalived.md +++ b/docs/ha-docker-swarm/keepalived.md @@ -1,4 +1,4 @@ -# Introduction +# Keepalived While having a self-healing, scalable docker swarm is great for availability and scalability, none of that is any good if nobody can connect to your cluster. diff --git a/docs/ha-docker-swarm/shared-storage-ceph.md b/docs/ha-docker-swarm/shared-storage-ceph.md new file mode 100644 index 0000000..7289d44 --- /dev/null +++ b/docs/ha-docker-swarm/shared-storage-ceph.md @@ -0,0 +1,186 @@ +# Shared Storage (Ceph) + +While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node. + +## Design + +### Why not GlusterFS? +I originally provided shared storage to my nodes using GlusterFS (see the next recipe for details), but found it difficult to deal with because: + +1. GlusterFS requires (n) "bricks", where (n) **has** to be a multiple of your replica count. I.e., if you want 2 copies of everything on shared storage (the minimum to provide redundancy), you **must** have either 2, 4, 6 (etc..) bricks. The HA swarm design calls for minimum of 3 nodes, and so under GlusterFS, my third node can't participate in shared storage at all, unless I start doubling up on bricks-per-node (which then impacts redundancy) +2. GlusterFS turns out to be a giant PITA when you want to restore a failed node. There are at [least 14 steps to follow](https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/sect-Replacing_Hosts.html) to replace a brick. +3. I'm pretty sure I messed up the 14-step process above anyway. My replaced brick synced with my "original" brick, but produced errors when querying status via the CLI, and hogged 100% of 1 CPU on the replaced node. Inexperienced with GlusterFS, and unable to diagnose the fault, I switched to a Ceph cluster instead. + +### Why Ceph? + +1. I'm more familiar with Ceph - I use it in the OpenStack designs I manage +2. Replacing a failed node is **easy**, provided you can put up with the I/O load of rebalancing OSDs after the replacement. +3. CentOS Atomic includes the ceph client in the OS, so while the Ceph OSD/Mon/MSD are running under containers, I can keep an eye (and later, automatically monitor) the status of Ceph from the base OS. + +## Ingredients + +!!! summary "Ingredients" + 3 x Virtual Machines (configured earlier), each with: + + * [X] CentOS/Fedora Atomic + * [X] At least 1GB RAM + * [X] At least 20GB disk space (_but it'll be tight_) + * [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_) + * [ ] A second disk dedicated to the Ceph OSD + +## Preparation + +### SELinux + +Since our Ceph components will be containerized, we need to ensure the SELinux context on the base OS's ceph files is set correctly: + +``` +chcon -Rt svirt_sandbox_file_t /etc/ceph +chcon -Rt svirt_sandbox_file_t /var/lib/ceph +``` +### Setup Monitors + +Pick a node, and run the following to stand up the first Ceph mon. Be sure to replace the values for **MON_IP** and **CEPH_PUBLIC_NETWORK** to those specific to your deployment: + +``` +docker run -d --net=host \ +--restart always \ +-v /etc/ceph:/etc/ceph \ +-v /var/lib/ceph/:/var/lib/ceph/ \ +-e MON_IP=192.168.31.11 \ +-e CEPH_PUBLIC_NETWORK=192.168.31.0/24 \ +--name="ceph-mon" \ +ceph/daemon mon +``` + +Now **copy** the contents of /etc/ceph on this first node to the remaining nodes, and **then** run the docker command above (_customizing MON_IP as you go_) on each remaining node. You'll end up with a cluster with 3 monitors (odd number is required for quorum, same as Docker Swarm), and no OSDs (yet) + + +### Setup OSDs + +Since we have a OSD-less mon-only cluster currently, prepare for OSD creation by dumping the auth credentials for the OSDs into the appropriate location on the base OS: + +``` +ceph auth get client.bootstrap-osd -o \ +/var/lib/ceph/bootstrap-osd/ceph.keyring +``` + +On each node, you need a dedicated disk for the OSD. In the example below, I used _/dev/vdd_ (the entire disk, no partitions) for the OSD. + +Run the following command on every node: + +``` +docker run -d --net=host \ +--privileged=true \ +--pid=host \ +-v /etc/ceph:/etc/ceph \ +-v /var/lib/ceph/:/var/lib/ceph/ \ +-v /dev/:/dev/ \ +-e OSD_DEVICE=/dev/vdd \ +-e OSD_TYPE=disk \ +--name="ceph-osd" \ +--restart=always \ +ceph/daemon osd +``` + +Watch the output by running ```docker logs ceph-osd -f```, and confirm success. + +!!! note "Zapping the device" + The Ceph OSD container will refuse to destroy a partition containing existing data, so it may be necessary to "zap" the target disk, using: + ``` + docker run -d --privileged=true \ + -v /dev/:/dev/ \ + -e OSD_DEVICE=/dev/sdd \ + ceph/daemon zap_device + ``` + +### Setup MDSs + +In order to mount our ceph pools as filesystems, we'll need Ceph MDS(s). Run the following on each node: + +``` +docker run -d --net=host \ +--name ceph-mds \ +--restart always \ +-v /var/lib/ceph/:/var/lib/ceph/ \ +-v /etc/ceph:/etc/ceph \ +-e CEPHFS_CREATE=1 \ +-e CEPHFS_DATA_POOL_PG=256 \ +-e CEPHFS_METADATA_POOL_PG=256 \ +ceph/daemon mds +``` +### Apply tweaks + +The ceph container seems to configure a pool default of 3 replicas (3 copies of each block are retained), which is one too many for our cluster (we are only protecting against the failure of a single node). + +Run the following on any node to reduce the size of the pool to 2 replicas: + +``` +ceph osd pool set cephfs_data size 2 +ceph osd pool set cephfs_metadata size 2 +``` + +Disabled "scrubbing" (which can be IO-intensive, and is unnecessary on a VM) with: + +``` +ceph osd set noscrub +ceph osd set nodeep-scrub +``` + + +### Create credentials for swarm + +In order to mount the ceph volume onto our base host, we need to provide cephx authentication credentials. + +On **one** node, create a client for the docker swarm: + +``` +ceph auth get-or-create client.dockerswarm osd \ +'allow rw' mon 'allow r' mds 'allow' > /etc/ceph/keyring.dockerswarm +``` + +Grab the secret associated with the new user (you'll need this for the /etc/fstab entry below) by running: + +``` +ceph-authtool /etc/ceph/keyring.dockerswarm -p -n client.dockerswarm +``` + +### Mount MDS volume + +On each noie, create a mountpoint for the data, by running ```mkdir /var/data```, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually _mounted_ if there's a network / boot delay getting access to the gluster volume: + +``` +mkdir /var/data + +MYHOST=`hostname -s` +echo -e " +# Mount cephfs volume \n +$MYHOST:6789:/ /var/data/ ceph \ +name=dockerswarm\ +,secret=\ +,noatime,_netdev,context=system_u:object_r:svirt_sandbox_file_t:s0\ +0 2" >> /etc/fstab +mount -a +``` +### Install docker-volume plugin + +Upstream bug for docker-latest reported at https://bugs.centos.org/view.php?id=13609 + +And the alpine fault: +https://github.com/gliderlabs/docker-alpine/issues/317 + + +## Serving + +After completing the above, you should have: + +``` +[X] Persistent storage available to every node +[X] Resiliency in the event of the failure of a single node +``` + +## Chef's Notes + +Future enhancements to this recipe include: + +1. Rather than pasting a secret key into /etc/fstab (which feels wrong), I'd prefer to be able to set "secretfile" in /etc/fstab (which just points ceph.mount to a file containing the secret), but under the current CentOS Atomic, we're stuck with "secret", per https://bugzilla.redhat.com/show_bug.cgi?id=1030402 diff --git a/docs/ha-docker-swarm/shared-storage.md b/docs/ha-docker-swarm/shared-storage-gluster.md similarity index 94% rename from docs/ha-docker-swarm/shared-storage.md rename to docs/ha-docker-swarm/shared-storage-gluster.md index db215bf..8ce2aad 100644 --- a/docs/ha-docker-swarm/shared-storage.md +++ b/docs/ha-docker-swarm/shared-storage-gluster.md @@ -1,7 +1,13 @@ -# Introduction +# Shared Storage (GlusterFS) While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node. +## Design + +### Why GlusterFS? + +This GlusterFS recipe was my original design for shared storage, but I [found it to be flawed](ha-docker-swarm/shared-storage-ceph/#why-not-glusterfs), and I replaced it with a [design which employs Ceph instead](http://localhost:8000/ha-docker-swarm/shared-storage-ceph/#why-ceph). This recipe is an alternate to the Ceph design, if you happen to prefer GlusterFS. + ## Ingredients !!! summary "Ingredients" diff --git a/docs/ha-docker-swarm/traefik.md b/docs/ha-docker-swarm/traefik.md index 3acb18d..28ad343 100644 --- a/docs/ha-docker-swarm/traefik.md +++ b/docs/ha-docker-swarm/traefik.md @@ -1,4 +1,4 @@ -# Introduction +# Traefik The platforms we plan to run on our cloud are generally web-based, and each listening on their own unique TCP port. When a container in a swarm exposes a port, then connecting to **any** swarm member on that port will result in your request being forwarded to the appropriate host running the container. (_Docker calls this the swarm "[routing mesh](https://docs.docker.com/engine/swarm/ingress/)"_) @@ -21,7 +21,7 @@ The traefik container is aware of the __other__ docker containers in the swarm, Run the following to build and activate policy to permit containers to access docker.sock: -```` +``` mkdir ~/dockersock cd ~/dockersock curl -O https://raw.githubusercontent.com/dpw/\ @@ -29,7 +29,7 @@ selinux-dockersock/master/Makefile curl -O https://raw.githubusercontent.com/dpw/\ selinux-dockersock/master/dockersock.te make && semodule -i dockersock.pp -```` +``` ### Prepare traefik.toml diff --git a/docs/ha-docker-swarm/vms.md b/docs/ha-docker-swarm/vms.md index dcea5b4..868d2eb 100644 --- a/docs/ha-docker-swarm/vms.md +++ b/docs/ha-docker-swarm/vms.md @@ -1,4 +1,4 @@ -# Introduction +# Virtual Machines Let's start building our cloud with virtual machines. You could use bare-metal machines as well, the configuration would be the same. Given that most readers (myself included) will be using virtual infrastructure, from now on I'll be referring strictly to VMs. diff --git a/docs/recipies/mail.md b/docs/recipies/mail.md new file mode 100644 index 0000000..1315878 --- /dev/null +++ b/docs/recipies/mail.md @@ -0,0 +1,152 @@ +# Mail Server + +Many of the recipies that follow require email access of some kind. It's normally possible to use a hosted service such as SendGrid, or just a gmail account. If (like me) you'd like to self-host email for your stacks, then the following recipe provides a full-stack mail server running on the docker HA swarm. + +Of value to me in choosing docker-mailserver were: + +1. Automatically renews LetsEncrypt certificates +2. Creation of email accounts across multiple domains (i.e., the same container gives me mailbox wekan@wekan.example.com, and gitlab@gitlab.example.com) +3. The entire configuration is based on flat files, so there's no database or persistence to worry about + +docker-mailserver doesn't include a webmail client, and one is not strictly needed. Rainloop can be added either as another service within the stack, or as a standalone service. Rainloop will be covered in a future recipe. + +## Ingredients + +1. [Docker swarm cluster](/ha-docker-swarm/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md) +2. [Traefik](/ha-docker-swarm/traefik) configured per design +3. LetsEncrypt authorized email address for domain +4. Access to manage DNS records for domains + +## Preparation + +### Setup data locations + +We'll need several directories to bind-mount into our container, so create them in /var/data/mailserver: + +``` +cd /var/data +mkdir mailserver +cd mailserver +mkdir {maildata,mailstate,config,letsencrypt} +``` + +### Get LetsEncrypt certificate + +Decide on the FQDN to assign to your mailserver. You can service multiple domains from a single mailserver - i.e., bob@dev.example.com and daphne@prod.example.com can both be served by **mail.example.com**. + +The docker-mailserver container can _renew_ our LetsEncrypt certs for us, but it can't generate them. To do this, we need to run certbot (from a container) to request the initial certs and create the appropriate directory structure. + +In the example below, since I'm already using Traefik to manage the LE certs for my web platforms, I opted to use the DNS challenge to prove my ownership of the domain. The certbot client will prompt you to add a DNS record for domain verification. + +``` +docker run -ti --rm -v \ +"$(pwd)"/letsencrypt:/etc/letsencrypt certbot/certbot \ +--manual --preferred-challenges dns certonly \ +-d mail.example.com +``` + +### Get setup.sh + +docker-mailserver comes with a handy bash script for managing the stack (which is just really a wrapper around the container.) It'll make our setup easier, so download it into the root of your configuration/data directory, and make it executable: + +``` +curl -o setup.sh \ +https://raw.githubusercontent.com/tomav/docker-mailserver/master/setup.sh \ +chmod a+x ./setup.sh +``` +### Create email accounts + +For every email address required, run ```./setup.sh email add ``` to create the account. The command returns no output. + +You can run ```./setup.sh email list``` to confirm all of your addresses have been created. + +### Create DKIM DNS entries + +Run ```./setup.sh config dkim``` to create the necessary DKIM entries. The command returns no output. + +Examine the keys created by opendkim to identify the DNS TXT records required: + +``` +for i in `find config/opendkim/keys/ -name mail.txt`; do \ +echo $i; \ +cat $i; \ +done +``` + +You'll end up with something like this: + +``` +config/opendkim/keys/gitlab.example.com/mail.txt +mail._domainkey IN TXT ( "v=DKIM1; k=rsa; " + "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB" ) ; ----- DKIM key mail for gitlab.example.com +[root@ds1 mail]# +``` + +Create the necessary DNS TXT entries for your domain(s). Note that although opendkim splits the record across two lines, the actual record should be concatenated on creation. I.e., the DNS TXT record above should read: + +``` +"v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB" +``` + +### Setup Docker Swarm + +Create a docker swarm config file in docker-compose syntax (v3), something like this: + +``` +version: '3' + +services: + mail: + image: tvial/docker-mailserver:latest + ports: + - "25:25" + - "587:587" + - "993:993" + volumes: + - /var/data/mail/maildata:/var/mail + - /var/data/mail/mailstate:/var/mail-state + - /var/data/mail/config:/tmp/docker-mailserver + - /var/data/mail/letsencrypt:/etc/letsencrypt + env_file: /var/data/mail/.env + networks: + - internal + deploy: + replicas: 1 + +networks: + traefik: + external: true + internal: + driver: overlay + ipam: + config: + - subnet: 172.16.2.0/24 +``` + +!!! tip + Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. + +A sample .env file looks like this: + +``` +ENABLE_SPAMASSASSIN=1 +ENABLE_CLAMAV=1 +ENABLE_POSTGREY=1 +ONE_DIR=1 +OVERRIDE_HOSTNAME=mail.example.com +OVERRIDE_DOMAINNAME=mail.example.com +POSTMASTER_ADDRESS=admin@example.com +PERMIT_DOCKER=network +SSL_TYPE=letsencrypt +``` + + +## Serving + +### Launch mailserver + +Launch the mail server stack by running ```docker stack deploy mailserver -c ``` + +## Chef's Notes + +1. One of the elements of this design which I didn't appreciate at first is that since the config is entirely file-based, **setup.sh** can be run on any container host, provided it has the shared data mounted. This means that even though docker-mailserver was not designed with docker swarm in mind, it works perfectl with swarm. I.e., from any node, regardless of where the container is actually running, you're able to add/delete email addresses, view logs, etc. diff --git a/examples/ceph.sh b/examples/ceph.sh new file mode 100644 index 0000000..1078d55 --- /dev/null +++ b/examples/ceph.sh @@ -0,0 +1,67 @@ +sudo chcon -Rt svirt_sandbox_file_t /etc/ceph +sudo chcon -Rt svirt_sandbox_file_t /var/lib/ceph + +docker run -d --net=host \ +--privileged=true \ +--pid=host \ +-v /etc/ceph:/etc/ceph \ +-v /var/lib/ceph/:/var/lib/ceph/ \ +-v /dev/:/dev/ \ +-e OSD_DEVICE=/dev/vdd \ +-e OSD_TYPE=disk \ +--name="ceph-osd" \ +--restart=always \ +ceph/daemon osd + + + +docker run -d --net=host \ +--restart always \ +-v /etc/ceph:/etc/ceph \ +-v /var/lib/ceph/:/var/lib/ceph/ \ +-e MON_IP=192.168.31.11 \ +-e CEPH_PUBLIC_NETWORK=192.168.31.0/24 \ +--name="ceph-mon" \ +ceph/daemon mon + +On other nodes + +ceph auth get client.bootstrap-osd -o /var/lib/ceph/bootstrap-osd/ceph.keyring + + +docker run -d --net=host \ +--name ceph-mds \ +--restart always \ +-v /var/lib/ceph/:/var/lib/ceph/ \ +-v /etc/ceph:/etc/ceph \ +-e CEPHFS_CREATE=0 \ +ceph/daemon mds + + +ceph auth get-or-create client.dockerswarm osd 'allow rw' mon 'allow r' mds 'allow' > /etc/ceph/keyring.dockerswarm + +ceph-authtool /etc/ceph/keyring.dockerswarm -p -n client.dockerswarm + +Note that current design seems to provide 3 replicas, which is probably overkill: + +[root@ds3 traefik]# ceph osd pool get cephfs_data size +size: 3 +[root@ds3 traefik]# + + +So I set it to 2 + +[root@ds3 traefik]# ceph osd pool set cephfs_data size 2 +set pool 1 size to 2 +[root@ds3 traefik]# ceph osd pool get cephfs_data size +size: 2 +[root@ds3 traefik]# + +Would like to be able to set secretfile in /etc/fstab, but for now it loosk like we're stuch with --secret, per https://bugzilla.redhat.com/show_bug.cgi?id=1030402 + +Euught. ceph writes are slow (surprise!) + +I disabled scrubbing with: + +ceph osd set noscrub +ceph osd set nodeep-scrub diff --git a/mkdocs.yml b/mkdocs.yml index 2685e87..1b408e1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -4,7 +4,8 @@ site_author: 'David Young' site_url: 'https://geeks-cookbook.funkypenguin.co.nz' # Repository -# repo_name: 'funkypenguin/geek-cookbook' +#repo_name: 'funkypenguin/geek-cookbook' +#repo_url: 'https://github.com/funkypenguin/geeks-cookbook' # repo_url: 'https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook' # Copyright @@ -17,14 +18,16 @@ pages: - Introduction: - README: README.md - whoami: whoami.md - - HA Docker Swarm: + - Essential: - Design: ha-docker-swarm/design.md - VMs: ha-docker-swarm/vms.md - - Shared Storage: ha-docker-swarm/shared-storage.md + - Shared Storage (Ceph): ha-docker-swarm/shared-storage-ceph.md + - Shared Storage (GlusterFS): ha-docker-swarm/shared-storage-gluster.md - Keepalived: ha-docker-swarm/keepalived.md - Docker Swarm Mode: ha-docker-swarm/docker-swarm-mode.md - Traefik: ha-docker-swarm/traefik.md -# - Tiny Tiny RSS: + - Recommended: + - Mail Server: recipies/mail.md # - Basic: advanced/tiny-tiny-rss.md # - Plugins: advanced/tiny-tiny-rss.md # - Themes: advanced/tiny-tiny-rss.md @@ -54,6 +57,7 @@ pages: extra: + disqus: 'geeks-cookbook' logo: 'images/site-logo.png' feature: tabs: false @@ -70,9 +74,9 @@ extra: link: 'https://twitter.com/funkypenguin' # Google Analytics -#google_analytics: -# - 'UA-XXXXXXXX-X' -# - 'auto' +google_analytics: + - 'UA-139253-18' + - 'auto' extra_javascript: - 'extras/javascript/piwik.js' diff --git a/site/404.html b/site/404.html new file mode 100644 index 0000000..4df7ca7 --- /dev/null +++ b/site/404.html @@ -0,0 +1,429 @@ + + + + + + + + + + + + + + + + + + + + + + + Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + + +
+
+ +

404 - Not found

+ + + +

Comments

+
+ + + +
+
+
+
+ + +
+ +
+
+
+ +
+ Copyright © 2016 - 2017 David Young +
+ + powered by + MkDocs + and + + Material for MkDocs +
+ + +
+ + + + + + + +
+ + +
+
+
+ +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/README/index.html b/site/README/index.html new file mode 100644 index 0000000..d0aa1d6 --- /dev/null +++ b/site/README/index.html @@ -0,0 +1,580 @@ + + + + + + + + + + + + + + + + + + + + + + + + + README - Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

How to read this book

+

Structure

+
    +
  1. "Recipies" generally follow on from each other. I.e., if a particular recipe requires a mail server, that mail server would have been described in an earlier recipe.
    2. +
  2. Each recipe contains enough detail in a single page to take a project from start to completion.
    3. +
  3. When there are optional add-ons/integrations possible to a project (i.e., the addition of "smart LED bulbs" to Home Assistant), this will be reflected either as a brief "Chef's note" after the recipe, or if they're substantial enough, as a sub-page of the main project
    4. +
+

Conventions

+
    +
  1. When creating swarm networks, we always explicitly set the subnet in the overlay network, to avoid potential conflicts (which docker won't prevent, but which will generate errors) (https://github.com/moby/moby/issues/26912)
    2. +
+ + + + + + +

Comments

+
+ + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/assets/images/favicon.ico b/site/assets/images/favicon.ico new file mode 100644 index 0000000..e85006a Binary files /dev/null and b/site/assets/images/favicon.ico differ diff --git a/site/assets/images/icons/bitbucket-670608a71a.svg b/site/assets/images/icons/bitbucket-670608a71a.svg new file mode 100644 index 0000000..7d95cb2 --- /dev/null +++ b/site/assets/images/icons/bitbucket-670608a71a.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/site/assets/images/icons/github-1da075986e.svg b/site/assets/images/icons/github-1da075986e.svg new file mode 100644 index 0000000..3cacb2e --- /dev/null +++ b/site/assets/images/icons/github-1da075986e.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/site/assets/images/icons/gitlab-5ad3f9f9e5.svg b/site/assets/images/icons/gitlab-5ad3f9f9e5.svg new file mode 100644 index 0000000..b036a9b --- /dev/null +++ b/site/assets/images/icons/gitlab-5ad3f9f9e5.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/site/assets/javascripts/application-c35428f87f.js b/site/assets/javascripts/application-c35428f87f.js new file mode 100644 index 0000000..b67be4c --- /dev/null +++ b/site/assets/javascripts/application-c35428f87f.js @@ -0,0 +1 @@ +window.app=function(e){function t(r){if(n[r])return n[r].exports;var i=n[r]={i:r,l:!1,exports:{}};return e[r].call(i.exports,i,i.exports,t),i.l=!0,i.exports}var n={};return t.m=e,t.c=n,t.d=function(e,n,r){t.o(e,n)||Object.defineProperty(e,n,{configurable:!1,enumerable:!0,get:r})},t.n=function(e){var n=e&&e.__esModule?function(){return e.default}:function(){return e};return t.d(n,"a",n),n},t.o=function(e,t){return Object.prototype.hasOwnProperty.call(e,t)},t.p="",t(t.s=35)}([function(e,t,n){"use strict";var r=n(23)("wks"),i=n(14),o=n(1).Symbol,a="function"==typeof o;(e.exports=function(e){return r[e]||(r[e]=a&&o[e]||(a?o:i)("Symbol."+e))}).store=r},function(e,t,n){"use strict";var r=e.exports="undefined"!=typeof window&&window.Math==Math?window:"undefined"!=typeof self&&self.Math==Math?self:Function("return this")();"number"==typeof __g&&(__g=r)},function(e,t,n){"use strict";var r=n(10),i=n(25);e.exports=n(5)?function(e,t,n){return r.f(e,t,i(1,n))}:function(e,t,n){return e[t]=n,e}},function(e,t,n){"use strict";var r=n(11);e.exports=function(e){if(!r(e))throw TypeError(e+" is not an object!");return e}},function(e,t,n){"use strict";var r=n(1),i=n(2),o=n(6),a=n(14)("src"),s=Function.toString,c=(""+s).split("toString");n(7).inspectSource=function(e){return s.call(e)},(e.exports=function(e,t,n,s){var u="function"==typeof n;u&&(o(n,"name")||i(n,"name",t)),e[t]!==n&&(u&&(o(n,a)||i(n,a,e[t]?""+e[t]:c.join(String(t)))),e===r?e[t]=n:s?e[t]?e[t]=n:i(e,t,n):(delete e[t],i(e,t,n)))})(Function.prototype,"toString",function(){return"function"==typeof this&&this[a]||s.call(this)})},function(e,t,n){"use strict";e.exports=!n(24)(function(){return 7!=Object.defineProperty({},"a",{get:function(){return 7}}).a})},function(e,t,n){"use strict";var r={}.hasOwnProperty;e.exports=function(e,t){return r.call(e,t)}},function(e,t,n){"use strict";var r=e.exports={version:"2.4.0"};"number"==typeof __e&&(__e=r)},function(e,t,n){"use strict";e.exports={}},function(e,t,n){"use strict";var r={}.toString;e.exports=function(e){return r.call(e).slice(8,-1)}},function(e,t,n){"use strict";var r=n(3),i=n(38),o=n(39),a=Object.defineProperty;t.f=n(5)?Object.defineProperty:function(e,t,n){if(r(e),t=o(t,!0),r(n),i)try{return a(e,t,n)}catch(e){}if("get"in n||"set"in n)throw TypeError("Accessors not supported!");return"value"in n&&(e[t]=n.value),e}},function(e,t,n){"use strict";var r="function"==typeof Symbol&&"symbol"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&"function"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?"symbol":typeof e};e.exports=function(e){return"object"===(void 0===e?"undefined":r(e))?null!==e:"function"==typeof e}},function(e,t,n){"use strict";var r=n(18);e.exports=function(e,t,n){if(r(e),void 0===t)return e;switch(n){case 1:return function(n){return e.call(t,n)};case 2:return function(n,r){return e.call(t,n,r)};case 3:return function(n,r,i){return e.call(t,n,r,i)}}return function(){return e.apply(t,arguments)}}},function(e,t,n){"use strict";var r=n(9),i=n(0)("toStringTag"),o="Arguments"==r(function(){return arguments}()),a=function(e,t){try{return e[t]}catch(e){}};e.exports=function(e){var t,n,s;return void 0===e?"Undefined":null===e?"Null":"string"==typeof(n=a(t=Object(e),i))?n:o?r(t):"Object"==(s=r(t))&&"function"==typeof t.callee?"Arguments":s}},function(e,t,n){"use strict";var r=0,i=Math.random();e.exports=function(e){return"Symbol(".concat(void 0===e?"":e,")_",(++r+i).toString(36))}},function(e,t,n){"use strict";var r=n(11),i=n(1).document,o=r(i)&&r(i.createElement);e.exports=function(e){return o?i.createElement(e):{}}},function(e,t,n){"use strict";var r=Math.ceil,i=Math.floor;e.exports=function(e){return isNaN(e=+e)?0:(e>0?i:r)(e)}},function(e,t,n){"use strict";e.exports=function(e){if(void 0==e)throw TypeError("Can't call method on "+e);return e}},function(e,t,n){"use strict";e.exports=function(e){if("function"!=typeof e)throw TypeError(e+" is not a function!");return e}},function(e,t,n){"use strict";var r=n(47),i=n(17);e.exports=function(e){return r(i(e))}},function(e,t,n){"use strict";var r=n(23)("keys"),i=n(14);e.exports=function(e){return r[e]||(r[e]=i(e))}},function(e,t,n){"use strict";var r=n(10).f,i=n(6),o=n(0)("toStringTag");e.exports=function(e,t,n){e&&!i(e=n?e:e.prototype,o)&&r(e,o,{configurable:!0,value:t})}},function(e,t,n){"use strict";Object.defineProperty(t,"__esModule",{value:!0}),t.default={createElement:function(e,t){var n=document.createElement(e);t&&Array.prototype.forEach.call(Object.keys(t),function(e){n.setAttribute(e,t[e])});for(var r=arguments.length,i=Array(r>2?r-2:0),o=2;o0?i(r(e),9007199254740991):0}},function(e,t,n){"use strict";e.exports="constructor,hasOwnProperty,isPrototypeOf,propertyIsEnumerable,toLocaleString,toString,valueOf".split(",")},function(e,t,n){"use strict";e.exports=n(1).document&&document.documentElement},function(e,t,n){"use strict";var r,i,o,a=n(12),s=n(63),c=n(31),u=n(15),l=n(1),f=l.process,h=l.setImmediate,d=l.clearImmediate,p=l.MessageChannel,m=0,y={},v=function(){var e=+this;if(y.hasOwnProperty(e)){var t=y[e];delete y[e],t()}},g=function(e){v.call(e.data)};h&&d||(h=function(e){for(var t=[],n=1;arguments.length>n;)t.push(arguments[n++]);return y[++m]=function(){s("function"==typeof e?e:Function(e),t)},r(m),m},d=function(e){delete y[e]},"process"==n(9)(f)?r=function(e){f.nextTick(a(v,e,1))}:p?(i=new p,o=i.port2,i.port1.onmessage=g,r=a(o.postMessage,o,1)):l.addEventListener&&"function"==typeof postMessage&&!l.importScripts?(r=function(e){l.postMessage(e+"","*")},l.addEventListener("message",g,!1)):r="onreadystatechange"in u("script")?function(e){c.appendChild(u("script")).onreadystatechange=function(){c.removeChild(this),v.call(e)}}:function(e){setTimeout(a(v,e,1),0)}),e.exports={set:h,clear:d}},function(e,t){(function(t){e.exports=t}).call(t,{})},function(e,t,n){"use strict";function r(e,t){if(!(e instanceof t))throw new TypeError("Cannot call a class as a function")}var i=function(){function e(e,t){for(var n=0;n=t.length?{value:void 0,done:!0}:(e=r(t,n),this._i+=e.length,{value:e,done:!1})})},function(e,t,n){"use strict";var r=n(16),i=n(17);e.exports=function(e){return function(t,n){var o,a,s=String(i(t)),c=r(n),u=s.length;return c<0||c>=u?e?"":void 0:(o=s.charCodeAt(c),o<55296||o>56319||c+1===u||(a=s.charCodeAt(c+1))<56320||a>57343?e?s.charAt(c):o:e?s.slice(c,c+2):a-56320+(o-55296<<10)+65536)}}},function(e,t,n){"use strict";var r=n(43),i=n(25),o=n(21),a={};n(2)(a,n(0)("iterator"),function(){return this}),e.exports=function(e,t,n){e.prototype=r(a,{next:i(1,n)}),o(e,t+" Iterator")}},function(e,t,n){"use strict";var r=n(3),i=n(44),o=n(30),a=n(20)("IE_PROTO"),s=function(){},c=function(){var e,t=n(15)("iframe"),r=o.length;for(t.style.display="none",n(31).appendChild(t),t.src="javascript:",e=t.contentWindow.document,e.open(),e.write(" + diff --git a/site/ha-docker-swarm/design/index.html b/site/ha-docker-swarm/design/index.html new file mode 100644 index 0000000..dddd1c0 --- /dev/null +++ b/site/ha-docker-swarm/design/index.html @@ -0,0 +1,767 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Design - Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

Design

+

In the design described below, the "private cloud" platform is:

+
    +
  • Highly-available (can tolerate the failure of a single component)
    • +
  • Scalable (can add resource or capacity as required)
    • +
  • Portable (run it on your garage server today, run it in AWS tomorrow)
    • +
  • Secure (access protected with LetsEncrypt certificates)
    • +
  • Automated (requires minimal care and feeding)
    • +
+

Design Decisions

+

Where possible, services will be highly available.

+

This means that:

+
    +
  • At least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure.
    • +
  • GlusterFS is employed for share filesystem, because it too can be made tolerant of a single failure.
    • +
+

Where multiple solutions to a requirement exist, preference will be given to the most portable solution.

+

This means that:

+
    +
  • Services are defined using docker-compose v3 YAML syntax
    • +
  • Services are portable, meaning a particular stack could be shut down and moved to a new provider with minimal effort.
    • +
+

Security

+

Under this design, the only inbound connections we're permitting to our docker swarm are:

+

Network Flows

+
    +
  • HTTP (TCP 80) : Redirects to https
    • +
  • HTTPS (TCP 443) : Serves individual docker containers via SSL-encrypted reverse proxy
    • +
+

Authentication

+
    +
  • Where the proxied application provides a trusted level of authentication, or where the application requires public exposure,
    • +
+

High availability

+

Normal function

+

Assuming 3 nodes, under normal circumstances the following is illustrated:

+
    +
  • All 3 nodes provide shared storage via GlusterFS, which is provided by a docker container on each node. (i.e., not running in swarm mode)
    • +
  • All 3 nodes participate in the Docker Swarm as managers.
    • +
  • The various containers belonging to the application "stacks" deployed within Docker Swarm are automatically distributed amongst the swarm nodes.
    • +
  • Persistent storage for the containers is provide via GlusterFS mount.
    • +
  • The traefik service (in swarm mode) receives incoming requests (on http and https), and forwards them to individual containers. Traefik knows the containers names because it's able to access the docker socket.
    • +
  • All 3 nodes run keepalived, at different priorities. Since traefik is running as a swarm service and listening on TCP 80/443, requests made to the keepalived VIP and arriving at any of the swarm nodes will be forwarded to the traefik container (no matter which node it's on), and then onto the target backend.
    • +
+

HA function

+

Node failure

+

In the case of a failure (or scheduled maintenance) of one of the nodes, the following is illustrated:

+
    +
  • The failed node no longer participates in GlusterFS, but the remaining nodes provide enough fault-tolerance for the cluster to operate.
    • +
  • The remaining two nodes in Docker Swarm achieve a quorum and agree that the failed node is to be removed.
    • +
  • The (possibly new) leader manager node reschedules the containers known to be running on the failed node, onto other nodes.
    • +
  • The traefik service is either restarted or unaffected, and as the backend containers stop/start and change IP, traefik is aware and updates accordingly.
    • +
  • The keepalived VIP continues to function on the remaining nodes, and docker swarm continues to forward any traffic received on TCP 80/443 to the appropriate node.
    • +
+

HA function

+

Node restore

+

When the failed (or upgraded) host is restored to service, the following is illustrated:

+
    +
  • GlusterFS regains full redundancy
    • +
  • Docker Swarm managers become aware of the recovered node, and will use it for scheduling new containers
    • +
  • Existing containers which were migrated off the node are not migrated backend
    • +
  • Keepalived VIP regains full redundancy
    • +
+

HA function

+

Total cluster failure

+

A day after writing this, my environment suffered a fault whereby all 3 VMs were unexpectedly and simultaneously powered off.

+

Upon restore, docker failed to start on one of the VMs due to local disk space issue1. However, the other two VMs started, established the swarm, mounted their shared storage, and started up all the containers (services) which were managed by the swarm.

+

In summary, although I suffered an unplanned power outage to all of my infrastructure, followed by a failure of a third of my hosts... all my platforms are 100% available with absolutely no manual intervention.

+
+
+
    +
  1. +

    Since there's no impact to availability, I can fix (or just reinstall) the failed node whenever convenient. ↩

    +
    2. +
+
+ + + + + + +

Comments

+
+ + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/ha-docker-swarm/docker-swarm-mode/index.html b/site/ha-docker-swarm/docker-swarm-mode/index.html new file mode 100644 index 0000000..e0be272 --- /dev/null +++ b/site/ha-docker-swarm/docker-swarm-mode/index.html @@ -0,0 +1,976 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Docker Swarm Mode - Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

Docker Swarm Mode

+

For truly highly-available services with Docker containers, we need an orchestration system. Docker Swarm (as defined at 1.13) is the simplest way to achieve redundancy, such that a single docker host could be turned off, and none of our services will be interrupted.

+

Ingredients

+
    +
  • 3 x CentOS Atomic hosts (bare-metal or VMs). A reasonable minimum would be:
    • +
  • 1 x vCPU
    • +
  • 1GB repo_name
    • +
  • 10GB HDD
    • +
  • Hosts must be within the same subnet, and connected on a low-latency link (i.e., no WAN links)
    • +
+

Preparation

+

Release the swarm!

+

Now, to launch my swarm:

+

docker swarm init

+

Yeah, that was it. Now I have a 1-node swarm.

+
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
[root@ds1 ~]# docker swarm init
+Swarm initialized: current node (b54vls3wf8xztwfz79nlkivt8) is now a manager.
+
+To add a worker to this swarm, run the following command:
+
+    docker swarm join \
+    --token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-bsud7xnvhv4cicwi7l6c9s6l0 \
+    202.170.164.47:2377
+
+To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
+
+[root@ds1 ~]#
+
+
+ +

Run docker node ls to confirm that I have a 1-node swarm:

+
1
+2
+3
+4
[root@ds1 ~]# docker node ls
+ID                           HOSTNAME                STATUS  AVAILABILITY  MANAGER STATUS
+b54vls3wf8xztwfz79nlkivt8 *  ds1.funkypenguin.co.nz  Ready   Active        Leader
+[root@ds1 ~]#
+
+
+ +

Note that when I ran docker swarm init above, the CLI output gave me a command to run to join further nodes to my swarm. This would join the nodes as workers (as opposed to managers). Workers can easily be promoted to managers (and demoted again), but since we know that we want our other two nodes to be managers too, it's simpler just to add them to the swarm as managers immediately.

+

On the first swarm node, generate the necessary token to join another manager by running docker swarm join-token manager:

+
1
+2
+3
+4
+5
+6
+7
+8
[root@ds1 ~]# docker swarm join-token manager
+To add a manager to this swarm, run the following command:
+
+    docker swarm join \
+    --token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-cfm24bq2zvfkcwujwlp5zqxta \
+    202.170.164.47:2377
+
+[root@ds1 ~]#
+
+
+ +

Run the command provided on your second node to join it to the swarm as a manager. After adding the second node, the output of docker node ls (on either host) should reflect two nodes:

+
1
+2
+3
+4
+5
[root@ds2 davidy]# docker node ls
+ID                           HOSTNAME                STATUS  AVAILABILITY  MANAGER STATUS
+b54vls3wf8xztwfz79nlkivt8    ds1.funkypenguin.co.nz  Ready   Active        Leader
+xmw49jt5a1j87a6ihul76gbgy *  ds2.funkypenguin.co.nz  Ready   Active        Reachable
+[root@ds2 davidy]#
+
+
+ +

Repeat the process to add your third node. You need a new token for the third node, don't re-use the manager token you generated for the second node.

+
+

Seriously. Don't use a token more than once, else it's swarm-rebuilding time.

+
+

Finally, docker node ls should reflect that you have 3 reachable manager nodes, one of whom is the "Leader":

+
1
+2
+3
+4
+5
+6
[root@ds3 ~]# docker node ls
+ID                           HOSTNAME                      STATUS  AVAILABILITY  MANAGER STATUS
+36b4twca7i3hkb7qr77i0pr9i    ds1.openstack.dev.safenz.net  Ready   Active        Reachable
+l14rfzazbmibh1p9wcoivkv1s *  ds3.openstack.dev.safenz.net  Ready   Active        Reachable
+tfsgxmu7q23nuo51wwa4ycpsj    ds2.openstack.dev.safenz.net  Ready   Active        Leader
+[root@ds3 ~]#
+
+
+ +

Create registry mirror

+

Although we now have shared storage for our persistent container data, our docker nodes don't share any other docker data, such as container images. This results in an inefficiency - every node which participates in the swarm will, at some point, need the docker image for every container deployed in the swarm.

+

When dealing with large container (looking at you, GitLab!), this can result in several gigabytes of wasted bandwidth per-node, and long delays when restarting containers on an alternate node. (It also wastes disk space on each node, but we'll get to that in the next section)

+

The solution is to run an official Docker registry container as a "pull-through" cache, or "registry mirror". By using our persistent storage for the registry cache, we can ensure we have a single copy of all the containers we've pulled at least once. After the first pull, any subsequent pulls from our nodes will use the cached version from our registry mirror. As a result, services are available more quickly when restarting container nodes, and we can be more aggressive about cleaning up unused containers on our nodes (more later)

+

The registry mirror runs as a swarm stack, using a simple docker-compose.yml. Customize your mirror FQDN below, so that Traefik will generate the appropriate LetsEncrypt certificates for it, and make it available via HTTPS.

+
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
version: "3"
+
+services:
+
+  registry-mirror:
+    image: registry:2
+    networks:
+      - traefik
+    deploy:
+      labels:
+        - traefik.frontend.rule=Host:<your mirror FQDN>
+        - traefik.docker.network=traefik
+        - traefik.port=5000
+    ports:
+      - 5000:5000
+    volumes:
+      - /var/data/registry/registry-mirror-data:/var/lib/registry
+      - /var/data/registry/registry-mirror-config.yml:/etc/docker/registry/config.yml
+
+networks:
+  traefik:
+    external: true
+
+
+ +
+

Unencrypted registry

+

We create this registry without consideration for SSL, which will fail if we attempt to use the registry directly. However, we're going to use the HTTPS-proxied version via Traefik, leveraging Traefik to manage the LetsEncrypt certificates required.

+
+

Create registry/registry-mirror-config.yml as follows: +
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
version: 0.1
+log:
+  fields:
+    service: registry
+storage:
+  cache:
+    blobdescriptor: inmemory
+  filesystem:
+    rootdirectory: /var/lib/registry
+  delete:
+    enabled: true
+http:
+  addr: :5000
+  headers:
+    X-Content-Type-Options: [nosniff]
+health:
+  storagedriver:
+    enabled: true
+    interval: 10s
+    threshold: 3
+proxy:
+  remoteurl: https://registry-1.docker.io
+
+

+

Enable registry mirror and experimental features

+

To tell docker to use the registry mirror, and in order to be able to watch the logs of any service from any manager node (an experimental feature in the current Atomic docker build), edit /etc/docker-latest/daemon.json on each node, and change from:

+
1
+2
+3
+4
{
+    "log-driver": "journald",
+    "signature-verification": false
+}
+
+
+ +

To:

+
1
+2
+3
+4
+5
+6
{
+    "log-driver": "journald",
+    "signature-verification": false,
+    "experimental": true,
+    "registry-mirrors": ["https://<your registry mirror FQDN>"]
+}
+
+
+ +
+

Note the extra comma required after "false" above

+
+

Setup automated cleanup

+

This needs to be a docker-compose.yml file, excluding trusted images (like glusterfs, traefik, etc) +
1
+2
+3
+4
docker run -d  \
+-v /var/run/docker.sock:/var/run/docker.sock:rw \
+-v /var/lib/docker:/var/lib/docker:rw  \
+meltwater/docker-cleanup:latest
+
+

+

Tweaks

+

Add some handy bash auto-completion for docker. Without this, you'll get annoyed that you can't autocomplete docker stack deploy <blah> -c <blah.yml> commands.

+
1
+2
cd /etc/bash_completion.d/
+curl -O https://raw.githubusercontent.com/docker/cli/b75596e1e4d5295ac69b9934d1bd8aff691a0de8/contrib/completion/bash/docker
+
+
+ +

Install some useful bash aliases on each host +
1
+2
+3
cd ~
+curl -O https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook-recipies/raw/master/bash/gcb-aliases.sh
+echo 'source ~/gcb-aliases.sh' >> ~/.bash_profile
+
+

+
1
+2
+3
+4
+5
mkdir ~/dockersock
+cd ~/dockersock
+curl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/Makefile
+curl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/dockersock.te
+make && semodule -i dockersock.pp
+
+
+ +

Setup registry

+

docker run -d \ + -p 5000:5000 \ + --restart=always \ + --name registry \ + -v /mnt/registry:/var/lib/registry \ + registry:2

+

{ +"log-driver": "journald", +"signature-verification": false, +"experimental": true, +"registry-mirrors": ["https://registry-mirror.funkypenguin.co.nz"] +}

+

registry-mirror: + image: registry:2 + ports: + - 5000:5000 + environment: + volumes: + - /var/data/registry:/var/lib/registry

+
1
+2
+3
+4
+5
+6
+7
+8
  [root@ds1 dockersock]# docker swarm join-token manager
+  To add a manager to this swarm, run the following command:
+
+      docker swarm join \
+      --token SWMTKN-1-09c94wv0opw0y6xg67uzjl13pnv8lxxn586hrg5f47spso9l6j-6zn3dxk7c4zkb19r61owasi15 \
+      192.168.31.11:2377
+
+  [root@ds1 dockersock]#
+
+
+ + + + + + +

Comments

+
+ + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/ha-docker-swarm/images/docker-swarm-ha-function.png b/site/ha-docker-swarm/images/docker-swarm-ha-function.png new file mode 100644 index 0000000..754c662 Binary files /dev/null and b/site/ha-docker-swarm/images/docker-swarm-ha-function.png differ diff --git a/site/ha-docker-swarm/images/docker-swarm-node-failure.png b/site/ha-docker-swarm/images/docker-swarm-node-failure.png new file mode 100644 index 0000000..8114937 Binary files /dev/null and b/site/ha-docker-swarm/images/docker-swarm-node-failure.png differ diff --git a/site/ha-docker-swarm/images/docker-swarm-node-restore.png b/site/ha-docker-swarm/images/docker-swarm-node-restore.png new file mode 100644 index 0000000..8ee2a0b Binary files /dev/null and b/site/ha-docker-swarm/images/docker-swarm-node-restore.png differ diff --git a/site/ha-docker-swarm/images/shared-storage-replicated-gluster-volume.png b/site/ha-docker-swarm/images/shared-storage-replicated-gluster-volume.png new file mode 100644 index 0000000..135a63f Binary files /dev/null and b/site/ha-docker-swarm/images/shared-storage-replicated-gluster-volume.png differ diff --git a/site/ha-docker-swarm/keepalived/index.html b/site/ha-docker-swarm/keepalived/index.html new file mode 100644 index 0000000..c9f9427 --- /dev/null +++ b/site/ha-docker-swarm/keepalived/index.html @@ -0,0 +1,709 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Keepalived - Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

Keepalived

+

While having a self-healing, scalable docker swarm is great for availability and scalability, none of that is any good if nobody can connect to your cluster.

+

In order to provide seamless external access to clustered resources, regardless of which node they're on and tolerant of node failure, you need to present a single IP to the world for external access.

+

Normally this is done using a HA loadbalancer, but since Docker Swarm aready provides the load-balancing capabilities (routing mesh), all we need for seamless HA is a virtual IP which will be provided by more than one docker node.

+

This is accomplished with the use of keepalived on at least two nodes.

+

Ingredients

+
1
+2
+3
+4
+5
+6
Already deployed:
+[X] At least 2 x CentOS/Fedora Atomic VMs
+[X] low-latency link (i.e., no WAN links)
+
+New:
+[ ] 3 x IPv4 addresses (one for each node and one for the virtual IP)
+
+
+ +

Preparation

+

Enable IPVS module

+

On all nodes which will participate in keepalived, we need the "ip_vs" kernel module, in order to permit serivces to bind to non-local interface addresses.

+

Set this up once for both the primary and secondary nodes, by running:

+
1
+2
echo "modprobe ip_vs" >> /etc/rc.local
+modprobe ip_vs
+
+
+ +

Setup nodes

+

Assuming your IPs are as follows:

+
    +
  • 192.168.4.1 : Primary
    • +
  • 192.168.4.2 : Secondary
    • +
  • 192.168.4.3 : Virtual
    • +
+

Run the following on the primary +
1
+2
+3
+4
+5
+6
docker run -d --name keepalived --restart=always \
+  --cap-add=NET_ADMIN --net=host \
+  -e KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:['192.168.4.1', '192.168.4.2']" \
+  -e KEEPALIVED_VIRTUAL_IPS=192.168.4.3 \
+  -e KEEPALIVED_PRIORITY=200 \
+  osixia/keepalived:1.3.5
+
+

+

And on the secondary: +
1
+2
+3
+4
+5
+6
docker run -d --name keepalived --restart=always \
+  --cap-add=NET_ADMIN --net=host \
+  -e KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:['192.168.4.1', '192.168.4.2']" \
+  -e KEEPALIVED_VIRTUAL_IPS=192.168.4.3 \
+  -e KEEPALIVED_PRIORITY=100 \
+  osixia/keepalived:1.3.5
+
+

+

Serving

+

That's it. Each node will talk to the other via unicast (no need to un-firewall multicast addresses), and the node with the highest priority gets to be the master. When ingress traffic arrives on the master node via the VIP, docker's routing mesh will deliver it to the appropriate docker node.

+

Chef's notes

+
    +
  1. Some hosting platforms (OpenStack, for one) won't allow you to simply "claim" a virtual IP. Each node is only able to receive traffic targetted to its unique IP. In this case, keepalived is not the right solution, and a platform-specific load-balancing solution should be used. In OpenStack, this is Neutron's "Load Balancer As A Service" (LBAAS) component. AWS and Azure would likely include similar protections.
    2. +
  2. More than 2 nodes can participate in keepalived. Simply ensure that each node has the appropriate priority set, and the node with the highest priority will become the master.
    3. +
+ + + + + + +

Comments

+
+ + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/ha-docker-swarm/shared-storage-ceph/index.html b/site/ha-docker-swarm/shared-storage-ceph/index.html new file mode 100644 index 0000000..3a4724b --- /dev/null +++ b/site/ha-docker-swarm/shared-storage-ceph/index.html @@ -0,0 +1,976 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Shared Storage (Ceph) - Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

Shared Storage (Ceph)

+

While Docker Swarm is great for keeping containers running (and restarting those that fail), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (hint: you do!), you need to provide shared storage to every docker node.

+

Design

+

Why not GlusterFS?

+

I originally provided shared storage to my nodes using GlusterFS (see the next recipe for details), but found it difficult to deal with because:

+
    +
  1. GlusterFS requires (n) "bricks", where (n) has to be a multiple of your replica count. I.e., if you want 2 copies of everything on shared storage (the minimum to provide redundancy), you must have either 2, 4, 6 (etc..) bricks. The HA swarm design calls for minimum of 3 nodes, and so under GlusterFS, my third node can't participate in shared storage at all, unless I start doubling up on bricks-per-node (which then impacts redundancy)
    2. +
  2. GlusterFS turns out to be a giant PITA when you want to restore a failed node. There are at least 14 steps to follow to replace a brick.
    3. +
  3. I'm pretty sure I messed up the 14-step process above anyway. My replaced brick synced with my "original" brick, but produced errors when querying status via the CLI, and hogged 100% of 1 CPU on the replaced node. Inexperienced with GlusterFS, and unable to diagnose the fault, I switched to a Ceph cluster instead.
    4. +
+

Why Ceph?

+
    +
  1. I'm more familiar with Ceph - I use it in the OpenStack designs I manage
    2. +
  2. Replacing a failed node is easy, provided you can put up with the I/O load of rebalancing OSDs after the replacement.
    3. +
  3. CentOS Atomic includes the ceph client in the OS, so while the Ceph OSD/Mon/MSD are running under containers, I can keep an eye (and later, automatically monitor) the status of Ceph from the base OS.
    4. +
+

Ingredients

+
+

Ingredients

+

3 x Virtual Machines (configured earlier), each with:

+
    +
  • CentOS/Fedora Atomic
    • +
  • At least 1GB RAM
    • +
  • At least 20GB disk space (but it'll be tight)
    • +
  • Connectivity to each other within the same subnet, and on a low-latency link (i.e., no WAN links)
    • +
  • A second disk dedicated to the Ceph OSD
    • +
+
+

Preparation

+

SELinux

+

Since our Ceph components will be containerized, we need to ensure the SELinux context on the base OS's ceph files is set correctly:

+
1
+2
chcon -Rt svirt_sandbox_file_t /etc/ceph
+chcon -Rt svirt_sandbox_file_t /var/lib/ceph
+
+
+ +

Setup Monitors

+

Pick a node, and run the following to stand up the first Ceph mon. Be sure to replace the values for MON_IP and CEPH_PUBLIC_NETWORK to those specific to your deployment:

+
1
+2
+3
+4
+5
+6
+7
+8
docker run -d --net=host \
+--restart always \
+-v /etc/ceph:/etc/ceph \
+-v /var/lib/ceph/:/var/lib/ceph/ \
+-e MON_IP=192.168.31.11 \
+-e CEPH_PUBLIC_NETWORK=192.168.31.0/24 \
+--name="ceph-mon" \
+ceph/daemon mon
+
+
+ +

Now copy the contents of /etc/ceph on this first node to the remaining nodes, and then run the docker command above (customizing MON_IP as you go) on each remaining node. You'll end up with a cluster with 3 monitors (odd number is required for quorum, same as Docker Swarm), and no OSDs (yet)

+

Setup OSDs

+

Since we have a OSD-less mon-only cluster currently, prepare for OSD creation by dumping the auth credentials for the OSDs into the appropriate location on the base OS:

+
1
+2
ceph auth get client.bootstrap-osd -o \
+/var/lib/ceph/bootstrap-osd/ceph.keyring
+
+
+ +

On each node, you need a dedicated disk for the OSD. In the example below, I used /dev/vdd (the entire disk, no partitions) for the OSD.

+

Run the following command on every node:

+
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
docker run -d --net=host \
+--privileged=true \
+--pid=host \
+-v /etc/ceph:/etc/ceph \
+-v /var/lib/ceph/:/var/lib/ceph/ \
+-v /dev/:/dev/ \
+-e OSD_DEVICE=/dev/vdd \
+-e OSD_TYPE=disk \
+--name="ceph-osd" \
+--restart=always \
+ceph/daemon osd
+
+
+ +

Watch the output by running docker logs ceph-osd -f, and confirm success.

+
+

Zapping the device

+

The Ceph OSD container will refuse to destroy a partition containing existing data, so it may be necessary to "zap" the target disk, using: +
1
+2
+3
+4
docker run -d --privileged=true \
+-v /dev/:/dev/ \
+-e OSD_DEVICE=/dev/sdd \
+ceph/daemon zap_device
+
+

+
+

Setup MDSs

+

In order to mount our ceph pools as filesystems, we'll need Ceph MDS(s). Run the following on each node:

+
1
+2
+3
+4
+5
+6
+7
+8
+9
docker run -d --net=host \
+--name ceph-mds \
+--restart always \
+-v /var/lib/ceph/:/var/lib/ceph/ \
+-v /etc/ceph:/etc/ceph \
+-e CEPHFS_CREATE=1 \
+-e CEPHFS_DATA_POOL_PG=256 \
+-e CEPHFS_METADATA_POOL_PG=256 \
+ceph/daemon mds
+
+
+ +

Apply tweaks

+

The ceph container seems to configure a pool default of 3 replicas (3 copies of each block are retained), which is one too many for our cluster (we are only protecting against the failure of a single node).

+

Run the following on any node to reduce the size of the pool to 2 replicas:

+
1
+2
ceph osd pool set cephfs_data size 2
+ceph osd pool set cephfs_metadata size 2
+
+
+ +

Disabled "scrubbing" (which can be IO-intensive, and is unnecessary on a VM) with:

+
1
+2
ceph osd set noscrub
+ceph osd set nodeep-scrub
+
+
+ +

Create credentials for swarm

+

In order to mount the ceph volume onto our base host, we need to provide cephx authentication credentials.

+

On one node, create a client for the docker swarm:

+
1
+2
ceph auth get-or-create client.dockerswarm osd \
+'allow rw' mon 'allow r' mds 'allow' > /etc/ceph/keyring.dockerswarm
+
+
+ +

Grab the secret associated with the new user (you'll need this for the /etc/fstab entry below) by running:

+
1
ceph-authtool /etc/ceph/keyring.dockerswarm -p -n client.dockerswarm
+
+
+ +

Mount MDS volume

+

On each noie, create a mountpoint for the data, by running mkdir /var/data, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually mounted if there's a network / boot delay getting access to the gluster volume:

+
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
mkdir /var/data
+
+MYHOST=`hostname -s`
+echo -e "
+# Mount cephfs volume \n
+$MYHOST:6789:/      /var/data/      ceph      \
+name=dockerswarm\
+,secret=<YOUR SECRET HERE>\
+,noatime,_netdev,context=system_u:object_r:svirt_sandbox_file_t:s0\
+0 2" >> /etc/fstab
+mount -a
+
+
+ +

Install docker-volume plugin

+

Upstream bug for docker-latest reported at https://bugs.centos.org/view.php?id=13609

+

And the alpine fault: +https://github.com/gliderlabs/docker-alpine/issues/317

+

Serving

+

After completing the above, you should have:

+
1
+2
[X] Persistent storage available to every node
+[X] Resiliency in the event of the failure of a single node
+
+
+ +

Chef's Notes

+

Future enhancements to this recipe include:

+
    +
  1. Rather than pasting a secret key into /etc/fstab (which feels wrong), I'd prefer to be able to set "secretfile" in /etc/fstab (which just points ceph.mount to a file containing the secret), but under the current CentOS Atomic, we're stuck with "secret", per https://bugzilla.redhat.com/show_bug.cgi?id=1030402
    2. +
+ + + + + + +

Comments

+
+ + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/ha-docker-swarm/shared-storage-gluster/index.html b/site/ha-docker-swarm/shared-storage-gluster/index.html new file mode 100644 index 0000000..80e619f --- /dev/null +++ b/site/ha-docker-swarm/shared-storage-gluster/index.html @@ -0,0 +1,904 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Shared Storage (GlusterFS) - Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

Shared Storage (GlusterFS)

+

While Docker Swarm is great for keeping containers running (and restarting those that fail), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (hint: you do!), you need to provide shared storage to every docker node.

+

Design

+

Why GlusterFS?

+

This GlusterFS recipe was my original design for shared storage, but I found it to be flawed, and I replaced it with a design which employs Ceph instead. This recipe is an alternate to the Ceph design, if you happen to prefer GlusterFS.

+

Ingredients

+
+

Ingredients

+

3 x Virtual Machines (configured earlier), each with:

+
    +
  • CentOS/Fedora Atomic
    • +
  • At least 1GB RAM
    • +
  • At least 20GB disk space (but it'll be tight)
    • +
  • Connectivity to each other within the same subnet, and on a low-latency link (i.e., no WAN links)
    • +
  • A second disk, or adequate space on the primary disk for a dedicated data partition
    • +
+
+

Preparation

+

Create Gluster "bricks"

+

To build our Gluster volume, we need 2 out of the 3 VMs to provide one "brick". The bricks will be used to create the replicated volume. Assuming a replica count of 2 (i.e., 2 copies of the data are kept in gluster), our total number of bricks must be divisible by our replica count. (I.e., you can't have 3 bricks if you want 2 replicas. You can have 4 though - We have to have minimum 3 swarm manager nodes for fault-tolerance, but only 2 of those nodes need to run as gluster servers.)

+

On each host, run a variation following to create your bricks, adjusted for the path to your disk.

+
+

The example below assumes /dev/vdb is dedicated to the gluster volume

+
+
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
(
+echo o # Create a new empty DOS partition table
+echo n # Add a new partition
+echo p # Primary partition
+echo 1 # Partition number
+echo   # First sector (Accept default: 1)
+echo   # Last sector (Accept default: varies)
+echo w # Write changes
+) | sudo fdisk /dev/vdb
+
+mkfs.xfs -i size=512 /dev/vdb1
+mkdir -p /var/no-direct-write-here/brick1
+echo '' >> /etc/fstab >> /etc/fstab
+echo '# Mount /dev/vdb1 so that it can be used as a glusterfs volume' >> /etc/fstab
+echo '/dev/vdb1 /var/no-direct-write-here/brick1 xfs defaults 1 2' >> /etc/fstab
+mount -a && mount
+
+
+ +
+

Don't provision all your LVM space

+

Atomic uses LVM to store docker data, and automatically grows Docker's volumes as requried. If you commit all your free LVM space to your brick, you'll quickly find (as I did) that docker will start to fail with error messages about insufficient space. If you're going to slice off a portion of your LVM space in /dev/atomicos, make sure you leave enough space for Docker storage, where "enough" depends on how much you plan to pull images, make volumes, etc. I ate through 20GB very quickly doing development, so I ended up provisioning 50GB for atomic alone, with a separate volume for the brick.

+
+

Create glusterfs container

+

Atomic doesn't include the Gluster server components. This means we'll have to run glusterd from within a container, with privileged access to the host. Although convoluted, I've come to prefer this design since it once again makes the OS "disposable", moving all the config into containers and code.

+

Run the following on each host: +
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
docker run \
+   -h glusterfs-server \
+   -v /etc/glusterfs:/etc/glusterfs:z \
+   -v /var/lib/glusterd:/var/lib/glusterd:z \
+   -v /var/log/glusterfs:/var/log/glusterfs:z \
+   -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
+   -v /var/no-direct-write-here/brick1:/var/no-direct-write-here/brick1 \
+   -d --privileged=true --net=host \
+   --restart=always \
+   --name="glusterfs-server" \
+   gluster/gluster-centos
+
+

+

Create trusted pool

+

On a single node (doesn't matter which), run docker exec -it glusterfs-server bash to launch a shell inside the container.

+

From the node, run +gluster peer probe <other host>

+

Example output: +
1
+2
+3
[root@glusterfs-server /]# gluster peer probe ds1
+peer probe: success.
+[root@glusterfs-server /]#
+
+

+

Run gluster peer status on both nodes to confirm that they're properly connected to each other:

+

Example output: +
1
+2
+3
+4
+5
+6
+7
[root@glusterfs-server /]# gluster peer status
+Number of Peers: 1
+
+Hostname: ds3
+Uuid: 3e115ba9-6a4f-48dd-87d7-e843170ff499
+State: Peer in Cluster (Connected)
+[root@glusterfs-server /]#
+
+

+

Create gluster volume

+

Now we create a replicated volume out of our individual "bricks".

+

Create the gluster volume by running +
1
+2
+3
gluster volume create gv0 replica 2 \
+ server1:/var/no-direct-write-here/brick1 \
+ server2:/var/no-direct-write-here/brick1
+
+

+

Example output: +
1
+2
+3
[root@glusterfs-server /]# gluster volume create gv0 replica 2 ds1:/var/no-direct-write-here/brick1/gv0  ds3:/var/no-direct-write-here/brick1/gv0
+volume create: gv0: success: please start the volume to access data
+[root@glusterfs-server /]#
+
+

+

Start the volume by running gluster volume start gv0

+
1
+2
+3
[root@glusterfs-server /]# gluster volume start gv0
+volume start: gv0: success
+[root@glusterfs-server /]#
+
+
+ +

The volume is only present on the host you're shelled into though. To add the other hosts to the volume, run gluster peer probe <servername>. Don't probe host from itself.

+

From one other host, run docker exec -it glusterfs-server bash to shell into the gluster-server container, and run gluster peer probe <original server name> to update the name of the host which started the volume.

+

Mount gluster volume

+

On the host (i.e., outside of the container - type exit if you're still shelled in), create a mountpoint for the data, by running mkdir /var/data, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually mounted if there's a network / boot delay getting access to the gluster volume:

+
1
+2
+3
+4
+5
+6
mkdir /var/data
+MYHOST=`hostname -s`
+echo '' >> /etc/fstab >> /etc/fstab
+echo '# Mount glusterfs volume' >> /etc/fstab
+echo "$MYHOST:/gv0                /var/data      glusterfs       defaults,_netdev,context="system_u:object_r:svirt_sandbox_file_t:s0"  0  0"  >> /etc/fstab
+mount -a
+
+
+ +

For some reason, my nodes won't auto-mount this volume on boot. I even tried the trickery below, but they stubbornly refuse to automount. +
1
+2
+3
echo -e "\n\n# Give GlusterFS 10s to start before \
+mounting\nsleep 10s && mount -a" >> /etc/rc.local
+systemctl enable rc-local.service
+
+

+

For non-gluster nodes, you'll need to replace $MYHOST above with the name of one of the gluster hosts (I haven't worked out how to make this fully HA yet)

+

Serving

+

After completing the above, you should have:

+
1
+2
[X] Persistent storage available to every node
+[X] Resiliency in the event of the failure of a single (gluster) node
+
+
+ +

Chef's Notes

+

Future enhancements to this recipe include:

+
    +
  1. Migration of shared storage from GlusterFS to Ceph ()#2)
    2. +
  2. Correct the fact that volumes don't automount on boot (#3)
    3. +
+ + + + + + +

Comments

+
+ + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/ha-docker-swarm/traefik/index.html b/site/ha-docker-swarm/traefik/index.html new file mode 100644 index 0000000..8dc2208 --- /dev/null +++ b/site/ha-docker-swarm/traefik/index.html @@ -0,0 +1,869 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Traefik - Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

Traefik

+

The platforms we plan to run on our cloud are generally web-based, and each listening on their own unique TCP port. When a container in a swarm exposes a port, then connecting to any swarm member on that port will result in your request being forwarded to the appropriate host running the container. (Docker calls this the swarm "routing mesh")

+

So we get a rudimentary load balancer built into swarm. We could stop there, just exposing a series of ports on our hosts, and making them HA using keepalived.

+

There are some gaps to this approach though:

+
    +
  • No consideration is given to HTTPS. Implementation would have to be done manually, per-container.
    • +
  • No mechanism is provided for authentication outside of that which the container providers. We may not want to expose every interface on every container to the world, especially if we are playing with tools or containers whose quality and origin are unknown.
    • +
+

To deal with these gaps, we need a front-end load-balancer, and in this design, that role is provided by Traefik.

+

Ingredients

+

Preparation

+

Prepare the host

+

The traefik container is aware of the other docker containers in the swarm, because it has access to the docker socket at /var/run/docker.sock. This allows traefik to dynamically configure itself based on the labels found on containers in the swarm, which is hugely useful. To make this functionality work on our SELinux-enabled Atomic hosts, we need to add custom SELinux policy.

+

Run the following to build and activate policy to permit containers to access docker.sock:

+
1
+2
+3
+4
+5
+6
+7
mkdir ~/dockersock
+cd ~/dockersock
+curl -O https://raw.githubusercontent.com/dpw/\
+selinux-dockersock/master/Makefile
+curl -O https://raw.githubusercontent.com/dpw/\
+selinux-dockersock/master/dockersock.te
+make && semodule -i dockersock.pp
+
+
+ +

Prepare traefik.toml

+

While it's possible to configure traefik via docker command arguments, I prefer to create a config file (traefik.toml). This allows me to change traefik's behaviour by simply changing the file, and keeps my docker config simple.

+

Create /var/data/traefik/traefik.toml as follows:

+
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
+23
+24
+25
+26
+27
+28
+29
+30
+31
+32
+33
+34
checkNewVersion = true
+defaultEntryPoints = ["http", "https"]
+
+# This section enable LetsEncrypt automatic certificate generation / renewal
+[acme]
+email = "<your LetsEncrypt email address>"
+storage = "acme.json" # or "traefik/acme/account" if using KV store
+entryPoint = "https"
+acmeLogging = true
+onDemand = true
+OnHostRule = true
+
+[[acme.domains]]
+  main = "<your primary domain>"
+
+# Redirect all HTTP to HTTPS (why wouldn't you?)
+[entryPoints]
+  [entryPoints.http]
+  address = ":80"
+    [entryPoints.http.redirect]
+      entryPoint = "https"
+  [entryPoints.https]
+  address = ":443"
+    [entryPoints.https.tls]
+
+[web]
+address = ":8080"
+watch = true
+
+[docker]
+endpoint = "tcp://127.0.0.1:2375"
+domain = "<your primary domain>"
+watch = true
+swarmmode = true
+
+
+ +

Prepare the docker service config

+

Create /var/data/traefik/docker-compose.yml as follows:

+
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
+23
+24
+25
+26
+27
+28
+29
+30
+31
+32
+33
+34
+35
+36
+37
+38
+39
+40
version: "3.2"
+
+services:
+  traefik:
+    image: traefik
+    command: --web --docker --docker.swarmmode --docker.watch --docker.domain=funkypenguin.co.nz --logLevel=DEBUG
+    ports:
+      - target: 80
+        published: 80
+        protocol: tcp
+        mode: host
+      - target: 443
+        published: 443
+        protocol: tcp
+        mode: host
+      - target: 8080
+        published: 8080
+        protocol: tcp
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /var/data/traefik/traefik.toml:/traefik.toml:ro
+      - /var/data/traefik/acme.json:/acme.json
+    labels:
+      - "traefik.enable=false"
+    networks:
+      - public
+    deploy:
+      mode: global
+      placement:
+        constraints: [node.role == manager]
+      restart_policy:
+        condition: on-failure
+
+networks:
+  public:
+    driver: overlay
+    ipam:
+      driver: default
+      config:
+      - subnet: 10.1.0.0/24
+
+
+ +

Docker won't start an image with a bind-mount to a non-existent file, so prepare acme.json by running touch /var/data/traefik/acme.json.

+

Launch

+

Deploy traefik with docker stack deploy traefik -c /var/data/traefik/docker-compose.yml

+

Confirm traefik is running with docker stack ps traefik

+

Serving

+

You now have:

+
    +
  1. Frontend proxy which will dynamically configure itself for new backend containers
    2. +
  2. Automatic SSL support for all proxied resources
    3. +
+

Chef's Notes

+

Additional features I'd like to see in this recipe are:

+
    +
  1. Include documentation of oauth2_proxy container for protecting individual backends
    2. +
  2. Traefik webUI is available via HTTPS, protected with oauth_proxy
    3. +
  3. Pending a feature in docker-swarm to avoid NAT on routing-mesh-delivered traffic, update the design
    4. +
+ + + + + + +

Comments

+
+ + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/ha-docker-swarm/vms/index.html b/site/ha-docker-swarm/vms/index.html new file mode 100644 index 0000000..61b0fbe --- /dev/null +++ b/site/ha-docker-swarm/vms/index.html @@ -0,0 +1,731 @@ + + + + + + + + + + + + + + + + + + + + + + + + + VMs - Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

Virtual Machines

+

Let's start building our cloud with virtual machines. You could use bare-metal machines as well, the configuration would be the same. Given that most readers (myself included) will be using virtual infrastructure, from now on I'll be referring strictly to VMs.

+

I chose the "Atomic" CentOS/Fedora image for the VM layer because:

+
    +
  1. I want less responsibility for maintaining the system, including ensuring regular software updates and reboots. Atomic's idempotent nature means the OS is largely real-only, and updates/rollbacks are "atomic" (haha) procedures, which can be easily rolled back if required.
    2. +
  2. For someone used to administrating servers individually, Atomic is a PITA. You have to employ tricky tricks to get it to install in a non-cloud environment. It's not designed for tweaking or customizing beyond what cloud-config is capable of. For my purposes, this is good, because it forces me to change my thinking - to consider every daemon as a container, and every config as code, to be checked in and version-controlled. Atomic forces this thinking on you.
    3. +
  3. I want the design to be as "portable" as possible. While I run it on VPSs now, I may want to migrate it to a "cloud" provider in the future, and I'll want the most portable, reproducible design.
    4. +
+

Ingredients

+
+

Ingredients

+

3 x Virtual Machines, each with:

+
    +
  • CentOS/Fedora Atomic
    • +
  • At least 1GB RAM
    • +
  • At least 20GB disk space (but it'll be tight)
    • +
  • Connectivity to each other within the same subnet, and on a low-latency link (i.e., no WAN links)
    • +
+
+

Preparation

+

Install Virtual machines

+
    +
  1. Install / launch virtual machines.
    2. +
  2. The default username on CentOS atomic is "centos", and you'll have needed to supply your SSH key during the build process.
    3. +
+
+

Tip

+

If you're not using a platform with cloud-init support (i.e., you're building a VM manually, not provisioning it through a cloud provider), you'll need to refer to trick #1 and #2 for a means to override the automated setup, apply a manual password to the CentOS account, and enable SSH password logins.

+
+

Prefer docker-latest

+

Run the following on each node to replace the default docker 1.12 with docker 1.13 (which we need for swarm mode): +
1
+2
+3
systemctl disable docker --now
+systemctl enable docker-latest --now
+sed -i '/DOCKERBINARY/s/^#//g' /etc/sysconfig/docker
+
+

+

Upgrade Atomic

+

Finally, apply any Atomic host updates, and reboot, by running: atomic host upgrade && systemctl reboot.

+

Permit connectivity between VMs

+

By default, Atomic only permits incoming SSH. We'll want to allow all traffic between our nodes, so add something like this to /etc/sysconfig/iptables:

+
1
+2
# Allow all inter-node communication
+-A INPUT -s 192.168.31.0/24 -j ACCEPT
+
+
+ +

And restart iptables with systemctl restart iptables

+

Enable host resolution

+

Depending on your hosting environment, you may have DNS automatically setup for your VMs. If not, it's useful to set up static entries in /etc/hosts for the nodes. For example, I setup the following:

+
1
+2
+3
192.168.31.11   ds1     ds1.funkypenguin.co.nz
+192.168.31.12   ds2     ds2.funkypenguin.co.nz
+192.168.31.13   ds3     ds3.funkypenguin.co.nz
+
+
+ +

Serving

+

After completing the above, you should have:

+
1
+2
[X] 3 x fresh atomic instances, at the latest releases,
+    running Docker v1.13 (docker-latest)
+
+
+ + + + + + +

Comments

+
+ + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/images/site-logo.png b/site/images/site-logo.png new file mode 100644 index 0000000..7edb223 Binary files /dev/null and b/site/images/site-logo.png differ diff --git a/site/index.html b/site/index.html new file mode 100644 index 0000000..c0eecda --- /dev/null +++ b/site/index.html @@ -0,0 +1,634 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

Index

+

The "Geek's Cookbook" is a collection of guides for establishing your own highly-available "private cloud" 1. This cloud enables you to run self-hosted services such as GitLab, Plex, NextCloud, etc.

+

Who is this for?

+

You already have a familiarity with concepts such as virtual machines, Docker containers, LetsEncrypt SSL certificates, databases, and command-line interfaces.

+

You've probably played with self-hosting some mainstream apps yourself, like Plex, OwnCloud, Wordpress or even SandStorm.

+

Why should I read this?

+

So if you're familiar enough with the tools, and you've done self-hosting before, why would you read this book?

+
    +
  1. You want to upskill. You want to do container orchestration, LetsEncrypt certificates, git collaboration.
    2. +
  2. You want to play. You want a safe sandbox to test new tools, keeping the ones you want and tossing the ones you don't.
    3. +
  3. You want reliability. Once you go from playing with a tool to actually using it, you want it to be available when you need it. Having to "quickly ssh into the host and restart the webserver" doesn't cut it when your wife wants to know why her phone won't sync!
    4. +
+

What do you want from me?

+

I want your money.

+

No, seriously (but yes, I do want your money - see below), If the above applies to you, then you're like me. I want everything I wrote above, so I ended up learning all this as I went along. I enjoy it, and I'm good at it. So I created this website, partly to make sure I documented my own setup properly.

+

How can I support you?

+

Buy my book 📖

+

I'm also writing it as a formal book, on Leanpub (https://leanpub.com/geeks-cookbook). Buy it for $0.99 (which is really just a token gesture of support) - you can get it for free (in PDF, mobi, or epub format), or pay me what you think it's worth!

+

Patreonize me 💰

+

Become a Patron! +- My Patreon page!

+

Hire me 🏢

+

Need some system design work done? I do freelance consulting - contact me for details.

+
+
+
    +
  1. +

    Sorry for the buzzword, I couldn't think of a better description! ↩

    +
    2. +
+
+ + + + + + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/mkdocs/js/lunr.min.js b/site/mkdocs/js/lunr.min.js new file mode 100644 index 0000000..b0198df --- /dev/null +++ b/site/mkdocs/js/lunr.min.js @@ -0,0 +1,7 @@ +/** + * lunr - http://lunrjs.com - A bit like Solr, but much smaller and not as bright - 0.7.0 + * Copyright (C) 2016 Oliver Nightingale + * MIT Licensed + * @license + */ +!function(){var t=function(e){var n=new t.Index;return n.pipeline.add(t.trimmer,t.stopWordFilter,t.stemmer),e&&e.call(n,n),n};t.version="0.7.0",t.utils={},t.utils.warn=function(t){return function(e){t.console&&console.warn&&console.warn(e)}}(this),t.utils.asString=function(t){return void 0===t||null===t?"":t.toString()},t.EventEmitter=function(){this.events={}},t.EventEmitter.prototype.addListener=function(){var t=Array.prototype.slice.call(arguments),e=t.pop(),n=t;if("function"!=typeof e)throw new TypeError("last argument must be a function");n.forEach(function(t){this.hasHandler(t)||(this.events[t]=[]),this.events[t].push(e)},this)},t.EventEmitter.prototype.removeListener=function(t,e){if(this.hasHandler(t)){var n=this.events[t].indexOf(e);this.events[t].splice(n,1),this.events[t].length||delete this.events[t]}},t.EventEmitter.prototype.emit=function(t){if(this.hasHandler(t)){var e=Array.prototype.slice.call(arguments,1);this.events[t].forEach(function(t){t.apply(void 0,e)})}},t.EventEmitter.prototype.hasHandler=function(t){return t in this.events},t.tokenizer=function(e){return arguments.length&&null!=e&&void 0!=e?Array.isArray(e)?e.map(function(e){return t.utils.asString(e).toLowerCase()}):e.toString().trim().toLowerCase().split(t.tokenizer.seperator):[]},t.tokenizer.seperator=/[\s\-]+/,t.tokenizer.load=function(t){var e=this.registeredFunctions[t];if(!e)throw new Error("Cannot load un-registered function: "+t);return e},t.tokenizer.label="default",t.tokenizer.registeredFunctions={"default":t.tokenizer},t.tokenizer.registerFunction=function(e,n){n in this.registeredFunctions&&t.utils.warn("Overwriting existing tokenizer: "+n),e.label=n,this.registeredFunctions[n]=e},t.Pipeline=function(){this._stack=[]},t.Pipeline.registeredFunctions={},t.Pipeline.registerFunction=function(e,n){n in this.registeredFunctions&&t.utils.warn("Overwriting existing registered function: "+n),e.label=n,t.Pipeline.registeredFunctions[e.label]=e},t.Pipeline.warnIfFunctionNotRegistered=function(e){var n=e.label&&e.label in this.registeredFunctions;n||t.utils.warn("Function is not registered with pipeline. This may cause problems when serialising the index.\n",e)},t.Pipeline.load=function(e){var n=new t.Pipeline;return e.forEach(function(e){var i=t.Pipeline.registeredFunctions[e];if(!i)throw new Error("Cannot load un-registered function: "+e);n.add(i)}),n},t.Pipeline.prototype.add=function(){var e=Array.prototype.slice.call(arguments);e.forEach(function(e){t.Pipeline.warnIfFunctionNotRegistered(e),this._stack.push(e)},this)},t.Pipeline.prototype.after=function(e,n){t.Pipeline.warnIfFunctionNotRegistered(n);var i=this._stack.indexOf(e);if(-1==i)throw new Error("Cannot find existingFn");i+=1,this._stack.splice(i,0,n)},t.Pipeline.prototype.before=function(e,n){t.Pipeline.warnIfFunctionNotRegistered(n);var i=this._stack.indexOf(e);if(-1==i)throw new Error("Cannot find existingFn");this._stack.splice(i,0,n)},t.Pipeline.prototype.remove=function(t){var e=this._stack.indexOf(t);-1!=e&&this._stack.splice(e,1)},t.Pipeline.prototype.run=function(t){for(var e=[],n=t.length,i=this._stack.length,r=0;n>r;r++){for(var o=t[r],s=0;i>s&&(o=this._stack[s](o,r,t),void 0!==o&&""!==o);s++);void 0!==o&&""!==o&&e.push(o)}return e},t.Pipeline.prototype.reset=function(){this._stack=[]},t.Pipeline.prototype.toJSON=function(){return this._stack.map(function(e){return t.Pipeline.warnIfFunctionNotRegistered(e),e.label})},t.Vector=function(){this._magnitude=null,this.list=void 0,this.length=0},t.Vector.Node=function(t,e,n){this.idx=t,this.val=e,this.next=n},t.Vector.prototype.insert=function(e,n){this._magnitude=void 0;var i=this.list;if(!i)return this.list=new t.Vector.Node(e,n,i),this.length++;if(en.idx?n=n.next:(i+=e.val*n.val,e=e.next,n=n.next);return i},t.Vector.prototype.similarity=function(t){return this.dot(t)/(this.magnitude()*t.magnitude())},t.SortedSet=function(){this.length=0,this.elements=[]},t.SortedSet.load=function(t){var e=new this;return e.elements=t,e.length=t.length,e},t.SortedSet.prototype.add=function(){var t,e;for(t=0;t1;){if(o===t)return r;t>o&&(e=r),o>t&&(n=r),i=n-e,r=e+Math.floor(i/2),o=this.elements[r]}return o===t?r:-1},t.SortedSet.prototype.locationFor=function(t){for(var e=0,n=this.elements.length,i=n-e,r=e+Math.floor(i/2),o=this.elements[r];i>1;)t>o&&(e=r),o>t&&(n=r),i=n-e,r=e+Math.floor(i/2),o=this.elements[r];return o>t?r:t>o?r+1:void 0},t.SortedSet.prototype.intersect=function(e){for(var n=new t.SortedSet,i=0,r=0,o=this.length,s=e.length,a=this.elements,h=e.elements;;){if(i>o-1||r>s-1)break;a[i]!==h[r]?a[i]h[r]&&r++:(n.add(a[i]),i++,r++)}return n},t.SortedSet.prototype.clone=function(){var e=new t.SortedSet;return e.elements=this.toArray(),e.length=e.elements.length,e},t.SortedSet.prototype.union=function(t){var e,n,i;this.length>=t.length?(e=this,n=t):(e=t,n=this),i=e.clone();for(var r=0,o=n.toArray();rp;p++)c[p]===a&&d++;h+=d/f*l.boost}}this.tokenStore.add(a,{ref:o,tf:h})}n&&this.eventEmitter.emit("add",e,this)},t.Index.prototype.remove=function(t,e){var n=t[this._ref],e=void 0===e?!0:e;if(this.documentStore.has(n)){var i=this.documentStore.get(n);this.documentStore.remove(n),i.forEach(function(t){this.tokenStore.remove(t,n)},this),e&&this.eventEmitter.emit("remove",t,this)}},t.Index.prototype.update=function(t,e){var e=void 0===e?!0:e;this.remove(t,!1),this.add(t,!1),e&&this.eventEmitter.emit("update",t,this)},t.Index.prototype.idf=function(t){var e="@"+t;if(Object.prototype.hasOwnProperty.call(this._idfCache,e))return this._idfCache[e];var n=this.tokenStore.count(t),i=1;return n>0&&(i=1+Math.log(this.documentStore.length/n)),this._idfCache[e]=i},t.Index.prototype.search=function(e){var n=this.pipeline.run(this.tokenizerFn(e)),i=new t.Vector,r=[],o=this._fields.reduce(function(t,e){return t+e.boost},0),s=n.some(function(t){return this.tokenStore.has(t)},this);if(!s)return[];n.forEach(function(e,n,s){var a=1/s.length*this._fields.length*o,h=this,u=this.tokenStore.expand(e).reduce(function(n,r){var o=h.corpusTokens.indexOf(r),s=h.idf(r),u=1,l=new t.SortedSet;if(r!==e){var c=Math.max(3,r.length-e.length);u=1/Math.log(c)}o>-1&&i.insert(o,a*s*u);for(var f=h.tokenStore.get(r),d=Object.keys(f),p=d.length,v=0;p>v;v++)l.add(f[d[v]].ref);return n.union(l)},new t.SortedSet);r.push(u)},this);var a=r.reduce(function(t,e){return t.intersect(e)});return a.map(function(t){return{ref:t,score:i.similarity(this.documentVector(t))}},this).sort(function(t,e){return e.score-t.score})},t.Index.prototype.documentVector=function(e){for(var n=this.documentStore.get(e),i=n.length,r=new t.Vector,o=0;i>o;o++){var s=n.elements[o],a=this.tokenStore.get(s)[e].tf,h=this.idf(s);r.insert(this.corpusTokens.indexOf(s),a*h)}return r},t.Index.prototype.toJSON=function(){return{version:t.version,fields:this._fields,ref:this._ref,tokenizer:this.tokenizerFn.label,documentStore:this.documentStore.toJSON(),tokenStore:this.tokenStore.toJSON(),corpusTokens:this.corpusTokens.toJSON(),pipeline:this.pipeline.toJSON()}},t.Index.prototype.use=function(t){var e=Array.prototype.slice.call(arguments,1);e.unshift(this),t.apply(this,e)},t.Store=function(){this.store={},this.length=0},t.Store.load=function(e){var n=new this;return n.length=e.length,n.store=Object.keys(e.store).reduce(function(n,i){return n[i]=t.SortedSet.load(e.store[i]),n},{}),n},t.Store.prototype.set=function(t,e){this.has(t)||this.length++,this.store[t]=e},t.Store.prototype.get=function(t){return this.store[t]},t.Store.prototype.has=function(t){return t in this.store},t.Store.prototype.remove=function(t){this.has(t)&&(delete this.store[t],this.length--)},t.Store.prototype.toJSON=function(){return{store:this.store,length:this.length}},t.stemmer=function(){var t={ational:"ate",tional:"tion",enci:"ence",anci:"ance",izer:"ize",bli:"ble",alli:"al",entli:"ent",eli:"e",ousli:"ous",ization:"ize",ation:"ate",ator:"ate",alism:"al",iveness:"ive",fulness:"ful",ousness:"ous",aliti:"al",iviti:"ive",biliti:"ble",logi:"log"},e={icate:"ic",ative:"",alize:"al",iciti:"ic",ical:"ic",ful:"",ness:""},n="[^aeiou]",i="[aeiouy]",r=n+"[^aeiouy]*",o=i+"[aeiou]*",s="^("+r+")?"+o+r,a="^("+r+")?"+o+r+"("+o+")?$",h="^("+r+")?"+o+r+o+r,u="^("+r+")?"+i,l=new RegExp(s),c=new RegExp(h),f=new RegExp(a),d=new RegExp(u),p=/^(.+?)(ss|i)es$/,v=/^(.+?)([^s])s$/,g=/^(.+?)eed$/,m=/^(.+?)(ed|ing)$/,y=/.$/,S=/(at|bl|iz)$/,w=new RegExp("([^aeiouylsz])\\1$"),k=new RegExp("^"+r+i+"[^aeiouwxy]$"),x=/^(.+?[^aeiou])y$/,b=/^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/,E=/^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/,F=/^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/,_=/^(.+?)(s|t)(ion)$/,z=/^(.+?)e$/,O=/ll$/,P=new RegExp("^"+r+i+"[^aeiouwxy]$"),T=function(n){var i,r,o,s,a,h,u;if(n.length<3)return n;if(o=n.substr(0,1),"y"==o&&(n=o.toUpperCase()+n.substr(1)),s=p,a=v,s.test(n)?n=n.replace(s,"$1$2"):a.test(n)&&(n=n.replace(a,"$1$2")),s=g,a=m,s.test(n)){var T=s.exec(n);s=l,s.test(T[1])&&(s=y,n=n.replace(s,""))}else if(a.test(n)){var T=a.exec(n);i=T[1],a=d,a.test(i)&&(n=i,a=S,h=w,u=k,a.test(n)?n+="e":h.test(n)?(s=y,n=n.replace(s,"")):u.test(n)&&(n+="e"))}if(s=x,s.test(n)){var T=s.exec(n);i=T[1],n=i+"i"}if(s=b,s.test(n)){var T=s.exec(n);i=T[1],r=T[2],s=l,s.test(i)&&(n=i+t[r])}if(s=E,s.test(n)){var T=s.exec(n);i=T[1],r=T[2],s=l,s.test(i)&&(n=i+e[r])}if(s=F,a=_,s.test(n)){var T=s.exec(n);i=T[1],s=c,s.test(i)&&(n=i)}else if(a.test(n)){var T=a.exec(n);i=T[1]+T[2],a=c,a.test(i)&&(n=i)}if(s=z,s.test(n)){var T=s.exec(n);i=T[1],s=c,a=f,h=P,(s.test(i)||a.test(i)&&!h.test(i))&&(n=i)}return s=O,a=c,s.test(n)&&a.test(n)&&(s=y,n=n.replace(s,"")),"y"==o&&(n=o.toLowerCase()+n.substr(1)),n};return T}(),t.Pipeline.registerFunction(t.stemmer,"stemmer"),t.generateStopWordFilter=function(t){var e=t.reduce(function(t,e){return t[e]=e,t},{});return function(t){return t&&e[t]!==t?t:void 0}},t.stopWordFilter=t.generateStopWordFilter(["a","able","about","across","after","all","almost","also","am","among","an","and","any","are","as","at","be","because","been","but","by","can","cannot","could","dear","did","do","does","either","else","ever","every","for","from","get","got","had","has","have","he","her","hers","him","his","how","however","i","if","in","into","is","it","its","just","least","let","like","likely","may","me","might","most","must","my","neither","no","nor","not","of","off","often","on","only","or","other","our","own","rather","said","say","says","she","should","since","so","some","than","that","the","their","them","then","there","these","they","this","tis","to","too","twas","us","wants","was","we","were","what","when","where","which","while","who","whom","why","will","with","would","yet","you","your"]),t.Pipeline.registerFunction(t.stopWordFilter,"stopWordFilter"),t.trimmer=function(t){return t.replace(/^\W+/,"").replace(/\W+$/,"")},t.Pipeline.registerFunction(t.trimmer,"trimmer"),t.TokenStore=function(){this.root={docs:{}},this.length=0},t.TokenStore.load=function(t){var e=new this;return e.root=t.root,e.length=t.length,e},t.TokenStore.prototype.add=function(t,e,n){var n=n||this.root,i=t.charAt(0),r=t.slice(1);return i in n||(n[i]={docs:{}}),0===r.length?(n[i].docs[e.ref]=e,void(this.length+=1)):this.add(r,e,n[i])},t.TokenStore.prototype.has=function(t){if(!t)return!1;for(var e=this.root,n=0;n":">",'"':""","'":"'","/":"/"};function escapeHtml(string){return String(string).replace(/[&<>"'\/]/g,function(s){return entityMap[s]})}var whiteRe=/\s*/;var spaceRe=/\s+/;var equalsRe=/\s*=/;var curlyRe=/\s*\}/;var tagRe=/#|\^|\/|>|\{|&|=|!/;function parseTemplate(template,tags){if(!template)return[];var sections=[];var tokens=[];var spaces=[];var hasTag=false;var nonSpace=false;function stripSpace(){if(hasTag&&!nonSpace){while(spaces.length)delete tokens[spaces.pop()]}else{spaces=[]}hasTag=false;nonSpace=false}var openingTagRe,closingTagRe,closingCurlyRe;function compileTags(tags){if(typeof tags==="string")tags=tags.split(spaceRe,2);if(!isArray(tags)||tags.length!==2)throw new Error("Invalid tags: "+tags);openingTagRe=new RegExp(escapeRegExp(tags[0])+"\\s*");closingTagRe=new RegExp("\\s*"+escapeRegExp(tags[1]));closingCurlyRe=new RegExp("\\s*"+escapeRegExp("}"+tags[1]))}compileTags(tags||mustache.tags);var scanner=new Scanner(template);var start,type,value,chr,token,openSection;while(!scanner.eos()){start=scanner.pos;value=scanner.scanUntil(openingTagRe);if(value){for(var i=0,valueLength=value.length;i0?sections[sections.length-1][4]:nestedTokens;break;default:collector.push(token)}}return nestedTokens}function Scanner(string){this.string=string;this.tail=string;this.pos=0}Scanner.prototype.eos=function(){return this.tail===""};Scanner.prototype.scan=function(re){var match=this.tail.match(re);if(!match||match.index!==0)return"";var string=match[0];this.tail=this.tail.substring(string.length);this.pos+=string.length;return string};Scanner.prototype.scanUntil=function(re){var index=this.tail.search(re),match;switch(index){case-1:match=this.tail;this.tail="";break;case 0:match="";break;default:match=this.tail.substring(0,index);this.tail=this.tail.substring(index)}this.pos+=match.length;return match};function Context(view,parentContext){this.view=view;this.cache={".":this.view};this.parent=parentContext}Context.prototype.push=function(view){return new Context(view,this)};Context.prototype.lookup=function(name){var cache=this.cache;var value;if(name in cache){value=cache[name]}else{var context=this,names,index,lookupHit=false;while(context){if(name.indexOf(".")>0){value=context.view;names=name.split(".");index=0;while(value!=null&&index")value=this._renderPartial(token,context,partials,originalTemplate);else if(symbol==="&")value=this._unescapedValue(token,context);else if(symbol==="name")value=this._escapedValue(token,context);else if(symbol==="text")value=this._rawValue(token);if(value!==undefined)buffer+=value}return buffer};Writer.prototype._renderSection=function(token,context,partials,originalTemplate){var self=this;var buffer="";var value=context.lookup(token[1]);function subRender(template){return self.render(template,context,partials)}if(!value)return;if(isArray(value)){for(var j=0,valueLength=value.length;jthis.depCount&&!this.defined){if(G(l)){if(this.events.error&&this.map.isDefine||g.onError!==ca)try{f=i.execCb(c,l,b,f)}catch(d){a=d}else f=i.execCb(c,l,b,f);this.map.isDefine&&void 0===f&&((b=this.module)?f=b.exports:this.usingExports&& +(f=this.exports));if(a)return a.requireMap=this.map,a.requireModules=this.map.isDefine?[this.map.id]:null,a.requireType=this.map.isDefine?"define":"require",w(this.error=a)}else f=l;this.exports=f;if(this.map.isDefine&&!this.ignore&&(r[c]=f,g.onResourceLoad))g.onResourceLoad(i,this.map,this.depMaps);y(c);this.defined=!0}this.defining=!1;this.defined&&!this.defineEmitted&&(this.defineEmitted=!0,this.emit("defined",this.exports),this.defineEmitComplete=!0)}}else this.fetch()}},callPlugin:function(){var a= +this.map,b=a.id,d=p(a.prefix);this.depMaps.push(d);q(d,"defined",u(this,function(f){var l,d;d=m(aa,this.map.id);var e=this.map.name,P=this.map.parentMap?this.map.parentMap.name:null,n=i.makeRequire(a.parentMap,{enableBuildCallback:!0});if(this.map.unnormalized){if(f.normalize&&(e=f.normalize(e,function(a){return c(a,P,!0)})||""),f=p(a.prefix+"!"+e,this.map.parentMap),q(f,"defined",u(this,function(a){this.init([],function(){return a},null,{enabled:!0,ignore:!0})})),d=m(h,f.id)){this.depMaps.push(f); +if(this.events.error)d.on("error",u(this,function(a){this.emit("error",a)}));d.enable()}}else d?(this.map.url=i.nameToUrl(d),this.load()):(l=u(this,function(a){this.init([],function(){return a},null,{enabled:!0})}),l.error=u(this,function(a){this.inited=!0;this.error=a;a.requireModules=[b];B(h,function(a){0===a.map.id.indexOf(b+"_unnormalized")&&y(a.map.id)});w(a)}),l.fromText=u(this,function(f,c){var d=a.name,e=p(d),P=M;c&&(f=c);P&&(M=!1);s(e);t(j.config,b)&&(j.config[d]=j.config[b]);try{g.exec(f)}catch(h){return w(C("fromtexteval", +"fromText eval for "+b+" failed: "+h,h,[b]))}P&&(M=!0);this.depMaps.push(e);i.completeLoad(d);n([d],l)}),f.load(a.name,n,l,j))}));i.enable(d,this);this.pluginMaps[d.id]=d},enable:function(){V[this.map.id]=this;this.enabling=this.enabled=!0;v(this.depMaps,u(this,function(a,b){var c,f;if("string"===typeof a){a=p(a,this.map.isDefine?this.map:this.map.parentMap,!1,!this.skipMap);this.depMaps[b]=a;if(c=m(L,a.id)){this.depExports[b]=c(this);return}this.depCount+=1;q(a,"defined",u(this,function(a){this.defineDep(b, +a);this.check()}));this.errback?q(a,"error",u(this,this.errback)):this.events.error&&q(a,"error",u(this,function(a){this.emit("error",a)}))}c=a.id;f=h[c];!t(L,c)&&(f&&!f.enabled)&&i.enable(a,this)}));B(this.pluginMaps,u(this,function(a){var b=m(h,a.id);b&&!b.enabled&&i.enable(a,this)}));this.enabling=!1;this.check()},on:function(a,b){var c=this.events[a];c||(c=this.events[a]=[]);c.push(b)},emit:function(a,b){v(this.events[a],function(a){a(b)});"error"===a&&delete this.events[a]}};i={config:j,contextName:b, +registry:h,defined:r,urlFetched:S,defQueue:A,Module:Z,makeModuleMap:p,nextTick:g.nextTick,onError:w,configure:function(a){a.baseUrl&&"/"!==a.baseUrl.charAt(a.baseUrl.length-1)&&(a.baseUrl+="/");var b=j.shim,c={paths:!0,bundles:!0,config:!0,map:!0};B(a,function(a,b){c[b]?(j[b]||(j[b]={}),U(j[b],a,!0,!0)):j[b]=a});a.bundles&&B(a.bundles,function(a,b){v(a,function(a){a!==b&&(aa[a]=b)})});a.shim&&(B(a.shim,function(a,c){H(a)&&(a={deps:a});if((a.exports||a.init)&&!a.exportsFn)a.exportsFn=i.makeShimExports(a); +b[c]=a}),j.shim=b);a.packages&&v(a.packages,function(a){var b,a="string"===typeof a?{name:a}:a;b=a.name;a.location&&(j.paths[b]=a.location);j.pkgs[b]=a.name+"/"+(a.main||"main").replace(ia,"").replace(Q,"")});B(h,function(a,b){!a.inited&&!a.map.unnormalized&&(a.map=p(b))});if(a.deps||a.callback)i.require(a.deps||[],a.callback)},makeShimExports:function(a){return function(){var b;a.init&&(b=a.init.apply(ba,arguments));return b||a.exports&&da(a.exports)}},makeRequire:function(a,e){function j(c,d,m){var n, +q;e.enableBuildCallback&&(d&&G(d))&&(d.__requireJsBuild=!0);if("string"===typeof c){if(G(d))return w(C("requireargs","Invalid require call"),m);if(a&&t(L,c))return L[c](h[a.id]);if(g.get)return g.get(i,c,a,j);n=p(c,a,!1,!0);n=n.id;return!t(r,n)?w(C("notloaded",'Module name "'+n+'" has not been loaded yet for context: '+b+(a?"":". Use require([])"))):r[n]}J();i.nextTick(function(){J();q=s(p(null,a));q.skipMap=e.skipMap;q.init(c,d,m,{enabled:!0});D()});return j}e=e||{};U(j,{isBrowser:z,toUrl:function(b){var d, +e=b.lastIndexOf("."),k=b.split("/")[0];if(-1!==e&&(!("."===k||".."===k)||1e.attachEvent.toString().indexOf("[native code"))&& +!Y?(M=!0,e.attachEvent("onreadystatechange",b.onScriptLoad)):(e.addEventListener("load",b.onScriptLoad,!1),e.addEventListener("error",b.onScriptError,!1)),e.src=d,J=e,D?y.insertBefore(e,D):y.appendChild(e),J=null,e;if(ea)try{importScripts(d),b.completeLoad(c)}catch(m){b.onError(C("importscripts","importScripts failed for "+c+" at "+d,m,[c]))}};z&&!q.skipDataMain&&T(document.getElementsByTagName("script"),function(b){y||(y=b.parentNode);if(I=b.getAttribute("data-main"))return s=I,q.baseUrl||(E=s.split("/"), +s=E.pop(),O=E.length?E.join("/")+"/":"./",q.baseUrl=O),s=s.replace(Q,""),g.jsExtRegExp.test(s)&&(s=I),q.deps=q.deps?q.deps.concat(s):[s],!0});define=function(b,c,d){var e,g;"string"!==typeof b&&(d=c,c=b,b=null);H(c)||(d=c,c=null);!c&&G(d)&&(c=[],d.length&&(d.toString().replace(ka,"").replace(la,function(b,d){c.push(d)}),c=(1===d.length?["require"]:["require","exports","module"]).concat(c)));if(M){if(!(e=J))N&&"interactive"===N.readyState||T(document.getElementsByTagName("script"),function(b){if("interactive"=== +b.readyState)return N=b}),e=N;e&&(b||(b=e.getAttribute("data-requiremodule")),g=F[e.getAttribute("data-requirecontext")])}(g?g.defQueue:R).push([b,c,d])};define.amd={jQuery:!0};g.exec=function(b){return eval(b)};g(q)}})(this); diff --git a/site/mkdocs/js/search-results-template.mustache b/site/mkdocs/js/search-results-template.mustache new file mode 100644 index 0000000..a8b3862 --- /dev/null +++ b/site/mkdocs/js/search-results-template.mustache @@ -0,0 +1,4 @@ + diff --git a/site/mkdocs/js/search.js b/site/mkdocs/js/search.js new file mode 100644 index 0000000..d5c8661 --- /dev/null +++ b/site/mkdocs/js/search.js @@ -0,0 +1,88 @@ +require([ + base_url + '/mkdocs/js/mustache.min.js', + base_url + '/mkdocs/js/lunr.min.js', + 'text!search-results-template.mustache', + 'text!../search_index.json', +], function (Mustache, lunr, results_template, data) { + "use strict"; + + function getSearchTerm() + { + var sPageURL = window.location.search.substring(1); + var sURLVariables = sPageURL.split('&'); + for (var i = 0; i < sURLVariables.length; i++) + { + var sParameterName = sURLVariables[i].split('='); + if (sParameterName[0] == 'q') + { + return decodeURIComponent(sParameterName[1].replace(/\+/g, '%20')); + } + } + } + + var index = lunr(function () { + this.field('title', {boost: 10}); + this.field('text'); + this.ref('location'); + }); + + data = JSON.parse(data); + var documents = {}; + + for (var i=0; i < data.docs.length; i++){ + var doc = data.docs[i]; + doc.location = base_url + doc.location; + index.add(doc); + documents[doc.location] = doc; + } + + var search = function(){ + + var query = document.getElementById('mkdocs-search-query').value; + var search_results = document.getElementById("mkdocs-search-results"); + while (search_results.firstChild) { + search_results.removeChild(search_results.firstChild); + } + + if(query === ''){ + return; + } + + var results = index.search(query); + + if (results.length > 0){ + for (var i=0; i < results.length; i++){ + var result = results[i]; + doc = documents[result.ref]; + doc.base_url = base_url; + doc.summary = doc.text.substring(0, 200); + var html = Mustache.to_html(results_template, doc); + search_results.insertAdjacentHTML('beforeend', html); + } + } else { + search_results.insertAdjacentHTML('beforeend', "

No results found

"); + } + + if(jQuery){ + /* + * We currently only automatically hide bootstrap models. This + * requires jQuery to work. + */ + jQuery('#mkdocs_search_modal a').click(function(){ + jQuery('#mkdocs_search_modal').modal('hide'); + }); + } + + }; + + var search_input = document.getElementById('mkdocs-search-query'); + + var term = getSearchTerm(); + if (term){ + search_input.value = term; + search(); + } + + search_input.addEventListener("keyup", search); + +}); diff --git a/site/mkdocs/js/text.js b/site/mkdocs/js/text.js new file mode 100644 index 0000000..17921b6 --- /dev/null +++ b/site/mkdocs/js/text.js @@ -0,0 +1,390 @@ +/** + * @license RequireJS text 2.0.12 Copyright (c) 2010-2014, The Dojo Foundation All Rights Reserved. + * Available via the MIT or new BSD license. + * see: http://github.com/requirejs/text for details + */ +/*jslint regexp: true */ +/*global require, XMLHttpRequest, ActiveXObject, + define, window, process, Packages, + java, location, Components, FileUtils */ + +define(['module'], function (module) { + 'use strict'; + + var text, fs, Cc, Ci, xpcIsWindows, + progIds = ['Msxml2.XMLHTTP', 'Microsoft.XMLHTTP', 'Msxml2.XMLHTTP.4.0'], + xmlRegExp = /^\s*<\?xml(\s)+version=[\'\"](\d)*.(\d)*[\'\"](\s)*\?>/im, + bodyRegExp = /]*>\s*([\s\S]+)\s*<\/body>/im, + hasLocation = typeof location !== 'undefined' && location.href, + defaultProtocol = hasLocation && location.protocol && location.protocol.replace(/\:/, ''), + defaultHostName = hasLocation && location.hostname, + defaultPort = hasLocation && (location.port || undefined), + buildMap = {}, + masterConfig = (module.config && module.config()) || {}; + + text = { + version: '2.0.12', + + strip: function (content) { + //Strips declarations so that external SVG and XML + //documents can be added to a document without worry. Also, if the string + //is an HTML document, only the part inside the body tag is returned. + if (content) { + content = content.replace(xmlRegExp, ""); + var matches = content.match(bodyRegExp); + if (matches) { + content = matches[1]; + } + } else { + content = ""; + } + return content; + }, + + jsEscape: function (content) { + return content.replace(/(['\\])/g, '\\$1') + .replace(/[\f]/g, "\\f") + .replace(/[\b]/g, "\\b") + .replace(/[\n]/g, "\\n") + .replace(/[\t]/g, "\\t") + .replace(/[\r]/g, "\\r") + .replace(/[\u2028]/g, "\\u2028") + .replace(/[\u2029]/g, "\\u2029"); + }, + + createXhr: masterConfig.createXhr || function () { + //Would love to dump the ActiveX crap in here. Need IE 6 to die first. + var xhr, i, progId; + if (typeof XMLHttpRequest !== "undefined") { + return new XMLHttpRequest(); + } else if (typeof ActiveXObject !== "undefined") { + for (i = 0; i < 3; i += 1) { + progId = progIds[i]; + try { + xhr = new ActiveXObject(progId); + } catch (e) {} + + if (xhr) { + progIds = [progId]; // so faster next time + break; + } + } + } + + return xhr; + }, + + /** + * Parses a resource name into its component parts. Resource names + * look like: module/name.ext!strip, where the !strip part is + * optional. + * @param {String} name the resource name + * @returns {Object} with properties "moduleName", "ext" and "strip" + * where strip is a boolean. + */ + parseName: function (name) { + var modName, ext, temp, + strip = false, + index = name.indexOf("."), + isRelative = name.indexOf('./') === 0 || + name.indexOf('../') === 0; + + if (index !== -1 && (!isRelative || index > 1)) { + modName = name.substring(0, index); + ext = name.substring(index + 1, name.length); + } else { + modName = name; + } + + temp = ext || modName; + index = temp.indexOf("!"); + if (index !== -1) { + //Pull off the strip arg. + strip = temp.substring(index + 1) === "strip"; + temp = temp.substring(0, index); + if (ext) { + ext = temp; + } else { + modName = temp; + } + } + + return { + moduleName: modName, + ext: ext, + strip: strip + }; + }, + + xdRegExp: /^((\w+)\:)?\/\/([^\/\\]+)/, + + /** + * Is an URL on another domain. Only works for browser use, returns + * false in non-browser environments. Only used to know if an + * optimized .js version of a text resource should be loaded + * instead. + * @param {String} url + * @returns Boolean + */ + useXhr: function (url, protocol, hostname, port) { + var uProtocol, uHostName, uPort, + match = text.xdRegExp.exec(url); + if (!match) { + return true; + } + uProtocol = match[2]; + uHostName = match[3]; + + uHostName = uHostName.split(':'); + uPort = uHostName[1]; + uHostName = uHostName[0]; + + return (!uProtocol || uProtocol === protocol) && + (!uHostName || uHostName.toLowerCase() === hostname.toLowerCase()) && + ((!uPort && !uHostName) || uPort === port); + }, + + finishLoad: function (name, strip, content, onLoad) { + content = strip ? text.strip(content) : content; + if (masterConfig.isBuild) { + buildMap[name] = content; + } + onLoad(content); + }, + + load: function (name, req, onLoad, config) { + //Name has format: some.module.filext!strip + //The strip part is optional. + //if strip is present, then that means only get the string contents + //inside a body tag in an HTML string. For XML/SVG content it means + //removing the declarations so the content can be inserted + //into the current doc without problems. + + // Do not bother with the work if a build and text will + // not be inlined. + if (config && config.isBuild && !config.inlineText) { + onLoad(); + return; + } + + masterConfig.isBuild = config && config.isBuild; + + var parsed = text.parseName(name), + nonStripName = parsed.moduleName + + (parsed.ext ? '.' + parsed.ext : ''), + url = req.toUrl(nonStripName), + useXhr = (masterConfig.useXhr) || + text.useXhr; + + // Do not load if it is an empty: url + if (url.indexOf('empty:') === 0) { + onLoad(); + return; + } + + //Load the text. Use XHR if possible and in a browser. + if (!hasLocation || useXhr(url, defaultProtocol, defaultHostName, defaultPort)) { + text.get(url, function (content) { + text.finishLoad(name, parsed.strip, content, onLoad); + }, function (err) { + if (onLoad.error) { + onLoad.error(err); + } + }); + } else { + //Need to fetch the resource across domains. Assume + //the resource has been optimized into a JS module. Fetch + //by the module name + extension, but do not include the + //!strip part to avoid file system issues. + req([nonStripName], function (content) { + text.finishLoad(parsed.moduleName + '.' + parsed.ext, + parsed.strip, content, onLoad); + }); + } + }, + + write: function (pluginName, moduleName, write, config) { + if (buildMap.hasOwnProperty(moduleName)) { + var content = text.jsEscape(buildMap[moduleName]); + write.asModule(pluginName + "!" + moduleName, + "define(function () { return '" + + content + + "';});\n"); + } + }, + + writeFile: function (pluginName, moduleName, req, write, config) { + var parsed = text.parseName(moduleName), + extPart = parsed.ext ? '.' + parsed.ext : '', + nonStripName = parsed.moduleName + extPart, + //Use a '.js' file name so that it indicates it is a + //script that can be loaded across domains. + fileName = req.toUrl(parsed.moduleName + extPart) + '.js'; + + //Leverage own load() method to load plugin value, but only + //write out values that do not have the strip argument, + //to avoid any potential issues with ! in file names. + text.load(nonStripName, req, function (value) { + //Use own write() method to construct full module value. + //But need to create shell that translates writeFile's + //write() to the right interface. + var textWrite = function (contents) { + return write(fileName, contents); + }; + textWrite.asModule = function (moduleName, contents) { + return write.asModule(moduleName, fileName, contents); + }; + + text.write(pluginName, nonStripName, textWrite, config); + }, config); + } + }; + + if (masterConfig.env === 'node' || (!masterConfig.env && + typeof process !== "undefined" && + process.versions && + !!process.versions.node && + !process.versions['node-webkit'])) { + //Using special require.nodeRequire, something added by r.js. + fs = require.nodeRequire('fs'); + + text.get = function (url, callback, errback) { + try { + var file = fs.readFileSync(url, 'utf8'); + //Remove BOM (Byte Mark Order) from utf8 files if it is there. + if (file.indexOf('\uFEFF') === 0) { + file = file.substring(1); + } + callback(file); + } catch (e) { + if (errback) { + errback(e); + } + } + }; + } else if (masterConfig.env === 'xhr' || (!masterConfig.env && + text.createXhr())) { + text.get = function (url, callback, errback, headers) { + var xhr = text.createXhr(), header; + xhr.open('GET', url, true); + + //Allow plugins direct access to xhr headers + if (headers) { + for (header in headers) { + if (headers.hasOwnProperty(header)) { + xhr.setRequestHeader(header.toLowerCase(), headers[header]); + } + } + } + + //Allow overrides specified in config + if (masterConfig.onXhr) { + masterConfig.onXhr(xhr, url); + } + + xhr.onreadystatechange = function (evt) { + var status, err; + //Do not explicitly handle errors, those should be + //visible via console output in the browser. + if (xhr.readyState === 4) { + status = xhr.status || 0; + if (status > 399 && status < 600) { + //An http 4xx or 5xx error. Signal an error. + err = new Error(url + ' HTTP status: ' + status); + err.xhr = xhr; + if (errback) { + errback(err); + } + } else { + callback(xhr.responseText); + } + + if (masterConfig.onXhrComplete) { + masterConfig.onXhrComplete(xhr, url); + } + } + }; + xhr.send(null); + }; + } else if (masterConfig.env === 'rhino' || (!masterConfig.env && + typeof Packages !== 'undefined' && typeof java !== 'undefined')) { + //Why Java, why is this so awkward? + text.get = function (url, callback) { + var stringBuffer, line, + encoding = "utf-8", + file = new java.io.File(url), + lineSeparator = java.lang.System.getProperty("line.separator"), + input = new java.io.BufferedReader(new java.io.InputStreamReader(new java.io.FileInputStream(file), encoding)), + content = ''; + try { + stringBuffer = new java.lang.StringBuffer(); + line = input.readLine(); + + // Byte Order Mark (BOM) - The Unicode Standard, version 3.0, page 324 + // http://www.unicode.org/faq/utf_bom.html + + // Note that when we use utf-8, the BOM should appear as "EF BB BF", but it doesn't due to this bug in the JDK: + // http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4508058 + if (line && line.length() && line.charAt(0) === 0xfeff) { + // Eat the BOM, since we've already found the encoding on this file, + // and we plan to concatenating this buffer with others; the BOM should + // only appear at the top of a file. + line = line.substring(1); + } + + if (line !== null) { + stringBuffer.append(line); + } + + while ((line = input.readLine()) !== null) { + stringBuffer.append(lineSeparator); + stringBuffer.append(line); + } + //Make sure we return a JavaScript string and not a Java string. + content = String(stringBuffer.toString()); //String + } finally { + input.close(); + } + callback(content); + }; + } else if (masterConfig.env === 'xpconnect' || (!masterConfig.env && + typeof Components !== 'undefined' && Components.classes && + Components.interfaces)) { + //Avert your gaze! + Cc = Components.classes; + Ci = Components.interfaces; + Components.utils['import']('resource://gre/modules/FileUtils.jsm'); + xpcIsWindows = ('@mozilla.org/windows-registry-key;1' in Cc); + + text.get = function (url, callback) { + var inStream, convertStream, fileObj, + readData = {}; + + if (xpcIsWindows) { + url = url.replace(/\//g, '\\'); + } + + fileObj = new FileUtils.File(url); + + //XPCOM, you so crazy + try { + inStream = Cc['@mozilla.org/network/file-input-stream;1'] + .createInstance(Ci.nsIFileInputStream); + inStream.init(fileObj, 1, 0, false); + + convertStream = Cc['@mozilla.org/intl/converter-input-stream;1'] + .createInstance(Ci.nsIConverterInputStream); + convertStream.init(inStream, "utf-8", inStream.available(), + Ci.nsIConverterInputStream.DEFAULT_REPLACEMENT_CHARACTER); + + convertStream.readString(inStream.available(), readData); + convertStream.close(); + inStream.close(); + callback(readData.value); + } catch (e) { + throw new Error((fileObj && fileObj.path || '') + ': ' + e); + } + }; + } + return text; +}); diff --git a/site/mkdocs/search_index.json b/site/mkdocs/search_index.json new file mode 100644 index 0000000..4199c79 --- /dev/null +++ b/site/mkdocs/search_index.json @@ -0,0 +1,559 @@ +{ + "docs": [ + { + "location": "/", + "text": "Index\n\n\nThe \"\nGeek's Cookbook\n\" is a collection of guides for establishing your own highly-available \"private cloud\" \n1\n. This cloud enables you to run self-hosted services such as \nGitLab\n, \nPlex\n, \nNextCloud\n, etc.\n\n\nWho is this for?\n\n\nYou already have a familiarity with concepts such as \nvirtual\n \nmachines\n, \nDocker\n containers, \nLetsEncrypt SSL certificates\n, databases, and command-line interfaces.\n\n\nYou've probably played with self-hosting some mainstream apps yourself, like \nPlex\n, \nOwnCloud\n, \nWordpress\n or even \nSandStorm\n.\n\n\nWhy should I read this?\n\n\nSo if you're familiar enough with the tools, and you've done self-hosting before, why would you read this book?\n\n\n\n\nYou want to upskill. You want to do container orchestration, LetsEncrypt certificates, git collaboration.\n\n\nYou want to play. You want a safe sandbox to test new tools, keeping the ones you want and tossing the ones you don't.\n\n\nYou want reliability. Once you go from \nplaying\n with a tool to actually \nusing\n it, you want it to be available when you need it. Having to \"\nquickly ssh into the host and restart the webserver\n\" doesn't cut it when your wife wants to know why her phone won't sync!\n\n\n\n\nWhat do you want from me?\n\n\nI want your money.\n\n\nNo, seriously (\nbut yes, I do want your money - see below\n), If the above applies to you, then you're like me. I want everything I wrote above, so I ended up learning all this as I went along. I enjoy it, and I'm good at it. So I created this website, partly to make sure I documented my own setup properly.\n\n\nHow can I support you?\n\n\nBuy my book \ud83d\udcd6\n\n\nI'm also writing it as a formal book, on Leanpub (\nhttps://leanpub.com/geeks-cookbook\n). Buy it for $0.99 (which is really just a token gesture of support) - you can get it for free (in PDF, mobi, or epub format), or pay me what you think it's worth!\n\n\nPatreonize me \ud83d\udcb0\n\n\nBecome a Patron!\n\n- \nMy Patreon page\n!\n\n\nHire me \ud83c\udfe2\n\n\nNeed some system design work done? I do freelance consulting - \ncontact\n me for details.\n\n\n\n\n\n\n\n\n\n\nSorry for the buzzword, I couldn't think of a better description!", + "title": "Home" + }, + { + "location": "/#index", + "text": "The \" Geek's Cookbook \" is a collection of guides for establishing your own highly-available \"private cloud\" 1 . This cloud enables you to run self-hosted services such as GitLab , Plex , NextCloud , etc.", + "title": "Index" + }, + { + "location": "/#who-is-this-for", + "text": "You already have a familiarity with concepts such as virtual machines , Docker containers, LetsEncrypt SSL certificates , databases, and command-line interfaces. You've probably played with self-hosting some mainstream apps yourself, like Plex , OwnCloud , Wordpress or even SandStorm .", + "title": "Who is this for?" + }, + { + "location": "/#why-should-i-read-this", + "text": "So if you're familiar enough with the tools, and you've done self-hosting before, why would you read this book? You want to upskill. You want to do container orchestration, LetsEncrypt certificates, git collaboration. You want to play. You want a safe sandbox to test new tools, keeping the ones you want and tossing the ones you don't. You want reliability. Once you go from playing with a tool to actually using it, you want it to be available when you need it. Having to \" quickly ssh into the host and restart the webserver \" doesn't cut it when your wife wants to know why her phone won't sync!", + "title": "Why should I read this?" + }, + { + "location": "/#what-do-you-want-from-me", + "text": "I want your money. No, seriously ( but yes, I do want your money - see below ), If the above applies to you, then you're like me. I want everything I wrote above, so I ended up learning all this as I went along. I enjoy it, and I'm good at it. So I created this website, partly to make sure I documented my own setup properly.", + "title": "What do you want from me?" + }, + { + "location": "/#how-can-i-support-you", + "text": "", + "title": "How can I support you?" + }, + { + "location": "/#buy-my-book", + "text": "I'm also writing it as a formal book, on Leanpub ( https://leanpub.com/geeks-cookbook ). Buy it for $0.99 (which is really just a token gesture of support) - you can get it for free (in PDF, mobi, or epub format), or pay me what you think it's worth!", + "title": "Buy my book \ud83d\udcd6" + }, + { + "location": "/#patreonize-me", + "text": "Become a Patron! \n- My Patreon page !", + "title": "Patreonize me \ud83d\udcb0" + }, + { + "location": "/#hire-me", + "text": "Need some system design work done? I do freelance consulting - contact me for details. Sorry for the buzzword, I couldn't think of a better description!", + "title": "Hire me \ud83c\udfe2" + }, + { + "location": "/README/", + "text": "How to read this book\n\n\nStructure\n\n\n\n\n\"Recipies\" generally follow on from each other. I.e., if a particular recipe requires a mail server, that mail server would have been described in an earlier recipe.\n\n\nEach recipe contains enough detail in a single page to take a project from start to completion.\n\n\nWhen there are optional add-ons/integrations possible to a project (i.e., the addition of \"smart LED bulbs\" to Home Assistant), this will be reflected either as a brief \"Chef's note\" after the recipe, or if they're substantial enough, as a sub-page of the main project\n\n\n\n\nConventions\n\n\n\n\nWhen creating swarm networks, we always explicitly set the subnet in the overlay network, to avoid potential conflicts (which docker won't prevent, but which will generate errors) (\nhttps://github.com/moby/moby/issues/26912\n)", + "title": "README" + }, + { + "location": "/README/#how-to-read-this-book", + "text": "", + "title": "How to read this book" + }, + { + "location": "/README/#structure", + "text": "\"Recipies\" generally follow on from each other. I.e., if a particular recipe requires a mail server, that mail server would have been described in an earlier recipe. Each recipe contains enough detail in a single page to take a project from start to completion. When there are optional add-ons/integrations possible to a project (i.e., the addition of \"smart LED bulbs\" to Home Assistant), this will be reflected either as a brief \"Chef's note\" after the recipe, or if they're substantial enough, as a sub-page of the main project", + "title": "Structure" + }, + { + "location": "/README/#conventions", + "text": "When creating swarm networks, we always explicitly set the subnet in the overlay network, to avoid potential conflicts (which docker won't prevent, but which will generate errors) ( https://github.com/moby/moby/issues/26912 )", + "title": "Conventions" + }, + { + "location": "/whoami/", + "text": "Welcome to Funky Penguin's Geek Cookbook\n\n\nHello world,\n\n\nI'm \nDavid\n.\n\n\nI've spent 20+ years working with technology. My current role is \nSenior Infrastructure Architect\n at \nProphecy Networks Ltd\n in New Zealand, with a specific interest in networking, systems, open-source, and business management.\n\n\nI've had a \nbook published\n, and I \nblog\n on topics that interest me.\n\n\nWhy Funky Penguin?\n\n\nMy first \"real\" job, out of high-school, was working the IT helpdesk in a typical pre-2000 organization in South Africa. I enjoyed experimenting with Linux, and cut my teeth by replacing the organization's Exchange 5.5 mail platform with a 15-site \nqmail-ldap\n cluster, with \namavis\n virus-scanning.\n\n\nOne of our suppliers asked me to quote to do the same for their organization. With nothing to loose, and half-expecting to be turned down, I quoted a generous fee, and chose a cheeky company name. The supplier immediately accepted my quote, and the name (\"\nFunky Penguin\n\") stuck.\n\n\nTechnical Documentation\n\n\nDuring the same \"real\" job above, I wanted to deploy \njabberd\n, for internal instant messaging within the organization, and as a means to control the sprawl of ad-hoc instant-messaging among staff, using ICQ, MSN, and Yahoo Messenger.\n\n\nTo get management approval to deploy, I wrote a logger (with web UI) for jabber conversations (\nBandersnatch\n), and a \n75-page user manual\n (in \nDocbook XML\n for a spunky Russian WinXP jabber client, \nJAJC\n.\n\n\nDue to my contributions to \nphpList\n, I was approached in 2011 by \nPackt Publishing\n, to \nwrite a book\n about using PHPList.\n\n\nContact Me\n\n\nContact me by:\n\n\n\n\nEmail (\n)\n\n\nTwitter (\n@funkypenguin\n)\n\n\nMastodon (\n@davidy@funkypenguin.co.nz\n)\n\n\n\n\nOr by using the form below:", + "title": "whoami" + }, + { + "location": "/whoami/#welcome-to-funky-penguins-geek-cookbook", + "text": "", + "title": "Welcome to Funky Penguin's Geek Cookbook" + }, + { + "location": "/whoami/#hello-world", + "text": "I'm David . I've spent 20+ years working with technology. My current role is Senior Infrastructure Architect at Prophecy Networks Ltd in New Zealand, with a specific interest in networking, systems, open-source, and business management. I've had a book published , and I blog on topics that interest me.", + "title": "Hello world," + }, + { + "location": "/whoami/#why-funky-penguin", + "text": "My first \"real\" job, out of high-school, was working the IT helpdesk in a typical pre-2000 organization in South Africa. I enjoyed experimenting with Linux, and cut my teeth by replacing the organization's Exchange 5.5 mail platform with a 15-site qmail-ldap cluster, with amavis virus-scanning. One of our suppliers asked me to quote to do the same for their organization. With nothing to loose, and half-expecting to be turned down, I quoted a generous fee, and chose a cheeky company name. The supplier immediately accepted my quote, and the name (\" Funky Penguin \") stuck.", + "title": "Why Funky Penguin?" + }, + { + "location": "/whoami/#technical-documentation", + "text": "During the same \"real\" job above, I wanted to deploy jabberd , for internal instant messaging within the organization, and as a means to control the sprawl of ad-hoc instant-messaging among staff, using ICQ, MSN, and Yahoo Messenger. To get management approval to deploy, I wrote a logger (with web UI) for jabber conversations ( Bandersnatch ), and a 75-page user manual (in Docbook XML for a spunky Russian WinXP jabber client, JAJC . Due to my contributions to phpList , I was approached in 2011 by Packt Publishing , to write a book about using PHPList.", + "title": "Technical Documentation" + }, + { + "location": "/whoami/#contact-me", + "text": "Contact me by: Email ( ) Twitter ( @funkypenguin ) Mastodon ( @davidy@funkypenguin.co.nz ) Or by using the form below:", + "title": "Contact Me" + }, + { + "location": "/ha-docker-swarm/design/", + "text": "Design\n\n\nIn the design described below, the \"private cloud\" platform is:\n\n\n\n\nHighly-available\n (\ncan tolerate the failure of a single component\n)\n\n\nScalable\n (\ncan add resource or capacity as required\n)\n\n\nPortable\n (\nrun it on your garage server today, run it in AWS tomorrow\n)\n\n\nSecure\n (\naccess protected with LetsEncrypt certificates\n)\n\n\nAutomated\n (\nrequires minimal care and feeding\n)\n\n\n\n\nDesign Decisions\n\n\nWhere possible, services will be highly available.\n\n\nThis means that:\n\n\n\n\nAt least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure.\n\n\nGlusterFS is employed for share filesystem, because it too can be made tolerant of a single failure.\n\n\n\n\nWhere multiple solutions to a requirement exist, preference will be given to the most portable solution.\n\n\nThis means that:\n\n\n\n\nServices are defined using docker-compose v3 YAML syntax\n\n\nServices are portable, meaning a particular stack could be shut down and moved to a new provider with minimal effort.\n\n\n\n\nSecurity\n\n\nUnder this design, the only inbound connections we're permitting to our docker swarm are:\n\n\nNetwork Flows\n\n\n\n\nHTTP (TCP 80) : Redirects to https\n\n\nHTTPS (TCP 443) : Serves individual docker containers via SSL-encrypted reverse proxy\n\n\n\n\nAuthentication\n\n\n\n\nWhere the proxied application provides a trusted level of authentication, or where the application requires public exposure,\n\n\n\n\nHigh availability\n\n\nNormal function\n\n\nAssuming 3 nodes, under normal circumstances the following is illustrated:\n\n\n\n\nAll 3 nodes provide shared storage via GlusterFS, which is provided by a docker container on each node. (i.e., not running in swarm mode)\n\n\nAll 3 nodes participate in the Docker Swarm as managers.\n\n\nThe various containers belonging to the application \"stacks\" deployed within Docker Swarm are automatically distributed amongst the swarm nodes.\n\n\nPersistent storage for the containers is provide via GlusterFS mount.\n\n\nThe \ntraefik\n service (in swarm mode) receives incoming requests (on http and https), and forwards them to individual containers. Traefik knows the containers names because it's able to access the docker socket.\n\n\nAll 3 nodes run keepalived, at different priorities. Since traefik is running as a swarm service and listening on TCP 80/443, requests made to the keepalived VIP and arriving at \nany\n of the swarm nodes will be forwarded to the traefik container (no matter which node it's on), and then onto the target backend.\n\n\n\n\n\n\nNode failure\n\n\nIn the case of a failure (or scheduled maintenance) of one of the nodes, the following is illustrated:\n\n\n\n\nThe failed node no longer participates in GlusterFS, but the remaining nodes provide enough fault-tolerance for the cluster to operate.\n\n\nThe remaining two nodes in Docker Swarm achieve a quorum and agree that the failed node is to be removed.\n\n\nThe (possibly new) leader manager node reschedules the containers known to be running on the failed node, onto other nodes.\n\n\nThe \ntraefik\n service is either restarted or unaffected, and as the backend containers stop/start and change IP, traefik is aware and updates accordingly.\n\n\nThe keepalived VIP continues to function on the remaining nodes, and docker swarm continues to forward any traffic received on TCP 80/443 to the appropriate node.\n\n\n\n\n\n\nNode restore\n\n\nWhen the failed (or upgraded) host is restored to service, the following is illustrated:\n\n\n\n\nGlusterFS regains full redundancy\n\n\nDocker Swarm managers become aware of the recovered node, and will use it for scheduling \nnew\n containers\n\n\nExisting containers which were migrated off the node are not migrated backend\n\n\nKeepalived VIP regains full redundancy\n\n\n\n\n\n\nTotal cluster failure\n\n\nA day after writing this, my environment suffered a fault whereby all 3 VMs were unexpectedly and simultaneously powered off.\n\n\nUpon restore, docker failed to start on one of the VMs due to local disk space issue\n1\n. However, the other two VMs started, established the swarm, mounted their shared storage, and started up all the containers (services) which were managed by the swarm.\n\n\nIn summary, although I suffered an \nunplanned power outage to all of my infrastructure\n, followed by a \nfailure of a third of my hosts\n... \nall my platforms are 100% available with \nabsolutely no manual intervention\n.\n\n\n\n\n\n\n\n\n\n\nSince there's no impact to availability, I can fix (or just reinstall) the failed node whenever convenient.", + "title": "Design" + }, + { + "location": "/ha-docker-swarm/design/#design", + "text": "In the design described below, the \"private cloud\" platform is: Highly-available ( can tolerate the failure of a single component ) Scalable ( can add resource or capacity as required ) Portable ( run it on your garage server today, run it in AWS tomorrow ) Secure ( access protected with LetsEncrypt certificates ) Automated ( requires minimal care and feeding )", + "title": "Design" + }, + { + "location": "/ha-docker-swarm/design/#design-decisions", + "text": "Where possible, services will be highly available. This means that: At least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure. GlusterFS is employed for share filesystem, because it too can be made tolerant of a single failure. Where multiple solutions to a requirement exist, preference will be given to the most portable solution. This means that: Services are defined using docker-compose v3 YAML syntax Services are portable, meaning a particular stack could be shut down and moved to a new provider with minimal effort.", + "title": "Design Decisions" + }, + { + "location": "/ha-docker-swarm/design/#security", + "text": "Under this design, the only inbound connections we're permitting to our docker swarm are:", + "title": "Security" + }, + { + "location": "/ha-docker-swarm/design/#network-flows", + "text": "HTTP (TCP 80) : Redirects to https HTTPS (TCP 443) : Serves individual docker containers via SSL-encrypted reverse proxy", + "title": "Network Flows" + }, + { + "location": "/ha-docker-swarm/design/#authentication", + "text": "Where the proxied application provides a trusted level of authentication, or where the application requires public exposure,", + "title": "Authentication" + }, + { + "location": "/ha-docker-swarm/design/#high-availability", + "text": "", + "title": "High availability" + }, + { + "location": "/ha-docker-swarm/design/#normal-function", + "text": "Assuming 3 nodes, under normal circumstances the following is illustrated: All 3 nodes provide shared storage via GlusterFS, which is provided by a docker container on each node. (i.e., not running in swarm mode) All 3 nodes participate in the Docker Swarm as managers. The various containers belonging to the application \"stacks\" deployed within Docker Swarm are automatically distributed amongst the swarm nodes. Persistent storage for the containers is provide via GlusterFS mount. The traefik service (in swarm mode) receives incoming requests (on http and https), and forwards them to individual containers. Traefik knows the containers names because it's able to access the docker socket. All 3 nodes run keepalived, at different priorities. Since traefik is running as a swarm service and listening on TCP 80/443, requests made to the keepalived VIP and arriving at any of the swarm nodes will be forwarded to the traefik container (no matter which node it's on), and then onto the target backend.", + "title": "Normal function" + }, + { + "location": "/ha-docker-swarm/design/#node-failure", + "text": "In the case of a failure (or scheduled maintenance) of one of the nodes, the following is illustrated: The failed node no longer participates in GlusterFS, but the remaining nodes provide enough fault-tolerance for the cluster to operate. The remaining two nodes in Docker Swarm achieve a quorum and agree that the failed node is to be removed. The (possibly new) leader manager node reschedules the containers known to be running on the failed node, onto other nodes. The traefik service is either restarted or unaffected, and as the backend containers stop/start and change IP, traefik is aware and updates accordingly. The keepalived VIP continues to function on the remaining nodes, and docker swarm continues to forward any traffic received on TCP 80/443 to the appropriate node.", + "title": "Node failure" + }, + { + "location": "/ha-docker-swarm/design/#node-restore", + "text": "When the failed (or upgraded) host is restored to service, the following is illustrated: GlusterFS regains full redundancy Docker Swarm managers become aware of the recovered node, and will use it for scheduling new containers Existing containers which were migrated off the node are not migrated backend Keepalived VIP regains full redundancy", + "title": "Node restore" + }, + { + "location": "/ha-docker-swarm/design/#total-cluster-failure", + "text": "A day after writing this, my environment suffered a fault whereby all 3 VMs were unexpectedly and simultaneously powered off. Upon restore, docker failed to start on one of the VMs due to local disk space issue 1 . However, the other two VMs started, established the swarm, mounted their shared storage, and started up all the containers (services) which were managed by the swarm. In summary, although I suffered an unplanned power outage to all of my infrastructure , followed by a failure of a third of my hosts ... all my platforms are 100% available with absolutely no manual intervention . Since there's no impact to availability, I can fix (or just reinstall) the failed node whenever convenient.", + "title": "Total cluster failure" + }, + { + "location": "/ha-docker-swarm/vms/", + "text": "Virtual Machines\n\n\nLet's start building our cloud with virtual machines. You could use bare-metal machines as well, the configuration would be the same. Given that most readers (myself included) will be using virtual infrastructure, from now on I'll be referring strictly to VMs.\n\n\nI chose the \"\nAtomic\n\" CentOS/Fedora image for the VM layer because:\n\n\n\n\nI want less responsibility for maintaining the system, including ensuring regular software updates and reboots. Atomic's idempotent nature means the OS is largely real-only, and updates/rollbacks are \"atomic\" (haha) procedures, which can be easily rolled back if required.\n\n\nFor someone used to administrating servers individually, Atomic is a PITA. You have to employ \ntricky\n \ntricks\n to get it to install in a non-cloud environment. It's not designed for tweaking or customizing beyond what cloud-config is capable of. For my purposes, this is good, because it forces me to change my thinking - to consider every daemon as a container, and every config as code, to be checked in and version-controlled. Atomic forces this thinking on you.\n\n\nI want the design to be as \"portable\" as possible. While I run it on VPSs now, I may want to migrate it to a \"cloud\" provider in the future, and I'll want the most portable, reproducible design.\n\n\n\n\nIngredients\n\n\n\n\nIngredients\n\n\n3 x Virtual Machines, each with:\n\n\n\n\n CentOS/Fedora Atomic\n\n\n At least 1GB RAM\n\n\n At least 20GB disk space (\nbut it'll be tight\n)\n\n\n Connectivity to each other within the same subnet, and on a low-latency link (\ni.e., no WAN links\n)\n\n\n\n\n\n\nPreparation\n\n\nInstall Virtual machines\n\n\n\n\nInstall / launch virtual machines.\n\n\nThe default username on CentOS atomic is \"centos\", and you'll have needed to supply your SSH key during the build process.\n\n\n\n\n\n\nTip\n\n\nIf you're not using a platform with cloud-init support (i.e., you're building a VM manually, not provisioning it through a cloud provider), you'll need to refer to \ntrick #1\n and \n#2\n for a means to override the automated setup, apply a manual password to the CentOS account, and enable SSH password logins.\n\n\n\n\nPrefer docker-latest\n\n\nRun the following on each node to replace the default docker 1.12 with docker 1.13 (\nwhich we need for swarm mode\n):\n\n1\n2\n3\nsystemctl disable docker --now\nsystemctl enable docker-latest --now\nsed -i \n/DOCKERBINARY/s/^#//g\n /etc/sysconfig/docker\n\n\n\n\n\nUpgrade Atomic\n\n\nFinally, apply any Atomic host updates, and reboot, by running: \natomic host upgrade \n systemctl reboot\n.\n\n\nPermit connectivity between VMs\n\n\nBy default, Atomic only permits incoming SSH. We'll want to allow all traffic between our nodes, so add something like this to /etc/sysconfig/iptables:\n\n\n1\n2\n# Allow all inter-node communication\n-A INPUT -s 192.168.31.0/24 -j ACCEPT\n\n\n\n\n\n\nAnd restart iptables with \nsystemctl restart iptables\n\n\nEnable host resolution\n\n\nDepending on your hosting environment, you may have DNS automatically setup for your VMs. If not, it's useful to set up static entries in /etc/hosts for the nodes. For example, I setup the following:\n\n\n1\n2\n3\n192.168.31.11 ds1 ds1.funkypenguin.co.nz\n192.168.31.12 ds2 ds2.funkypenguin.co.nz\n192.168.31.13 ds3 ds3.funkypenguin.co.nz\n\n\n\n\n\n\nServing\n\n\nAfter completing the above, you should have:\n\n\n1\n2\n[X] 3 x fresh atomic instances, at the latest releases,\n running Docker v1.13 (docker-latest)", + "title": "VMs" + }, + { + "location": "/ha-docker-swarm/vms/#virtual-machines", + "text": "Let's start building our cloud with virtual machines. You could use bare-metal machines as well, the configuration would be the same. Given that most readers (myself included) will be using virtual infrastructure, from now on I'll be referring strictly to VMs. I chose the \" Atomic \" CentOS/Fedora image for the VM layer because: I want less responsibility for maintaining the system, including ensuring regular software updates and reboots. Atomic's idempotent nature means the OS is largely real-only, and updates/rollbacks are \"atomic\" (haha) procedures, which can be easily rolled back if required. For someone used to administrating servers individually, Atomic is a PITA. You have to employ tricky tricks to get it to install in a non-cloud environment. It's not designed for tweaking or customizing beyond what cloud-config is capable of. For my purposes, this is good, because it forces me to change my thinking - to consider every daemon as a container, and every config as code, to be checked in and version-controlled. Atomic forces this thinking on you. I want the design to be as \"portable\" as possible. While I run it on VPSs now, I may want to migrate it to a \"cloud\" provider in the future, and I'll want the most portable, reproducible design.", + "title": "Virtual Machines" + }, + { + "location": "/ha-docker-swarm/vms/#ingredients", + "text": "Ingredients 3 x Virtual Machines, each with: CentOS/Fedora Atomic At least 1GB RAM At least 20GB disk space ( but it'll be tight ) Connectivity to each other within the same subnet, and on a low-latency link ( i.e., no WAN links )", + "title": "Ingredients" + }, + { + "location": "/ha-docker-swarm/vms/#preparation", + "text": "", + "title": "Preparation" + }, + { + "location": "/ha-docker-swarm/vms/#install-virtual-machines", + "text": "Install / launch virtual machines. The default username on CentOS atomic is \"centos\", and you'll have needed to supply your SSH key during the build process. Tip If you're not using a platform with cloud-init support (i.e., you're building a VM manually, not provisioning it through a cloud provider), you'll need to refer to trick #1 and #2 for a means to override the automated setup, apply a manual password to the CentOS account, and enable SSH password logins.", + "title": "Install Virtual machines" + }, + { + "location": "/ha-docker-swarm/vms/#prefer-docker-latest", + "text": "Run the following on each node to replace the default docker 1.12 with docker 1.13 ( which we need for swarm mode ): 1\n2\n3 systemctl disable docker --now\nsystemctl enable docker-latest --now\nsed -i /DOCKERBINARY/s/^#//g /etc/sysconfig/docker", + "title": "Prefer docker-latest" + }, + { + "location": "/ha-docker-swarm/vms/#upgrade-atomic", + "text": "Finally, apply any Atomic host updates, and reboot, by running: atomic host upgrade systemctl reboot .", + "title": "Upgrade Atomic" + }, + { + "location": "/ha-docker-swarm/vms/#permit-connectivity-between-vms", + "text": "By default, Atomic only permits incoming SSH. We'll want to allow all traffic between our nodes, so add something like this to /etc/sysconfig/iptables: 1\n2 # Allow all inter-node communication\n-A INPUT -s 192.168.31.0/24 -j ACCEPT And restart iptables with systemctl restart iptables", + "title": "Permit connectivity between VMs" + }, + { + "location": "/ha-docker-swarm/vms/#enable-host-resolution", + "text": "Depending on your hosting environment, you may have DNS automatically setup for your VMs. If not, it's useful to set up static entries in /etc/hosts for the nodes. For example, I setup the following: 1\n2\n3 192.168.31.11 ds1 ds1.funkypenguin.co.nz\n192.168.31.12 ds2 ds2.funkypenguin.co.nz\n192.168.31.13 ds3 ds3.funkypenguin.co.nz", + "title": "Enable host resolution" + }, + { + "location": "/ha-docker-swarm/vms/#serving", + "text": "After completing the above, you should have: 1\n2 [X] 3 x fresh atomic instances, at the latest releases,\n running Docker v1.13 (docker-latest)", + "title": "Serving" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/", + "text": "Shared Storage (Ceph)\n\n\nWhile Docker Swarm is great for keeping containers running (\nand restarting those that fail\n), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (\nhint: you do!\n), you need to provide shared storage to every docker node.\n\n\nDesign\n\n\nWhy not GlusterFS?\n\n\nI originally provided shared storage to my nodes using GlusterFS (see the next recipe for details), but found it difficult to deal with because:\n\n\n\n\nGlusterFS requires (n) \"bricks\", where (n) \nhas\n to be a multiple of your replica count. I.e., if you want 2 copies of everything on shared storage (the minimum to provide redundancy), you \nmust\n have either 2, 4, 6 (etc..) bricks. The HA swarm design calls for minimum of 3 nodes, and so under GlusterFS, my third node can't participate in shared storage at all, unless I start doubling up on bricks-per-node (which then impacts redundancy)\n\n\nGlusterFS turns out to be a giant PITA when you want to restore a failed node. There are at \nleast 14 steps to follow\n to replace a brick.\n\n\nI'm pretty sure I messed up the 14-step process above anyway. My replaced brick synced with my \"original\" brick, but produced errors when querying status via the CLI, and hogged 100% of 1 CPU on the replaced node. Inexperienced with GlusterFS, and unable to diagnose the fault, I switched to a Ceph cluster instead.\n\n\n\n\nWhy Ceph?\n\n\n\n\nI'm more familiar with Ceph - I use it in the OpenStack designs I manage\n\n\nReplacing a failed node is \neasy\n, provided you can put up with the I/O load of rebalancing OSDs after the replacement.\n\n\nCentOS Atomic includes the ceph client in the OS, so while the Ceph OSD/Mon/MSD are running under containers, I can keep an eye (and later, automatically monitor) the status of Ceph from the base OS.\n\n\n\n\nIngredients\n\n\n\n\nIngredients\n\n\n3 x Virtual Machines (configured earlier), each with:\n\n\n\n\n CentOS/Fedora Atomic\n\n\n At least 1GB RAM\n\n\n At least 20GB disk space (\nbut it'll be tight\n)\n\n\n Connectivity to each other within the same subnet, and on a low-latency link (\ni.e., no WAN links\n)\n\n\n A second disk dedicated to the Ceph OSD\n\n\n\n\n\n\nPreparation\n\n\nSELinux\n\n\nSince our Ceph components will be containerized, we need to ensure the SELinux context on the base OS's ceph files is set correctly:\n\n\n1\n2\nchcon -Rt svirt_sandbox_file_t /etc/ceph\nchcon -Rt svirt_sandbox_file_t /var/lib/ceph\n\n\n\n\n\n\nSetup Monitors\n\n\nPick a node, and run the following to stand up the first Ceph mon. Be sure to replace the values for \nMON_IP\n and \nCEPH_PUBLIC_NETWORK\n to those specific to your deployment:\n\n\n1\n2\n3\n4\n5\n6\n7\n8\ndocker run -d --net=host \\\n--restart always \\\n-v /etc/ceph:/etc/ceph \\\n-v /var/lib/ceph/:/var/lib/ceph/ \\\n-e MON_IP=192.168.31.11 \\\n-e CEPH_PUBLIC_NETWORK=192.168.31.0/24 \\\n--name=\nceph-mon\n \\\nceph/daemon mon\n\n\n\n\n\n\nNow \ncopy\n the contents of /etc/ceph on this first node to the remaining nodes, and \nthen\n run the docker command above (\ncustomizing MON_IP as you go\n) on each remaining node. You'll end up with a cluster with 3 monitors (odd number is required for quorum, same as Docker Swarm), and no OSDs (yet)\n\n\nSetup OSDs\n\n\nSince we have a OSD-less mon-only cluster currently, prepare for OSD creation by dumping the auth credentials for the OSDs into the appropriate location on the base OS:\n\n\n1\n2\nceph auth get client.bootstrap-osd -o \\\n/var/lib/ceph/bootstrap-osd/ceph.keyring\n\n\n\n\n\n\nOn each node, you need a dedicated disk for the OSD. In the example below, I used \n/dev/vdd\n (the entire disk, no partitions) for the OSD.\n\n\nRun the following command on every node:\n\n\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\ndocker run -d --net=host \\\n--privileged=true \\\n--pid=host \\\n-v /etc/ceph:/etc/ceph \\\n-v /var/lib/ceph/:/var/lib/ceph/ \\\n-v /dev/:/dev/ \\\n-e OSD_DEVICE=/dev/vdd \\\n-e OSD_TYPE=disk \\\n--name=\nceph-osd\n \\\n--restart=always \\\nceph/daemon osd\n\n\n\n\n\n\nWatch the output by running \ndocker logs ceph-osd -f\n, and confirm success.\n\n\n\n\nZapping the device\n\n\nThe Ceph OSD container will refuse to destroy a partition containing existing data, so it may be necessary to \"zap\" the target disk, using:\n\n1\n2\n3\n4\ndocker run -d --privileged=true \\\n-v /dev/:/dev/ \\\n-e OSD_DEVICE=/dev/sdd \\\nceph/daemon zap_device\n\n\n\n\n\n\n\nSetup MDSs\n\n\nIn order to mount our ceph pools as filesystems, we'll need Ceph MDS(s). Run the following on each node:\n\n\n1\n2\n3\n4\n5\n6\n7\n8\n9\ndocker run -d --net=host \\\n--name ceph-mds \\\n--restart always \\\n-v /var/lib/ceph/:/var/lib/ceph/ \\\n-v /etc/ceph:/etc/ceph \\\n-e CEPHFS_CREATE=1 \\\n-e CEPHFS_DATA_POOL_PG=256 \\\n-e CEPHFS_METADATA_POOL_PG=256 \\\nceph/daemon mds\n\n\n\n\n\n\nApply tweaks\n\n\nThe ceph container seems to configure a pool default of 3 replicas (3 copies of each block are retained), which is one too many for our cluster (we are only protecting against the failure of a single node).\n\n\nRun the following on any node to reduce the size of the pool to 2 replicas:\n\n\n1\n2\nceph osd pool set cephfs_data size 2\nceph osd pool set cephfs_metadata size 2\n\n\n\n\n\n\nDisabled \"scrubbing\" (which can be IO-intensive, and is unnecessary on a VM) with:\n\n\n1\n2\nceph osd set noscrub\nceph osd set nodeep-scrub\n\n\n\n\n\n\nCreate credentials for swarm\n\n\nIn order to mount the ceph volume onto our base host, we need to provide cephx authentication credentials.\n\n\nOn \none\n node, create a client for the docker swarm:\n\n\n1\n2\nceph auth get-or-create client.dockerswarm osd \\\n\nallow rw\n mon \nallow r\n mds \nallow\n \n /etc/ceph/keyring.dockerswarm\n\n\n\n\n\n\nGrab the secret associated with the new user (you'll need this for the /etc/fstab entry below) by running:\n\n\n1\nceph-authtool /etc/ceph/keyring.dockerswarm -p -n client.dockerswarm\n\n\n\n\n\n\nMount MDS volume\n\n\nOn each noie, create a mountpoint for the data, by running \nmkdir /var/data\n, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually \nmounted\n if there's a network / boot delay getting access to the gluster volume:\n\n\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\nmkdir /var/data\n\nMYHOST=`hostname -s`\necho -e \n\n# Mount cephfs volume \\n\n$MYHOST:6789:/ /var/data/ ceph \\\nname=dockerswarm\\\n,secret=\nYOUR SECRET HERE\n\\\n,noatime,_netdev,context=system_u:object_r:svirt_sandbox_file_t:s0\\\n0 2\n \n /etc/fstab\nmount -a\n\n\n\n\n\n\nInstall docker-volume plugin\n\n\nUpstream bug for docker-latest reported at \nhttps://bugs.centos.org/view.php?id=13609\n\n\nAnd the alpine fault:\n\nhttps://github.com/gliderlabs/docker-alpine/issues/317\n\n\nServing\n\n\nAfter completing the above, you should have:\n\n\n1\n2\n[X] Persistent storage available to every node\n[X] Resiliency in the event of the failure of a single node\n\n\n\n\n\n\nChef's Notes\n\n\nFuture enhancements to this recipe include:\n\n\n\n\nRather than pasting a secret key into /etc/fstab (which feels wrong), I'd prefer to be able to set \"secretfile\" in /etc/fstab (which just points ceph.mount to a file containing the secret), but under the current CentOS Atomic, we're stuck with \"secret\", per \nhttps://bugzilla.redhat.com/show_bug.cgi?id=1030402", + "title": "Shared Storage (Ceph)" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#shared-storage-ceph", + "text": "While Docker Swarm is great for keeping containers running ( and restarting those that fail ), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts ( hint: you do! ), you need to provide shared storage to every docker node.", + "title": "Shared Storage (Ceph)" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#design", + "text": "", + "title": "Design" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#why-not-glusterfs", + "text": "I originally provided shared storage to my nodes using GlusterFS (see the next recipe for details), but found it difficult to deal with because: GlusterFS requires (n) \"bricks\", where (n) has to be a multiple of your replica count. I.e., if you want 2 copies of everything on shared storage (the minimum to provide redundancy), you must have either 2, 4, 6 (etc..) bricks. The HA swarm design calls for minimum of 3 nodes, and so under GlusterFS, my third node can't participate in shared storage at all, unless I start doubling up on bricks-per-node (which then impacts redundancy) GlusterFS turns out to be a giant PITA when you want to restore a failed node. There are at least 14 steps to follow to replace a brick. I'm pretty sure I messed up the 14-step process above anyway. My replaced brick synced with my \"original\" brick, but produced errors when querying status via the CLI, and hogged 100% of 1 CPU on the replaced node. Inexperienced with GlusterFS, and unable to diagnose the fault, I switched to a Ceph cluster instead.", + "title": "Why not GlusterFS?" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#why-ceph", + "text": "I'm more familiar with Ceph - I use it in the OpenStack designs I manage Replacing a failed node is easy , provided you can put up with the I/O load of rebalancing OSDs after the replacement. CentOS Atomic includes the ceph client in the OS, so while the Ceph OSD/Mon/MSD are running under containers, I can keep an eye (and later, automatically monitor) the status of Ceph from the base OS.", + "title": "Why Ceph?" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#ingredients", + "text": "Ingredients 3 x Virtual Machines (configured earlier), each with: CentOS/Fedora Atomic At least 1GB RAM At least 20GB disk space ( but it'll be tight ) Connectivity to each other within the same subnet, and on a low-latency link ( i.e., no WAN links ) A second disk dedicated to the Ceph OSD", + "title": "Ingredients" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#preparation", + "text": "", + "title": "Preparation" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#selinux", + "text": "Since our Ceph components will be containerized, we need to ensure the SELinux context on the base OS's ceph files is set correctly: 1\n2 chcon -Rt svirt_sandbox_file_t /etc/ceph\nchcon -Rt svirt_sandbox_file_t /var/lib/ceph", + "title": "SELinux" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#setup-monitors", + "text": "Pick a node, and run the following to stand up the first Ceph mon. Be sure to replace the values for MON_IP and CEPH_PUBLIC_NETWORK to those specific to your deployment: 1\n2\n3\n4\n5\n6\n7\n8 docker run -d --net=host \\\n--restart always \\\n-v /etc/ceph:/etc/ceph \\\n-v /var/lib/ceph/:/var/lib/ceph/ \\\n-e MON_IP=192.168.31.11 \\\n-e CEPH_PUBLIC_NETWORK=192.168.31.0/24 \\\n--name= ceph-mon \\\nceph/daemon mon Now copy the contents of /etc/ceph on this first node to the remaining nodes, and then run the docker command above ( customizing MON_IP as you go ) on each remaining node. You'll end up with a cluster with 3 monitors (odd number is required for quorum, same as Docker Swarm), and no OSDs (yet)", + "title": "Setup Monitors" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#setup-osds", + "text": "Since we have a OSD-less mon-only cluster currently, prepare for OSD creation by dumping the auth credentials for the OSDs into the appropriate location on the base OS: 1\n2 ceph auth get client.bootstrap-osd -o \\\n/var/lib/ceph/bootstrap-osd/ceph.keyring On each node, you need a dedicated disk for the OSD. In the example below, I used /dev/vdd (the entire disk, no partitions) for the OSD. Run the following command on every node: 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11 docker run -d --net=host \\\n--privileged=true \\\n--pid=host \\\n-v /etc/ceph:/etc/ceph \\\n-v /var/lib/ceph/:/var/lib/ceph/ \\\n-v /dev/:/dev/ \\\n-e OSD_DEVICE=/dev/vdd \\\n-e OSD_TYPE=disk \\\n--name= ceph-osd \\\n--restart=always \\\nceph/daemon osd Watch the output by running docker logs ceph-osd -f , and confirm success. Zapping the device The Ceph OSD container will refuse to destroy a partition containing existing data, so it may be necessary to \"zap\" the target disk, using: 1\n2\n3\n4 docker run -d --privileged=true \\\n-v /dev/:/dev/ \\\n-e OSD_DEVICE=/dev/sdd \\\nceph/daemon zap_device", + "title": "Setup OSDs" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#setup-mdss", + "text": "In order to mount our ceph pools as filesystems, we'll need Ceph MDS(s). Run the following on each node: 1\n2\n3\n4\n5\n6\n7\n8\n9 docker run -d --net=host \\\n--name ceph-mds \\\n--restart always \\\n-v /var/lib/ceph/:/var/lib/ceph/ \\\n-v /etc/ceph:/etc/ceph \\\n-e CEPHFS_CREATE=1 \\\n-e CEPHFS_DATA_POOL_PG=256 \\\n-e CEPHFS_METADATA_POOL_PG=256 \\\nceph/daemon mds", + "title": "Setup MDSs" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#apply-tweaks", + "text": "The ceph container seems to configure a pool default of 3 replicas (3 copies of each block are retained), which is one too many for our cluster (we are only protecting against the failure of a single node). Run the following on any node to reduce the size of the pool to 2 replicas: 1\n2 ceph osd pool set cephfs_data size 2\nceph osd pool set cephfs_metadata size 2 Disabled \"scrubbing\" (which can be IO-intensive, and is unnecessary on a VM) with: 1\n2 ceph osd set noscrub\nceph osd set nodeep-scrub", + "title": "Apply tweaks" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#create-credentials-for-swarm", + "text": "In order to mount the ceph volume onto our base host, we need to provide cephx authentication credentials. On one node, create a client for the docker swarm: 1\n2 ceph auth get-or-create client.dockerswarm osd \\ allow rw mon allow r mds allow /etc/ceph/keyring.dockerswarm Grab the secret associated with the new user (you'll need this for the /etc/fstab entry below) by running: 1 ceph-authtool /etc/ceph/keyring.dockerswarm -p -n client.dockerswarm", + "title": "Create credentials for swarm" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#mount-mds-volume", + "text": "On each noie, create a mountpoint for the data, by running mkdir /var/data , add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually mounted if there's a network / boot delay getting access to the gluster volume: 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11 mkdir /var/data\n\nMYHOST=`hostname -s`\necho -e \n# Mount cephfs volume \\n\n$MYHOST:6789:/ /var/data/ ceph \\\nname=dockerswarm\\\n,secret= YOUR SECRET HERE \\\n,noatime,_netdev,context=system_u:object_r:svirt_sandbox_file_t:s0\\\n0 2 /etc/fstab\nmount -a", + "title": "Mount MDS volume" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#install-docker-volume-plugin", + "text": "Upstream bug for docker-latest reported at https://bugs.centos.org/view.php?id=13609 And the alpine fault: https://github.com/gliderlabs/docker-alpine/issues/317", + "title": "Install docker-volume plugin" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#serving", + "text": "After completing the above, you should have: 1\n2 [X] Persistent storage available to every node\n[X] Resiliency in the event of the failure of a single node", + "title": "Serving" + }, + { + "location": "/ha-docker-swarm/shared-storage-ceph/#chefs-notes", + "text": "Future enhancements to this recipe include: Rather than pasting a secret key into /etc/fstab (which feels wrong), I'd prefer to be able to set \"secretfile\" in /etc/fstab (which just points ceph.mount to a file containing the secret), but under the current CentOS Atomic, we're stuck with \"secret\", per https://bugzilla.redhat.com/show_bug.cgi?id=1030402", + "title": "Chef's Notes" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/", + "text": "Shared Storage (GlusterFS)\n\n\nWhile Docker Swarm is great for keeping containers running (\nand restarting those that fail\n), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (\nhint: you do!\n), you need to provide shared storage to every docker node.\n\n\nDesign\n\n\nWhy GlusterFS?\n\n\nThis GlusterFS recipe was my original design for shared storage, but I \nfound it to be flawed\n, and I replaced it with a \ndesign which employs Ceph instead\n. This recipe is an alternate to the Ceph design, if you happen to prefer GlusterFS.\n\n\nIngredients\n\n\n\n\nIngredients\n\n\n3 x Virtual Machines (configured earlier), each with:\n\n\n\n\n CentOS/Fedora Atomic\n\n\n At least 1GB RAM\n\n\n At least 20GB disk space (\nbut it'll be tight\n)\n\n\n Connectivity to each other within the same subnet, and on a low-latency link (\ni.e., no WAN links\n)\n\n\n A second disk, or adequate space on the primary disk for a dedicated data partition\n\n\n\n\n\n\nPreparation\n\n\nCreate Gluster \"bricks\"\n\n\nTo build our Gluster volume, we need 2 out of the 3 VMs to provide one \"brick\". The bricks will be used to create the replicated volume. Assuming a replica count of 2 (\ni.e., 2 copies of the data are kept in gluster\n), our total number of bricks must be divisible by our replica count. (\nI.e., you can't have 3 bricks if you want 2 replicas. You can have 4 though - We have to have minimum 3 swarm manager nodes for fault-tolerance, but only 2 of those nodes need to run as gluster servers.\n)\n\n\nOn each host, run a variation following to create your bricks, adjusted for the path to your disk.\n\n\n\n\nThe example below assumes /dev/vdb is dedicated to the gluster volume\n\n\n\n\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n(\necho o # Create a new empty DOS partition table\necho n # Add a new partition\necho p # Primary partition\necho 1 # Partition number\necho # First sector (Accept default: 1)\necho # Last sector (Accept default: varies)\necho w # Write changes\n) | sudo fdisk /dev/vdb\n\nmkfs.xfs -i size=512 /dev/vdb1\nmkdir -p /var/no-direct-write-here/brick1\necho \n \n /etc/fstab \n /etc/fstab\necho \n# Mount /dev/vdb1 so that it can be used as a glusterfs volume\n \n /etc/fstab\necho \n/dev/vdb1 /var/no-direct-write-here/brick1 xfs defaults 1 2\n \n /etc/fstab\nmount -a \n mount\n\n\n\n\n\n\n\n\nDon't provision all your LVM space\n\n\nAtomic uses LVM to store docker data, and \nautomatically grows\n Docker's volumes as requried. If you commit all your free LVM space to your brick, you'll quickly find (as I did) that docker will start to fail with error messages about insufficient space. If you're going to slice off a portion of your LVM space in /dev/atomicos, make sure you leave enough space for Docker storage, where \"enough\" depends on how much you plan to pull images, make volumes, etc. I ate through 20GB very quickly doing development, so I ended up provisioning 50GB for atomic alone, with a separate volume for the brick.\n\n\n\n\nCreate glusterfs container\n\n\nAtomic doesn't include the Gluster server components. This means we'll have to run glusterd from within a container, with privileged access to the host. Although convoluted, I've come to prefer this design since it once again makes the OS \"disposable\", moving all the config into containers and code.\n\n\nRun the following on each host:\n\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\ndocker run \\\n -h glusterfs-server \\\n -v /etc/glusterfs:/etc/glusterfs:z \\\n -v /var/lib/glusterd:/var/lib/glusterd:z \\\n -v /var/log/glusterfs:/var/log/glusterfs:z \\\n -v /sys/fs/cgroup:/sys/fs/cgroup:ro \\\n -v /var/no-direct-write-here/brick1:/var/no-direct-write-here/brick1 \\\n -d --privileged=true --net=host \\\n --restart=always \\\n --name=\nglusterfs-server\n \\\n gluster/gluster-centos\n\n\n\n\n\nCreate trusted pool\n\n\nOn a single node (doesn't matter which), run \ndocker exec -it glusterfs-server bash\n to launch a shell inside the container.\n\n\nFrom the node, run\n\ngluster peer probe \nother host\n\n\nExample output:\n\n1\n2\n3\n[root@glusterfs-server /]# gluster peer probe ds1\npeer probe: success.\n[root@glusterfs-server /]#\n\n\n\n\n\nRun \ngluster peer status\n on both nodes to confirm that they're properly connected to each other:\n\n\nExample output:\n\n1\n2\n3\n4\n5\n6\n7\n[root@glusterfs-server /]# gluster peer status\nNumber of Peers: 1\n\nHostname: ds3\nUuid: 3e115ba9-6a4f-48dd-87d7-e843170ff499\nState: Peer in Cluster (Connected)\n[root@glusterfs-server /]#\n\n\n\n\n\nCreate gluster volume\n\n\nNow we create a \nreplicated volume\n out of our individual \"bricks\".\n\n\nCreate the gluster volume by running\n\n1\n2\n3\ngluster volume create gv0 replica 2 \\\n server1:/var/no-direct-write-here/brick1 \\\n server2:/var/no-direct-write-here/brick1\n\n\n\n\n\nExample output:\n\n1\n2\n3\n[root@glusterfs-server /]# gluster volume create gv0 replica 2 ds1:/var/no-direct-write-here/brick1/gv0 ds3:/var/no-direct-write-here/brick1/gv0\nvolume create: gv0: success: please start the volume to access data\n[root@glusterfs-server /]#\n\n\n\n\n\nStart the volume by running \ngluster volume start gv0\n\n\n1\n2\n3\n[root@glusterfs-server /]# gluster volume start gv0\nvolume start: gv0: success\n[root@glusterfs-server /]#\n\n\n\n\n\n\nThe volume is only present on the host you're shelled into though. To add the other hosts to the volume, run \ngluster peer probe \nservername\n. Don't probe host from itself.\n\n\nFrom one other host, run \ndocker exec -it glusterfs-server bash\n to shell into the gluster-server container, and run \ngluster peer probe \noriginal server name\n to update the name of the host which started the volume.\n\n\nMount gluster volume\n\n\nOn the host (i.e., outside of the container - type \nexit\n if you're still shelled in), create a mountpoint for the data, by running \nmkdir /var/data\n, add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually \nmounted\n if there's a network / boot delay getting access to the gluster volume:\n\n\n1\n2\n3\n4\n5\n6\nmkdir /var/data\nMYHOST=`hostname -s`\necho \n \n /etc/fstab \n /etc/fstab\necho \n# Mount glusterfs volume\n \n /etc/fstab\necho \n$MYHOST:/gv0 /var/data glusterfs defaults,_netdev,context=\nsystem_u:object_r:svirt_sandbox_file_t:s0\n 0 0\n \n /etc/fstab\nmount -a\n\n\n\n\n\n\nFor some reason, my nodes won't auto-mount this volume on boot. I even tried the trickery below, but they stubbornly refuse to automount.\n\n1\n2\n3\necho -e \n\\n\\n# Give GlusterFS 10s to start before \\\nmounting\\nsleep 10s \n mount -a\n \n /etc/rc.local\nsystemctl enable rc-local.service\n\n\n\n\n\nFor non-gluster nodes, you'll need to replace $MYHOST above with the name of one of the gluster hosts (I haven't worked out how to make this fully HA yet)\n\n\nServing\n\n\nAfter completing the above, you should have:\n\n\n1\n2\n[X] Persistent storage available to every node\n[X] Resiliency in the event of the failure of a single (gluster) node\n\n\n\n\n\n\nChef's Notes\n\n\nFuture enhancements to this recipe include:\n\n\n\n\nMigration of shared storage from GlusterFS to Ceph ()\n#2\n)\n\n\nCorrect the fact that volumes don't automount on boot (\n#3\n)", + "title": "Shared Storage (GlusterFS)" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#shared-storage-glusterfs", + "text": "While Docker Swarm is great for keeping containers running ( and restarting those that fail ), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts ( hint: you do! ), you need to provide shared storage to every docker node.", + "title": "Shared Storage (GlusterFS)" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#design", + "text": "", + "title": "Design" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#why-glusterfs", + "text": "This GlusterFS recipe was my original design for shared storage, but I found it to be flawed , and I replaced it with a design which employs Ceph instead . This recipe is an alternate to the Ceph design, if you happen to prefer GlusterFS.", + "title": "Why GlusterFS?" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#ingredients", + "text": "Ingredients 3 x Virtual Machines (configured earlier), each with: CentOS/Fedora Atomic At least 1GB RAM At least 20GB disk space ( but it'll be tight ) Connectivity to each other within the same subnet, and on a low-latency link ( i.e., no WAN links ) A second disk, or adequate space on the primary disk for a dedicated data partition", + "title": "Ingredients" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#preparation", + "text": "", + "title": "Preparation" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#create-gluster-bricks", + "text": "To build our Gluster volume, we need 2 out of the 3 VMs to provide one \"brick\". The bricks will be used to create the replicated volume. Assuming a replica count of 2 ( i.e., 2 copies of the data are kept in gluster ), our total number of bricks must be divisible by our replica count. ( I.e., you can't have 3 bricks if you want 2 replicas. You can have 4 though - We have to have minimum 3 swarm manager nodes for fault-tolerance, but only 2 of those nodes need to run as gluster servers. ) On each host, run a variation following to create your bricks, adjusted for the path to your disk. The example below assumes /dev/vdb is dedicated to the gluster volume 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16 (\necho o # Create a new empty DOS partition table\necho n # Add a new partition\necho p # Primary partition\necho 1 # Partition number\necho # First sector (Accept default: 1)\necho # Last sector (Accept default: varies)\necho w # Write changes\n) | sudo fdisk /dev/vdb\n\nmkfs.xfs -i size=512 /dev/vdb1\nmkdir -p /var/no-direct-write-here/brick1\necho /etc/fstab /etc/fstab\necho # Mount /dev/vdb1 so that it can be used as a glusterfs volume /etc/fstab\necho /dev/vdb1 /var/no-direct-write-here/brick1 xfs defaults 1 2 /etc/fstab\nmount -a mount Don't provision all your LVM space Atomic uses LVM to store docker data, and automatically grows Docker's volumes as requried. If you commit all your free LVM space to your brick, you'll quickly find (as I did) that docker will start to fail with error messages about insufficient space. If you're going to slice off a portion of your LVM space in /dev/atomicos, make sure you leave enough space for Docker storage, where \"enough\" depends on how much you plan to pull images, make volumes, etc. I ate through 20GB very quickly doing development, so I ended up provisioning 50GB for atomic alone, with a separate volume for the brick.", + "title": "Create Gluster \"bricks\"" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#create-glusterfs-container", + "text": "Atomic doesn't include the Gluster server components. This means we'll have to run glusterd from within a container, with privileged access to the host. Although convoluted, I've come to prefer this design since it once again makes the OS \"disposable\", moving all the config into containers and code. Run the following on each host: 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11 docker run \\\n -h glusterfs-server \\\n -v /etc/glusterfs:/etc/glusterfs:z \\\n -v /var/lib/glusterd:/var/lib/glusterd:z \\\n -v /var/log/glusterfs:/var/log/glusterfs:z \\\n -v /sys/fs/cgroup:/sys/fs/cgroup:ro \\\n -v /var/no-direct-write-here/brick1:/var/no-direct-write-here/brick1 \\\n -d --privileged=true --net=host \\\n --restart=always \\\n --name= glusterfs-server \\\n gluster/gluster-centos", + "title": "Create glusterfs container" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#create-trusted-pool", + "text": "On a single node (doesn't matter which), run docker exec -it glusterfs-server bash to launch a shell inside the container. From the node, run gluster peer probe other host Example output: 1\n2\n3 [root@glusterfs-server /]# gluster peer probe ds1\npeer probe: success.\n[root@glusterfs-server /]# Run gluster peer status on both nodes to confirm that they're properly connected to each other: Example output: 1\n2\n3\n4\n5\n6\n7 [root@glusterfs-server /]# gluster peer status\nNumber of Peers: 1\n\nHostname: ds3\nUuid: 3e115ba9-6a4f-48dd-87d7-e843170ff499\nState: Peer in Cluster (Connected)\n[root@glusterfs-server /]#", + "title": "Create trusted pool" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#create-gluster-volume", + "text": "Now we create a replicated volume out of our individual \"bricks\". Create the gluster volume by running 1\n2\n3 gluster volume create gv0 replica 2 \\\n server1:/var/no-direct-write-here/brick1 \\\n server2:/var/no-direct-write-here/brick1 Example output: 1\n2\n3 [root@glusterfs-server /]# gluster volume create gv0 replica 2 ds1:/var/no-direct-write-here/brick1/gv0 ds3:/var/no-direct-write-here/brick1/gv0\nvolume create: gv0: success: please start the volume to access data\n[root@glusterfs-server /]# Start the volume by running gluster volume start gv0 1\n2\n3 [root@glusterfs-server /]# gluster volume start gv0\nvolume start: gv0: success\n[root@glusterfs-server /]# The volume is only present on the host you're shelled into though. To add the other hosts to the volume, run gluster peer probe servername . Don't probe host from itself. From one other host, run docker exec -it glusterfs-server bash to shell into the gluster-server container, and run gluster peer probe original server name to update the name of the host which started the volume.", + "title": "Create gluster volume" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#mount-gluster-volume", + "text": "On the host (i.e., outside of the container - type exit if you're still shelled in), create a mountpoint for the data, by running mkdir /var/data , add an entry to fstab to ensure the volume is auto-mounted on boot, and ensure the volume is actually mounted if there's a network / boot delay getting access to the gluster volume: 1\n2\n3\n4\n5\n6 mkdir /var/data\nMYHOST=`hostname -s`\necho /etc/fstab /etc/fstab\necho # Mount glusterfs volume /etc/fstab\necho $MYHOST:/gv0 /var/data glusterfs defaults,_netdev,context= system_u:object_r:svirt_sandbox_file_t:s0 0 0 /etc/fstab\nmount -a For some reason, my nodes won't auto-mount this volume on boot. I even tried the trickery below, but they stubbornly refuse to automount. 1\n2\n3 echo -e \\n\\n# Give GlusterFS 10s to start before \\\nmounting\\nsleep 10s mount -a /etc/rc.local\nsystemctl enable rc-local.service For non-gluster nodes, you'll need to replace $MYHOST above with the name of one of the gluster hosts (I haven't worked out how to make this fully HA yet)", + "title": "Mount gluster volume" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#serving", + "text": "After completing the above, you should have: 1\n2 [X] Persistent storage available to every node\n[X] Resiliency in the event of the failure of a single (gluster) node", + "title": "Serving" + }, + { + "location": "/ha-docker-swarm/shared-storage-gluster/#chefs-notes", + "text": "Future enhancements to this recipe include: Migration of shared storage from GlusterFS to Ceph () #2 ) Correct the fact that volumes don't automount on boot ( #3 )", + "title": "Chef's Notes" + }, + { + "location": "/ha-docker-swarm/keepalived/", + "text": "Keepalived\n\n\nWhile having a self-healing, scalable docker swarm is great for availability and scalability, none of that is any good if nobody can connect to your cluster.\n\n\nIn order to provide seamless external access to clustered resources, regardless of which node they're on and tolerant of node failure, you need to present a single IP to the world for external access.\n\n\nNormally this is done using a HA loadbalancer, but since Docker Swarm aready provides the load-balancing capabilities (routing mesh), all we need for seamless HA is a virtual IP which will be provided by more than one docker node.\n\n\nThis is accomplished with the use of keepalived on at least two nodes.\n\n\nIngredients\n\n\n1\n2\n3\n4\n5\n6\nAlready deployed:\n[X] At least 2 x CentOS/Fedora Atomic VMs\n[X] low-latency link (i.e., no WAN links)\n\nNew:\n[ ] 3 x IPv4 addresses (one for each node and one for the virtual IP)\n\n\n\n\n\n\nPreparation\n\n\nEnable IPVS module\n\n\nOn all nodes which will participate in keepalived, we need the \"ip_vs\" kernel module, in order to permit serivces to bind to non-local interface addresses.\n\n\nSet this up once for both the primary and secondary nodes, by running:\n\n\n1\n2\necho \nmodprobe ip_vs\n \n /etc/rc.local\nmodprobe ip_vs\n\n\n\n\n\n\nSetup nodes\n\n\nAssuming your IPs are as follows:\n\n\n\n\n192.168.4.1 : Primary\n\n\n192.168.4.2 : Secondary\n\n\n192.168.4.3 : Virtual\n\n\n\n\nRun the following on the primary\n\n1\n2\n3\n4\n5\n6\ndocker run -d --name keepalived --restart=always \\\n --cap-add=NET_ADMIN --net=host \\\n -e KEEPALIVED_UNICAST_PEERS=\n#PYTHON2BASH:[\n192.168.4.1\n, \n192.168.4.2\n]\n \\\n -e KEEPALIVED_VIRTUAL_IPS=192.168.4.3 \\\n -e KEEPALIVED_PRIORITY=200 \\\n osixia/keepalived:1.3.5\n\n\n\n\n\nAnd on the secondary:\n\n1\n2\n3\n4\n5\n6\ndocker run -d --name keepalived --restart=always \\\n --cap-add=NET_ADMIN --net=host \\\n -e KEEPALIVED_UNICAST_PEERS=\n#PYTHON2BASH:[\n192.168.4.1\n, \n192.168.4.2\n]\n \\\n -e KEEPALIVED_VIRTUAL_IPS=192.168.4.3 \\\n -e KEEPALIVED_PRIORITY=100 \\\n osixia/keepalived:1.3.5\n\n\n\n\n\nServing\n\n\nThat's it. Each node will talk to the other via unicast (no need to un-firewall multicast addresses), and the node with the highest priority gets to be the master. When ingress traffic arrives on the master node via the VIP, docker's routing mesh will deliver it to the appropriate docker node.\n\n\nChef's notes\n\n\n\n\nSome hosting platforms (OpenStack, for one) won't allow you to simply \"claim\" a virtual IP. Each node is only able to receive traffic targetted to its unique IP. In this case, keepalived is not the right solution, and a platform-specific load-balancing solution should be used. In OpenStack, this is Neutron's \"Load Balancer As A Service\" (LBAAS) component. AWS and Azure would likely include similar protections.\n\n\nMore than 2 nodes can participate in keepalived. Simply ensure that each node has the appropriate priority set, and the node with the highest priority will become the master.", + "title": "Keepalived" + }, + { + "location": "/ha-docker-swarm/keepalived/#keepalived", + "text": "While having a self-healing, scalable docker swarm is great for availability and scalability, none of that is any good if nobody can connect to your cluster. In order to provide seamless external access to clustered resources, regardless of which node they're on and tolerant of node failure, you need to present a single IP to the world for external access. Normally this is done using a HA loadbalancer, but since Docker Swarm aready provides the load-balancing capabilities (routing mesh), all we need for seamless HA is a virtual IP which will be provided by more than one docker node. This is accomplished with the use of keepalived on at least two nodes.", + "title": "Keepalived" + }, + { + "location": "/ha-docker-swarm/keepalived/#ingredients", + "text": "1\n2\n3\n4\n5\n6 Already deployed:\n[X] At least 2 x CentOS/Fedora Atomic VMs\n[X] low-latency link (i.e., no WAN links)\n\nNew:\n[ ] 3 x IPv4 addresses (one for each node and one for the virtual IP)", + "title": "Ingredients" + }, + { + "location": "/ha-docker-swarm/keepalived/#preparation", + "text": "", + "title": "Preparation" + }, + { + "location": "/ha-docker-swarm/keepalived/#enable-ipvs-module", + "text": "On all nodes which will participate in keepalived, we need the \"ip_vs\" kernel module, in order to permit serivces to bind to non-local interface addresses. Set this up once for both the primary and secondary nodes, by running: 1\n2 echo modprobe ip_vs /etc/rc.local\nmodprobe ip_vs", + "title": "Enable IPVS module" + }, + { + "location": "/ha-docker-swarm/keepalived/#setup-nodes", + "text": "Assuming your IPs are as follows: 192.168.4.1 : Primary 192.168.4.2 : Secondary 192.168.4.3 : Virtual Run the following on the primary 1\n2\n3\n4\n5\n6 docker run -d --name keepalived --restart=always \\\n --cap-add=NET_ADMIN --net=host \\\n -e KEEPALIVED_UNICAST_PEERS= #PYTHON2BASH:[ 192.168.4.1 , 192.168.4.2 ] \\\n -e KEEPALIVED_VIRTUAL_IPS=192.168.4.3 \\\n -e KEEPALIVED_PRIORITY=200 \\\n osixia/keepalived:1.3.5 And on the secondary: 1\n2\n3\n4\n5\n6 docker run -d --name keepalived --restart=always \\\n --cap-add=NET_ADMIN --net=host \\\n -e KEEPALIVED_UNICAST_PEERS= #PYTHON2BASH:[ 192.168.4.1 , 192.168.4.2 ] \\\n -e KEEPALIVED_VIRTUAL_IPS=192.168.4.3 \\\n -e KEEPALIVED_PRIORITY=100 \\\n osixia/keepalived:1.3.5", + "title": "Setup nodes" + }, + { + "location": "/ha-docker-swarm/keepalived/#serving", + "text": "That's it. Each node will talk to the other via unicast (no need to un-firewall multicast addresses), and the node with the highest priority gets to be the master. When ingress traffic arrives on the master node via the VIP, docker's routing mesh will deliver it to the appropriate docker node.", + "title": "Serving" + }, + { + "location": "/ha-docker-swarm/keepalived/#chefs-notes", + "text": "Some hosting platforms (OpenStack, for one) won't allow you to simply \"claim\" a virtual IP. Each node is only able to receive traffic targetted to its unique IP. In this case, keepalived is not the right solution, and a platform-specific load-balancing solution should be used. In OpenStack, this is Neutron's \"Load Balancer As A Service\" (LBAAS) component. AWS and Azure would likely include similar protections. More than 2 nodes can participate in keepalived. Simply ensure that each node has the appropriate priority set, and the node with the highest priority will become the master.", + "title": "Chef's notes" + }, + { + "location": "/ha-docker-swarm/docker-swarm-mode/", + "text": "Docker Swarm Mode\n\n\nFor truly highly-available services with Docker containers, we need an orchestration system. Docker Swarm (as defined at 1.13) is the simplest way to achieve redundancy, such that a single docker host could be turned off, and none of our services will be interrupted.\n\n\nIngredients\n\n\n\n\n3 x CentOS Atomic hosts (bare-metal or VMs). A reasonable minimum would be:\n\n\n1 x vCPU\n\n\n1GB repo_name\n\n\n10GB HDD\n\n\nHosts must be within the same subnet, and connected on a low-latency link (i.e., no WAN links)\n\n\n\n\nPreparation\n\n\nRelease the swarm!\n\n\nNow, to launch my swarm:\n\n\ndocker swarm init\n\n\nYeah, that was it. Now I have a 1-node swarm.\n\n\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n[root@ds1 ~]# docker swarm init\nSwarm initialized: current node (b54vls3wf8xztwfz79nlkivt8) is now a manager.\n\nTo add a worker to this swarm, run the following command:\n\n docker swarm join \\\n --token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-bsud7xnvhv4cicwi7l6c9s6l0 \\\n 202.170.164.47:2377\n\nTo add a manager to this swarm, run \ndocker swarm join-token manager\n and follow the instructions.\n\n[root@ds1 ~]#\n\n\n\n\n\n\nRun \ndocker node ls\n to confirm that I have a 1-node swarm:\n\n\n1\n2\n3\n4\n[root@ds1 ~]# docker node ls\nID HOSTNAME STATUS AVAILABILITY MANAGER STATUS\nb54vls3wf8xztwfz79nlkivt8 * ds1.funkypenguin.co.nz Ready Active Leader\n[root@ds1 ~]#\n\n\n\n\n\n\nNote that when I ran \ndocker swarm init\n above, the CLI output gave me a command to run to join further nodes to my swarm. This would join the nodes as \nworkers\n (as opposed to \nmanagers\n). Workers can easily be promoted to managers (and demoted again), but since we know that we want our other two nodes to be managers too, it's simpler just to add them to the swarm as managers immediately.\n\n\nOn the first swarm node, generate the necessary token to join another manager by running \ndocker swarm join-token manager\n:\n\n\n1\n2\n3\n4\n5\n6\n7\n8\n[root@ds1 ~]# docker swarm join-token manager\nTo add a manager to this swarm, run the following command:\n\n docker swarm join \\\n --token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-cfm24bq2zvfkcwujwlp5zqxta \\\n 202.170.164.47:2377\n\n[root@ds1 ~]#\n\n\n\n\n\n\nRun the command provided on your second node to join it to the swarm as a manager. After adding the second node, the output of \ndocker node ls\n (on either host) should reflect two nodes:\n\n\n1\n2\n3\n4\n5\n[root@ds2 davidy]# docker node ls\nID HOSTNAME STATUS AVAILABILITY MANAGER STATUS\nb54vls3wf8xztwfz79nlkivt8 ds1.funkypenguin.co.nz Ready Active Leader\nxmw49jt5a1j87a6ihul76gbgy * ds2.funkypenguin.co.nz Ready Active Reachable\n[root@ds2 davidy]#\n\n\n\n\n\n\nRepeat the process to add your third node. \nYou need a new token for the third node, don't re-use the manager token you generated for the second node\n.\n\n\n\n\nSeriously. Don't use a token more than once, else it's swarm-rebuilding time.\n\n\n\n\nFinally, \ndocker node ls\n should reflect that you have 3 reachable manager nodes, one of whom is the \"Leader\":\n\n\n1\n2\n3\n4\n5\n6\n[root@ds3 ~]# docker node ls\nID HOSTNAME STATUS AVAILABILITY MANAGER STATUS\n36b4twca7i3hkb7qr77i0pr9i ds1.openstack.dev.safenz.net Ready Active Reachable\nl14rfzazbmibh1p9wcoivkv1s * ds3.openstack.dev.safenz.net Ready Active Reachable\ntfsgxmu7q23nuo51wwa4ycpsj ds2.openstack.dev.safenz.net Ready Active Leader\n[root@ds3 ~]#\n\n\n\n\n\n\nCreate registry mirror\n\n\nAlthough we now have shared storage for our persistent container data, our docker nodes don't share any other docker data, such as container images. This results in an inefficiency - every node which participates in the swarm will, at some point, need the docker image for every container deployed in the swarm.\n\n\nWhen dealing with large container (looking at you, GitLab!), this can result in several gigabytes of wasted bandwidth per-node, and long delays when restarting containers on an alternate node. (\nIt also wastes disk space on each node, but we'll get to that in the next section\n)\n\n\nThe solution is to run an official Docker registry container as a \n\"pull-through\" cache, or \"registry mirror\"\n. By using our persistent storage for the registry cache, we can ensure we have a single copy of all the containers we've pulled at least once. After the first pull, any subsequent pulls from our nodes will use the cached version from our registry mirror. As a result, services are available more quickly when restarting container nodes, and we can be more aggressive about cleaning up unused containers on our nodes (more later)\n\n\nThe registry mirror runs as a swarm stack, using a simple docker-compose.yml. Customize \nyour mirror FQDN\n below, so that Traefik will generate the appropriate LetsEncrypt certificates for it, and make it available via HTTPS.\n\n\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\nversion: \n3\n\n\nservices:\n\n registry-mirror:\n image: registry:2\n networks:\n - traefik\n deploy:\n labels:\n - traefik.frontend.rule=Host:\nyour mirror FQDN\n\n - traefik.docker.network=traefik\n - traefik.port=5000\n ports:\n - 5000:5000\n volumes:\n - /var/data/registry/registry-mirror-data:/var/lib/registry\n - /var/data/registry/registry-mirror-config.yml:/etc/docker/registry/config.yml\n\nnetworks:\n traefik:\n external: true\n\n\n\n\n\n\n\n\nUnencrypted registry\n\n\nWe create this registry without consideration for SSL, which will fail if we attempt to use the registry directly. However, we're going to use the HTTPS-proxied version via Traefik, leveraging Traefik to manage the LetsEncrypt certificates required.\n\n\n\n\nCreate registry/registry-mirror-config.yml as follows:\n\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\nversion\n:\n \n0.1\n\n\nlog\n:\n\n \nfields\n:\n\n \nservice\n:\n \nregistry\n\n\nstorage\n:\n\n \ncache\n:\n\n \nblobdescriptor\n:\n \ninmemory\n\n \nfilesystem\n:\n\n \nrootdirectory\n:\n \n/var/lib/\nregistry\n\n \ndelete\n:\n\n \nenabled\n:\n \ntrue\n\n\nhttp\n:\n\n \naddr\n:\n \n:\n5000\n\n \nheaders\n:\n\n \nX\n-\nContent\n-\nType\n-\nOptions\n:\n \n[\nnosniff\n]\n\n\nhealth\n:\n\n \nstoragedriver\n:\n\n \nenabled\n:\n \ntrue\n\n \ninterval\n:\n \n10\ns\n\n \nthreshold\n:\n \n3\n\n\nproxy\n:\n\n \nremoteurl\n:\n \nhttps\n://\nregistry\n-\n1\n.\ndocker\n.\nio\n\n\n\n\n\n\nEnable registry mirror and experimental features\n\n\nTo tell docker to use the registry mirror, and in order to be able to watch the logs of any service from any manager node (\nan experimental feature in the current Atomic docker build\n), edit \n/etc/docker-latest/daemon.json\n on each node, and change from:\n\n\n1\n2\n3\n4\n{\n \nlog-driver\n: \njournald\n,\n \nsignature-verification\n: false\n}\n\n\n\n\n\n\nTo:\n\n\n1\n2\n3\n4\n5\n6\n{\n \nlog-driver\n: \njournald\n,\n \nsignature-verification\n: false,\n \nexperimental\n: true,\n \nregistry-mirrors\n: [\nhttps://\nyour registry mirror FQDN\n]\n}\n\n\n\n\n\n\n\n\nNote the extra comma required after \"false\" above\n\n\n\n\nSetup automated cleanup\n\n\nThis needs to be a docker-compose.yml file, excluding trusted images (like glusterfs, traefik, etc)\n\n1\n2\n3\n4\ndocker run -d \\\n-v /var/run/docker.sock:/var/run/docker.sock:rw \\\n-v /var/lib/docker:/var/lib/docker:rw \\\nmeltwater/docker-cleanup:latest\n\n\n\n\n\nTweaks\n\n\nAdd some handy bash auto-completion for docker. Without this, you'll get annoyed that you can't autocomplete \ndocker stack deploy \nblah\n -c \nblah.yml\n commands.\n\n\n1\n2\ncd /etc/bash_completion.d/\ncurl -O https://raw.githubusercontent.com/docker/cli/b75596e1e4d5295ac69b9934d1bd8aff691a0de8/contrib/completion/bash/docker\n\n\n\n\n\n\nInstall some useful bash aliases on each host\n\n1\n2\n3\ncd ~\ncurl -O https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook-recipies/raw/master/bash/gcb-aliases.sh\necho \nsource ~/gcb-aliases.sh\n \n ~/.bash_profile\n\n\n\n\n\n1\n2\n3\n4\n5\nmkdir ~/dockersock\ncd ~/dockersock\ncurl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/Makefile\ncurl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/dockersock.te\nmake \n semodule -i dockersock.pp\n\n\n\n\n\n\nSetup registry\n\n\ndocker run -d \\\n -p 5000:5000 \\\n --restart=always \\\n --name registry \\\n -v /mnt/registry:/var/lib/registry \\\n registry:2\n\n\n{\n\"log-driver\": \"journald\",\n\"signature-verification\": false,\n\"experimental\": true,\n\"registry-mirrors\": [\"\nhttps://registry-mirror.funkypenguin.co.nz\n\"]\n}\n\n\nregistry-mirror:\n image: registry:2\n ports:\n - 5000:5000\n environment:\n volumes:\n - /var/data/registry:/var/lib/registry\n\n\n1\n2\n3\n4\n5\n6\n7\n8\n [root@ds1 dockersock]# docker swarm join-token manager\n To add a manager to this swarm, run the following command:\n\n docker swarm join \\\n --token SWMTKN-1-09c94wv0opw0y6xg67uzjl13pnv8lxxn586hrg5f47spso9l6j-6zn3dxk7c4zkb19r61owasi15 \\\n 192.168.31.11:2377\n\n [root@ds1 dockersock]#", + "title": "Docker Swarm Mode" + }, + { + "location": "/ha-docker-swarm/docker-swarm-mode/#docker-swarm-mode", + "text": "For truly highly-available services with Docker containers, we need an orchestration system. Docker Swarm (as defined at 1.13) is the simplest way to achieve redundancy, such that a single docker host could be turned off, and none of our services will be interrupted.", + "title": "Docker Swarm Mode" + }, + { + "location": "/ha-docker-swarm/docker-swarm-mode/#ingredients", + "text": "3 x CentOS Atomic hosts (bare-metal or VMs). A reasonable minimum would be: 1 x vCPU 1GB repo_name 10GB HDD Hosts must be within the same subnet, and connected on a low-latency link (i.e., no WAN links)", + "title": "Ingredients" + }, + { + "location": "/ha-docker-swarm/docker-swarm-mode/#preparation", + "text": "", + "title": "Preparation" + }, + { + "location": "/ha-docker-swarm/docker-swarm-mode/#release-the-swarm", + "text": "Now, to launch my swarm: docker swarm init Yeah, that was it. Now I have a 1-node swarm. 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12 [root@ds1 ~]# docker swarm init\nSwarm initialized: current node (b54vls3wf8xztwfz79nlkivt8) is now a manager.\n\nTo add a worker to this swarm, run the following command:\n\n docker swarm join \\\n --token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-bsud7xnvhv4cicwi7l6c9s6l0 \\\n 202.170.164.47:2377\n\nTo add a manager to this swarm, run docker swarm join-token manager and follow the instructions.\n\n[root@ds1 ~]# Run docker node ls to confirm that I have a 1-node swarm: 1\n2\n3\n4 [root@ds1 ~]# docker node ls\nID HOSTNAME STATUS AVAILABILITY MANAGER STATUS\nb54vls3wf8xztwfz79nlkivt8 * ds1.funkypenguin.co.nz Ready Active Leader\n[root@ds1 ~]# Note that when I ran docker swarm init above, the CLI output gave me a command to run to join further nodes to my swarm. This would join the nodes as workers (as opposed to managers ). Workers can easily be promoted to managers (and demoted again), but since we know that we want our other two nodes to be managers too, it's simpler just to add them to the swarm as managers immediately. On the first swarm node, generate the necessary token to join another manager by running docker swarm join-token manager : 1\n2\n3\n4\n5\n6\n7\n8 [root@ds1 ~]# docker swarm join-token manager\nTo add a manager to this swarm, run the following command:\n\n docker swarm join \\\n --token SWMTKN-1-2orjbzjzjvm1bbo736xxmxzwaf4rffxwi0tu3zopal4xk4mja0-cfm24bq2zvfkcwujwlp5zqxta \\\n 202.170.164.47:2377\n\n[root@ds1 ~]# Run the command provided on your second node to join it to the swarm as a manager. After adding the second node, the output of docker node ls (on either host) should reflect two nodes: 1\n2\n3\n4\n5 [root@ds2 davidy]# docker node ls\nID HOSTNAME STATUS AVAILABILITY MANAGER STATUS\nb54vls3wf8xztwfz79nlkivt8 ds1.funkypenguin.co.nz Ready Active Leader\nxmw49jt5a1j87a6ihul76gbgy * ds2.funkypenguin.co.nz Ready Active Reachable\n[root@ds2 davidy]# Repeat the process to add your third node. You need a new token for the third node, don't re-use the manager token you generated for the second node . Seriously. Don't use a token more than once, else it's swarm-rebuilding time. Finally, docker node ls should reflect that you have 3 reachable manager nodes, one of whom is the \"Leader\": 1\n2\n3\n4\n5\n6 [root@ds3 ~]# docker node ls\nID HOSTNAME STATUS AVAILABILITY MANAGER STATUS\n36b4twca7i3hkb7qr77i0pr9i ds1.openstack.dev.safenz.net Ready Active Reachable\nl14rfzazbmibh1p9wcoivkv1s * ds3.openstack.dev.safenz.net Ready Active Reachable\ntfsgxmu7q23nuo51wwa4ycpsj ds2.openstack.dev.safenz.net Ready Active Leader\n[root@ds3 ~]#", + "title": "Release the swarm!" + }, + { + "location": "/ha-docker-swarm/docker-swarm-mode/#create-registry-mirror", + "text": "Although we now have shared storage for our persistent container data, our docker nodes don't share any other docker data, such as container images. This results in an inefficiency - every node which participates in the swarm will, at some point, need the docker image for every container deployed in the swarm. When dealing with large container (looking at you, GitLab!), this can result in several gigabytes of wasted bandwidth per-node, and long delays when restarting containers on an alternate node. ( It also wastes disk space on each node, but we'll get to that in the next section ) The solution is to run an official Docker registry container as a \"pull-through\" cache, or \"registry mirror\" . By using our persistent storage for the registry cache, we can ensure we have a single copy of all the containers we've pulled at least once. After the first pull, any subsequent pulls from our nodes will use the cached version from our registry mirror. As a result, services are available more quickly when restarting container nodes, and we can be more aggressive about cleaning up unused containers on our nodes (more later) The registry mirror runs as a swarm stack, using a simple docker-compose.yml. Customize your mirror FQDN below, so that Traefik will generate the appropriate LetsEncrypt certificates for it, and make it available via HTTPS. 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22 version: 3 \n\nservices:\n\n registry-mirror:\n image: registry:2\n networks:\n - traefik\n deploy:\n labels:\n - traefik.frontend.rule=Host: your mirror FQDN \n - traefik.docker.network=traefik\n - traefik.port=5000\n ports:\n - 5000:5000\n volumes:\n - /var/data/registry/registry-mirror-data:/var/lib/registry\n - /var/data/registry/registry-mirror-config.yml:/etc/docker/registry/config.yml\n\nnetworks:\n traefik:\n external: true Unencrypted registry We create this registry without consideration for SSL, which will fail if we attempt to use the registry directly. However, we're going to use the HTTPS-proxied version via Traefik, leveraging Traefik to manage the LetsEncrypt certificates required. Create registry/registry-mirror-config.yml as follows: 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22 version : 0.1 log : \n fields : \n service : registry storage : \n cache : \n blobdescriptor : inmemory \n filesystem : \n rootdirectory : /var/lib/ registry \n delete : \n enabled : true http : \n addr : : 5000 \n headers : \n X - Content - Type - Options : [ nosniff ] health : \n storagedriver : \n enabled : true \n interval : 10 s \n threshold : 3 proxy : \n remoteurl : https :// registry - 1 . docker . io", + "title": "Create registry mirror" + }, + { + "location": "/ha-docker-swarm/docker-swarm-mode/#enable-registry-mirror-and-experimental-features", + "text": "To tell docker to use the registry mirror, and in order to be able to watch the logs of any service from any manager node ( an experimental feature in the current Atomic docker build ), edit /etc/docker-latest/daemon.json on each node, and change from: 1\n2\n3\n4 {\n log-driver : journald ,\n signature-verification : false\n} To: 1\n2\n3\n4\n5\n6 {\n log-driver : journald ,\n signature-verification : false,\n experimental : true,\n registry-mirrors : [ https:// your registry mirror FQDN ]\n} Note the extra comma required after \"false\" above", + "title": "Enable registry mirror and experimental features" + }, + { + "location": "/ha-docker-swarm/docker-swarm-mode/#setup-automated-cleanup", + "text": "This needs to be a docker-compose.yml file, excluding trusted images (like glusterfs, traefik, etc) 1\n2\n3\n4 docker run -d \\\n-v /var/run/docker.sock:/var/run/docker.sock:rw \\\n-v /var/lib/docker:/var/lib/docker:rw \\\nmeltwater/docker-cleanup:latest", + "title": "Setup automated cleanup" + }, + { + "location": "/ha-docker-swarm/docker-swarm-mode/#tweaks", + "text": "Add some handy bash auto-completion for docker. Without this, you'll get annoyed that you can't autocomplete docker stack deploy blah -c blah.yml commands. 1\n2 cd /etc/bash_completion.d/\ncurl -O https://raw.githubusercontent.com/docker/cli/b75596e1e4d5295ac69b9934d1bd8aff691a0de8/contrib/completion/bash/docker Install some useful bash aliases on each host 1\n2\n3 cd ~\ncurl -O https://gitlab.funkypenguin.co.nz/funkypenguin/geeks-cookbook-recipies/raw/master/bash/gcb-aliases.sh\necho source ~/gcb-aliases.sh ~/.bash_profile 1\n2\n3\n4\n5 mkdir ~/dockersock\ncd ~/dockersock\ncurl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/Makefile\ncurl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/dockersock.te\nmake semodule -i dockersock.pp", + "title": "Tweaks" + }, + { + "location": "/ha-docker-swarm/docker-swarm-mode/#setup-registry", + "text": "docker run -d \\\n -p 5000:5000 \\\n --restart=always \\\n --name registry \\\n -v /mnt/registry:/var/lib/registry \\\n registry:2 {\n\"log-driver\": \"journald\",\n\"signature-verification\": false,\n\"experimental\": true,\n\"registry-mirrors\": [\" https://registry-mirror.funkypenguin.co.nz \"]\n} registry-mirror:\n image: registry:2\n ports:\n - 5000:5000\n environment:\n volumes:\n - /var/data/registry:/var/lib/registry 1\n2\n3\n4\n5\n6\n7\n8 [root@ds1 dockersock]# docker swarm join-token manager\n To add a manager to this swarm, run the following command:\n\n docker swarm join \\\n --token SWMTKN-1-09c94wv0opw0y6xg67uzjl13pnv8lxxn586hrg5f47spso9l6j-6zn3dxk7c4zkb19r61owasi15 \\\n 192.168.31.11:2377\n\n [root@ds1 dockersock]#", + "title": "Setup registry" + }, + { + "location": "/ha-docker-swarm/traefik/", + "text": "Traefik\n\n\nThe platforms we plan to run on our cloud are generally web-based, and each listening on their own unique TCP port. When a container in a swarm exposes a port, then connecting to \nany\n swarm member on that port will result in your request being forwarded to the appropriate host running the container. (\nDocker calls this the swarm \"\nrouting mesh\n\"\n)\n\n\nSo we get a rudimentary load balancer built into swarm. We could stop there, just exposing a series of ports on our hosts, and making them HA using keepalived.\n\n\nThere are some gaps to this approach though:\n\n\n\n\nNo consideration is given to HTTPS. Implementation would have to be done manually, per-container.\n\n\nNo mechanism is provided for authentication outside of that which the container providers. We may not \nwant\n to expose every interface on every container to the world, especially if we are playing with tools or containers whose quality and origin are unknown.\n\n\n\n\nTo deal with these gaps, we need a front-end load-balancer, and in this design, that role is provided by \nTraefik\n.\n\n\nIngredients\n\n\nPreparation\n\n\nPrepare the host\n\n\nThe traefik container is aware of the \nother\n docker containers in the swarm, because it has access to the docker socket at \n/var/run/docker.sock\n. This allows traefik to dynamically configure itself based on the labels found on containers in the swarm, which is hugely useful. To make this functionality work on our SELinux-enabled Atomic hosts, we need to add custom SELinux policy.\n\n\nRun the following to build and activate policy to permit containers to access docker.sock:\n\n\n1\n2\n3\n4\n5\n6\n7\nmkdir ~/dockersock\ncd ~/dockersock\ncurl -O https://raw.githubusercontent.com/dpw/\\\nselinux-dockersock/master/Makefile\ncurl -O https://raw.githubusercontent.com/dpw/\\\nselinux-dockersock/master/dockersock.te\nmake \n semodule -i dockersock.pp\n\n\n\n\n\n\nPrepare traefik.toml\n\n\nWhile it's possible to configure traefik via docker command arguments, I prefer to create a config file (traefik.toml). This allows me to change traefik's behaviour by simply changing the file, and keeps my docker config simple.\n\n\nCreate /var/data/traefik/traefik.toml as follows:\n\n\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\ncheckNewVersion = true\ndefaultEntryPoints = [\nhttp\n, \nhttps\n]\n\n# This section enable LetsEncrypt automatic certificate generation / renewal\n[acme]\nemail = \nyour LetsEncrypt email address\n\nstorage = \nacme.json\n # or \ntraefik/acme/account\n if using KV store\nentryPoint = \nhttps\n\nacmeLogging = true\nonDemand = true\nOnHostRule = true\n\n[[acme.domains]]\n main = \nyour primary domain\n\n\n# Redirect all HTTP to HTTPS (why wouldn\nt you?)\n[entryPoints]\n [entryPoints.http]\n address = \n:80\n\n [entryPoints.http.redirect]\n entryPoint = \nhttps\n\n [entryPoints.https]\n address = \n:443\n\n [entryPoints.https.tls]\n\n[web]\naddress = \n:8080\n\nwatch = true\n\n[docker]\nendpoint = \ntcp://127.0.0.1:2375\n\ndomain = \nyour primary domain\n\nwatch = true\nswarmmode = true\n\n\n\n\n\n\nPrepare the docker service config\n\n\nCreate /var/data/traefik/docker-compose.yml as follows:\n\n\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\nversion: \n3.2\n\n\nservices:\n traefik:\n image: traefik\n command: --web --docker --docker.swarmmode --docker.watch --docker.domain=funkypenguin.co.nz --logLevel=DEBUG\n ports:\n - target: 80\n published: 80\n protocol: tcp\n mode: host\n - target: 443\n published: 443\n protocol: tcp\n mode: host\n - target: 8080\n published: 8080\n protocol: tcp\n volumes:\n - /var/run/docker.sock:/var/run/docker.sock:ro\n - /var/data/traefik/traefik.toml:/traefik.toml:ro\n - /var/data/traefik/acme.json:/acme.json\n labels:\n - \ntraefik.enable=false\n\n networks:\n - public\n deploy:\n mode: global\n placement:\n constraints: [node.role == manager]\n restart_policy:\n condition: on-failure\n\nnetworks:\n public:\n driver: overlay\n ipam:\n driver: default\n config:\n - subnet: 10.1.0.0/24\n\n\n\n\n\n\nDocker won't start an image with a bind-mount to a non-existent file, so prepare acme.json by running \ntouch /var/data/traefik/acme.json\n.\n\n\nLaunch\n\n\nDeploy traefik with \ndocker stack deploy traefik -c /var/data/traefik/docker-compose.yml\n\n\nConfirm traefik is running with \ndocker stack ps traefik\n\n\nServing\n\n\nYou now have:\n\n\n\n\nFrontend proxy which will dynamically configure itself for new backend containers\n\n\nAutomatic SSL support for all proxied resources\n\n\n\n\nChef's Notes\n\n\nAdditional features I'd like to see in this recipe are:\n\n\n\n\nInclude documentation of oauth2_proxy container for protecting individual backends\n\n\nTraefik webUI is available via HTTPS, protected with oauth_proxy\n\n\nPending a feature in docker-swarm to avoid NAT on routing-mesh-delivered traffic, update the design", + "title": "Traefik" + }, + { + "location": "/ha-docker-swarm/traefik/#traefik", + "text": "The platforms we plan to run on our cloud are generally web-based, and each listening on their own unique TCP port. When a container in a swarm exposes a port, then connecting to any swarm member on that port will result in your request being forwarded to the appropriate host running the container. ( Docker calls this the swarm \" routing mesh \" ) So we get a rudimentary load balancer built into swarm. We could stop there, just exposing a series of ports on our hosts, and making them HA using keepalived. There are some gaps to this approach though: No consideration is given to HTTPS. Implementation would have to be done manually, per-container. No mechanism is provided for authentication outside of that which the container providers. We may not want to expose every interface on every container to the world, especially if we are playing with tools or containers whose quality and origin are unknown. To deal with these gaps, we need a front-end load-balancer, and in this design, that role is provided by Traefik .", + "title": "Traefik" + }, + { + "location": "/ha-docker-swarm/traefik/#ingredients", + "text": "", + "title": "Ingredients" + }, + { + "location": "/ha-docker-swarm/traefik/#preparation", + "text": "", + "title": "Preparation" + }, + { + "location": "/ha-docker-swarm/traefik/#prepare-the-host", + "text": "The traefik container is aware of the other docker containers in the swarm, because it has access to the docker socket at /var/run/docker.sock . This allows traefik to dynamically configure itself based on the labels found on containers in the swarm, which is hugely useful. To make this functionality work on our SELinux-enabled Atomic hosts, we need to add custom SELinux policy. Run the following to build and activate policy to permit containers to access docker.sock: 1\n2\n3\n4\n5\n6\n7 mkdir ~/dockersock\ncd ~/dockersock\ncurl -O https://raw.githubusercontent.com/dpw/\\\nselinux-dockersock/master/Makefile\ncurl -O https://raw.githubusercontent.com/dpw/\\\nselinux-dockersock/master/dockersock.te\nmake semodule -i dockersock.pp", + "title": "Prepare the host" + }, + { + "location": "/ha-docker-swarm/traefik/#prepare-traefiktoml", + "text": "While it's possible to configure traefik via docker command arguments, I prefer to create a config file (traefik.toml). This allows me to change traefik's behaviour by simply changing the file, and keeps my docker config simple. Create /var/data/traefik/traefik.toml as follows: 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34 checkNewVersion = true\ndefaultEntryPoints = [ http , https ]\n\n# This section enable LetsEncrypt automatic certificate generation / renewal\n[acme]\nemail = your LetsEncrypt email address \nstorage = acme.json # or traefik/acme/account if using KV store\nentryPoint = https \nacmeLogging = true\nonDemand = true\nOnHostRule = true\n\n[[acme.domains]]\n main = your primary domain \n\n# Redirect all HTTP to HTTPS (why wouldn t you?)\n[entryPoints]\n [entryPoints.http]\n address = :80 \n [entryPoints.http.redirect]\n entryPoint = https \n [entryPoints.https]\n address = :443 \n [entryPoints.https.tls]\n\n[web]\naddress = :8080 \nwatch = true\n\n[docker]\nendpoint = tcp://127.0.0.1:2375 \ndomain = your primary domain \nwatch = true\nswarmmode = true", + "title": "Prepare traefik.toml" + }, + { + "location": "/ha-docker-swarm/traefik/#prepare-the-docker-service-config", + "text": "Create /var/data/traefik/docker-compose.yml as follows: 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40 version: 3.2 \n\nservices:\n traefik:\n image: traefik\n command: --web --docker --docker.swarmmode --docker.watch --docker.domain=funkypenguin.co.nz --logLevel=DEBUG\n ports:\n - target: 80\n published: 80\n protocol: tcp\n mode: host\n - target: 443\n published: 443\n protocol: tcp\n mode: host\n - target: 8080\n published: 8080\n protocol: tcp\n volumes:\n - /var/run/docker.sock:/var/run/docker.sock:ro\n - /var/data/traefik/traefik.toml:/traefik.toml:ro\n - /var/data/traefik/acme.json:/acme.json\n labels:\n - traefik.enable=false \n networks:\n - public\n deploy:\n mode: global\n placement:\n constraints: [node.role == manager]\n restart_policy:\n condition: on-failure\n\nnetworks:\n public:\n driver: overlay\n ipam:\n driver: default\n config:\n - subnet: 10.1.0.0/24 Docker won't start an image with a bind-mount to a non-existent file, so prepare acme.json by running touch /var/data/traefik/acme.json .", + "title": "Prepare the docker service config" + }, + { + "location": "/ha-docker-swarm/traefik/#launch", + "text": "Deploy traefik with docker stack deploy traefik -c /var/data/traefik/docker-compose.yml Confirm traefik is running with docker stack ps traefik", + "title": "Launch" + }, + { + "location": "/ha-docker-swarm/traefik/#serving", + "text": "You now have: Frontend proxy which will dynamically configure itself for new backend containers Automatic SSL support for all proxied resources", + "title": "Serving" + }, + { + "location": "/ha-docker-swarm/traefik/#chefs-notes", + "text": "Additional features I'd like to see in this recipe are: Include documentation of oauth2_proxy container for protecting individual backends Traefik webUI is available via HTTPS, protected with oauth_proxy Pending a feature in docker-swarm to avoid NAT on routing-mesh-delivered traffic, update the design", + "title": "Chef's Notes" + }, + { + "location": "/recipies/mail/", + "text": "Mail Server\n\n\nMany of the recipies that follow require email access of some kind. It's quite accepmatebl normally possible to use a hosted service such as SendGrid, or just a gmail account. If (like me) you'd like to self-host email for your stacks, then the following recipe provides a full-stack mail server running on the docker HA swarm.\n\n\nOf value to me in choosing docker-mailserver were:\n\n\n\n\nAutomatically renews LetsEncrypt certificates\n\n\nCreation of email accounts across multiple domains (i.e., the same container gives me mailbox \n, and \n)\n\n\nThe entire configuration is based on flat files, so there's no database or persistence to worry about\n\n\n\n\ndocker-mailserver doesn't include a webmail client, and one is not strictly needed. Rainloop can be added either as another service within the stack, or as a standalone service. Rainloop will be covered in a future recipe.\n\n\nIngredients\n\n\n\n\nDocker swarm cluster\n with \npersistent shared storage\n\n\nTraefik\n configured per design\n\n\nLetsEncrypt authorized email address for domain\n\n\nAccess to manage DNS records for domains\n\n\n\n\nPreparation\n\n\nSetup data locations\n\n\nWe'll need several directories to bind-mount into our container, so create them in /var/data/mailserver:\n\n\n1\n2\n3\n4\ncd /var/data\nmkdir mailserver\ncd mailserver\nmkdir {maildata,mailstate,config,letsencrypt}\n\n\n\n\n\n\nGet LetsEncrypt certificate\n\n\nDecide on the FQDN to assign to your mailserver. You can service multiple domains from a single mailserver - i.e., \n and \n can both be served by \nmail.example.com\n.\n\n\nThe docker-mailserver container can \nrenew\n our LetsEncrypt certs for us, but it can't generate them. To do this, we need to run certbot (from a container) to request the initial certs and create the appropriate directory structure.\n\n\nIn the example below, since I'm already using Traefik to manage the LE certs for my web platforms, I opted to use the DNS challenge to prove my ownership of the domain. The certbot client will prompt you to add a DNS record for domain verification.\n\n\n1\n2\n3\n4\ndocker run -ti --rm -v \\\n\n$(pwd)\n/letsencrypt:/etc/letsencrypt certbot/certbot \\\n--manual --preferred-challenges dns certonly \\\n-d mail.example.com\n\n\n\n\n\n\nGet setup.sh\n\n\ndocker-mailserver comes with a handy bash script for managing the stack (which is just really a wrapper around the container.) It'll make our setup easier, so download it into the root of your configuration/data directory, and make it executable:\n\n\n1\n2\n3\ncurl -o setup.sh \\\nhttps://raw.githubusercontent.com/tomav/docker-mailserver/master/setup.sh \\\nchmod a+x ./setup.sh\n\n\n\n\n\n\nCreate email accounts\n\n\nFor every email address required, run \n./setup.sh email add \nemail\n \npassword\n to create the account. The command returns no output.\n\n\nYou can run \n./setup.sh email list\n to confirm all of your addresses have been created.\n\n\nCreate DKIM DNS entries\n\n\nRun \n./setup.sh config dkim\n to create the necessary DKIM entries. The command returns no output.\n\n\nExamine the keys created by opendkim to identify the DNS TXT records required:\n\n\n1\n2\n3\n4\nfor i in `find config/opendkim/keys/ -name mail.txt`; do \\\necho $i; \\\ncat $i; \\\ndone\n\n\n\n\n\n\nYou'll end up with something like this:\n\n\n1\n2\n3\n4\nconfig/opendkim/keys/gitlab.example.com/mail.txt\nmail._domainkey IN TXT ( \nv=DKIM1; k=rsa; \n\n \np=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB\n ) ; ----- DKIM key mail for gitlab.example.com\n[root@ds1 mail]#\n\n\n\n\n\n\nCreate the necessary DNS TXT entries for your domain(s). Note that although opendkim splits the record across two lines, the actual record should be concatenated on creation. I.e., the DNS TXT record above should read:\n\n\n1\nv=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB\n\n\n\n\n\n\n\nSetup Docker Swarm\n\n\nCreate a docker swarm config file in docker-compose syntax (v3), something like this:\n\n\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\nversion: \n3\n\n\nservices:\n mail:\n image: tvial/docker-mailserver:latest\n ports:\n - \n25:25\n\n - \n587:587\n\n - \n993:993\n\n volumes:\n - /var/data/mail/maildata:/var/mail\n - /var/data/mail/mailstate:/var/mail-state\n - /var/data/mail/config:/tmp/docker-mailserver\n - /var/data/mail/letsencrypt:/etc/letsencrypt\n env_file: /var/data/mail/.env\n networks:\n - internal\n deploy:\n replicas: 1\n\nnetworks:\n traefik:\n external: true\n internal:\n driver: overlay\n ipam:\n config:\n - subnet: 172.16.2.0/24\n\n\n\n\n\n\n\n\nTip\n\n\nSetup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot.\n\n\n\n\nA sample .env file looks like this:\n\n\n1\n2\n3\n4\n5\n6\n7\n8\n9\nENABLE_SPAMASSASSIN=1\nENABLE_CLAMAV=1\nENABLE_POSTGREY=1\nONE_DIR=1\nOVERRIDE_HOSTNAME=mail.example.com\nOVERRIDE_DOMAINNAME=mail.example.com\nPOSTMASTER_ADDRESS=admin@example.com\nPERMIT_DOCKER=network\nSSL_TYPE=letsencrypt\n\n\n\n\n\n\nServing\n\n\nLaunch mailserver\n\n\nLaunch the mail server stack by running \ndocker stack deploy mailserver -c \npath -to-docker-compose.yml\n\n\nChef's Notes\n\n\n\n\nOne of the elements of this design which I didn't appreciate at first is that since the config is entirely file-based, \nsetup.sh\n can be run on any container host, provided it has the shared data mounted. This means that even though docker-mailserver was not designed with docker swarm in mind, it works perfectl with swarm. I.e., from any node, regardless of where the container is actually running, you're able to add/delete email addresses, view logs, etc.", + "title": "Mail Server" + }, + { + "location": "/recipies/mail/#mail-server", + "text": "Many of the recipies that follow require email access of some kind. It's quite accepmatebl normally possible to use a hosted service such as SendGrid, or just a gmail account. If (like me) you'd like to self-host email for your stacks, then the following recipe provides a full-stack mail server running on the docker HA swarm. Of value to me in choosing docker-mailserver were: Automatically renews LetsEncrypt certificates Creation of email accounts across multiple domains (i.e., the same container gives me mailbox , and ) The entire configuration is based on flat files, so there's no database or persistence to worry about docker-mailserver doesn't include a webmail client, and one is not strictly needed. Rainloop can be added either as another service within the stack, or as a standalone service. Rainloop will be covered in a future recipe.", + "title": "Mail Server" + }, + { + "location": "/recipies/mail/#ingredients", + "text": "Docker swarm cluster with persistent shared storage Traefik configured per design LetsEncrypt authorized email address for domain Access to manage DNS records for domains", + "title": "Ingredients" + }, + { + "location": "/recipies/mail/#preparation", + "text": "", + "title": "Preparation" + }, + { + "location": "/recipies/mail/#setup-data-locations", + "text": "We'll need several directories to bind-mount into our container, so create them in /var/data/mailserver: 1\n2\n3\n4 cd /var/data\nmkdir mailserver\ncd mailserver\nmkdir {maildata,mailstate,config,letsencrypt}", + "title": "Setup data locations" + }, + { + "location": "/recipies/mail/#get-letsencrypt-certificate", + "text": "Decide on the FQDN to assign to your mailserver. You can service multiple domains from a single mailserver - i.e., and can both be served by mail.example.com . The docker-mailserver container can renew our LetsEncrypt certs for us, but it can't generate them. To do this, we need to run certbot (from a container) to request the initial certs and create the appropriate directory structure. In the example below, since I'm already using Traefik to manage the LE certs for my web platforms, I opted to use the DNS challenge to prove my ownership of the domain. The certbot client will prompt you to add a DNS record for domain verification. 1\n2\n3\n4 docker run -ti --rm -v \\ $(pwd) /letsencrypt:/etc/letsencrypt certbot/certbot \\\n--manual --preferred-challenges dns certonly \\\n-d mail.example.com", + "title": "Get LetsEncrypt certificate" + }, + { + "location": "/recipies/mail/#get-setupsh", + "text": "docker-mailserver comes with a handy bash script for managing the stack (which is just really a wrapper around the container.) It'll make our setup easier, so download it into the root of your configuration/data directory, and make it executable: 1\n2\n3 curl -o setup.sh \\\nhttps://raw.githubusercontent.com/tomav/docker-mailserver/master/setup.sh \\\nchmod a+x ./setup.sh", + "title": "Get setup.sh" + }, + { + "location": "/recipies/mail/#create-email-accounts", + "text": "For every email address required, run ./setup.sh email add email password to create the account. The command returns no output. You can run ./setup.sh email list to confirm all of your addresses have been created.", + "title": "Create email accounts" + }, + { + "location": "/recipies/mail/#create-dkim-dns-entries", + "text": "Run ./setup.sh config dkim to create the necessary DKIM entries. The command returns no output. Examine the keys created by opendkim to identify the DNS TXT records required: 1\n2\n3\n4 for i in `find config/opendkim/keys/ -name mail.txt`; do \\\necho $i; \\\ncat $i; \\\ndone You'll end up with something like this: 1\n2\n3\n4 config/opendkim/keys/gitlab.example.com/mail.txt\nmail._domainkey IN TXT ( v=DKIM1; k=rsa; \n p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB ) ; ----- DKIM key mail for gitlab.example.com\n[root@ds1 mail]# Create the necessary DNS TXT entries for your domain(s). Note that although opendkim splits the record across two lines, the actual record should be concatenated on creation. I.e., the DNS TXT record above should read: 1 v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB", + "title": "Create DKIM DNS entries" + }, + { + "location": "/recipies/mail/#setup-docker-swarm", + "text": "Create a docker swarm config file in docker-compose syntax (v3), something like this: 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28 version: 3 \n\nservices:\n mail:\n image: tvial/docker-mailserver:latest\n ports:\n - 25:25 \n - 587:587 \n - 993:993 \n volumes:\n - /var/data/mail/maildata:/var/mail\n - /var/data/mail/mailstate:/var/mail-state\n - /var/data/mail/config:/tmp/docker-mailserver\n - /var/data/mail/letsencrypt:/etc/letsencrypt\n env_file: /var/data/mail/.env\n networks:\n - internal\n deploy:\n replicas: 1\n\nnetworks:\n traefik:\n external: true\n internal:\n driver: overlay\n ipam:\n config:\n - subnet: 172.16.2.0/24 Tip Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. A sample .env file looks like this: 1\n2\n3\n4\n5\n6\n7\n8\n9 ENABLE_SPAMASSASSIN=1\nENABLE_CLAMAV=1\nENABLE_POSTGREY=1\nONE_DIR=1\nOVERRIDE_HOSTNAME=mail.example.com\nOVERRIDE_DOMAINNAME=mail.example.com\nPOSTMASTER_ADDRESS=admin@example.com\nPERMIT_DOCKER=network\nSSL_TYPE=letsencrypt", + "title": "Setup Docker Swarm" + }, + { + "location": "/recipies/mail/#serving", + "text": "", + "title": "Serving" + }, + { + "location": "/recipies/mail/#launch-mailserver", + "text": "Launch the mail server stack by running docker stack deploy mailserver -c path -to-docker-compose.yml", + "title": "Launch mailserver" + }, + { + "location": "/recipies/mail/#chefs-notes", + "text": "One of the elements of this design which I didn't appreciate at first is that since the config is entirely file-based, setup.sh can be run on any container host, provided it has the shared data mounted. This means that even though docker-mailserver was not designed with docker swarm in mind, it works perfectl with swarm. I.e., from any node, regardless of where the container is actually running, you're able to add/delete email addresses, view logs, etc.", + "title": "Chef's Notes" + } + ] +} \ No newline at end of file diff --git a/site/recipies/mail/index.html b/site/recipies/mail/index.html new file mode 100644 index 0000000..01a5d39 --- /dev/null +++ b/site/recipies/mail/index.html @@ -0,0 +1,882 @@ + + + + + + + + + + + + + + + + + + + + + + + + + Mail Server - Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

Mail Server

+

Many of the recipies that follow require email access of some kind. It's quite accepmatebl normally possible to use a hosted service such as SendGrid, or just a gmail account. If (like me) you'd like to self-host email for your stacks, then the following recipe provides a full-stack mail server running on the docker HA swarm.

+

Of value to me in choosing docker-mailserver were:

+
    +
  1. Automatically renews LetsEncrypt certificates
    2. +
  2. Creation of email accounts across multiple domains (i.e., the same container gives me mailbox wekan@wekan.example.com, and gitlab@gitlab.example.com)
    3. +
  3. The entire configuration is based on flat files, so there's no database or persistence to worry about
    4. +
+

docker-mailserver doesn't include a webmail client, and one is not strictly needed. Rainloop can be added either as another service within the stack, or as a standalone service. Rainloop will be covered in a future recipe.

+

Ingredients

+
    +
  1. Docker swarm cluster with persistent shared storage
    2. +
  2. Traefik configured per design
    3. +
  3. LetsEncrypt authorized email address for domain
    4. +
  4. Access to manage DNS records for domains
    5. +
+

Preparation

+

Setup data locations

+

We'll need several directories to bind-mount into our container, so create them in /var/data/mailserver:

+
1
+2
+3
+4
cd /var/data
+mkdir mailserver
+cd mailserver
+mkdir {maildata,mailstate,config,letsencrypt}
+
+
+ +

Get LetsEncrypt certificate

+

Decide on the FQDN to assign to your mailserver. You can service multiple domains from a single mailserver - i.e., bob@dev.example.com and daphne@prod.example.com can both be served by mail.example.com.

+

The docker-mailserver container can renew our LetsEncrypt certs for us, but it can't generate them. To do this, we need to run certbot (from a container) to request the initial certs and create the appropriate directory structure.

+

In the example below, since I'm already using Traefik to manage the LE certs for my web platforms, I opted to use the DNS challenge to prove my ownership of the domain. The certbot client will prompt you to add a DNS record for domain verification.

+
1
+2
+3
+4
docker run -ti --rm -v \
+"$(pwd)"/letsencrypt:/etc/letsencrypt certbot/certbot \
+--manual --preferred-challenges dns certonly \
+-d mail.example.com
+
+
+ +

Get setup.sh

+

docker-mailserver comes with a handy bash script for managing the stack (which is just really a wrapper around the container.) It'll make our setup easier, so download it into the root of your configuration/data directory, and make it executable:

+
1
+2
+3
curl -o setup.sh \
+https://raw.githubusercontent.com/tomav/docker-mailserver/master/setup.sh \
+chmod a+x ./setup.sh
+
+
+ +

Create email accounts

+

For every email address required, run ./setup.sh email add <email> <password> to create the account. The command returns no output.

+

You can run ./setup.sh email list to confirm all of your addresses have been created.

+

Create DKIM DNS entries

+

Run ./setup.sh config dkim to create the necessary DKIM entries. The command returns no output.

+

Examine the keys created by opendkim to identify the DNS TXT records required:

+
1
+2
+3
+4
for i in `find config/opendkim/keys/ -name mail.txt`; do \
+echo $i; \
+cat $i; \
+done
+
+
+ +

You'll end up with something like this:

+
1
+2
+3
+4
config/opendkim/keys/gitlab.example.com/mail.txt
+mail._domainkey IN  TXT ( "v=DKIM1; k=rsa; "
+      "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB" )  ; ----- DKIM key mail for gitlab.example.com
+[root@ds1 mail]#
+
+
+ +

Create the necessary DNS TXT entries for your domain(s). Note that although opendkim splits the record across two lines, the actual record should be concatenated on creation. I.e., the DNS TXT record above should read:

+
1
"v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB"
+
+
+ +

Setup Docker Swarm

+

Create a docker swarm config file in docker-compose syntax (v3), something like this:

+
 1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
+23
+24
+25
+26
+27
+28
version: '3'
+
+services:
+  mail:
+    image: tvial/docker-mailserver:latest
+    ports:
+      - "25:25"
+      - "587:587"
+      - "993:993"
+    volumes:
+      - /var/data/mail/maildata:/var/mail
+      - /var/data/mail/mailstate:/var/mail-state
+      - /var/data/mail/config:/tmp/docker-mailserver
+      - /var/data/mail/letsencrypt:/etc/letsencrypt
+    env_file: /var/data/mail/.env
+    networks:
+      - internal
+    deploy:
+      replicas: 1
+
+networks:
+  traefik:
+    external: true
+  internal:
+    driver: overlay
+    ipam:
+      config:
+        - subnet: 172.16.2.0/24
+
+
+ +
+

Tip

+

Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot.

+
+

A sample .env file looks like this:

+
1
+2
+3
+4
+5
+6
+7
+8
+9
ENABLE_SPAMASSASSIN=1
+ENABLE_CLAMAV=1
+ENABLE_POSTGREY=1
+ONE_DIR=1
+OVERRIDE_HOSTNAME=mail.example.com
+OVERRIDE_DOMAINNAME=mail.example.com
+POSTMASTER_ADDRESS=admin@example.com
+PERMIT_DOCKER=network
+SSL_TYPE=letsencrypt
+
+
+ +

Serving

+

Launch mailserver

+

Launch the mail server stack by running docker stack deploy mailserver -c <path -to-docker-compose.yml>

+

Chef's Notes

+
    +
  1. One of the elements of this design which I didn't appreciate at first is that since the config is entirely file-based, setup.sh can be run on any container host, provided it has the shared data mounted. This means that even though docker-mailserver was not designed with docker swarm in mind, it works perfectl with swarm. I.e., from any node, regardless of where the container is actually running, you're able to add/delete email addresses, view logs, etc.
    2. +
+ + + + + + +

Comments

+
+ + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/site/sitemap.xml b/site/sitemap.xml new file mode 100644 index 0000000..65e00af --- /dev/null +++ b/site/sitemap.xml @@ -0,0 +1,84 @@ + + + + + + https://geeks-cookbook.funkypenguin.co.nz/ + 2017-07-30 + daily + + + + + + + https://geeks-cookbook.funkypenguin.co.nz/README/ + 2017-07-30 + daily + + + + https://geeks-cookbook.funkypenguin.co.nz/whoami/ + 2017-07-30 + daily + + + + + + + + https://geeks-cookbook.funkypenguin.co.nz/ha-docker-swarm/design/ + 2017-07-30 + daily + + + + https://geeks-cookbook.funkypenguin.co.nz/ha-docker-swarm/vms/ + 2017-07-30 + daily + + + + https://geeks-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-ceph/ + 2017-07-30 + daily + + + + https://geeks-cookbook.funkypenguin.co.nz/ha-docker-swarm/shared-storage-gluster/ + 2017-07-30 + daily + + + + https://geeks-cookbook.funkypenguin.co.nz/ha-docker-swarm/keepalived/ + 2017-07-30 + daily + + + + https://geeks-cookbook.funkypenguin.co.nz/ha-docker-swarm/docker-swarm-mode/ + 2017-07-30 + daily + + + + https://geeks-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik/ + 2017-07-30 + daily + + + + + + + + https://geeks-cookbook.funkypenguin.co.nz/recipies/mail/ + 2017-07-30 + daily + + + + + \ No newline at end of file diff --git a/site/whoami/index.html b/site/whoami/index.html new file mode 100644 index 0000000..5edc1f4 --- /dev/null +++ b/site/whoami/index.html @@ -0,0 +1,620 @@ + + + + + + + + + + + + + + + + + + + + + + + + + whoami - Funky Penguin's Geek's Cookbook + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +
+ + +
+
+ + +
+
+
+ +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+ + + +

Welcome to Funky Penguin's Geek Cookbook

+

Hello world,

+

I'm David.

+

I've spent 20+ years working with technology. My current role is Senior Infrastructure Architect at Prophecy Networks Ltd in New Zealand, with a specific interest in networking, systems, open-source, and business management.

+

I've had a book published, and I blog on topics that interest me.

+

Why Funky Penguin?

+

My first "real" job, out of high-school, was working the IT helpdesk in a typical pre-2000 organization in South Africa. I enjoyed experimenting with Linux, and cut my teeth by replacing the organization's Exchange 5.5 mail platform with a 15-site qmail-ldap cluster, with amavis virus-scanning.

+

One of our suppliers asked me to quote to do the same for their organization. With nothing to loose, and half-expecting to be turned down, I quoted a generous fee, and chose a cheeky company name. The supplier immediately accepted my quote, and the name ("Funky Penguin") stuck.

+

Technical Documentation

+

During the same "real" job above, I wanted to deploy jabberd, for internal instant messaging within the organization, and as a means to control the sprawl of ad-hoc instant-messaging among staff, using ICQ, MSN, and Yahoo Messenger.

+

To get management approval to deploy, I wrote a logger (with web UI) for jabber conversations (Bandersnatch), and a 75-page user manual (in Docbook XML for a spunky Russian WinXP jabber client, JAJC.

+

Due to my contributions to phpList, I was approached in 2011 by Packt Publishing, to write a book about using PHPList.

+

Contact Me

+

Contact me by:

+ +

Or by using the form below:

+
+ +
+ + + + + + +

Comments

+
+ + + +
+
+
+
+ + + + +
+ + + + + + + + + + + + + + + + \ No newline at end of file