From 2dd09c0cbc5e17ed61f3ae045b91e7e8fc89716f Mon Sep 17 00:00:00 2001 From: David Young Date: Thu, 16 May 2019 10:00:47 +1200 Subject: [PATCH] Avoid commiting staging directory --- .gitignore | 3 ++ manuscript/book.txt | 2 +- .../ha-docker-swarm/docker-swarm-mode.md | 49 ++++++++++++------- .../ha-docker-swarm/{vms.md => nodes.md} | 36 +++++++------- .../ha-docker-swarm/traefik-forward-auth.md | 8 +-- manuscript/ha-docker-swarm/traefik.md | 4 +- manuscript/recipes/bitwarden.md | 22 ++++++--- manuscript/stylesheets/mailchimp.css | 27 ++++++++++ mkdocs.yml | 7 ++- scripts/build.sh | 28 +++++++++++ scripts/recipe-footer.md | 28 +++++++++++ 11 files changed, 159 insertions(+), 55 deletions(-) rename manuscript/ha-docker-swarm/{vms.md => nodes.md} (66%) create mode 100644 manuscript/stylesheets/mailchimp.css create mode 100755 scripts/build.sh create mode 100644 scripts/recipe-footer.md diff --git a/.gitignore b/.gitignore index 4ece16d..eb6088c 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,9 @@ # Don't include built site site/ +# Don't include staging area for publishing +publish/ + # Don't include random notes notes/ diff --git a/manuscript/book.txt b/manuscript/book.txt index 1c187ca..b62f082 100644 --- a/manuscript/book.txt +++ b/manuscript/book.txt @@ -5,7 +5,7 @@ whoami.md sections/ha-docker-swarm.md ha-docker-swarm/design.md -ha-docker-swarm/vms.md +ha-docker-swarm/nodes.md ha-docker-swarm/shared-storage-ceph.md ha-docker-swarm/shared-storage-gluster.md ha-docker-swarm/keepalived.md diff --git a/manuscript/ha-docker-swarm/docker-swarm-mode.md b/manuscript/ha-docker-swarm/docker-swarm-mode.md index 7e0c828..00151e5 100644 --- a/manuscript/ha-docker-swarm/docker-swarm-mode.md +++ b/manuscript/ha-docker-swarm/docker-swarm-mode.md @@ -5,14 +5,34 @@ For truly highly-available services with Docker containers, we need an orchestra ## Ingredients !!! summary - * [X] 3 x modern linux hosts (*bare-metal or VMs*). A reasonable minimum would be: - * 2 x vCPU - * 2GB RAM - * 20GB HDD - * [X] Hosts must be within the same subnet, and connected on a low-latency link (*i.e., no WAN links*) + Existing + + * [X] 3 x nodes (*bare-metal or VMs*), each with: + * A mainstream Linux OS (*tested on either [CentOS](https://www.centos.org) 7+ or [Ubuntu](http://releases.ubuntu.com) 16.04+*) + * At least 2GB RAM + * At least 20GB disk space (_but it'll be tight_) + * [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_) ## Preparation +### Bash auto-completion + +Add some handy bash auto-completion for docker. Without this, you'll get annoyed that you can't autocomplete ```docker stack deploy -c ``` commands. + +``` +cd /etc/bash_completion.d/ +curl -O https://raw.githubusercontent.com/docker/cli/b75596e1e4d5295ac69b9934d1bd8aff691a0de8/contrib/completion/bash/docker +``` + +Install some useful bash aliases on each host +``` +cd ~ +curl -O https://raw.githubusercontent.com/funkypenguin/geek-cookbook/master/examples/scripts/gcb-aliases.sh +echo 'source ~/gcb-aliases.sh' >> ~/.bash_profile +``` + +## Serving + ### Release the swarm! Now, to launch a swarm. Pick a target node, and run `docker swarm init` @@ -145,22 +165,13 @@ services: Launch shepherd by running ```docker stack deploy shepherd -c /var/data/config/shepherd/shepherd.yml```, and then just forget about it, comfortable in the knowledge that every day, Shepherd will check that your images are the latest available, and if not, will destroy and recreate the container on the latest available image. +### Summary -### Tweaks +!!! summary + Created + + * [X] [Docker swarm cluster](/ha-docker-swarm/design/) -Add some handy bash auto-completion for docker. Without this, you'll get annoyed that you can't autocomplete ```docker stack deploy -c ``` commands. - -``` -cd /etc/bash_completion.d/ -curl -O https://raw.githubusercontent.com/docker/cli/b75596e1e4d5295ac69b9934d1bd8aff691a0de8/contrib/completion/bash/docker -``` - -Install some useful bash aliases on each host -``` -cd ~ -curl -O https://raw.githubusercontent.com/funkypenguin/geek-cookbook/master/examples/scripts/gcb-aliases.sh -echo 'source ~/gcb-aliases.sh' >> ~/.bash_profile -``` ## Chef's Notes diff --git a/manuscript/ha-docker-swarm/vms.md b/manuscript/ha-docker-swarm/nodes.md similarity index 66% rename from manuscript/ha-docker-swarm/vms.md rename to manuscript/ha-docker-swarm/nodes.md index 4da5f1e..83db364 100644 --- a/manuscript/ha-docker-swarm/vms.md +++ b/manuscript/ha-docker-swarm/nodes.md @@ -1,6 +1,6 @@ -# Virtual Machines +# Nodes -Let's start building our cluster. You can use either bare-metal machines or virtual machines - the configuration would be the same. Given that most readers (myself included) will be using virtual infrastructure, from now on I'll be referring strictly to VMs. +Let's start building our cluster. You can use either bare-metal machines or virtual machines - the configuration would be the same. To avoid confusion, I'll be referring to these as "nodes" from now on. !!! note In 2017, I **initially** chose the "[Atomic](https://www.projectatomic.io/)" CentOS/Fedora image for the swarm hosts, but later found its outdated version of Docker to be problematic with advanced features like GPU transcoding (in [Plex](/recipes/plex/)), [Swarmprom](/recipes/swarmprom/), etc. In the end, I went mainstream and simply preferred a modern Ubuntu installation. @@ -8,25 +8,18 @@ Let's start building our cluster. You can use either bare-metal machines or virt ## Ingredients !!! summary "Ingredients" - 3 x Virtual Machines, each with: + New in this recipe: - * [ ] A mainstream Linux OS (*tested on either [CentOS](https://www.centos.org) 7+ or [Ubuntu](http://releases.ubuntu.com) 16.04+*) - * [ ] At least 2GB RAM - * [ ] At least 20GB disk space (_but it'll be tight_) + * [ ] 3 x nodes (*bare-metal or VMs*), each with: + * A mainstream Linux OS (*tested on either [CentOS](https://www.centos.org) 7+ or [Ubuntu](http://releases.ubuntu.com) 16.04+*) + * At least 2GB RAM + * At least 20GB disk space (_but it'll be tight_) * [ ] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_) ## Preparation -### Install Virtual machines - -1. Install / launch virtual machines. -2. The default username on CentOS atomic is "centos", and you'll have needed to supply your SSH key during the build process. - -!!! tip - If you're not using a platform with cloud-init support (i.e., you're building a VM manually, not provisioning it through a cloud provider), you'll need to refer to [trick #1](https://spinningmatt.wordpress.com/2014/01/08/a-recipe-for-starting-cloud-images-with-virt-install/) and [trick #2](http://blog.oddbit.com/2015/03/10/booting-cloud-images-with-libvirt/) for a means to override the automated setup, apply a manual password to the CentOS account, and enable SSH password logins. - -### Permit connectivity between hosts +### Permit connectivity Most modern Linux distributions include firewall rules which only only permit minimal required incoming connections (like SSH). We'll want to allow all traffic between our nodes. The steps to achieve this in CentOS/Ubuntu are a little different... @@ -52,7 +45,7 @@ Install the (*non-default*) persistent iptables tools, by running `apt-get insta And refresh your running iptables rules with `iptables-restore < /etc/iptables/rules.v4` -### Enable host resolution +### Enable hostname resolution Depending on your hosting environment, you may have DNS automatically setup for your VMs. If not, it's useful to set up static entries in /etc/hosts for the nodes. For example, I setup the following: @@ -74,9 +67,14 @@ ln -sf /usr/share/zoneinfo/ /etc/localtime After completing the above, you should have: -``` -[X] 3 x fresh linux instances, ready to become swarm nodes -``` +!!! summary "Summary" + Deployed in this recipe: + + * [X] 3 x nodes (*bare-metal or VMs*), each with: + * A mainstream Linux OS (*tested on either [CentOS](https://www.centos.org) 7+ or [Ubuntu](http://releases.ubuntu.com) 16.04+*) + * At least 2GB RAM + * At least 20GB disk space (_but it'll be tight_) + * [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_) ## Chef's Notes diff --git a/manuscript/ha-docker-swarm/traefik-forward-auth.md b/manuscript/ha-docker-swarm/traefik-forward-auth.md index 13ba122..9809030 100644 --- a/manuscript/ha-docker-swarm/traefik-forward-auth.md +++ b/manuscript/ha-docker-swarm/traefik-forward-auth.md @@ -42,14 +42,14 @@ COOKIE_DOMAINS=example.com ### Prepare the docker service config -This is a small container, you can simply add the following content to the existing `traefik.yml` deployed in the previous [Traefik](/recipes/traefik/) recipe: +This is a small container, you can simply add the following content to the existing `traefik-app.yml` deployed in the previous [Traefik](/recipes/traefik/) recipe: ``` traefik-forward-auth: image: thomseddon/traefik-forward-auth env_file: /var/data/config/keycloak/traefik-forward-auth.env networks: - - public + - traefik_public # Uncomment these lines if you're using auth host mode #deploy: # labels: @@ -66,7 +66,7 @@ If you're not confident that forward authentication is working, add a simple "wh whoami: image: containous/whoami networks: - - public + - traefik_public deploy: labels: - traefik.frontend.rule=Host:whoami.example.com @@ -85,7 +85,7 @@ If you're not confident that forward authentication is working, add a simple "wh ### Launch -Redeploy traefik with ```docker stack deploy traefik -c /var/data/traefik/docker-compose.yml```, to launch the traefik-forward-auth container. +Redeploy traefik with ```docker stack deploy traefik-app -c /var/data/traefik/traeifk-app.yml```, to launch the traefik-forward-auth container. ### Test diff --git a/manuscript/ha-docker-swarm/traefik.md b/manuscript/ha-docker-swarm/traefik.md index 92d24f4..7698015 100644 --- a/manuscript/ha-docker-swarm/traefik.md +++ b/manuscript/ha-docker-swarm/traefik.md @@ -16,11 +16,11 @@ To deal with these gaps, we need a front-end load-balancer, and in this design, ## Ingredients !!! summary "You'll need" - Already deployed: + Existing * [X] [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph) - New to this recipe: + New * [ ] Access to update your DNS records for manual/automated [LetsEncrypt](https://letsencrypt.org/docs/challenge-types/) DNS-01 validation, or ingress HTTP/HTTPS for HTTP-01 validation diff --git a/manuscript/recipes/bitwarden.md b/manuscript/recipes/bitwarden.md index 6c213dd..f9851ec 100644 --- a/manuscript/recipes/bitwarden.md +++ b/manuscript/recipes/bitwarden.md @@ -22,9 +22,12 @@ Bitwarden is a free and open source password management solution for individuals ## Ingredients -1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md) -2. [Traefik](/ha-docker-swarm/traefik_public) configured per design -3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP +!!! summary "Ingredients" + Existing: + + 1. [X] [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md) + 2. [X] [Traefik](/ha-docker-swarm/traefik_public) configured per design + 3. [X] DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP ## Preparation @@ -35,6 +38,12 @@ We'll need to create a directory to bind-mount into our container, so create `/v ``` mkdir /var/data/bitwarden ``` +### Setup environment + +Create `/var/data/config/bitwarden/bitwarden.env`, and **leave it empty for now**. + +!!! question + What, why an empty env file? Well, the container supports lots of customizations via environment variables, for things like toggling self-registration, 2FA, etc. These are too complex to go into for this recipe, but readers are recommended to review the [dani-garcia/bitwarden_rs wiki](https://github.com/dani-garcia/bitwarden_rs), and customize their installation to suite. ### Setup Docker Swarm @@ -88,10 +97,7 @@ Once you've created your account, jump over to https://bitwarden.com/#download a ## Chef's Notes 1. You'll notice we're not using the *official* container images (*[all 6 of them required](https://help.bitwarden.com/article/install-on-premise/#install-bitwarden)!)*, but rather a [more lightweight version ideal for self-hosting](https://hub.docker.com/r/mprasil/bitwarden). All of the elements are contained within a single container, and SQLite is used for the database backend. -2. The inclusion of Bitwarden was due to the efforts of @gkoerk in our [Discord server](http://chat.funkypenguin.co.nz)- Thanks Gerry! +2. As mentioned above, readers should refer to the [dani-garcia/bitwarden_rs wiki](https://github.com/dani-garcia/bitwarden_rs) for details on customizing the behaviour of Bitwarden. +3. The inclusion of Bitwarden was due to the efforts of @gkoerk in our [Discord server](http://chat.funkypenguin.co.nz)- Thanks Gerry! -### Tip your waiter (support me) 👏 -Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏 - -### Your comments? 💬 diff --git a/manuscript/stylesheets/mailchimp.css b/manuscript/stylesheets/mailchimp.css new file mode 100644 index 0000000..28acd85 --- /dev/null +++ b/manuscript/stylesheets/mailchimp.css @@ -0,0 +1,27 @@ +#mailchimp { + background: #f8f8f8; + color: #3D6392; + padding: 20px 15px; +} +#mailchimp input { + border: medium none; + color: gray; + font-size: .8rem; + margin-bottom: 10px; + padding: 8px 10px; + width: 300px; +} +#mailchimp input.email { background: #fff } +#mailchimp input.name { background: #fff} +#mailchimp input[type="submit"] { + background: #795548; + color: #fff; + cursor: pointer; + width: 35%; + padding: 8px 0; + font-size: .8rem; + transition: background-color .25s cubic-bezier(.1,.7,.1,1),color .25s cubic-bezier(.1,.7,.1,1); +} +#mailchimp input[type="seeubmit"]:hover { + color: #F2C94C; +} \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 9ed94b4..d205808 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -15,7 +15,7 @@ repo_url: 'https://github.com/funkypenguin/geek-cookbook' docs_dir: 'manuscript' # Copyright -copyright: 'Copyright © 2016 - 2019 David Young' +copyright: 'Copyright © 2016 - 2019 David Young, Funky Penguin Limited' #theme_dir: mkdocs-material @@ -27,7 +27,7 @@ nav: - whoami: whoami.md - Docker Swarm: - Design: ha-docker-swarm/design.md - - VMs: ha-docker-swarm/vms.md + - Nodes: ha-docker-swarm/nodes.md - Shared Storage (Ceph): ha-docker-swarm/shared-storage-ceph.md - Shared Storage (GlusterFS): ha-docker-swarm/shared-storage-gluster.md - Keepalived: ha-docker-swarm/keepalived.md @@ -170,6 +170,9 @@ google_analytics: - 'UA-139253-18' - 'auto' +extra_css: + - 'stylesheets/mailchimp.css' + #extra_javascript: # - 'extras/javascript/discord.js' diff --git a/scripts/build.sh b/scripts/build.sh new file mode 100755 index 0000000..8f91240 --- /dev/null +++ b/scripts/build.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# This script prepares mkdocs for a build (there are some adjustments to be made to the recipes before publishing) + +# Copy the contents of "manuscript" to a new "publish" folder +mkdir -p publish +mkdir -p publish/overrides +cp -r manuscript publish/ +cp mkdocs.yml publish/ + +# Append a common footer to all recipes +for i in `find publish/manuscript -name "*.md"` +do + # Does this recipe already have a "tip your waiter" section? + grep -q "Tip your waiter" $i + if [ $? -eq 1 ] + then + echo -e "\n" >> $i + cat scripts/recipe-footer.md >> $i + else + echo "WARNING - hard-coded footer exists in $i" + fi +done + +# Now build the docs for publishing +mkdocs build -f publish/mkdocs.yml + +# Setup any necessary netlify redirects +cp netlify_redirects.txt publish/site/_redirects diff --git a/scripts/recipe-footer.md b/scripts/recipe-footer.md new file mode 100644 index 0000000..223c4ba --- /dev/null +++ b/scripts/recipe-footer.md @@ -0,0 +1,28 @@ +### Tip your waiter (support me) 👏 + +Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏 + +### Flirt with your waiter (subscribe to updates) :kissing_heart::love_letter: + +Want to know now when this recipe gets updated, or when future recipes are added? Subscribe to the [RSS feed](https://mastodon.social/@geekcookbook_changes.atom), or leave your email address below, and we'll keep you updated. (*double-opt-in, no monkey business, no spam either - check the [archive](https://us16.campaign-archive.com/home/?u=a1d9cee4402be76497a2baf49&id=10e284530a) for proof!*) + + +
+
+
+ +
+ +
+
+ + +
+ + +
+
+
+ + +### Your comments? 💬