Highly Available Docker Swarm Design

In the design described below, our "private cloud" platform is:

• Highly-available (can tolerate the failure of a single component)
• Scalable (can add resource or capacity as required)
• Portable (run it on your garage server today, run it in AWS tomorrow)
• Secure (access protected with LetsEncrypt certificates and optional OIDC with 2FA)
• Automated (requires minimal care and feeding)

Design Decisions

Where possible, services will be highly available.**

This means that:

• At least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure.
• Ceph is employed for share storage, because it too can be made tolerant of a single failure.

Where multiple solutions to a requirement exist, preference will be given to the most portable solution.

This means that:

• Services are defined using docker-compose v3 YAML syntax
• Services are portable, meaning a particular stack could be shut down and moved to a new provider with minimal effort.

Security

Under this design, the only inbound connections we're permitting to our docker swarm in a minimal configuration (you may add custom services later, like UniFi Controller) are:

Network Flows

• HTTP (TCP 80) : Redirects to https
• HTTPS (TCP 443) : Serves individual docker containers via SSL-encrypted reverse proxy

Authentication

• Where the hosted application provides a trusted level of authentication (i.e., NextCloud), or where the application requires public exposure (i.e. Privatebin), no additional layer of authentication will be required.
• Where the hosted application provides inadequate (i.e. NZBGet) or no authentication (i.e. Gollum), a further authentication against an OAuth provider will be required.

High availability

Normal function

Assuming a 3-node configuration, under normal circumstances the following is illustrated:

• All 3 nodes provide shared storage via Ceph, which is provided by a docker container on each node.
• All 3 nodes participate in the Docker Swarm as managers.
• The various containers belonging to the application "stacks" deployed within Docker Swarm are automatically distributed amongst the swarm nodes.
• Persistent storage for the containers is provide via cephfs mount.
• The traefik service (in swarm mode) receives incoming requests (on HTTP and HTTPS), and forwards them to individual containers. Traefik knows the containers names because it's able to read the docker socket.
• All 3 nodes run keepalived, at varying priorities. Since traefik is running as a swarm service and listening on TCP 80/443, requests made to the keepalived VIP and arriving at any of the swarm nodes will be forwarded to the traefik container (no matter which node it's on), and then onto the target backend.

Node failure

In the case of a failure (or scheduled maintenance) of one of the nodes, the following is illustrated:

• The failed node no longer participates in Ceph, but the remaining nodes provide enough fault-tolerance for the cluster to operate.
• The remaining two nodes in Docker Swarm achieve a quorum and agree that the failed node is to be removed.
• The (possibly new) leader manager node reschedules the containers known to be running on the failed node, onto other nodes.
• The traefik service is either restarted or unaffected, and as the backend containers stop/start and change IP, traefik is aware and updates accordingly.
• The keepalived VIP continues to function on the remaining nodes, and docker swarm continues to forward any traffic received on TCP 80/443 to the appropriate node.

Node restore

When the failed (or upgraded) host is restored to service, the following is illustrated:

• Ceph regains full redundancy
• Docker Swarm managers become aware of the recovered node, and will use it for scheduling new containers
• Existing containers which were migrated off the node are not migrated backend
• Keepalived VIP regains full redundancy

Total cluster failure

A day after writing this, my environment suffered a fault whereby all 3 VMs were unexpectedly and simultaneously powered off.

Upon restore, docker failed to start on one of the VMs due to local disk space issue¹. However, the other two VMs started, established the swarm, mounted their shared storage, and started up all the containers (services) which were managed by the swarm.

In summary, although I suffered an unplanned power outage to all of my infrastructure, followed by a failure of a third of my hosts... all my platforms are 100% available¹ with absolutely no manual intervention.

Nodes

Let's start building our cluster. You can use either bare-metal machines or virtual machines - the configuration would be the same. To avoid confusion, I'll be referring to these as "nodes" from now on.

Ingredients

Preparation

Permit connectivity

Most modern Linux distributions include firewall rules which only only permit minimal required incoming connections (like SSH). We'll want to allow all traffic between our nodes. The steps to achieve this in CentOS/Ubuntu are a little different...

CentOS

Add something like this to /etc/sysconfig/iptables:

# Allow all inter-node communication
-A INPUT -s 192.168.31.0/24 -j ACCEPT

And restart iptables with systemctl restart iptables

Ubuntu

Install the (non-default) persistent iptables tools, by running apt-get install iptables-persistent, establishing some default rules (dkpg will prompt you to save current ruleset), and then add something like this to /etc/iptables/rules.v4:

# Allow all inter-node communication
-A INPUT -s 192.168.31.0/24 -j ACCEPT

And refresh your running iptables rules with iptables-restore < /etc/iptables/rules.v4

Enable hostname resolution

Depending on your hosting environment, you may have DNS automatically setup for your VMs. If not, it's useful to set up static entries in /etc/hosts for the nodes. For example, I setup the following:

• 192.168.31.11 ds1 ds1.funkypenguin.co.nz
• 192.168.31.12 ds2 ds2.funkypenguin.co.nz
• 192.168.31.13 ds3 ds3.funkypenguin.co.nz

Set timezone

Set your local timezone, by running:

ln -sf /usr/share/zoneinfo/<your timezone> /etc/localtime

Serving

After completing the above, you should have:

Introduction

My first introduction to Kubernetes was a children's story:

Why Kubernetes?

Why would you want to Kubernetes for your self-hosted recipes, over simple Docker Swarm? Here's my personal take..

Docker Swarm is dead

Sorry to say, but from where I sit, there's no innovation or development happening in docker swarm.

Yes, I know, after Docker Inc sold its platform business to Mirantis in Nov 2019, in Feb 2020 Mirantis back-tracked on their original plan to sunset swarm after 2 years, and stated that they'd continue to invest in swarm. But seriously, look around. Nobody is interested in swarm right now...

... Not even Mirantis! As of Nov 2021, the Mirantis blog tag "kubernetes" had 8 posts within the past month. The tag "docker" has 8 posts in the past 2 years, the 8^th being the original announcement of the Docker aquisition. The tag "docker swarm" has only 2 posts, ever.

Dead. Extinct. Like the doodoo.

Once you go Kubernetes, you can't go back

For years now, I've provided Kubernetes design consulting to small clients and large enterprises. The implementation details in each case vary widely, but there are some primitives which I've come to take for granted, and I wouldn't easily do without. A few examples:

• CLI drives API from anywhere. From my laptop, I can use my credentials to manage any number of Kubernetes clusters, simply by switching kubectl "context". Each interaction is an API call against an HTTPS endpoint. No SSHing to hosts and manually running docker command as root!
• GitOps is magic. There are multiple ways to achieve it, but having changes you commit to a repo automatically applied to a cluster, "Just Works™". The process removes so much friction from making changes that it makes you more productive, and a better "gitizen" ;P
• Controllers are trustworthy. I've come to trust that when I tell Kubernetes to run 3 replicas on separate hosts, to scale up a set of replicas based on CPU load metrics, or provision a blob of storage for a given workloa, that this will be done in a consistent and visible way. I'll be able to see logs / details for each action taken by the controller, and adjust my own instructions/configuration accordingly if necessary.

Uggh, it's so complicated!

Yes, it's more complex than Docker Swarm. And that complexity can definately be a barrier, although with improved tooling, it's continually becoming less-so. However, you don't need to be a mechanic to drive a car or to use a chainsaw. You just need a basic understanding of some core primitives, and then you get on with using the tool to achieve your goals, without needing to know every detail about how it works!

Your end-goal is probably "I want to reliably self-host services I care about", and not "I want to fully understand a complex, scalable, and highly sophisticated container orchestrator". ¹

So let's get on with learning how to use the tool...

Mm.. maaaaybe, how do I start?

Primarily you need 2 things:

1. A cluster
2. A way to deploy workloads into the cluster

Practically, you need some extras too, but you can mix-and-match these.