public/geek-cookbook
1
0
Fork 0
mirror of https://github.com/funkypenguin/geek-cookbook/ synced 2025-12-20 13:11:44 +00:00
Code Issues Releases Wiki Activity

Update for leanpub preview

Browse Source
This commit is contained in:
AutoPenguin
2020-06-03 01:42:01 +00:00
parent 65dd34c7ea
commit 2e8e16157b
193 changed files with 12667 additions and 155 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
manuscript/kubernetes/cluster.md
View File

@@ -7,7 +7,7 @@ IMO, the easiest Kubernetes cloud provider to experiment with is [DigitalOcean](
## Ingredients
1. [DigitalOcean](https://www.digitalocean.com/?refcode=e33b78ad621b) account, either linked to a credit card or (_my preference for a trial_) topped up with $5 credit from PayPal. (_yes, this is a referral link, making me some to buy _)
2. Geek-Fu required : 🐱 (easy - even has screenshots!)
2. Geek-Fu required : (easy - even has screenshots!)
## Preparation

86
manuscript/kubernetes/cluster.mde Normal file
View File

@@ -0,0 +1,86 @@
# Kubernetes on DigitalOcean
IMO, the easiest Kubernetes cloud provider to experiment with is [DigitalOcean](https://m.do.co/c/e33b78ad621b) (_this is a referral link_). I've included instructions below to start a basic cluster.
![Kubernetes on Digital Ocean](/images/kubernetes-on-digitalocean.jpg)
## Ingredients
1. [DigitalOcean](https://www.digitalocean.com/?refcode=e33b78ad621b) account, either linked to a credit card or (_my preference for a trial_) topped up with $5 credit from PayPal. (_yes, this is a referral link, making me some 💰 to buy 🍷_)
2. Geek-Fu required : 🐱 (easy - even has screenshots!)
## Preparation
### Create DigitalOcean Account
Create a project, and then from your project page, click **Manage** -> **Kubernetes (LTD)** in the left-hand panel:
![Kubernetes on Digital Ocean Screenshot #1](/images/kubernetes-on-digitalocean-screenshot-1.png)
Until DigitalOcean considers their Kubernetes offering to be "production ready", you'll need the additional step of clicking on **Enable Limited Access**:
![Kubernetes on Digital Ocean Screenshot #2](/images/kubernetes-on-digitalocean-screenshot-2.png)
The _Enable Limited Access_ button changes to read _Create a Kubernetes Cluster_ . Cleeeek it:
![Kubernetes on Digital Ocean Screenshot #3](/images/kubernetes-on-digitalocean-screenshot-3.png)
When prompted, choose some defaults for your first node pool (_your pool of "compute" resources for your cluster_), and give it a name. In more complex deployments, you can use this concept of "node pools" to run certain applications (_like an inconsequential nightly batch job_) on a particular class of compute instance (_such as cheap, preemptible instances_)
![Kubernetes on Digital Ocean Screenshot #4](/images/kubernetes-on-digitalocean-screenshot-4.png)
That's it! Have a sip of your 🍷, a bite of your :cheese:, and wait for your cluster to build. While you wait, follow the instructions to setup kubectl (if you don't already have it)
![Kubernetes on Digital Ocean Screenshot #5](/images/kubernetes-on-digitalocean-screenshot-5.png)
DigitalOcean will provide you with a "kubeconfig" file to use to access your cluster. It's at the bottom of the page (_illustrated below_), and easy to miss (_in my experience_).
![Kubernetes on Digital Ocean Screenshot #6](/images/kubernetes-on-digitalocean-screenshot-6.png)
## Release the kubectl!
Save your kubeconfig file somewhere, and test it our by running ```kubectl --kubeconfig=<PATH TO KUBECONFIG> get nodes```
Example output:
```
[davidy:~/Downloads] 130 % kubectl --kubeconfig=penguins-are-the-sexiest-geeks-kubeconfig.yaml get nodes
NAME STATUS ROLES AGE VERSION
festive-merkle-8n9e Ready <none> 20s v1.13.1
[davidy:~/Downloads] %
```
In the example above, my nodes were being deployed. Repeat the command to see your nodes spring into existence:
```
[davidy:~/Downloads] % kubectl --kubeconfig=penguins-are-the-sexiest-geeks-kubeconfig.yaml get nodes
NAME STATUS ROLES AGE VERSION
festive-merkle-8n96 Ready <none> 6s v1.13.1
festive-merkle-8n9e Ready <none> 34s v1.13.1
[davidy:~/Downloads] %
[davidy:~/Downloads] % kubectl --kubeconfig=penguins-are-the-sexiest-geeks-kubeconfig.yaml get nodes
NAME STATUS ROLES AGE VERSION
festive-merkle-8n96 Ready <none> 30s v1.13.1
festive-merkle-8n9a Ready <none> 17s v1.13.1
festive-merkle-8n9e Ready <none> 58s v1.13.1
[davidy:~/Downloads] %
```
That's it. You have a beautiful new kubernetes cluster ready for some action!
## Move on..
Still with me? Good. Move on to creating your own external load balancer..
* [Start](/kubernetes/start/) - Why Kubernetes?
* [Design](/kubernetes/design/) - How does it fit together?
* Cluster (this page) - Setup a basic cluster
* [Load Balancer](/kubernetes/loadbalancer/) - Setup inbound access
* [Snapshots](/kubernetes/snapshots/) - Automatically backup your persistent data
* [Helm](/kubernetes/helm/) - Uber-recipes from fellow geeks
* [Traefik](/kubernetes/traefik/) - Traefik Ingress via Helm
## Chef's Notes
1. Ok, yes, there's not much you can do with your cluster _yet_. But stay tuned, more Kubernetes fun to come!

129
manuscript/kubernetes/design.mde Normal file
View File

@@ -0,0 +1,129 @@
# Design
Like the [Docker Swarm](ha-docker-swarm/design/) "_private cloud_" design, the Kubernetes design is:
* **Highly-available** (_can tolerate the failure of a single component_)
* **Scalable** (_can add resource or capacity as required_)
* **Portable** (_run it in DigitalOcean today, AWS tomorrow and Azure on Thursday_)
* **Secure** (_access protected with LetsEncrypt certificates_)
* **Automated** (_requires minimal care and feeding_)
*Unlike* the Docker Swarm design, the Kubernetes design is:
* **Cloud-Native** (_While you **can** [run your own Kubernetes cluster](https://microk8s.io/), it's far simpler to let someone else manage the infrastructure, freeing you to play with the fun stuff_)
* **Complex** (_Requires more basic elements, more verbose configuration, and provides more flexibility and customisability_)
## Design Decisions
**The design and recipes are provider-agnostic**
This means that:
* The design should work on GKE, AWS, DigitalOcean, Azure, or even MicroK8s
* Custom service elements specific to individual providers are avoided
**The simplest solution to achieve the desired result will be preferred**
This means that:
* Persistent volumes from the cloud provider are used for all persistent storage
* We'll do things the "_Kubernetes way_", i.e., using secrets and configmaps, rather than trying to engineer around the Kubernetes basic building blocks.
**Insofar as possible, the format of recipes will align with Docker Swarm**
This means that:
* We use Kubernetes namespaces to replicate Docker Swarm's "_per-stack_" networking and service discovery
## Security
Under this design, the only inbound connections we're permitting to our Kubernetes swarm are:
### Network Flows
* HTTPS (TCP 443) : Serves individual docker containers via SSL-encrypted reverse proxy (_Traefik_)
* Individual additional ports we choose to expose for specific recipes (_i.e., port 8443 for [MQTT](/recipes/mqtt/)_)
### Authentication
* Other than when an SSL-served application provides a trusted level of authentication, or where the application requires public exposure, applications served via Traefik will be protected with an OAuth proxy.
## The challenges of external access
Because we're Cloude-Native now, it's complex to get traffic **into** our cluster from outside. We basically have 3 options:
1. **HostIP**: Map a port on the host to a service. This is analogous to Docker's port exposure, but lacking in that it restricts us to one host port per-container, and it's not possible to anticipate _which_ of your Kubernetes hosts is running a given container. Kubernetes does not have Docker Swarm's "routing mesh", allowing for simple load-balancing of incoming connections.
2. **LoadBalancer**: Purchase a "loadbalancer" per-service from your cloud provider. While this is the simplest way to assure a fixed IP and port combination will always exist for your service, it has 2 significant limitations:
1. Cost is prohibitive, at roughly $US10/month per port
2. You won't get the _same_ fixed IP for multiple ports. So if you wanted to expose 443 and 25 (_webmail and smtp server, for example_), you'd find yourself assigned a port each on two **unique** IPs, a challenge for a single DNS-based service, like "_mail.batman.com_"
3. **NodePort**: Expose our service as a port (_between 30000-32767_) on the host which happens to be running the service. This is challenging because you might want to expose port 443, but that's not possible with NodePort.
To further complicate options #1 and #3 above, our cloud provider may, without notice, change the IP of the host running your containers (_O hai, Google!_).
Our solution to these challenges is to employ a simple-but-effective solution which places an HAProxy instance in front of the services exposed by NodePort. For example, this allows us to expose a container on 443 as NodePort 30443, and to cause HAProxy to listen on port 443, and forward all requests to our Node's IP on port 30443, after which it'll be forwarded onto our container on the original port 443.
We use a phone-home container, which calls a simple webhook on our haproxy VM, advising HAProxy to update its backend for the calling IP. This means that when our provider changes the host's IP, we automatically update HAProxy and keep-on-truckin'!
Here's a high-level diagram:
![Kubernetes Design](/images/kubernetes-cluster-design.png)
## Overview
So what's happening in the diagram above? I'm glad you asked - let's go through it!
### Setting the scene
In the diagram, we have a Kubernetes cluster comprised of 3 nodes. You'll notice that there's no visible master node. This is because most cloud providers will give you "_free_" master node, but you don't get to access it. The master node is just a part of the Kubernetes "_as-a-service_" which you're purchasing.
Our nodes are partitioned into several namespaces, which logically separate our individual recipes. (_I.e., allowing both a "gitlab" and a "nextcloud" namespace to include a service named "db", which would be challenging without namespaces_)
Outside of our cluster (_could be anywhere on the internet_) is a single VM servicing as a load-balancer, running HAProxy and a webhook service. This load-balancer is described in detail, [in its own section](/kubernetes/loadbalancer/), but what's important up-front is that this VM is the **only element of the design for which we need to provide a fixed IP address**.
### 1 : The mosquitto pod
In the "mqtt" namespace, we have a single pod, running 2 containers - the mqtt broker, and a "phone-home" container.
Why 2 containers in one pod, instead of 2 independent pods? Because all the containers in a pod are **always** run on the same physical host. We're using the phone-home container as a simple way to call a webhook on the not-in-the-cluster VM.
The phone-home container calls the webhook, and tells HAProxy to listen on port 8443, and to forward any incoming requests to port 30843 (_within the NodePort range_) on the IP of the host running the container (_and because of the pod, tho phone-home container is guaranteed to be on the same host as the MQTT container_).
### 2 : The Traefik Ingress
In the "default" namespace, we have a Traefik "Ingress Controller". An Ingress controller is a way to use a single port (_say, 443_) plus some intelligence (_say, a defined mapping of URLs to services_) to route incoming requests to the appropriate containers (_via services_). Basically, the Trafeik ingress does what [Traefik does for us under Docker Swarm](/docker-ha-swarm/traefik/).
What's happening in the diagram is that a phone-home pod is tied to the traefik pod using affinity, so that both containers will be executed on the same host. Again, the phone-home container calls a webhook on the HAProxy VM, auto-configuring HAproxy to send any HTTPs traffic to its calling address and customer NodePort port number.
When an inbound HTTPS request is received by Traefik, based on some internal Kubernetes elements (ingresses), Traefik provides SSL termination, and routes the request to the appropriate service (_In this case, either the GitLab UI or teh UniFi UI_)
### 3 : The UniFi pod
What's happening in the UniFi pod is a combination of #1 and #2 above. UniFi controller provides a webUI (_typically 8443, but we serve it via Traefik on 443_), plus some extra ports for device adoption, which are using a proprietary protocol, and can't be proxied with Traefik.
To make both the webUI and the adoption ports work, we use a combination of an ingress for the webUI (_see #2 above_), and a phone-home container to tell HAProxy to forward port 8080 (_the adoption port_) directly to the host, using a NodePort-exposed service.
This allows us to retain the use of a single IP for all controller functions, as accessed outside of the cluster.
### 4 : The webhook
Each phone-home container is calling a webhook on the HAProxy VM, secured with a secret shared token. The phone-home container passes the desired frontend port (i.e., 443), the corresponding NodeIP port (i.e., 30443), and the node's current public IP address.
The webhook uses the provided details to update HAProxy for the combination of values, validate the config, and then restart HAProxy.
### 5 : The user
Finally, the DNS for all externally-accessible services is pointed to the IP of the HAProxy VM. On receiving an inbound request (_be it port 443, 8080, or anything else configured_), HAProxy will forward the request to the IP and NodePort port learned from the phone-home container.
## Move on..
Still with me? Good. Move on to creating your cluster!
* [Start](/kubernetes/start/) - Why Kubernetes?
* Design (this page) - How does it fit together?
* [Cluster](/kubernetes/cluster/) - Setup a basic cluster
* [Load Balancer](/kubernetes/loadbalancer/) - Setup inbound access
* [Snapshots](/kubernetes/snapshots/) - Automatically backup your persistent data
* [Helm](/kubernetes/helm/) - Uber-recipes from fellow geeks
* [Traefik](/kubernetes/traefik/) - Traefik Ingress via Helm

10
manuscript/kubernetes/diycluster.md
View File

@@ -29,7 +29,7 @@ If you want to use minikube, there is a guide below but again, I recommend using
they add in additional complexities to the installation as they
require running a Linux based image running in a VM,
that although minikube will manage, adds to the complexities. And
even then, who uses Windows or macOS in production anyways? 🙂
even then, who uses Windows or macOS in production anyways?
If you are serious about running on windows/macOS,
check the official MiniKube guides
[here](https://minikube.sigs.k8s.io/docs/start/)
@@ -82,7 +82,7 @@ Then spin yourself up as many systems as you need with the following guide
!!! note
I am running a 3 node cluster, with nodes running on Ubuntu 19.04, all virtualized with VMWare ESXi
Your setup doesn't need to be as complex as mine, you can use 3 old Dell OptiPlex if you really want 🙂
Your setup doesn't need to be as complex as mine, you can use 3 old Dell OptiPlex if you really want
1. Insert your installation medium into the machine, and boot it.
2. Select your language
@@ -183,7 +183,7 @@ thomas-k3s-node1$ curl -sfL https://get.k3s.io | sh -
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
```
@@ -284,7 +284,7 @@ That is all! You have yourself a Kubernetes cluster for you and your dog to enjo
DRP or Digital Rebar Provisioning Tool is a tool designed to automatically setup your cluster, installing an operating system for you, and doing all the configuration like we did in the k3s setup.
This section is WIP, instead, try using the K3S guide above 🙂
This section is WIP, instead, try using the K3S guide above
## Where from now
@@ -304,7 +304,7 @@ This article, believe it or not, was not diced up by your regular chef (funkypen
Instead, today's article was diced up by HexF, a fellow kiwi (hence a lot of kiwi references) who enjoys his sysadmin time.
Feel free to talk to today's chef in the discord, or see one of his many other links that you can follow below
[Twitter](https://hexf.me/api/social/twitter/geekcookbook) • [Website](https://hexf.me/api/social/website/geekcookbook) • [Github](https://hexf.me/api/social/github/geekcookbook)
[Twitter](https://hexf.me/api/social/twitter/geekcookbook) [Website](https://hexf.me/api/social/website/geekcookbook) [Github](https://hexf.me/api/social/github/geekcookbook)
<!--
The links above are just redirect links incase anything ever changes, and it has analytics too

311
manuscript/kubernetes/diycluster.mde Normal file
View File

@@ -0,0 +1,311 @@
# DIY Kubernetes
If you are looking for a little more of a challenge, or just don't have the money to fork out to managed Kubernetes, you're in luck.
Kubernetes provides many ways to run a cluster, by far the simplest method is with `minikube` but there are other methods like `k3s` and using `drp` to deploy a cluster.
After all, DIY its in our DNA.
## Ingredients
1. Basic knowledge of Kubernetes terms (Will come in handy) [Start](/kubernetes/start)
2. Some Linux machines (Depends on what recipe you follow)
## Minikube
First, what is minikube?
Minikube is a method of running Kubernetes on your local machine.
It is mainly targeted at developers looking to test if their application will work with Kubernetes without deploying it to a production cluster. For this reason,
I do not recommend running your cluster on minikube as it isn't designed for deployment, and is only a single node cluster.
If you want to use minikube, there is a guide below but again, I recommend using something more production-ready like `k3s` or `drp`
### Ingredients
1. A Fresh Linux Machine
2. Some basic Linux knowledge (or can just copy-paste)
!!! note
Make sure you are running a SystemD based distro like Ubuntu.
Although minikube will run on macOS and Windows,
they add in additional complexities to the installation as they
require running a Linux based image running in a VM,
that although minikube will manage, adds to the complexities. And
even then, who uses Windows or macOS in production anyways? 🙂
If you are serious about running on windows/macOS,
check the official MiniKube guides
[here](https://minikube.sigs.k8s.io/docs/start/)
### Installation
After booting yourself up a fresh Linux machine and getting to a console,
you can now install minikube.
Download and install our minikube binary
```sh
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
```
Now we can boot up our cluster
```sh
sudo minikube start --vm-driver=none
#Start our minikube instance, and make it use the machine to host the cluster, instead of a VM
sudo minikube config set vm-driver none #Set our default vm driver to none
```
You are now set up with minikube!
!!! warning
MiniKube is not a production-grade method of deploying Kubernetes
## K3S
What is k3s?
K3s is a production-ready method of deploying Kubernetes on many machines,
where a full Kubernetes deployment is not required, AKA - your cluster (unless your a big SaaS company, in that case, can I get a job?).
### Ingredients
1. A handful of Linux machines (3 or more, virtualized or not)
2. Some Linux knowledge.
3. Patience.
### Setting your Linux Machines up
Firstly, my flavour of choice for deployment is Ubuntu Server,
although it is not as enterprise-friendly as RHEL (That's Red Hat Enterprise Linux for my less geeky readers) or CentOS (The free version of RHEL).
Ubuntu ticks all the boxes for k3s to run on and allows you to follow lots of other guides on managing and maintaining your Ubuntu server.
Firstly, download yourself a version of Ubuntu Server from [here](https://ubuntu.com/download/server) (Whatever is latest)
Then spin yourself up as many systems as you need with the following guide
!!! note
I am running a 3 node cluster, with nodes running on Ubuntu 19.04, all virtualized with VMWare ESXi
Your setup doesn't need to be as complex as mine, you can use 3 old Dell OptiPlex if you really want 🙂
1. Insert your installation medium into the machine, and boot it.
2. Select your language
3. Select your keyboard layout
4. Select `Install Ubuntu`
5. Check and modify your network settings if required, make sure to write down your IPs
6. Select Done on Proxy, unless you use a proxy
7. Select Done on Mirror, as it has picked the best mirror for you unless you have a local mirror you want to use (in that case you are uber-geek)
8. Select `Use An Entire Disk` for Filesystem, and basically hit enter for the rest of the disk setup,
just make sure to read the prompts and understand what you are doing
9. Now that you are up to setting up the profile, this is where things change.
You are going to want to set up the same account on all the machines, but change the server name just a tad every time.
![Profile Setup for Node 1](../images/diycluster-k3s-profile-setup.png)
![Profile Setup for Node 2](../images/diycluster-k3s-profile-setup-node2.png)
10. Now install OpenSSH on the server, if you wish to import your existing SSH key from GitHub or Launchpad,
you can do that now and save yourself a step later.
11. Skip over Featured Server snaps by clicking `Done`
12. Wait for your server to install everything and drop you to a Linux prompt
13. Repeat for all your nodes
### Pre-installation of k3s
For the rest of this guide, you will need some sort of Linux/macOS based terminal.
On Windows you can do this with Windows Subsystem for Linux (WSL) see [here for information on WSL.](https://aka.ms/wslinstall)
The rest of this guide will all be from your local terminal.
If you already have an SSH key generated or added an existing one, skip this step.
From your PC,run `ssh-keygen` to generate a public and private key pair
(You can use this instead of typing your password in every time you want to connect via ssh)
```sh
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/thomas/.ssh/id_rsa): [enter]
Enter passphrase (empty for no passphrase): [password]
Enter same passphrase again: [password]
Your identification has been saved in /home/thomas/.ssh/id_rsa.
Your public key has been saved in /home/thomas/.ssh/id_rsa.pub.
The key fingerprint is:
...
The key's randomart image is:
...
```
If you have already imported a key from GitHub or Launchpad, skip this step.
```sh
$ ssh-copy-id [username]@[hostname]
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/thomas/.ssh/id_rsa.pub"
The authenticity of host 'thomas-k3s-node1 (theipaddress)' can't be established.
ECDSA key fingerprint is SHA256:...
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
thomas@thomas-k3s-node1's password: [insert your password now]
Number of key(s) added: 1
```
You will want to do this once for every machine, replacing the hostname with the other next nodes hostname each time.
!!! note
If your hostnames aren't resolving correct, try adding them to your `/etc/hosts` file
### Installation
If you have access to the premix repository, you can download the ansible-playbook and follow the steps contained in there, if not sit back and prepare to do it manually.
!!! tip
Becoming a patron will allow you to get the ansible-playbook to setup k3s on your own hosts. For as little as 5$/m you can get access to the ansible playbooks for this recipe, and more!
See [funkypenguin's Patreon](https://www.patreon.com/funkypenguin) for more!
<!---
(Just someone needs to remind me (HexF) to write such playbook)
-->
Select one node to become your master, in my case `thomas-k3s-node1`.
Now SSH into this node, and run the following:
```sh
localpc$ ssh thomas@thomas-k3s-node1
Enter passphrase for key '/home/thomas/.ssh/id_rsa': [ssh key password]
thomas-k3s-node1$ curl -sfL https://get.k3s.io | sh -
[sudo] password for thomas: [password entered in setup]
[INFO] Finding latest release
[INFO] Using v1.0.0 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.0.0/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.0.0/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
```
Before we log out of the master, we need the token from it.
Make sure to note this token down
(please don't write it on paper, use something like `notepad` or `vim`, it's ~100 characters)
```sh
thomas-k3s-node1$ sudo cat /var/lib/rancher/k3s/server/node-token
K1097e226f95f56d90a4bab7151...
```
Make sure all nodes can access each other by hostname, whether you add them to `/etc/hosts` or to your DNS server
Now that you have your master node setup, you can now add worker nodes
SSH into the other nodes, and run the following making sure to replace values with ones that suit your installation
```sh
localpc$ ssh thomas@thomas-k3s-node2
Enter passphrase for key '/home/thomas/.ssh/id_rsa': [ssh key password]
thomas-k3s-node2$ curl -sfL https://get.k3s.io | K3S_URL=https://thomas-k3s-node1:6443 K3S_TOKEN=K1097e226f95f56d90a4bab7151... sh -
```
Now test your installation!
SSH into your master node
```sh
ssh thomas@thomas-k3s-node1
Enter passphrase for key '/home/thomas/.ssh/id_rsa': [ssh key password]
thomas-k3s-node1$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
thomas-k3s-node1 Ready master 15m3s v1.16.3-k3s.2
thomas-k3s-node2 Ready <none> 6m58s v1.16.3-k3s.2
thomas-k3s-node3 Ready <none> 6m12s v1.16.3-k3s.2
```
If you got Ready for all your nodes, Well Done! Your k3s cluster is now running! If not try getting help in our discord.
### Post-Installation
Now you can get yourself a kubeconfig for your cluster.
SSH into your master node, and run the following
```sh
localpc$ ssh thomas@thomas-k3s-node1
Enter passphrase for key '/home/thomas/.ssh/id_rsa': [ssh key password]
thomas-k3s-node1$ sudo kubectl config view --flatten
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBD...
server: https://127.0.0.1:6443
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
password: thisishowtolosecontrolofyourk3s
username: admin
```
Make sure to change `clusters.cluster.server` to have the master node's name instead of `127.0.0.1`, in my case making it `https://thomas-k3s-node1:6443`
!!! warning
This kubeconfig file can grant full access to your Kubernetes installation, I recommend you protect this file just as well as you protect your passwords
You will probably want to save this kubeconfig file into a file on your local machine, say `my-k3s-cluster.yml` or `where-8-hours-of-my-life-went.yml`.
Now test it out!
```sh
localpc$ kubectl --kubeconfig=my-k3s-cluster.yml get nodes
NAME STATUS ROLES AGE VERSION
thomas-k3s-node1 Ready master 495m v1.16.3-k3s.2
thomas-k3s-node2 Ready <none> 488m v1.16.3-k3s.2
thomas-k3s-node3 Ready <none> 487m v1.16.3-k3s.2
```
<!--
To the reader concerned about my health, no I did not actually spend 8 hours writing this guide, Instead I spent most of it helping you guys on the discord (👍) and other stuff
-->
That is all! You have yourself a Kubernetes cluster for you and your dog to enjoy.
## DRP
DRP or Digital Rebar Provisioning Tool is a tool designed to automatically setup your cluster, installing an operating system for you, and doing all the configuration like we did in the k3s setup.
This section is WIP, instead, try using the K3S guide above 🙂
## Where from now
Now that you have wasted half a lifetime on installing your very own cluster, you can install more to it. Like a load balancer!
* [Start](/kubernetes/start/) - Why Kubernetes?
* [Design](/kubernetes/design/) - How does it fit together?
* Cluster (this page) - Setup a basic cluster
* [Load Balancer](/kubernetes/loadbalancer/) - Setup inbound access
* [Snapshots](/kubernetes/snapshots/) - Automatically backup your persistent data
* [Helm](/kubernetes/helm/) - Uber-recipes from fellow geeks
* [Traefik](/kubernetes/traefik/) - Traefik Ingress via Helm
## About your Chef
This article, believe it or not, was not diced up by your regular chef (funkypenguin).
Instead, today's article was diced up by HexF, a fellow kiwi (hence a lot of kiwi references) who enjoys his sysadmin time.
Feel free to talk to today's chef in the discord, or see one of his many other links that you can follow below
[Twitter](https://hexf.me/api/social/twitter/geekcookbook) • [Website](https://hexf.me/api/social/website/geekcookbook) • [Github](https://hexf.me/api/social/github/geekcookbook)
<!--
The links above are just redirect links incase anything ever changes, and it has analytics too
-->

2
manuscript/kubernetes/helm.md
View File

@@ -10,7 +10,7 @@
## Ingredients
1. [Kubernetes cluster](https://geek-cookbook.funkypenguin.co.nz/)kubernetes/cluster/)
2. Geek-Fu required : 🐤 (_easy - copy and paste_)
2. Geek-Fu required : (_easy - copy and paste_)
## Preparation

62
manuscript/kubernetes/helm.mde Normal file
View File

@@ -0,0 +1,62 @@
# Helm
[Helm](https://github.com/helm/helm) is a tool for managing Kubernetes "charts" (_think of it as an uber-polished collection of recipes_). Using one simple command, and by tweaking one simple config file (values.yaml), you can launch a complex stack. There are many publicly available helm charts for popular packages like [elasticsearch](https://github.com/helm/charts/tree/master/stable/elasticsearch), [ghost](https://github.com/helm/charts/tree/master/stable/ghost), [grafana](https://github.com/helm/charts/tree/master/stable/grafana), [mediawiki](https://github.com/helm/charts/tree/master/stable/mediawiki), etc.
![Kubernetes Snapshots](/images/kubernetes-helm.png)
!!! note
Given enough interest, I may provide a helm-compatible version of the pre-mix repository for [supporters](/support/). [Hit me up](/whoami/#contact-me) if you're interested!
## Ingredients
1. [Kubernetes cluster](/kubernetes/cluster/)
2. Geek-Fu required : 🐤 (_easy - copy and paste_)
## Preparation
### Install Helm
This section is from the Helm README:
Binary downloads of the Helm client can be found on [the Releases page](https://github.com/helm/helm/releases/latest).
Unpack the `helm` binary and add it to your PATH and you are good to go!
If you want to use a package manager:
- [Homebrew](https://brew.sh/) users can use `brew install kubernetes-helm`.
- [Chocolatey](https://chocolatey.org/) users can use `choco install kubernetes-helm`.
- [Scoop](https://scoop.sh/) users can use `scoop install helm`.
- [GoFish](https://gofi.sh/) users can use `gofish install helm`.
To rapidly get Helm up and running, start with the [Quick Start Guide](https://docs.helm.sh/using_helm/#quickstart-guide).
See the [installation guide](https://docs.helm.sh/using_helm/#installing-helm) for more options,
including installing pre-releases.
## Serving
### Initialise Helm
After installing Helm, initialise it by running ```helm init```. This will install "tiller" pod into your cluster, which works with the locally installed helm binaries to launch/update/delete Kubernetes elements based on helm charts.
That's it - not very exciting I know, but we'll need helm for the next and final step in building our Kubernetes cluster - deploying the [Traefik ingress controller (via helm)](/kubernetes/traefik/)!
## Move on..
Still with me? Good. Move on to understanding Helm charts...
* [Start](/kubernetes/start/) - Why Kubernetes?
* [Design](/kubernetes/design/) - How does it fit together?
* [Cluster](/kubernetes/cluster/) - Setup a basic cluster
* [Load Balancer](/kubernetes/loadbalancer/) Setup inbound access
* [Snapshots](/kubernetes/snapshots/) - Automatically backup your persistent data
* Helm (this page) - Uber-recipes from fellow geeks
* [Traefik](/kubernetes/traefik/) - Traefik Ingress via Helm
## Chef's Notes
1. Of course, you can have lots of fun deploying all sorts of things via Helm. Check out https://github.com/helm/charts for some examples.

4
manuscript/kubernetes/loadbalancer.md
View File

@@ -14,7 +14,7 @@ This recipe details a simple design to permit the exposure of as many ports as y
1. [Kubernetes cluster](https://geek-cookbook.funkypenguin.co.nz/)kubernetes/cluster/)
2. VM _outside_ of Kubernetes cluster, with a fixed IP address. Perhaps, on a [$5/month Digital Ocean Droplet](https://www.digitalocean.com/?refcode=e33b78ad621b).. (_yes, another referral link. Mooar for me!_)
3. Geek-Fu required : 🐧🐧🐧 (_complex - inline adjustments required_)
3. Geek-Fu required : (_complex - inline adjustments required_)
## Preparation
@@ -331,4 +331,4 @@ Still with me? Good. Move on to setting up an ingress SSL terminating proxy with
## Chef's Notes
1. This is MVP of the load balancer solution. Any suggestions for improvements are welcome 😉
1. This is MVP of the load balancer solution. Any suggestions for improvements are welcome

334
manuscript/kubernetes/loadbalancer.mde Normal file
View File

@@ -0,0 +1,334 @@
# Load Balancer
One of the issues I encountered early on in migrating my Docker Swarm workloads to Kubernetes on GKE, was how to reliably permit inbound traffic into the cluster.
There were several complications with the "traditional" mechanisms of providing a load-balanced ingress, not the least of which was cost. I also found that even if I paid my cloud provider (_Google_) for a load-balancer Kubernetes service, this service required a unique IP per exposed port, which was incompatible with my mining pool empire (_mining pools need to expose multiple ports on the same DNS name_).
See further examination of the problem and possible solutions in the [Kubernetes design](kubernetes/design/#the-challenges-of-external-access) page.
This recipe details a simple design to permit the exposure of as many ports as you like, on a single public IP, to a cluster of Kubernetes nodes running as many pods/containers as you need, with services exposed via NodePort.
![Kubernetes Design](/images/kubernetes-cluster-design.png)
## Ingredients
1. [Kubernetes cluster](/kubernetes/cluster/)
2. VM _outside_ of Kubernetes cluster, with a fixed IP address. Perhaps, on a [$5/month Digital Ocean Droplet](https://www.digitalocean.com/?refcode=e33b78ad621b).. (_yes, another referral link. Mooar 🍷 for me!_)
3. Geek-Fu required : 🐧🐧🐧 (_complex - inline adjustments required_)
## Preparation
### Summary
### Create LetsEncrypt certificate
!!! warning
Safety first, folks. You wouldn't run a webhook exposed to the big bad ol' internet without first securing it with a valid SSL certificate? Of course not, I didn't think so!
Use whatever method you prefer to generate (and later, renew) your LetsEncrypt cert. The example below uses the CertBot docker image for CloudFlare DNS validation, since that's what I've used elsewhere.
We **could** run our webhook as a simple HTTP listener, but really, in a world where LetsEncrypt cacn assign you a wildcard certificate in under 30 seconds, thaht's unforgivable. Use the following **general** example to create a LetsEncrypt wildcard cert for your host:
In my case, since I use CloudFlare, I create /etc/webhook/letsencrypt/cloudflare.ini:
```
dns_cloudflare_email=davidy@funkypenguin.co.nz
dns_cloudflare_api_key=supersekritnevergonnatellyou
```
I request my cert by running:
```
cd /etc/webhook/
docker run -ti --rm -v "$(pwd)"/letsencrypt:/etc/letsencrypt certbot/dns-cloudflare --preferred-challenges dns certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini -d ''*.funkypenguin.co.nz'
```
!!! question
Why use a wildcard cert? So my enemies can't examine my certs to enumerate my various services and discover my weaknesses, of course!
I add the following as a cron command to renew my certs every day:
```
cd /etc/webhook && docker run -ti --rm -v "$(pwd)"/letsencrypt:/etc/letsencrypt certbot/dns-cloudflare renew --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini
```
Once you've confirmed you've got a valid LetsEncrypt certificate stored in ```/etc/webhook/letsencrypt/live/<your domain>/fullcert.pem```, proceed to the next step..
### Install webhook
We're going to use https://github.com/adnanh/webhook to run our webhook. On some distributions (_❤️ ya, Debian!_), webhook and its associated systemd config can be installed by running ```apt-get install webhook```.
### Create webhook config
We'll create a single webhook, by creating ```/etc/webhook/hooks.json``` as follows. Choose a nice secure random string for your MY_TOKEN value!
```
mkdir /etc/webhook
export MY_TOKEN=ilovecheese
echo << EOF > /etc/webhook/hooks.json
[
{
"id": "update-haproxy",
"execute-command": "/etc/webhook/update-haproxy.sh",
"command-working-directory": "/etc/webhook",
"pass-arguments-to-command":
[
{
"source": "payload",
"name": "name"
},
{
"source": "payload",
"name": "frontend-port"
},
{
"source": "payload",
"name": "backend-port"
},
{
"source": "payload",
"name": "dst-ip"
},
{
"source": "payload",
"name": "action"
}
],
"trigger-rule":
{
"match":
{
"type": "value",
"value": "$MY_TOKEN",
"parameter":
{
"source": "header",
"name": "X-Funkypenguin-Token"
}
}
}
}
]
EOF
```
!!! note
Note that to avoid any bozo from calling our we're matching on a token header in the request called ```X-Funkypenguin-Token```. Webhook will **ignore** any request which doesn't include a matching token in the request header.
### Update systemd for webhook
!!! note
This section is particular to Debian Stretch and its webhook package. If you're using another OS for your VM, just ensure that you can start webhook with a config similar to the one illustrated below.
Since we want to force webhook to run in secure mode (_no point having a token if it can be extracted from a simple packet capture!_) I ran ```systemctl edit webhook```, and pasted in the following:
```
[Service]
# Override the default (non-secure) behaviour of webhook by passing our certificate details and custom hooks.json location
ExecStart=
ExecStart=/usr/bin/webhook -hooks /etc/webhook/hooks.json -verbose -secure -cert /etc/webhook/letsencrypt/live/funkypenguin.co.nz/fullchain.pem -key /etc/webhook/letsencrypt/live/funkypenguin.co.nz/privkey.pem
```
Then I restarted webhook by running ```systemctl enable webhook && systemctl restart webhook```. I watched the subsequent logs by running ```journalctl -u webhook -f```
### Create /etc/webhook/update-haproxy.sh
When successfully authenticated with our top-secret token, our webhook will execute a local script, defined as follows (_yes, you should create this file_):
```
#!/bin/bash
NAME=$1
FRONTEND_PORT=$2
BACKEND_PORT=$3
DST_IP=$4
ACTION=$5
# Bail if we haven't received our expected parameters
if [[ "$#" -ne 5 ]]
then
echo "illegal number of parameters"
exit 2;
fi
# Either add or remove a service based on $ACTION
case $ACTION in
add)
# Create the portion of haproxy config
cat << EOF > /etc/webhook/haproxy/$FRONTEND_PORT.inc
### >> Used to run $NAME:${FRONTEND_PORT}
frontend ${FRONTEND_PORT}_frontend
bind *:$FRONTEND_PORT
mode tcp
default_backend ${FRONTEND_PORT}_backend
backend ${FRONTEND_PORT}_backend
mode tcp
balance roundrobin
stick-table type ip size 200k expire 30m
stick on src
server s1 $DST_IP:$BACKEND_PORT
### << Used to run $NAME:$FRONTEND_PORT
EOF
;;
delete)
rm /etc/webhook/haproxy/$FRONTEND_PORT.inc
;;
*)
echo "Invalid action $ACTION"
exit 2
esac
# Concatenate all the haproxy configs into a single file
cat /etc/webhook/haproxy/global /etc/webhook/haproxy/*.inc > /etc/webhook/haproxy/pre_validate.cfg
# Validate the generated config
haproxy -f /etc/webhook/haproxy/pre_validate.cfg -c
# If validation was successful, only _then_ copy it over to /etc/haproxy/haproxy.cfg, and reload
if [[ $? -gt 0 ]]
then
echo "HAProxy validation failed, not continuing"
exit 2
else
# Remember what the original file looked like
m1=$(md5sum "/etc/haproxy/haproxy.cfg")
# Overwrite the original file
cp /etc/webhook/haproxy/pre_validate.cfg /etc/haproxy/haproxy.cfg
# Get MD5 of new file
m2=$(md5sum "/etc/haproxy/haproxy.cfg")
# Only if file has changed, then we need to reload haproxy
if [ "$m1" != "$m2" ] ; then
echo "HAProxy config has changed, reloading"
systemctl reload haproxy
fi
fi
```
### Create /etc/webhook/haproxy/global
Create ```/etc/webhook/haproxy/global``` and populate with something like the following. This will be the non-dynamically generated part of our HAProxy config:
```
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
log global
mode tcp
option tcplog
option dontlognull
timeout connect 5000
timeout client 5000000
timeout server 5000000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
```
## Serving
### Take the bait!
Whew! We now have all the components of our automated load-balancing solution in place. Browse to your VM's FQDN at https://whatever.it.is:9000/hooks/update-haproxy, and you should see the text "_Hook rules were not satisfied_", with a valid SSL certificate (_You didn't send a token_).
If you don't see the above, then check the following:
1. Does the webhook verbose log (```journalctl -u webhook -f```) complain about invalid arguments or missing files?
2. Is port 9000 open to the internet on your VM?
### Apply to pods
You'll see me use this design in any Kubernetes-based recipe which requires container-specific ports, like UniFi. Here's an excerpt of the .yml which defines the UniFi controller:
```
<snip>
spec:
containers:
- image: linuxserver/unifi
name: controller
volumeMounts:
- name: controller-volumeclaim
mountPath: /config
- image: funkypenguin/poor-mans-k8s-lb
imagePullPolicy: Always
name: 8080-phone-home
env:
- name: REPEAT_INTERVAL
value: "600"
- name: FRONTEND_PORT
value: "8080"
- name: BACKEND_PORT
value: "30808"
- name: NAME
value: "unifi-adoption"
- name: WEBHOOK
value: "https://my-secret.url.wouldnt.ya.like.to.know:9000/hooks/update-haproxy"
- name: WEBHOOK_TOKEN
valueFrom:
secretKeyRef:
name: unifi-credentials
key: webhook_token.secret
<snip>
```
The takeaways here are:
1. We add the funkypenguin/poor-mans-k8s-lb containier to any pod which has special port requirements, forcing the container to run on the same node as the other containers in the pod (_in this case, the UniFi controller_)
2. We use a Kubernetes secret for the webhook token, so that our .yml can be shared without exposing sensitive data
Here's what the webhook logs look like when the above is added to the UniFi deployment:
```
Feb 06 23:04:28 haproxy2 webhook[1433]: [webhook] 2019/02/06 23:04:28 Started POST /hooks/update-haproxy
Feb 06 23:04:28 haproxy2 webhook[1433]: [webhook] 2019/02/06 23:04:28 update-haproxy got matched
Feb 06 23:04:28 haproxy2 webhook[1433]: [webhook] 2019/02/06 23:04:28 update-haproxy hook triggered successfully
Feb 06 23:04:28 haproxy2 webhook[1433]: [webhook] 2019/02/06 23:04:28 Completed 200 OK in 2.123921ms
Feb 06 23:04:28 haproxy2 webhook[1433]: [webhook] 2019/02/06 23:04:28 executing /etc/webhook/update-haproxy.sh (/etc/webhook/update-haproxy.sh) with arguments ["/etc/webhook/update-haproxy.sh" "unifi-adoption" "8080" "30808" "35.244.91.178" "add"] and environment [] using /etc/webhook as cwd
Feb 06 23:04:28 haproxy2 webhook[1433]: [webhook] 2019/02/06 23:04:28 command output: Configuration file is valid
<HAProxy restarts>
```
## Move on..
Still with me? Good. Move on to setting up an ingress SSL terminating proxy with Traefik..
* [Start](/kubernetes/start/) - Why Kubernetes?
* [Design](/kubernetes/design/) - How does it fit together?
* [Cluster](/kubernetes/cluster/) - Setup a basic cluster
* Load Balancer (this page) - Setup inbound access
* [Snapshots](/kubernetes/snapshots/) - Automatically backup your persistent data
* [Helm](/kubernetes/helm/) - Uber-recipes from fellow geeks
* [Traefik](/kubernetes/traefik/) - Traefik Ingress via Helm
## Chef's Notes
1. This is MVP of the load balancer solution. Any suggestions for improvements are welcome 😉

2
manuscript/kubernetes/snapshots.md
View File

@@ -15,7 +15,7 @@ This recipe employs a clever tool ([miracle2k/k8s-snapshots](https://github.com/
## Ingredients
1. [Kubernetes cluster](https://geek-cookbook.funkypenguin.co.nz/)kubernetes/cluster/) with either AWS or GKE (currently, but apparently other providers are [easy to implement](https://github.com/miracle2k/k8s-snapshots/blob/master/k8s_snapshots/backends/abstract.py))
2. Geek-Fu required : 🐒🐒 (_medium - minor adjustments may be required_)
2. Geek-Fu required : (_medium - minor adjustments may be required_)
## Preparation

180
manuscript/kubernetes/snapshots.mde Normal file
View File

@@ -0,0 +1,180 @@
# Snapshots
Before we get carried away creating pods, services, deployments etc, let's spare a thought for _security_... (_DevSecPenguinOps, here we come!_). In the context of this recipe, security refers to safe-guarding your data from accidental loss, as well as malicious impact.
Under [Docker Swarm](/ha-docker-swarm/design/), we used [shared storage](/ha-docker-swarm/shared-storage-ceph/) with [Duplicity](/recipes/duplicity/) (or [ElkarBackup](recipes/elkarbackup/)) to automate backups of our persistent data.
Now that we're playing in the deep end with Kubernetes, we'll need a Cloud-native backup solution...
It bears repeating though - don't be like [Cameron](http://haltandcatchfire.wikia.com/wiki/Cameron_Howe). Backup your stuff.
<iframe width="560" height="315" src="https://www.youtube.com/embed/1UtFeMoqVHQ" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
This recipe employs a clever tool ([miracle2k/k8s-snapshots](https://github.com/miracle2k/k8s-snapshots)), running _inside_ your cluster, to trigger automated snapshots of your persistent volumes, using your cloud provider's APIs.
## Ingredients
1. [Kubernetes cluster](/kubernetes/cluster/) with either AWS or GKE (currently, but apparently other providers are [easy to implement](https://github.com/miracle2k/k8s-snapshots/blob/master/k8s_snapshots/backends/abstract.py))
2. Geek-Fu required : 🐒🐒 (_medium - minor adjustments may be required_)
## Preparation
### Create RoleBinding (GKE only)
If you're running GKE, run the following to create a RoleBinding, allowing your user to grant rights-it-doesn't-currently-have to the service account responsible for creating the snapshots:
```kubectl create clusterrolebinding your-user-cluster-admin-binding \
--clusterrole=cluster-admin --user=<your user@yourdomain>```
!!! question
Why do we have to do this? Check [this blog post](https://www.funkypenguin.co.nz/workaround-blocked-attempt-to-grant-extra-privileges-on-gke/) for details
### Apply RBAC
If your cluster is RBAC-enabled (_it probably is_), you'll need to create a ClusterRole and ClusterRoleBinding to allow k8s_snapshots to see your PVs and friends:
```
kubectl apply -f https://raw.githubusercontent.com/miracle2k/k8s-snapshots/master/rbac.yaml
```
## Serving
### Deploy the pod
Ready? Run the following to create a deployment in to the kube-system namespace:
```
cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: k8s-snapshots
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
app: k8s-snapshots
spec:
containers:
- name: k8s-snapshots
image: elsdoerfer/k8s-snapshots:v2.0
EOF
```
Confirm your pod is running and happy by running ```kubectl get pods -n kubec-system```, and ```kubectl -n kube-system logs k8s-snapshots<tab-to-auto-complete>```
### Pick PVs to snapshot
k8s-snapshots relies on annotations to tell it how frequently to snapshot your PVs. A PV requires the ```backup.kubernetes.io/deltas``` annotation in order to be snapshotted.
From the k8s-snapshots README:
```
The generations are defined by a list of deltas formatted as ISO 8601 durations (this differs from tarsnapper). PT60S or PT1M means a minute, PT12H or P0.5D is half a day, P1W or P7D is a week. The number of backups in each generation is implied by it's and the parent generation's delta.
For example, given the deltas PT1H P1D P7D, the first generation will consist of 24 backups each one hour older than the previous (or the closest approximation possible given the available backups), the second generation of 7 backups each one day older than the previous, and backups older than 7 days will be discarded for good.
The most recent backup is always kept.
The first delta is the backup interval.
```
To add the annotation to an existing PV, run something like this:
```
kubectl patch pv pvc-01f74065-8fe9-11e6-abdd-42010af00148 -p \
'{"metadata": {"annotations": {"backup.kubernetes.io/deltas": "P1D P30D P360D"}}}'
```
To add the annotation to a _new_ PV, add the following annotation to your **PVC**:
```
backup.kubernetes.io/deltas: PT1H P2D P30D P180D
```
Here's an example of the PVC for the UniFi recipe, which includes 7 daily snapshots of the PV:
```
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: controller-volumeclaim
namespace: unifi
annotations:
backup.kubernetes.io/deltas: P1D P7D
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```
And here's what my snapshot list looks like after a few days:
![Kubernetes Snapshots](/images/kubernetes-snapshots.png)
### Snapshot a non-Kubernetes volume (optional)
If you're running traditional compute instances with your cloud provider (I do this for my poor man's load balancer), you might want to backup _these_ volumes as well.
To do so, first create a custom resource, ```SnapshotRule```:
```
cat <<EOF | kubectl create -f -
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: snapshotrules.k8s-snapshots.elsdoerfer.com
spec:
group: k8s-snapshots.elsdoerfer.com
version: v1
scope: Namespaced
names:
plural: snapshotrules
singular: snapshotrule
kind: SnapshotRule
shortNames:
- sr
EOF
```
Then identify the volume ID of your volume, and create an appropriate ```SnapshotRule```:
```
cat <<EOF | kubectl apply -f -
apiVersion: "k8s-snapshots.elsdoerfer.com/v1"
kind: SnapshotRule
metadata:
name: haproxy-badass-loadbalancer
spec:
deltas: P1D P7D
backend: google
disk:
name: haproxy2
zone: australia-southeast1-a
EOF
```
!!! note
Example syntaxes for the SnapshotRule for different providers can be found at https://github.com/miracle2k/k8s-snapshots/tree/master/examples
## Move on..
Still with me? Good. Move on to understanding Helm charts...
* [Start](/kubernetes/start/) - Why Kubernetes?
* [Design](/kubernetes/design/) - How does it fit together?
* [Cluster](/kubernetes/cluster/) - Setup a basic cluster
* [Load Balancer](/kubernetes/loadbalancer/) Setup inbound access
* Snapshots (this page) - Automatically backup your persistent data
* [Helm](/kubernetes/helm/) - Uber-recipes from fellow geeks
* [Traefik](/kubernetes/traefik/) - Traefik Ingress via Helm
## Chef's Notes
1. I've submitted [2 PRs](https://github.com/miracle2k/k8s-snapshots/pulls/funkypenguin) to the k8s-snapshots repo. The first [updates the README for GKE RBAC requirements](https://github.com/miracle2k/k8s-snapshots/pull/71), and the second [fixes a minor typo](https://github.com/miracle2k/k8s-snapshots/pull/74).

67
manuscript/kubernetes/start.mde Normal file
View File

@@ -0,0 +1,67 @@
# Why Kubernetes?
My first introduction to Kubernetes was a children's story:
<iframe width="560" height="315" src="https://www.youtube.com/embed/4ht22ReBjno" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
## Wait, what?
Why would you want to use Kubernetes for your self-hosted recipes over simple Docker Swarm? Here's my personal take..
I use Docker swarm both at home (_on a single-node swarm_), and on a trio of Ubuntu 16.04 VPSs in a shared lab OpenStack environment.
In both cases above, I'm responsible for maintaining the infrastructure supporting Docker - either the physical host, or the VPS operating systems.
I started experimenting with Kubernetes as a plan to improve the reliability of my cryptocurrency mining pools (_the contended lab VPSs negatively impacted the likelihood of finding a block_), and as a long-term replacement for my aging home server.
What I enjoy about building recipes and self-hosting is **not** the operating system maintenance, it's the tools and applications that I can quickly launch in my swarms. If I could **only** play with the applications, and not bother with the maintenance, I totally would.
Kubernetes (_on a cloud provider, mind you!_) does this for me. I feed Kubernetes a series of YAML files, and it takes care of all the rest, including version upgrades, node failures/replacements, disk attach/detachments, etc.
## Uggh, it's so complicated!
Yes, but that's a necessary sacrifice for the maturity, power and flexibility it offers. Like docker-compose syntax, Kubernetes uses YAML to define its various, interworking components.
Let's talk some definitions. Kubernetes.io provides a [glossary](https://kubernetes.io/docs/reference/glossary/?fundamental=true). My definitions are below:
* **Node** : A compute instance which runs docker containers, managed by a cluster master.
* **Cluster** : One or more "worker nodes" which run containers. Very similar to a Docker Swarm node. In most cloud provider deployments, the [master node for your cluster is provided free of charge](https://www.sdxcentral.com/articles/news/google-eliminates-gke-management-fees-kubernetes-clusters/2017/11/), but you don't get to access it.
* **Pod** : A collection of one or more the containers. If a pod runs multiple containers, these containers always run on the same node.
* **Deployment** : A definition of a desired state. I.e., "I want a pod with containers A and B running". The Kubernetes master then ensures that any changes necessary to maintain the state are taken. (_I.e., if a pod crashes, but is supposed to be running, a new pod will be started_)
* **Service** : Unlike Docker Swarm, service discovery is not _built in_ to Kubernetes. For your pods to discover each other (say, to have "webserver" talk to "database"), you create a service for each pod, and refer to these services when you want your containers (_in pods_) to talk to each other. Complicated, yes, but the abstraction allows you to do powerful things, like auto-scale-up a bunch of database "pods" behind a service called "database", or perform a rolling container image upgrade with zero impact.
* **External access** : Services not only allow pods to discover each other, but they're also the mechanism through which the outside world can talk to a container. At the simplest level, this is akin to exposing a container port on a docker host.
* **Ingress** : When mapping ports to applications is inadequate (think virtual web hosts), an ingress is a sort of "inbound router" which can receive requests on one port (i.e., HTTPS), and forward them to a variety of internal pods, based on things like VHOST, etc. For us, this is the functional equivalent of what Traefik does in Docker Swarm. In fact, we use a Traefik Ingress in Kubernetes to accomplish the same.
* **Persistent Volume** : A virtual disk which is attached to a pod, storing persistent data. Meets the requirement for shared storage from Docker Swarm. I.e., if a persistent volume (PV) is bound to a pod, and the pod dies and is recreated, or get upgraded to a new image, the PV the data is bound to the new container. PVs can be "claimed" in a YAML definition, so that your Kubernetes provider will auto-create a PV when you launch your pod. PVs can be snapshotted.
* **Namespace** : An abstraction to separate a collection of pods, services, ingresses, etc. A "virtual cluster within a cluster". Can be used for security, or simplicity. For example, since we don't have individual docker stacks anymore, if you commonly name your database container "db", and you want to deploy two applications which both use a database container, how will you name your services? Use namespaces to keep each application ("nextcloud" vs "kanboard") separate. Namespaces also allow you to allocate resources **limits** to the aggregate of containers in a namespace, so you could, for example, limit the "nextcloud" namespace to 2.3 CPUs and 1200MB RAM.
## Mm.. maaaaybe, how do I start?
If you're like me, and you learn by doing, either play with the examples at https://labs.play-with-k8s.com/, or jump right in by setting up a Google Cloud trial (_you get $300 credit for 12 months_), or a small cluster on [Digital Ocean](/kubernetes/digitalocean/).
If you're the learn-by-watching type, just search for "Kubernetes introduction video". There's a **lot** of great content available.
## I'm ready, gimme some recipes!
As of Jan 2019, our first (_and only!_) Kubernetes recipe is a WIP for the Mosquitto [MQTT](/recipes/mqtt/) broker. It's a good, simple starter if you're into home automation (_shoutout to [Home Assistant](/recipes/homeassistant/)!_), since it only requires a single container, and a simple NodePort service.
I'd love for your [feedback](/support/) on the Kubernetes recipes, as well as suggestions for what to add next. The current rough plan is to replicate the Chef's Favorites recipes (_see the left-hand panel_) into Kubernetes first.
## Move on..
Still with me? Good. Move on to reviewing the design elements
* Start (this page) - Why Kubernetes?
* [Design](/kubernetes/design/) - How does it fit together?
* [Cluster](/kubernetes/cluster/) - Setup a basic cluster
* [Load Balancer](/kubernetes/loadbalancer/) - Setup inbound access
* [Snapshots](/kubernetes/snapshots/) - Automatically backup your persistent data
* [Helm](/kubernetes/helm/) - Uber-recipes from fellow geeks
* [Traefik](/kubernetes/traefik/) - Traefik Ingress via Helm

2
manuscript/kubernetes/traefik.md
View File

@@ -188,7 +188,7 @@ helm upgrade --values values.yml traefik stable/traefik --recreate-pods
## Review
We're doneburgers! 🍔 We now have all the pieces to safely deploy recipes into our Kubernetes cluster, knowing:
We're doneburgers! We now have all the pieces to safely deploy recipes into our Kubernetes cluster, knowing:
1. Our HTTPS traffic will be secured with LetsEncrypt (thanks Traefik!)
2. Our non-HTTPS ports (like UniFi adoption) will be load-balanced using an free-to-scale [external load balancer](https://geek-cookbook.funkypenguin.co.nz/)kubernetes/loadbalancer/)

214
manuscript/kubernetes/traefik.mde Normal file
View File

@@ -0,0 +1,214 @@
# Traefik
This recipe utilises the [traefik helm chart](https://github.com/helm/charts/tree/master/stable/traefik) to proving LetsEncrypt-secured HTTPS access to multiple containers within your cluster.
## Ingredients
1. [Kubernetes cluster](/kubernetes/cluster/)
2. [Helm](/kubernetes/helm/) installed and initialised in your cluster
## Preparation
### Clone helm charts
Clone the helm charts, by running:
```
git clone https://github.com/helm/charts
```
Change to stable/traefik:
```
cd charts/stable/traefik
```
### Edit values.yaml
The beauty of the helm approach is that all the complexity of the Kubernetes elements' YAML files are hidden from you (created using templates), and all your changes go into values.yaml.
These are my values, you'll need to adjust for your own situation:
```
imageTag: alpine
serviceType: NodePort
# yes, we're not listening on 80 or 443 because we don't want to pay for a loadbalancer IP to do this. I use poor-mans-k8s-lb instead
service:
nodePorts:
http: 30080
https: 30443
cpuRequest: 1m
memoryRequest: 100Mi
cpuLimit: 1000m
memoryLimit: 500Mi
ssl:
enabled: true
enforced: true
debug:
enabled: false
rbac:
enabled: true
dashboard:
enabled: true
domain: traefik.funkypenguin.co.nz
kubernetes:
# set these to all the namespaces you intend to use. I standardize on one-per-stack. You can always add more later
namespaces:
- kube-system
- unifi
- kanboard
- nextcloud
- huginn
- miniflux
accessLogs:
enabled: true
acme:
persistence:
enabled: true
# Add the necessary annotation to backup ACME store with k8s-snapshots
annotations: { "backup.kubernetes.io/deltas: P1D P7D" }
staging: false
enabled: true
logging: true
email: "<my letsencrypt email>"
challengeType: "dns-01"
dnsProvider:
name: cloudflare
cloudflare:
CLOUDFLARE_EMAIL: "<my cloudlare email"
CLOUDFLARE_API_KEY: "<my cloudflare API key>"
domains:
enabled: true
domainsList:
- main: "*.funkypenguin.co.nz" # name of the wildcard domain name for the certificate
- sans:
- "funkypenguin.co.nz"
metrics:
prometheus:
enabled: true
```
!!! note
The helm chart doesn't enable the Traefik dashboard by default. I intend to add an oauth_proxy pod to secure this, in a future recipe update.
### Prepare phone-home pod
[Remember](/kubernetes/loadbalancer/) how our load balancer design ties a phone-home container to another container using a pod, so that the phone-home container can tell our external load balancer (_using a webhook_) where to send our traffic?
Since we deployed Traefik using helm, we need to take a slightly different approach, so we'll create a pod with an affinity which ensures it runs on the same host which runs the Traefik container (_more precisely, containers with the label app=traefik_).
Create phone-home.yaml as follows:
```
apiVersion: v1
kind: Pod
metadata:
name: phonehome-traefik
spec:
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- traefik
topologyKey: failure-domain.beta.kubernetes.io/zone
containers:
- image: funkypenguin/poor-mans-k8s-lb
imagePullPolicy: Always
name: phonehome-traefik
env:
- name: REPEAT_INTERVAL
value: "600"
- name: FRONTEND_PORT
value: "443"
- name: BACKEND_PORT
value: "30443"
- name: NAME
value: "traefik"
- name: WEBHOOK
value: "https://<your loadbalancer hostname>:9000/hooks/update-haproxy"
- name: WEBHOOK_TOKEN
valueFrom:
secretKeyRef:
name: traefik-credentials
key: webhook_token.secret
```
Create your webhook token secret by running:
```
echo -n "imtoosecretformyshorts" > webhook_token.secret
kubectl create secret generic traefik-credentials --from-file=webhook_token.secret
```
!!! warning
Yes, the "-n" in the echo statement is needed. [Read here for why](https://www.funkypenguin.co.nz/beware-the-hidden-newlines-in-kubernetes-secrets/).
## Serving
### Install the chart
To install the chart, simply run ```helm install stable/traefik --name traefik --namespace kube-system```
That's it, traefik is running.
You can confirm this by running ```kubectl get pods```, and even watch the traefik logs, by running ```kubectl logs -f traefik<tab-to-autocomplete>```
### Deploy the phone-home pod
We still can't access traefik yet, since it's listening on port 30443 on node it happens to be running on. We'll launch our phone-home pod, to tell our [load balancer](/kubernetes/loadbalancer/) where to send incoming traffic on port 443.
Optionally, on your loadbalancer VM, run ```journalctl -u webhook -f``` to watch for the container calling the webhook.
Run ```kubectl create -f phone-home.yaml``` to create the pod.
Run ```kubectl get pods -o wide``` to confirm that both the phone-home pod and the traefik pod are on the same node:
```
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
phonehome-traefik 1/1 Running 0 20h 10.56.2.55 gke-penguins-are-sexy-8b85ef4d-2c9g
traefik-69db67f64c-5666c 1/1 Running 0 10d 10.56.2.30 gkepenguins-are-sexy-8b85ef4d-2c9g
```
Now browse to https://<your load balancer>, and you should get a valid SSL cert, along with a 404 error (_you haven't deployed any other recipes yet_)
### Making changes
If you change a value in values.yaml, and want to update the traefik pod, run:
```
helm upgrade --values values.yml traefik stable/traefik --recreate-pods
```
## Review
We're doneburgers! 🍔 We now have all the pieces to safely deploy recipes into our Kubernetes cluster, knowing:
1. Our HTTPS traffic will be secured with LetsEncrypt (thanks Traefik!)
2. Our non-HTTPS ports (like UniFi adoption) will be load-balanced using an free-to-scale [external load balancer](/kubernetes/loadbalancer/)
3. Our persistent data will be [automatically backed up](/kubernetes/snapshots/)
Here's a recap:
* [Start](/kubernetes/start/) - Why Kubernetes?
* [Design](/kubernetes/design/) - How does it fit together?
* [Cluster](/kubernetes/cluster/) - Setup a basic cluster
* [Load Balancer](/kubernetes/loadbalancer/) Setup inbound access
* [Snapshots](/kubernetes/snapshots/) - Automatically backup your persistent data
* [Helm](/kubernetes/helm/) - Uber-recipes from fellow geeks
* Traefik (this page) - Traefik Ingress via Helm
## Where to next?
I'll be adding more Kubernetes versions of existing recipes soon. Check out the [MQTT](/recipes/mqtt/) recipe for a start!
## Chef's Notes
1. It's kinda lame to be able to bring up Traefik but not to use it. I'll be adding the oauth_proxy element shortly, which will make this last step a little more conclusive and exciting!