public/geek-cookbook
1
0
Fork 0
mirror of https://github.com/funkypenguin/geek-cookbook/ synced 2025-12-13 09:46:23 +00:00
Code Issues Releases Wiki Activity

Experiment with PDF generation

Browse Source 
Signed-off-by: David Young <davidy@funkypenguin.co.nz>
This commit is contained in:
David Young
2022-08-19 16:40:53 +12:00
parent c051e0bdad
commit abf9309cb1
317 changed files with 124 additions and 546 deletions
Download Patch File Download Diff File Expand all files Collapse all files

314
docs/kubernetes/backup/index.md Normal file
View File

@@ -0,0 +1,314 @@
# Miniflux
Miniflux is a lightweight RSS reader, developed by [Frédéric Guillot](https://github.com/fguillot). (_Who also happens to be the developer of the favorite Open Source Kanban app, [Kanboard](/recipes/kanboard/)_)
![Miniflux Screenshot](/images/miniflux.png){ loading=lazy }
I've [reviewed Miniflux in detail on my blog](https://www.funkypenguin.co.nz/review/miniflux-lightweight-self-hosted-rss-reader/), but features (among many) that I appreciate:
* Compatible with the Fever API, read your feeds through existing mobile and desktop clients (_This is the killer feature for me. I hardly ever read RSS on my desktop, I typically read on my iPhone or iPad, using [Fiery Feeds](http://cocoacake.net/apps/fiery/) or my new squeeze, [Unread](https://www.goldenhillsoftware.com/unread/)_)
* Send your bookmarks to Pinboard, Wallabag, Shaarli or Instapaper (_I use this to automatically pin my bookmarks for collection on my [blog](https://www.funkypenguin.co.nz/)_)
* Feeds can be configured to download a "full" version of the content (_rather than an excerpt_)
* Use the Bookmarklet to subscribe to a website directly from any browsers
!!! abstract "2.0+ is a bit different"
[Some things changed](https://docs.miniflux.net/en/latest/migration.html) when Miniflux 2.0 was released. For one thing, the only supported database is now postgresql (_no more SQLite_). External themes are gone, as is PHP (_in favor of golang_). It's been a controversial change, but I'm keen on minimal and single-purpose, so I'm still very happy with the direction of development. The developer has laid out his [opinions](https://docs.miniflux.net/en/latest/opinionated.html) re the decisions he's made in the course of development.
## Ingredients
1. A [Kubernetes Cluster](/kubernetes/design/) including [Traefik Ingress](/kubernetes/traefik/)
2. A DNS name for your miniflux instance (*miniflux.example.com*, below) pointing to your [load balancer](/kubernetes/loadbalancer/), fronting your Traefik ingress
## Preparation
### Prepare traefik for namespace
When you deployed [Traefik via the helm chart](/kubernetes/traefik/), you would have customized ```values.yml``` for your deployment. In ```values.yml``` is a list of namespaces which Traefik is permitted to access. Update ```values.yml``` to include the *miniflux* namespace, as illustrated below:
```yaml
<snip>
kubernetes:
namespaces:
- kube-system
- nextcloud
- kanboard
- miniflux
<snip>
```
If you've updated ```values.yml```, upgrade your traefik deployment via helm, by running ```helm upgrade --values values.yml traefik stable/traefik --recreate-pods```
### Create data locations
Although we could simply bind-mount local volumes to a local Kubuernetes cluster, since we're targetting a cloud-based Kubernetes deployment, we only need a local path to store the YAML files which define the various aspects of our Kubernetes deployment.
```bash
mkdir /var/data/config/miniflux
```
### Create namespace
We use Kubernetes namespaces for service discovery and isolation between our stacks, so create a namespace for the miniflux stack with the following .yml:
```bash
cat <<EOF > /var/data/config/miniflux/namespace.yml
apiVersion: v1
kind: Namespace
metadata:
name: miniflux
EOF
kubectl create -f /var/data/config/miniflux/namespace.yaml
```
### Create persistent volume claim
Persistent volume claims are a streamlined way to create a persistent volume and assign it to a container in a pod. Create a claim for the miniflux postgres database:
```bash
cat <<EOF > /var/data/config/miniflux/db-persistent-volumeclaim.yml
kkind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: miniflux-db
namespace: miniflux
annotations:
backup.kubernetes.io/deltas: P1D P7D
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF
kubectl create -f /var/data/config/miniflux/db-persistent-volumeclaim.yaml
```
!!! question "What's that annotation about?"
The annotation is used by [k8s-snapshots](/kubernetes/snapshots/) to create daily incremental snapshots of your persistent volumes. In this case, our volume is snapshotted daily, and copies kept for 7 days.
### Create secrets
It's not always desirable to have sensitive data stored in your .yml files. Maybe you want to check your config into a git repository, or share it. Using Kubernetes Secrets means that you can create "secrets", and use these in your deployments by name, without exposing their contents. Run the following, replacing ```imtoosexyformyadminpassword```, and the ```mydbpass``` value in both postgress-password.secret **and** database-url.secret:
```bash
echo -n "imtoosexyformyadminpassword" > admin-password.secret
echo -n "mydbpass" > postgres-password.secret
echo -n "postgres://miniflux:mydbpass@db/miniflux?sslmode=disable" > database-url.secret
kubectl create secret -n mqtt generic miniflux-credentials \
--from-file=admin-password.secret \
--from-file=database-url.secret \
--from-file=database-url.secret
```
!!! tip "Why use ```echo -n```?"
Because. See [my blog post here](https://www.funkypenguin.co.nz/blog/beware-the-hidden-newlines-in-kubernetes-secrets/) for the pain of hunting invisible newlines, that's why!
## Serving
Now that we have a [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/), a [persistent volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/), and a [configmap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/), we can create [deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), [services](https://kubernetes.io/docs/concepts/services-networking/service/), and an [ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) for the miniflux [pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/).
### Create db deployment
Deployments tell Kubernetes about the desired state of the pod (*which it will then attempt to maintain*). Create the db deployment by excecuting the following. Note that the deployment refers to the secrets created above.
--8<-- "premix-cta.md"
```bash
cat <<EOF > /var/data/miniflux/db-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: miniflux
name: db
labels:
app: db
spec:
replicas: 1
selector:
matchLabels:
app: db
template:
metadata:
labels:
app: db
spec:
containers:
- image: postgres:11
name: db
volumeMounts:
- name: miniflux-db
mountPath: /var/lib/postgresql/data
env:
- name: POSTGRES_USER
value: "miniflux"
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: miniflux-credentials
key: postgres-password.secret
volumes:
- name: miniflux-db
persistentVolumeClaim:
claimName: miniflux-db
```
### Create app deployment
Create the app deployment by excecuting the following. Again, note that the deployment refers to the secrets created above.
```bash
cat <<EOF > /var/data/miniflux/app-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: miniflux
name: app
labels:
app: app
spec:
replicas: 1
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
spec:
containers:
- image: miniflux/miniflux
name: app
env:
# This is necessary for the miniflux to update the db schema, even on an empty DB
- name: CREATE_ADMIN
value: "1"
- name: RUN_MIGRATIONS
value: "1"
- name: ADMIN_USERNAME
value: "admin"
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: miniflux-credentials
key: admin-password.secret
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: miniflux-credentials
key: database-url.secret
EOF
kubectl create -f /var/data/miniflux/deployment.yml
```
### Check pods
Check that your deployment is running, with ```kubectl get pods -n miniflux```. After a minute or so, you should see 2 "Running" pods, as illustrated below:
```bash
[funkypenguin:~] % kubectl get pods -n miniflux
NAME READY STATUS RESTARTS AGE
app-667c667b75-5jjm9 1/1 Running 0 4d
db-fcd47b88f-9vvqt 1/1 Running 0 4d
[funkypenguin:~] %
```
### Create db service
The db service resource "advertises" the availability of PostgreSQL's port (TCP 5432) in your pod, to the rest of the cluster (*constrained within your namespace*). It seems a little like overkill coming from the Docker Swarm's automated "service discovery" model, but the Kubernetes design allows for load balancing, rolling upgrades, and health checks of individual pods, without impacting the rest of the cluster elements.
```bash
cat <<EOF > /var/data/miniflux/db-service.yml
kind: Service
apiVersion: v1
metadata:
name: db
namespace: miniflux
spec:
selector:
app: db
ports:
- protocol: TCP
port: 5432
clusterIP: None
EOF
kubectl create -f /var/data/miniflux/service.yml
```
### Create app service
The app service resource "advertises" the availability of miniflux's HTTP listener port (TCP 8080) in your pod. This is the service which will be referred to by the ingress (below), so that Traefik can route incoming traffic to the miniflux app.
```bash
cat <<EOF > /var/data/miniflux/app-service.yml
kind: Service
apiVersion: v1
metadata:
name: app
namespace: miniflux
spec:
selector:
app: app
ports:
- protocol: TCP
port: 8080
clusterIP: None
EOF
kubectl create -f /var/data/miniflux/app-service.yml
```
### Check services
Check that your services are deployed, with ```kubectl get services -n miniflux```. You should see something like this:
```bash
[funkypenguin:~] % kubectl get services -n miniflux
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
app ClusterIP None <none> 8080/TCP 55d
db ClusterIP None <none> 5432/TCP 55d
[funkypenguin:~] %
```
### Create ingress
The ingress resource tells Traefik what to forward inbound requests for *miniflux.example.com* to your service (defined above), which in turn passes the request to the "app" pod. Adjust the config below for your domain.
```bash
cat <<EOF > /var/data/miniflux/ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: app
namespace: miniflux
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: miniflux.example.com
http:
paths:
- backend:
serviceName: app
servicePort: 8080
EOF
kubectl create -f /var/data/miniflux/ingress.yml
```
Check that your service is deployed, with ```kubectl get ingress -n miniflux```. You should see something like this:
```bash
[funkypenguin:~] 130 % kubectl get ingress -n miniflux
NAME HOSTS ADDRESS PORTS AGE
app miniflux.funkypenguin.co.nz 80 55d
[funkypenguin:~] %
```
### Access Miniflux
At this point, you should be able to access your instance on your chosen DNS name (*i.e. <https://miniflux.example.com>*)
### Troubleshooting
To look at the Miniflux pod's logs, run ```kubectl logs -n miniflux <name of pod per above> -f```. For further troubleshooting hints, see [Troubleshooting](/reference/kubernetes/troubleshooting/).
--8<-- "recipe-footer.md"

77
docs/kubernetes/cluster/digitalocean.md Normal file
View File

@@ -0,0 +1,77 @@
---
description: Creating a Kubernetes cluster on DigitalOcean
---
# Kubernetes on DigitalOcean
IMO, the easiest Kubernetes cloud provider to experiment with is [DigitalOcean](https://m.do.co/c/e33b78ad621b) (_this is a referral link_). I've included instructions below to start a basic cluster.
![Kubernetes on Digital Ocean](/images/kubernetes-on-digitalocean.jpg)
## Ingredients
1. [DigitalOcean](https://www.digitalocean.com/?refcode=e33b78ad621b) account, either linked to a credit card or (_my preference for a trial_) topped up with $5 credit from PayPal. (_yes, this is a referral link, making me some 💰 to buy 🍷_)
2. Geek-Fu required : 🐱 (easy - even has screenshots!)
## Preparation
### Create DigitalOcean Account
Create a project, and then from your project page, click **Manage** -> **Kubernetes (LTD)** in the left-hand panel:
![Kubernetes on Digital Ocean Screenshot #1](/images/kubernetes-on-digitalocean-screenshot-1.png){ loading=lazy }
Until DigitalOcean considers their Kubernetes offering to be "production ready", you'll need the additional step of clicking on **Enable Limited Access**:
![Kubernetes on Digital Ocean Screenshot #2](/images/kubernetes-on-digitalocean-screenshot-2.png){ loading=lazy }
The _Enable Limited Access_ button changes to read _Create a Kubernetes Cluster_ . Cleeeek it:
![Kubernetes on Digital Ocean Screenshot #3](/images/kubernetes-on-digitalocean-screenshot-3.png){ loading=lazy }
When prompted, choose some defaults for your first node pool (_your pool of "compute" resources for your cluster_), and give it a name. In more complex deployments, you can use this concept of "node pools" to run certain applications (_like an inconsequential nightly batch job_) on a particular class of compute instance (_such as cheap, preemptible instances_)
![Kubernetes on Digital Ocean Screenshot #4](/images/kubernetes-on-digitalocean-screenshot-4.png){ loading=lazy }
That's it! Have a sip of your 🍷, a bite of your :cheese:, and wait for your cluster to build. While you wait, follow the instructions to setup kubectl (if you don't already have it)
![Kubernetes on Digital Ocean Screenshot #5](/images/kubernetes-on-digitalocean-screenshot-5.png){ loading=lazy }
DigitalOcean will provide you with a "kubeconfig" file to use to access your cluster. It's at the bottom of the page (_illustrated below_), and easy to miss (_in my experience_).
![Kubernetes on Digital Ocean Screenshot #6](/images/kubernetes-on-digitalocean-screenshot-6.png){ loading=lazy }
## Release the kubectl!
Save your kubeconfig file somewhere, and test it our by running ```kubectl --kubeconfig=<PATH TO KUBECONFIG> get nodes``` [^1]
Example output:
```bash
[davidy:~/Downloads] 130 % kubectl --kubeconfig=penguins-are-the-sexiest-geeks-kubeconfig.yaml get nodes
NAME STATUS ROLES AGE VERSION
festive-merkle-8n9e Ready <none> 20s v1.13.1
[davidy:~/Downloads] %
```
In the example above, my nodes were being deployed. Repeat the command to see your nodes spring into existence:
```bash
[davidy:~/Downloads] % kubectl --kubeconfig=penguins-are-the-sexiest-geeks-kubeconfig.yaml get nodes
NAME STATUS ROLES AGE VERSION
festive-merkle-8n96 Ready <none> 6s v1.13.1
festive-merkle-8n9e Ready <none> 34s v1.13.1
[davidy:~/Downloads] %
[davidy:~/Downloads] % kubectl --kubeconfig=penguins-are-the-sexiest-geeks-kubeconfig.yaml get nodes
NAME STATUS ROLES AGE VERSION
festive-merkle-8n96 Ready <none> 30s v1.13.1
festive-merkle-8n9a Ready <none> 17s v1.13.1
festive-merkle-8n9e Ready <none> 58s v1.13.1
[davidy:~/Downloads] %
```
That's it. You have a beautiful new kubernetes cluster ready for some action!
[^1]: Do you live in the CLI? Install the kubectl autocompletion for [bash](https://kubernetes.io/docs/tasks/tools/included/optional-kubectl-configs-bash-linux/) or [zsh](https://kubernetes.io/docs/tasks/tools/included/optional-kubectl-configs-zsh/) to make your life much easier!
--8<-- "recipe-footer.md"

65
docs/kubernetes/cluster/index.md Normal file
View File

@@ -0,0 +1,65 @@
---
title: How to choose a managed Kubernetes cluster vs build your own
description: So you want to play with Kubernetes? The first decision you need to make is how your cluster will run. Do you choose (and pay a premium for) a managed Kubernetes cloud provider, or do you "roll your own" with kubeadm on bare-metal, VMs, or k3s?
---
# Kubernetes Cluster
There are an ever-increasing amount of ways to deploy and run Kubernetes. The primary distinction to be aware of is whether to fork out for a managed Kubernetes instance or not. Managed instances have some advantages, which I'll detail below, but these come at additional cost.
## Managed (Cloud Provider)
### Popular Options
Popular options are:
* [DigitalOcean](/kubernetes/cluster/digitalocean/)
* Google Kubernetes Engine (GKE)
* Amazon Elastic Kubernetes Service (EKS)
* Azure Kubernetes Service (AKS)
### Upgrades
A managed Kubernetes provider will typically provide a way to migrate to pre-tested and trusted versions of Kuberenetes, as they're released and then tested. This [doesn't mean that upgrades will be trouble-free](https://www.digitalocean.com/community/tech_talks/20-000-upgrades-later-lessons-from-a-year-of-managed-kubernetes-upgrades), but they're likely to be less of a PITA. With Kubernetes' 4-month release cadence, you'll want to keep an eye on updates, and avoid becoming too out-of-date.
### Horizontal Scaling
One of the key drawcards for Kubernetes is horizonal scaling. You want to be able to expand/contract your cluster as your workloads change, even if just for one day a month. Doing this on your own hardware is.. awkward.
### Load Balancing
Even if you had enough hardware capacity to handle any unexpected scaling requirements, ensuring that traffic can reliably reach your cluster is a complicated problem. You need to present a "virtual" IP for external traffic to ingress the cluster on. There are popular solutions to provide LoadBalancer services to a self-managed cluster (*i.e., [MetalLB](/kubernetes/loadbalancer/metallb/)*), but they do represent extra complexity, and won't necessarily be resilient to outages outside of the cluster (*network devices, power, etc*).
### Storage
Cloud providers make it easy to connect their storage solutions to your cluster, but you'll pay as you scale, and in most cases, I/O on cloud block storage is throttled along with your provisioned size. (*So a 1Gi volume will have terrible IOPS compared to a 100Gi volume*)
### Services
Some things just "work better" in a cloud provider environment. For example, to run a highly available Postgres instance on Kubernetes requires at least 3 nodes, and 3 x storage, plus manual failover/failback in the event of an actual issue. This can represent a huge cost if you simply need a PostgreSQL database to provide (*for example*) a backend to an authentication service like Keycloak. Cloud providers will have a range of managed database solutions which will cost far less than do-it-yourselfing, and integrate easily and securely into their kubernetes offerings.
### Summary
Go with a managed provider if you want your infrastructure to be resilient to your own hardware/connectivity issues. I.e., there's a material impact to a power/network/hardware outage, and the cost of the managed provider is less than the cost of an outage.
## DIY (Cloud Provider, Bare Metal, VMs)
### Popular Options
Popular options are:
* Rancher's K3s
* Ubuntu's Charmed Kubernetes
### Flexible
With self-hosted Kubernetes, you're free to mix/match your configuration as you see fit. You can run a single k3s node on a raspberry pi, or a fully HA pi-cluster, or a handful of combined master/worker nodes on a bunch of proxmox VMs, or on plain bare-metal.
### Education
You'll learn more about how to care for and feed your cluster if you build it yourself. But you'll definately spend more time on it, and it won't always be when you expect!
### Summary
Go with a self-hosted cluster if you want to learn more, you'd rather spend time than money, or you've already got significant investment in local infructure and technical skillz.
--8<-- "recipe-footer.md"

161
docs/kubernetes/cluster/k3s.md Normal file
View File

@@ -0,0 +1,161 @@
---
title: Quickly (and simply) create a k8s cluster with k3s
description: Creating a Kubernetes cluster on k3s
---
# Deploy your k8s cluster on k3s
If you're wanting to self-host your own Kubernetes cluster, one of the simplest and most widely-supported approach is Rancher's [k3s](https://k3s.io/).
## Why k3s vs k8s?
!!! question "k3s vs k8s - which is better to start with?"
**Question**: If you're wanting to learn about Kubernetes, isn't it "better" to just jump into the "deep end", and use "full" k8s? Is k3s a "lite" version of k8s?
**Answer**: It depends on what you want to learn. If you want to deep-dive into the interaction between the apiserver, schedule, etcd and SSL certificates, then k3s will hide much of this from you, and you'd probably prefer to learn [Kubernetes The Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way). If, however, you want to learn how to **drive** Kubernetes as an operator / user, then k3s abstracts a lot of the (*unnecessary?*) complexity around cluster setup, bootstrapping, and upgrading.
Some of the "let's-just-get-started" advantages to k3s are:
* Packaged as a single binary.
* Lightweight storage backend based on sqlite3 as the default storage mechanism. etcd3, MySQL, Postgres also still available.
* Simple but powerful “batteries-included” features have been added, such as: a local storage provider, a service load balancer, a Helm controller, and the Traefik ingress controller (I prefer to leave some of these out)
## k3s requirements
!!! summary "Ingredients"
* [ ] One or more "modern" Linux hosts to serve as cluster masters. (*Using an odd number of masters is required for HA*). Additional steps are required for [Raspbian Buster](https://rancher.com/docs/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster), [Alpine](https://rancher.com/docs/k3s/latest/en/advanced/#additional-preparation-for-alpine-linux-setup), or [RHEL/CentOS](https://rancher.com/docs/k3s/latest/en/advanced/#additional-preparation-for-red-hat-centos-enterprise-linux).
* [ ] Ensure you have sudo access to your nodes, and that each node meets the [installation requirements](https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/).
Optional:
* [ ] Additional hosts to serve as cluster agents (*assuming that not everybody gets to be a master!*)
!!! question "Which host OS to use for k8s?"
Strictly, it doesn't matter. I prefer the latest Ubuntu LTS server version, but that's because I like to standardize my toolset across different clusters / platforms - I find this makes it easier to manage the "cattle" :cow: over time!
## k3s single node setup
If you only want a single-node k3s cluster, then simply run the following to do the deployment:
```bash
MYSECRET=iambatman
curl -fL https://get.k3s.io | K3S_TOKEN=${MYSECRET} \
sh -s - --disable traefik server
```
!!! question "Why no k3s traefik?"
k3s comes with the traefik ingress "built-in", so why not deploy it? Because we'd rather deploy it **later** (*if we even want it*), using the same [deployment strategy](/kubernetes/deployment/flux/) which we use with all of our other services, so that we can easily update/configure it.
## k3s multi master setup
### Deploy first master
You may only have one node now, but it's a good idea to prepare for future expansion by bootstrapping k3s in "embedded etcd" multi master HA mode. Pick a secret to use for your server token, and run the following:
```bash
MYSECRET=iambatman
curl -fL https://get.k3s.io | K3S_TOKEN=${MYSECRET} \
sh -s - --disable traefik --disable servicelb server --cluster-init
```
!!! question "y no servicelb or k3s traefik?"
K3s includes a [rudimentary load balancer](/kubernetes/loadbalancer/k3s/) which utilizes host ports to make a given port available on all nodes. If you plan to deploy one, and only one k3s node, then this is a viable configuration, and you can leave out the `--disable servicelb` text above. If you plan for more nodes and you want to run k3s HA though, then you're better off deploying [MetalLB](/kubernetes/loadbalancer/metallb/) to do "real" loadbalancing.
You should see output which looks something like this:
```bash
root@shredder:~# curl -fL https://get.k3s.io | K3S_TOKEN=${MYSECRET} \
> sh -s - --disable traefik server --cluster-init
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 27318 100 27318 0 0 144k 0 --:--:-- --:--:-- --:--:-- 144k
[INFO] Finding release for channel stable
[INFO] Using v1.21.5+k3s2 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.21.5+k3s2/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.21.5+k3s2/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
root@shredder:~#
```
Provided the last line of output says `Starting k3s` and not something more troublesome-sounding.. you have a cluster! Run `k3s kubectl get nodes -o wide` to confirm this, which has the useful side-effect of printing out your first master's IP address (*which we'll need for the next step*)
```bash
root@shredder:~# k3s kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
shredder Ready control-plane,etcd,master 83s v1.21.5+k3s2 192.168.39.201 <none> Ubuntu 20.04.3 LTS 5.4.0-70-generic containerd://1.4.11-k3s1
root@shredder:~#
```
!!! tip "^Z undo undo ..."
Oops! Did you mess something up? Just run `k3s-uninstall.sh` to wipe all traces of K3s, and start over!
### Deploy other k3s master nodes (optional)
Now that the first master is deploy, add additional masters (*remember to keep the total number of masters to an odd number*) by referencing the secret, and the IP address of the first master, on all the others:
```bash
MYSECRET=iambatman
curl -fL https://get.k3s.io | K3S_TOKEN=${MYSECRET} \
sh -s - server --disable servicelb --server https://<IP OF FIRST MASTER>:6443
```
Run `k3s kubectl get nodes` to see your new master node make friends with the others:
```bash
root@shredder:~# k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
bebop Ready control-plane,etcd,master 4m13s v1.21.5+k3s2
rocksteady Ready control-plane,etcd,master 4m42s v1.21.5+k3s2
shredder Ready control-plane,etcd,master 8m54s v1.21.5+k3s2
root@shredder:~#
```
### Deploy k3s worker nodes (optional)
If you have more nodes which you want _not_ to be considered masters, then run the following on each. Note that the command syntax differs slightly from the masters (*which is why k3s deploys this as k3s-agent instead*)
```bash
MYSECRET=iambatman
curl -fL https://get.k3s.io | K3S_TOKEN=${MYSECRET} \
K3S_URL=https://<IP OF FIRST MASTER>:6443 \
sh -s -
```
!!! question "y no kubectl on k3s-agent?"
If you tried to run `k3s kubectl` on an agent, you'll notice that it returns an error about `localhost:8080` being refused. This is **normal**, and it happens because agents aren't necessarily "trusted" to the same degree that masters are, and so the cluster admin credentials are **not** saved to the filesystem, as they are with masters.
!!! tip "^Z undo undo ..."
Oops! Did you mess something up? Just run `k3s-agent-uninstall.sh` to wipe all traces of K3s agent, and start over!
## Cuddle your cluster with k3s kubectl!
k3s will have saved your kubeconfig file on the masters to `/etc/rancher/k3s/k3s.yaml`. This file contains the necessary config and certificates to administer your cluster, and should be treated with the same respect and security as your root password. To interact with the cluster, you need to tell the kubectl command where to find this `KUBECONFIG` file. There are a few ways to do this...
1. Prefix your `kubectl` commands with `k3s`. i.e., `kubectl cluster-info` becomes `k3s kubectl cluster-info`
2. Update your environment variables in your shell to set `KUBECONFIG` to `/etc/rancher/k3s/k3s.yaml`
3. Copy ``/etc/rancher/k3s/k3s.yaml` to `~/.kube/config`, which is the default location `kubectl` will look for
Cuddle your beautiful new cluster by running `kubectl cluster-info` [^1] - if that doesn't work, check your k3s logs[^2].
[^1]: Do you live in the CLI? Install the kubectl autocompletion for [bash](https://kubernetes.io/docs/tasks/tools/included/optional-kubectl-configs-bash-linux/) or [zsh](https://kubernetes.io/docs/tasks/tools/included/optional-kubectl-configs-zsh/) to make your life much easier!
[^2]: Looking for your k3s logs? Under Ubuntu LTS, run `journalctl -u k3s` to show your logs
[^3]: k3s is not the only "lightweight kubernetes" game in town. Minikube (*virtualization-based*) and mikrok8s (*possibly better for Ubuntu users since it's installed in a "snap" - haha*) are also popular options. One day I'll write a "mikrok8s vs k3s" review, but it doesn't really matter for our cluster operations - as I understand it, microk8s makes HA clustering slightly easire than k3s, but you get slightly less "out-of-the-box" in return, so mikrok8s may be more suitable for experience users / production edge deployments.
--8<-- "recipe-footer.md"

66
docs/kubernetes/deployment/flux/design.md Normal file
View File

@@ -0,0 +1,66 @@
---
title: Using fluxcd/fluxv2 to deploy from helm with GitOps
description: This page details a deployment design pattern which facilitates the use of fluxcd/fluxv2 to provided a tiered structure of helm releases, so that you can manage your cluster services via GitOps using a single repository.
---
# Using helm with GitOps via fluxv2/fluxv2
!!! question "Shouldn't a design **precede** installation instructions?"
In this case, I felt that an [installation](/kubernetes/deployment/flux/install/) and a practical demonstration upfront, would help readers to understand the flux design, and make it simpler to then explain how to [operate](/kubernetes/deployment/flux/operate/) flux themselves! 💪
Flux is power and flexible enough to fit many use-cases. After some experience and dead-ends, I've worked out a way to deploy Flux with enough flexibility but structure to make it an almost-invisible part of how my cluster "just works" on an ongoing basis..
## Illustration
Consider this entity relationship diagram:
``` mermaid
erDiagram
repo-path-flux-system ||..|{ app-namespace : "contains yaml for"
repo-path-flux-system ||..|{ app-kustomization : "contains yaml for"
repo-path-flux-system ||..|{ helmrepositories : "contains yaml for"
app-kustomization ||..|| repo-path-app : "points flux at"
flux-system-kustomization ||..|| repo-path-flux-system : "points flux at"
repo-path-app ||..|{ app-helmreleases: "contains yaml for"
repo-path-app ||..|{ app-configmap: "contains yaml for"
repo-path-app ||..|o app-sealed-secrets: "contains yaml for"
app-configmap ||..|| app-helmreleases : configures
helmrepositories ||..|| app-helmreleases : "host charts for"
app-helmreleases ||..|{ app-containers : deploys
app-containers }|..|o app-sealed-secrets : references
```
## Description
And here's what it all means, starting from the top...
1. The flux-system **Kustomization** tells flux to look in the repo in `/flux-system`, and apply any YAMLs it finds (*with optional kustomize templating, if you're an uber-ninja!*).
2. Within `/bootstrap`, we've defined (for convenience), 3 subfolders, containing YAML for:
1. `namespaces` : Any other **Namespaces** we want to deploy for our apps
2. `helmrepositories` : Any **HelmRepositories** we later want to pull helm charts from
3. `kustomizations` : An **Kustomizations** we need to tell flux to import YAMLs from **elsewhere** in the repository
3. In turn, each app's **Kustomization** (*which we just defined above*) tells flux to look in the repo in the `/<app name>` path, and apply any YAMLs it finds (*with optional kustomize templating, if you're an uber-ninja!*).
4. Within the `/<app name>` path, we define **at least** the following:
1. A **HelmRelease** for the app, telling flux which version of what chart to apply from which **HelmRepository**
2. A **ConfigMap** for the HelmRelease, which contains all the custom (*and default!*) values for the chart
5. Of course, we can also put any **other** YAML into the `/<app name>` path in the repo, which may include additional ConfigMaps, SealedSecrets (*for safely storing secrets in a repo*), Ingresses, etc.
!!! question "That seems overly complex!"
> "Why not just stick all the YAML into one folder and let flux reconcile it all-at-once?"
Several reasons:
* We need to be able to deploy multiple copies of the same helm chart into different namespaces. Imagine if you wanted to deploy a "postgres" helm chart into a namespace for Keycloak, plus another one for NextCloud. Putting each HelmRelease resource into its own namespace allows us to do this, while sourcing them all from a common HelmRepository
* As your cluster grows in complexity, you end up with dependency issues, and sometimes you need one chart deployed first, in order to create CRDs which are depended upon by a second chart (*like Prometheus' ServiceMonitor*). Isolating apps to a kustomization-per-app means you can implement dependencies and health checks to allow a complex cluster design without chicken vs egg problems!
## Got it?
Good! I describe how to put this design into action on the [next page](/kubernetes/deployment/flux/operate/)...
[^1]: ERDs are fancy diagrams for nERDs which [represent cardinality between entities](https://en.wikipedia.org/wiki/Entity%E2%80%93relationship_model#Crow's_foot_notation) scribbled using the foot of a crow 🐓
--8<-- "recipe-footer.md"

11
docs/kubernetes/deployment/flux/index.md Normal file
View File

@@ -0,0 +1,11 @@
---
title: Using flux for deployment in Kubernetes
---
# Deployment
In a break from tradition, the flux design is best understood *after* installing it, so this section makes the most sense read in the following order:
1. [Install](/kubernetes/deployment/flux/install/)
2. [Design](/kubernetes/deployment/flux/design/)
3. [Operate](/kubernetes/deployment/flux/operate/)

136
docs/kubernetes/deployment/flux/install.md Normal file
View File

@@ -0,0 +1,136 @@
---
title: Install fluxcd/fluxv2 with custom layout for future flexibility
description: Installing fluxcd/fluxv2 is a trivial operation, but sometimes the defaults are not conducive to future flexibility. This pages details a few tricks to ensure Flux can be easily managed / extended over time.
---
# Install fluxcd/fluxv2
[Flux](https://fluxcd.io/) is a set of continuous and progressive delivery solutions for Kubernetes that are open and extensible.
Using flux to manage deployments into the cluster means:
1. All change is version-controlled (*i.e. "GitOps"*)
2. It's not necessary to expose the cluster API (*i.e., which would otherwise be the case if you were using CI*)
3. Deployments can be paused, rolled back, examine, debugged using Kubernetes primitives and tooling
!!! summary "Ingredients"
* [x] [Install the flux CLI tools](https://fluxcd.io/docs/installation/#install-the-flux-cli) on a host which has access to your cluster's apiserver.
* [x] Create a GitHub [personal access token](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line) that can create repositories by checking all permissions under repo.
* [x] Create a private GitHub repository dedicated to your flux deployments
## Fluxv2 components
Here's a simplified way to think about the various flux components..
1. You need a source for flux to look at. This is usually a Git repository, although it can also be a helm repository, an S3 bucket. A source defines the entire repo (*not a path or a folder structure*).
2. Within your source, you define one or more kustomizations. Each kustomization is a _location_ on your source (*i.e., myrepo/nginx*) containing YAML files to be applied directly to the API server.
3. The YAML files inside the kustomization include:
1. HelmRepositories (*think of these as the repos you'd add to helm with `helm repo`*)
2. HelmReleases (*these are charts which live in HelmRepositories*)
3. Any other valid Kubernetes YAML manifests (*i.e., ConfigMaps, etc)*
## Preparation
### Install flux CLI
This section is a [direct copy of the official docs](https://fluxcd.io/docs/installation/#install-the-flux-cli), to save you having to open another tab..
=== "HomeBrew (MacOS/Linux)"
With [Homebrew](https://brew.sh/) for macOS and Linux:
```bash
brew install fluxcd/tap/flux
```
=== "Bash (MacOS/Linux)"
With Bash for macOS and Linux:
```bash
curl -s https://fluxcd.io/install.sh | sudo bash
```
=== "Chocolatey"
With [Chocolatey](https://chocolatey.org/) for Windows:
```bash
choco install flux
```
### Create GitHub Token
Create a GitHub [personal access token](https://github.com/settings/tokens) that can create repositories by checking all permissions under repo, as well as all options under `admin:public_key `. (*we'll use the token in the bootstrapping step below*)
### Create GitHub Repo
Now we'll create a repo for flux - it can (*and probably should!*) be private. I've created a [template repo to get you started](https://github.com/geek-cookbook/template-flux/generate), but you could simply start with a blank repo too (*although you'll need at least a `bootstrap` directory included or the command below will fail*).[^1]
### Bootstrap Flux
Having prepared all of the above, we're now ready to deploy flux. Before we start, take a look at all the running pods in the cluster, with `kubectl get pods -A`. You should see something like this...
```bash
root@shredder:~# k3s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-7448499f4d-qfszx 1/1 Running 0 6m32s
kube-system local-path-provisioner-5ff76fc89d-rqh52 1/1 Running 0 6m32s
kube-system metrics-server-86cbb8457f-25688 1/1 Running 0 6m32s
```
Now, run a customized version of the following:
```bash
GITHUB_TOKEN=<your-token>
flux bootstrap github \
--owner=my-github-username \
--repository=my-github-username/my-repository \
--personal \
--path bootstrap
```
Once the flux bootstrap is completed without errors, list the pods in the cluster again, with `kubectl get pods -A`. This time, you see something like this:
```bash
root@shredder:~# k3s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
flux-system helm-controller-f7c5b6c56-nk7rm 1/1 Running 0 5m48s
flux-system kustomize-controller-55db56f44f-4kqs2 1/1 Running 0 5m48s
flux-system notification-controller-77f68bf8f4-9zlw9 1/1 Running 0 5m48s
flux-system source-controller-8457664f8f-8qhhm 1/1 Running 0 5m48s
kube-system coredns-7448499f4d-qfszx 1/1 Running 0 15m
kube-system local-path-provisioner-5ff76fc89d-rqh52 1/1 Running 0 15m
kube-system metrics-server-86cbb8457f-25688 1/1 Running 0 15m
traefik svclb-traefik-ppvhr 2/2 Running 0 5m31s
traefik traefik-f48b94477-d476p 1/1 Running 0 5m31s
root@shredder:~#
```
### What just happened?
Flux installed its controllers into the `flux-system` namespace, and created two new objects:
1. A **GitRepository** called `flux-system`, pointing to your GitHub repo.
2. A **Kustomization** called `flux-system`, pointing to the `flux-system` directory in the above repo.
If you used my template repo, some extra things also happened..
1. I'd pre-populated the `bootstrap` directory in the template repo with 3 folders:
1. [helmrepositories](https://github.com/geek-cookbook/template-flux/tree/main/bootstrap/helmrepositories), for storing repositories used for deploying helm charts
2. [kustomizations](https://github.com/geek-cookbook/template-flux/tree/main/bootstrap/kustomizations), for storing additional kustomizations *(which in turn can reference other paths in the repo*)
3. [namespaces](https://github.com/geek-cookbook/template-flux/tree/main/bootstrap/namespaces), for storing namespace manifests (*since these need to exist before we can deploy helmreleases into them*)
2. Because the `bootstrap` Kustomization includes everything **recursively** under `bootstrap` path in the repo, all of the above were **also** applied to the cluster
3. I'd pre-prepared a [Namespace](https://github.com/geek-cookbook/template-flux/blob/main/bootstrap/namespaces/namespace-podinfo.yaml), [HelmRepository](https://github.com/geek-cookbook/template-flux/blob/main/bootstrap/helmrepositories/helmrepository-podinfo.yaml), and [Kustomization](https://github.com/geek-cookbook/template-flux/blob/main/bootstrap/kustomizations/kustomization-podinfo.yaml) for "podinfo", a simple example application, so these were applied to the cluster
4. The kustomization we added for podinfo refers to the `/podinfo` path in the repo, so everything in **this** folder was **also** applied to the cluster
5. In the `/podinfo` path of the repo is a [HelmRelease](https://github.com/geek-cookbook/template-flux/blob/main/podinfo/helmrelease-podinfo.yaml) (*an object describing how to deploy a helm chart*), and a [ConfigMap](https://github.com/geek-cookbook/template-flux/blob/main/podinfo/configmap-pofinfo-helm-chart-value-overrides-configmap.yaml) (*which ontain the `values.yaml` for the podinfo helm chart*)
6. Flux recognized the podinfo **HelmRelease**, applied it along with the values in the **ConfigMap**, and consequently we have podinfo deployed from the latest helm chart, into the cluster, and managed by Flux! 💪
## Wait, but why?
That's best explained on the [next page](/kubernetes/deployment/flux/design/), describing the design we're using...
--8<-- "recipe-footer.md"
[^1]: The [template repo](https://github.com/geek-cookbook/template-flux/) also "bootstraps" a simple example re how to [operate flux](/kubernetes/deployment/flux/operate/), by deploying the podinfo helm chart.

160
docs/kubernetes/deployment/flux/operate.md Normal file
View File

@@ -0,0 +1,160 @@
---
title: Using fluxcd/fluxv2 to "GitOps" multiple helm charts from a single repository
description: Having described a installation and design pattern for fluxcd/fluxv2, this page describes how to _use_ and extend the design to manage multilpe helm releases in your cluster, from a single repository.
---
# Operate fluxcd/fluxv2 from a single repository
Having described [how to install flux](/kubernetes/deployment/flux/install/), and [how our flux deployment design works](/kubernetes/deployment/flux/design/), let's finish by exploring how to **use** flux to deploy helm charts into a cluster!
## Deploy App
We'll need 5 files per-app, to deploy and manage our apps using flux. The example below will use the following highlighted files:
```hl_lines="4 6 8 10 11"
├── README.md
├── bootstrap
│   ├── flux-system
│   ├── helmrepositories
│   │   └── helmrepository-podinfo.yaml
│   ├── kustomizations
│   │   └── kustomization-podinfo.yaml
│   └── namespaces
│   └── namespace-podinfo.yaml
└── podinfo
├── configmap-podinfo-helm-chart-value-overrides.yaml
└── helmrelease-podinfo.yaml
```
???+ question "5 files! That seems overly complex!"
> "Why not just stick all the YAML into one folder and let flux reconcile it all-at-once?"
Several reasons:
* We need to be able to deploy multiple copies of the same helm chart into different namespaces. Imagine if you wanted to deploy a "postgres" helm chart into a namespace for Keycloak, plus another one for NextCloud. Putting each HelmRelease resource into its own namespace allows us to do this, while sourcing them all from a common HelmRepository
* As your cluster grows in complexity, you end up with dependency issues, and sometimes you need one chart deployed first, in order to create CRDs which are depended upon by a second chart (*like Prometheus' ServiceMonitor*). Isolating apps to a kustomization-per-app means you can implement dependencies and health checks to allow a complex cluster design without chicken vs egg problems!
* I like to use the one-object-per-yaml-file approach. Kubernetes is complex enough without trying to define multiple objects in one file, or having confusingly-generic filenames such as `app.yaml`! 🤦‍♂️
### Identify target helm chart
Identify your target helm chart. Let's take podinfo as an example. Here's the [official chart](https://github.com/stefanprodan/podinfo/tree/master/charts/podinfo), and here's the [values.yaml](https://github.com/stefanprodan/podinfo/tree/master/charts/podinfo/values.yaml) which describes the default values passed to the chart (*and the options the user has to make changes*).
### Create HelmRepository
The README instructs users to add the repo "podinfo" with the URL `ttps://stefanprodan.github.io/podinfo`, so
create a suitable HelmRepository YAML in `bootstrap/helmrepositories/helmrepository-podinfo.yaml`. Here's [my example](https://github.com/geek-cookbook/template-flux/blob/main/bootstrap/helmrepositories/helmrepository-podinfo.yaml).
!!! question "Why such obtuse file names?"
> Why not just call the HelmRepository YAML `podinfo.yaml`? Why prefix the filename with the API object `helmrepository-`?
We're splitting the various "bits" which define this app into multiple YAMLs, and we'll soon have multiple apps in our repo, each with their own set of "bits". It gets very confusing quickly, when comparing git commit diffs, if you're not explicitly clear on what file you're working on, or which changes you're reviewing. Plus, adding the API object name to the filename provides extra "metadata" to the file structure, and makes "fuzzy searching" for quick-opening of files in tools like VSCode more effective.
### Create Namespace
Create a namespace for the chart. Typically you'd name this the same as your chart name. Here's [my namespace-podinfo.yaml](https://github.com/geek-cookbook/template-flux/blob/main/bootstrap/namespaces/namespace-podinfo.yaml).
??? example "Here's an example Namespace..."
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: podinfo
```
### Create Kustomization
Create a kustomization for the chart, pointing flux to a path in the repo where the chart-specific YAMLs will be found. Here's my [kustomization-podinfo.yaml](https://github.com/geek-cookbook/template-flux/blob/main/bootstrap/kustomizations/kustomization-podinfo.yaml).
??? example "Here's an example Kustomization..."
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: podinfo
namespace: flux-system
spec:
interval: 15m
path: podinfo
prune: true # remove any elements later removed from the above path
timeout: 2m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: podinfo
namespace: podinfo
```
### Create HelmRelease
Now create a HelmRelease for the chart - the HelmRelease defines how the (generic) chart from the HelmRepository will be installed into our cluster. Here's my [podinfo/helmrelease-podinfo.yaml](https://github.com/geek-cookbook/template-flux/blob/main/podinfo/helmrelease-podinfo.yaml).
??? example "Here's an example HelmRelease..."
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: podinfo
namespace: podinfo
spec:
chart:
spec:
chart: podinfo # Must be the same as the upstream chart name
version: 10.x # Pin to semver major versions to avoid breaking changes but still get bugfixes/updates
sourceRef:
kind: HelmRepository
name: podinfo # References the HelmRepository you created earlier
namespace: flux-system # All HelmRepositories exist in the flux-system namespace
interval: 15m
timeout: 5m
releaseName: podinfo # _may_ be different from the upstream chart name, but could cause confusion
valuesFrom:
- kind: ConfigMap
name: podinfo-helm-chart-value-overrides # Align with the name of the ConfigMap containing all values
valuesKey: values.yaml # This is the default, but best to be explicit for clarity
```
### Create ConfigMap
Finally, create a ConfigMap to be used to pass helm chart values to the chart. Note that it is **possible** to pass values directly in the HelmRelease, but.. it's messy. I find it easier to let the HelmRelease **describe** the release, and to let the configmap **configure** the release. It also makes tracking changes more straightforward.
As a second note, it's strictly only necessary to include in the ConfigMap the values you want to **change** from the chart's defaults. I find this to be too confusing as charts are continually updated by their developers, and this can obsucre valuable options over time. So I place in my ConfigMaps the **entire** contents of the chart's `values.yaml` file, and then I explicitly overwrite the values I want to change.
!!! tip "Making chart updates simpl(er)"
This also makes updating my values for an upstream chart refactor a simple process - I duplicate the ConfigMap, paste-overwrite with the values.yaml for the refactored/updated chart, and compare the old and new versions side-by-side, to ensure I'm still up-to-date.
It's too large to display nicely below, but here's my [podinfo/configmap-podinfo-helm-chart-value-overrides.yaml](https://github.com/geek-cookbook/template-flux/blob/main/podinfo/configmap-podinfo-helm-chart-value-overrides.yaml)
!!! tip "Yes, I am sticking to my super-obtuse file naming convention!"
Doesn't it make it easier to understand, at a glance, exactly what this YAML file is intended to be?
### Commit the changes
Simply commit your changes, sit back, and wait for flux to do its 1-min update. If you like to watch the fun, you could run `watch -n1 flux get kustomizations` so that you'll see the reconciliation take place (*if you're quick*). You can also force flux to check the repo for changes manually, by running `flux reconcile source git flux-system`.
## Making changes
Let's say you decide that instead of 1 replica of the podinfo pod, you'd like 3 replicas. Edit your configmap, and change `replicaCount: 1` to `replicaCount: 3`.
Commit your changes, and once again do the waiting / impatient-reconciling jig. This time you'll have to wait up to 15 minutes though...
!!! question "Why 15 minutes?"
> I thought we check the repo every minute?
Yes, we check the entire GitHub repository for changes every 1 min, and changes to a kustomization are applied immediately. I.e., your podinfo ConfigMap gets updated within a minute (roughly). But the interval value for the HelmRelease is set to 15 minutes, so you could be waiting for as long as 15 minutes for flux to re-reconcile your HelmRelease with the ConfigMap, and to apply any changes. I've found that setting the HelmRelease interval too low causes (a) lots of unnecessary resource usage on behalf of flux, and (b) less stability when you have a large number of HelmReleases, some of whom depend on each other.
You can force a HelmRelease to reconcile, by running `flux reconcile helmrelease -n <namespace> <name of helmrelease>`
## Success!
We did it. The Holy Grail. We deployed an application into the cluster, without touching the cluster. Pinch yourself, and then prove it worked by running `flux get kustomizations`, or `kubectl get helmreleases -n podinfo`.
--8<-- "recipe-footer.md"
[^1]: Got suggestions for improvements here? Shout out in the comments below!

22
docs/kubernetes/deployment/index.md Normal file
View File

@@ -0,0 +1,22 @@
---
description: Kubernetes deployment strategies
---
# Deployment
So far our Kubernetes journey has been fairly linear - your standard "geek follows instructions" sort of deal.
When it comes to a deployment methodology, there are a few paths you can take, and it's possible to "mix-and-match" if you want to (*and if you enjoy extra pain and frustration!*)
Being imperative, Kubernetes is "driven" by your definitions of an intended state. I.e., "*I want a minecraft server and a 3-node redis cluster*". The state is defined by resources (pod, deployment, PVC) etc, which you apply to the Kubernetes apiserver, normally using YAML.
Now you _could_ hand-craft some YAML files, and manually apply these to the apiserver, but there are much smarter and more scalable ways to drive Kubernetes.
The typical methods of deploying applications into Kubernetes, sorted from least to most desirable and safe are:
1. A human applies YAML directly to the apiserver.
2. A human applies a helm chart directly to the apiserver.
3. A human updates a version-controlled set of configs, and a CI process applies YAML/helm chart directly to the apiserver.
4. A human updates a version-controlled set of configs, and a trusted process _within_ the cluster "reaches out" to the config, and applies it to itself.
In our case, #4 is achieved with [Flux](/kubernetes/deployment/flux/).

132
docs/kubernetes/design.md Normal file
View File

@@ -0,0 +1,132 @@
# Design
Like the [Docker Swarm](/docker-swarm/design/) "_private cloud_" design, the Kubernetes design is:
- **Highly-available** (_can tolerate the failure of a single component_)
- **Scalable** (_can add resource or capacity as required_)
- **Portable** (_run it in DigitalOcean today, AWS tomorrow and Azure on Thursday_)
- **Secure** (_access protected with LetsEncrypt certificates_)
- **Automated** (_requires minimal care and feeding_)
_Unlike_ the Docker Swarm design, the Kubernetes design is:
- **Cloud-Native** (_While you **can** [run your own Kubernetes cluster](https://microk8s.io/), it's far simpler to let someone else manage the infrastructure, freeing you to play with the fun stuff_)
- **Complex** (_Requires more basic elements, more verbose configuration, and provides more flexibility and customisability_)
## Design Decisions
### The design and recipes are provider-agnostic**
This means that:
- The design should work on GKE, AWS, DigitalOcean, Azure, or even MicroK8s
- Custom service elements specific to individual providers are avoided
### The simplest solution to achieve the desired result will be preferred**
This means that:
- Persistent volumes from the cloud provider are used for all persistent storage
- We'll do things the "_Kubernetes way_", i.e., using secrets and configmaps, rather than trying to engineer around the Kubernetes basic building blocks.
### Insofar as possible, the format of recipes will align with Docker Swarm**
This means that:
- We use Kubernetes namespaces to replicate Docker Swarm's "_per-stack_" networking and service discovery
## Security
Under this design, the only inbound connections we're permitting to our Kubernetes swarm are:
### Network Flows
- HTTPS (TCP 443) : Serves individual docker containers via SSL-encrypted reverse proxy (_Traefik_)
- Individual additional ports we choose to expose for specific recipes (_i.e., port 8443 for [MQTT](/recipes/mqtt/)_)
### Authentication
- Other than when an SSL-served application provides a trusted level of authentication, or where the application requires public exposure, applications served via Traefik will be protected with an OAuth proxy.
## The challenges of external access
Because we're Cloude-Native now, it's complex to get traffic **into** our cluster from outside. We basically have 3 options:
1. **HostIP**: Map a port on the host to a service. This is analogous to Docker's port exposure, but lacking in that it restricts us to one host port per-container, and it's not possible to anticipate _which_ of your Kubernetes hosts is running a given container. Kubernetes does not have Docker Swarm's "routing mesh", allowing for simple load-balancing of incoming connections.
2. **LoadBalancer**: Purchase a "loadbalancer" per-service from your cloud provider. While this is the simplest way to assure a fixed IP and port combination will always exist for your service, it has 2 significant limitations:
1. Cost is prohibitive, at roughly \$US10/month per port
2. You won't get the _same_ fixed IP for multiple ports. So if you wanted to expose 443 and 25 (_webmail and smtp server, for example_), you'd find yourself assigned a port each on two **unique** IPs, a challenge for a single DNS-based service, like "_mail.batman.com_"
3. **NodePort**: Expose our service as a port (_between 30000-32767_) on the host which happens to be running the service. This is challenging because you might want to expose port 443, but that's not possible with NodePort.
To further complicate options #1 and #3 above, our cloud provider may, without notice, change the IP of the host running your containers (_O hai, Google!_).
Our solution to these challenges is to employ a simple-but-effective solution which places an HAProxy instance in front of the services exposed by NodePort. For example, this allows us to expose a container on 443 as NodePort 30443, and to cause HAProxy to listen on port 443, and forward all requests to our Node's IP on port 30443, after which it'll be forwarded onto our container on the original port 443.
We use a phone-home container, which calls a simple webhook on our haproxy VM, advising HAProxy to update its backend for the calling IP. This means that when our provider changes the host's IP, we automatically update HAProxy and keep-on-truckin'!
Here's a high-level diagram:
![Kubernetes Design](/images/kubernetes-cluster-design.png){ loading=lazy }
## Overview
So what's happening in the diagram above? I'm glad you asked - let's go through it!
### Setting the scene
In the diagram, we have a Kubernetes cluster comprised of 3 nodes. You'll notice that there's no visible master node. This is because most cloud providers will give you "_free_" master node, but you don't get to access it. The master node is just a part of the Kubernetes "_as-a-service_" which you're purchasing.
Our nodes are partitioned into several namespaces, which logically separate our individual recipes. (_I.e., allowing both a "gitlab" and a "nextcloud" namespace to include a service named "db", which would be challenging without namespaces_)
Outside of our cluster (_could be anywhere on the internet_) is a single VM servicing as a load-balancer, running HAProxy and a webhook service. This load-balancer is described in detail, [in its own section](/kubernetes/loadbalancer/), but what's important up-front is that this VM is the **only element of the design for which we need to provide a fixed IP address**.
### 1 : The mosquitto pod
In the "mqtt" namespace, we have a single pod, running 2 containers - the mqtt broker, and a "phone-home" container.
Why 2 containers in one pod, instead of 2 independent pods? Because all the containers in a pod are **always** run on the same physical host. We're using the phone-home container as a simple way to call a webhook on the not-in-the-cluster VM.
The phone-home container calls the webhook, and tells HAProxy to listen on port 8443, and to forward any incoming requests to port 30843 (_within the NodePort range_) on the IP of the host running the container (_and because of the pod, tho phone-home container is guaranteed to be on the same host as the MQTT container_).
### 2 : The Traefik Ingress
In the "default" namespace, we have a Traefik "Ingress Controller". An Ingress controller is a way to use a single port (_say, 443_) plus some intelligence (_say, a defined mapping of URLs to services_) to route incoming requests to the appropriate containers (_via services_). Basically, the Trafeik ingress does what [Traefik does for us under Docker Swarm](/docker-swarm/traefik/).
What's happening in the diagram is that a phone-home pod is tied to the traefik pod using affinity, so that both containers will be executed on the same host. Again, the phone-home container calls a webhook on the HAProxy VM, auto-configuring HAproxy to send any HTTPs traffic to its calling address and customer NodePort port number.
When an inbound HTTPS request is received by Traefik, based on some internal Kubernetes elements (ingresses), Traefik provides SSL termination, and routes the request to the appropriate service (_In this case, either the GitLab UI or teh UniFi UI_)
### 3 : The UniFi pod
What's happening in the UniFi pod is a combination of #1 and #2 above. UniFi controller provides a webUI (_typically 8443, but we serve it via Traefik on 443_), plus some extra ports for device adoption, which are using a proprietary protocol, and can't be proxied with Traefik.
To make both the webUI and the adoption ports work, we use a combination of an ingress for the webUI (_see #2 above_), and a phone-home container to tell HAProxy to forward port 8080 (_the adoption port_) directly to the host, using a NodePort-exposed service.
This allows us to retain the use of a single IP for all controller functions, as accessed outside of the cluster.
### 4 : The webhook
Each phone-home container is calling a webhook on the HAProxy VM, secured with a secret shared token. The phone-home container passes the desired frontend port (i.e., 443), the corresponding NodeIP port (i.e., 30443), and the node's current public IP address.
The webhook uses the provided details to update HAProxy for the combination of values, validate the config, and then restart HAProxy.
### 5 : The user
Finally, the DNS for all externally-accessible services is pointed to the IP of the HAProxy VM. On receiving an inbound request (_be it port 443, 8080, or anything else configured_), HAProxy will forward the request to the IP and NodePort port learned from the phone-home container.
## Move on..
Still with me? Good. Move on to creating your cluster!
- [Start](/kubernetes/) - Why Kubernetes?
- Design (this page) - How does it fit together?
- [Cluster](/kubernetes/cluster/) - Setup a basic cluster
- [Load Balancer](/kubernetes/loadbalancer/) - Setup inbound access
- [Snapshots](/kubernetes/snapshots/) - Automatically backup your persistent data
- [Helm](/kubernetes/helm/) - Uber-recipes from fellow geeks
- [Traefik](/kubernetes/traefik/) - Traefik Ingress via Helm
--8<-- "recipe-footer.md"

1079
docs/kubernetes/external-dns.md Normal file
View File

File diff suppressed because it is too large Load Diff

56
docs/kubernetes/helm.md Normal file
View File

@@ -0,0 +1,56 @@
# Helm
[Helm](https://github.com/helm/helm) is a tool for managing Kubernetes "charts" (_think of it as an uber-polished collection of recipes_). Using one simple command, and by tweaking one simple config file (values.yaml), you can launch a complex stack. There are many publicly available helm charts for popular packages like [elasticsearch](https://github.com/helm/charts/tree/master/stable/elasticsearch), [ghost](https://github.com/helm/charts/tree/master/stable/ghost), [grafana](https://github.com/helm/charts/tree/master/stable/grafana), [mediawiki](https://github.com/helm/charts/tree/master/stable/mediawiki), etc.
![Kubernetes Snapshots](/images/kubernetes-helm.png){ loading=lazy }
## Ingredients
1. [Kubernetes cluster](/kubernetes/cluster/)
2. Geek-Fu required : 🐤 (_easy - copy and paste_)
## Preparation
### Install Helm
This section is from the Helm README:
Binary downloads of the Helm client can be found on [the Releases page](https://github.com/helm/helm/releases/latest).
Unpack the `helm` binary and add it to your PATH and you are good to go!
If you want to use a package manager:
- [Homebrew](https://brew.sh/) users can use `brew install kubernetes-helm`.
- [Chocolatey](https://chocolatey.org/) users can use `choco install kubernetes-helm`.
- [Scoop](https://scoop.sh/) users can use `scoop install helm`.
- [GoFish](https://gofi.sh/) users can use `gofish install helm`.
To rapidly get Helm up and running, start with the [Quick Start Guide](https://helm.sh/docs/intro/quickstart/).
See the [installation guide](https://helm.sh/docs/intro/install/) for more options,
including installing pre-releases.
## Serving
### Initialise Helm
After installing Helm, initialise it by running ```helm init```. This will install "tiller" pod into your cluster, which works with the locally installed helm binaries to launch/update/delete Kubernetes elements based on helm charts.
That's it - not very exciting I know, but we'll need helm for the next and final step in building our Kubernetes cluster - deploying the [Traefik ingress controller (via helm)](/kubernetes/traefik/)!
## Move on..
Still with me? Good. Move on to understanding Helm charts...
- [Start](/kubernetes/) - Why Kubernetes?
- [Design](/kubernetes/design/) - How does it fit together?
- [Cluster](/kubernetes/cluster/) - Setup a basic cluster
- [Load Balancer](/kubernetes/loadbalancer/) Setup inbound access
- [Snapshots](/kubernetes/snapshots/) - Automatically backup your persistent data
- Helm (this page) - Uber-recipes from fellow geeks
- [Traefik](/kubernetes/traefik/) - Traefik Ingress via Helm
[^1]: Of course, you can have lots of fun deploying all sorts of things via Helm. Check out <https://artifacthub.io> for some examples.
--8<-- "recipe-footer.md"

51
docs/kubernetes/index.md Normal file
View File

@@ -0,0 +1,51 @@
---
title: Docker Swarm vs Kubernetes (the winner)
description: I cut my cloud-teeth on Docker swarm, but since swarm is all-but-abandoned by Docker/Mirantis, I'm a happy convert to Kubernetes. Here's why...
---
My first introduction to Kubernetes was a children's story:
<!-- markdownlint-disable MD033 -->
<iframe width="560" height="315" src="https://www.youtube.com/embed/R9-SOzep73w" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
## Why Kubernetes?
Why would you want to Kubernetes for your self-hosted recipes, over simple [Docker Swarm](/docker-swarm/)? Here's my personal take..
### Docker Swarm is dead
Sorry to say, but from where I sit, there's no innovation or development happening in docker swarm.
Yes, I know, after Docker Inc [sold its platform business to Mirantis in Nov 2019](https://www.mirantis.com/blog/mirantis-acquires-docker-enterprise-platform-business/), in Feb 2020 Mirantis [back-tracked](https://www.mirantis.com/blog/mirantis-will-continue-to-support-and-develop-docker-swarm/) on their original plan to sunset swarm after 2 years, and stated that they'd continue to invest in swarm. But seriously, look around. Nobody is interested in swarm right now...
... Not even Mirantis! As of Nov 2021, the Mirantis blog tag "[kubernetes](https://www.mirantis.com/tag/kubernetes/)" had 8 posts within the past month. The tag "[docker](https://www.mirantis.com/tag/docker/)" has 8 posts in the past **2 years**, the 8th being the original announcement of the Docker aquisition. The tag "[docker swarm](https://www.mirantis.com/tag/docker-swarm/)" has only 2 posts, **ever**.
Dead. [Extinct. Like the doodoo](https://youtu.be/NxnZC9L_YXE?t=47).
### Once you go Kubernetes, you can't go back
For years now, [I've provided Kubernetes design consulting](https://www.funkypenguin.co.nz/work-with-me/) to small clients and large enterprises. The implementation details in each case vary widely, but there are some primitives which I've come to take for granted, and I wouldn't easily do without. A few examples:
* **CLI drives API from anywhere**. From my laptop, I can use my credentials to manage any number of Kubernetes clusters, simply by switching kubectl "context". Each interaction is an API call against an HTTPS endpoint. No SSHing to hosts and manually running docker command as root!
* **GitOps is magic**. There are multiple ways to achieve it, but having changes you commit to a repo automatically applied to a cluster, "Just Works(tm)". The process removes so much friction from making changes that it makes you more productive, and a better "gitizen" ;P
* **Controllers are trustworthy**. I've come to trust that when I tell Kubernetes to run 3 replicas on separate hosts, to scale up a set of replicas based on CPU load metrics, or provision a blob of storage for a given workloa, that this will be done in a consistent and visible way. I'll be able to see logs / details for each action taken by the controller, and adjust my own instructions/configuration accordingly if necessary.
## Uggh, it's so complicated!
Yes, it's more complex than Docker Swarm. And that complexity can definately be a barrier, although with improved tooling, it's continually becoming less-so. However, you don't need to be a mechanic to drive a car or to use a chainsaw. You just need a basic understanding of some core primitives, and then you get on with using the tool to achieve your goals, without needing to know every detail about how it works!
Your end-goal is probably "*I want to reliably self-host services I care about*", and not "*I want to fully understand a complex, scalable, and highly sophisticated container orchestrator*". [^1]
So let's get on with learning how to use the tool...
## Mm.. maaaaybe, how do I start?
Primarily you need 2 things:
1. A [cluster](/kubernetes/cluster/)
2. A way to [deploy workloads](/kubernetes/deployment/) into the cluster
Practically, you need some extras too, but you can mix-and-match these.
--8<-- "recipe-footer.md"
[^1]: Of course, if you **do** enjoy understanding the intricacies of how your tools work, you're in good company!

19
docs/kubernetes/ingress/index.md Normal file
View File

@@ -0,0 +1,19 @@
---
description: What is a Kubernetes Ingress?
---
# Ingresses
In Kubernetes, an Ingress is a way to describe how to route traffic coming **into** the cluster, so that (*for example*) <https://radarr.example.com> will end up on a [Radarr][radarr] pod, but <https://sonarr.example.com> will end up on a [Sonarr][sonarr] pod.
![Ingress illustration](/images/ingress.jpg)
There are many popular Ingress Controllers, we're going to cover two equally useful options:
1. [Traefik](/kubernetes/ingress/traefik/)
2. [Nginx](/kubernetes/ingress/nginx/)
Choose at least one of the above (*there may be valid reasons to use both!* [^1]), so that you can expose applications via Ingress.
--8<-- "recipe-footer.md"
[^1]: One cluster I manage uses traefik Traefik for public services, but Nginx for internal management services such as Prometheus, etc. The idea is that you'd need one type of Ingress to help debug problems with the _other_ type!

240
docs/kubernetes/ingress/nginx.md Normal file
View File

@@ -0,0 +1,240 @@
---
title: Install nginx ingress controller into Kubernetes with Flux
---
# Nginx Ingress Controller for Kubernetes - the "flux way"
The [Nginx Ingress Controller](https://kubernetes.github.io/ingress-nginx/) is the grandpappy of Ingress Controllers, with releases dating back ot at least 2016. Of course, Nginx itself is a battle-tested rock, [released in 2004](https://en.wikipedia.org/wiki/Nginx) and has been constantly updated / improved ever since.
Having such a pedigree though can make it a little awkward for the unfamiliar to configure Nginx, whereas something like [Traefik](/kubernetes/ingress/traefik/), being newer-on-the-scene, is more user-friendly, and offers (*among other features*) a free **dashboard**. (*Nginx's dashboard is only available in the commercial Nginx+ package, which is a [monumental PITA](https://www.nginx.com/blog/deploying-nginx-nginx-plus-docker/) to run*)
Nginx Ingress Controller does make for a nice, simple "default" Ingress controller, if you don't want to do anything fancy.
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
* [x] A [load-balancer](/kubernetes/loadbalancer/) solution (*either [k3s](/kubernetes/loadbalancer/k3s/) or [MetalLB](/kubernetes/loadbalancer/metallb/)*)
Optional:
* [x] [Cert-Manager](/kubernetes/ssl-certificates/cert-manager/) deployed to request/renew certificates
* [x] [External DNS](/kubernetes/external-dns/) configured to respond to ingresses, or with a wildcard DNS entry
## Preparation
### Namespace
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/namespaces/namespace-nginx-ingress-controller.yaml`:
??? example "Example NameSpace (click to expand)"
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: nginx-ingress-controller
```
### HelmRepository
Next, we need to define a HelmRepository (*a repository of helm charts*), to which we'll refer when we create the HelmRelease. We only need to do this once per-repository. In this case, we're using the (*prolific*) [bitnami chart repository](https://github.com/bitnami/charts/tree/master/bitnami), so per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/helmrepositories/helmrepository-bitnami.yaml`:
??? example "Example HelmRepository (click to expand)"
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: bitnami
namespace: flux-system
spec:
interval: 15m
url: https://charts.bitnami.com/bitnami
```
### Kustomization
Now that the "global" elements of this deployment (*Namespace and HelmRepository*) have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at `/nginx-ingress-controller`. I create this example Kustomization in my flux repo at `bootstrap/kustomizations/kustomization-nginx-ingress-controller.yaml`:
??? example "Example Kustomization (click to expand)"
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: nginx-ingress-controller
namespace: flux-system
spec:
interval: 15m
path: ./nginx-ingress-controller
prune: true # remove any elements later removed from the above path
timeout: 2m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: nginx-ingress-controller
namespace: nginx-ingress-controller
```
### ConfigMap
Now we're into the nginx-ingress-controller-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/bitnami/charts/blob/master/bitnami/nginx-ingress-controller/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo at `nginx-ingress-controller/configmap-nginx-ingress-controller-helm-chart-value-overrides.yaml`:
??? example "Example ConfigMap (click to expand)"
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: nginx-ingress-controller-helm-chart-value-overrides
namespace: nginx-ingress-controller
data:
values.yaml: |-
# paste chart values.yaml (indented) here and alter as required
```
--8<-- "kubernetes-why-full-values-in-configmap.md"
Then work your way through the values you pasted, and change any which are specific to your configuration. It may not be necessary to change anything.
### HelmRelease
Lastly, having set the scene above, we define the HelmRelease which will actually deploy nginx-ingress-controller into the cluster, with the config and extra ConfigMap we defined above. I save this in my flux repo as `nginx-ingress-controller/helmrelease-nginx-ingress-controller.yaml`:
??? example "Example HelmRelease (click to expand)"
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: nginx-ingress-controller
namespace: nginx-ingress-controller
spec:
chart:
spec:
chart: nginx-ingress-controller
version: 9.x
sourceRef:
kind: HelmRepository
name: bitnami
namespace: flux-system
interval: 15m
timeout: 5m
releaseName: nginx-ingress-controller
valuesFrom:
- kind: ConfigMap
name: nginx-ingress-controller-helm-chart-value-overrides
valuesKey: values.yaml # This is the default, but best to be explicit for clarity
```
--8<-- "kubernetes-why-not-config-in-helmrelease.md"
## Deploy nginx-ingress-controller
Having committed the above to your flux repository, you should shortly see a nginx-ingress-controller kustomization, and in the `nginx-ingress-controller` namespace, a controller and a speaker pod for every node:
```bash
demo@shredder:~$ kubectl get pods -n nginx-ingress-controller
NAME READY STATUS RESTARTS AGE
nginx-ingress-controller-5b849b4fbd-svbxk 1/1 Running 0 24h
nginx-ingress-controller-5b849b4fbd-xt7vc 1/1 Running 0 24h
nginx-ingress-controller-default-backend-867d86fb8f-t27j9 1/1 Running 0 24h
demo@shredder:~$
```
### How do I know it's working?
#### Test Service
By default, the chart will deploy nginx ingress controller's service in [LoadBalancer](/kubernetes/loadbalancer/) mode. When you use kubectl to display the service (`kubectl get services -n nginx-ingress-controller`), you'll see the external IP displayed:
```bash
demo@shredder:~$ kubectl get services -n nginx-ingress-controller
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-controller LoadBalancer 10.152.183.162 172.168.209.1 80:30756/TCP,443:30462/TCP 24h
nginx-ingress-controller-default-backend ClusterIP 10.152.183.200 <none> 80/TCP 24h
demo@shredder:~$
```
!!! question "Where does the external IP come from?"
If you're using [k3s's load balancer](/kubernetes/loadbalancer/k3s/), the external IP will likely be the IP of the the nodes running k3s. If you're using [MetalLB](/kubernetes/loadbalancer/metallb/), the external IP should come from the list of addresses in the pool you allocated.
Pointing your web browser to the external IP displayed should result in the default backend page (*or an nginx-branded 404*). Congratulations, you have external access to the ingress controller! 🥳
#### Test Ingress
Still, you didn't deploy an ingress controller to look at 404 pages! If you used my [template repository](https://github.com/geek-cookbook/template-flux) to start off your [flux deployment strategy](/kubernetes/deployment/flux/), then the podinfo helm chart has already been deployed. By default, the podinfo configmap doesn't deploy an Ingress, but you can change this using the magic of GitOps... 🪄
Edit your podinfo helmrelease configmap (`/podinfo/configmap-podinfo-helm-chart-value-overrides.yaml`), and change `ingress.enabled` to `true`, and set the host name to match your local domain name (*already configured using [External DNS](/kubernetes/external-dns/)*):
``` yaml hl_lines="2 8"
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: podinfo.local
```
To:
``` yaml hl_lines="2 8"
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: podinfo.<your domain name>
```
Commit your changes, wait for a reconciliation, and run `kubectl get ingress -n podinfo`. You should see an ingress created matching the host defined above, and the ADDRESS value should match the service address of the nginx-ingress-controller service.
```bash
root@cn1:~# kubectl get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
podinfo podinfo <none> podinfo.example.com 172.168.209.1 80, 443 91d
```
!!! question "Why is there no class value?"
You don't **have** to define an ingress class if you only have one **class** of ingress, since typically your ingress controller will assume the default class. When you run multiple ingress controllers (say, nginx **and** [traeifk](/kubernetes/ingress/traefik/), or multiple nginx instances with different access controls) then classes become more important.
Now assuming your [DNS is correct](/kubernetes/external-dns/), you should be able to point your browser to the hostname you chose, and see the beautiful podinfo page! 🥳🥳
#### Test SSL
Ha, but we're not done yet! We have exposed a service via our load balancer, we've exposed a route to a service via an Ingress, but let's get rid of that nasty "insecure" message in the browser when using HTTPS...
Since you setup [SSL certificates,](/kubernetes/ssl-certificates/) including [secret-replicator](/kubernetes/ssl-certificates/secret-replicator/), you should end up with a `letsencrypt-wildcard-cert` secret in every namespace, including `podinfo`.
So once again, alter the podinfo ConfigMap to change this:
```yaml hl_lines="2 4"
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
```
To this:
```yaml hl_lines="2 4"
tls:
- secretName: letsencrypt-wildcard-cert
hosts:
- podinfo.<your domain name>
```
Commit your changes, wait for the reconciliation, and the next time you point your browser at your ingress, you should get a beautiful, valid, officially-signed SSL certificate[^1]! 🥳🥳🥳
### Troubleshooting
Are things not working as expected? Watch the nginx-ingress-controller's logs with ```kubectl logs -n nginx-ingress-controller -l app.kubernetes.io/name=nginx-ingress-controller -f```.
--8<-- "recipe-footer.md"
[^1]: The beauty of this design is that the same process will now work for any other application you deploy, without any additional manual effort for DNS or SSL setup!

20
docs/kubernetes/ingress/traefik/dashboard.md Normal file
View File

@@ -0,0 +1,20 @@
---
title: Traefik Ingress Controller's Dashboard
description: Unlike competing ingresses (*cough* nginx *cough*), the beautiful Traefik dashboard UI is free for all.
---
# Traefik Dashboard
One of the advantages [Traefik](/kubernetes/ingress/traefik/) offers over [Nginx](/kubernetes/ingress/nginx/), is a native dashboard available in the open-source version (*Nginx+, the commercially-supported version, also includes a dashboard*).
![Traefik Dashboard Screenshot](/images/traefik-dashboard.png){ loading=lazy }
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
* [x] A [load-balancer](/kubernetes/loadbalancer/) solution (*either [k3s](/kubernetes/loadbalancer/k3s/) or [MetalLB](/kubernetes/loadbalancer/metallb/)*)
* [x] [Traefik](/kubernetes/ingress/traefik/) deployed per-design
--8<-- "recipe-footer.md"
[^1]: The beauty of this design is that the same process will now work for any other application you deploy, without any additional manual effort for DNS or SSL setup!

242
docs/kubernetes/ingress/traefik/index.md Normal file
View File

@@ -0,0 +1,242 @@
---
title: Why I use Traefik Ingress Controller
description: Among other advantages, I no longer need to replicate SSL certificate secrets for nginx-ingress-controller to consume, once-per-namespace!
---
# Traefik Ingress Controller
Unlike grumpy ol' man [Nginx](/kubernetes/ingress/nginx/) :older_man:, Traefik, a microservice-friendly reverse proxy, is relatively fresh in the "cloud-native" space, having been "born" :baby_bottle: [in the same year that Kubernetes was launched](https://techcrunch.com/2020/09/23/five-years-after-creating-traefik-application-proxy-open-source-project-hits-2b-downloads/).
Traefik natively includes some features which Nginx lacks:
* [x] Ability to use cross-namespace TLS certificates (*this may be accidental, but it totally works currently*)
* [x] An elegant "middleware" implementation allowing certain requests to pass through additional layers of authentication
* [x] A beautiful dashboard
![Traefik Screenshot](/images/traefik.png){ loading=lazy }
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
* [x] A [load-balancer](/kubernetes/loadbalancer/) solution (*either [k3s](/kubernetes/loadbalancer/k3s/) or [MetalLB](/kubernetes/loadbalancer/metallb/)*)
Optional:
* [x] [Cert-Manager](/kubernetes/ssl-certificates/cert-manager/) deployed to request/renew certificates
* [x] [External DNS](/kubernetes/external-dns/) configured to respond to ingresses, or with a wildcard DNS entry
## Preparation
### Namespace
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo:
```yaml title="/bootstrap/namespaces/namespace-traefik.yaml"
apiVersion: v1
kind: Namespace
metadata:
name: traefik
```
### HelmRepository
Next, we need to define a HelmRepository (*a repository of helm charts*), to which we'll refer when we create the HelmRelease. We only need to do this once per-repository. In this case, we're using the official [Traefik helm chart](https://github.com/traefik/traefik-helm-chart), so per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo:
```yaml title="/bootstrap/helmrepositories/helmrepository-traefik.yaml"
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: traefik
namespace: flux-system
spec:
interval: 15m
url: https://helm.traefik.io/traefik
```
### Kustomization
Now that the "global" elements of this deployment (*Namespace and HelmRepository*) have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at `/traefik`. I create this example Kustomization in my flux repo:
```yaml title="/bootstrap/kustomizations/kustomization-traefik.yaml"
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: traefik
namespace: flux-system
spec:
interval: 15m
path: ./traefik
prune: true # remove any elements later removed from the above path
timeout: 2m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: traefik
namespace: traefik
```
### ConfigMap
Now we're into the traefik-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo:
```yaml title="/traefik/configmap-traefik-helm-chart-value-overrides.yaml"
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: traefik-helm-chart-value-overrides
namespace: traefik
data:
values.yaml: |- # (1)!
# <upstream values go here>
```
1. Paste in the contents of the upstream `values.yaml` here, intended 4 spaces, and then change the values you need as illustrated below.
--8<-- "kubernetes-why-full-values-in-configmap.md"
Then work your way through the values you pasted, and change any which are specific to your configuration. It may not be necessary to change anything.
### HelmRelease
Lastly, having set the scene above, we define the HelmRelease which will actually deploy traefik into the cluster, with the config and extra ConfigMap we defined above. I save this in my flux repo as `traefik/helmrelease-traefik.yaml`:
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: traefik
namespace: traefik
spec:
chart:
spec:
chart: traefik
version: 10.x # (1)!
sourceRef:
kind: HelmRepository
name: traefik
namespace: flux-system
interval: 15m
timeout: 5m
releaseName: traefik
valuesFrom:
- kind: ConfigMap
name: traefik-helm-chart-value-overrides
valuesKey: values.yaml # This is the default, but best to be explicit for clarity
```
1. Use `9.x` for Kubernetes versions older than 1.22, as described [here](https://github.com/traefik/traefik-helm-chart/tree/master/traefik#kubernetes-version-support).
--8<-- "kubernetes-why-not-config-in-helmrelease.md"
## Deploy traefik
Having committed the above to your flux repository, you should shortly see a traefik kustomization, and in the `traefik` namespace, a controller and a speaker pod for every node:
```bash
demo@shredder:~$ kubectl get pods -n traefik
NAME READY STATUS RESTARTS AGE
traefik-5b849b4fbd-svbxk 1/1 Running 0 24h
traefik-5b849b4fbd-xt7vc 1/1 Running 0 24h
demo@shredder:~$
```
### How do I know it's working?
#### Test Service
By default, the chart will deploy Traefik in [LoadBalancer](/kubernetes/loadbalancer/) mode. When you use kubectl to display the service (`kubectl get services -n traefik`), you'll see the external IP displayed:
```bash
demo@shredder:~$ kubectl get services -n traefik
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
traefik LoadBalancer 10.152.183.162 172.168.209.1 80:30756/TCP,443:30462/TCP 24h
demo@shredder:~$
```
!!! question "Where does the external IP come from?"
If you're using [k3s's load balancer](/kubernetes/loadbalancer/k3s/), the external IP will likely be the IP of the the nodes running k3s. If you're using [MetalLB](/kubernetes/loadbalancer/metallb/), the external IP should come from the list of addresses in the pool you allocated.
Pointing your web browser to the external IP displayed should result in a 404 page. Congratulations, you have external access to the Traefik ingress controller! 🥳
#### Test Ingress
Still, you didn't deploy an ingress controller to look at 404 pages! If you used my [template repository](https://github.com/geek-cookbook/template-flux) to start off your [flux deployment strategy](/kubernetes/deployment/flux/), then the podinfo helm chart has already been deployed. By default, the podinfo configmap doesn't deploy an Ingress, but you can change this using the magic of GitOps... 🪄
Edit your podinfo helmrelease configmap (`/podinfo/configmap-podinfo-helm-chart-value-overrides.yaml`), and change `ingress.enabled` to `true`, and set the host name to match your local domain name (*already configured using [External DNS](/kubernetes/external-dns/)*):
``` yaml hl_lines="2 8"
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: podinfo.local
```
To:
``` yaml hl_lines="2 8"
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: podinfo.<your domain name>
```
Commit your changes, wait for a reconciliation, and run `kubectl get ingress -n podinfo`. You should see an ingress created matching the host defined above, and the ADDRESS value should match the service address of the traefik service.
```bash
root@cn1:~# kubectl get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
podinfo podinfo <none> podinfo.example.com 172.168.209.1 80, 443 91d
```
!!! question "Why is there no class value?"
You don't **have** to define an ingress class if you only have one **class** of ingress, since typically your ingress controller will assume the default class. When you run multiple ingress controllers (say, nginx **and** [traeifk](/kubernetes/ingress/traefik/), or multiple nginx instances with different access controls) then classes become more important.
Now assuming your [DNS is correct](/kubernetes/external-dns/), you should be able to point your browser to the hostname you chose, and see the beautiful podinfo page! 🥳🥳
#### Test SSL
Ha, but we're not done yet! We have exposed a service via our load balancer, we've exposed a route to a service via an Ingress, but let's get rid of that nasty "insecure" message in the browser when using HTTPS...
Since you setup [SSL certificates,](/kubernetes/ssl-certificates/) including [secret-replicator](/kubernetes/ssl-certificates/secret-replicator/), you should end up with a `letsencrypt-wildcard-cert` secret in every namespace, including `podinfo`.
So once again, alter the podinfo ConfigMap to change this:
```yaml hl_lines="2 4"
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
```
To this:
```yaml hl_lines="2 4"
tls:
- secretName: letsencrypt-wildcard-cert
hosts:
- podinfo.<your domain name>
```
Commit your changes, wait for the reconciliation, and the next time you point your browser at your ingress, you should get a beautiful, valid, officially-signed SSL certificate[^1]! 🥳🥳🥳
### Troubleshooting
Are things not working as expected? Watch the traefik's logs with ```kubectl logs -n traefik -l app.kubernetes.io/name=traefik -f```.
--8<-- "recipe-footer.md"
[^1]: The beauty of this design is that the same process will now work for any other application you deploy, without any additional manual effort for DNS or SSL setup!

55
docs/kubernetes/loadbalancer/index.md Normal file
View File

@@ -0,0 +1,55 @@
---
title: What loadbalancer to use in self-hosted Kubernetes?
description: Here's a simply way to work out which load balancer you'll need for your self-hosted Kubernetes cluster
---
# Loadbalancing Services
## TL;DR
1. I have multiple nodes (*you'd benefit from [MetalLB](/kubernetes/loadbalancer/metallb/)*)
2. I only need/want one node (*just go with [k3s svclb](/kubernetes/loadbalancer/k3s/)*)
## But why?
In Kubernetes, you don't access your containers / pods "*directly*", other than for debugging purposes. Rather, we have a construct called a "*service*", which is "in front of" one or more pods.
Consider that this is how containers talk to each other under Docker Swarm:
```mermaid
sequenceDiagram
wordpress->>+mysql: Are you there?
mysql->>+wordpress: Yes, ready to serve!
```
But **this** is how containers (pods) talk to each other under Kubernetes:
```mermaid
sequenceDiagram
wordpress->>+mysql-service: Are you there?
mysql-service->>+mysql-pods: Are you there?
mysql-pods->>+wordpress: Yes, ready to serve!
```
Why do we do this?
1. A service isn't pinned to a particular node, it's a virtual IP which lives in the cluster and doesn't change as pods/nodes come and go.
2. Using a service "in front of" pods means that rolling updates / scaling of the pods can take place, but communication with the service is uninterrupted (*assuming correct configuration*).
Here's some [more technical detail](https://kubernetes.io/docs/concepts/services-networking/service/) into how it works, but what you need to know is that when you want to interact with your containers in Kubernetes (*either from other containers or from outside, as a human*), you'll be talking to **services.**
Also, services are not exposed outside of the cluster by default. There are 3 levels of "exposure" for your Kubernetes services, briefly:
1. ClusterIP (*A service is only available to other services in the cluster - this is the default*)
2. NodePort (*A mostly-random high-port on the node running the pod is forwarded to the pod*)[^1]
3. LoadBalancer (*Some external help is required to forward a particular IP into the cluster, terminating on the node running your pod*)
For anything vaguely useful, only `LoadBalancer` is a viable option. Even though `NodePort` may allow you to access services directly, who wants to remember that they need to access [Radarr][radarr] on `192.168.1.44:34542` and Homer on `192.168.1.44:34532`? Ugh.
Assuming you only had a single Kubernetes node (*say, a small k3s deployment*), you'd want 100% of all incoming traffic to be directed to that node, and so you wouldn't **need** a loadbalancer. You'd just point some DNS entries / firewall NATs at the IP of the cluster, and be done.
(*This is [the way k3s works](/kubernetes/loadbalancer/k3s/) by default, although it's still called a LoadBalancer*)
--8<-- "recipe-footer.md"
[^1]: It is possible to be prescriptive about which port is used for a Nodeport-exposed service, and this is occasionally [a valid deployment strategy](https://github.com/portainer/k8s/#using-nodeport-on-a-localremote-cluster), but you're usually limited to ports between 30000 and 32768.

28
docs/kubernetes/loadbalancer/k3s.md Normal file
View File

@@ -0,0 +1,28 @@
---
title: klipper loadbalancer with k3s
description: klipper - k3s' lightweight loadbalancer
---
# K3s Load Balancing with Klipper
If your cluster is using K3s, and you have only one node, then you could be adequately served by the [built in "klipper" loadbalbancer provided with k3s](https://rancher.com/docs/k3s/latest/en/networking/#service-load-balancer).
If you want more than one node in your cluster[^1] (*either now or in future*), I'd steer you towards [MetalLB](/kubernetes/loadbalancer/metallb/) instead).
## How does it work?
When **not** deployed with `--disable servicelb`, every time you create a service of type `LoadBalancer`, k3s will deploy a daemonset (*a collection of pods which run on every host in the cluster*), listening on that given port on the host. So deploying a LoadBalancer service for nginx on ports 80 and 443, for example, would result in **every** cluster host listening on ports 80 and 443, and sending any incoming traffic to the nginx service.
## Well that's great, isn't it?
Yes, to get you started. But consider the following limitations:
1. This magic can only happen **once** per port. So you can't, for example, run two mysql instances on port 3306.
2. Because **every** host listens on the exposed ports, you can't run anything **else** on the hosts, which listens on those ports
3. Having multiple hosts listening on a given port still doesn't solve the problem of how to reliably direct traffic to all hosts, and how to gracefully fail over if one of the hosts fails.
To tackle these issues, you need some more advanced network configuration, along with [MetalLB](/kubernetes/loadbalancer/metallb/).
--8<-- "recipe-footer.md"
[^1]: And seriously, if you're building a Kubernetes cluster, of **course** you'll want more than one host!

288
docs/kubernetes/loadbalancer/metallb/index.md Normal file
View File

@@ -0,0 +1,288 @@
---
title: MetalLB - Kubernetes Bare-Metal Loadbalancing
description: MetalLB - Load-balancing for bare-metal Kubernetes clusters, deployed with Helm via flux
---
# MetalLB on Kubernetes, via Helm
[MetalLB](https://metallb.universe.tf/) offers a network [load balancer](/kubernetes/loadbalancer/) implementation which workes on "bare metal" (*as opposed to a cloud provider*).
MetalLB does two jobs:
1. Provides address allocation to services out of a pool of addresses which you define
2. Announces these addresses to devices outside the cluster, either using ARP/NDP (L2) or BGP (L3)
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
* [x] If k3s is used, then it was deployed with `--disable servicelb`
Optional:
* [ ] Network firewall/router supporting BGP (*ideal but not required*)
## MetalLB Requirements
### Allocations
You'll need to make some decisions re IP allocations.
* What is the range of addresses you want to use for your LoadBalancer service pool? If you're using BGP, this can be a dedicated subnet (*i.e. a /24*), and if you're not, this should be a range of IPs in your existing network space for your cluster nodes (*i.e., 192.168.1.100-200*)
* If you're using BGP, pick two [private AS numbers](https://datatracker.ietf.org/doc/html/rfc6996#section-5) between 64512 and 65534 inclusively.
### Namespace
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/namespaces/namespace-metallb-system.yaml`:
??? example "Example NameSpace (click to expand)"
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: metallb-system
```
### HelmRepository
Next, we need to define a HelmRepository (*a repository of helm charts*), to which we'll refer when we create the HelmRelease. We only need to do this once per-repository. In this case, we're using the (*prolific*) [bitnami chart repository](https://github.com/bitnami/charts/tree/master/bitnami), so per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/helmrepositories/helmrepository-bitnami.yaml`:
??? example "Example HelmRepository (click to expand)"
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: bitnami
namespace: flux-system
spec:
interval: 15m
url: https://charts.bitnami.com/bitnami
```
### Kustomization
Now that the "global" elements of this deployment (*Namespace and HelmRepository*) have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at `/metallb-system`. I create this example Kustomization in my flux repo at `bootstrap/kustomizations/kustomization-metallb.yaml`:
??? example "Example Kustomization (click to expand)"
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: metallb--metallb-system
namespace: flux-system
spec:
interval: 15m
path: ./metallb-system
prune: true # remove any elements later removed from the above path
timeout: 2m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: metallb-controller
namespace: metallb-system
```
!!! question "What's with that screwy name?"
> Why'd you call the kustomization `metallb--metallb-system`?
I keep my file and object names as consistent as possible. In most cases, the helm chart is named the same as the namespace, but in some cases, by upstream chart or historical convention, the namespace is different to the chart name. MetalLB is one of these - the helmrelease/chart name is `metallb`, but the typical namespace it's deployed in is `metallb-system`. (*Appending `-system` seems to be a convention used in some cases for applications which support the entire cluster*). To avoid confusion when I list all kustomizations with `kubectl get kustomization -A`, I give these oddballs a name which identifies both the helmrelease and the namespace.
### ConfigMap (for HelmRelease)
Now we're into the metallb-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/bitnami/charts/blob/master/bitnami/metallb/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo at `metallb-system/configmap-metallb-helm-chart-value-overrides.yaml`:
??? example "Example ConfigMap (click to expand)"
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: metallb-helm-chart-value-overrides
namespace: metallb-system
data:
values.yaml: |-
## @section Global parameters
## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry, imagePullSecrets and storageClass
## @param global.imageRegistry Global Docker image registry
## @param global.imagePullSecrets Global Docker registry secret names as an array
##
global:
imageRegistry: ""
## E.g.
## imagePullSecrets:
## - myRegistryKeySecretName
<snip>
prometheus:
## Prometheus Operator service monitors
##
serviceMonitor:
## @param speaker.prometheus.serviceMonitor.enabled Enable support for Prometheus Operator
##
enabled: false
## @param speaker.prometheus.serviceMonitor.jobLabel Job label for scrape target
##
jobLabel: "app.kubernetes.io/name"
## @param speaker.prometheus.serviceMonitor.interval Scrape interval. If not set, the Prometheus default scrape interval is used
##
interval: ""
## @param speaker.prometheus.serviceMonitor.metricRelabelings Specify additional relabeling of metrics
##
metricRelabelings: []
## @param speaker.prometheus.serviceMonitor.relabelings Specify general relabeling
##
relabelings: []
```
--8<-- "kubernetes-why-full-values-in-configmap.md"
Then work your way through the values you pasted, and change any which are specific to your configuration. I'd recommend changing the following:
* `existingConfigMap: metallb-config`: I prefer to set my MetalLB config independently of the chart config, so I set this to `metallb-config`, which I then define below.
* `commonAnnotations`: Anticipating the future use of Reloader to bounce applications when their config changes, I add the `configmap.reloader.stakater.com/reload: "metallb-config"` annotation to all deployed objects, which will instruct Reloader to bounce the daemonset if the ConfigMap changes.
### ConfigMap (for MetalLB)
Finally, it's time to actually configure MetalLB! As discussed above, I prefer to configure the helm chart to apply config from an existing ConfigMap, so that I isolate my application configuration from my chart configuration (*and make tracking changes easier*). In my setup, I'm using BGP against a pair of pfsense[^1] firewalls, so per the [official docs](https://metallb.universe.tf/configuration/), I use the following configuration, saved in my flux repo as `metallb-system/configmap-metallb-config.yaml`:
??? example "Example ConfigMap (click to expand)"
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: metallb-config
data:
config: |
peers:
- peer-address: 192.168.33.2
peer-asn: 64501
my-asn: 64500
- peer-address: 192.168.33.4
peer-asn: 64501
my-asn: 64500
address-pools:
- name: default
protocol: bgp
avoid-buggy-ips: true
addresses:
- 192.168.32.0/24
```
!!! question "What does that mean?"
In the config referenced above, I define one pool of addresses (`192.168.32.0/24`) which MetalLB is responsible for allocating to my services. MetalLB will then "advertise" these addresses to my firewalls (`192.168.33.2` and `192.168.33.4`), in an eBGP relationship where the firewalls' ASN is `64501` and MetalLB's ASN is `64500`. Provided I'm using my firewalls as my default gateway (*a VIP*), when I try to access one of the `192.168.32.x` IPs from any subnet connected to my firewalls, the traffic will be routed from the firewall to one of the cluster nodes running the pods selected by that service.
!!! note "Dude, that's too complicated!"
There's an easier way, with some limitations. If you configure MetalLB in L2 mode, all you need to do is to define a range of IPs within your existing node subnet, like this:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: metallb-config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.1.240-192.168.1.250
```
### HelmRelease
Lastly, having set the scene above, we define the HelmRelease which will actually deploy MetalLB into the cluster, with the config and extra ConfigMap we defined above. I save this in my flux repo as `metallb-system/helmrelease-metallb.yaml`:
??? example "Example HelmRelease (click to expand)"
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: metallb
namespace: metallb-system
spec:
chart:
spec:
chart: metallb
version: 2.x
sourceRef:
kind: HelmRepository
name: bitnami
namespace: flux-system
interval: 15m
timeout: 5m
releaseName: metallb
valuesFrom:
- kind: ConfigMap
name: metallb-helm-chart-value-overrides
valuesKey: values.yaml # This is the default, but best to be explicit for clarity
```
--8<-- "kubernetes-why-not-config-in-helmrelease.md"
## Deploy MetalLB
Having committed the above to your flux repository, you should shortly see a metallb kustomization, and in the `metallb-system` namespace, a controller and a speaker pod for every node:
```bash
root@cn1:~# kubectl get pods -n metallb-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
metallb-controller-779d8686f6-mgb4s 1/1 Running 0 21d 10.0.6.19 wn3 <none> <none>
metallb-speaker-2qh2d 1/1 Running 0 21d 192.168.33.24 wn4 <none> <none>
metallb-speaker-7rz24 1/1 Running 0 21d 192.168.33.22 wn2 <none> <none>
metallb-speaker-gbm5r 1/1 Running 0 21d 192.168.33.23 wn3 <none> <none>
metallb-speaker-gzgd2 1/1 Running 0 21d 192.168.33.21 wn1 <none> <none>
metallb-speaker-nz6kd 1/1 Running 0 21d 192.168.33.25 wn5 <none> <none>
root@cn1:~#
```
!!! question "Why are there no speakers on my masters?"
In some cluster setups, master nodes are "tainted" to prevent workloads running on them and consuming capacity required for "mastering". If this is the case for you, but you actually **do** want to run some externally-exposed workloads on your masters, you'll need to update the `speaker.tolerations` value for the HelmRelease config to include:
```yaml
- key: "node-role.kubernetes.io/master"
effect: "NoSchedule"
```
### How do I know it's working?
If you used my [template repository](https://github.com/geek-cookbook/template-flux) to start off your [flux deployment strategy](/kubernetes/deployment/flux/), then the podinfo helm chart has already been deployed. By default, the podinfo service is in `ClusterIP` mode, so it's only reachable within the cluster.
Edit your podinfo helmrelease configmap (`/podinfo/configmap-podinfo-helm-chart-value-overrides.yaml`), and change this:
``` yaml hl_lines="6"
<snip>
# Kubernetes Service settings
service:
enabled: true
annotations: {}
type: ClusterIP
<snip>
```
To:
``` yaml hl_lines="6"
<snip>
# Kubernetes Service settings
service:
enabled: true
annotations: {}
type: LoadBalancer
<snip>
```
Commit your changes, wait for a reconciliation, and run `kubectl get services -n podinfo`. All going well, you should see that the service now has an IP assigned from the pool you chose for MetalLB!
--8<-- "recipe-footer.md"
[^1]: I've documented an example re [how to configure BGP between MetalLB and pfsense](/kubernetes/loadbalancer/metallb/pfsense/).

80
docs/kubernetes/loadbalancer/metallb/pfsense.md Normal file
View File

@@ -0,0 +1,80 @@
---
title: MetalLB BGP config for pfSense - Kubernetes load balancing
description: Using MetalLB with pfsense and BGP
---
# MetalLB on Kubernetes with pfSense
This is an addendum to the MetalLB recipe, explaining how to configure MetalLB to perform BGP peering with a pfSense firewall.
!!! summary "Ingredients"
* [X] A [Kubernetes cluster](/kubernetes/cluster/)
* [X] [MetalLB](/kubernetes/loadbalancer/metallb/) deployed
* [X] One or more pfSense firewalls
* [X] Basic familiarity with pfSense operation
## Preparation
Complete the [MetalLB](/kubernetes/loadbalancer/metallb/) installation, including the process of identifying ASNs for both your pfSense firewall and your MetalLB configuration.
Install the FRR package in pfsense, under **System -> Package Manager -> Available Packages**
### Configure FRR Global/Zebra
Under **Services -> FRR Global/Zebra**, enable FRR, set your router ID (*this will be your router's peer IP in MetalLB config*), and set a master password (*because apparently you have to, even though we don't use it*):
![Enabling BGP routing](/images/metallb-pfsense-00.png){ loading=lazy }
### Configure FRR BGP
Under **Services -> FRR BGP**, globally enable BGP, and set your local AS and router ID:
![Enabling BGP routing](/images/metallb-pfsense-01.png){ loading=lazy }
### Configure FRR BGP Advanced
Use the tabs at the top of the FRR configuration to navigate to "**Advanced**"...
![Enabling BGP routing](/images/metallb-pfsense-02.png){ loading=lazy }
... and scroll down to **eBGP**. Check the checkbox titled "**Disable eBGP Require Policy**:
![Enabling BGP routing](/images/metallb-pfsense-03.png){ loading=lazy }
!!! question "Isn't disabling a policy check a Bad Idea(tm)?"
If you're an ISP, sure. If you're only using eBGP to share routes between MetalLB and pfsense, then applying policy is an unnecessary complication.[^1]
### Configure BGP neighbors
#### Peer Group
It's useful to bundle our configurations within a "peer group" (*a collection of settings which applies to all neighbors who are members of that group*), so start off by creating a neighbor with the name of "**metallb**" (*this will become a peer-group*). Set the remote AS (*because you have to*), and leave the rest of the settings as default.
!!! question "Why bother with a peer group?"
> If we're not changing any settings, why are we bothering with a peer group?
We may later want to change settings which affect all the peers, such as prefix lists, route-maps, etc. We're doing this now for the benefit of our future selves 💪
#### Individual Neighbors
Now add each node running MetalLB, as a BGP neighbor. Pick the peer-group you created above, and configure each neighbor's ASN:
![Enabling BGP routing](/images/metallb-pfsense-04.png){ loading=lazy }
## Serving
Once you've added your neighbors, you should be able to use the FRR tab navigation (*it's weird, I know!*) to get to Status / BGP, and identify your neighbors, and all the routes learned from them. In the screenshot below, you'll note that **most** routes are learned from all the neighbors - that'll be service backed by a daemonset, running on all nodes. The `192.168.32.3/32` route, however, is only received from `192.168.33.22`, meaning only one node is running the pods backing this service, so only those pods are advertising the route to pfSense:
![BGP route-](/images/metallb-pfsense-05.png){ loading=lazy }
### Troubleshooting
If you're not receiving any routes from MetalLB, or if the neighbors aren't in an established state, here are a few suggestions for troubleshooting:
1. Confirm on PFSense that the BGP connections (*TCP port 179*) are not being blocked by the firewall
2. Examine the metallb speaker logs in the cluster, by running `kubectl logs -n metallb-system -l app.kubernetes.io/name=metallb`
3. SSH to the pfsense, start a shell and launch the FFR shell by running `vtysh`. Now you're in a cisco-like console where commands like `show ip bgp sum` and `show ip bgp neighbors <neighbor ip> received-routes` will show you interesting debugging things.
--8<-- "recipe-footer.md"
[^1]: If you decide to deploy some policy with route-maps, prefix-lists, etc, it's all found under **Services -> FRR Global/Zebra** 🦓

314
docs/kubernetes/monitoring/index.md Normal file
View File

@@ -0,0 +1,314 @@
# Miniflux
Miniflux is a lightweight RSS reader, developed by [Frédéric Guillot](https://github.com/fguillot). (_Who also happens to be the developer of the favorite Open Source Kanban app, [Kanboard](/recipes/kanboard/)_)
![Miniflux Screenshot](/images/miniflux.png){ loading=lazy }
I've [reviewed Miniflux in detail on my blog](https://www.funkypenguin.co.nz/review/miniflux-lightweight-self-hosted-rss-reader/), but features (among many) that I appreciate:
* Compatible with the Fever API, read your feeds through existing mobile and desktop clients (_This is the killer feature for me. I hardly ever read RSS on my desktop, I typically read on my iPhone or iPad, using [Fiery Feeds](http://cocoacake.net/apps/fiery/) or my new squeeze, [Unread](https://www.goldenhillsoftware.com/unread/)_)
* Send your bookmarks to Pinboard, Wallabag, Shaarli or Instapaper (_I use this to automatically pin my bookmarks for collection on my [blog](https://www.funkypenguin.co.nz/)_)
* Feeds can be configured to download a "full" version of the content (_rather than an excerpt_)
* Use the Bookmarklet to subscribe to a website directly from any browsers
!!! abstract "2.0+ is a bit different"
[Some things changed](https://docs.miniflux.net/en/latest/migration.html) when Miniflux 2.0 was released. For one thing, the only supported database is now postgresql (_no more SQLite_). External themes are gone, as is PHP (_in favor of golang_). It's been a controversial change, but I'm keen on minimal and single-purpose, so I'm still very happy with the direction of development. The developer has laid out his [opinions](https://docs.miniflux.net/en/latest/opinionated.html) re the decisions he's made in the course of development.
## Ingredients
1. A [Kubernetes Cluster](/kubernetes/design/) including [Traefik Ingress](/kubernetes/traefik/)
2. A DNS name for your miniflux instance (*miniflux.example.com*, below) pointing to your [load balancer](/kubernetes/loadbalancer/), fronting your Traefik ingress
## Preparation
### Prepare traefik for namespace
When you deployed [Traefik via the helm chart](/kubernetes/traefik/), you would have customized ```values.yml``` for your deployment. In ```values.yml``` is a list of namespaces which Traefik is permitted to access. Update ```values.yml``` to include the *miniflux* namespace, as illustrated below:
```yaml
<snip>
kubernetes:
namespaces:
- kube-system
- nextcloud
- kanboard
- miniflux
<snip>
```
If you've updated ```values.yml```, upgrade your traefik deployment via helm, by running ```helm upgrade --values values.yml traefik stable/traefik --recreate-pods```
### Create data locations
Although we could simply bind-mount local volumes to a local Kubuernetes cluster, since we're targetting a cloud-based Kubernetes deployment, we only need a local path to store the YAML files which define the various aspects of our Kubernetes deployment.
```bash
mkdir /var/data/config/miniflux
```
### Create namespace
We use Kubernetes namespaces for service discovery and isolation between our stacks, so create a namespace for the miniflux stack with the following .yml:
```bash
cat <<EOF > /var/data/config/miniflux/namespace.yml
apiVersion: v1
kind: Namespace
metadata:
name: miniflux
EOF
kubectl create -f /var/data/config/miniflux/namespace.yaml
```
### Create persistent volume claim
Persistent volume claims are a streamlined way to create a persistent volume and assign it to a container in a pod. Create a claim for the miniflux postgres database:
```bash
cat <<EOF > /var/data/config/miniflux/db-persistent-volumeclaim.yml
kkind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: miniflux-db
namespace: miniflux
annotations:
backup.kubernetes.io/deltas: P1D P7D
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF
kubectl create -f /var/data/config/miniflux/db-persistent-volumeclaim.yaml
```
!!! question "What's that annotation about?"
The annotation is used by [k8s-snapshots](/kubernetes/snapshots/) to create daily incremental snapshots of your persistent volumes. In this case, our volume is snapshotted daily, and copies kept for 7 days.
### Create secrets
It's not always desirable to have sensitive data stored in your .yml files. Maybe you want to check your config into a git repository, or share it. Using Kubernetes Secrets means that you can create "secrets", and use these in your deployments by name, without exposing their contents. Run the following, replacing ```imtoosexyformyadminpassword```, and the ```mydbpass``` value in both postgress-password.secret **and** database-url.secret:
```bash
echo -n "imtoosexyformyadminpassword" > admin-password.secret
echo -n "mydbpass" > postgres-password.secret
echo -n "postgres://miniflux:mydbpass@db/miniflux?sslmode=disable" > database-url.secret
kubectl create secret -n mqtt generic miniflux-credentials \
--from-file=admin-password.secret \
--from-file=database-url.secret \
--from-file=database-url.secret
```
!!! tip "Why use ```echo -n```?"
Because. See [my blog post here](https://www.funkypenguin.co.nz/blog/beware-the-hidden-newlines-in-kubernetes-secrets/) for the pain of hunting invisible newlines, that's why!
## Serving
Now that we have a [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/), a [persistent volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/), and a [configmap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/), we can create [deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), [services](https://kubernetes.io/docs/concepts/services-networking/service/), and an [ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) for the miniflux [pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/).
### Create db deployment
Deployments tell Kubernetes about the desired state of the pod (*which it will then attempt to maintain*). Create the db deployment by excecuting the following. Note that the deployment refers to the secrets created above.
--8<-- "premix-cta.md"
```bash
cat <<EOF > /var/data/miniflux/db-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: miniflux
name: db
labels:
app: db
spec:
replicas: 1
selector:
matchLabels:
app: db
template:
metadata:
labels:
app: db
spec:
containers:
- image: postgres:11
name: db
volumeMounts:
- name: miniflux-db
mountPath: /var/lib/postgresql/data
env:
- name: POSTGRES_USER
value: "miniflux"
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: miniflux-credentials
key: postgres-password.secret
volumes:
- name: miniflux-db
persistentVolumeClaim:
claimName: miniflux-db
```
### Create app deployment
Create the app deployment by excecuting the following. Again, note that the deployment refers to the secrets created above.
```bash
cat <<EOF > /var/data/miniflux/app-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: miniflux
name: app
labels:
app: app
spec:
replicas: 1
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
spec:
containers:
- image: miniflux/miniflux
name: app
env:
# This is necessary for the miniflux to update the db schema, even on an empty DB
- name: CREATE_ADMIN
value: "1"
- name: RUN_MIGRATIONS
value: "1"
- name: ADMIN_USERNAME
value: "admin"
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: miniflux-credentials
key: admin-password.secret
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: miniflux-credentials
key: database-url.secret
EOF
kubectl create -f /var/data/miniflux/deployment.yml
```
### Check pods
Check that your deployment is running, with ```kubectl get pods -n miniflux```. After a minute or so, you should see 2 "Running" pods, as illustrated below:
```bash
[funkypenguin:~] % kubectl get pods -n miniflux
NAME READY STATUS RESTARTS AGE
app-667c667b75-5jjm9 1/1 Running 0 4d
db-fcd47b88f-9vvqt 1/1 Running 0 4d
[funkypenguin:~] %
```
### Create db service
The db service resource "advertises" the availability of PostgreSQL's port (TCP 5432) in your pod, to the rest of the cluster (*constrained within your namespace*). It seems a little like overkill coming from the Docker Swarm's automated "service discovery" model, but the Kubernetes design allows for load balancing, rolling upgrades, and health checks of individual pods, without impacting the rest of the cluster elements.
```bash
cat <<EOF > /var/data/miniflux/db-service.yml
kind: Service
apiVersion: v1
metadata:
name: db
namespace: miniflux
spec:
selector:
app: db
ports:
- protocol: TCP
port: 5432
clusterIP: None
EOF
kubectl create -f /var/data/miniflux/service.yml
```
### Create app service
The app service resource "advertises" the availability of miniflux's HTTP listener port (TCP 8080) in your pod. This is the service which will be referred to by the ingress (below), so that Traefik can route incoming traffic to the miniflux app.
```bash
cat <<EOF > /var/data/miniflux/app-service.yml
kind: Service
apiVersion: v1
metadata:
name: app
namespace: miniflux
spec:
selector:
app: app
ports:
- protocol: TCP
port: 8080
clusterIP: None
EOF
kubectl create -f /var/data/miniflux/app-service.yml
```
### Check services
Check that your services are deployed, with ```kubectl get services -n miniflux```. You should see something like this:
```bash
[funkypenguin:~] % kubectl get services -n miniflux
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
app ClusterIP None <none> 8080/TCP 55d
db ClusterIP None <none> 5432/TCP 55d
[funkypenguin:~] %
```
### Create ingress
The ingress resource tells Traefik what to forward inbound requests for *miniflux.example.com* to your service (defined above), which in turn passes the request to the "app" pod. Adjust the config below for your domain.
```bash
cat <<EOF > /var/data/miniflux/ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: app
namespace: miniflux
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: miniflux.example.com
http:
paths:
- backend:
serviceName: app
servicePort: 8080
EOF
kubectl create -f /var/data/miniflux/ingress.yml
```
Check that your service is deployed, with ```kubectl get ingress -n miniflux```. You should see something like this:
```bash
[funkypenguin:~] 130 % kubectl get ingress -n miniflux
NAME HOSTS ADDRESS PORTS AGE
app miniflux.funkypenguin.co.nz 80 55d
[funkypenguin:~] %
```
### Access Miniflux
At this point, you should be able to access your instance on your chosen DNS name (*i.e. <https://miniflux.example.com>*)
### Troubleshooting
To look at the Miniflux pod's logs, run ```kubectl logs -n miniflux <name of pod per above> -f```. For further troubleshooting hints, see [Troubleshooting](/reference/kubernetes/troubleshooting/).
--8<-- "recipe-footer.md"

11
docs/kubernetes/persistence/index.md Normal file
View File

@@ -0,0 +1,11 @@
# Persistence
So we've gone as far as we can with our cluster, without any form of persistence. As soon as we want to retain data, be it a database, metrics history, or objects, we need one or more ways to persist data within the cluster.
Here are some popular options, ranked in difficulty/complexity, in vaguely ascending order:
* [Local Path Provisioner](/kubernetes/persistence/local-path-provisioner/) (on k3s)
* [TopoLVM](/kubernetes/persistence/topolvm/)
* OpenEBS (coming soon)
* [Rook Ceph](/kubernetes/persistence/rook-ceph/)
* Longhorn (coming soon)

45
docs/kubernetes/persistence/local-path-provisioner.md Normal file
View File

@@ -0,0 +1,45 @@
# Local Path Provisioner
[k3s](/kubernetes/cluster/k3s/) installs itself with "Local Path Provisioner", a simple controller whose job it is to create local volumes on each k3s node. If you only have one node, or you just want something simple to start learning with, then `local-path` is ideal, since it requires no further setup.
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/) deployed with [k3s](/kubernetes/cluster/k3s/)
Here's how you know you've got the StorageClass:
```bash
root@shredder:~# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 60m
root@shredder:~#
```
## Preparation
### Basics
A few things you should know:
1. This is not **network storage**. The volume you create will forever be found to the k3s node its pod is executed on. If you later take that node down for maintenance, the pods will not be able to start on other nodes, because they won't find their volumes.
2. The default path for the volumes is `/opt/local-path-provisioner`, although this can be changed by [editing a ConfigMap](https://github.com/rancher/local-path-provisioner/blob/master/README.md#customize-the-configmap). Make sure you have enough disk space! [^1]
3. There's no support for resizing a volume. If you create a volume and later work out that it's too small, you'll have to destroy it and recreate it. (*More sophisticated provisioners like [rook-ceph](/kubernetes/persistence/rook-ceph/) and [topolvm](/kubernetes/persistence/topolvm/) allow for dynamic resizing of volumes*)
### When to use it
* When you don't care much about your storage. This seems backwards, but sometimes you need large amounts of storage for relatively ephemeral reasons, like batch processing, or log aggregation. You may decide the convenience of using Local Path Provisioner for quick, hard-drive-speed storage outweighs the minor hassle of loosing your metrics data if you were to have a node outage.
* When [TopoLVM](/kubernetes/persistence/topolvm/) is not a viable option, and you'd rather use available disk space on your existing, formatted filesystems
### When not to use it
* When you have any form of redundancy requirement on your persisted data.
* When you're not using k3s.
* You may one day want to resize your volumes.
### Summary
In summary, Local Path Provisioner is fine if you have very specifically sized workloads and you don't care about node redundancy.
--8<-- "recipe-footer.md"
[^1]: [TopoLVM](/kubernetes/persistence/topolvm/) also creates per-node volumes which aren't "portable" between nodes, but because it relies on LVM, it is "capacity-aware", and is able to distribute storage among multiple nodes based on available capacity.

3
docs/kubernetes/persistence/longhorn.md Normal file
View File

@@ -0,0 +1,3 @@
# Longhorn
Coming soon!

3
docs/kubernetes/persistence/openebs.md Normal file
View File

@@ -0,0 +1,3 @@
# Open EBS
Coming soon!

421
docs/kubernetes/persistence/rook-ceph/cluster.md Normal file
View File

@@ -0,0 +1,421 @@
---
title: Deploy Rook Ceph's operater-managed Cluster for Persistent Storage in Kubernetes
description: Step #2 - Having the operator available, now we deploy the ceph cluster itself
---
# Persistent storage in Kubernetes with Rook Ceph / CephFS - Cluster
[Ceph](https://docs.ceph.com/en/quincy/) is a highly-reliable, scalable network storage platform which uses individual disks across participating nodes to provide fault-tolerant storage.
[Rook](https://rook.io) provides an operator for Ceph, decomposing the [10-year-old](https://en.wikipedia.org/wiki/Ceph_(software)#Release_history), at-time-arcane, platform into cloud-native components, created declaratively, whose lifecycle is managed by an operator.
In the [previous recipe](/kubernetes/persistence/rook-ceph/operator/), we deployed the operator, and now to actually deploy a Ceph cluster, we need to deploy a custom resource (*a "CephCluster"*), which will instruct the operator on we'd like our cluster to be deployed.
We'll end up with multilpe storageClasses which we can use to allocate storage to pods from either Ceph RBD (*block storage*), or CephFS (*a mounted filesystem*). In many cases, CephFS is a useful choice, because it can be mounted from more than one pod **at the same time**, which makes it suitable for apps which need to share access to the same data ([NZBGet][nzbget], [Sonarr][sonarr], and [Plex][plex], for example)
## Rook Ceph Cluster requirements
!!! summary "Ingredients"
Already deployed:
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
* [x] Rook Ceph's [Operator](/kubernetes/persistence/rook-ceph/operator/)
## Preparation
### Namespace
We already deployed a `rook-ceph` namespace when deploying the Rook Ceph [Operator](/kubernetes/persistence/rook-ceph/operator/), so we don't need to create this again :thumbsup: [^1]
### HelmRepository
Likewise, we'll install the `rook-ceph-cluster` helm chart from the same Rook-managed repository as we did the `rook-ceph` (operator) chart, so we don't need to create a new HelmRepository.
### Kustomization
We do, however, need a separate Kustomization for rook-ceph-cluster, telling flux to deploy any YAMLs found in the repo at `/rook-ceph-cluster`. I create this example Kustomization in my flux repo:
!!! question "Why a separate Kustomization if both are needed for rook-ceph?"
While technically we **could** use the same Kustomization to deploy both `rook-ceph` and `rook-ceph-cluster`, we'd run into dependency issues. It's simpler and cleaner to deploy `rook-ceph` first, and then list it as a dependency for `rook-ceph-cluster`.
```yaml title="/bootstrap/kustomizations/kustomization-rook-ceph-cluster.yaml"
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: rook-ceph-cluster--rook-ceph
namespace: flux-system
spec:
dependsOn:
- name: "rook-ceph" # (1)!
interval: 30m
path: ./rook-ceph-cluster
prune: true # remove any elements later removed from the above path
timeout: 10m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
```
1. Note that we use the `spec.dependsOn` to ensure that this Kustomization is only applied **after** the rook-ceph operator is deployed and operational. This ensures that the necessary CRDs are in place, and avoids a dry-run error on the reconciliation.
--8<-- "premix-cta-kubernetes.md"
### ConfigMap
Now we're into the app-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/rook/rook/blob/master/deploy/charts/rook-ceph/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo:
```yaml title="/rook-ceph-cluster/configmap-rook-ceph-cluster-helm-chart-value-overrides.yaml"
apiVersion: v1
kind: ConfigMap
metadata:
name: rook-ceph-cluster-helm-chart-value-overrides
namespace: rook-ceph-cluster
data:
values.yaml: |- # (1)!
# <upstream values go here>
```
1. Paste in the contents of the upstream `values.yaml` here, intended 4 spaces, and then change the values you need as illustrated below.
Here are some suggested changes to the defaults which you should consider:
```yaml
toolbox:
enabled: true # (1)!
monitoring:
# enabling will also create RBAC rules to allow Operator to create ServiceMonitors
enabled: true # (2)!
# whether to create the prometheus rules
createPrometheusRules: true # (3)!
pspEnable: false # (4)!
ingress:
dashboard: {} # (5)!
```
1. It's useful to have a "toolbox" pod to shell into to run ceph CLI commands
2. Consider enabling if you already have Prometheus installed
3. Consider enabling if you already have Prometheus installed
4. PSPs are deprecated, and will eventually be removed in Kubernetes 1.25, at which point this will cause breakage.
5. Customize the ingress configuration for your dashboard
Further to the above, decide which disks you want to dedicate to Ceph, and add to the `cephClusterSpec` section.
The default configuration (below) will cause the operator to use any un-formatted disks found on any of your nodes. If this is what you **want** to happen, then you don't need to change anything.
```yaml
cephClusterSpec:
storage: # cluster level storage configuration and selection
useAllNodes: true
useAllDevices: true
```
If you'd rather be a little more selective / declarative about which disks are used in a homogenous cluster, you could consider using `deviceFilter`, like this:
```yaml
cephClusterSpec:
storage: # cluster level storage configuration and selection
useAllNodes: true
useAllDevices: false
deviceFilter: sdc #(1)!
```
1. A regex to use to filter target devices found on each node
If your cluster nodes are a little more snowflakey :snowflake:, here's a complex example:
```yaml
cephClusterSpec:
storage: # cluster level storage configuration and selection
useAllNodes: false
useAllDevices: false
nodes:
- name: "teeny-tiny-node"
deviceFilter: "." #(1)!
- name: "bigass-node"
devices:
- name: "/dev/disk/by-path/pci-0000:01:00.0-sas-exp0x500404201f43b83f-phy11-lun-0" #(2)!
config:
metadataDevice: "/dev/osd-metadata/11"
- name: "nvme0n1" #(3)!
- name: "nvme1n1"
```
1. Match any devices found on this node
2. Match a very-specific device path, and pair this device with a faster device for OSD metadata
3. Match devices with simple regex string matches
### HelmRelease
Finally, having set the scene above, we define the HelmRelease which will actually deploy the rook-ceph operator into the cluster. I save this in my flux repo:
```yaml title="/rook-ceph-cluster/helmrelease-rook-ceph-cluster.yaml"
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: rook-ceph-cluster
namespace: rook-ceph
spec:
chart:
spec:
chart: rook-ceph-cluster
version: 1.9.x
sourceRef:
kind: HelmRepository
name: rook-release
namespace: flux-system
interval: 30m
timeout: 10m
install:
remediation:
retries: 3
upgrade:
remediation:
retries: -1 # keep trying to remediate
crds: CreateReplace # Upgrade CRDs on package update
releaseName: rook-ceph-cluster
valuesFrom:
- kind: ConfigMap
name: rook-ceph-cluster-helm-chart-value-overrides
valuesKey: values.yaml # (1)!
```
1. This is the default, but best to be explicit for clarity
## Install Rook Ceph Operator!
Commit the changes to your flux repository, and either wait for the reconciliation interval, or force a reconcilliation using `flux reconcile source git flux-system`. You should see the kustomization appear...
```bash
~ ❯ flux get kustomizations rook-ceph-cluster
NAME READY MESSAGE REVISION SUSPENDED
rook-ceph-cluster True Applied revision: main/345ee5e main/345ee5e False
~ ❯
```
The helmrelease should be reconciled...
```bash
~ ❯ flux get helmreleases -n rook-ceph rook-ceph
NAME READY MESSAGE REVISION SUSPENDED
rook-ceph-cluster True Release reconciliation succeeded v1.9.9 False
~ ❯
```
And you should have happy rook-ceph operator pods:
```bash
~ ❯ k get pods -n rook-ceph -l app=rook-ceph-operator
NAME READY STATUS RESTARTS AGE
rook-ceph-operator-7c94b7446d-nwsss 1/1 Running 0 5m14s
~ ❯
```
To watch the operator do its magic, you can tail its logs, using:
```bash
k logs -n rook-ceph -f -l app=rook-ceph-operator
```
You can **get** or **describe** the status of your cephcluster:
```bash
~ ❯ k get cephclusters.ceph.rook.io -n rook-ceph
NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL
rook-ceph /var/lib/rook 3 6d22h Ready Cluster created successfully HEALTH_OK
~ ❯
```
### How do I know it's working?
So we have a ceph cluster now, but how do we know we can actually provision volumes?
#### Create PVCs
Create two ceph-block PVCs (*persistent volume claim*), by running:
```bash
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ceph-block-pvc-1
labels:
test: ceph
funkypenguin-is: a-smartass
spec:
accessModes:
- ReadWriteOnce
storageClassName: ceph-block
resources:
requests:
storage: 128Mi
EOF
```
And:
```bash
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ceph-block-pvc-2
labels:
test: ceph
funkypenguin-is: a-smartass
spec:
accessModes:
- ReadWriteOnce
storageClassName: ceph-block
resources:
requests:
storage: 128Mi
EOF
```
Now create a ceph-filesystem (RWX) PVC, by running:
```bash
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ceph-filesystem-pvc
labels:
test: ceph
funkypenguin-is: a-smartass
spec:
accessModes:
- ReadWriteMany
storageClassName: ceph-filesystem
resources:
requests:
storage: 128Mi
EOF
```
Examine the PVCs by running:
```bash
kubectl get pvc -l test=ceph
```
#### Create Pod
Now create pods to consume the PVCs, by running:
```bash
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: ceph-test-1
labels:
test: ceph
funkypenguin-is: a-smartass
spec:
containers:
- name: volume-test
image: nginx:stable-alpine
imagePullPolicy: IfNotPresent
volumeMounts:
- name: ceph-block-is-rwo
mountPath: /rwo
- name: ceph-filesystem-is-rwx
mountPath: /rwx
ports:
- containerPort: 80
volumes:
- name: ceph-block-is-rwo
persistentVolumeClaim:
claimName: ceph-block-pvc-1
EOF
```
And:
```bash
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: ceph-test-2
labels:
test: ceph
funkypenguin-is: a-smartass
spec:
containers:
- name: volume-test
image: nginx:stable-alpine
imagePullPolicy: IfNotPresent
volumeMounts:
- name: ceph-block-is-rwo
mountPath: /rwo
- name: ceph-filesystem-is-rwx
mountPath: /rwx
ports:
- containerPort: 80
volumes:
- name: ceph-block-is-rwo
persistentVolumeClaim:
claimName: ceph-block-pvc-2
- name: ceph-filesystem-is-rwx
persistentVolumeClaim:
claimName: ceph-filesystem-pvc
EOF
```
Ensure the pods have started successfully (*this indicates the PVCs were correctly attached*) by running:
```bash
kubectl get pod -l test=ceph
```
#### Clean up
Assuming that the pod is in a `Running` state, then TopoLVM is working!
Clean up your mess, little bare-metal-cave-monkey :monkey_face:, by running:
```bash
kubectl delete pod -l funkypenguin-is=a-smartass
kubectl delete pvc -l funkypenguin-is=a-smartass #(1)!
```
1. Label selectors are powerful!
### View Ceph Dashboard
Assuming you have an Ingress Controller setup, and you've either picked a default IngressClass, or defined the dashboard ingress appropriately, you should be able to access your Ceph Dashboard, at the URL identified by the ingress (*this is a good opportunity to check that the ingress deployed correctly*):
```bash
~ ❯ k get ingress -n rook-ceph
NAME CLASS HOSTS ADDRESS PORTS AGE
rook-ceph-mgr-dashboard nginx rook-ceph.batcave.awesome.me 172.16.237.1 80, 443 177d
~ ❯
```
The dashboard credentials are automatically generated for you by the operator, and stored in a Kubernetes secret. To retrieve your credentials, run:
```bash
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o \
jsonpath="{['data']['password']}" | base64 --decode && echo
```
## Summary
What have we achieved? We're half-way to getting a ceph cluster, having deployed the operator which will manage the lifecycle of the [ceph cluster](/kubernetes/persistence/rook-ceph/cluster/) we're about to create!
!!! summary "Summary"
Created:
* [X] Ceph cluster has been deployed
* [X] StorageClasses are available so that the cluster storage can be consumed by your pods
* [X] Pretty graphs are viewable in the Ceph Dashboard
--8<-- "recipe-footer.md"
[^1]: Unless you **wanted** to deploy your cluster components in a separate namespace to the operator, of course!

19
docs/kubernetes/persistence/rook-ceph/index.md Normal file
View File

@@ -0,0 +1,19 @@
---
title: How to use Rook Ceph for Persistent Storage in Kubernetes
description: How to deploy Rook Ceph into your Kubernetes cluster for persistent storage
---
# Persistent storage in Kubernetes with Rook Ceph / CephFS
[Ceph](https://docs.ceph.com/en/quincy/) is a highly-reliable, scalable network storage platform which uses individual disks across participating nodes to provide fault-tolerant storage.
![Ceph Screenshot](/images/ceph.png){ loading=lazy }
[Rook](https://rook.io) provides an operator for Ceph, decomposing the [10-year-old](https://en.wikipedia.org/wiki/Ceph_(software)#Release_history), at-time-arcane, platform into cloud-native components, created declaratively, whose lifecycle is managed by an operator.
The simplest way to think about running rook-ceph is separate the [operator](/kubernetes/persistence/rook-ceph/operator/) (*a generic worker which manages the lifecycle of your cluster*) from your desired [cluster](/kubernetes/persistence/rook-ceph/cluster/) config itself (*spec*).
To this end, I've defined each as a separate component, below:
1. First, install the [operator](/kubernetes/persistence/rook-ceph/operator/)
2. Then, define your [cluster](/kubernetes/persistence/rook-ceph/cluster/)
3. Win!

182
docs/kubernetes/persistence/rook-ceph/operator.md Normal file
View File

@@ -0,0 +1,182 @@
---
title: Deploy Rook Ceph Operator for Persistent Storage in Kubernetes
description: Start your Rook Ceph deployment by installing the operator into your Kubernetes cluster
---
# Persistent storage in Kubernetes with Rook Ceph / CephFS - Operator
[Ceph](https://docs.ceph.com/en/quincy/) is a highly-reliable, scalable network storage platform which uses individual disks across participating nodes to provide fault-tolerant storage.
[Rook](https://rook.io) provides an operator for Ceph, decomposing the [10-year-old](https://en.wikipedia.org/wiki/Ceph_(software)#Release_history), at-time-arcane, platform into cloud-native components, created declaratively, whose lifecycle is managed by an operator.
To start off with, we need to deploy the ceph operator into the cluster, after which, we'll be able to actually deploy our [ceph cluster itself](/kubernetes/persistence/rook-ceph/cluster/).
## Rook Ceph requirements
!!! summary "Ingredients"
Already deployed:
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
## Preparation
### Namespace
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `/bootstrap/namespaces/namespace-rook-system.yaml`:
```yaml title="/bootstrap/namespaces/namespace-rook-ceph.yaml"
apiVersion: v1
kind: Namespace
metadata:
name: rook-system
```
### HelmRepository
We're going to install a helm chart from the Rook Ceph chart repository, so I create the following in my flux repo:
```yaml title="/bootstrap/helmrepositories/gitepository-rook-release.yaml"
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: rook-release
namespace: flux-system
spec:
interval: 15m
url: https://charts.rook.io/release
```
### Kustomization
Now that the "global" elements of this deployment (*just the HelmRepository in this case*) have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at `/rook-ceph`. I create this example Kustomization in my flux repo:
```yaml title="/bootstrap/kustomizations/kustomization-rook-ceph.yaml"
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: rook-ceph
namespace: flux-system
spec:
interval: 30m
path: ./rook-ceph
prune: true # remove any elements later removed from the above path
timeout: 10m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
healthChecks:
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: cephblockpools.ceph.rook.io
```
--8<-- "premix-cta-kubernetes.md"
### ConfigMap
Now we're into the app-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/rook/rook/blob/master/deploy/charts/rook-ceph/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo:
```yaml title="rook-ceph/configmap-rook-ceph-helm-chart-value-overrides.yaml"
apiVersion: v1
kind: ConfigMap
metadata:
name: rook-ceph-helm-chart-value-overrides
namespace: rook-ceph
data:
values.yaml: |- # (1)!
# <upstream values go here>
```
1. Paste in the contents of the upstream `values.yaml` here, intended 4 spaces, and then change the values you need as illustrated below.
Values I change from the default are:
```yaml
pspEnable: false # (1)!
```
1. PSPs are deprecated, and will eventually be removed in Kubernetes 1.25, at which point this will cause breakage.
### HelmRelease
Finally, having set the scene above, we define the HelmRelease which will actually deploy the rook-ceph operator into the cluster. I save this in my flux repo:
```yaml title="/rook-ceph/helmrelease-rook-ceph.yaml"
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
chart:
spec:
chart: rook-ceph
version: 1.9.x
sourceRef:
kind: HelmRepository
name: rook-release
namespace: flux-system
interval: 30m
timeout: 10m
install:
remediation:
retries: 3
upgrade:
remediation:
retries: -1 # keep trying to remediate
crds: CreateReplace # Upgrade CRDs on package update
releaseName: rook-ceph
valuesFrom:
- kind: ConfigMap
name: rook-ceph-helm-chart-value-overrides
valuesKey: values.yaml # (1)!
```
1. This is the default, but best to be explicit for clarity
## Install Rook Ceph Operator!
Commit the changes to your flux repository, and either wait for the reconciliation interval, or force a reconcilliation using `flux reconcile source git flux-system`. You should see the kustomization appear...
```bash
~ ❯ flux get kustomizations rook-ceph
NAME READY MESSAGE REVISION SUSPENDED
rook-ceph True Applied revision: main/70da637 main/70da637 False
~ ❯
```
The helmrelease should be reconciled...
```bash
~ ❯ flux get helmreleases -n rook-ceph rook-ceph
NAME READY MESSAGE REVISION SUSPENDED
rook-ceph True Release reconciliation succeeded v1.9.9 False
~ ❯
```
And you should have happy rook-ceph operator pods:
```bash
~ ❯ k get pods -n rook-ceph -l app=rook-ceph-operator
NAME READY STATUS RESTARTS AGE
rook-ceph-operator-7c94b7446d-nwsss 1/1 Running 0 5m14s
~ ❯
```
## Summary
What have we achieved? We're half-way to getting a ceph cluster, having deployed the operator which will manage the lifecycle of the [ceph cluster](/kubernetes/persistence/rook-ceph/cluster/) we're about to create!
!!! summary "Summary"
Created:
* [X] Rook ceph operator running and ready to deploy a cluster!
Next:
* [ ] Deploy the ceph [cluster](/kubernetes/persistence/rook-ceph/cluster/) using a CR
--8<-- "recipe-footer.md"

282
docs/kubernetes/persistence/topolvm.md Normal file
View File

@@ -0,0 +1,282 @@
---
title: TopoLVM - Capacity-aware LVM-based storage on Kubernetes
---
# TopoLVM on Kubernetes
TopoLVM is **like** [Local Path Provisioner](/kubernetes/persistence/local-path-provisioner/), in that it deals with local volumes specific to each Kubernetes node, but it offers more flexibility, and is more suited for a production deployment.
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
* [x] A dedicated disk, or free LVM volume space, for provisioning volumes
Additional benefits offered by TopoLVM are:
* Volumes can by dynamically expanded
* The scheduler is capacity-aware, and can schedule pods to nodes with enough capacity for the pods' storage requirements
* Multiple storageclasses are supported, so you could, for example, create a storageclass for HDD-backed volumes, and another for SSD-backed volumes
## Preparation
### Volume Group
Finally you get to do something on your nodes without YAML or git, like a pre-GitOps, bare-metal-cavemonkey! :monkey_face:
On each node, you'll need an LVM Volume Group (VG) for TopoLVM to consume. The most straightforward to to arrange this is to dedicate a disk to TopoLVM, and create a dedicated PV and VG for it.
In brief, assuming `/dev/sdb` is the disk (*and it's unused*), you'd do the following to create a VG called `VG-topolvm`:
```bash
pvcreate /dev/sdb
vgcreate VG-topolvm /dev/sdb
```
!!! tip
If you don't have a dedicated disk, you could try installing your OS using LVM partitioning, and leave some space unused, for TopoLVM to consume. Run `vgs` from an installed node to work out what the VG name is that the OS installer chose.
### Namespace
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/namespaces/namespace-topolvm.yaml`:
??? example "Example NameSpace (click to expand)"
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: topolvm-system
```
### HelmRepository
Next, we need to define a HelmRepository (*a repository of helm charts*), to which we'll refer when we create the HelmRelease. We only need to do this once per-repository. In this case, we're using the official [TopoLVM helm chart](https://github.com/topolvm/topolvm/tree/main/charts/topolvm), so per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/helmrepositories/helmrepository-topolvm.yaml`:
??? example "Example HelmRepository (click to expand)"
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: topolvm
namespace: flux-system
spec:
interval: 15m
url: https://topolvm.github.io/topolvm
```
### Kustomization
Now that the "global" elements of this deployment (*Namespace and HelmRepository*) have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at `/topolvm`. I create this example Kustomization in my flux repo at `bootstrap/kustomizations/kustomization-topolvm.yaml`:
??? example "Example Kustomization (click to expand)"
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: topolvm--topolvm-system
namespace: flux-system
spec:
interval: 15m
path: ./topolvm-system
prune: true # remove any elements later removed from the above path
timeout: 2m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: topolvm-controller
namespace: topolvm-system
- apiVersion: apps/v1
kind: DaemonSet
name: topolvm-lvmd-0
namespace: topolvm-system
- apiVersion: apps/v1
kind: DaemonSet
name: topolvm-node
namespace: topolvm-system
- apiVersion: apps/v1
kind: DaemonSet
name: topolvm-scheduler
namespace: topolvm-system
```
!!! question "What's with that screwy name?"
> Why'd you call the kustomization `topolvm--topolvm-system`?
I keep my file and object names as consistent as possible. In most cases, the helm chart is named the same as the namespace, but in some cases, by upstream chart or historical convention, the namespace is different to the chart name. TopoLVM is one of these - the helmrelease/chart name is `topolvm`, but the typical namespace it's deployed in is `topolvm-system`. (*Appending `-system` seems to be a convention used in some cases for applications which support the entire cluster*). To avoid confusion when I list all kustomizations with `kubectl get kustomization -A`, I give these oddballs a name which identifies both the helmrelease and the namespace.
### ConfigMap
Now we're into the topolvm-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/topolvm/topolvm/blob/main/charts/topolvm/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo at `topolvm/configmap-topolvm-helm-chart-value-overrides.yaml`:
??? example "Example ConfigMap (click to expand)"
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: topolvm-helm-chart-value-overrides
namespace: topolvm
data:
values.yaml: |-
# paste chart values.yaml (indented) here and alter as required>
```
--8<-- "kubernetes-why-full-values-in-configmap.md"
Then work your way through the values you pasted, and change any which are specific to your configuration. You might want to start off by changing the following to match the name of the [volume group you created above](#volume-group).[^1]
```yaml hl_lines="10-13"
lvmd:
# lvmd.managed -- If true, set up lvmd service with DaemonSet.
managed: true
# lvmd.socketName -- Specify socketName.
socketName: /run/topolvm/lvmd.sock
# lvmd.deviceClasses -- Specify the device-class settings.
deviceClasses:
- name: ssd
volume-group: myvg1
default: true
spare-gb: 10
```
### HelmRelease
Lastly, having set the scene above, we define the HelmRelease which will actually deploy TopoLVM into the cluster, with the config we defined above. I save this in my flux repo as `topolvm/helmrelease-topolvm.yaml`:
??? example "Example HelmRelease (click to expand)"
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: topolvm
namespace: topolvm-system
spec:
chart:
spec:
chart: topolvm
version: 3.x
sourceRef:
kind: HelmRepository
name: topolvm
namespace: flux-system
interval: 15m
timeout: 5m
releaseName: topolvm
valuesFrom:
- kind: ConfigMap
name: topolvm-helm-chart-value-overrides
valuesKey: values.yaml # This is the default, but best to be explicit for clarity
```
--8<-- "kubernetes-why-not-config-in-helmrelease.md"
## Serving
### Deploy TopoLVM
Having committed the above to your flux repository, you should shortly see a topolvm kustomization, and in the `topolvm-system` namespace, a bunch of pods:
```bash
demo@shredder:~$ kubectl get pods -n topolvm-system
NAME READY STATUS RESTARTS AGE
topolvm-controller-85698b44dd-65fd9 4/4 Running 0 133m
topolvm-controller-85698b44dd-dmncr 4/4 Running 0 133m
topolvm-lvmd-0-98h4q 1/1 Running 0 133m
topolvm-lvmd-0-b29t8 1/1 Running 0 133m
topolvm-lvmd-0-c5vnf 1/1 Running 0 133m
topolvm-lvmd-0-hmmq5 1/1 Running 0 133m
topolvm-lvmd-0-zfldv 1/1 Running 0 133m
topolvm-node-6p4qz 3/3 Running 0 133m
topolvm-node-7vdgt 3/3 Running 0 133m
topolvm-node-mlp4x 3/3 Running 0 133m
topolvm-node-sxtn5 3/3 Running 0 133m
topolvm-node-xf265 3/3 Running 0 133m
topolvm-scheduler-jlwsh 1/1 Running 0 133m
topolvm-scheduler-nj8nz 1/1 Running 0 133m
topolvm-scheduler-tg72z 1/1 Running 0 133m
demo@shredder:~$
```
### How do I know it's working?
So the controllers etc are running, but how do we know we can actually provision volumes?
#### Create PVC
Create a PVC, by running:
```bash
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: topolvm-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: topolvm-provisioner
resources:
requests:
storage: 128Mi
EOF
```
Examine the PVC by running `kubectl describe pvc topolvm-pvc`
#### Create Pod
Now create a pod to consume the PVC, by running:
```bash
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: topolvm-test
spec:
containers:
- name: volume-test
image: nginx:stable-alpine
imagePullPolicy: IfNotPresent
volumeMounts:
- name: topolvm-rocks
mountPath: /data
ports:
- containerPort: 80
volumes:
- name: topolvm-rocks
persistentVolumeClaim:
claimName: topolvm-pvc
EOF
```
Examine the pod by running `kubectl describe pod topolvm-test`.
#### Clean up
Assuming that the pod is in a `Running` state, then TopoLVM is working!
Clean up your mess, little bare-metal-cave-monkey :monkey_face:, by running:
```bash
kubectl delete pod topolvm-test
kubectl delete pvc topolvm-pvc
```
### Troubleshooting
Are things not working as expected? Try one of the following to look for issues:
1. Watch the lvmd logs, by running `kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=topolvm-lvmd`
2. Watch the node logs, by running `kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=topolvm-node`
3. Watch the scheduler logs, by running `kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=scheduler`
4. Watch the controller node logs, by running `kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=controller`
--8<-- "recipe-footer.md"
[^1]: This is where you'd add multiple Volume Groups if you wanted a storageclass per Volume Group

632
docs/kubernetes/sealed-secrets.md Normal file
View File

@@ -0,0 +1,632 @@
---
description: Securely store your secrets in plain sight
---
# Sealed Secrets
So you're sold on GitOps, you're using the [flux deployment strategy](/kubernetes/deployment/flux/) to deploy all your applications into your cluster, and you sleep like a baby 🍼 at night, knowing that you could rebuild your cluster with a few commands, given every change is stored in git's history.
But what about your secrets?
In Kubernetes, a "Secret" is a "teeny-weeny" bit more secure ConfigMap, in that it's base-64 encoded to prevent shoulder-surfing, and access to secrets can be restricted (*separately to ConfigMaps*) using Kubernetes RBAC. In some cases, applications deployed via helm expect to find existing secrets within the cluster, containing things like AWS credentials (*External DNS, Cert Manager*), admin passwords (*Grafana*), etc.
They're still not very secret though, and you certainly wouldn't want to be storing base64-encoded secrets in a git repository, public or otherwise!
An elegant solution to this problem is Bitnami Labs' Sealed Secrets.
![Sealed Secrets illustration](/images/sealed-secrets.png){ loading=lazy }
A "[SealedSecret](https://github.com/bitnami-labs/sealed-secrets)" can only be decrypted (*and turned back into a regular Secret*) by the controller in the target cluster. (*or by a controller in another cluster which has been primed with your own private/public pair)* This means the SealedSecret is safe to store and expose anywhere.
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
Optional:
* [ ] Your own private/public PEM certificate pair for secret encryption/decryption (*ideal but not required*)
## Preparation
### Install kubeseal CLI
=== "HomeBrew (MacOS/Linux)"
With [Homebrew](https://brew.sh/) for macOS and Linux:
```bash
brew install kubeseal
```
=== "Bash (Linux)"
With Bash for macOS and Linux:
(Update for whatever the [latest release](https://github.com/bitnami-labs/sealed-secrets/releases) is)
```bash
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.17.0/kubeseal-linux-amd64 -O kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
```
### Namespace
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo:
```yaml title="/bootstrap/namespaces/namespace-sealed-secrets.yaml"
apiVersion: v1
kind: Namespace
metadata:
name: sealed-secrets
```
### HelmRepository
Next, we need to define a HelmRepository (*a repository of helm charts*), to which we'll refer when we create the HelmRelease. We only need to do this once per-repository. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/helmrepositories/helmrepository-sealedsecrets.yaml`:
??? example "Example HelmRepository (click to expand)"
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: sealed-secrets
namespace: flux-system
spec:
interval: 15m
url: https://bitnami-labs.github.io/sealed-secrets
```
### Kustomization
Now that the "global" elements of this deployment (*just the HelmRepository in this case*z*) have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at `/sealed-secrets`. I create this example Kustomization in my flux repo at `bootstrap/kustomizations/kustomization-sealed-secrets.yaml`:
??? example "Example Kustomization (click to expand)"
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: sealed-secrets
namespace: flux-system
spec:
interval: 15m
path: ./sealed-secrets
prune: true # remove any elements later removed from the above path
timeout: 2m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: sealed-secrets
namespace: sealed-secrets
```
### ConfigMap
{% raw %}
Now we're into the sealed-secrets-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/bitnami-labs/sealed-secrets/blob/main/helm/sealed-secrets/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo at `sealed-secrets/configmap-sealed-secrets-helm-chart-value-overrides.yaml`:
??? example "Example ConfigMap (click to expand)"
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: sealed-secrets-helm-chart-value-overrides
namespace: sealed-secrets
data:
values.yaml: |-
## @section Common parameters
## @param kubeVersion Override Kubernetes version
##
kubeVersion: ""
## @param nameOverride String to partially override sealed-secrets.fullname
##
nameOverride: ""
## @param fullnameOverride String to fully override sealed-secrets.fullname
##
fullnameOverride: ""
## @param namespace Namespace where to deploy the Sealed Secrets controller
##
namespace: ""
## @param extraDeploy [array] Array of extra objects to deploy with the release
##
extraDeploy: []
## @section Sealed Secrets Parameters
## Sealed Secrets image
## ref: https://quay.io/repository/bitnami/sealed-secrets-controller?tab=tags
## @param image.registry Sealed Secrets image registry
## @param image.repository Sealed Secrets image repository
## @param image.tag Sealed Secrets image tag (immutable tags are recommended)
## @param image.pullPolicy Sealed Secrets image pull policy
## @param image.pullSecrets [array] Sealed Secrets image pull secrets
##
image:
registry: quay.io
repository: bitnami/sealed-secrets-controller
tag: v0.17.2
## Specify a imagePullPolicy
## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
## e.g:
## pullSecrets:
## - myRegistryKeySecretName
##
pullSecrets: []
## @param createController Specifies whether the Sealed Secrets controller should be created
##
createController: true
## @param secretName The name of an existing TLS secret containing the key used to encrypt secrets
##
secretName: "sealed-secrets-key"
## Sealed Secret resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
## @param resources.limits [object] The resources limits for the Sealed Secret containers
## @param resources.requests [object] The requested resources for the Sealed Secret containers
##
resources:
limits: {}
requests: {}
## Configure Pods Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
## @param podSecurityContext.enabled Enabled Sealed Secret pods' Security Context
## @param podSecurityContext.fsGroup Set Sealed Secret pod's Security Context fsGroup
##
podSecurityContext:
enabled: true
fsGroup: 65534
## Configure Container Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
## @param containerSecurityContext.enabled Enabled Sealed Secret containers' Security Context
## @param containerSecurityContext.readOnlyRootFilesystem Whether the Sealed Secret container has a read-only root filesystem
## @param containerSecurityContext.runAsNonRoot Indicates that the Sealed Secret container must run as a non-root user
## @param containerSecurityContext.runAsUser Set Sealed Secret containers' Security Context runAsUser
##
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1001
## @param podLabels [object] Extra labels for Sealed Secret pods
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
##
podLabels: {}
## @param podAnnotations [object] Annotations for Sealed Secret pods
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
##
podAnnotations: {}
## @param priorityClassName Sealed Secret pods' priorityClassName
##
priorityClassName: ""
## @param affinity [object] Affinity for Sealed Secret pods assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
##
affinity: {}
## @param nodeSelector [object] Node labels for Sealed Secret pods assignment
## ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## @param tolerations [array] Tolerations for Sealed Secret pods assignment
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
tolerations: []
## @param updateStatus Specifies whether the Sealed Secrets controller should update the status subresource
##
updateStatus: true
## @section Traffic Exposure Parameters
## Sealed Secret service parameters
##
service:
## @param service.type Sealed Secret service type
##
type: ClusterIP
## @param service.port Sealed Secret service HTTP port
##
port: 8080
## @param service.nodePort Node port for HTTP
## Specify the nodePort value for the LoadBalancer and NodePort service types
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
## NOTE: choose port between <30000-32767>
##
nodePort: ""
## @param service.annotations [object] Additional custom annotations for Sealed Secret service
##
annotations: {}
## Sealed Secret ingress parameters
## ref: http://kubernetes.io/docs/user-guide/ingress/
##
ingress:
## @param ingress.enabled Enable ingress record generation for Sealed Secret
##
enabled: false
## @param ingress.pathType Ingress path type
##
pathType: ImplementationSpecific
## @param ingress.apiVersion Force Ingress API version (automatically detected if not set)
##
apiVersion: ""
## @param ingress.ingressClassName IngressClass that will be be used to implement the Ingress
## This is supported in Kubernetes 1.18+ and required if you have more than one IngressClass marked as the default for your cluster.
## ref: https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/
##
ingressClassName: ""
## @param ingress.hostname Default host for the ingress record
##
hostname: sealed-secrets.local
## @param ingress.path Default path for the ingress record
##
path: /v1/cert.pem
## @param ingress.annotations [object] Additional annotations for the Ingress resource. To enable certificate autogeneration, place here your cert-manager annotations.
## Use this parameter to set the required annotations for cert-manager, see
## ref: https://cert-manager.io/docs/usage/ingress/#supported-annotations
## e.g:
## annotations:
## kubernetes.io/ingress.class: nginx
## cert-manager.io/cluster-issuer: cluster-issuer-name
##
annotations:
## @param ingress.tls Enable TLS configuration for the host defined at `ingress.hostname` parameter
## TLS certificates will be retrieved from a TLS secret with name: `{{- printf "%s-tls" .Values.ingress.hostname }}`
## You can:
## - Use the `ingress.secrets` parameter to create this TLS secret
## - Relay on cert-manager to create it by setting the corresponding annotations
## - Relay on Helm to create self-signed certificates by setting `ingress.selfSigned=true`
##
tls: false
## @param ingress.selfSigned Create a TLS secret for this ingress record using self-signed certificates generated by Helm
##
selfSigned: false
## @param ingress.extraHosts [array] An array with additional hostname(s) to be covered with the ingress record
## e.g:
## extraHosts:
## - name: sealed-secrets.local
## path: /
##
extraHosts: []
## @param ingress.extraPaths [array] An array with additional arbitrary paths that may need to be added to the ingress under the main host
## e.g:
## extraPaths:
## - path: /*
## backend:
## serviceName: ssl-redirect
## servicePort: use-annotation
##
extraPaths: []
## @param ingress.extraTls [array] TLS configuration for additional hostname(s) to be covered with this ingress record
## ref: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
## e.g:
## extraTls:
## - hosts:
## - sealed-secrets.local
## secretName: sealed-secrets.local-tls
##
extraTls: []
## @param ingress.secrets [array] Custom TLS certificates as secrets
## NOTE: 'key' and 'certificate' are expected in PEM format
## NOTE: 'name' should line up with a 'secretName' set further up
## If it is not set and you're using cert-manager, this is unneeded, as it will create a secret for you with valid certificates
## If it is not set and you're NOT using cert-manager either, self-signed certificates will be created valid for 365 days
## It is also possible to create and manage the certificates outside of this helm chart
## Please see README.md for more information
## e.g:
## secrets:
## - name: sealed-secrets.local-tls
## key: |-
## -----BEGIN RSA PRIVATE KEY-----
## ...
## -----END RSA PRIVATE KEY-----
## certificate: |-
## -----BEGIN CERTIFICATE-----
## ...
## -----END CERTIFICATE-----
##
secrets: []
## Network policies
## Ref: https://kubernetes.io/docs/concepts/services-networking/network-policies/
##
networkPolicy:
## @param networkPolicy.enabled Specifies whether a NetworkPolicy should be created
##
enabled: false
## @section Other Parameters
## ServiceAccount configuration
##
serviceAccount:
## @param serviceAccount.create Specifies whether a ServiceAccount should be created
##
create: true
## @param serviceAccount.labels Extra labels to be added to the ServiceAccount
##
labels: {}
## @param serviceAccount.name The name of the ServiceAccount to use.
## If not set and create is true, a name is generated using the sealed-secrets.fullname template
##
name: ""
## RBAC configuration
##
rbac:
## @param rbac.create Specifies whether RBAC resources should be created
##
create: true
## @param rbac.labels Extra labels to be added to RBAC resources
##
labels: {}
## @param rbac.pspEnabled PodSecurityPolicy
##
pspEnabled: false
## @section Metrics parameters
metrics:
## Prometheus Operator ServiceMonitor configuration
##
serviceMonitor:
## @param metrics.serviceMonitor.enabled Specify if a ServiceMonitor will be deployed for Prometheus Operator
##
enabled: false
## @param metrics.serviceMonitor.namespace Namespace where Prometheus Operator is running in
##
namespace: ""
## @param metrics.serviceMonitor.labels Extra labels for the ServiceMonitor
##
labels: {}
## @param metrics.serviceMonitor.annotations Extra annotations for the ServiceMonitor
##
annotations: {}
## @param metrics.serviceMonitor.interval How frequently to scrape metrics
## e.g:
## interval: 10s
##
interval: ""
## @param metrics.serviceMonitor.scrapeTimeout Timeout after which the scrape is ended
## e.g:
## scrapeTimeout: 10s
##
scrapeTimeout: ""
## @param metrics.serviceMonitor.metricRelabelings [array] Specify additional relabeling of metrics
##
metricRelabelings: []
## @param metrics.serviceMonitor.relabelings [array] Specify general relabeling
##
relabelings: []
## Grafana dashboards configuration
##
dashboards:
## @param metrics.dashboards.create Specifies whether a ConfigMap with a Grafana dashboard configuration should be created
## ref https://github.com/helm/charts/tree/master/stable/grafana#configuration
##
create: false
## @param metrics.dashboards.labels Extra labels to be added to the Grafana dashboard ConfigMap
##
labels: {}
## @param metrics.dashboards.namespace Namespace where Grafana dashboard ConfigMap is deployed
##
namespace: ""
```
--8<-- "kubernetes-why-full-values-in-configmap.md"
Then work your way through the values you pasted, and change any which are specific to your configuration (*I stick with the defaults*).
### HelmRelease
Lastly, having set the scene above, we define the HelmRelease which will actually deploy the sealed-secrets controller into the cluster, with the config we defined above. I save this in my flux repo as `sealed-secrets/helmrelease-sealed-secrets.yaml`:
??? example "Example HelmRelease (click to expand)"
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: sealed-secrets
namespace: sealed-secrets
spec:
chart:
spec:
chart: sealed-secrets
version: 1.x
sourceRef:
kind: HelmRepository
name: sealed-secrets
namespace: flux-system
interval: 15m
timeout: 5m
releaseName: sealed-secrets
valuesFrom:
- kind: ConfigMap
name: sealed-secrets-helm-chart-value-overrides
valuesKey: values.yaml # This is the default, but best to be explicit for clarity
```
--8<-- "kubernetes-why-not-config-in-helmrelease.md"
## Serving
Commit your files to your flux repo, and wait until you see pods show up in the `sealed-secrets` namespace.
Now you're ready to seal some secrets!
### Sealing a secret
To generate sealed secrets, we need the public key that the controller has generated. On a host with a valid `KUBECONFIG` env var, pointing to a kubeconfig file with cluster-admin privileges, run the following to retrieve the public key for the sealed secrets (*this is the public key, it doesn't need to be specifically protected*)
{% endraw %}
```bash
kubeseal --fetch-cert \
--controller-name=sealed-secrets \
--controller-namespace=sealed-secrets \
> pub-cert.pem
```
Now generate a kubernetes secret locally, using `kubectl --dry-run=client`, as illustrated below:
```bash
echo -n batman | kubectl create secret \
generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o json
```
The result should look like this:
```yaml
{
"kind": "Secret",
"apiVersion": "v1",
"metadata": {
"name": "mysecret",
"creationTimestamp": null
},
"data": {
"foo": "YmF0bWFu"
}
}
```
Note that "*YmF0bWFu*", [base64 decoded](https://www.base64decode.org/), will reveal the top-secret secret. Not so secret, Batman!
Next, pipe the secret (*in json format*) to kubeseal, referencing the public key, and you'll get a totally un-decryptable "sealed" secret in return:
```bash
echo -n batman | kubectl create secret \
generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o json \
| kubeseal --cert pub-cert.pem
```
Resulting in something like this:
```json
{
"kind": "SealedSecret",
"apiVersion": "bitnami.com/v1alpha1",
"metadata": {
"name": "mysecret",
"namespace": "default",
"creationTimestamp": null
},
"spec": {
"template": {
"metadata": {
"name": "mysecret",
"namespace": "default",
"creationTimestamp": null
},
"data": null
},
"encryptedData": {
"foo": "AgAywfMzHx/4QFa3sa68zUbpmejT/MjuHUnfI/p2eo5xFKf2SsdGiRK4q2gl2yaSeEcAlA/P1vKZpsM+Jlh5WqrFxTtJjTYgXilzTSSTkK8hilZMflCnL1xs7ywH/lk+4gHdI7z0QS7FQztc649Z+SP2gjunOmTnRTczyCbzYlYSdHS9bB7xqLvGIofvn4dtQvapiTIlaFKhr+sDNtd8WVVzJ1eLuGgc9g6u1UjhuGa8NhgQnzXBd4zQ7678pKEpkXpUmINEKMzPchp9+ME5tIDASfV/R8rxkKvwN3RO3vbCNyLXw7KXRdyhd276kfHP4p4s9nUWDHthefsh19C6lT0ixup3PiG6gT8eFPa0v4jenxqtKNczmTwN9+dF4ZqHh93cIRvffZ7RS9IUOc9kUObQgvp3fZlo2B4m36G7or30ZfuontBh4h5INQCH8j/U3tXegGwaShGmKWg+kRFYQYC4ZqHCbNQJtvTHWKELQTStoAiyHyM+T36K6nCoJTixGZ/Nq4NzIvVfcp7I8LGzEbRSTdaO+MlTT3d32HjsJplXZwSzygSNrRRGwHKr5wfo5rTTdBVuZ0A1u1a6aQPQiJYSluKZwAIJKGQyfZC5Fbo+NxSxKS8MoaZjQh5VUPB+Q92WoPJoWbqZqlU2JZOuoyDWz5x7ZS812x1etQCy6QmuLYe+3nXOuQx85drJFdNw4KXzoQs2uSA="
}
}
}
```
!!! question "Who set the namespace to default?"
By default, sealed secrets can only be "unsealed" in the same namespace for which the original secret was created. In the example above, we didn't explicitly specity a namespace when creating our secret, so the default namespace was used.
Apply the sealed secret to the cluster...
```bash
echo -n batman | kubectl create secret \
generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o json \
| kubeseal --cert pub-cert.pem \
| kubectl create -f -
```
And watch the sealed-secrets controller decrypt it, and turn it into a regular secrets, using `kubectl logs -n sealed-secrets -l app.kubernetes.io/name=sealed-secrets`
```bash
2021/11/16 10:37:16 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"mysecret",
UID:"82ac8c4b-c167-400e-8768-51957364f6b9", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"147314",
FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
```
Finally, confirm that the secret now exists in the `default` namespace:
```yaml
root@shredder:/tmp# kubectl get secret mysecret -o yaml
apiVersion: v1
data:
foo: YmF0bWFu
kind: Secret
metadata:
creationTimestamp: "2021-11-16T10:37:16Z"
name: mysecret
namespace: default
ownerReferences:
- apiVersion: bitnami.com/v1alpha1
controller: true
kind: SealedSecret
name: mysecret
uid: 82ac8c4b-c167-400e-8768-51957364f6b9
resourceVersion: "147315"
uid: 6f6ba81c-c9a2-45bc-877c-7a8b50afde83
type: Opaque
root@shredder:/tmp#
```
So we now have a means to store an un-decryptable secret in our flux repo, and have only our cluster be able to convert that sealedsecret into a regular secret!
Based on our [flux deployment strategy](/kubernetes/deployment/flux/), we simply seal up any necessary secrets into the appropriate folder in the flux repository, and have them decrypted and unsealed into the running cluster. For example, if we needed a secret for metallb called "magic-password", containing a key "location-of-rabbit", we'd do this:
```bash
kubectl create secret generic magic-password \
--namespace metallb-system \
--dry-run=client \
--from-literal=location-of-rabbit=top-hat -o json \
| kubeseal --cert pub-cert.pem \
| kubectl create -f - \
> <path to repo>/metallb/sealedsecret-magic-password.yaml
```
Once flux reconciled the above sealedsecret, the sealedsecrets controller in the cluster would confirm that it's able to decrypt the secret, and would create the corresponding regular secret.
### Using our own keypair
One flaw in the process above is that we rely on the sealedsecrets controller to generate its own public/private keypair. This means that the pair (*and therefore all the encrypted secrets*) are specific to this cluster (*and this instance of the sealedsecrets controller*) only.
To go "fully GitOps", we'd want to be able to rebuild our entire cluster "from scratch" using our flux repository. If the keypair is recreated when a new cluster is built, then the existing sealedsecrets would remain forever "sealed"..
The solution here is to [generate our own public/private keypair](https://github.com/bitnami-labs/sealed-secrets/blob/main/docs/bring-your-own-certificates.md), and to store the private key safely and securely outside of the flux repo[^1]. We'll only need the key once, when deploying a fresh instance of the sealedsecrets controller.
Once you've got the public/private key pair, create them as kubernetes secrets directly in the cluster, like this:
```bash
kubectl -n sealed-secrets create secret tls my-own-certs \
--cert="<path to public key>" --key="<path to private key>"
```
And then "label" the secret you just created, so that the sealedsecrets controller knows that it's special:
```bash
kubectl -n sealed-secrets label secret my-own-certs \
sealedsecrets.bitnami.com/sealed-secrets-key=active
```
Restart the sealedsecret controller deployment, to force it to detect the new secret:
```bash
root@shredder:~# kubectl rollout restart -n sealed-secrets deployment sealed-secrets
deployment.apps/sealed-secrets restarted
root@shredder:~#
```
And now when you create your seadsecrets, refer to the public key you just created using `--cert <path to cert>`. These secrets will be decryptable by **any** sealedsecrets controller bootstrapped with the same keypair (*above*).
--8<-- "recipe-footer.md"
[^1]: There's no harm in storing the **public** key in the repo though, which means it's easy to refer to when sealing secrets.

179
docs/kubernetes/snapshots.md Normal file
View File

@@ -0,0 +1,179 @@
# Snapshots
Before we get carried away creating pods, services, deployments etc, let's spare a thought for _security_... (_DevSecPenguinOps, here we come!_). In the context of this recipe, security refers to safe-guarding your data from accidental loss, as well as malicious impact.
Under [Docker Swarm](/docker-swarm/design/), we used [shared storage](/docker-swarm/shared-storage-ceph/) with [Duplicity](/recipes/duplicity/) (or [ElkarBackup](/recipes/elkarbackup/)) to automate backups of our persistent data.
Now that we're playing in the deep end with Kubernetes, we'll need a Cloud-native backup solution...
It bears repeating though - don't be like [Cameron](http://haltandcatchfire.wikia.com/wiki/Cameron_Howe). Backup your stuff.
<!-- markdownlint-disable MD033 -->
<iframe width="560" height="315" src="https://www.youtube.com/embed/1UtFeMoqVHQ" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
This recipe employs a clever tool ([miracle2k/k8s-snapshots](https://github.com/miracle2k/k8s-snapshots)), running _inside_ your cluster, to trigger automated snapshots of your persistent volumes, using your cloud provider's APIs.
## Ingredients
1. [Kubernetes cluster](/kubernetes/cluster/) with either AWS or GKE (currently, but apparently other providers are [easy to implement](https://github.com/miracle2k/k8s-snapshots/blob/master/k8s_snapshots/backends/abstract.py))
2. Geek-Fu required : 🐒🐒 (_medium - minor adjustments may be required_)
## Preparation
### Create RoleBinding (GKE only)
If you're running GKE, run the following to create a RoleBinding, allowing your user to grant rights-it-doesn't-currently-have to the service account responsible for creating the snapshots:
````kubectl create clusterrolebinding your-user-cluster-admin-binding \
--clusterrole=cluster-admin --user=<your user@yourdomain>```
!!! question
Why do we have to do this? Check [this blog post](https://www.funkypenguin.co.nz/workaround-blocked-attempt-to-grant-extra-privileges-on-gke/) for details
### Apply RBAC
If your cluster is RBAC-enabled (_it probably is_), you'll need to create a ClusterRole and ClusterRoleBinding to allow k8s_snapshots to see your PVs and friends:
````bash
kubectl apply -f https://raw.githubusercontent.com/miracle2k/k8s-snapshots/master/rbac.yaml
```
## Serving
### Deploy the pod
Ready? Run the following to create a deployment in to the kube-system namespace:
```bash
cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: k8s-snapshots
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
app: k8s-snapshots
spec:
containers: - name: k8s-snapshots
image: elsdoerfer/k8s-snapshots:v2.0
EOF
````
Confirm your pod is running and happy by running ```kubectl get pods -n kubec-system```, and ```kubectl -n kube-system logs k8s-snapshots<tab-to-auto-complete>```
### Pick PVs to snapshot
k8s-snapshots relies on annotations to tell it how frequently to snapshot your PVs. A PV requires the ```backup.kubernetes.io/deltas``` annotation in order to be snapshotted.
From the k8s-snapshots README:
> The generations are defined by a list of deltas formatted as ISO 8601 durations (this differs from tarsnapper). PT60S or PT1M means a minute, PT12H or P0.5D is half a day, P1W or P7D is a week. The number of backups in each generation is implied by it's and the parent generation's delta.
>
> For example, given the deltas PT1H P1D P7D, the first generation will consist of 24 backups each one hour older than the previous (or the closest approximation possible given the available backups), the second generation of 7 backups each one day older than the previous, and backups older than 7 days will be discarded for good.
>
> The most recent backup is always kept.
>
> The first delta is the backup interval.
To add the annotation to an existing PV, run something like this:
```bash
kubectl patch pv pvc-01f74065-8fe9-11e6-abdd-42010af00148 -p \
'{"metadata": {"annotations": {"backup.kubernetes.io/deltas": "P1D P30D P360D"}}}'
```
To add the annotation to a _new_ PV, add the following annotation to your **PVC**:
```yaml
backup.kubernetes.io/deltas: PT1H P2D P30D P180D
```
Here's an example of the PVC for the UniFi recipe, which includes 7 daily snapshots of the PV:
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: controller-volumeclaim
namespace: unifi
annotations:
backup.kubernetes.io/deltas: P1D P7D
spec:
accessModes: - ReadWriteOnce
resources:
requests:
storage: 1Gi
````
And here's what my snapshot list looks like after a few days:
![Kubernetes Snapshots](/images/kubernetes-snapshots.png){ loading=lazy }
### Snapshot a non-Kubernetes volume (optional)
If you're running traditional compute instances with your cloud provider (I do this for my poor man's load balancer), you might want to backup _these_ volumes as well.
To do so, first create a custom resource, ```SnapshotRule```:
````bash
cat <<EOF | kubectl create -f -
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: snapshotrules.k8s-snapshots.elsdoerfer.com
spec:
group: k8s-snapshots.elsdoerfer.com
version: v1
scope: Namespaced
names:
plural: snapshotrules
singular: snapshotrule
kind: SnapshotRule
shortNames: - sr
EOF
````
Then identify the volume ID of your volume, and create an appropriate ```SnapshotRule```:
````bash
cat <<EOF | kubectl apply -f -
apiVersion: "k8s-snapshots.elsdoerfer.com/v1"
kind: SnapshotRule
metadata:
name: haproxy-badass-loadbalancer
spec:
deltas: P1D P7D
backend: google
disk:
name: haproxy2
zone: australia-southeast1-a
EOF
```
!!! note
Example syntaxes for the SnapshotRule for different providers can be found at https://github.com/miracle2k/k8s-snapshots/tree/master/examples
## Move on..
Still with me? Good. Move on to understanding Helm charts...
* [Start](/kubernetes/) - Why Kubernetes?
* [Design](/kubernetes/design/) - How does it fit together?
* [Cluster](/kubernetes/cluster/) - Setup a basic cluster
* [Load Balancer](/kubernetes/loadbalancer/) Setup inbound access
* Snapshots (this page) - Automatically backup your persistent data
* [Helm](/kubernetes/helm/) - Uber-recipes from fellow geeks
* [Traefik](/kubernetes/traefik/) - Traefik Ingress via Helm
[^1]: I've submitted [2 PRs](https://github.com/miracle2k/k8s-snapshots/pulls/funkypenguin) to the k8s-snapshots repo. The first [updates the README for GKE RBAC requirements](https://github.com/miracle2k/k8s-snapshots/pull/71), and the second [fixes a minor typo](https://github.com/miracle2k/k8s-snapshots/pull/74).
```
--8<-- "recipe-footer.md"

140
docs/kubernetes/ssl-certificates/cert-manager.md Normal file
View File

@@ -0,0 +1,140 @@
---
description: Cert Manager generates and renews LetsEncrypt certificates
---
# Cert Manager
To interact with your cluster externally, you'll almost certainly be using a web browser, and you'll almost certainly be wanting your browsing session to be SSL-secured. Some Ingress Controllers (i.e. Traefik) will include a default, self-signed, nasty old cert which will permit you to use SSL, but it's faaaar better to use valid certs.
Cert Manager adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using those certificates.
![Sealed Secrets illustration](/images/cert-manager.svg)
It can issue certificates from a variety of supported sources, including Let’s Encrypt, HashiCorp Vault, and Venafi as well as private PKI.
It will ensure certificates are valid and up to date, and attempt to renew certificates at a configured time before expiry.
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
## Preparation
### Namespace
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/namespaces/namespace-cert-manager.yaml`:
??? example "Example Namespace (click to expand)"
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager
```
### HelmRepository
Next, we need to define a HelmRepository (*a repository of helm charts*), to which we'll refer when we create the HelmRelease. We only need to do this once per-repository. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/helmrepositories/helmrepository-jetstack.yaml`:
??? example "Example HelmRepository (click to expand)"
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: jetstack
namespace: flux-system
spec:
interval: 15m
url: https://charts.jetstack.io
```
### Kustomization
Now that the "global" elements of this deployment (*just the HelmRepository in this case*z*) have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at `/cert-manager`. I create this example Kustomization in my flux repo at `bootstrap/kustomizations/kustomization-cert-manager.yaml`:
??? example "Example Kustomization (click to expand)"
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: cert-manager
namespace: flux-system
spec:
interval: 15m
path: ./cert-manager
prune: true # remove any elements later removed from the above path
timeout: 2m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: cert-manager
namespace: cert-manager
```
### ConfigMap
Now we're into the cert-manager-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/bitnami-labs/cert-manager/blob/main/helm/cert-manager/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo at `cert-manager/configmap-cert-manager-helm-chart-value-overrides.yaml`:
??? example "Example ConfigMap (click to expand)"
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: cert-manager-helm-chart-value-overrides
namespace: cert-manager
data:
values.yaml: |-
# paste chart values.yaml (indented) here and alter as required>
```
--8<-- "kubernetes-why-full-values-in-configmap.md"
Then work your way through the values you pasted, and change any which are specific to your configuration.
### HelmRelease
Lastly, having set the scene above, we define the HelmRelease which will actually deploy the cert-manager controller into the cluster, with the config we defined above. I save this in my flux repo as `cert-manager/helmrelease-cert-manager.yaml`:
??? example "Example HelmRelease (click to expand)"
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: cert-manager
namespace: cert-manager
spec:
chart:
spec:
chart: cert-manager
version: 1.6.x
sourceRef:
kind: HelmRepository
name: jetstack
namespace: flux-system
interval: 15m
timeout: 5m
releaseName: cert-manager
valuesFrom:
- kind: ConfigMap
name: cert-manager-helm-chart-value-overrides
valuesKey: values.yaml # This is the default, but best to be explicit for clarity
```
--8<-- "kubernetes-why-not-config-in-helmrelease.md"
## Serving
Once you've committed your YAML files into your repo, you should soon see some pods appear in the `cert-manager` namespace!
What do we have now? Well, we've got the cert-manager controller **running**, but it won't **do** anything until we define some certificate issuers, credentials, and certificates..
### Troubleshooting
If your certificate is not created **aren't** created as you expect, then the best approach is to check the cert-manager logs, by running `kubectl logs -n cert-manager -l app.kubernetes.io/name=cert-manager`.
--8<-- "recipe-footer.md"
[^1]: Why yes, I **have** accidentally rate-limited myself by deleting/recreating my prod certificates a few times!

22
docs/kubernetes/ssl-certificates/index.md Normal file
View File

@@ -0,0 +1,22 @@
# SSL Certificates
When you expose applications running within your cluster to the outside world, you're going to want to protect these with SSL certificates. Typically, this'll be SSL certificates used by browsers to access your Ingress resources over HTTPS, but SSL certificates would be used for other externally-facing services, for example OpenLDAP, docker-mailserver, etc.
!!! question "Why do I need SSL if it's just internal?"
It's true that you could expose applications via HTTP only, and **not** bother with SSL. By doing so, however, you "train yourself"[^1] to ignore SSL certificates / browser security warnings.
One day, this behaviour will bite you in the ass.
If you want to be a person who relies on privacy and security, then insist on privacy and security **everywhere**.
Plus, once you put in the effort to setup automated SSL certificates _once_, it's literally **no** extra effort to use them everywhere!
I've split this section, conceptually, into 3 separate tasks:
1. Setup [Cert Manager](/kubernetes/ssl-certificates/cert-manager/), a controller whose job it is to request / renew certificates
2. Setup "[Issuers](/kubernetes/ssl-certificates/letsencrypt-issuers/)" for LetsEncrypt, which Cert Manager will use to request certificates
3. Setup a [wildcard certificate](/kubernetes/ssl-certificates/wildcard-certificate/) in such a way that it can be used by Ingresses like Traefik or Nginx
--8<-- "recipe-footer.md"
[^1]: I had a really annoying but smart boss once who taught me this. Hi Mark! :wave:

109
docs/kubernetes/ssl-certificates/letsencrypt-issuers.md Normal file
View File

@@ -0,0 +1,109 @@
# LetsEncrypt Issuers
Certificates are issued by certificate authorities. By far the most common issuer will be LetsEncrypt.
In order for Cert Manager to request/renew certificates, we have to tell it about our **Issuers**.
!!! note
There's a minor distinction between an **Issuer** (*only issues certificates within one namespace*) and a **ClusterIssuer** (*issues certificates throughout the cluster*). Typically a **ClusterIssuer** will be suitable.
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
* [x] [Cert-Manager](/kubernetes/ssl-certificates/cert-manager/) deployed to request/renew certificates
* [x] API credentials for a [supported DNS01 provider](https://cert-manager.io/docs/configuration/acme/dns01/) for LetsEncrypt wildcard certs
## Preparation
### LetsEncrypt Staging
The ClusterIssuer resource below represents a certificate authority which is able to request certificates for any namespace within the cluster.
I save this in my flux repo as `letsencrypt-wildcard-cert/cluster-issuer-letsencrypt-staging.yaml`. I've highlighted the areas you'll need to pay attention to:
???+ example "ClusterIssuer for LetsEncrypt Staging"
```yaml hl_lines="8 15 17-21"
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: batman@example.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- selector:
dnsZones:
- "example.com"
dns01:
cloudflare:
email: batman@example.com
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token
```
Deploying this issuer YAML into the cluster would provide Cert Manager with the details necessary to start issuing certificates from the LetsEncrypt staging server (*always good to test in staging first!*)
!!! note
The example above is specific to [Cloudflare](https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/), but the syntax for [other providers](https://cert-manager.io/docs/configuration/acme/dns01/) is similar.
### LetsEncrypt Prod
As you'd imagine, the "prod" version of the LetsEncrypt issues is very similar, and I save this in my flux repo as `letsencrypt-wildcard-cert/cluster-issuer-letsencrypt-prod.yaml`
???+ example "ClusterIssuer for LetsEncrypt Prod"
```yaml hl_lines="8 15 17-21"
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: batman@example.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- selector:
dnsZones:
- "example.com"
dns01:
cloudflare:
email: batman@example.com
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token
```
!!! note
You'll note that there are two secrets referred to above - `privateKeySecretRef`, referencing `letsencrypt-prod` is for cert-manager to populate as a result of its ACME schenanigans - you don't have to do anything about this particular secret! The cloudflare-specific secret (*and this will change based on your provider*) is expected to be found in the same namespace as the certificate we'll be issuing, and will be discussed when we create our [wildcard certificate](/kubernetes/ssl-certificates/wildcard-certificate/).
## Serving
### How do we know it works?
We're not quite ready to issue certificates yet, but we can now test whether the Issuers are configured correctly for LetsEncrypt. To check their status, **describe** the ClusterIssuers (i.e., `kubectl describe clusterissuer letsencrypt-prod`), which (*truncated*) shows something like this:
```yaml
Status:
Acme:
Last Registered Email: admin@example.com
Uri: https://acme-v02.api.letsencrypt.org/acme/acct/34523
Conditions:
Last Transition Time: 2021-11-18T22:54:20Z
Message: The ACME account was registered with the ACME server
Observed Generation: 1
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
```
Provided your account is registered, you're ready to proceed with [creating a wildcard certificate](/kubernetes/ssl-certificates/wildcard-certificate/)!
--8<-- "recipe-footer.md"
[^1]: Since a ClusterIssuer is not a namespaced resource, it doesn't exist in any specific namespace. Therefore, my assumption is that the `apiTokenSecretRef` secret is only "looked for" when a certificate (*which __is__ namespaced*) requires validation.

175
docs/kubernetes/ssl-certificates/secret-replicator.md Normal file
View File

@@ -0,0 +1,175 @@
# Secret Replicator
As explained when creating our [LetsEncrypt Wildcard certificates](/kubernetes/ssl-certificates/wildcard-certificate/), it can be problematic that Certificates can't be **shared** between namespaces. One simple solution to this problem is simply to "replicate" secrets from one "source" namespace into all other namespaces.
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
* [x] [secret-replicator](/kubernetes/ssl-certificates/secret-replicator/) deployed to request/renew certificates
* [x] [LetsEncrypt Wildcard Certificates](/kubernetes/ssl-certificates/wildcard-certificate/) created in the `letsencrypt-wildcard-cert` namespace
Kiwigrid's "[Secret Replicator](https://github.com/kiwigrid/secret-replicator)" is a simple controller which replicates secrets from one namespace to another.[^1]
## Preparation
### Namespace
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/namespaces/namespace-secret-replicator.yaml`:
??? example "Example Namespace (click to expand)"
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: secret-replicator
```
### HelmRepository
Next, we need to define a HelmRepository (*a repository of helm charts*), to which we'll refer when we create the HelmRelease. We only need to do this once per-repository. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/helmrepositories/helmrepository-kiwigrid.yaml`:
??? example "Example HelmRepository (click to expand)"
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: kiwigrid
namespace: flux-system
spec:
interval: 15m
url: https://kiwigrid.github.io
```
### Kustomization
Now that the "global" elements of this deployment have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at `/secret-replicator`. I create this example Kustomization in my flux repo at `bootstrap/kustomizations/kustomization-secret-replicator.yaml`:
??? example "Example Kustomization (click to expand)"
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: secret-replicator
namespace: flux-system
spec:
interval: 15m
path: ./secret-replicator
prune: true # remove any elements later removed from the above path
timeout: 2m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: secret-replicator
namespace: secret-replicator
```
### ConfigMap
Now we're into the secret-replicator-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/kiwigrid/helm-charts/blob/master/charts/secret-replicator/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo at `secret-replicator/configmap-secret-replicator-helm-chart-value-overrides.yaml`:
??? example "Example ConfigMap (click to expand)"
```yaml hl_lines="21 27"
apiVersion: v1
kind: ConfigMap
metadata:
name: secret-replicator-helm-chart-value-overrides
namespace: secret-replicator
data:
values.yaml: |-
# Default values for secret-replicator.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
image:
repository: kiwigrid/secret-replicator
tag: 0.2.0
pullPolicy: IfNotPresent
## Specify ImagePullSecrets for Pods
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
# pullSecrets: myregistrykey
# csv list of secrets
secretList: "letsencrypt-wildcard-cert"
# secretList: "secret1,secret2
ignoreNamespaces: "kube-system,kube-public"
# If defined, allow secret-replicator to watch for secrets in _another_ namespace
secretNamespace: letsencrypt-wildcard-cert"
rbac:
enabled: true
resources: {}
# limits:
# cpu: 50m
# memory: 20Mi
# requests:
# cpu: 20m
# memory: 20Mi
nodeSelector: {}
tolerations: []
affinity: {}
```
--8<-- "kubernetes-why-full-values-in-configmap.md"
Note that the following values changed from default, above:
* `secretList`: `letsencrypt-wildcard-cert`
* `secretNamespace`: `letsencrypt-wildcard-cert`
### HelmRelease
Lastly, having set the scene above, we define the HelmRelease which will actually deploy the secret-replicator controller into the cluster, with the config we defined above. I save this in my flux repo as `secret-replicator/helmrelease-secret-replicator.yaml`:
??? example "Example HelmRelease (click to expand)"
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: secret-replicator
namespace: secret-replicator
spec:
chart:
spec:
chart: secret-replicator
version: 0.6.x
sourceRef:
kind: HelmRepository
name: kiwigrid
namespace: flux-system
interval: 15m
timeout: 5m
releaseName: secret-replicator
valuesFrom:
- kind: ConfigMap
name: secret-replicator-helm-chart-value-overrides
valuesKey: values.yaml # This is the default, but best to be explicit for clarity
```
--8<-- "kubernetes-why-not-config-in-helmrelease.md"
## Serving
Once you've committed your YAML files into your repo, you should soon see some pods appear in the `secret-replicator` namespace!
### How do we know it worked?
Look for secrets across the whole cluster, by running `kubectl get secrets -A | grep letsencrypt-wildcard-cert`. What you should see is an identical secret in every namespace. Note that the **Certificate** only exists in the `letsencrypt-wildcard-cert` namespace, but the secret it **generates** is what gets replicated to every other namespace.
### Troubleshooting
If your certificate is not created **aren't** created as you expect, then the best approach is to check the secret-replicator logs, by running `kubectl logs -n secret-replicator -l app.kubernetes.io/name=secret-replicator`.
--8<-- "recipe-footer.md"
[^1]: To my great New Zealandy confusion, "Kiwigrid GmbH" is a German company :shrug:

158
docs/kubernetes/ssl-certificates/wildcard-certificate.md Normal file
View File

@@ -0,0 +1,158 @@
# Wildcard Certificate
Now that we have an [Issuer](/kubernetes/ssl-certificates/letsencrypt-issuers/) and the necessary credentials, we can create a wildcard certificate, which we can then feed to our [Ingresses](/kubernetes/ingress/).
!!! summary "Ingredients"
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
* [x] [Cert-Manager](/kubernetes/ssl-certificates/cert-manager/) deployed to request/renew certificates
* [x] [LetsEncrypt ClusterIssuers](/kubernetes/ssl-certificates/letsencrypt-issuers/) created using DNS01 validation solvers
Certificates are Kubernetes secrets, and so are subject to the same limitations / RBAC controls as other secrets. Importantly, they are **namespaced**, so it's not possible to refer to a secret in one namespace, from a pod in **another** namespace. This restriction also applies to Ingress resources (*although there are workarounds*) - An Ingress can only refer to TLS secrets in its own namespace.
This behaviour can be prohibitive, because (a) we don't want to have to request/renew certificates for every single FQDN served by our cluster, and (b) we don't want more than one wildcard certificate if possible, to avoid being rate-limited at request/renewal time.
To take advantage of the various workarounds available, I find it best to put the certificates into a dedicated namespace, which I name.. `letsencrypt-wildcard-cert`.
!!! question "Why not the cert-manager namespace?"
Because cert-manager is a _controller_, whose job it is to act on resources. I should be able to remove cert-manager entirely (even its namespace) from my cluster, and re-add it, without impacting the resources it acts upon. If the certificates lived in the `cert-manager` namespace, then I wouldn't be able to remove the namespace without also destroying the certificates.
Furthermore, we can't deploy ClusterIssuers (a CRD) in the same kustomization which deploys the helmrelease which creates those CRDs in the first place. Flux won't be able to apply the ClusterIssuers until the CRD is created, and so will fail to reconcile.
## Preparation
### Namespace
We need a namespace to deploy our certificates and associated secrets into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/namespaces/namespace-letsencrypt-wildcard-cert.yaml`:
??? example "Example Namespace (click to expand)"
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: letsencrypt-wildcard-cert
```
### Kustomization
Now we need a kustomization to tell Flux to install any YAMLs it finds in `/letsencrypt-wildcard-cert`. I create this example Kustomization in my flux repo at `bootstrap/kustomizations/kustomization-letsencrypt-wildcard-cert.yaml`.
!!! tip
Importantly, note that we define a **dependsOn**, to tell Flux not to try to reconcile this kustomization before the cert-manager and sealedsecrets kustomizations are reconciled. Cert-manager creates the CRDs used to define certificates, so prior to Cert Manager being installed, the cluster won't know what to do with the ClusterIssuers/Certificate resources.
??? example "Example Kustomization (click to expand)"
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: letsencrypt-wildcard-cert
namespace: flux-system
spec:
interval: 15m
path: ./letsencrypt-wildcard-cert
dependsOn:
- name: "cert-manager"
- name: "sealed-secrets"
prune: true # remove any elements later removed from the above path
timeout: 2m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
validation: server
```
### DNS01 Validation Secret
The simplest way to validate ownership of a domain to LetsEncrypt is to use DNS-01 validation. In this mode, we "prove" our ownership of a domain name by creating a special TXT record, which LetsEncrypt will check and confirm for validity, before issuing us any certificates for that domain name.
The [ClusterIssuers we created earlier](/kubernetes/ssl-certificates/letsencrypt-issuers/) included a field `solvers.dns01.cloudflare.apiTokenSecretRef.name`. This value points to a secret (*in the same namespace as the certificate[^1]*) containing credentials necessary to create DNS records automatically. (*again, my examples are for cloudflare, but the [other supported providers](https://cert-manager.io/docs/configuration/acme/dns01/) will have similar secret requirements*)
Thanks to [Sealed Secrets](/kubernetes/sealed-secrets/), we have a safe way of committing secrets into our repository, so to create necessary secret, you'd run something like this:
```bash
kubectl create secret generic cloudflare-api-token-secret \
--namespace letsencrypt-wildcard-cert \
--dry-run=client \
--from-literal=api-token=gobbledegook -o json \
| kubeseal --cert <path to public cert> \
| kubectl create -f - \
> <path to repo>/letsencrypt-wildcard-cert/sealedsecret-cloudflare-api-token-secret.yaml
```
### Staging Certificate
Finally, we create our certificates! Here's an example certificate resource which uses the letsencrypt-staging issuer (*to avoid being rate-limited while learning!*). I save this in my flux repo as `/letsencrypt-wildcard-cert/certificate-wildcard-cert-letsencrypt-staging.yaml`
???+ example "Example certificate requested from LetsEncrypt staging"
```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: letsencrypt-wildcard-cert-example.com-staging
namespace: letsencrypt-wildcard-cert
spec:
# secretName doesn't have to match the certificate name, but it may as well, for simplicity!
secretName: letsencrypt-wildcard-cert-example.com-staging
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
dnsNames:
- "example.com"
- "*.example.com"
```
## Serving
### Did it work?
After committing the above to the repo, provided the YAML syntax is correct, you should end up with a "Certificate" resource in the `letsencrypt-wildcard-cert` namespace. This doesn't mean that the certificate has been issued by LetsEncrypt yet though - describe the certificate for more details, using `kubectl describe certificate -n letsencrypt-wildcard-cert letsencrypt-wildcard-cert-staging`. The `status` field will show you whether the certificate is issued or not:
```yaml
Status:
Conditions:
Last Transition Time: 2021-11-19T01:09:32Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2022-02-17T00:09:26Z
Not Before: 2021-11-19T00:09:27Z
Renewal Time: 2022-01-18T00:09:26Z
Revision: 1
```
### Troubleshooting
If your certificate does not become `Ready` within a few minutes [^1], try watching the logs of cert-manager to identify the issue, using `kubectl logs -f -n cert-manager -l app.kubernetes.io/name=cert-manager`.
### Production Certificate
Once you know you can happily deploy a staging certificate, it's safe enough to attempt your "prod" certificate. I save this in my flux repo as `/letsencrypt-wildcard-cert/certificate-wildcard-cert-letsencrypt-prod.yaml`
???+ example "Example certificate requested from LetsEncrypt prod"
```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: letsencrypt-wildcard-cert-example.com
namespace: letsencrypt-wildcard-cert
spec:
# secretName doesn't have to match the certificate name, but it may as well, for simplicity!
secretName: letsencrypt-wildcard-cert-example.com
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
dnsNames:
- "example.com"
- "*.example.com"
```
Commit the certificate and follow the steps above to confirm that your prod certificate has been issued.
--8<-- "recipe-footer.md"
[^1]: This process can take a frustratingly long time, and watching the cert-manager logs at least gives some assurance that it's progressing!

213
docs/kubernetes/traefik.md Normal file
View File

@@ -0,0 +1,213 @@
# Traefik
This recipe utilises the [traefik helm chart](https://github.com/helm/charts/tree/master/stable/traefik) to proving LetsEncrypt-secured HTTPS access to multiple containers within your cluster.
## Ingredients
1. [Kubernetes cluster](/kubernetes/cluster/)
2. [Helm](/kubernetes/helm/) installed and initialised in your cluster
## Preparation
### Clone helm charts
Clone the helm charts, by running:
```bash
git clone https://github.com/helm/charts
```
Change to stable/traefik:
```bash
cd charts/stable/traefik
```
### Edit values.yaml
The beauty of the helm approach is that all the complexity of the Kubernetes elements' YAML files are hidden from you (created using templates), and all your changes go into values.yaml.
These are my values, you'll need to adjust for your own situation:
```yaml
imageTag: alpine
serviceType: NodePort
# yes, we're not listening on 80 or 443 because we don't want to pay for a loadbalancer IP to do this. I use poor-mans-k8s-lb instead
service:
nodePorts:
http: 30080
https: 30443
cpuRequest: 1m
memoryRequest: 100Mi
cpuLimit: 1000m
memoryLimit: 500Mi
ssl:
enabled: true
enforced: true
debug:
enabled: false
rbac:
enabled: true
dashboard:
enabled: true
domain: traefik.funkypenguin.co.nz
kubernetes:
# set these to all the namespaces you intend to use. I standardize on one-per-stack. You can always add more later
namespaces:
- kube-system
- unifi
- kanboard
- nextcloud
- huginn
- miniflux
accessLogs:
enabled: true
acme:
persistence:
enabled: true
# Add the necessary annotation to backup ACME store with k8s-snapshots
annotations: { "backup.kubernetes.io/deltas: P1D P7D" }
staging: false
enabled: true
logging: true
email: "<my letsencrypt email>"
challengeType: "dns-01"
dnsProvider:
name: cloudflare
cloudflare:
CLOUDFLARE_EMAIL: "<my cloudlare email"
CLOUDFLARE_API_KEY: "<my cloudflare API key>"
domains:
enabled: true
domainsList:
- main: "*.funkypenguin.co.nz" # name of the wildcard domain name for the certificate
- sans:
- "funkypenguin.co.nz"
metrics:
prometheus:
enabled: true
```
!!! note
The helm chart doesn't enable the Traefik dashboard by default. I intend to add an oauth_proxy pod to secure this, in a future recipe update.
### Prepare phone-home pod
[Remember](/kubernetes/loadbalancer/) how our load balancer design ties a phone-home container to another container using a pod, so that the phone-home container can tell our external load balancer (_using a webhook_) where to send our traffic?
Since we deployed Traefik using helm, we need to take a slightly different approach, so we'll create a pod with an affinity which ensures it runs on the same host which runs the Traefik container (_more precisely, containers with the label app=traefik_).
Create phone-home.yaml as per the following example:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: phonehome-traefik
spec:
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- traefik
topologyKey: failure-domain.beta.kubernetes.io/zone
containers:
- image: funkypenguin/poor-mans-k8s-lb
imagePullPolicy: Always
name: phonehome-traefik
env:
- name: REPEAT_INTERVAL
value: "600"
- name: FRONTEND_PORT
value: "443"
- name: BACKEND_PORT
value: "30443"
- name: NAME
value: "traefik"
- name: WEBHOOK
value: "https://<your loadbalancer hostname>:9000/hooks/update-haproxy"
- name: WEBHOOK_TOKEN
valueFrom:
secretKeyRef:
name: traefik-credentials
key: webhook_token.secret
```
Create your webhook token secret by running:
```bash
echo -n "imtoosecretformyshorts" > webhook_token.secret
kubectl create secret generic traefik-credentials --from-file=webhook_token.secret
```
!!! warning
Yes, the "-n" in the echo statement is needed. [Read here for why](https://www.funkypenguin.co.nz/beware-the-hidden-newlines-in-kubernetes-secrets/).
## Serving
### Install the chart
To install the chart, simply run ```helm install stable/traefik --name traefik --namespace kube-system```
That's it, traefik is running.
You can confirm this by running ```kubectl get pods```, and even watch the traefik logs, by running ```kubectl logs -f traefik<tab-to-autocomplete>```
### Deploy the phone-home pod
We still can't access traefik yet, since it's listening on port 30443 on node it happens to be running on. We'll launch our phone-home pod, to tell our [load balancer](/kubernetes/loadbalancer/) where to send incoming traffic on port 443.
Optionally, on your loadbalancer VM, run ```journalctl -u webhook -f``` to watch for the container calling the webhook.
Run ```kubectl create -f phone-home.yaml``` to create the pod.
Run ```kubectl get pods -o wide``` to confirm that both the phone-home pod and the traefik pod are on the same node:
```bash
# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
phonehome-traefik 1/1 Running 0 20h 10.56.2.55 gke-penguins-are-sexy-8b85ef4d-2c9g
traefik-69db67f64c-5666c 1/1 Running 0 10d 10.56.2.30 gkepenguins-are-sexy-8b85ef4d-2c9g
```
Now browse to `https://<your load balancer`, and you should get a valid SSL cert, along with a 404 error (_you haven't deployed any other recipes yet_)
### Making changes
If you change a value in values.yaml, and want to update the traefik pod, run:
```bash
helm upgrade --values values.yml traefik stable/traefik --recreate-pods
```
## Review
We're doneburgers! 🍔 We now have all the pieces to safely deploy recipes into our Kubernetes cluster, knowing:
1. Our HTTPS traffic will be secured with LetsEncrypt (thanks Traefik!)
2. Our non-HTTPS ports (like UniFi adoption) will be load-balanced using an free-to-scale [external load balancer](/kubernetes/loadbalancer/)
3. Our persistent data will be [automatically backed up](/kubernetes/snapshots/)
Here's a recap:
* [Start](/kubernetes/) - Why Kubernetes?
* [Design](/kubernetes/design/) - How does it fit together?
* [Cluster](/kubernetes/cluster/) - Setup a basic cluster
* [Load Balancer](/kubernetes/loadbalancer/) Setup inbound access
* [Snapshots](/kubernetes/snapshots/) - Automatically backup your persistent data
* [Helm](/kubernetes/helm/) - Uber-recipes from fellow geeks
* Traefik (this page) - Traefik Ingress via Helm
## Where to next?
I'll be adding more Kubernetes versions of existing recipes soon. Check out the [MQTT](/recipes/mqtt/) recipe for a start!
[^1]: It's kinda lame to be able to bring up Traefik but not to use it. I'll be adding the oauth_proxy element shortly, which will make this last step a little more conclusive and exciting!
--8<-- "recipe-footer.md"