Hello, Kubernetes

manuscript/CHANGELOG.md

@@ -13,13 +13,14 @@
 
 ## Recently added recipes
 
+* Added the beginning of the **Kubernetes** design, including a getting started on using [Digital Ocean,](/kubernetes/digitalocean/) and a WIP recipe for an [MQTT](/recipes/mqtt/) broker (_21 Jan 2019_)
 * [ElkarBackup](/recipes/elkarbackup/), a beautiful GUI-based backup solution built on rsync/rsnapshot (_1 Jan 2019_)
 * Added [Collabora Online](/recipes/collabora-online), an rich document editor within [NextCloud](/recipes/nextcloud/) (_think "headless LibreOffice"_)
 * Added [phpIPAM](/recipes/phpipam), an IP address management tool (_18 Dec 2018_)
 * Added [KeyCloak](/recipes/keycloak), an open source identity and access management solution which backends neatly into [OpenLDAP](/recipes/openldap/) (_among other providers_), providing true SSO (_13 Dec 2018_)
 * Added [OpenLDAP](/recipes/openldap/), a 20-year old project which [refuses to die](https://www.youtube.com/watch?v=cnQEo4bazIo), underpinning many of today's authentication platforms, and providing a single authentication backend for multiple recipes (_9 Dec 2018_)
 * Added [Wetty](/recipes/wetty/), a remote terminal client in your web browser (_22 Nov 2018_)
-* Added [PrivateBin](/recipes/privatebin/), a self-hosted pastebin alternative (_5 Nov 2018_)
+
 
 
 ## Recent improvements

manuscript/recipes/mqtt.md

@@ -0,0 +1,213 @@
+hero: Kubernetes. The hero we deserve.
+
+!!! danger "This recipe is a work in progress"
+    This recipe is **incomplete**, and is featured to align the [patrons](https://www.patreon.com/funkypenguin)'s "premix" repository with the cookbook.  "_premix_" is a private git repository available to [all Patreon patrons](https://www.patreon.com/funkypenguin), which includes all the necessary .yml files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```kubectl create -f *.yml``` 👍
+
+    So... There may be errors and inaccuracies. Jump into [Discord](http://chat.funkypenguin.co.nz) if you're encountering issues 😁
+
+# MQTT broker
+
+I use Elias Kotlyar's [excellent custom firmware](https://github.com/EliasKotlyar/Xiaomi-Dafang-Hacks) for Xiaomi DaFang/XiaoFang cameras, enabling RTSP, MQTT, motion tracking, and other features, integrating directly with [Home Assistant](/recipes/homeassistant/).
+
+There's currently a [mysterious bug](https://github.com/EliasKotlyar/Xiaomi-Dafang-Hacks/issues/638) though, which prevents TCP communication between Home Assistant and the camera, when MQTT services are enabled on the camera and the mqtt broker runs on the same Raspberry Pi as Home Assistant, using [Hass.io](https://www.home-assistant.io/hassio/).
+
+A workaround to this bug is to run an MQTT broker **external** to the raspberry pi, which makes the whole problem GoAway(tm). Since an MQTT broker is a single, self-contained container, I've written this recipe as an introduction to our Kubernetes cluster design.
+
+![MQTT Screenshot](../images/mqtt.png)
+
+[MQTT](https://mqtt.org/faq) stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks. The design principles are to minimise network bandwidth and device resource requirements whilst also attempting to ensure reliability and some degree of assurance of delivery. These principles also turn out to make the protocol ideal of the emerging “machine-to-machine” (M2M) or “Internet of Things” world of connected devices, and for mobile applications where bandwidth and battery power are at a premium.
+
+## Ingredients
+
+1. A [Kubernetes cluster](/kubernetes/digital-ocean/)
+
+## Preparation
+
+### Create data locations
+
+Although we could simply bind-mount local volumes to a local Kubuernetes cluster, since we're targetting a cloud-based Kubernetes deployment, we only need a local path to store the YAML files which define the various aspects of our Kubernetes deployment.
+
+```
+mkdir /var/data/config/mqtt
+```
+
+### Create namespace
+
+We use Kubernetes namespaces for service discovery and isolation between our stacks, so create a namespace for the mqtt stack by creating the following .yaml:
+
+```
+cat <<EOF > /var/data/mqtt/namespace.yml
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: mqtt
+EOF
+kubectl create -f /var/data/mqtt/namespace.yaml
+```
+
+### Create persistent volume claim
+
+Persistent volume claims are a streamlined way to create a persistent volume and assign it to a container in a pod. Create a claim for the certbot data:
+
+```
+cat <<EOF > /var/data/mqtt/persistent-volumeclaim.yml
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+  name: mqtt-volumeclaim
+  namespace: mqtt
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+EOF
+kubectl create -f /var/data/mqtt/mqtt-volumeclaim.yaml
+```
+
+### Create nodeport service
+
+I like to expose my services using nodeport (_limited to ports 30000-32767_), and then use an external haproxy load balancer to make these available externally. (_This avoids having to pay per-port changes for a loadbalancer from the cloud provider_)
+
+```
+cat <<EOF > /var/data/mqtt/service-nodeport.yml
+kind: Service
+apiVersion: v1
+metadata:
+  name: mqtt-nodeport
+  namespace: mqtt
+spec:
+  selector:
+    app: mqtt
+  type: NodePort
+  ports:
+  - name: mqtts
+    port: 8883
+    protocol: TCP
+    nodePort : 30883
+EOF
+kubectl create -f /var/data/mqtt/service-nodeport.yml
+```
+### Create secrets
+
+It's not always desirable to have sensitive data stored in your .yml files. Maybe you want to check your config into a git repository, or share it. Using Kubernetes Secrets means that you can create "secrets", and use these in your deployments by name, without exposing their contents.
+
+```
+echo -n "myapikeyissosecret" > cloudflare-key.secret
+echo -n "myemailaddress" > cloudflare-email.secret
+echo -n "myemailaddress" > letsencrypt-email.secret
+
+kubectl create secret -n mqtt generic mqtt-credentials \
+   --from-file=cloudflare-key.secret \
+   --from-file=cloudflare-email.secret \
+   --from-file=letsencrypt-email.secret
+```
+
+!!! tip "Why use ```echo -n```?"
+    Because. See [my blog post here](https://www.funkypenguin.co.nz/beware-the-hidden-newlines-in-kubernetes-secrets/) for the pain of hunting invisible newlines, that's why!
+
+## Serving
+
+### Create deployment
+
+Now that we have a volume, a service, and a namespace, we can create a deployment for the mqtt pod. Note below the use of volume mounts, environment variables, as well as the secrets.
+
+!!! tip
+        I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary .yml files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```kubectl create -f *.yml``` 👍
+
+```
+cat <<EOF > /var/data/mqtt/mqtt.yml
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  namespace: mqtt
+  name: mqtt
+  labels:
+    app: mqtt
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: mqtt
+  template:
+    metadata:
+      labels:
+        app: mqtt
+    spec:
+      containers:
+        - image: funkypenguin/mqtt-certbot-dns
+          imagePullPolicy: Always
+# only uncomment these to get the container to run so that we can transfer files into the PV
+#          command: [ "/bin/sleep" ]
+#          args: [ "1h" ]
+          env:
+          - name: DOMAIN
+            value: "*.funkypenguin.co.nz"
+          - name: EMAIL
+            valueFrom:
+              secretKeyRef:
+                name: mqtt-credentials
+                key: letsencrypt-email.secret
+          - name: CLOUDFLARE_EMAIL
+            valueFrom:
+              secretKeyRef:
+                name: mqtt-credentials
+                key: cloudflare-email.secret
+          - name: CLOUDFLARE_KEY
+            valueFrom:
+              secretKeyRef:
+                name: mqtt-credentials
+                key: cloudflare-key.secret
+# uncomment this to test LetsEncrypt validations
+#          - name: TESTCERT
+#            value: "true"
+          name: mqtt
+          resources:
+            requests:
+              memory: "50Mi"
+              cpu: "0.1"
+          volumeMounts:
+            # We need the LE certs to persist across reboots to avoid getting rate-limited (bad, bad)
+            - name: mqtt-volumeclaim
+              mountPath: /etc/letsencrypt
+            # A configmap for the mosquitto.conf file
+            - name: mosquitto-conf
+              mountPath: /mosquitto/conf/mosquitto.conf
+              subPath: mosquitto.conf
+            # A configmap for the mosquitto passwd file
+            - name: mosquitto-passwd
+              mountPath: /mosquitto/conf/passwd
+              subPath: passwd
+      volumes:
+        - name: mqtt-volumeclaim
+          persistentVolumeClaim:
+            claimName: mqtt-volumeclaim
+        - name: mosquitto-conf
+          configMap:
+            name: mosquitto.conf
+        - name: mosquitto-passwd
+          configMap:
+            name: passwd
+EOF
+kubectl create -f /var/data/mqtt/mqtt.yml
+```
+
+Check that your deployment is running, with ```kubectl get pods -n mqtt```. After a minute or so, you should see a "Running" pod, as illustrated below:
+
+```
+[davidy:~/Documents/Personal/Projects/mqtt-k8s] 130 % kubectl get pods -n mqtt
+NAME                          READY     STATUS    RESTARTS   AGE
+mqtt-65f4d96945-bjj44         1/1       Running   0          5m
+[davidy:~/Documents/Personal/Projects/mqtt-k8s] %
+```
+
+To actually **use** your new MQTT broker, you'll need to connect to any one of your nodes (```kubectl get nodes -o wide```) on port 30883 (_the nodeport service we created earlier_). More info on that, and a loadbalancer design, to follow shortly :)
+
+## Chef's Notes
+
+### Tip your waiter (donate) 👏
+
+Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
+
+### Your comments? 💬

manuscript/recipes/template-k8s.md

@@ -0,0 +1,133 @@
+hero: Not all heroes wear capes
+
+!!! danger "This recipe is a work in progress"
+    This recipe is **incomplete**, and is featured to align the [patrons](https://www.patreon.com/funkypenguin)'s "premix" repository with the cookbook.  "_premix_" is a private git repository available to [all Patreon patrons](https://www.patreon.com/funkypenguin), which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
+
+    So... There may be errors and inaccuracies. Jump into [Discord](http://chat.funkypenguin.co.nz) if you're encountering issues 😁
+
+# NAME
+
+Intro
+
+![NAME Screenshot](../images/name.jpg)
+
+Details
+
+## Ingredients
+
+1. [Kubernetes cluster](/kubernetes/digital-ocean/)
+
+## Preparation
+
+### Create data locations
+
+```
+mkdir /var/data/config/mqtt
+```
+
+### Create namespace
+
+We use Kubernetes namespaces for service discovery and isolation between our stacks, so create a namespace for the mqtt stack by creating the following .yaml:
+
+```
+cat <<EOF > /var/data/mqtt/namespace.yaml
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: mqtt
+EOF
+kubectl create -f /var/data/mqtt/namespace.yaml
+```
+
+### Prepare environment
+
+Create wekan.env, and populate with the following variables
+```
+OAUTH2_PROXY_CLIENT_ID=
+OAUTH2_PROXY_CLIENT_SECRET=
+OAUTH2_PROXY_COOKIE_SECRET=
+MONGO_URL=mongodb://wekandb:27017/wekan
+ROOT_URL=https://wekan.example.com
+MAIL_URL=smtp://wekan@wekan.example.com:password@mail.example.com:587/
+MAIL_FROM="Wekan <wekan@wekan.example.com>"
+```
+
+### Setup Docker Swarm
+
+Create a docker swarm config file in docker-compose syntax (v3), something like this:
+
+!!! tip
+        I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
+
+
+```
+version: '3'
+
+services:
+
+  wekandb:
+    image: mongo:3.2.15
+    command: mongod --smallfiles --oplogSize 128
+    networks:
+      - internal
+    volumes:
+      - /var/data/wekan/wekan-db:/data/db
+      - /var/data/wekan/wekan-db-dump:/dump
+
+  proxy:
+    image: a5huynh/oauth2_proxy
+    env_file: /var/data/wekan/wekan.env
+    networks:
+      - traefik_public
+      - internal
+    deploy:
+      labels:
+        - traefik_public.frontend.rule=Host:wekan.example.com
+        - traefik_public.docker.network=traefik_public
+        - traefik_public.port=4180
+    command: |
+      -cookie-secure=false
+      -upstream=http://wekan:80
+      -redirect-url=https://wekan.example.com
+      -http-address=http://0.0.0.0:4180
+      -email-domain=example.com
+      -provider=github
+
+  wekan:
+    image: wekanteam/wekan:latest
+    networks:
+      - internal
+    env_file: /var/data/wekan/wekan.env
+
+networks:
+  traefik_public:
+    external: true
+  internal:
+    driver: overlay
+    ipam:
+      config:
+        - subnet: 172.16.3.0/24
+```
+
+!!! note
+    Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
+
+
+
+## Serving
+
+### Launch Wekan stack
+
+Launch the Wekan stack by running ```docker stack deploy wekan -c <path -to-docker-compose.yml>```
+
+Log into your new instance at https://**YOUR-FQDN**, with user "root" and the password you specified in gitlab.env.
+
+## Chef's Notes
+
+1. If you wanted to expose the Wekan UI directly, you could remove the oauth2_proxy from the design, and move the traefik_public-related labels directly to the wekan container. You'd also need to add the traefik_public network to the wekan container.
+
+### Tip your waiter (donate) 👏
+
+Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
+
+### Your comments? 💬

mkdocs.yml

@@ -112,6 +112,7 @@ pages:
     - Work-in-Progress:
 #       - MatterMost: recipes/mattermost.md
        - IPFS Cluster: recipes/ipfs-cluster.md
+       - MQTT: recipes/mqtt.md       
 #       - HackMD: recipes/hackmd.md
 #       - Mastodon: recipes/mastodon.md
 #       - Mayan EDMS: recipes/mayan-edms.md