public/geek-cookbook
1
0
Fork 0
mirror of https://github.com/funkypenguin/geek-cookbook/ synced 2025-12-12 17:26:19 +00:00
Code Issues Releases Wiki Activity

Add blog section, optimize images

Browse Source 
Signed-off-by: David Young <davidy@funkypenguin.co.nz>
This commit is contained in:
David Young
2023-02-07 11:04:58 +13:00
parent 9eec6cd985
commit d2153a7780
194 changed files with 727 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/blog/.authors.yml Normal file
View File

@@ -0,0 +1,4 @@
funkypenguin:
name: David Young
description: Geek Chef
avatar: https://github.com/funkypenguin.png

7
docs/blog/index.md Normal file
View File

@@ -0,0 +1,7 @@
# Funky Penguin's Geek Cookblog
Welcome to the Funky Blog!
After years of trying to use alternate platforms (*❤️ ya, [Ghost](../recipes/ghost.md) ! 👻*), I've given up and decided to move my technical blog entries here instead - I prefer the way [mkdocs-material](https://squidfunk.github.io/mkdocs-material/) lets me format documentation for technical consumption!
--8<-- "common-links.md"

5
docs/blog/posts/.meta.yml Normal file
View File

@@ -0,0 +1,5 @@
authors:
- funkypenguin
links:
- 🛟 Get help: support.md
- 🤝 Work with me: https://www.funkypenguin.co.nz/work-with-me

209
docs/blog/posts/2023/lessons-learned-velero-hardened-kubernetes-cluster.md Normal file
View File

@@ -0,0 +1,209 @@
---
date: 2023-02-03
categories:
- note
tags:
- kubernetes
- velero
---
# Using Velero in hardened Kubernetes with Istio
I'm approaching the end of the journey of applying Velero to a client's "hardened" Kubernetes cluster, and along the way I stubbed my toes on several issues, which I intend to lay out below...
<!-- more -->
## What is a hardened Kubernetes cluster?
In this particular case, the following apply:
* [X] Selective workloads/namespaces are protected by Istio, using strict mTlS
* [X] Kyverno is employed with policies enforcing mode using the "[restricted](https://github.com/kyverno/kyverno/tree/main/charts/kyverno-policies/templates/restricted)" baseline, with further policies applied (such as deny-exec, preventing arbitrary execing into pods).
* [X] Kubernetes best-practices are applied to all workloads, audited using Fairwinds Polaris, which includes running pods as non-root, with read-only root filesystems, whever possible.
## How does Velero work?
Velero runs within a cluster, listening for custom resources defining backups, restores, destinations, schedules, etc. Based on a combination of all of these, Velero scrapes the kubernetes API, works out what to backup, and does so, according to a schedule.
## Velero backup hooks
While Velero can backup persistent volumes using either snapshots or restic/kopia, if you're backing up in-use data, it's usually necessary to take some actions before the backup, to ensure the data is in a safe, restorable state. This is achieved using pre/post hooks, as illustrated below, a fairly generic config for postgresql instances based on Bitnami's postgresql chart [^1]:
```yaml
extraVolumes: # (1)!
- name: backup
emptyDir: {}
extraVolumeMounts: # (2)!
- name: backup
mountPath: /scratch
podAnnotations:
backup.velero.io/backup-volumes: backup
pre.hook.backup.velero.io/command: '["/bin/bash", "-c", "PGPASSWORD=$POSTGRES_PASSWORD pg_dump -U $POSTGRES_USER -d $POSTGRES_DB -F c -f /scratch/backup.psql"]'
pre.hook.backup.velero.io/timeout: 5m
pre.hook.restore.velero.io/timeout: 5m
post.hook.restore.velero.io/command: '["/bin/bash", "-c", "[ -f \"/scratch/backup.psql\" ] && \
sleep 1m && \
PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U $POSTGRES_USER -d $POSTGRES_USER --clean \
< /scratch/backup.psql && rm -f /scratch/backup.psql;"]' # !(3)
```
1. This defines an additional ephemeral volume to attach to the pod
2. This attaches the above volume at `/scratch`
3. It's necessary to sleep for "a period" before attempting the restore, so that postegresql has time to start up and be ready to interact with the `pg_restore` command.
[^1]: Details at https://github.com/bitnami/charts/tree/main/bitnami/postgresql
During the process of setting up the preHooks for various iterations of a postgresql instance, I discovered that Velero will not necessary check that carefully re whether the hooks returned successfully or not. It's best to completely simulate a restore/backup of your pods by execing into the pod, and running each hook command manually, ensuring that you get the expected result.
## Velero vs securityContexts
We apply best-practice securityContexts to our pods, including enforcing of readOnly root filesystems on the pod, the disabling of all capabilities, etc. Here's a sensible example:
```yaml
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
readOnlyRootFilesystem: true
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
```
However, on the node-restore agent, we need to make a few changes to the helm chart above:
```yaml
# Extra volumes for the node-agent daemonset. Optional.
extraVolumes:
- name: tmp
emptyDir:
sizeLimit: 1Gi
# Extra volumeMounts for the node-agent daemonset. Optional.
extraVolumeMounts:
- name: tmp
mountPath: /tmp # (1)
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
add: ["CHOWN"] # (2)!
readOnlyRootFilesystem: true
seccompProfile:
type: RuntimeDefault
```
1. node-agent tries to write a credential file to `/tmp`. We create this emptydir so that we don't need to enabel a RW filesystem for the entire container
2. Necessary for restic restores, since after a restic restore, a CHOWN will be performed
## Velero vs Kyverno policies
We use a Kyverno policy as illustrated below this, to permit users from execing into containers. It was necessary to make an exeception to permit Velero to exec into pods:
```yaml
{% raw %}
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: deny-exec
annotations:
policies.kyverno.io/title: Block Pod Exec by Namespace Label
policies.kyverno.io/category: Sample
policies.kyverno.io/minversion: 1.4.2
policies.kyverno.io/subject: Pod
policies.kyverno.io/description: >-
The `exec` command may be used to gain shell access, or run other commands, in a Pod's container. While this can
be useful for troubleshooting purposes, it could represent an attack vector and is discouraged.
This policy blocks Pod exec commands to Pods unless their namespace is labeled with "kyverno/permit-exec:true"
spec:
validationFailureAction: enforce
background: false
rules:
- name: deny-exec
match:
resources:
kinds:
- PodExecOptions
namespaceSelector:
matchExpressions:
- key: kyverno/permit-exec
operator: NotIn
values:
- "true"
preconditions:
all:
- key: "{{ request.operation || 'BACKGROUND' }}"
operator: Equals
value: CONNECT
validate:
message: Invalid request from {{ request.userInfo.username }}. Pods may not be exec'd into unless their namespace is labeled with "kyverno/permit-exec:true"
deny:
conditions:
all:
- key: "{{ request.namespace }}"
operator: NotEquals
value: sandbox # (1)!
- key: "{{ request.userInfo.username }}" # (2)!
operator: NotEquals
value: system:serviceaccount:velero:velero-server
{% endraw %}
```
1. "sandbox" is a special, unprotected namespace for development
2. Here we permit the velero-server service account to exec into containers, which is necessary for executing the hooks!
## Velero vs Istio
If you're running Istio sidecars on your workloads, then you may find that your hooks mysteriously fail. It turns out that this happens because Velero, by default, targets the **first** container in your pod. In the case of an Istio-augmented pod, this pod is the `istio-proxy` sidecar, which is probably not where you **intended** to run your hooks!
Add 2 additional annotations to your workload, as illustrated below, to tell Velero **which** container to exec into:
```yaml
pre.hook.backup.velero.io/container: keycloak-postgresql # (1)!
post.hook.restore.velero.io/container: keycloak-postgresql
```
1. Set this to the value of your target container name.
## Velero vs Filesystem state
Docker-mailserver runs postfix, as well as many other components using an init-sort of process. This makes it hard to backup directly via a filesystem backup, since the various state files may be in use at any point. The solution here was to avoid directly backing up the data volume (*and no, you can't selectively exclude folders!*), and to implement the backup, once again, using pre/post hooks:
```yaml
{% raw %}
additionalVolumeMounts:
- name: backup
mountPath: /scratch
additionalVolumes:
- name: backup
persistentVolumeClaim:
claimName: docker-mailserver-backup
pod:
# pod.dockermailserver section refers to the configuration of the docker-mailserver pod itself. Note that teh many environment variables which define the behaviour of docker-mailserver are configured here
dockermailserver:
annotations:
sidecar.istio.io/inject: "false"
backup.velero.io/backup-volumes: backup
pre.hook.backup.velero.io/command: '["/bin/bash", "-c", "cat /dev/null > /scratch/backup.tar.gz && tar -czf /scratch/backup.tar.gz /var/mail /var/mail-state || echo done-with-harmeless-errors"]' # (1)!
pre.hook.backup.velero.io/timeout: 5m
post.hook.restore.velero.io/timeout: 5m
post.hook.restore.velero.io/command: '["/bin/bash", "-c", "[ -f \"/scratch/backup.tar.gz\" ] && tar zxfp /scratch/backup.tar.gz && rm -f /scratch/backup.tar.gz;"]'
{% endraw %}
```
1. Avoid exiting with a non-zero exit code and causing a partial failure
## PartiallyFailed can't be trusted
The Velero helm chart allows the setup of PrometheusRules, which will raise an alert in AlertManager if a backup fully (*or partially*) fails. This is what prompted our initial overhaul of our backups, since we wanted **one** alert to advise us that **all** backups were successful. Just one failure backing up one pod therefore causes the entire backup to be "PartiallyFailed", so we felt it worth investing in getting 100% success across every backup. The alternative would have been to silence the "partial" failures (*in some cases, these fail for known reasons, like empty folders*), but that would leave us blind to **new** failures, and severely compromise the entire purpose of the backups!
## Summary
In summary, Velero is a hugely useful tool, but lots of care and attention should be devoted to ensuring it **actually** works, and the state of backups should be monitored (*i.e., with `PrometheusRules` via AlertManager*).
--8<-- "blog-footer.md"

19
docs/blog/posts/changelog/added-kavita-swarm.md Normal file
View File

@@ -0,0 +1,19 @@
---
date: 2022-07-27
categories:
- CHANGELOG
tags:
- kavita
links:
- Kavita recipe: recipes/kavita.md
---
# New Recipe: Kavita - "Rocket-fueled" reader for manga/comics/ebooks, able to save reading position across devices/sessions
So you've just watched a bunch of superhero movies, and you're suddenly inspired to deep-dive into the weird world of comic books? You're already rocking AutoPirate with Mylar and NZBGet to grab content, but how to manage and enjoy your growing collection?
<!-- more -->
[Kavita Reader][kavita] is a "rocket fueled self-hosted digital library which supports a vast array of file formats". Primarily used for cosuming Manga (but quite capable of managing ebooks too), Kavita's killer feature is an OPDS server for integration with other mobile apps such as Chunky on iPad, and the ability to save your reading position across multiple devices.
--8<-- "common-links.md"

21
docs/blog/posts/changelog/added-mastodon-kubernetes.md Normal file
View File

@@ -0,0 +1,21 @@
---
date: 2022-08-08
categories:
- CHANGELOG
tags:
- mastodon
links:
- Mastodon Review: review/mastodon.md
- Mastodon Kubernetes recipe: recipes/kubernetes/mastodon.md
- Mastodon Docker Swarm recipe: recipes/mastodon.md
---
# New Recipe: Mastodon - Federated social network. Think "like twitter but also like email"
New recipe - Mastodon, like Twitter on the Fediverse. Check out the [Kubernetes recipe][k8s/mastodon]!
<!-- more -->
Mastodon is an open-source, federated (*i.e., decentralized*) social network, inspired by Twitter's "microblogging" format, and used by upwards of 6.4M early-adopters, to share links, pictures, video and text.
--8<-- "common-links.md"

21
docs/blog/posts/changelog/added-mastodon-swarm.md Normal file
View File

@@ -0,0 +1,21 @@
---
date: 2022-08-05
categories:
- CHANGELOG
tags:
- mastodon
links:
- Mastodon Review: review/mastodon.md
- Mastodon Kubernetes recipe: recipes/kubernetes/mastodon.md
- Mastodon Docker Swarm recipe: recipes/mastodon.md
---
# New Recipe: Mastodon - Federated social network. Think "like twitter but also like email"
New recipe - Mastodon, like Twitter on the Fediverse. Check out the [Docker Swarm recipe][mastodon]!
<!-- more -->
Mastodon is an open-source, federated (*i.e., decentralized*) social network, inspired by Twitter's "microblogging" format, and used by upwards of 6.4M early-adopters, to share links, pictures, video and text.
--8<-- "common-links.md"

21
docs/blog/posts/changelog/added-pixelfed.md Normal file
View File

@@ -0,0 +1,21 @@
---
date: 2022-11-10
categories:
- CHANGELOG
tags:
- pixelfed
links:
- Get Support!: support.md
---
# New Recipe: Pixelfed - Federated image sharing. Think "looks like instagram, works like Mastodon"
New recipe - Pixelfed, like Instagram on the Fediverse. Check it out [here](/docs/recipes/pixelfed.md)
<!-- more -->
[Pixelfed](https://pixelfed.org) is a free and ethical, open-source, federated (*i.e., decentralized*) social image sharing platform. As [Mastodon][mastodon] is to Twitter, so Pixelfed is to Instagram. Pixelfed uses the ActivityPub protocol, allowing users to interact with other users (*on other servers*) within the protocol, such as Mastodon, PeerTube, and Friendica, making Pixelfed a part of the Fediverse.
Much like Mastodon, Pixelfed implements chronological timelines with no implementation of content manipulation algorithms and is privacy-focused with no third party analytics or tracking. It only allows users over 16 years old to use.
--8<-- "common-links.md"

21
docs/blog/posts/changelog/review-immich.md Normal file
View File

@@ -0,0 +1,21 @@
---
date: 2022-11-10
categories:
- CHANGELOG
tags:
- immich
links:
- Get Support!: support.md
---
# New Review: Immich
New recipe - Pixelfed, like Instagram on the Fediverse. Check it out [here](/docs/recipes/pixelfed.md)
<!-- more -->
[Pixelfed](https://pixelfed.org) is a free and ethical, open-source, federated (*i.e., decentralized*) social image sharing platform. As [Mastodon][mastodon] is to Twitter, so Pixelfed is to Instagram. Pixelfed uses the ActivityPub protocol, allowing users to interact with other users (*on other servers*) within the protocol, such as Mastodon, PeerTube, and Friendica, making Pixelfed a part of the Fediverse.
Much like Mastodon, Pixelfed implements chronological timelines with no implementation of content manipulation algorithms and is privacy-focused with no third party analytics or tracking. It only allows users over 16 years old to use.
--8<-- "common-links.md"

24
docs/blog/posts/changelog/review-mastodon.md Normal file
View File

@@ -0,0 +1,24 @@
---
date: 2022-11-10
categories:
- CHANGELOG
tags:
- mastodon
links:
- Mastodon Review: review/mastodon.md
- Mastodon Kubernetes recipe: recipes/kubernetes/mastodon.md
- Mastodon Docker Swarm recipe: recipes/mastodon.md
- Get Support!: support.md
---
# New Review: Mastodon
New recipe - Pixelfed, like Instagram on the Fediverse. Check it out [here](/docs/recipes/pixelfed.md)
<!-- more -->
[Pixelfed](https://pixelfed.org) is a free and ethical, open-source, federated (*i.e., decentralized*) social image sharing platform. As [Mastodon][mastodon] is to Twitter, so Pixelfed is to Instagram. Pixelfed uses the ActivityPub protocol, allowing users to interact with other users (*on other servers*) within the protocol, such as Mastodon, PeerTube, and Friendica, making Pixelfed a part of the Fediverse.
Much like Mastodon, Pixelfed implements chronological timelines with no implementation of content manipulation algorithms and is privacy-focused with no third party analytics or tracking. It only allows users over 16 years old to use.
--8<-- "common-links.md"

21
docs/blog/posts/changelog/review-pixelfed.md Normal file
View File

@@ -0,0 +1,21 @@
---
date: 2022-11-10
categories:
- CHANGELOG
tags:
- pixelfed
links:
- Get Support!: support.md
---
# New Review: Pixelfed
New recipe - Pixelfed, like Instagram on the Fediverse. Check it out [here](/docs/recipes/pixelfed.md)
<!-- more -->
[Pixelfed](https://pixelfed.org) is a free and ethical, open-source, federated (*i.e., decentralized*) social image sharing platform. As [Mastodon][mastodon] is to Twitter, so Pixelfed is to Instagram. Pixelfed uses the ActivityPub protocol, allowing users to interact with other users (*on other servers*) within the protocol, such as Mastodon, PeerTube, and Friendica, making Pixelfed a part of the Fediverse.
Much like Mastodon, Pixelfed implements chronological timelines with no implementation of content manipulation algorithms and is privacy-focused with no third party analytics or tracking. It only allows users over 16 years old to use.
--8<-- "common-links.md"

24
docs/blog/posts/changelog/updated-metallb-v4.md Normal file
View File

@@ -0,0 +1,24 @@
---
date: 2023-01-16
categories:
- CHANGELOG
tags:
- kavita
links:
- MetalLB recipe: /kubernetes/loadbalancer/metallb.md
---
# Updated MetalLB recipe for CRDs
Prior to v0.13, [MetalLB][metallb] was configured using a ConfigMap. This has all changed now, and CRDs are required to perform configuration (which improves syntax checking, abong other things)
<!-- more -->
[MetalLB](https://metallb.universe.tf/) offers a network [load balancer](/kubernetes/loadbalancer/) implementation which workes on "bare metal" (*as opposed to a cloud provider*).
MetalLB does two jobs:
1. Provides address allocation to services out of a pool of addresses which you define
2. Announces these addresses to devices outside the cluster, either using ARP/NDP (L2) or BGP (L3)
--8<-- "common-links.md"

6
docs/blog/tags.md Normal file
View File

@@ -0,0 +1,6 @@
# Tags
Following is a list of relevant tags:
[TAGS]