mirror of
https://github.com/funkypenguin/geek-cookbook/
synced 2026-10-07 09:32:34 +00:00
Experiment with PDF generation
Signed-off-by: David Young <davidy@funkypenguin.co.nz>
This commit is contained in:
David Young
@@ -0,0 +1,11 @@
|
||||
# Persistence
|
||||
|
||||
So we've gone as far as we can with our cluster, without any form of persistence. As soon as we want to retain data, be it a database, metrics history, or objects, we need one or more ways to persist data within the cluster.
|
||||
|
||||
Here are some popular options, ranked in difficulty/complexity, in vaguely ascending order:
|
||||
|
||||
* [Local Path Provisioner](/kubernetes/persistence/local-path-provisioner/) (on k3s)
|
||||
* [TopoLVM](/kubernetes/persistence/topolvm/)
|
||||
* OpenEBS (coming soon)
|
||||
* [Rook Ceph](/kubernetes/persistence/rook-ceph/)
|
||||
* Longhorn (coming soon)
|
||||
@@ -0,0 +1,45 @@
|
||||
# Local Path Provisioner
|
||||
|
||||
[k3s](/kubernetes/cluster/k3s/) installs itself with "Local Path Provisioner", a simple controller whose job it is to create local volumes on each k3s node. If you only have one node, or you just want something simple to start learning with, then `local-path` is ideal, since it requires no further setup.
|
||||
|
||||
!!! summary "Ingredients"
|
||||
|
||||
* [x] A [Kubernetes cluster](/kubernetes/cluster/) deployed with [k3s](/kubernetes/cluster/k3s/)
|
||||
|
||||
Here's how you know you've got the StorageClass:
|
||||
|
||||
```bash
|
||||
root@shredder:~# kubectl get sc
|
||||
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
|
||||
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 60m
|
||||
root@shredder:~#
|
||||
```
|
||||
|
||||
## Preparation
|
||||
|
||||
### Basics
|
||||
|
||||
A few things you should know:
|
||||
|
||||
1. This is not **network storage**. The volume you create will forever be found to the k3s node its pod is executed on. If you later take that node down for maintenance, the pods will not be able to start on other nodes, because they won't find their volumes.
|
||||
2. The default path for the volumes is `/opt/local-path-provisioner`, although this can be changed by [editing a ConfigMap](https://github.com/rancher/local-path-provisioner/blob/master/README.md#customize-the-configmap). Make sure you have enough disk space! [^1]
|
||||
3. There's no support for resizing a volume. If you create a volume and later work out that it's too small, you'll have to destroy it and recreate it. (*More sophisticated provisioners like [rook-ceph](/kubernetes/persistence/rook-ceph/) and [topolvm](/kubernetes/persistence/topolvm/) allow for dynamic resizing of volumes*)
|
||||
|
||||
### When to use it
|
||||
|
||||
* When you don't care much about your storage. This seems backwards, but sometimes you need large amounts of storage for relatively ephemeral reasons, like batch processing, or log aggregation. You may decide the convenience of using Local Path Provisioner for quick, hard-drive-speed storage outweighs the minor hassle of loosing your metrics data if you were to have a node outage.
|
||||
* When [TopoLVM](/kubernetes/persistence/topolvm/) is not a viable option, and you'd rather use available disk space on your existing, formatted filesystems
|
||||
|
||||
### When not to use it
|
||||
|
||||
* When you have any form of redundancy requirement on your persisted data.
|
||||
* When you're not using k3s.
|
||||
* You may one day want to resize your volumes.
|
||||
|
||||
### Summary
|
||||
|
||||
In summary, Local Path Provisioner is fine if you have very specifically sized workloads and you don't care about node redundancy.
|
||||
|
||||
--8<-- "recipe-footer.md"
|
||||
|
||||
[^1]: [TopoLVM](/kubernetes/persistence/topolvm/) also creates per-node volumes which aren't "portable" between nodes, but because it relies on LVM, it is "capacity-aware", and is able to distribute storage among multiple nodes based on available capacity.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Longhorn
|
||||
|
||||
Coming soon!
|
||||
@@ -0,0 +1,3 @@
|
||||
# Open EBS
|
||||
|
||||
Coming soon!
|
||||
@@ -0,0 +1,421 @@
|
||||
---
|
||||
title: Deploy Rook Ceph's operater-managed Cluster for Persistent Storage in Kubernetes
|
||||
description: Step #2 - Having the operator available, now we deploy the ceph cluster itself
|
||||
---
|
||||
|
||||
# Persistent storage in Kubernetes with Rook Ceph / CephFS - Cluster
|
||||
|
||||
[Ceph](https://docs.ceph.com/en/quincy/) is a highly-reliable, scalable network storage platform which uses individual disks across participating nodes to provide fault-tolerant storage.
|
||||
|
||||
[Rook](https://rook.io) provides an operator for Ceph, decomposing the [10-year-old](https://en.wikipedia.org/wiki/Ceph_(software)#Release_history), at-time-arcane, platform into cloud-native components, created declaratively, whose lifecycle is managed by an operator.
|
||||
|
||||
In the [previous recipe](/kubernetes/persistence/rook-ceph/operator/), we deployed the operator, and now to actually deploy a Ceph cluster, we need to deploy a custom resource (*a "CephCluster"*), which will instruct the operator on we'd like our cluster to be deployed.
|
||||
|
||||
We'll end up with multilpe storageClasses which we can use to allocate storage to pods from either Ceph RBD (*block storage*), or CephFS (*a mounted filesystem*). In many cases, CephFS is a useful choice, because it can be mounted from more than one pod **at the same time**, which makes it suitable for apps which need to share access to the same data ([NZBGet][nzbget], [Sonarr][sonarr], and [Plex][plex], for example)
|
||||
|
||||
## Rook Ceph Cluster requirements
|
||||
|
||||
!!! summary "Ingredients"
|
||||
|
||||
Already deployed:
|
||||
|
||||
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
|
||||
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
|
||||
* [x] Rook Ceph's [Operator](/kubernetes/persistence/rook-ceph/operator/)
|
||||
|
||||
## Preparation
|
||||
|
||||
### Namespace
|
||||
|
||||
We already deployed a `rook-ceph` namespace when deploying the Rook Ceph [Operator](/kubernetes/persistence/rook-ceph/operator/), so we don't need to create this again :thumbsup: [^1]
|
||||
|
||||
### HelmRepository
|
||||
|
||||
Likewise, we'll install the `rook-ceph-cluster` helm chart from the same Rook-managed repository as we did the `rook-ceph` (operator) chart, so we don't need to create a new HelmRepository.
|
||||
|
||||
### Kustomization
|
||||
|
||||
We do, however, need a separate Kustomization for rook-ceph-cluster, telling flux to deploy any YAMLs found in the repo at `/rook-ceph-cluster`. I create this example Kustomization in my flux repo:
|
||||
|
||||
!!! question "Why a separate Kustomization if both are needed for rook-ceph?"
|
||||
While technically we **could** use the same Kustomization to deploy both `rook-ceph` and `rook-ceph-cluster`, we'd run into dependency issues. It's simpler and cleaner to deploy `rook-ceph` first, and then list it as a dependency for `rook-ceph-cluster`.
|
||||
|
||||
```yaml title="/bootstrap/kustomizations/kustomization-rook-ceph-cluster.yaml"
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: rook-ceph-cluster--rook-ceph
|
||||
namespace: flux-system
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: "rook-ceph" # (1)!
|
||||
interval: 30m
|
||||
path: ./rook-ceph-cluster
|
||||
prune: true # remove any elements later removed from the above path
|
||||
timeout: 10m # if not set, this defaults to interval duration, which is 1h
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
validation: server
|
||||
```
|
||||
|
||||
1. Note that we use the `spec.dependsOn` to ensure that this Kustomization is only applied **after** the rook-ceph operator is deployed and operational. This ensures that the necessary CRDs are in place, and avoids a dry-run error on the reconciliation.
|
||||
|
||||
--8<-- "premix-cta-kubernetes.md"
|
||||
|
||||
### ConfigMap
|
||||
|
||||
Now we're into the app-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/rook/rook/blob/master/deploy/charts/rook-ceph/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo:
|
||||
|
||||
```yaml title="/rook-ceph-cluster/configmap-rook-ceph-cluster-helm-chart-value-overrides.yaml"
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: rook-ceph-cluster-helm-chart-value-overrides
|
||||
namespace: rook-ceph-cluster
|
||||
data:
|
||||
values.yaml: |- # (1)!
|
||||
# <upstream values go here>
|
||||
```
|
||||
|
||||
1. Paste in the contents of the upstream `values.yaml` here, intended 4 spaces, and then change the values you need as illustrated below.
|
||||
|
||||
Here are some suggested changes to the defaults which you should consider:
|
||||
|
||||
```yaml
|
||||
toolbox:
|
||||
enabled: true # (1)!
|
||||
monitoring:
|
||||
# enabling will also create RBAC rules to allow Operator to create ServiceMonitors
|
||||
enabled: true # (2)!
|
||||
# whether to create the prometheus rules
|
||||
createPrometheusRules: true # (3)!
|
||||
pspEnable: false # (4)!
|
||||
ingress:
|
||||
dashboard: {} # (5)!
|
||||
```
|
||||
|
||||
1. It's useful to have a "toolbox" pod to shell into to run ceph CLI commands
|
||||
2. Consider enabling if you already have Prometheus installed
|
||||
3. Consider enabling if you already have Prometheus installed
|
||||
4. PSPs are deprecated, and will eventually be removed in Kubernetes 1.25, at which point this will cause breakage.
|
||||
5. Customize the ingress configuration for your dashboard
|
||||
|
||||
Further to the above, decide which disks you want to dedicate to Ceph, and add to the `cephClusterSpec` section.
|
||||
|
||||
The default configuration (below) will cause the operator to use any un-formatted disks found on any of your nodes. If this is what you **want** to happen, then you don't need to change anything.
|
||||
|
||||
```yaml
|
||||
cephClusterSpec:
|
||||
storage: # cluster level storage configuration and selection
|
||||
useAllNodes: true
|
||||
useAllDevices: true
|
||||
```
|
||||
|
||||
If you'd rather be a little more selective / declarative about which disks are used in a homogenous cluster, you could consider using `deviceFilter`, like this:
|
||||
|
||||
```yaml
|
||||
cephClusterSpec:
|
||||
storage: # cluster level storage configuration and selection
|
||||
useAllNodes: true
|
||||
useAllDevices: false
|
||||
deviceFilter: sdc #(1)!
|
||||
```
|
||||
|
||||
1. A regex to use to filter target devices found on each node
|
||||
|
||||
If your cluster nodes are a little more snowflakey :snowflake:, here's a complex example:
|
||||
|
||||
```yaml
|
||||
cephClusterSpec:
|
||||
storage: # cluster level storage configuration and selection
|
||||
useAllNodes: false
|
||||
useAllDevices: false
|
||||
nodes:
|
||||
- name: "teeny-tiny-node"
|
||||
deviceFilter: "." #(1)!
|
||||
- name: "bigass-node"
|
||||
devices:
|
||||
- name: "/dev/disk/by-path/pci-0000:01:00.0-sas-exp0x500404201f43b83f-phy11-lun-0" #(2)!
|
||||
config:
|
||||
metadataDevice: "/dev/osd-metadata/11"
|
||||
- name: "nvme0n1" #(3)!
|
||||
- name: "nvme1n1"
|
||||
```
|
||||
|
||||
1. Match any devices found on this node
|
||||
2. Match a very-specific device path, and pair this device with a faster device for OSD metadata
|
||||
3. Match devices with simple regex string matches
|
||||
|
||||
### HelmRelease
|
||||
|
||||
Finally, having set the scene above, we define the HelmRelease which will actually deploy the rook-ceph operator into the cluster. I save this in my flux repo:
|
||||
|
||||
```yaml title="/rook-ceph-cluster/helmrelease-rook-ceph-cluster.yaml"
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2beta1
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
name: rook-ceph-cluster
|
||||
namespace: rook-ceph
|
||||
spec:
|
||||
chart:
|
||||
spec:
|
||||
chart: rook-ceph-cluster
|
||||
version: 1.9.x
|
||||
sourceRef:
|
||||
kind: HelmRepository
|
||||
name: rook-release
|
||||
namespace: flux-system
|
||||
interval: 30m
|
||||
timeout: 10m
|
||||
install:
|
||||
remediation:
|
||||
retries: 3
|
||||
upgrade:
|
||||
remediation:
|
||||
retries: -1 # keep trying to remediate
|
||||
crds: CreateReplace # Upgrade CRDs on package update
|
||||
releaseName: rook-ceph-cluster
|
||||
valuesFrom:
|
||||
- kind: ConfigMap
|
||||
name: rook-ceph-cluster-helm-chart-value-overrides
|
||||
valuesKey: values.yaml # (1)!
|
||||
```
|
||||
|
||||
1. This is the default, but best to be explicit for clarity
|
||||
|
||||
## Install Rook Ceph Operator!
|
||||
|
||||
Commit the changes to your flux repository, and either wait for the reconciliation interval, or force a reconcilliation using `flux reconcile source git flux-system`. You should see the kustomization appear...
|
||||
|
||||
```bash
|
||||
~ ❯ flux get kustomizations rook-ceph-cluster
|
||||
NAME READY MESSAGE REVISION SUSPENDED
|
||||
rook-ceph-cluster True Applied revision: main/345ee5e main/345ee5e False
|
||||
~ ❯
|
||||
```
|
||||
|
||||
The helmrelease should be reconciled...
|
||||
|
||||
```bash
|
||||
~ ❯ flux get helmreleases -n rook-ceph rook-ceph
|
||||
NAME READY MESSAGE REVISION SUSPENDED
|
||||
rook-ceph-cluster True Release reconciliation succeeded v1.9.9 False
|
||||
~ ❯
|
||||
```
|
||||
|
||||
And you should have happy rook-ceph operator pods:
|
||||
|
||||
```bash
|
||||
~ ❯ k get pods -n rook-ceph -l app=rook-ceph-operator
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
rook-ceph-operator-7c94b7446d-nwsss 1/1 Running 0 5m14s
|
||||
~ ❯
|
||||
```
|
||||
|
||||
To watch the operator do its magic, you can tail its logs, using:
|
||||
|
||||
```bash
|
||||
k logs -n rook-ceph -f -l app=rook-ceph-operator
|
||||
```
|
||||
|
||||
You can **get** or **describe** the status of your cephcluster:
|
||||
|
||||
```bash
|
||||
~ ❯ k get cephclusters.ceph.rook.io -n rook-ceph
|
||||
NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL
|
||||
rook-ceph /var/lib/rook 3 6d22h Ready Cluster created successfully HEALTH_OK
|
||||
~ ❯
|
||||
```
|
||||
|
||||
### How do I know it's working?
|
||||
|
||||
So we have a ceph cluster now, but how do we know we can actually provision volumes?
|
||||
|
||||
#### Create PVCs
|
||||
|
||||
Create two ceph-block PVCs (*persistent volume claim*), by running:
|
||||
|
||||
```bash
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: ceph-block-pvc-1
|
||||
labels:
|
||||
test: ceph
|
||||
funkypenguin-is: a-smartass
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: ceph-block
|
||||
resources:
|
||||
requests:
|
||||
storage: 128Mi
|
||||
EOF
|
||||
```
|
||||
|
||||
And:
|
||||
|
||||
```bash
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: ceph-block-pvc-2
|
||||
labels:
|
||||
test: ceph
|
||||
funkypenguin-is: a-smartass
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: ceph-block
|
||||
resources:
|
||||
requests:
|
||||
storage: 128Mi
|
||||
EOF
|
||||
```
|
||||
|
||||
Now create a ceph-filesystem (RWX) PVC, by running:
|
||||
|
||||
```bash
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: ceph-filesystem-pvc
|
||||
labels:
|
||||
test: ceph
|
||||
funkypenguin-is: a-smartass
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: ceph-filesystem
|
||||
resources:
|
||||
requests:
|
||||
storage: 128Mi
|
||||
EOF
|
||||
```
|
||||
|
||||
Examine the PVCs by running:
|
||||
|
||||
```bash
|
||||
kubectl get pvc -l test=ceph
|
||||
```
|
||||
|
||||
#### Create Pod
|
||||
|
||||
Now create pods to consume the PVCs, by running:
|
||||
|
||||
```bash
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: ceph-test-1
|
||||
labels:
|
||||
test: ceph
|
||||
funkypenguin-is: a-smartass
|
||||
spec:
|
||||
containers:
|
||||
- name: volume-test
|
||||
image: nginx:stable-alpine
|
||||
imagePullPolicy: IfNotPresent
|
||||
volumeMounts:
|
||||
- name: ceph-block-is-rwo
|
||||
mountPath: /rwo
|
||||
- name: ceph-filesystem-is-rwx
|
||||
mountPath: /rwx
|
||||
ports:
|
||||
- containerPort: 80
|
||||
volumes:
|
||||
- name: ceph-block-is-rwo
|
||||
persistentVolumeClaim:
|
||||
claimName: ceph-block-pvc-1
|
||||
EOF
|
||||
```
|
||||
|
||||
And:
|
||||
|
||||
```bash
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: ceph-test-2
|
||||
labels:
|
||||
test: ceph
|
||||
funkypenguin-is: a-smartass
|
||||
spec:
|
||||
containers:
|
||||
- name: volume-test
|
||||
image: nginx:stable-alpine
|
||||
imagePullPolicy: IfNotPresent
|
||||
volumeMounts:
|
||||
- name: ceph-block-is-rwo
|
||||
mountPath: /rwo
|
||||
- name: ceph-filesystem-is-rwx
|
||||
mountPath: /rwx
|
||||
ports:
|
||||
- containerPort: 80
|
||||
volumes:
|
||||
- name: ceph-block-is-rwo
|
||||
persistentVolumeClaim:
|
||||
claimName: ceph-block-pvc-2
|
||||
- name: ceph-filesystem-is-rwx
|
||||
persistentVolumeClaim:
|
||||
claimName: ceph-filesystem-pvc
|
||||
EOF
|
||||
```
|
||||
|
||||
Ensure the pods have started successfully (*this indicates the PVCs were correctly attached*) by running:
|
||||
|
||||
```bash
|
||||
kubectl get pod -l test=ceph
|
||||
```
|
||||
|
||||
#### Clean up
|
||||
|
||||
Assuming that the pod is in a `Running` state, then TopoLVM is working!
|
||||
|
||||
Clean up your mess, little bare-metal-cave-monkey :monkey_face:, by running:
|
||||
|
||||
```bash
|
||||
kubectl delete pod -l funkypenguin-is=a-smartass
|
||||
kubectl delete pvc -l funkypenguin-is=a-smartass #(1)!
|
||||
```
|
||||
|
||||
1. Label selectors are powerful!
|
||||
|
||||
### View Ceph Dashboard
|
||||
|
||||
Assuming you have an Ingress Controller setup, and you've either picked a default IngressClass, or defined the dashboard ingress appropriately, you should be able to access your Ceph Dashboard, at the URL identified by the ingress (*this is a good opportunity to check that the ingress deployed correctly*):
|
||||
|
||||
```bash
|
||||
~ ❯ k get ingress -n rook-ceph
|
||||
NAME CLASS HOSTS ADDRESS PORTS AGE
|
||||
rook-ceph-mgr-dashboard nginx rook-ceph.batcave.awesome.me 172.16.237.1 80, 443 177d
|
||||
~ ❯
|
||||
```
|
||||
|
||||
The dashboard credentials are automatically generated for you by the operator, and stored in a Kubernetes secret. To retrieve your credentials, run:
|
||||
|
||||
```bash
|
||||
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o \
|
||||
jsonpath="{['data']['password']}" | base64 --decode && echo
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
What have we achieved? We're half-way to getting a ceph cluster, having deployed the operator which will manage the lifecycle of the [ceph cluster](/kubernetes/persistence/rook-ceph/cluster/) we're about to create!
|
||||
|
||||
!!! summary "Summary"
|
||||
Created:
|
||||
|
||||
* [X] Ceph cluster has been deployed
|
||||
* [X] StorageClasses are available so that the cluster storage can be consumed by your pods
|
||||
* [X] Pretty graphs are viewable in the Ceph Dashboard
|
||||
|
||||
--8<-- "recipe-footer.md"
|
||||
|
||||
[^1]: Unless you **wanted** to deploy your cluster components in a separate namespace to the operator, of course!
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
title: How to use Rook Ceph for Persistent Storage in Kubernetes
|
||||
description: How to deploy Rook Ceph into your Kubernetes cluster for persistent storage
|
||||
---
|
||||
# Persistent storage in Kubernetes with Rook Ceph / CephFS
|
||||
|
||||
[Ceph](https://docs.ceph.com/en/quincy/) is a highly-reliable, scalable network storage platform which uses individual disks across participating nodes to provide fault-tolerant storage.
|
||||
|
||||
{ loading=lazy }
|
||||
|
||||
[Rook](https://rook.io) provides an operator for Ceph, decomposing the [10-year-old](https://en.wikipedia.org/wiki/Ceph_(software)#Release_history), at-time-arcane, platform into cloud-native components, created declaratively, whose lifecycle is managed by an operator.
|
||||
|
||||
The simplest way to think about running rook-ceph is separate the [operator](/kubernetes/persistence/rook-ceph/operator/) (*a generic worker which manages the lifecycle of your cluster*) from your desired [cluster](/kubernetes/persistence/rook-ceph/cluster/) config itself (*spec*).
|
||||
|
||||
To this end, I've defined each as a separate component, below:
|
||||
|
||||
1. First, install the [operator](/kubernetes/persistence/rook-ceph/operator/)
|
||||
2. Then, define your [cluster](/kubernetes/persistence/rook-ceph/cluster/)
|
||||
3. Win!
|
||||
@@ -0,0 +1,182 @@
|
||||
---
|
||||
title: Deploy Rook Ceph Operator for Persistent Storage in Kubernetes
|
||||
description: Start your Rook Ceph deployment by installing the operator into your Kubernetes cluster
|
||||
---
|
||||
|
||||
# Persistent storage in Kubernetes with Rook Ceph / CephFS - Operator
|
||||
|
||||
[Ceph](https://docs.ceph.com/en/quincy/) is a highly-reliable, scalable network storage platform which uses individual disks across participating nodes to provide fault-tolerant storage.
|
||||
|
||||
[Rook](https://rook.io) provides an operator for Ceph, decomposing the [10-year-old](https://en.wikipedia.org/wiki/Ceph_(software)#Release_history), at-time-arcane, platform into cloud-native components, created declaratively, whose lifecycle is managed by an operator.
|
||||
|
||||
To start off with, we need to deploy the ceph operator into the cluster, after which, we'll be able to actually deploy our [ceph cluster itself](/kubernetes/persistence/rook-ceph/cluster/).
|
||||
|
||||
## Rook Ceph requirements
|
||||
|
||||
!!! summary "Ingredients"
|
||||
|
||||
Already deployed:
|
||||
|
||||
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
|
||||
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
|
||||
|
||||
## Preparation
|
||||
|
||||
### Namespace
|
||||
|
||||
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `/bootstrap/namespaces/namespace-rook-system.yaml`:
|
||||
|
||||
```yaml title="/bootstrap/namespaces/namespace-rook-ceph.yaml"
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: rook-system
|
||||
```
|
||||
|
||||
### HelmRepository
|
||||
|
||||
We're going to install a helm chart from the Rook Ceph chart repository, so I create the following in my flux repo:
|
||||
|
||||
```yaml title="/bootstrap/helmrepositories/gitepository-rook-release.yaml"
|
||||
apiVersion: source.toolkit.fluxcd.io/v1beta1
|
||||
kind: HelmRepository
|
||||
metadata:
|
||||
name: rook-release
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 15m
|
||||
url: https://charts.rook.io/release
|
||||
```
|
||||
|
||||
### Kustomization
|
||||
|
||||
Now that the "global" elements of this deployment (*just the HelmRepository in this case*) have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at `/rook-ceph`. I create this example Kustomization in my flux repo:
|
||||
|
||||
```yaml title="/bootstrap/kustomizations/kustomization-rook-ceph.yaml"
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: rook-ceph
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 30m
|
||||
path: ./rook-ceph
|
||||
prune: true # remove any elements later removed from the above path
|
||||
timeout: 10m # if not set, this defaults to interval duration, which is 1h
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
validation: server
|
||||
healthChecks:
|
||||
- apiVersion: apiextensions.k8s.io/v1
|
||||
kind: CustomResourceDefinition
|
||||
name: cephblockpools.ceph.rook.io
|
||||
```
|
||||
|
||||
--8<-- "premix-cta-kubernetes.md"
|
||||
|
||||
### ConfigMap
|
||||
|
||||
Now we're into the app-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/rook/rook/blob/master/deploy/charts/rook-ceph/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo:
|
||||
|
||||
```yaml title="rook-ceph/configmap-rook-ceph-helm-chart-value-overrides.yaml"
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: rook-ceph-helm-chart-value-overrides
|
||||
namespace: rook-ceph
|
||||
data:
|
||||
values.yaml: |- # (1)!
|
||||
# <upstream values go here>
|
||||
```
|
||||
|
||||
1. Paste in the contents of the upstream `values.yaml` here, intended 4 spaces, and then change the values you need as illustrated below.
|
||||
|
||||
Values I change from the default are:
|
||||
|
||||
```yaml
|
||||
pspEnable: false # (1)!
|
||||
```
|
||||
|
||||
1. PSPs are deprecated, and will eventually be removed in Kubernetes 1.25, at which point this will cause breakage.
|
||||
|
||||
### HelmRelease
|
||||
|
||||
Finally, having set the scene above, we define the HelmRelease which will actually deploy the rook-ceph operator into the cluster. I save this in my flux repo:
|
||||
|
||||
```yaml title="/rook-ceph/helmrelease-rook-ceph.yaml"
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2beta1
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
name: rook-ceph
|
||||
namespace: rook-ceph
|
||||
spec:
|
||||
chart:
|
||||
spec:
|
||||
chart: rook-ceph
|
||||
version: 1.9.x
|
||||
sourceRef:
|
||||
kind: HelmRepository
|
||||
name: rook-release
|
||||
namespace: flux-system
|
||||
interval: 30m
|
||||
timeout: 10m
|
||||
install:
|
||||
remediation:
|
||||
retries: 3
|
||||
upgrade:
|
||||
remediation:
|
||||
retries: -1 # keep trying to remediate
|
||||
crds: CreateReplace # Upgrade CRDs on package update
|
||||
releaseName: rook-ceph
|
||||
valuesFrom:
|
||||
- kind: ConfigMap
|
||||
name: rook-ceph-helm-chart-value-overrides
|
||||
valuesKey: values.yaml # (1)!
|
||||
```
|
||||
|
||||
1. This is the default, but best to be explicit for clarity
|
||||
|
||||
## Install Rook Ceph Operator!
|
||||
|
||||
Commit the changes to your flux repository, and either wait for the reconciliation interval, or force a reconcilliation using `flux reconcile source git flux-system`. You should see the kustomization appear...
|
||||
|
||||
```bash
|
||||
~ ❯ flux get kustomizations rook-ceph
|
||||
NAME READY MESSAGE REVISION SUSPENDED
|
||||
rook-ceph True Applied revision: main/70da637 main/70da637 False
|
||||
~ ❯
|
||||
```
|
||||
|
||||
The helmrelease should be reconciled...
|
||||
|
||||
```bash
|
||||
~ ❯ flux get helmreleases -n rook-ceph rook-ceph
|
||||
NAME READY MESSAGE REVISION SUSPENDED
|
||||
rook-ceph True Release reconciliation succeeded v1.9.9 False
|
||||
~ ❯
|
||||
```
|
||||
|
||||
And you should have happy rook-ceph operator pods:
|
||||
|
||||
```bash
|
||||
~ ❯ k get pods -n rook-ceph -l app=rook-ceph-operator
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
rook-ceph-operator-7c94b7446d-nwsss 1/1 Running 0 5m14s
|
||||
~ ❯
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
What have we achieved? We're half-way to getting a ceph cluster, having deployed the operator which will manage the lifecycle of the [ceph cluster](/kubernetes/persistence/rook-ceph/cluster/) we're about to create!
|
||||
|
||||
!!! summary "Summary"
|
||||
Created:
|
||||
|
||||
* [X] Rook ceph operator running and ready to deploy a cluster!
|
||||
|
||||
Next:
|
||||
|
||||
* [ ] Deploy the ceph [cluster](/kubernetes/persistence/rook-ceph/cluster/) using a CR
|
||||
|
||||
--8<-- "recipe-footer.md"
|
||||
@@ -0,0 +1,282 @@
|
||||
---
|
||||
title: TopoLVM - Capacity-aware LVM-based storage on Kubernetes
|
||||
---
|
||||
# TopoLVM on Kubernetes
|
||||
|
||||
TopoLVM is **like** [Local Path Provisioner](/kubernetes/persistence/local-path-provisioner/), in that it deals with local volumes specific to each Kubernetes node, but it offers more flexibility, and is more suited for a production deployment.
|
||||
|
||||
!!! summary "Ingredients"
|
||||
|
||||
* [x] A [Kubernetes cluster](/kubernetes/cluster/)
|
||||
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped
|
||||
* [x] A dedicated disk, or free LVM volume space, for provisioning volumes
|
||||
|
||||
Additional benefits offered by TopoLVM are:
|
||||
|
||||
* Volumes can by dynamically expanded
|
||||
* The scheduler is capacity-aware, and can schedule pods to nodes with enough capacity for the pods' storage requirements
|
||||
* Multiple storageclasses are supported, so you could, for example, create a storageclass for HDD-backed volumes, and another for SSD-backed volumes
|
||||
|
||||
## Preparation
|
||||
|
||||
### Volume Group
|
||||
|
||||
Finally you get to do something on your nodes without YAML or git, like a pre-GitOps, bare-metal-cavemonkey! :monkey_face:
|
||||
|
||||
On each node, you'll need an LVM Volume Group (VG) for TopoLVM to consume. The most straightforward to to arrange this is to dedicate a disk to TopoLVM, and create a dedicated PV and VG for it.
|
||||
|
||||
In brief, assuming `/dev/sdb` is the disk (*and it's unused*), you'd do the following to create a VG called `VG-topolvm`:
|
||||
|
||||
```bash
|
||||
pvcreate /dev/sdb
|
||||
vgcreate VG-topolvm /dev/sdb
|
||||
```
|
||||
|
||||
!!! tip
|
||||
If you don't have a dedicated disk, you could try installing your OS using LVM partitioning, and leave some space unused, for TopoLVM to consume. Run `vgs` from an installed node to work out what the VG name is that the OS installer chose.
|
||||
|
||||
### Namespace
|
||||
|
||||
We need a namespace to deploy our HelmRelease and associated ConfigMaps into. Per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/namespaces/namespace-topolvm.yaml`:
|
||||
|
||||
??? example "Example NameSpace (click to expand)"
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: topolvm-system
|
||||
```
|
||||
|
||||
### HelmRepository
|
||||
|
||||
Next, we need to define a HelmRepository (*a repository of helm charts*), to which we'll refer when we create the HelmRelease. We only need to do this once per-repository. In this case, we're using the official [TopoLVM helm chart](https://github.com/topolvm/topolvm/tree/main/charts/topolvm), so per the [flux design](/kubernetes/deployment/flux/), I create this example yaml in my flux repo at `bootstrap/helmrepositories/helmrepository-topolvm.yaml`:
|
||||
|
||||
??? example "Example HelmRepository (click to expand)"
|
||||
```yaml
|
||||
apiVersion: source.toolkit.fluxcd.io/v1beta1
|
||||
kind: HelmRepository
|
||||
metadata:
|
||||
name: topolvm
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 15m
|
||||
url: https://topolvm.github.io/topolvm
|
||||
```
|
||||
|
||||
### Kustomization
|
||||
|
||||
Now that the "global" elements of this deployment (*Namespace and HelmRepository*) have been defined, we do some "flux-ception", and go one layer deeper, adding another Kustomization, telling flux to deploy any YAMLs found in the repo at `/topolvm`. I create this example Kustomization in my flux repo at `bootstrap/kustomizations/kustomization-topolvm.yaml`:
|
||||
|
||||
??? example "Example Kustomization (click to expand)"
|
||||
```yaml
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: topolvm--topolvm-system
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 15m
|
||||
path: ./topolvm-system
|
||||
prune: true # remove any elements later removed from the above path
|
||||
timeout: 2m # if not set, this defaults to interval duration, which is 1h
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
validation: server
|
||||
healthChecks:
|
||||
- apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: topolvm-controller
|
||||
namespace: topolvm-system
|
||||
- apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
name: topolvm-lvmd-0
|
||||
namespace: topolvm-system
|
||||
- apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
name: topolvm-node
|
||||
namespace: topolvm-system
|
||||
- apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
name: topolvm-scheduler
|
||||
namespace: topolvm-system
|
||||
```
|
||||
|
||||
!!! question "What's with that screwy name?"
|
||||
> Why'd you call the kustomization `topolvm--topolvm-system`?
|
||||
|
||||
I keep my file and object names as consistent as possible. In most cases, the helm chart is named the same as the namespace, but in some cases, by upstream chart or historical convention, the namespace is different to the chart name. TopoLVM is one of these - the helmrelease/chart name is `topolvm`, but the typical namespace it's deployed in is `topolvm-system`. (*Appending `-system` seems to be a convention used in some cases for applications which support the entire cluster*). To avoid confusion when I list all kustomizations with `kubectl get kustomization -A`, I give these oddballs a name which identifies both the helmrelease and the namespace.
|
||||
|
||||
### ConfigMap
|
||||
|
||||
Now we're into the topolvm-specific YAMLs. First, we create a ConfigMap, containing the entire contents of the helm chart's [values.yaml](https://github.com/topolvm/topolvm/blob/main/charts/topolvm/values.yaml). Paste the values into a `values.yaml` key as illustrated below, indented 4 tabs (*since they're "encapsulated" within the ConfigMap YAML*). I create this example yaml in my flux repo at `topolvm/configmap-topolvm-helm-chart-value-overrides.yaml`:
|
||||
|
||||
??? example "Example ConfigMap (click to expand)"
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
creationTimestamp: null
|
||||
name: topolvm-helm-chart-value-overrides
|
||||
namespace: topolvm
|
||||
data:
|
||||
values.yaml: |-
|
||||
# paste chart values.yaml (indented) here and alter as required>
|
||||
```
|
||||
|
||||
--8<-- "kubernetes-why-full-values-in-configmap.md"
|
||||
|
||||
Then work your way through the values you pasted, and change any which are specific to your configuration. You might want to start off by changing the following to match the name of the [volume group you created above](#volume-group).[^1]
|
||||
|
||||
```yaml hl_lines="10-13"
|
||||
lvmd:
|
||||
# lvmd.managed -- If true, set up lvmd service with DaemonSet.
|
||||
managed: true
|
||||
|
||||
# lvmd.socketName -- Specify socketName.
|
||||
socketName: /run/topolvm/lvmd.sock
|
||||
|
||||
# lvmd.deviceClasses -- Specify the device-class settings.
|
||||
deviceClasses:
|
||||
- name: ssd
|
||||
volume-group: myvg1
|
||||
default: true
|
||||
spare-gb: 10
|
||||
```
|
||||
|
||||
### HelmRelease
|
||||
|
||||
Lastly, having set the scene above, we define the HelmRelease which will actually deploy TopoLVM into the cluster, with the config we defined above. I save this in my flux repo as `topolvm/helmrelease-topolvm.yaml`:
|
||||
|
||||
??? example "Example HelmRelease (click to expand)"
|
||||
```yaml
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2beta1
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
name: topolvm
|
||||
namespace: topolvm-system
|
||||
spec:
|
||||
chart:
|
||||
spec:
|
||||
chart: topolvm
|
||||
version: 3.x
|
||||
sourceRef:
|
||||
kind: HelmRepository
|
||||
name: topolvm
|
||||
namespace: flux-system
|
||||
interval: 15m
|
||||
timeout: 5m
|
||||
releaseName: topolvm
|
||||
valuesFrom:
|
||||
- kind: ConfigMap
|
||||
name: topolvm-helm-chart-value-overrides
|
||||
valuesKey: values.yaml # This is the default, but best to be explicit for clarity
|
||||
```
|
||||
|
||||
--8<-- "kubernetes-why-not-config-in-helmrelease.md"
|
||||
|
||||
## Serving
|
||||
|
||||
### Deploy TopoLVM
|
||||
|
||||
Having committed the above to your flux repository, you should shortly see a topolvm kustomization, and in the `topolvm-system` namespace, a bunch of pods:
|
||||
|
||||
```bash
|
||||
demo@shredder:~$ kubectl get pods -n topolvm-system
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
topolvm-controller-85698b44dd-65fd9 4/4 Running 0 133m
|
||||
topolvm-controller-85698b44dd-dmncr 4/4 Running 0 133m
|
||||
topolvm-lvmd-0-98h4q 1/1 Running 0 133m
|
||||
topolvm-lvmd-0-b29t8 1/1 Running 0 133m
|
||||
topolvm-lvmd-0-c5vnf 1/1 Running 0 133m
|
||||
topolvm-lvmd-0-hmmq5 1/1 Running 0 133m
|
||||
topolvm-lvmd-0-zfldv 1/1 Running 0 133m
|
||||
topolvm-node-6p4qz 3/3 Running 0 133m
|
||||
topolvm-node-7vdgt 3/3 Running 0 133m
|
||||
topolvm-node-mlp4x 3/3 Running 0 133m
|
||||
topolvm-node-sxtn5 3/3 Running 0 133m
|
||||
topolvm-node-xf265 3/3 Running 0 133m
|
||||
topolvm-scheduler-jlwsh 1/1 Running 0 133m
|
||||
topolvm-scheduler-nj8nz 1/1 Running 0 133m
|
||||
topolvm-scheduler-tg72z 1/1 Running 0 133m
|
||||
demo@shredder:~$
|
||||
```
|
||||
|
||||
### How do I know it's working?
|
||||
|
||||
So the controllers etc are running, but how do we know we can actually provision volumes?
|
||||
|
||||
#### Create PVC
|
||||
|
||||
Create a PVC, by running:
|
||||
|
||||
```bash
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: topolvm-pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: topolvm-provisioner
|
||||
resources:
|
||||
requests:
|
||||
storage: 128Mi
|
||||
EOF
|
||||
```
|
||||
|
||||
Examine the PVC by running `kubectl describe pvc topolvm-pvc`
|
||||
|
||||
#### Create Pod
|
||||
|
||||
Now create a pod to consume the PVC, by running:
|
||||
|
||||
```bash
|
||||
cat <<EOF | kubectl create -f -
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: topolvm-test
|
||||
spec:
|
||||
containers:
|
||||
- name: volume-test
|
||||
image: nginx:stable-alpine
|
||||
imagePullPolicy: IfNotPresent
|
||||
volumeMounts:
|
||||
- name: topolvm-rocks
|
||||
mountPath: /data
|
||||
ports:
|
||||
- containerPort: 80
|
||||
volumes:
|
||||
- name: topolvm-rocks
|
||||
persistentVolumeClaim:
|
||||
claimName: topolvm-pvc
|
||||
EOF
|
||||
```
|
||||
|
||||
Examine the pod by running `kubectl describe pod topolvm-test`.
|
||||
|
||||
#### Clean up
|
||||
|
||||
Assuming that the pod is in a `Running` state, then TopoLVM is working!
|
||||
|
||||
Clean up your mess, little bare-metal-cave-monkey :monkey_face:, by running:
|
||||
|
||||
```bash
|
||||
kubectl delete pod topolvm-test
|
||||
kubectl delete pvc topolvm-pvc
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
Are things not working as expected? Try one of the following to look for issues:
|
||||
|
||||
1. Watch the lvmd logs, by running `kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=topolvm-lvmd`
|
||||
2. Watch the node logs, by running `kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=topolvm-node`
|
||||
3. Watch the scheduler logs, by running `kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=scheduler`
|
||||
4. Watch the controller node logs, by running `kubectl logs -f -n topolvm-system -l app.kubernetes.io/name=controller`
|
||||
|
||||
--8<-- "recipe-footer.md"
|
||||
|
||||
[^1]: This is where you'd add multiple Volume Groups if you wanted a storageclass per Volume Group
|
||||
Reference in New Issue
Block a user