geek-cookbook/docs/kubernetes/persistence/aws-ebs.md

title, description, values_yaml_url, helm_chart_version, helm_chart_name, helm_chart_repo_name, helm_chart_repo_url, helmrelease_name, helmrelease_namespace, kustomization_name, slug, status, upstream, github_repo

title	description	values_yaml_url	helm_chart_version	helm_chart_name	helm_chart_repo_name	helm_chart_repo_url	helmrelease_name	helmrelease_namespace	kustomization_name	slug	status	upstream	github_repo
Support CSI VolumeSnapshots with snapshot-controller	Add CSI VolumeSnapshot support with snapshot support	https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/charts/aws-ebs-csi-driver/values.yaml	2.24.x	aws-ebs-csi-driver	aws-ebs-csi-driver		aws-ebs-csi-driver	aws-ebs-csi-driver	aws-ebs-csi-driver	EBS CSI Driver	new	https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html	https://github.com/kubernetes-sigs/aws-ebs-csi-driver

Install the AWS EBS CSI driver

The Amazon Elastic Block Store Container Storage Interface (CSI) Driver provides a CSI interface used by Container Orchestrators to manage the lifecycle of Amazon EBS volumes. It's a convenient way to consume EBS storage, which works consistently with other CSI-based tooling (for example, you can dynamically expand and snapshot volumes).

??? question "Tell me about the features..."

* Static Provisioning - Associate an externally-created EBS volume with a PersistentVolume (PV) for consumption within Kubernetes.
* Dynamic Provisioning - Automatically create EBS volumes and associated PersistentVolumes (PV) from PersistentVolumeClaims) (PVC). Parameters can be passed via a StorageClass for fine-grained control over volume creation.
* Mount Options - Mount options could be specified in the PersistentVolume (PV) resource to define how the volume should be mounted.
* NVMe Volumes - Consume NVMe volumes from EC2 Nitro instances.
* Block Volumes - Consume an EBS volume as a raw block device.
* Volume Snapshots - Create and restore snapshots taken from a volume in Kubernetes.
* Volume Resizing - Expand the volume by specifying a new size in the PersistentVolumeClaim (PVC).

!!! summary "Ingredients"

* [x] A [Kubernetes cluster](/kubernetes/cluster/) on [AWS EKS](/kubernetes/cluster/eks/)
* [x] [Flux deployment process](/kubernetes/deployment/flux/) bootstrapped

{% include 'kubernetes-flux-namespace.md' %} {% include 'kubernetes-flux-helmrepository.md' %} {% include 'kubernetes-flux-kustomization.md' %} {% include 'kubernetes-flux-helmrelease.md' %}

Install {{ page.meta.slug }}!

Commit the changes to your flux repository, and either wait for the reconciliation interval, or force a reconcilliation using flux reconcile source git flux-system. You should see the kustomization appear...

~ ❯ flux get kustomizations {{ page.meta.kustomization_name }}
NAME     	READY	MESSAGE                       	REVISION    	SUSPENDED
{{ page.meta.kustomization_name }}	True 	Applied revision: main/70da637	main/70da637	False
~ ❯

The helmrelease should be reconciled...

~ ❯ flux get helmreleases -n {{ page.meta.helmrelease_namespace }} {{ page.meta.helmrelease_name }}
NAME     	READY	MESSAGE                         	REVISION	SUSPENDED
{{ page.meta.helmrelease_name }}	True 	Release reconciliation succeeded	v{{ page.meta.helm_chart_version }}  	False
~ ❯

And you should have happy pods in the {{ page.meta.helmrelease_namespace }} namespace:

~ ❯ k get pods -n {{ page.meta.helmrelease_namespace }} -l app.kubernetes.io/name={{ page.meta.helmrelease_name }}
NAME                                  READY   STATUS    RESTARTS   AGE
ebs-csi-controller-77bddb4c95-2bzw5   5/5     Running   1 (10h ago)   37h
ebs-csi-controller-77bddb4c95-qr2hk   5/5     Running   0             37h
ebs-csi-node-4f8kz                    3/3     Running   0             37h
ebs-csi-node-fq8bn                    3/3     Running   0             37h
~ ❯

Setup IRSA

Before you can attach EBS volumes with aws-ebs-csi-driver, it's necessary to perform some AWS IAM acronym-salad first :salad: ..

The CSI driver pods need access to your AWS account in order to provision EBS volumes. You could feed them with classic access key/secret keys, but a more "sophisticated" method is to use "IAM roles for service accounts", or IRSA.

IRSA lets you associate a Kubernetes service account with an IAM role, so instead of stashing access secrets somewhere in a namespace (and in your GitOps repo¹ ), you simply tell AWS "grant the service account batcave-music in the namespace bat-ertainment the ability to use my streamToAlexa IAM role.

Before we start, we have to use eksctl to generate an IAM OIDC provider for your cluster, in case we don't have one. I ran:

eksctl utils associate-iam-oidc-provider --cluster=funkypenguin-authentik-test --approve

(It's harmless to run it more than once, if you already have an IAM OIDC provider associated, the command will just error)

Once complete, I ran the following to grant the aws-ebs-csi-driver service account in the aws-ebs-csi-driver namespace the power to use the AWS-managed AmazonEBSCSIDriverPolicy policy, which exists for exactly this purpose:

eksctl create iamserviceaccount \
    --name ebs-csi-controller-sa \
    --namespace aws-ebs-csi-driver \
    --cluster funkypenguin-authentik-test \
    --role-name AmazonEKS_EBS_CSI_DriverRole \
    --override-existing-serviceaccounts \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
    --approve

This will annotate the existing serviceaccount in the aws-ebs-csi-driver namespace, with the role to be attached.

Confirm it's worked by describing the serviceAccount - you should see an annotation indicating the role attached, like this:

Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::6831384437293:role/AmazonEKS_EBS_CSI_DriverRole

Troubleshooting

If it doesn't work for some reason (like you ran the command once with a typo!), you may find yourself unable to re-run the command. Cloudformation logs will show you that the action is failing because the role name already exits. To work around this, grab the ARN of the existing role, and change the command slightly:

eksctl create iamserviceaccount \
    --name ebs-csi-controller-sa \
    --namespace aws-ebs-csi-driver \
    --cluster funkypenguin-authentik-test \
    --attach-role-arn arn:aws:iam::683179697293:role/AmazonEKS_EBS_CSI_DriverRole \
    --override-existing-serviceaccounts \
    --approve

How do I know it's working?

So the AWS EBS CSI driver is installed, but how do we know it's working, especially that IRSA voodoo?

Check pod logs

First off, check the pod logs for any errors, by running:

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver

If you see nasty errors about EBS access denied, then revisit the IRSA magic above. If not, proceed with the acid test 🧪 below..

Create resources

Create PVCs

Create a PVCs (persistent volume claim), by running:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: aws-ebs-csi-test
  labels:
    test: aws-ebs-csi
    funkypenguin-is: a-smartass  
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 128Mi
EOF

Examine the PVC, and note that it's in a pending state (this is normal):

kubectl get pvc -l test=aws-ebs-csi

Create Pod

Now create a pod to consume the PVC, by running:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: aws-ebs-csi-test
  labels:
    test: aws-ebs-csi
    funkypenguin-is: a-smartass  
spec:
  containers:
  - name: volume-test
    image: nginx:stable-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: ebs-volume
      mountPath: /i-am-a-volume
    ports:
    - containerPort: 80
  volumes:
  - name: ebs-volume
    persistentVolumeClaim:
      claimName: aws-ebs-csi-test
EOF

Ensure the pods have started successfully (this indicates the PVCs were correctly attached) by running:

kubectl get pod -l test=aws-ebs-csi

Clean up

Assuming that the pod is in a Running state, then your EBS provisioning, and all the background AWS plumbing, worked!

Clean up your mess, little cloud-monkey 🐵, by running:

kubectl delete pod -l funkypenguin-is=a-smartass
kubectl delete pvc -l funkypenguin-is=a-smartass

Summary

What have we achieved? We're now able to persist data in our EKS cluster, and have left the door open for future options like snapshots, volume expansion, etc.

!!! summary "Summary" Created:

* [X] AWS EBS CSI driver installed and tested in our EKS cluster
* [X] Future support for [Velero][velero] with [csi-snapshots](/kubernetes/backup/csi-snapshots/), and volume expansion

{% include 'recipe-footer.md' %}

---

1. Negated somewhat with Sealed Secrets ↩︎