public/geek-cookbook
1
0
Fork 0
mirror of https://github.com/funkypenguin/geek-cookbook/ synced 2025-12-13 17:56:26 +00:00
Code Issues Releases Wiki Activity

Completed section on shared storage

Browse Source
This commit is contained in:
David Young
2017-07-18 23:06:38 +12:00
parent 307227ca54
commit 51c7aa5098
9 changed files with 185 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
docs/advanced/docker.md
View File

@@ -251,17 +251,6 @@ need to add something to rc.local to make glustetr fs mount
__ maybe __this works:
setsebool -P virt_sandbox_use_fusefs on
https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_glusterfs.html#selinux
Stupid cloud-init makes the system slow to boot:
[root@ds1 ~]# systemctl mask cloud-final.service
Created symlink from /etc/systemd/system/cloud-final.service to /dev/null.
[root@ds1 ~]# systemctl mask cloud-config.service
Created symlink from /etc/systemd/system/cloud-config.service to /dev/null.
[root@ds1 ~]#
@@ -287,10 +276,3 @@ To this:
}```
!!! note the comma after "false" above
docker run -d --name nfsd --restart always --privileged -v /mnt/data:/mnt/data -e SHARED_DIRECTORY=/mnt/data --network=host itsthenetwork/nfs-server-alpine:latest

13
docs/ha-docker-swarm/docker.md
View File

@@ -1,13 +0,0 @@
# Introduction
We start building our cloud with virtual machines. You could use bare-metal machines as well, the configuration would be the same. Given that most readers (myself included) will be using virtual infrastructure, from now on I'll be referring strictly to VMs.
## Ingredients
3 x Virtual Machines, each with:
* CentOS/Fedora Atomic
* At least 1GB RAM
* At least 20GB disk space (but it'll be tight)
* Connectivity to each other within the same subnet, and on a low-latency link (i.e., no WAN links)
## Preparation

70
docs/ha-docker-swarm/persistent-storage.md
View File

@@ -1,70 +0,0 @@
# Introduction
While Docker Swarm is great for keeping containers running (and restarting those that fail), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts, you need to think about storage.
## Ingredients
!!! summary "Ingredients"
3 x Virtual Machines (configured earlier), each with:
* [X] CentOS/Fedora Atomic
* [X] At least 1GB RAM
* [X] At least 20GB disk space (_but it'll be tight_)
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
* [ ] A second disk, or adequate space on the primary disk for a dedicated data partition
## Preparation
### Create Gluster "bricks"
To build our Gluster volume, we need each of the 3 VMs to provide one "brick". The bricks will be used to create the replicated volume.
On each host, run the following to create your bricks, adjusted for the path to your disk.
```
mkfs.xfs -i size=512 /dev/vdb1
mkdir -p /data/glusterfs/docker-persistent/brick1
echo '/dev/vdb1 /data/glusterfs/docker-persistent/brick1/ xfs defaults 1 2' >> /etc/fstab
mount -a && mount
```
!!! warning "Don't provision all your LVM space"
Atomic uses LVM to store docker data, and **automatically grows** Docker's volumes as requried. If you commit all your free LVM space to your brick, you'll quickly find (as I did) that docker will start to fail with error messages about insufficient space. If you're going to slice off a portion of your LVM space in /dev/atomicos, make sure you leave enough space for Docker storage, where "enough" depends on how much you plan to pull images, make volumes, etc. I ate through 20GB very quickly doing development, so I ended up provisioning 50GB for atomic alone, with a separate volume for the brick.
### Create glusterfs container
Atomic doesn't include the Gluster server components. This means we'll have to run glusterd from within a container, with privileged access to the host. Although convoluted, I actually prefer this design since it once again makes the OS "disposable", moving all the config into containers and code.
Run the following on each host:
````
docker run \
-h glusterfs-server \
-v /etc/glusterfs:/etc/glusterfs:z \
-v /var/lib/glusterd:/var/lib/glusterd:z \
-v /var/log/glusterfs:/var/log/glusterfs:z \
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-v /data/glusterfs/gv0/brick1:/data/glusterfs/gv0/brick1 \
-d --privileged=true --net=host \
--restart=always \
--name="glusterfs-server" \
gluster/gluster-centos
````
### Create gluster volume
Now we create a *replicated volume* out of our individual "bricks". On a single node (doesn't matter which), run ```docker exec -it glusterfs-server bash``` to launch a shell inside the container.
Create the gluster volume by running
```gluster volume create gv0 replica 2 server1:/data/glusterfs/gv0/brick1 server2:/data/glusterfs/gv0/brick1 server3:/data/glusterfs/gv0/brick1```
Start the volume by running ```gluster volume start gv0```
The volume is only present on the host you're shelled into though. To add the other hosts to the volume, run ```gluster peer probe <servername>```. Don't probe host from itself.
From one other host, run ```docker exec -it glusterfs-server bash``` to shell into the gluster-server container, and run ```gluster peer probe <original server name>``` to update the name of the host which started the volume.
### Mount gluster volume
On the host (i.e., outside of the container - type ```exit``` if you're still shelled in), create a mountpoint for the data, by running ```mkdir /srv/data```
Add an entry to fstab to ensure the volume is auto-mounted on boot:

142
docs/ha-docker-swarm/shared-storage.md Normal file
View File

@@ -0,0 +1,142 @@
# Introduction
While Docker Swarm is great for keeping containers running (_and restarting those that fail_), it does nothing for persistent storage. This means if you actually want your containers to keep any data persistent across restarts (_hint: you do!_), you need to provide shared storage to every docker node.
## Ingredients
!!! summary "Ingredients"
3 x Virtual Machines (configured earlier), each with:
* [X] CentOS/Fedora Atomic
* [X] At least 1GB RAM
* [X] At least 20GB disk space (_but it'll be tight_)
* [X] Connectivity to each other within the same subnet, and on a low-latency link (_i.e., no WAN links_)
* [ ] A second disk, or adequate space on the primary disk for a dedicated data partition
## Preparation
### Create Gluster "bricks"
To build our Gluster volume, we need 2 out of the 3 VMs to provide one "brick". The bricks will be used to create the replicated volume. Assuming a replica count of 2 (_i.e., 2 copies of the data are kept in gluster_), our total number of bricks must be divisible by our replica count. (_I.e., you can't have 3 bricks if you want 2 replicas. You can have 4 though - We have to have minimum 3 swarm manager nodes for fault-tolerance, but only 2 of those nodes need to run as gluster servers._)
On each host, run a variation following to create your bricks, adjusted for the path to your disk.
!!! note "The example below assumes /dev/vdb is dedicated to the gluster volume"
```
(
echo o # Create a new empty DOS partition table
echo n # Add a new partition
echo p # Primary partition
echo 1 # Partition number
echo # First sector (Accept default: 1)
echo # Last sector (Accept default: varies)
echo w # Write changes
) | sudo fdisk /dev/vdb
mkfs.xfs -i size=512 /dev/vdb1
mkdir -p /var/no-direct-write-here/brick1
echo '' >> /etc/fstab >> /etc/fstab
echo '# Mount /dev/vdb1 so that it can be used as a glusterfs volume' >> /etc/fstab
echo '/dev/vdb1 /var/no-direct-write-here/brick1 xfs defaults 1 2' >> /etc/fstab
mount -a && mount
```
!!! warning "Don't provision all your LVM space"
Atomic uses LVM to store docker data, and **automatically grows** Docker's volumes as requried. If you commit all your free LVM space to your brick, you'll quickly find (as I did) that docker will start to fail with error messages about insufficient space. If you're going to slice off a portion of your LVM space in /dev/atomicos, make sure you leave enough space for Docker storage, where "enough" depends on how much you plan to pull images, make volumes, etc. I ate through 20GB very quickly doing development, so I ended up provisioning 50GB for atomic alone, with a separate volume for the brick.
### Create glusterfs container
Atomic doesn't include the Gluster server components. This means we'll have to run glusterd from within a container, with privileged access to the host. Although convoluted, I've come to prefer this design since it once again makes the OS "disposable", moving all the config into containers and code.
Run the following on each host:
````
docker run \
-h glusterfs-server \
-v /etc/glusterfs:/etc/glusterfs:z \
-v /var/lib/glusterd:/var/lib/glusterd:z \
-v /var/log/glusterfs:/var/log/glusterfs:z \
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-v /var/no-direct-write-here/brick1:/var/no-direct-write-here/brick1 \
-d --privileged=true --net=host \
--restart=always \
--name="glusterfs-server" \
gluster/gluster-centos
````
### Create trusted pool
On a single node (doesn't matter which), run ```docker exec -it glusterfs-server bash``` to launch a shell inside the container.
From the node, run
```gluster peer probe <other host>```
Example output:
```
[root@glusterfs-server /]# gluster peer probe ds1
peer probe: success.
[root@glusterfs-server /]#
```
Run ```gluster peer status``` on both nodes to confirm that they're properly connected to each other:
Example output:
```
[root@glusterfs-server /]# gluster peer status
Number of Peers: 1
Hostname: ds3
Uuid: 3e115ba9-6a4f-48dd-87d7-e843170ff499
State: Peer in Cluster (Connected)
[root@glusterfs-server /]#
```
### Create gluster volume
Now we create a *replicated volume* out of our individual "bricks".
Create the gluster volume by running
```
gluster volume create gv0 replica 2 \
server1:/var/no-direct-write-here/brick1 \
server2:/var/no-direct-write-here/brick1
```
Example output:
```
[root@glusterfs-server /]# gluster volume create gv0 replica 2 ds1:/var/no-direct-write-here/brick1/gv0 ds3:/var/no-direct-write-here/brick1/gv0
volume create: gv0: success: please start the volume to access data
[root@glusterfs-server /]#
```
Start the volume by running ```gluster volume start gv0```
```
[root@glusterfs-server /]# gluster volume start gv0
volume start: gv0: success
[root@glusterfs-server /]#
```
The volume is only present on the host you're shelled into though. To add the other hosts to the volume, run ```gluster peer probe <servername>```. Don't probe host from itself.
From one other host, run ```docker exec -it glusterfs-server bash``` to shell into the gluster-server container, and run ```gluster peer probe <original server name>``` to update the name of the host which started the volume.
### Mount gluster volume
On the host (i.e., outside of the container - type ```exit``` if you're still shelled in), create a mountpoint for the data, by running ```mkdir /var/data```, and add an entry to fstab to ensure the volume is auto-mounted on boot:
```
mkdir /var/data
MYHOST=`hostname -s`
echo '' >> /etc/fstab >> /etc/fstab
echo '# Mount glusterfs volume' >> /etc/fstab
echo "$MYHOST:/gv0 /var/data glusterfs defaults,_netdev,context="system_u:object_r:svirt_sandbox_file_t:s0" 0 0" >> /etc/fstab
mount -a
```
For non-gluster nodes, you'll need to replace $MYHOST above with the name of one of the gluster hosts (I haven't worked out how to make this fully HA yet)
!!! summary "Ready to serve..."
After completing the above, you should have:
* [X] Persistent storage available to every node
* [X] Resiliency in the event of the failure of a single (gluster) node

7
docs/ha-docker-swarm/traefik.md
View File

@@ -0,0 +1,7 @@
````
mkdir ~/dockersock
cd ~/dockersock
curl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/Makefile
curl -O https://raw.githubusercontent.com/dpw/selinux-dockersock/master/dockersock.te
make && semodule -i dockersock.pp
````

20
docs/ha-docker-swarm/vms.md
View File

@@ -67,14 +67,32 @@ To:
!!! tip ""
Note the extra comma required after "false" above
Add some handy bash auto-completion for docker. Without this, you'll get annoyed that you can't autocomplete ```docker stack deploy <blah> -c <blah.yml>``` commands.
```
cd /etc/bash_completion.d/
curl -O https://raw.githubusercontent.com/docker/cli/b75596e1e4d5295ac69b9934d1bd8aff691a0de8/contrib/completion/bash/docker
```
### Upgrade Atomic
Finally, apply any Atomic host updates, and reboot, by running: ```atomic host upgrade && systemctl reboot```.
### Permit connectivity between VMs
By default, Atomic only permits incoming SSH. We'll want to allow all traffic between our nodes, so add something like this to /etc/sysconfig/iptables:
```
# Allow all inter-node communication
-A INPUT -s 192.168.31.0/24 -j ACCEPT
```
And restart iptables with ```systemctl restart iptables```
!!! summary "Ready to serve..."
After completing the above, you should have:
* [X] 3 fresh atomic instances, at the latest releases
* [X] Docker 1.13, with experimental features enabled
* [X] Docker 1.13, with experimental features enabled

8
examples/cloud-config.txt
View File

@@ -1,6 +1,4 @@
#cloud-config
ssh_authorized_keys:
- ssh-rsa ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFkmE2zK7uE9q75nzQxa9tQPHiCgaEUkIDj9xdoPL911 davidy@funkypenguin.co.nz
write_files:
- path: /etc/docker-latest/daemon.json
content: |
@@ -9,6 +7,12 @@ write_files:
"signature-verification": false,
"experimental": true
}
# Add users to the system. Users are added after groups are added.
users:
- name: davidy
sudo: ALL=(ALL) NOPASSWD:ALL
ssh-authorized-keys:
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFkmE2zK7uE9q75nzQxa9tQPHiCgaEUkIDj9xdoPL911 davidy@funkypenguin.co.nz
runcmd:
- systemctl disable docker --now
- systemctl enable docker-latest --now

10
examples/create_data_volume.sh Normal file
View File

@@ -0,0 +1,10 @@
#!/bin/bash
(
echo o # Create a new empty DOS partition table
echo n # Add a new partition
echo p # Primary partition
echo 1 # Partition number
echo # First sector (Accept default: 1)
echo # Last sector (Accept default: varies)
echo w # Write changes
) | sudo fdisk /dev/vdb

2
mkdocs.yml
View File

@@ -20,7 +20,7 @@ pages:
- HA Docker Swarm:
- Design: ha-docker-swarm/design.md
- VMs: ha-docker-swarm/vms.md
- Persistent Storage: ha-docker-swarm/persistent-storage.md
- Shared Storage: ha-docker-swarm/shared-storage.md
- Keepalived: ha-docker-swarm/keepalived.md
- Docker Swarm Mode: ha-docker-swarm/docker-swarm-mode.md
- Traefik: ha-docker-swarm/traefik.md