public/geek-cookbook
1
0
Fork 0
mirror of https://github.com/funkypenguin/geek-cookbook/ synced 2025-12-15 18:56:24 +00:00
Code Issues Releases Wiki Activity

Updated doc structure (#9)

Browse Source
This commit is contained in:
David Young
2017-08-04 22:34:41 +12:00
committed by GitHub
parent 05a146f11c
commit e9d0bb822e
89 changed files with 25 additions and 10363 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
manuscript/recipies/gitlab-runner.md Normal file
View File

@@ -0,0 +1,60 @@
# Gitlab Runner
Some features of GitLab require a "[runner](https://docs.gitlab.com/runner/)" (_in the sense of a "gopher" or a "minion"_). A runner "registers" itself with a GitLab instance, and is given tasks to run. Tasks include running Continuous Integration (CI) builds, and building container images.
While a runner isn't strictly required to use GitLab, if you want to do CI, you'll need at least one. There are many was to deploy a runner - this recipe focuses on the docker container model.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [GitLab](/ha-docker-swarm/gitlab) installation (see previous recipe)
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our runner containers, so create them in /var/data/gitlab:
```
cd /var/data
mkdir gitlab
cd gitlab
mkdir -p {runners/1,runners/2}
```
### Configure runners
From your GitLab UI, you can retrieve a "token" necessary to register a new runner. To register the runner, you can either create config.toml in each runner's bind-mounted folder (example below), or just "docker exec" into each runner container and execute ```gitlab-container register``` to interactively generate config.toml.
Sample runner config.toml:
```
concurrent = 1
check_interval = 0
[[runners]]
name = "myrunner1"
url = "https://gitlab.example.com"
token = "<long string here>"
executor = "docker"
[runners.docker]
tls_verify = false
image = "ruby:2.1"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
```
## Serving
### Launch runners
Launch the mail server stack by running ```docker stack deploy gitlab-runner -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user "root" and the password you specified in gitlab.env.
## Chef's Notes
1. You'll note that I setup 2 runners. One is locked to a single project (this cookbook build), and the other is a shared runner. I wanted to ensure that one runner was always available to run CI for this project, even if I'd tied up another runner on something heavy-duty, like a container build. Customize this to your use case.
2. Originally I deployed runners in the same stack as GitLab, but I found that they would frequently fail to start properly when I launched the stack. I think that this was because the runners started so quickly (and GitLab starts so slowly!), that they always started up reporting that the GitLab instance was invalid or unavailable. I had issues with CI builds stuck permanently in a "pending" state, which were only resolved by restarting the runner. Having the runners deployed in a separate stack to GitLab avoids this problem.

124
manuscript/recipies/gitlab.md Normal file
View File

@@ -0,0 +1,124 @@
# GitLab
GitLab is a self-hosted [alternative to GitHub](https://about.gitlab.com/comparison/). The most common use case is (a set of) developers with the desire for the rich feature-set of GitHub, but with unlimited private repositories.
Docker does maintain an [official "Omnibus" container](https://docs.gitlab.com/omnibus/docker/README.html), but for this recipe I prefer the "[dockerized gitlab](https://github.com/sameersbn/docker-gitlab)" project, since it allows distribution of the various Gitlab components across multiple swarm nodes.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/gitlab:
```
cd /var/data
mkdir gitlab
cd gitlab
mkdir -p {postgresql,redis,gitlab}
```
### Prepare environment
You'll need to know the following:
1. Choose a password for postgresql, you'll need it for DB_PASS in the compose file (below)
2. Generate 3 passwords using ```pwgen -Bsv1 64```. You'll use these for the XXX_KEY_BASE environment variables below
2. Create gitlab.env, and populate with **at least** the following variables (the full set is available at https://github.com/sameersbn/docker-gitlab#available-configuration-parameters):
```
DB_USER=gitlab
DB_PASS=<as determined above>
TZ=Pacific/Auckland
GITLAB_TIMEZONE=Auckland
GITLAB_HTTPS=true
SSL_SELF_SIGNED=false
GITLAB_HOST
GITLAB_PORT
GITLAB_SSH_PORT
GITLAB_SECRETS_DB_KEY_BASE
GITLAB_SECRETS_SECRET_KEY_BASE
GITLAB_SECRETS_OTP_KEY_BASE
GITLAB_ROOT_PASSWORD
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
```
version: '3'
services:
redis:
image: sameersbn/redis:latest
command:
- --loglevel warning
volumes:
- /var/data/gitlab/redis:/var/lib/redis:Z
networks:
- internal
postgresql:
image: sameersbn/postgresql:9.6-2
volumes:
- /var/data/gitlab/postgresql:/var/lib/postgresql:Z
networks:
- internal
environment:
- DB_USER=gitlab
- DB_PASS=<your db password>
- DB_NAME=gitlabhq_production
- DB_EXTENSION=pg_trgm
gitlab:
image: sameersbn/gitlab:latest
networks:
- internal
- traefik
deploy:
labels:
- traefik.frontend.rule=Host:gitlab.example.com
- traefik.docker.network=traefik
- traefik.port=80
restart_policy:
delay: 10s
max_attempts: 10
window: 60s
ports:
- "10022:22"
volumes:
- /var/data/gitlab/gitlab:/home/git/data:Z
env_file: gitlab.env
networks:
traefik:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.2.0/24
-
```
!!! tip
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch gitlab
Launch the mail server stack by running ```docker stack deploy gitlab -c <path -to-docker-compose.yml>```
Log into your new instance at https://[your FQDN], with user "root" and the password you specified in gitlab.env.
## Chef's Notes
A few comments on decisions taken in this design:
1. I use the **sameersbn/gitlab:latest** image, rather than a specific version. This lets me execute updates simply by redeploying the stack (and why **wouldn't** I want the latest version?)

152
manuscript/recipies/mail.md Normal file
View File

@@ -0,0 +1,152 @@
# Mail Server
Many of the recipies that follow require email access of some kind. It's normally possible to use a hosted service such as SendGrid, or just a gmail account. If (like me) you'd like to self-host email for your stacks, then the following recipe provides a full-stack mail server running on the docker HA swarm.
Of value to me in choosing docker-mailserver were:
1. Automatically renews LetsEncrypt certificates
2. Creation of email accounts across multiple domains (i.e., the same container gives me mailbox wekan@wekan.example.com, and gitlab@gitlab.example.com)
3. The entire configuration is based on flat files, so there's no database or persistence to worry about
docker-mailserver doesn't include a webmail client, and one is not strictly needed. Rainloop can be added either as another service within the stack, or as a standalone service. Rainloop will be covered in a future recipe.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. LetsEncrypt authorized email address for domain
4. Access to manage DNS records for domains
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/mailserver:
```
cd /var/data
mkdir mailserver
cd mailserver
mkdir {maildata,mailstate,config,letsencrypt}
```
### Get LetsEncrypt certificate
Decide on the FQDN to assign to your mailserver. You can service multiple domains from a single mailserver - i.e., bob@dev.example.com and daphne@prod.example.com can both be served by **mail.example.com**.
The docker-mailserver container can _renew_ our LetsEncrypt certs for us, but it can't generate them. To do this, we need to run certbot (from a container) to request the initial certs and create the appropriate directory structure.
In the example below, since I'm already using Traefik to manage the LE certs for my web platforms, I opted to use the DNS challenge to prove my ownership of the domain. The certbot client will prompt you to add a DNS record for domain verification.
```
docker run -ti --rm -v \
"$(pwd)"/letsencrypt:/etc/letsencrypt certbot/certbot \
--manual --preferred-challenges dns certonly \
-d mail.example.com
```
### Get setup.sh
docker-mailserver comes with a handy bash script for managing the stack (which is just really a wrapper around the container.) It'll make our setup easier, so download it into the root of your configuration/data directory, and make it executable:
```
curl -o setup.sh \
https://raw.githubusercontent.com/tomav/docker-mailserver/master/setup.sh \
chmod a+x ./setup.sh
```
### Create email accounts
For every email address required, run ```./setup.sh email add <email> <password>``` to create the account. The command returns no output.
You can run ```./setup.sh email list``` to confirm all of your addresses have been created.
### Create DKIM DNS entries
Run ```./setup.sh config dkim``` to create the necessary DKIM entries. The command returns no output.
Examine the keys created by opendkim to identify the DNS TXT records required:
```
for i in `find config/opendkim/keys/ -name mail.txt`; do \
echo $i; \
cat $i; \
done
```
You'll end up with something like this:
```
config/opendkim/keys/gitlab.example.com/mail.txt
mail._domainkey IN TXT ( "v=DKIM1; k=rsa; "
"p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB" ) ; ----- DKIM key mail for gitlab.example.com
[root@ds1 mail]#
```
Create the necessary DNS TXT entries for your domain(s). Note that although opendkim splits the record across two lines, the actual record should be concatenated on creation. I.e., the DNS TXT record above should read:
```
"v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB"
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
```
version: '3'
services:
mail:
image: tvial/docker-mailserver:latest
ports:
- "25:25"
- "587:587"
- "993:993"
volumes:
- /var/data/mail/maildata:/var/mail
- /var/data/mail/mailstate:/var/mail-state
- /var/data/mail/config:/tmp/docker-mailserver
- /var/data/mail/letsencrypt:/etc/letsencrypt
env_file: /var/data/mail/.env
networks:
- internal
deploy:
replicas: 1
networks:
traefik:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.2.0/24
```
!!! tip
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot.
A sample .env file looks like this:
```
ENABLE_SPAMASSASSIN=1
ENABLE_CLAMAV=1
ENABLE_POSTGREY=1
ONE_DIR=1
OVERRIDE_HOSTNAME=mail.example.com
OVERRIDE_DOMAINNAME=mail.example.com
POSTMASTER_ADDRESS=admin@example.com
PERMIT_DOCKER=network
SSL_TYPE=letsencrypt
```
## Serving
### Launch mailserver
Launch the mail server stack by running ```docker stack deploy mailserver -c <path -to-docker-compose.yml>```
## Chef's Notes
1. One of the elements of this design which I didn't appreciate at first is that since the config is entirely file-based, **setup.sh** can be run on any container host, provided it has the shared data mounted. This means that even though docker-mailserver was not designed with docker swarm in mind, it works perfectl with swarm. I.e., from any node, regardless of where the container is actually running, you're able to add/delete email addresses, view logs, etc.

115
manuscript/recipies/wekan.md Normal file
View File

@@ -0,0 +1,115 @@
# Wekan
Wekan is an open-source kanban board which allows a card-based task and to-do management, similar to tools like WorkFlowy or Trello.
![Wekan Screenshot](../images/wekan.jpg)
Wekan allows to create Boards, on which Cards can be moved around between a number of Columns. Boards can have many members, allowing for easy collaboration, just add everyone that should be able to work with you on the board to it, and you are good to go! You can assign colored Labels to cards to facilitate grouping and filtering, additionally you can add members to a card, for example to assign a task to someone.
There's a [video](https://www.youtube.com/watch?v=N3iMLwCNOro) of the developer showing off the app, as well as a f[unctional demo](https://wekan.indie.host/b/t2YaGmyXgNkppcFBq/wekan-fork-roadmap).
!!! note
For added privacy, this design secures wekan behind an [oauth2 proxy](/reference/oauth_proxy/), so that in order to gain access to the wekan UI at all, oauth2 authentication (to GitHub, GitLab, Google, etc) must have already occured.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/wekan:
```
mkdir /var/data/wekan
cd /var/data/wekan
mkdir -p {wekan-db,wekan-db-dump}
```
### Prepare environment
You'll need to know the following:
1. Choose an oauth provider, and obtain a client ID and secret
2. Create wekan.env, and populate with the following variables
```
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
MONGO_URL=mongodb://wekandb:27017/wekan
ROOT_URL=https://wekan.example.com
MAIL_URL=smtp://wekan@wekan.example.com:password@mail.example.com:587/
MAIL_FROM="Wekan <wekan@wekan.example.com>"
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
```
version: '3'
services:
wekandb:
image: mongo:3.2.15
command: mongod --smallfiles --oplogSize 128
networks:
- internal
volumes:
- /var/data/wekan/wekan-db:/data/db
- /var/data/wekan/wekan-db-dump:/dump
proxy:
image: zappi/oauth2_proxy
env_file: /var/data/wekan/wekan.env
networks:
- traefik
- internal
deploy:
labels:
- traefik.frontend.rule=Host:wekan.example.com
- traefik.docker.network=traefik
- traefik.port=4180
command: |
-cookie-secure=false
-upstream=http://wekan:80
-redirect-url=https://wekan.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
wekan:
image: wekanteam/wekan:latest
networks:
- internal
env_file: /var/data/wekan/wekan.env
networks:
traefik:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.3.0/24
```
!!! tip
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Wekan stack
Launch the Wekan stack by running ```docker stack deploy wekan -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user "root" and the password you specified in gitlab.env.
## Chef's Notes
1. If you wanted to expose the Wekan UI directly, you could remove the oauth2_proxy from the design, and move the traefik-related labels directly to the wekan container. You'd also need to add the traefik network to the wekan container.

8
manuscript/recipies/workflow.md Normal file
View File

@@ -0,0 +1,8 @@
The workflow for creating a recipe
1. In my gitlab repo for the cookbook, I create an issue to track the new recipe I want to create
2. From the issue, I create a branch
3. I pull the branch locally, "git pull" followed by "git co <branch>"
4. I "git add" my changes, commit, and push the branch
5. I create a merge request in the branch to track merging into produced
6. I set request to merge when pipeline succeeds