public/geek-cookbook
1
0 Fork 0
mirror of https://github.com/funkypenguin/geek-cookbook/ synced 2026-10-27 03:03:22 +00:00
Code Issues Releases Wiki Activity

Correct spelling of recipes once and for all

Browse Source
This commit is contained in:
David Young committed 2018-11-22 22:22:17 +13:00
1 parent 09fb52f3d4
commit 1d64d39e01
90 files changed
+1593 -505
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

manuscript/recipes/autopirate.md
+132
View File
@@ -0,0 +1,132 @@
hero: AutoPirate - A fully-featured recipe to automate finding, downloading, and organising your media 📺 🎥 🎵 📖
# AutoPirate
Once the cutting edge of the "internet" (_pre-world-wide-web and mosiac days_), Usenet is now a murky, geeky alternative to torrents for file-sharing. However, it's **cool** geeky, especially if you're into having a fully automated media platform.
A good starter for the usenet scene is https://www.reddit.com/r/usenet/. Because it's so damn complicated, a host of automated tools exist to automate the process of finding, downloading, and managing content. The tools included in this recipe are as follows:
![Autopirate Screenshot](../images/autopirate.png)
This recipe presents a method to combine these tools into a single swarm deployment, and make them available securely.
## Menu
Tools included in the AutoPirate stack are:
* **[SABnzbd](http://sabnzbd.org)** : downloads data from usenet servers based on .nzb definitions
* **[NZBGet](https://nzbget.net/)** : downloads data from usenet servers based on .nzb definitions, but written in C++ and designed with performance in mind to achieve maximum download speed by using very little system resources (_this is a popular alternative to SABnzbd_)
* **[RTorrent](https://github.com/rakshasa/rtorrent/wiki)** is a CLI-based torrent client, which when combined with **[ruTorrent](https://github.com/Novik/ruTorrent)** becomes a powerful and fully browser-managed torrent client. (_Yes, it's not Usenet, but Sonarr/Radarr will let fulfill your watchlist using either Usenet **or** torrents, so it's worth including_)
* **[NZBHydra](https://github.com/theotherp/nzbhydra)** : acts as a "meta-indexer", so that your downloading tools (_radarr, sonarr, etc_) only need to be setup for a single indexes. Also produces interesting stats on indexers, which helps when evaluating which indexers are performing well.
* **[NZBHydra2](https://github.com/theotherp/nzbhydra2)** : is a high-performance rewrite of the original NZBHydra, with extra features. While still in beta, this NZBHydra2 will eventually supercede NZBHydra
* **[Sonarr](https://sonarr.tv)** : finds, downloads and manages TV shows
* **[Radarr](https://radarr.video)** : finds, downloads and manages movies
* **[Mylar](https://github.com/evilhero/mylar)** : finds, downloads and manages comic books
* **[Headphones](https://github.com/rembo10/headphones)** : finds, downloads and manages music
* **[Lazy Librarian](https://github.com/itsmegb/LazyLibrarian)** : finds, downloads and manages ebooks
* **[Ombi](https://github.com/tidusjar/Ombi)** : provides an interface to request additions to a [Plex](/recipes/plex/)/[Emby](/recipes/emby/) library using the above tools
* **[Jackett](https://github.com/Jackett/Jackett)** : Provides an local, caching, API-based interface to torrent trackers, simplifying the way your tools search for torrents.
Since this recipe is so long, and so many of the tools are optional to the final result (_i.e., if you're not interested in comics, you won't want Mylar_), I've described each individual tool on its own sub-recipe page (_below_), even though most of them are deployed very similarly.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. Access to NZB indexers and Usenet servers
4. DNS entries configured for each of the NZB tools in this recipe that you want to use
## Preparation
### Setup data locations
We'll need a unique directories for each tool in the stack, bind-mounted into our containers, so create them upfront, in /var/data/autopirate:
```
mkdir /var/data/autopirate
cd /var/data/autopirate
mkdir -p {lazylibrarian,mylar,ombi,sonarr,radarr,headphones,plexpy,nzbhydra,sabnzbd,nzbget,rtorrent,jackett}
```
Create a directory for the storage of your downloaded media, i.e., something like:
```
mkdir /var/data/media
```
Create a user to "own" the above directories, and note the uid and gid of the created user. You'll need to specify the UID/GID in the environment variables passed to the container (in the example below, I used 4242 - twice the meaning of life).
### Secure public access
What you'll quickly notice about this recipe is that __every__ web interface is protected by an [OAuth proxy](/reference/oauth_proxy/).
Why? Because these tools are developed by a handful of volunteer developers who are focused on adding features, not necessarily implementing robust security. Most users wouldn't expose these tools directly to the internet, so the tools have rudimentary (if any) access control.
To mitigate the risk associated with public exposure of these tools (_you're on your smartphone and you want to add a movie to your watchlist, what do you do, hotshot?_), in order to gain access to each tool you'll first need to authenticate against your given OAuth provider.
This is tedious, but you only have to do it once. Each tool (Sonarr, Radarr, etc) to be protected by an OAuth proxy, requires unique configuration. I use github to provide my oauth, giving each tool a unique logo while I'm at it (make up your own random string for OAUTH2PROXYCOOKIE_SECRET)
For each tool, create /var/data/autopirate/<tool>.env, and set the following:
```
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
PUID=4242
PGID=4242
```
Create at least /var/data/autopirate/authenticated-emails.txt, containing at least your own email address with your OAuth provider. If you wanted to grant access to a specific tool to other users, you'd need a unique authenticated-emails-<tool>.txt which included both normal email address as well as any addresses to be granted tool-specific access.
### Setup components
#### Stack basics
**Start** with a swarm config file in docker-compose syntax, like this:
````
version: '3'
services:
````
And **end** with a stanza like this:
````
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.11.0/24
````
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
#### Assemble the tools..
Now work your way through the list of tools below, adding whichever tools your want to use, and finishing with the **end** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](/recipes/autopirate/mylar/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [End](/recipes/autopirate/end/) (launch the stack)
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/end.md
+20
View File
@@ -0,0 +1,20 @@
!!! warning
This is not a complete recipe - it's the conclusion to the [AutoPirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
### Launch Autopirate stack
Launch the AutoPirate stack by running ```docker stack deploy autopirate -c <path -to-docker-compose.yml>```
Confirm the container status by running "docker stack ps autopirate", and wait for all containers to enter the "Running" state.
Log into each of your new tools at its respective HTTPS URL. You'll be prompted to authenticate against your OAuth provider, and upon success, redirected to the tool's UI.
## Chef's Notes 📓
1. This is a complex stack. Sing out in the comments if you found a flaw or need a hand :)
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/headphones.md
+81
View File
@@ -0,0 +1,81 @@
hero: AutoPirate - A fully-featured recipe to automate finding, downloading, and organising your media 📺 🎥 🎵 📖
!!! warning
This is not a complete recipe - it's a component of the [autopirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Headphones
[Headphones](https://github.com/rembo10/headphones) is an automated music downloader for NZB and Torrent, written in Python. It supports SABnzbd, NZBget, Transmission, µTorrent, Deluge and Blackhole.
![Headphones Screenshot](../../images/headphones.png)
## Inclusion into AutoPirate
To include Headphones in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
headphones:
image: linuxserver/headphones:latest
env_file : /var/data/config/autopirate/headphones.env
volumes:
- /var/data/autopirate/headphones:/config
- /var/data/media:/media
networks:
- internal
headphones_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/headphones.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:headphones.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://headphones:8181
-redirect-url=https://headphones.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](https://github.com/evilhero/mylar)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* Headphones (this page)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/heimdall.md
+88
View File
@@ -0,0 +1,88 @@
!!! warning
This is not a complete recipe - it's a component of the [autopirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Heimdall
[Heimdall Application Dashboard](https://heimdall.site/) is a dashboard for all your web applications. It doesn't need to be limited to applications though, you can add links to anything you like.
Heimdall is an elegant solution to organise all your web applications. It’s dedicated to this purpose so you won’t lose your links in a sea of bookmarks.
Heimdall provides a single URL to manage access to all of your autopirate tools, and includes "enhanced" (_i.e., display stats within Heimdall without launching the app_) access to [NZBGet](/recipes/autopirate/nzbget.md), [SABnzbd](/recipes/autopirate/sabnzbd/), and friends.
![Heimdall Screenshot](../../images/heimdall.jpg)
## Inclusion into AutoPirate
To include Heimdall in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
heimdall:
image: linuxserver/heimdall:latest
env_file: /var/data/config/autopirate/heimdall.env
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/data/heimdall:/config
networks:
- internal
heimdall_proxy:
image: funkypenguin/oauth2_proxy:latest
env_file : /var/data/config/autopirate/heimdall.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:heimdall.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://heimdall:80
-redirect-url=https://heimdall.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](/recipes/autopirate/mylarr/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* Heimdall (this page)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
2. The inclusion of Heimdall was due to the efforts of @gkoerk in our [Discord server](http://chat.funkypenguin.co.nz). Thanks gkoerk!
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/jackett.md
+81
View File
@@ -0,0 +1,81 @@
!!! warning
This is not a complete recipe - it's a component of the [autopirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Jackett
[Jackett](https://github.com/Jackett/Jackett) works as a proxy server: it translates queries from apps (Sonarr, Radarr, Mylar, etc) into tracker-site-specific http queries, parses the html response, then sends results back to the requesting software.
This allows for getting recent uploads (like RSS) and performing searches. Jackett is a single repository of maintained indexer scraping & translation logic - removing the burden from other apps.
![Jackett Screenshot](../../images/jackett.png)
## Inclusion into AutoPirate
To include Jackett in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
jackett:
image: linuxserver/jackett:latest
env_file : /var/data/config/autopirate/jackett.env
volumes:
- /var/data/autopirate/jackett:/config
networks:
- internal
jackett_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/jackett.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:jackett.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://jackett:9117
-redirect-url=https://jackett.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](/recipes/autopirate/mylarr/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* Jackett (this page)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/lazylibrarian.md
+94
View File
@@ -0,0 +1,94 @@
!!! warning
This is not a complete recipe - it's a component of the [autopirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# LazyLibrarian
[LazyLibrarian](https://github.com/DobyTang/LazyLibrarian) is a tool to follow authors and grab metadata for all your digital reading needs. It uses a combination of Goodreads Librarything and optionally GoogleBooks as sources for author info and book info. Features include:
* Find authors and add them to the database
* List all books of an author and mark ebooks or audiobooks as 'wanted'.
* When processing the downloaded books it will save a cover picture (if available) and save all metadata into metadata.opf next to the bookfile (calibre compatible format)
* AutoAdd feature for book management tools like Calibre which must have books in flattened directory structure, or use calibre to import your books into an existing calibre library
* LazyLibrarian can also be used to search for and download magazines, and monitor for new issues
![Lazy Librarian Screenshot](../../images/lazylibrarian.png)
## Inclusion into AutoPirate
To include LazyLibrarian in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
lazylibrarian:
image: linuxserver/lazylibrarian:latest
env_file : /var/data/config/autopirate/lazylibrarian.env
volumes:
- /var/data/autopirate/lazylibrarian:/config
- /var/data/media:/media
networks:
- internal
lazylibrarian_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/lazylibrarian.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:lazylibrarian.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://lazylibrarian:5299
-redirect-url=https://lazylibrarian.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
calibre-server:
image: regueiro/calibre-server
volumes:
- /var/data/media/Ebooks/calibre/:/opt/calibre/library
networks:
- internal
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](https://github.com/evilhero/mylar)
* Lazy Librarian (this page)
* [Headphones](/recipes/autopirate/headphones)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. The calibre-server container co-exists within the Lazy Librarian (LL) containers so that LL can automatically add a book to Calibre using the calibre-server interface. The calibre library can then be properly viewed using the [calibre-web](/recipes/calibre-web) recipe.
2. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/lidarr.md
+83
View File
@@ -0,0 +1,83 @@
hero: AutoPirate - A fully-featured recipe to automate finding, downloading, and organising your media 📺 🎥 🎵 📖
!!! warning
This is not a complete recipe - it's a component of the [autopirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Lidarr
[Lidarr](https://lidarr.audio/) is an automated music downloader for NZB and Torrent. It performs the same function as [Headphones](/recipes/autopirate/headphones), but is written using the same(ish) codebase as [Radarr](/recipes/autopirate/radarr/) and [Sonarr](/recipes/autopirate/sonarr). It's blazingly fast, and includes beautiful album/artist art. Lidarr supports [SABnzbd](/recipes/autopirate/sabnzbd/), [NZBGet](/recipes/autopirate/nzbget/), Transmission, µTorrent, Deluge and Blackhole (_just like Sonarr / Radarr_)
![Lidarr Screenshot](../../images/lidarr.png)
## Inclusion into AutoPirate
To include Lidarr in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
lidarr:
image: linuxserver/lidarr:latest
env_file : /var/data/config/autopirate/lidarr.env
volumes:
- /var/data/autopirate/lidarr:/config
- /var/data/media:/media
networks:
- internal
lidarr_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/lidarr.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:lidarr.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://lidarr:8181
-redirect-url=https://lidarr.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](https://github.com/evilhero/mylar)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones/)
* Lidarr (this page)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
2. The addition of the Lidarr recipe was contributed by our very own @gpulido in Discord (http://chat.funkypenguin.co.nz) - Thanks Gabriel!
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/mylar.md
+79
View File
@@ -0,0 +1,79 @@
!!! warning
This is not a complete recipe - it's a component of the [autopirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Mylar
[Mylar](https://github.com/evilhero/mylar) is a tool for downloading and managing digital comic books.
![Mylar Screenshot](../../images/mylar.jpg)
## Inclusion into AutoPirate
To include Mylar in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
mylar:
image: linuxserver/mylar:latest
env_file : /var/data/config/autopirate/mylar.env
volumes:
- /var/data/autopirate/mylar:/config
- /var/data/media:/media
networks:
- internal
mylar_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/mylar.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:mylar.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://mylar:8090
-redirect-url=https://mylar.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* Mylar (this page)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/nzbget.md
+86
View File
@@ -0,0 +1,86 @@
!!! warning
This is not a complete recipe - it's a component of the [AutoPirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# NZBGet
## Introduction
NZBGet performs the same function as [SABnzbd](/recipes/autopirate/sabnzbd.md) (_downloading content from Usenet servers_), but it's lightweight and fast(er), written in C++ (_as opposed to Python_).
![NZBGet Screenshot](../../images/nzbget.jpg)
## Inclusion into AutoPirate
To include NZBGet in your [AutoPirate](/recipes/autopirate/) stack
(_The only reason you **wouldn't** use NZBGet, would be if you were using [SABnzbd](/recipes/autopirate/sabnzbd/) instead_), include the following in your autopirate.yml stack definition file:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
````
nzbget:
image: linuxserver/nzbget
env_file : /var/data/config/autopirate/nzbget.env
volumes:
- /var/data/autopirate/nzbget:/config
- /var/data/media:/data
networks:
- internal
nzbget_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/nzbget.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:nzbget.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://nzbget:6789
-redirect-url=https://nzbget.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! note
NZBGet uses a 401 header to prompt for authentication. When you use OAuth2_proxy, this seems to break. Since we trust OAuth to authenticate us, we can just disable NZGet's own authentication, by changing ControlPassword to null in nzbget.conf (i.e. ```ControlPassword=```)
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* NZBGet (this page)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](/recipes/autopirate/mylar/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones/)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/nzbhydra.md
+85
View File
@@ -0,0 +1,85 @@
!!! warning
This is not a complete recipe - it's a component of the [AutoPirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# NZBHydra
[NZBHydra](https://github.com/theotherp/nzbhydra) is a meta search for NZB indexers. It provides easy access to a number of raw and newznab based indexers. You can search all your indexers from one place and use it as indexer source for tools like Sonarr or CouchPotato. Features include:
* Search by IMDB, TMDB, TVDB, TVRage and TVMaze ID (including season and episode) and filter by age and size. If an ID is not supported by an indexer it is attempted to be converted (e.g. TMDB to IMDB)
* Query generation, meaning when you search for a movie using e.g. an IMDB ID a query will be generated for raw indexers. Searching for a series season 1 episode 2 will also generate queries for raw indexers, like s01e02 and 1x02
* Grouping of results with the same title and of duplicate results, accounting for result posting time, size, group and poster. By default only one of the duplicates is shown. You can provide an indexer score to influence which one that might be
* Compatible with Sonarr, CP, NZB 360, SickBeard, Mylar and Lazy Librarian (and others)
* Statistics on indexers (average response time, share of results, access errors), searches and downloads per time of day and day of week, NZB download history and search history (both via internal GUI and API)
![NZBHydra Screenshot](../../images/nzbhydra.png)
## Inclusion into AutoPirate
To include NZBHydra in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
nzbhydra:
image: linuxserver/hydra:latest
env_file : /var/data/config/autopirate/nzbhydra.env
volumes:
- /var/data/autopirate/nzbhydra:/config
networks:
- internal
nzbhydra_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/nzbhydra.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:nzbhydra.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://nzbhydra:5075
-redirect-url=https://nzbhydra.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](/recipes/autopirate/mylar/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones/)
* [Lidarr](/recipes/autopirate/lidarr/)
* NZBHydra (this page)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/nzbhydra2.md
+101
View File
@@ -0,0 +1,101 @@
!!! warning
This is not a complete recipe - it's a component of the [AutoPirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# NZBHydra 2
[NZBHydra 2](https://github.com/theotherp/nzbhydra2) is a meta search for NZB indexers. It provides easy access to a number of raw and newznab based indexers. You can search all your indexers from one place and use it as an indexer source for tools like Sonarr, Radarr or CouchPotato.
!!! note
NZBHydra 2 is a complete rewrite of [NZBHydra (1)](/recipes/autopirate/nzbhybra/). It's currently in Beta. It works mostly fine but some functions might not be completely done and incompatibilities with some tools might still exist. You might want to run both in parallel for migration / testing purposes, but ultimately you'll probably want to switch over to NZBHydra 2 exclusively.
![NZBHydra Screenshot](../../images/nzbhydra2.png)
Features include:
* Searches Anizb, BinSearch, NZBIndex and any newznab compatible indexers. Merges all results, filters them by a number of configurable restrictions, recognizes duplicates and returns them all in one place
* Add results to [NZBGet](/recipes/autopirate/nzbget/) or [SABnzbd](/recipes/autopirate/sabnzbd/)
* Support for all relevant media IDs (IMDB, TMDB, TVDB, TVRage, TVMaze) and conversion between them
* Query generation, meaning a query will be generated if only a media ID is provided in the search and the indexer doesn't support the ID or if no results were found
* Compatible with [Sonarr](/recipes/autopirate/sonarr/), [Radarr](/recipes/autopirate/radarr/), [NZBGet](/recipes/autopirate/nzbget.md), [SABnzbd](/recipes/autopirate/sabnzbd/), nzb360, CouchPotato, [Mylar](/recipes/autopirate/mylar/), [Lazy Librarian](/recipes/autopirate/lazylibrarian/), Sick Beard, [Jackett/Cardigann](/recipes/autopirate/jackett/), Watcher, etc.
* Search and download history and extensive stats. E.g. indexer response times, download shares, NZB age, etc.
* Authentication and multi-user support
* Automatic update of NZB download status by querying configured downloaders
* RSS support with configurable cache times
* Torrent support (_Although I prefer [Jackett](/recipes/autopirate/jackett/) for this_):
* For GUI searches, allowing you to download torrents to a blackhole folder
* A separate Torznab compatible endpoint for API requests, allowing you to merge multiple trackers
* Extensive configurability
* Migration of database and settings from v1
## Inclusion into AutoPirate
To include NZBHydra2 in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
nzbhydra2:
image: linuxserver/hydra2:latest
env_file : /var/data/config/autopirate/nzbhydra2.env
volumes:
- /var/data/autopirate/nzbhydra2:/config
networks:
- internal
nzbhydra2_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/nzbhydra2.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:nzbhydra2.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://nzbhydra2:5076
-redirect-url=https://nzbhydra2.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](/recipes/autopirate/mylar/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones/)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* NZBHydra2 (this page)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra2, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
2. Note that NZBHydra2 _can_ co-exist with NZBHydra (1), but if you want your tools (Sonarr, Radarr, etc) to use NZBHydra2, you'll need to change both the target hostname (_to "hydra2"_) and the target port (_to 5076_).
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/ombi.md
+86
View File
@@ -0,0 +1,86 @@
!!! warning
This is not a complete recipe - it's a component of the [AutoPirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Ombi
[Ombi](https://github.com/tidusjar/Ombi) is a useful addition to the [autopirate](/recipes/autopirate/) stack. Features include:
* Lets users request Movies and TV Shows (_whether it being the entire series, an entire season, or even single episodes._)
* Easily manage your requests
User management system (_supports plex.tv, Emby and local accounts_)
* A landing page that will give you the availability of your Plex/Emby server and also add custom notification text to inform your users of downtime.
* Allows your users to get custom notifications!
* Will show if the request is already on plex or even if it's already monitored.
Automatically updates the status of requests when they are available on Plex/Emby
![Ombi Screenshot](../../images/ombi.png)
## Inclusion into AutoPirate
To include Ombi in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
ombi:
image: linuxserver/ombi:latest
env_file : /var/data/config/autopirate/ombi.env
volumes:
- /var/data/autopirate/ombi:/config
networks:
- internal
ombi_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/ombi.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:ombi.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://ombi:3579
-redirect-url=https://ombi.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](/recipes/autopirate/mylar/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones/)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* Ombi (this page)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/radarr.md
+97
View File
@@ -0,0 +1,97 @@
!!! warning
This is not a complete recipe - it's a component of the [AutoPirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Radarr
[Radarr](https://radarr.video/) is a tool for finding, downloading and managing movies. Features include:
* Adding new movies with lots of information, such as trailers, ratings, etc.
* Can watch for better quality of the movies you have and do an automatic upgrade. eg. from DVD to Blu-Ray
* Automatic failed download handling will try another release if one fails
* Manual search so you can pick any release or to see why a release was not downloaded automatically
* Full integration with SABnzbd and NZBGet
* Automatically searching for releases as well as RSS Sync
* Automatically importing downloaded movies
* Recognizing Special Editions, Director's Cut, etc.
* Identifying releases with hardcoded subs
* Importing movies from various online sources, such as IMDb Watchlists (A complete list can be found here)
* Full integration with Kodi, Plex (notification, library update)
* And a beautiful UI
* Importing Metadata such as trailers or subtitles
![Radarr Screenshot](../../images/radarr.png)
!!! tip "Sponsored Project"
Sonarr is one of my [sponsored projects](/sponsored-projects/) - a project I financially support on a regular basis because of its utility to me. I forget it's there until I (reliably) receive an email with new and exciting updates 😁
## Inclusion into AutoPirate
To include Radarr in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
radarr:
image: linuxserver/radarr:latest
env_file : /var/data/config/autopirate/radarr.env
volumes:
- /var/data/autopirate/radarr:/config
- /var/data/media:/media
networks:
- internal
radarr_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/radarr.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:radarr.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://radarr:7878
-redirect-url=https://radarr.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* Radarr (this page)
* [Mylar](/recipes/autopirate/mylar/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones/)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/rtorrent.md
+86
View File
@@ -0,0 +1,86 @@
!!! warning
This is not a complete recipe - it's a component of the [AutoPirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# RTorrent / ruTorrent
[RTorrent](http://rakshasa.github.io/rtorrent) is a popular CLI-based bittorrent client, and [ruTorrent](https://github.com/Novik/ruTorrent) is a powerful web interface for rtorrent.
![Rtorrent Screenshot](../../images/rtorrent.png)
## Choose incoming port
When using a torrent client from behind NAT (_which swarm, by nature, is_), you typically need to set a static port for inbound torrent communications. In the example below, I've set the port to 36258. You'll need to configure /var/data/autopirate/rtorrent/rtorrent/rtorrent.rc with the equivalent port.
## Inclusion into AutoPirate
To include ruTorrent in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
```
rtorrent:
image: linuxserver/rutorrent
env_file : /var/data/config/autopirate/rtorrent.env
ports:
- 36258:36258
volumes:
- /var/data/media/:/media
- /var/data/autopirate/rtorrent:/config
networks:
- internal
rtorrent_proxy:
image: skippy/oauth2_proxy
env_file : /var/data/config/autopirate/rtorrent.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:rtorrent.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://rtorrent:80
-redirect-url=https://rtorrent.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
```
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* RTorrent (this page)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](/recipes/autopirate/mylar/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones/)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/sabnzbd.md
+93
View File
@@ -0,0 +1,93 @@
!!! warning
This is not a complete recipe - it's a component of the [AutoPirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# SABnzbd
## Introduction
SABnzbd is the workhorse of the stack. It takes .nzb files as input (_manually or from other [autopirate](/recipes/autopirate/) stack tools_), then connects to your chosen Usenet provider, downloads all the individual binaries referenced by the .nzb, and then tests/repairs/combines/uncompresses them all into the final result - media files.
![SABNZBD Screenshot](../../images/sabnzbd.png)
!!! tip "Sponsored Project"
SABnzbd is one of my [sponsored projects](/sponsored-projects/) - a project I financially support on a regular basis because of its utility to me. It's not sexy, but it's consistent and reliable, and I enjoy the fruits of its labor near-daily.
## Inclusion into AutoPirate
To include SABnzbd in your [AutoPirate](/recipes/autopirate/) stack
(_The only reason you **wouldn't** use SABnzbd, would be if you were using [NZBGet](/recipes/autopirate/nzbget.md) instead_), include the following in your autopirate.yml stack definition file:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
````
sabnzbd:
image: linuxserver/sabnzbd:latest
env_file : /var/data/config/autopirate/sabnzbd.env
volumes:
- /var/data/autopirate/sabnzbd:/config
- /var/data/media:/media
networks:
- internal
sabnzbd_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/sabnzbd.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:sabnzbd.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://sabnzbd:8080
-redirect-url=https://sabnzbd.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! warning "Important Note re hostname validation"
(**Updated 10 June 2018**) : In SABnzbd [2.3.3](https://sabnzbd.org/wiki/extra/hostname-check.html), hostname verification was added as a mandatory check. SABnzbd will refuse inbound connections which weren't addressed to its own (_initially, autodetected_) hostname. This presents a problem within Docker Swarm, where container hostnames are random and disposable.
You'll need to edit sabnzbd.ini (_only created after your first launch_), and **replace** the value in ```host_whitelist``` configuration (_it's comma-separated_) with the name of your service within the swarm definition, as well as your FQDN as accessed via traefik.
For example, mine simply reads ```host_whitelist = sabnzbd.funkypenguin.co.nz, sabnzbd```
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* SABnzbd (this page)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* [Sonarr](/recipes/autopirate/sonarr/)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](/recipes/autopirate/mylar/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones/)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/autopirate/sonarr.md
+83
View File
@@ -0,0 +1,83 @@
!!! warning
This is not a complete recipe - it's a component of the [AutoPirate](/recipes/autopirate/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Sonarr
[Sonarr](https://sonarr.tv/) is a tool for finding, downloading and managing your TV series.
![Sonarr Screenshot](../../images/sonarr.png)
!!! tip "Sponsored Project"
Sonarr is one of my [sponsored projects](/sponsored-projects/) - a project I financially support on a regular basis because of its utility to me. I forget it's there until I (reliably) receive an email with new and exciting updates 😁
## Inclusion into AutoPirate
To include Sonarr in your [AutoPirate](/recipes/autopirate/) stack, include the following in your autopirate.yml stack definition file:
````
sonarr:
image: linuxserver/sonarr:latest
env_file : /var/data/config/autopirate/sonarr.env
volumes:
- /var/data/autopirate/sonarr:/config
- /var/data/media:/media
networks:
- internal
sonarr_proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/autopirate/sonarr.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:sonarr.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/autopirate/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://sonarr:8989
-redirect-url=https://sonarr.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
````
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
## Assemble more tools..
Continue through the list of tools below, adding whichever tools your want to use, and finishing with the **[end](/recipes/autopirate/end/)** section:
* [SABnzbd](/recipes/autopirate/sabnzbd.md)
* [NZBGet](/recipes/autopirate/nzbget.md)
* [RTorrent](/recipes/autopirate/rtorrent/)
* Sonarr (this page)
* [Radarr](/recipes/autopirate/radarr/)
* [Mylar](/recipes/autopirate/mylar/)
* [Lazy Librarian](/recipes/autopirate/lazylibrarian/)
* [Headphones](/recipes/autopirate/headphones/)
* [Lidarr](/recipes/autopirate/lidarr/)
* [NZBHydra](/recipes/autopirate/nzbhydra/)
* [NZBHydra2](/recipes/autopirate/nzbhydra2/)
* [Ombi](/recipes/autopirate/ombi/)
* [Jackett](/recipes/autopirate/jackett/)
* [Heimdall](/recipes/autopirate/heimdall/)
* [End](/recipes/autopirate/end/) (launch the stack)
## Chef's Notes 📓
1. In many cases, tools will integrate with each other. I.e., Radarr needs to talk to SABnzbd and NZBHydra, Ombi needs to talk to Radarr, etc. Since each tool runs within the stack under its own name, just refer to each tool by name (i.e. "radarr"), and docker swarm will resolve the name to the appropriate container. You can identify the tool-specific port by looking at the docker-compose service definition.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/bookstack.md
+152
View File
@@ -0,0 +1,152 @@
hero: Heroic Hero
# BookStack
BookStack is a simple, self-hosted, easy-to-use platform for organising and storing information.
A friendly middle ground between heavyweights like MediaWiki or Confluence and [Gollum](/recipes/gollum/), BookStack relies on a database backend (so searching and versioning is easy), but limits itself to a pre-defined, 3-tier structure (book, chapter, page). The result is a lightweight, approachable personal documentation stack, which includes search and Markdown editing.
![BookStack Screenshot](../images/bookstack.png)
I like to protect my public-facing web UIs with an [oauth_proxy](/reference/oauth_proxy), ensuring that if an application bug (or a user misconfiguration) exposes the app to unplanned public scrutiny, I have a second layer of defense.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik_public) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/bookstack:
```
mkdir -p /var/data/bookstack/database-dump
mkdir -p /var/data/runtime/bookstack/db
```
### Prepare environment
Create bookstack.env, and populate with the following variables. Set the [oauth_proxy](/reference/oauth_proxy) variables provided by your OAuth provider (if applicable.)
```
# For oauth-proxy (optional)
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
# For MariaDB/MySQL database
MYSQL_RANDOM_ROOT_PASSWORD=true
MYSQL_DATABASE=bookstack
MYSQL_USER=bookstack
MYSQL_PASSWORD=secret
# Bookstack-specific variables
DB_HOST=bookstack_db:3306
DB_DATABASE=bookstack
DB_USERNAME=bookstack
DB_PASSWORD=secret
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
db:
image: mariadb:10
env_file: /var/data/config/bookstack/bookstack.env
networks:
- internal
volumes:
- /var/data/runtime/bookstack/db:/var/lib/mysql
proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/bookstack/bookstack.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:bookstack.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/bookstack/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://app
-redirect-url=https://bookstack.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
app:
image: solidnerd/bookstack
env_file: /var/data/config/bookstack/bookstack.env
networks:
- internal
db-backup:
image: mariadb:10
env_file: /var/data/config/bookstack/bookstack.env
volumes:
- /var/data/bookstack/database-dump:/dump
- /etc/localtime:/etc/localtime:ro
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
sleep 2m
while /bin/true; do
mysqldump -h db --all-databases | gzip -c > /dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.sql.gz
(ls -t /dump/dump*.sql.gz|head -n $$BACKUP_NUM_KEEP;ls /dump/dump*.sql.gz)|sort|uniq -u|xargs rm -- {}
sleep $$BACKUP_FREQUENCY
done
EOF'
networks:
- internal
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.33.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Bookstack stack
Launch the BookStack stack by running ```docker stack deploy bookstack -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, authenticate with oauth_proxy, and then login with username 'admin@admin.com' and password 'password'.
## Chef's Notes
1. If you wanted to expose the BookStack UI directly, you could remove the oauth2_proxy from the design, and move the traefik_public-related labels directly to the bookstack container. You'd also need to add the traefik_public network to the bookstack container.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/calibre-web.md
+134
View File
@@ -0,0 +1,134 @@
hero: Manage your ebook collection. Like a BOSS.
# Calibre-Web
The [AutoPirate](/recipes/autopirate/) recipe includes [Lazy Librarian](https://github.com/itsmegb/LazyLibrarian), a tool for tracking, finding, and downloading eBooks. However, after the eBooks are downloaded, Lazy Librarian is not much use for organising, tracking, and actually **reading** them.
[Calibre-Web](https://github.com/janeczku/calibre-web) could be described as "_[Plex](/recipes/plex/) (or [Emby](/recipes/emby/)) for eBooks_" - it's a web-based interface to manage your eBook library, screenshot below:
![Calibre-Web Screenshot](../images/calibre-web.png)
Of course, you probably already manage your eBooks using the excellent [Calibre](https://calibre-ebook.com/), but this is primarily a (_powerful_) desktop application. Calibre-Web is an alternative way to manage / view your existing Calibre database, meaning you can continue to use Calibre on your desktop if you wish.
As a long-time Kindle user, Calibre-Web brings (among [others](https://github.com/janeczku/calibre-web)) the following features which appeal to me:
* Filter and search by titles, authors, tags, **series** and language
* Create custom book collection (shelves)
Support for editing eBook metadata and deleting eBooks from Calibre library
* Support for converting eBooks from EPUB to Kindle format (mobi/azw)
* Send eBooks to Kindle devices with the click of a button
* Support for reading eBooks directly in the browser (.txt, .epub, .pdf, .cbr, .cbt, .cbz)
* Upload new books in PDF, epub, fb2 format
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need a directory to store some config data for Calibre-Web, container, so create /var/data/calibre-web, and ensure the directory is owned by the same use which owns your Calibre data (below)
```
mkdir /var/data/calibre-web
chown calibre:calibre /var/data/calibre-web # for example
```
Ensure that your Calibre library is accessible to the swarm (_i.e., exists on shared storage_), and that the same user who owns the config directory above, also owns the actual calibre library data (_including the ebooks managed by Calibre_).
### Prepare environment
We'll use an [oauth-proxy](/reference/oauth_proxy/) to protect the UI from public access, so create calibre-web.env, and populate with the following variables:
```
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=<make this a random string>
PUID=
PGID=
```
Follow the [instructions](https://github.com/bitly/oauth2_proxy) to setup your oauth provider. You need to setup a unique key/secret for each instance of the proxy you want to run, since in each case the callback URL will differ.
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
app:
image: technosoft2000/calibre-web
env_file : /var/data/config/calibre-web/calibre-web.env
volumes:
- /var/data/calibre-web:/config
- /srv/data/Archive/Ebooks/calibre:/books
networks:
- internal
proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/calibre-web/calibre-web.env
dns_search: hq.example.com
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:calibre-web.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/calibre-web/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://app:8083
-redirect-url=https://calibre-web.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.18.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Calibre-Web
Launch the Calibre-Web stack by running ```docker stack deploy calibre-web -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**. You'll be directed to the initial GUI configuraition. Set the first field (_Location of Calibre database_) to "_/books/_", and when complete, login using defaults username of "**admin**" with password "**admin123**".
## Chef's Notes
1. Yes, Calibre does provide a server component. But it's not as fully-featured as Calibre-Web (_i.e., you can't use it to send ebooks directly to your Kindle_)
2. A future enhancement might be integrating this recipe with the filestore for [NextCloud](/recipes/nextcloud/), so that the desktop database (Calibre) can be kept synced with Calibre-Web.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptominer.md
+42
View File
@@ -0,0 +1,42 @@
hero: We dig dig digga-dig dig!
# CryptoMiner
This is a diversion from my usual recipes - recently I've become interested in cryptocurrency, both in mining, and in investing.
I honestly didn't expect to enjoy the mining process as much as I did. Part of the enjoyment was getting my hands dirty with hardware.
Since a [mining rig](/recipes/cryptominer/mining-rig/) relies on hardware, we can't really use a docker swarm for this one!
![NAME Screenshot](../images/cryptominer.png)
This recipe isn't for everyone - if you just want to make some money from cryptocurrency, then you're better off learning to [invest](https://www.reddit.com/r/CryptoCurrency/) or [trade](https://www.reddit.com/r/CryptoMarkets/). However, if you want to (_ideally_) make money **and** you like tinkering, playing with hardware, optimising and monitoring, read on!
## Ingredients
1. Suitable system guts (_CPU, motherboard, RAM, PSU_) for your [mining rig](/recipes/cryptominer/mining-rig/)
2. [AMD](/recipes/cryptominer/amd-gpu/) / [Nvidia](/recipes/cryptominer/nvidia-gpu/) GPUs (_yes, plural, since although you **can** start with just one, you'll soon get hooked!_)
3. A friendly operating system ([Ubuntu](https://www.ubuntu.com/)/[Debian](https://www.debian.org/)/[CentOS7](https://www.centos.org/download/)) are known to work
4. Patience and time
## Preparation
For readability, I've split this recipe into multiple sub-recipes, which can be found below, or in the navigation links on the right-hand side:
1. Build your [mining rig](/recipes/cryptominer/mining-rig/) 💻
2. Setup your [AMD](/recipes/cryptominer/amd-gpu/) or [Nvidia](/recipes/cryptominer/nvidia-gpu/) GPUs 🎨
3. Sign up for [mining pools](/recipes/cryptominer/mining-pool/) :swimmer:
4. Setup your miners with [Miner Hotel](/recipes/cryptominer/minerhotel/) 🏨
5. Send your coins to [exchanges](/recipes/cryptominer/exchange/) or [wallets](/recipes/cryptominer/wallet/) 💹
6. [Monitor](/recipes/cryptominer/monitor/) your empire :heartbeat:
7. [Profit](/recipes/cryptominer/profit/)! 💰
## Chef's Notes
1. Ultimately I hope to move all the configuration / mining executables into docker containers, but for now, they're running on a CentOS7 host for direct access to GPUs. (Apparently it _may_ be possible to pass-thru the GPUs to docker containers, but I wanted stability first, before abstracting my hardware away from my miners)
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptominer/amd-gpu.md
+169
View File
@@ -0,0 +1,169 @@
!!! warning
This is not a complete recipe - it's a component of the [cryptominer](/recipes/cryptominer/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# AMD GPU
## Ingredients
1. [AMD drivers](http://support.amd.com/en-us/kb-articles/Pages/Radeon-Software-for-Linux-Release-Notes.aspx) for your GPU
2. [Linux version](https://bitcointalk.org/index.php?topic=1809527.0) of "atiflash" command
3. A [VBIOS rom](https://anorak.tech/c/downloads) compatible with your GPU model and memory manufacturer
## Preparation
### Install the drivers
There are links on the AMD driver download page (_linked above_) to drivers for RHEL/CentOS6, RHEL/CentOS7, and Ubuntu 16.04. As I write this, the latest version is **amdgpu-pro-17.50-511655**.
!!! note
You'll find reference online to the "blockchain" drivers. These were an earlier, [beta release](http://support.amd.com/en-us/kb-articles/Pages/AMDGPU-Pro-Beta-Mining-Driver-for-Linux-Release-Notes.aspx) which have been superseded by version 17.50 and later. You can ignore these.
Uncompress the drivers package, and run the following:
```./amdgpu-install --opencl=legacy --headless```
If you have a newer (_than my 5-year-old one!_) motherboard/CPU, you can also try the following, for ROCm support (_which might allow you some more software-based overclocking powers_):
```./amdgpu-install --opencl=legacy,rocm --headless```
Reboot upon completion.
### Flash the BIOS
Yes, this sounds scary, but it's not as bad as it sounds, if you want better performance from your GPUs, you **have** to flash your GPU BIOS.
#### Why flash BIOS?
Here's my noob-level version of why:
1. GPU-mining performance is all about the **memory speed** of your GPU - you get the best mining from the fastest internal timings. So you want to optimize your GPU to do really fast memory work, which is not how it's designed by default.
2. The **processor** on your GPU sits almost idle, so you **lower** the power to the processor (_undervolt_) to save some power.
3. As it turns out, the factory memory timings of the RX5xx series were particularly poor.
As an aside, here's an illustration re why you'd **want** to flash your BIOS. Below is the mining throughput of 2 AMD RX580s I purchased together. Guess which one had its BIOS flashed?
```
ETH: GPU0 30.115 Mh/s, GPU1 22.176 Mh/s
```
Here's the power consumption of the two GPUs while doing the above test:
GPU1 (original ROM)
```
GFX Clocks and Power:
1750 MHz (MCLK)
1411 MHz (SCLK)
144.107 W (VDDC)
16.0 W (VDDCI)
171.161 W (max GPU)
172.209 W (average GPU)
GPU Temperature: 67 C
GPU Load: 100 %
```
GPU0 (flashed ROM)
```
GFX Clocks and Power:
2050 MHz (MCLK)
1150 MHz (SCLK)
87.155 W (VDDC)
16.0 W (VDDCI)
117.152 W (max GPU)
116.1 W (average GPU)
GPU Temperature: 62 C
GPU Load: 100 %
```
So, by flashing the BIOS, I gained 8 MH/s (a 36% increase), while reducing power consumption by ~40W!
#### How to flash AMD GPU BIOS?
1. Get [atiflash for linux](https://bitcointalk.org/index.php?topic=1809527.0).
2. Identify which card you want to flash, by running ```./atiflash -i```
Example output below:
```
[root@kvm ~]# ./atiflash -i
adapter bn dn dID asic flash romsize test bios p/n
======= == == ==== =============== ============== ======= ==== ================
0 01 00 67DF Ellesmere M25P20/c 40000 pass 113-1E3660EU-O55
[root@kvm ~]#
```
3. Save the original, factory ROM, by running ```./atiflash -s <adapter number> <filename to save>```
Example below:
```
[root@kvm ~]# ./atiflash -s 0 rx580-4gb-299-1E366-101SA.orig.rom
0x40000 bytes saved, checksum = 0x7FBF
```
Now find an appropriate ROM to flash onto the card, and run ```atiflash -p <adatper number> <rom filename>
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes a range of RX580-compatible ROMs, some of which I've tweaked for my own GPUs. 👍
Example below:
```
[root@kvm ~]# ./atiflash -f -p 0 Insan1ty\ R9\ 390X\ BIOS\ v1.81/R9\ 290X/MEM\ MOD\ --\ ELPIDA/290X_ELPIDA_MOD_V1.8.rom
Old SSID: E285
New SSID: 9395
Old P/N: 113-E285FOC-U005
New P/N: 113-GRENADA_XT_C671_D5_8GB_HY_W
Old DeviceID: 67B1
New DeviceID: 67B0
Old Product Name: C67111 Hawaii PRO OC GDDR5 4GB 64Mx32 300e/150m
New Product Name: C67130 Grenada XT A0 GDDR5 8GB 128Mx32 300e/150m
Old BIOS Version: 015.044.000.011.000000
New BIOS Version: 015.049.000.000.000000
Flash type: M25P10/c
Burst size is 256
20000/20000h bytes programmed
20000/20000h bytes verified
Restart System To Complete VBIOS Update.
[root@kvm ~]#
```
Reboot the system, [hold onto your butts](https://www.youtube.com/watch?v=o0YWRXJsMyM), and wait for your newly-flashed GPU to fire up.
#### If it goes wrong
The safest way to do this is to run more than one GPU, and to flash the GPUs one-at-a-time, rebooting after each. That way, even if you make your GPU totally unresponsive, you'll still get access to your system to flash it back to the factory ROM.
That said, it's very unlikely that a flashed GPU won't let you boot at all though. In the (legion) cases where I overclocked my RX580 too far, I was able choose to boot into rescue mode in CentOS7 (bypassing the framebuffer / drm initialisation), and reflash my card back to its original BIOS.
#### Mooar tweaking! 🔧
If you want to tweak the BIOS yourself, download the [Polaris bios editor](https://github.com/jaschaknack/PolarisBiosEditor) and tweak away!
## Continue your adventure
Now, continue to the next stage of your grand mining adventure:
1. Build your [mining rig](/recipes/cryptominer/mining-rig/) 💻
2. Setup your AMD (_this page_) or [Nvidia](/recipes/cryptominer/nvidia-gpu/) GPUs 🎨
3. Sign up for [mining pools](/recipes/cryptominer/mining-pool/) :swimmer:
3. Setup your miners with [Miner Hotel](/recipes/cryptominer/minerhotel/) 🏨
4. Send your coins to [exchanges](/recipes/cryptominer/exchange/) or [wallets](/recipes/cryptominer/wallet/) 💹
5. [Monitor](/recipes/cryptominer/monitor/) your empire :heartbeat:
6. [Profit](/recipes/cryptominer/profit/)! 💰
## Chef's Notes
1. My two RX580 cards (_bought alongside each other_) perform slightly differently. GPU0 works with a 2050Mhz memory clock, but GPU1 only works at 2000Mhz. Anything over 2000Mhz causes system instability. YMMV.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptominer/exchange.md
+55
View File
@@ -0,0 +1,55 @@
!!! warning
This is not a complete recipe - it's a component of the [cryptominer](/recipes/cryptominer/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Exchange
You may be mining a particular coin, and want to hold onto it, in the hopes of long-term growth. In that case, stick it in a [wallet](/recipes/cryptominer/wallet/) and be done with it.
You may also not care too much about the coin (you're mining for money, right?), in which case you want to "cash out" your coins into something you can spend.
In this case, you'll want to configure your mining pool to send your coin-of-choice to an **exchange**, so that you can turn it into a **different** coin, or extract it into FIAT (_oldschool, cave-man currency_).
## Preparation
### Get verified at exchanges
Most exchanges (Binance is currently a notable exception) require some sort of verification of your ID before they'll let you trade, or withdraw coins as FIAT.
So, you may as well get yourself verified in anticipation (_it can take a while during periods of increased crypto-hype_).
Here are (_referral_) links to exchanges I've used personally:
* [Cryptopia](https://www.cryptopia.co.nz/Register?referrer=funkypenguin) : Trades obscure altcoins that other exchanges don't, and can withdraw to USD and NZD
* [Binance](https://www.binance.com/?ref=15312815) : Doesn't require verification for small-time traders/miners
* [Coinbase](https://www.coinbase.com/join/5a4d1ed0ee3de40195a695c8) : Beginner's exchange. Coins mined in Nicehash can be sent to coinbase with zero fees.
### Send coins to Exchanges
Now simply configure your mining pool (or your miner) to send your coins to your wallet's deposit address for each coin. Note that every coin has a unique wallet address.
!!! warning
Don't try to send one coin (i.e., LTC) to a different coin's (i.e. BTC) wallet. You will **loose** your money and be **sad** 😞
## Withdraw coins to FIAT (cash)
Once you have enough coins in your exchange wallet, you can "trade" them into the real-world currency of your choice. For example, if you mined 100 Ella to [Cryptopia](https://www.cryptopia.co.nz/Register?referrer=funkypenguin), you could trade it for NZDT or USDT, and withdraw it to your bank account.
## Continue your adventure
Now, continue to the next stage of your grand mining adventure:
1. Build your [mining rig](/recipes/cryptominer/mining-rig/) 💻
2. Setup your [AMD](/recipes/cryptominer/amd-gpu/) or [Nvidia](/recipes/cryptominer/nvidia-gpu/) GPUs 🎨
3. Sign up for [mining pools](/recipes/cryptominer/mining-pool/) :swimmer:
4. Setup your miners with [Miner Hotel](/recipes/cryptominer/minerhotel/) 🏨
5. Send your coins to exchanges (_This page_) or [wallets](/recipes/cryptominer/wallet/) 💹
6. [Monitor](/recipes/cryptominer/monitor/) your empire :heartbeat:
7. [Profit](/recipes/cryptominer/profit/)! 💰
## Chef's Notes
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptominer/minerhotel.md
+105
View File
@@ -0,0 +1,105 @@
# Minerhotel
!!! warning
This is not a complete recipe - it's a component of the [cryptominer](/recipes/cryptominer/) "_uber-recipe_", but has been split into its own page to reduce complexity.
So, you have GPUs. You can mine cryptocurrency. But **what** cryptocurrency should you mine?
1. You could manually keep track of [whattomine](http://whattomine.com/), and launch/stop miners based on profitability/convenience, as you see fit.
2. You can automate the process of mining the most profitable coin based on your GPUs' capabilities and the current market prices, and do better things with your free time! (_[receiving alerts](/recipes/crytominer/monitor/), of course, if anything stops working!_)
This recipe covers option #2 😁
[Miner hotel](http://minerhotel.com/) is a collection of scripts and config files to fully automate your mining across AMD or Nvidia cards.
## Ingredients
* [Latest Minerhotel release](http://minerhotel.com/download.html) for Linux
* Time and patience
## Preparation
### Unpack Minerhotel
Unpack the minerhotel release. You can technically unpack it anywhere, but this guide, and all pre-configured miners, expect an installation at /opt/minerhotel.
### Prepare miner.config
Copy /opt/minerhotel/miner.config.example to /opt/minerhotel/miner.config, and start making changes. Here's a rundown of the variables:
* **WALLET<WHATEVER\>** : Set these WALLET variables to your wallet addresses for all the currencies you want to mine. Your miner will fail to start without the wallet variable, but it won't confirm it's a **valid** wallet. **Now, double-check to confirm the wallet is correct, and you're not just mining coins to /dev/null, or someone else's wallet!** You can either use your [exchange](/recipes/cryptominer/exchange/) wallet address or your own [wallet](/recipes/cryptominer/wallet/).
* **WORKER** : Set this to the name you'll use to define your miner in the various pools you mine. Some pools (_i.e. NiceHash_) auto-create workers based on whatever worked name you specify, whereas others (_Supernova.cc_) will refuse to authenticate you unless you've manually created the worker/password in their UI first.
* **SUPRUSER** : Set this to your supernova.cc login username (**not** your worker name) (_optional, only use this if you want to use supernova.cc_)
* **SUPRPASS** : Set this to the password you've configured within Supernova.cc for your **worker** as defined by the WORKER variable. Note that this require syou to use the **same** worker name and password across all your supernova.cc pools (_optional, only necessary if you want to use supernova.cc_)
* **MPHUSER** : Set this to your miningpoolhub login username (_optional, only necessary if you want to use [miningpoolhub.com](https://miningpoolhub.com/)_)
* **TBFUSER** : Set this to your theblocksfactory login username (_optional, only necessary if you want to use t[heblocksfactory.com](https://theblocksfactory.com/)_)
* **VERTPOOLUSER/VERTPOOLPASS** : Set these to your vertpool user/password (_optional, only necessary if you want to use [vertpool.org](http://vertpool.org/)_)
### Install services
1. Run ```/opt/minerhotel/scripts/install-services.sh``` to install the necessary services for systemd
2. Run ```/opt/minerhotel/scripts/fixmods.sh``` to correctly set the filesystem permissions for the various miner executables
!!! note
fixmods.sh doesn't correctly set permissions on subdirectories, so until this is fixed, you also need to run ```chmod 755 /opt/minerhotel/bin/claymore/ethdcrminer64```
### Setup whattomine-linux
For the whattomine bot to select the most profitable coin to mine for **your** GPUs, you'll need to feed your cookie from https://whattomine.com
1. Start by installing [this](https://chrome.google.com/webstore/detail/cookie-inspector/jgbbilmfbammlbbhmmgaagdkbkepnijn) addon for Chrome, or [this](https://addons.mozilla.org/en-US/firefox/addon/firecookie/) addon for firefox
2. Then visit http://whattomine.com/ and tweak settings for you GPUs, power costs, etc.
3. Grab the cookie per the whattomine [README](http://git.minerhotel.com:3000/minerhotel/minerhotel/src/master/whattomine/README.md), and paste it (_about 2200 characters_) into /opt/minerhotel/whattomine/config.json
4. Ensure that only the coins/miners that you **want** are enabled in config.json - delete the others, or put a dash ("-") after the ones you want to disable. Set the service names as defined in /opt/minerhotel/services/
### Test miners
Before trusting the whattomine service to automatically launch your miners, test each one first by starting them manually, and then checking their status.
For example, to test the **miner-amd-eth-ethhash-ethermine** miner, run
1. ```systemctl start miner-amd-eth-ethhash-ethermine.service``` to start the service
2. And then watch the output by running ```journalctl -u miner-amd-eth-ethhash-ethermine -f```
3. When you're satisfied it's working correctly (_without errors and with a decent hashrate_), stop the miner again by running ```systemctl stop miner-amd-eth-ethhash-ethermine```, and move onto testing the next one.
## Serving
### Launch whattomine
Finally, run ```systemctl start minerhotel-whattomine``` and then ```journalctl -u minerhotel-whattomine -f``` to watch the output. Within a minute, you should see whattomime launching the most profitable miner, as illustrated below:
```
Jan 29 13:49:38 kvm.funkypenguin.co.nz whattomine-linux[2057]: 2018-01-29T13:49:38+1300 <INF> whattomine.js Loading whattominebot
Jan 29 13:49:38 kvm.funkypenguin.co.nz whattomine-linux[2057]: 2018-01-29T13:49:38+1300 <INF> whattomine.js Starting whattominebot now.
Jan 29 13:50:45 kvm.funkypenguin.co.nz whattomine-linux[2057]: 2018-01-29T13:50:45+1300 <INF> whattomine.js Mining Ethereum|ETH|Ethash|0.0089|0.00093|100
Jan 29 13:50:45 kvm.funkypenguin.co.nz whattomine-linux[2057]: 2018-01-29T13:50:45+1300 <INF> whattomine.js Could not find a miner for Ubiq.
Jan 29 13:51:39 kvm.funkypenguin.co.nz whattomine-linux[2057]: 2018-01-29T13:51:39+1300 <INF> whattomine.js Mining Ethereum|ETH|Ethash|0.0089|0.00094|100
Jan 29 13:51:39 kvm.funkypenguin.co.nz whattomine-linux[2057]: 2018-01-29T13:51:39+1300 <INF> whattomine.js Could not find a miner for Ubiq.
```
!!! note
The messages about "Could not find miner" can be ignored, they indicate that one of the preferred coins on whattomine does not have a miner defined.
To make whattomine start automatically in future, run ```systemctl enable minerhotel-whattomine```
## Continue your adventure
Now, continue to the next stage of your grand mining adventure:
1. Build your [mining rig](/recipes/cryptominer/mining-rig/) 💻
2. Setup your [AMD](/recipes/cryptominer/amd-gpu/) or [Nvidia](/recipes/cryptominer/nvidia-gpu/) GPUs 🎨
3. Sign up for [mining pools](/recipes/cryptominer/mining-pool/) :swimmer:
4. Setup your miners with Miner Hotel 🏨 (_This page_)
5. Send your coins to [exchanges](/recipes/cryptominer/exchange/) or [wallets](/recipes/cryptominer/wallet/) 💹
6. [Monitor](/recipes/cryptominer/monitor/) your empire :heartbeat:
7. [Profit](/recipes/cryptominer/profit/)! 💰
## Chef's Notes
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptominer/mining-pool.md
+58
View File
@@ -0,0 +1,58 @@
!!! warning
This is not a complete recipe - it's a component of the [cryptominer](/recipes/cryptominer/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Mining Pools
You and your puny GPUs don't have a snowball's chance of mining a block on your own. Your only option is to join a mass mining conglomerate (_a mining pool_), throw in your share of the effort for a share of the reward.
## Preparation
### Setup accounts at mining pools
This'll save you some frustration later... Next time you're watching a movie or doing something mindless, visit http://whattomine.com/, and take note of the 10-15 most profitable coins for your GPU type(s).
On your [exchanges](/recipes/cryptominer/exchange/), identify the "_deposit address_" for each popular coin, and note them down for the next step.
!!! note
If you're wanting to mine directly to a wallet for long-term holding, then substitute your wallet public address for this deposit address.
Now work your way through the following list of pools, creating an account on each one you want as you go. In the case of each pool/coin, setup your "payout address" to match your change address for the coin (above).
* [Mining Pool Hub](https://miningpoolhub.com/) (Lots of coins)
* [NiceHash](https://nicehash.com) (Ethereum, Decred)
* [suprnova](https://suprnova.cc/) - Lots of coins, but, you generally need a separate login for each pool. You _also_ need to create a worker in each pool with a common username and password, for [Minerhotel](/recipes/crytominer/minerhotel/).
* [nanopool](https://nanopool.org/) (Ethereum, Ethereum Classic, SiaCoin, ZCash, Monero, Pascal and Electroneum)
* [slushpool](https://slushpool.com/home/) (BTC and ZCash)
## Serving
### Avoid fees
As noted by IronicBadger [here](https://www.linuxserver.io/2018/01/20/how-to-build-a-cryptocurrency-mining-rig/), the name of the game is avoiding fees where possible. Here are a few tips:
* [Mining Pool Hub](https://miningpoolhub.com/) is a popular pool for multiple coins, and it allows you (_for a fee_) to auto-exchange the coins you mine for coins that you actually _want_.
* [NiceHash](https://nicehash.com) will allow you to send your earned bitcoin (_whatever you mine, they pay you in bitcoin_) to coinbase for free.
## Continue your adventure
Now, continue to the next stage of your grand mining adventure:
1. Build your [mining rig](/recipes/cryptominer/mining-rig/) 💻
2. Setup your [AMD](/recipes/cryptominer/amd-gpu/) or [Nvidia](/recipes/cryptominer/nvidia-gpu/) GPUs 🎨
3. Sign up for [mining pools](/recipes/cryptominer/mining-pool/) :swimmer:
4. Setup your miners with [Miner Hotel](/recipes/cryptominer/minerhotel/) 🏨
5. Send your coins to exchanges (_This page_) or [wallets](/recipes/cryptominer/wallet/) 💹
6. [Monitor](/recipes/cryptominer/monitor/) your empire :heartbeat:
7. [Profit](/recipes/cryptominer/profit/)! 💰
## Chef's Notes
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptominer/mining-rig.md
+55
View File
@@ -0,0 +1,55 @@
!!! warning
This is not a complete recipe - it's a component of the [cryptominer](/recipes/cryptominer/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Mining Rig
## Hardware
You can surely [find](https://www.reddit.com/r/gpumining/) a better tutorial on how to build a mining rig than this one. However, to summarise what I've learned:
1. You want a beefy power supply, with lots of PCI-e 8pin and 6pin cables.
2. You need 1 x PCI express (_PCI-e_) port per GPU
3. You don't need powerful CPU or much RAM - the GPUs do all the mining work. My current guts (_minus the PSU_) are 5 years old.
## Do I need a open-air rig?
Initially, no. You can use any old PC chassis. But as soon as you want more than one GPU, you're going to start to run into cooling problems.
You don't need anything fancy. Here's a photo of the rig my wife built me:
![My mining rig, naked](../../images/mining_rig_naked.jpg)
I recommend this design (_with the board with little holes in it_) - it takes up more space, but I have more room to place extra components (_PSUs, hard drives, etc_), as illustrated below:
!!! note
You'll note the hard drives in the picture - that's not part of the mining requirements, it's because my rig doubles as my [Plex](/recipes/plex/) server ;)
![My mining rig, populated](../../images/mining_rig_populated.jpg)
## Continue your adventure
Now, continue to the next stage of your grand mining adventure:
1. Build your mining rig 💻 (This page)
2. Setup your [AMD](/recipes/cryptominer/amd-gpu/) or [Nvidia](/recipes/cryptominer/nvidia-gpu/) GPUs 🎨
3. Sign up for [mining pools](/recipes/cryptominer/mining-pool/) :swimmer:
4. Setup your miners with [Miner Hotel](/recipes/cryptominer/minerhotel/) 🏨
5. Send your coins to [exchanges](/recipes/cryptominer/exchange/) or [wallets](/recipes/cryptominer/wallet/) 💹
6. [Monitor](/recipes/cryptominer/monitor/) your empire :heartbeat:
7. [Profit](/recipes/cryptominer/profit/)! 💰
## Chef's Notes
1. Pro-tip : You're going to spend some time overclocking. Which is going to make your mining host unstable.
Yes. It's the ultimate _#firstworldproblem_, but if you have a means to remotely reboot your host, use it! You can thank me later.
(_I hooked up a remote-controlled outlet to my rig, so that I can power-cycle it without having to crawl under the desk!_)
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptominer/monitor.md
+93
View File
@@ -0,0 +1,93 @@
# Monitor
!!! warning
This is not a complete recipe - it's a component of the [cryptominer](/recipes/cryptominer/) "_uber-recipe_", but has been split into its own page to reduce complexity.
So, you're a miner! But if you're not **actively** mining, are you still a miner? This page details how to **measure** your mining activity, and how to raise an alert when a profit-affecting issue affects your miners.
## Ingredients
1. [InfluxDB+Grafana](https://www.funkypenguin.co.nz/note/adding-custom-data-to-influxdb-and-grafana/) instance, for visualising data
2. [Icinga](https://www.icinga.com/), [Nagios](https://www.nagios.org/) etc for alarming on GPU/miner status
3. [Asi MPM](https://www.asimpm.com/) (iOS) for monitoring your miner/pool status
4. [Altpocket](https://altpocket.io/?ref=ilVqdeWbAv), [CoinTracking](https://cointracking.info?ref=F560640), etc for managing your crypto-asset portfolio (_referral links_)
## Preparation
### Visualising performance
![Visualise mining performance](../../images/cryptominer_grafana.png)
Since [Minerhotel](/recipes/crytominer/minerhotel/) switches currency based on what's most profitable in the moment, it's hard to gauge the impact of changes (overclocking, tweaking, mining pools) over time.
I hacked up a bash script which grabs performance data from the output of the miners, and throws it into an InfluxDB database, which can then be visualized using Grafana.
Here's an early version of the script (_it's since been updated for clockspeed and power usage too_):
<script src="https://gist.github.com/funkypenguin/5ec0581389be20ea6512e4c2bafb2a89.js"></script>
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes up-to-date versions of the InfluxDB /Grafana script mentioned above, as well as pre-setup Grafana graphs, so that patrons can simply "_git pull_" and start monitoring 👍
### Alarming on failure
![Visualise mining performance](../../images/cryptominer_alarm.png)
GPU mining can fail in subtle ways. On occasion, I've tweaked my GPUs to the point that the miner will start, but one or all GPUs will report a zero hash rate. I wanted to be alerted to such profit-affecting issues, so I wrote a bash script (_intended to be executed by NRPE from Icinga, Nagios, etc_).
The script tests the output of the currently active miner, and ensures the GPUs have a valid hashrate.
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes up-to-date versions of the Icinga scripts mentioned above, so that patrons can simply "_git pull_" and start monitoring 👍
### Monitoring pool/miner status
I've tried several iOS apps for monitoring my performance across various. The most useful app I've found thus far is [Asi MPM](https://www.asimpm.com/). It requires a little upfront effort to configure for all your coins/pools, but thereafter it's a handy way to keep tabs on your new obsession!
### Track your portfolio
Now that you've got your coins happily cha-chinging into you [wallets](/recipes/cryptominer/wallet/) (_and potentially various [exchanges](/recipes/cryptominer/exchange/)_), you'll want to monitor the performance of your portfolio over time.
#### Web Apps
There's a detailed breakdown of porfolio-management apps [here](https://www.cryptostache.com/2017/11/10/keeping-track-cryptocurrency-portfolio-best-apps-2017/).
Personally, I use:
* [Altpocket](https://altpocket.io/?ref=ilVqdeWbAv) (A free web app which can auto-sync with certain exchanges and wallets)
* [CoinTracking](https://cointracking.info?ref=F560640) - The top crypto-portfolio manager, by far. But it's expensive when you get to > 200 trades. You get what you pay for ;)
#### Mobile Apps
I've found the following iOS apps to be useful in tracking my portfolio (_really more for investing than mining though, since portfolio tracking requires a manual entry for each trade_)
* [Delta](https://itunes.apple.com/us/app/delta-crypto-ico-portfolio/id1288676542?mt=8) (iOS) - Track your portfolio (losses/gains) and alert you to changes in the coins you watch
* [Bitscreener](https://itunes.apple.com/app/apple-store/id1240849311?mt=8) )(iOS) - Track multiple currencies on a watchlist, and quickly view news/discussion per coin
!!! note
Some of the links above are referral links. I get some goodies when you use them.
## Continue your adventure
Now, continue to the next stage of your grand mining adventure:
1. Build your [mining rig](/recipes/cryptominer/mining-rig/) 💻
2. Setup your [AMD](/recipes/cryptominer/amd-gpu/) or [Nvidia](/recipes/cryptominer/nvidia-gpu/) GPUs 🎨
3. Sign up for [mining pools](/recipes/cryptominer/mining-pool/) :swimmer:
4. Setup your miners with [Miner Hotel](/recipes/cryptominer/minerhotel/) 🏨
5. Send your coins to [exchanges](/recipes/cryptominer/exchange/) or [wallets](/recipes/cryptominer/wallet/) 💹
6. Monitor your empire :heartbeat: (_this page_)
7. [Profit](/recipes/cryptominer/profit/)! 💰
## Chef's Notes
1. Ultimately I hope to move all the configuration / mining executables into docker containers, but for now, they're running on a CentOS7 host for direct access to GPUs. (_Apparently it **may** be possible to pass-thru the GPUs to docker containers, but I wanted stability first, before abstracting my hardware away from my miners_)
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptominer/nvidia-gpu.md
+164
View File
@@ -0,0 +1,164 @@
# NVidia GPU
!!! warning
This is not a complete recipe - it's a component of the [cryptominer](/recipes/cryptominer/) "_uber-recipe_", but has been split into its own page to reduce complexity.
## Ingredients
1. [Nvidia drivers](http://www.nvidia.com/Download/driverResults.aspx/104284/en-us) for your GPU
2. Some form of X11 GUI preconfigured on your linux host (yes, it's a PITA, but it's necessary for overclocking)
## Preparation
### Install kernel-devel and gcc
The nVidia drivers will need the kernel development packages for your OS installed, as well as gcc. Run the following (for CentOS - there will be an Ubuntu equivalent):
```yum install kernel-devel-$(uname -r) gcc```
### Remove nouveau
Your host probably already includes nouveau, free/libre drivers for Nvidia graphics card. These won't cut it for mining, so blacklist them to avoid conflict with the dirty, proprietary Nvidia drivers:
```
echo 'blacklist nouveau' >> /etc/modprobe.d/blacklist.conf
dracut /boot/initramfs-$(uname -r).img $(uname -r) --force
systemctl disable gdm
reboot
```
### Install Nvidia drivers
Download and uncompress the [Nvidia drivers](http://www.nvidia.com/Download/driverResults.aspx/104284/en-us), and execute the installation as root, with a command something like this:
```bash NVIDIA-Linux-x86_64-352.30.run```
Update your X11 config by running:
```
nvidia-xconfig
```
### Enable GUID
```
systemctl enable gdm
ln -s '/usr/lib/systemd/system/gdm.service' '/etc/systemd/system/display-manager.service'
reboot
```
## Overclock
### Preparation
!!! warning
Like overclocking itself, this process is still a work in progress. YMMV.
Of course, you want to squeeze the optimal performance out of your GPU. This is where the X11 environment is required - to adjust GPU clock/memory settings, you need to use the ```nvidia-settings``` command, which (_stupidly_) **requires** an X11 display, even if you're just using the command line.
The following command: configures X11 for a "fake" screen so that X11 will run, even on a headless machine managed by SSH only, and ensures that the PCI bus ID of every NVidia device is added to the xorg.conf file (to avoid errors about "_(EE) no screens found(EE)_")
```
nvidia-xconfig -a --allow-empty-initial-configuration --cool-bits=28 --use-display-device="DFP-0" --connected-monitor="DFP-0" --enable-all-gpus --separate-x-screens
```
!!! note
The script below was taken from https://github.com/Cyclenerd/ethereum_nvidia_miner
Make a directory for your overclocking script. Mine happens to be /root/overclock/, but use whatever you like.
Create settings.conf as follows:
```
# Known to work with Nvidia 1080ti, but probably not optimal. It's an eternal work-in-progress.
MY_WATT="200"
MY_CLOCK="100"
MY_MEM="400"
MY_FAN="60"
```
Then create nvidia-overclock.sh as follows:
```
#!/usr/bin/env bash
#
# nvidia-overclock.sh
# Author: Nils Knieling - https://github.com/Cyclenerd/ethereum_nvidia_miner
#
# Overclocking with nvidia-settings
#
# Load global settings settings.conf
if ! source ~/overclock/settings.conf; then
echo "FAILURE: Can not load global settings 'settings.conf'"
exit 9
fi
export DISPLAY=:0
# Graphics card 1 to 6
for MY_DEVICE in {0..5}
do
# Check if card exists
if nvidia-smi -i $MY_DEVICE >> /dev/null 2>&1; then
nvidia-settings -a "[gpu:$MY_DEVICE]/GPUPowerMizerMode=1"
# Fan speed
nvidia-settings -a "[gpu:$MY_DEVICE]/GPUFanControlState=1"
nvidia-settings -a "[fan:$MY_DEVICE]/GPUTargetFanSpeed=$MY_FAN"
# Graphics clock
nvidia-settings -a "[gpu:$MY_DEVICE]/GPUGraphicsClockOffset[3]=$MY_CLOCK"
# Memory clock
nvidia-settings -a "[gpu:$MY_DEVICE]/GPUMemoryTransferRateOffset[3]=$MY_MEM"
# Set watt/powerlimit. This is also set in miner.sh at autostart.
sudo nvidia-smi -i "$MY_DEVICE" -pl "$MY_WATT"
fi
done
echo
echo "Done"
echo
```
### Start your engine!
**Once** you've got X11 running correctly, execute ,/nvidia-overclock.sh, and you should see something like the following:
```
[root@kvm overclock]# ./nvidia-overclock.sh
Attribute 'GPUPowerMizerMode' (kvm.funkypenguin.co.nz:0[gpu:0]) assigned value 1.
Attribute 'GPUFanControlState' (kvm.funkypenguin.co.nz:0[gpu:0]) assigned value 1.
Attribute 'GPUTargetFanSpeed' (kvm.funkypenguin.co.nz:0[fan:0]) assigned value 60.
Attribute 'GPUGraphicsClockOffset' (kvm.funkypenguin.co.nz:0[gpu:0]) assigned value 100.
Attribute 'GPUMemoryTransferRateOffset' (kvm.funkypenguin.co.nz:0[gpu:0]) assigned value 400.
Power limit for GPU 00000000:04:00.0 was set to 150.00 W from 150.00 W.
All done.
Done
[root@kvm overclock]#
```
Play with changing your settings.conf file until you break it, and then go back one revision :)
## Continue your adventure
Now, continue to the next stage of your grand mining adventure:
1. Build your [mining rig](/recipes/cryptominer/mining-rig/) 💻
2. Setup your [AMD](/recipes/cryptominer/amd-gpu/) or Nvidia (_this page_) GPUs 🎨
3. Sign up for [mining pools](/recipes/cryptominer/mining-pool/) :swimmer:
4. Setup your miners with [Miner Hotel](/recipes/cryptominer/minerhotel/) 🏨
5. Send your coins to [exchanges](/recipes/cryptominer/exchange/) or [wallets](/recipes/cryptominer/wallet/) 💹
6. [Monitor](/recipes/cryptominer/monitor/) your empire :heartbeat:
7. [Profit](/recipes/cryptominer/profit/)! 💰
## Chef's Notes
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptominer/profit.md
+28
View File
@@ -0,0 +1,28 @@
# Profit! 💰
Well, that's it really. You're a cryptominer. Welcome to the party.
## Your adventure has only just begun!
To recap, you did all this:
1. Build your [mining rig](/recipes/cryptominer/mining-rig/) 💻
2. Setup your [AMD](/recipes/cryptominer/amd-gpu/) or [Nvidia](/recipes/cryptominer/nvidia-gpu/) GPUs 🎨
3. Sign up for [mining pools](/recipes/cryptominer/mining-pool/) :swimmer:
4. Setup your miners with [Miner Hotel](/recipes/cryptominer/minerhotel/) 🏨
5. Send your coins to [exchanges](/recipes/cryptominer/exchange/) or [wallets](/recipes/cryptominer/wallet/) 💹
6. [Monitor](/recipes/cryptominer/monitor/) your empire :heartbeat:
7. Profit! (_This page_) 💰
## What next?
Get in touch and share your experience - there's a special [discord](https://discord.gg/Y9aUhrj) channel if you're the IM type, else post a comment/thread at the [kitchen](http://discourse.geek-kitchen.funkypenguin.co.nz/) :)
## Chef's Notes
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptominer/wallet.md
+41
View File
@@ -0,0 +1,41 @@
!!! warning
This is not a complete recipe - it's a component of the [cryptominer](/recipes/cryptominer/) "_uber-recipe_", but has been split into its own page to reduce complexity.
# Wallet
You may be mining a particular coin, and want to hold onto it, in the hopes of long-term growth. The safest place to stick that coin, therefore, is an a wallet.
## Preparation
### Get your wallets
Your favorite coin probably has a link to various desktop wallets on their website. All you have to do is download the wallet to your desktop, fire it up, and then **backup your public/private key somewhere safe**.
### Wallets I use
I mine most of my coins to Exchanges, but I do have the following wallets:
* [Jaxx](https://itunes.apple.com/nz/app/jaxx-blockchain-wallet/id1084514516?mt=8) on my iPhone for popular coins (BTC, ETH, etc)
* Eleos wallet for [ZClassic](https://zclassic.org/)
## Continue your adventure
Now, continue to the next stage of your grand mining adventure:
1. Build your [mining rig](/recipes/cryptominer/mining-rig/) 💻
2. Setup your [AMD](/recipes/cryptominer/amd-gpu/) or [Nvidia](/recipes/cryptominer/nvidia-gpu/) GPUs 🎨
3. Sign up for [mining pools](/recipes/cryptominer/mining-pool/) :swimmer:
4. Setup your miners with [Miner Hotel](/recipes/cryptominer/minerhotel/) 🏨
5. Send your coins to [exchanges](/recipes/cryptominer/exchange/) or wallets (_This page_) 💹
6. [Monitor](/recipes/cryptominer/monitor/) your empire :heartbeat:
7. [Profit](/recipes/cryptominer/profit/)! 💰
## Chef's Notes
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/cryptonote-mining-pool.md
+16
View File
@@ -0,0 +1,16 @@
# CryptoNote Mining Pool
[Cryptocurrency miners](/recipes/cryptominer) will "pool" their GPU resources ("_hashpower_") into aggregate "_mining pools_", so that by the combined effort of all the miners, the pool will receive a reward for the blocks "mined" into the blockchain, and this reward will be distributed among the miners.
[CryptoNote](https://cryptonote.org/) is an open-source toolset designed to facilitate the creation of new privacy-focused [cryptocurrencies](https://cryptonote.org/coins)
(_CryptoNote = 'Kryptonite'. In a pool. Get it?_)
![CryptoNote Mining Pool Screenshot](/images/cryptonote-mining-pool.png)
The fact that all these currencies share a common ancestry means that a common mining pool platform can be used for miners. The following recipes all use variations of [Dvandal's cryptonote-nodejs-pool ](https://github.com/dvandal/cryptonote-nodejs-pool)
## Mining Pool Recipies
* [TurtleCoin](/recipes/turtle-pool/), the no-BS, fun baby cryptocurrency
* [Athena](/recipes/cryptonote-mining-pool/athena/), TurtleCoin's newborn baby sister
manuscript/recipes/cryptonote-mining-pool/athena.md
+385
View File
@@ -0,0 +1,385 @@
# Athena Mining Pool
[Cryptocurrency miners](/recipes/cryptominer) will "pool" their GPU resources ("_hashpower_") into aggregate "_mining pools_", so that by the combined effort of all the miners, the pool will receive a reward for the blocks "mined" into the blockchain, and this reward will be distributed among the miners.
![Athena Pool Screenshot](../../images/athena-mining-pool.png)
This recipe illustrates how to build a mining pool for [Athena](https://getathena.org), a newborn [CryptoNote](https://cryptonote.org/) [currency](https://cryptonote.org/coins) recently forked from [TurtleCoin](https://turtlecoin.lol)
The end result is a mining pool which looks like this: https://athx.heigh-ho.funkypenguin.co.nz/
!!! question "Isn't this just a copy/paste of your [TurtleCoin Pool Recipe](/recipes/turtle-pool/)?"
Why yes. Yes it is :) But it's adjusted for Athena, which uses different containers and wallet binary names, and it's running the improved [cryptonote-nodejs-pool software](https://github.com/dvandal/cryptonote-nodejs-pool), which is common to all the [cryptonote-mining-pool](/recipes/criptonote-mining-pool/) recipes!
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostnames (_pool and api_) you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
4. At least 8GB disk space (2.5GB used, 6.5GB for future growth)
## Preparation
### Create user account
The Athena pool elements won't (_and shouldn't_) run as root, but they'll need access to write data to some parts of the filesystem (_like logs, etc_).
To manage access control, we'll want to create a local user on **each docker node** with the same UID.
```
useradd -u 3508 athena-pool
```
### Setup Redis
The pool uses Redis for in-memory and persistent storage.
!!! warning "Playing it safe"
Be aware that by default, Redis stores some data **only** in memory, and writes to the filesystem at default intervals (_can be up to 5 minutes by default_). Given we don't **want** to loose 5 minutes of miner's data if we restart Redis (_what happens if we found a block during those 5 minutes but haven't paid any miners yet?_), we want to ensure that Redis runs in "appendonly" mode, which ensures that every change is immediately written to disk.
We also want to make sure that we retain all Redis logs persistently (_We're dealing with people's cryptocurrency here, it's a good idea to keep persistent logs for debugging/auditing_)
Create directories to hold Redis data. We use separate directories for future flexibility - One day, we may want to backup the data but not the logs, or move the data to an SSD partition but leave the logs on slower, cheaper disk.
```
mkdir -p /var/data/athena-pool/redis/config
mkdir -p /var/data/athena-pool/redis/data
mkdir -p /var/data/athena-pool/redis/logs
chown athena-pool /var/data/athena-pool/redis/data
chown athena-pool /var/data/athena-pool/redis/logs
```
Create **/var/data/athena-pool/redis/config/redis.conf** using http://download.redis.io/redis-stable/redis.conf as a guide. The following are the values I changed from default on my deployment (_but I'm not a Redis expert!_):
```
appendonly yes
appendfilename "appendonly.aof"
loglevel notice
logfile "/logs/redis.log"
protected-mode no
```
I also had to **disable** the following line, by commenting it out (_thus ensuring Redis container will respond to the other containers_):
```
bind 127.0.0.1
```
### Setup Nginx
We'll run a simple Nginx container to serve the static front-end of the web UI.
The simplest way to get the frontend is just to clone the upstream athena-pool repo, and mount the "/website" subdirectory into Nginx.
```
git clone https://github.com/funkypenguin/cryptonote-nodejs-pool.git /var/data/athena-pool/nginx
```
Edit **/var/data/athena-pool/nginx/website/config.js**, and change at least the following:
```
var api = "https://<CHOOSE A FQDN TO USE FOR YOUR API>";
var blockchainExplorer = "http://explorer.athx.org/?hash={id}#blockchain_block";
var transactionExplorer = "http://explorer.athx.org/?hash={id}#blockchain_transaction";
```
And optionally, set the following:
```
var telegram = "https://t.me/YourPool";
var discord = "https://chat.funkypenguin.co.nz";
```
### Setup Athena daemon
The first thing we'll need to participate in the great and powerful Athena network is a **node**. The node is responsible for communicating with the rest of the nodes in the blockchain, allowing our miners to receive new blocks to try to find.
Create a directory to hold the blockchain data:
```
mkdir -p /var/data/runtime/athena-pool/daemon/
```
### Setup Athena wallet
Our pool needs a wallet to be able to (a) receive rewards for blocks discovered, and (b) pay out our miners for their share of the reward.
!!! note
Under Athena, "walletd" was renamed to "services". I don't know why, but I've updated this recipe to reflect this.
Create directories to hold wallet data:
```
mkdir -p /var/data/athena-pool/services/config
mkdir -p /var/data/athena-pool/services/services
mkdir -p /var/data/athena-pool/services/logs
chown -R athena-pool /var/data/athena-pool/wallet/services
chown -R athena-pool /var/data/athena-pool/wallet/logs
```
Now create the initial wallet. You'll want to secure your wallet password, so the command below will **prompt** you for the key (no output from the prompt), and insert it into an environment variable. This means that the key won't be stored in plaintext in your bash history!
```
read PASS
```
After having entered your password (you can confirm by running ```env | grep PASS```), run the following to run the wallet daemon _once_, with the instruction to create a new wallet container:
```
docker run \
-v /var/data/athena-pool/wallet/container:/container \
--rm -ti --entrypoint /usr/local/bin/services funkypenguin/athena \
--container-file /container/wallet.container \
--container-password $PASS \
--generate-container
```
You'll get a lot of output. The following are relevant lines from a successful run with the extra stuff stripped out:
```
2018-May-01 11:14:57.662664 INFO walled v0.6.4 ()
2018-May-01 11:14:59.868087 INFO Generating new wallet
2018-May-01 11:14:59.919178 INFO Container initialized with view secret key, public view key <your view public key will be here>
2018-May-01 11:14:59.920932 INFO New wallet added athena<your wallet's public address>, creation timestamp 0
2018-May-01 11:14:59.932367 INFO Container shut down
2018-May-01 11:14:59.932419 INFO Loading container...
2018-May-01 11:14:59.961814 INFO Consumer added, consumer 0x55b0fb5bc070, count 1
2018-May-01 11:14:59.961996 INFO Starting...
2018-May-01 11:14:59.962173 INFO Container loaded, view public key <your view public key will be here>, wallet count 1, actual balance 0.00, pending balance 0.00
2018-May-01 11:14:59.962508 INFO New wallet is generated. Address: TRTL<your wallet's public address>
2018-May-01 11:14:59.962581 INFO Saving container...
2018-May-01 11:14:59.962683 INFO Stopping...
2018-May-01 11:14:59.962862 INFO Stopped
```
Take careful note of your wallet password, public view key, and wallet address
Create **/var/data/athena-pool/services/config/services.conf**, containing the following:
```
bind-address = 0.0.0.0
container-file = /services/wallet.container
container-password = <ENTER YOUR CONTAINER PASSWORD HERE>
rpc-password = <CHOOSE A PASSWORD TO ALLOW POOL TO TALK TO WALLET>
log-file = /dev/stdout
log-level = 3
daemon-address = daemon
```
Set permissions appropriately:
```
chown athena-pool /var/data/athena-pool/services/ -R
```
### Setup Athena mining pool
Following the convention we've set above, create directories to hold pool data:
```
mkdir -p /var/data/athena-pool/pool/config
mkdir -p /var/data/athena-pool/pool/logs
chown -R athena-pool /var/data/athena-pool/pool/logs
```
Now create **/var/data/athena-pool/pool/config/config.json**, copying https://github.com/funkypenguin/cryptonote-nodejs-pool/blob/master/config_examples/monero.json, and modifying according to https://github.com/athena-network/athena-pool/blob/master/config.json, adjusting at least the following:
Send logs to /logs/, so that they can potentially be stored / backed up separately from the config:
```
"logging": {
"files": {
"level": "debug",
"directory": "/logs",
"flushInterval": 5
},
```
Set the "poolAddress" field to your wallet address
```
"poolServer": {
"enabled": true,
"clusterForks": "auto",
"poolAddress": "<SET THIS TO YOUR WALLET ADDRESS GENERATED ABOVE>",
```
Add the "host" value to the api section, since our API will run on its own container, and choose a password you'll use for the webUI admin page
```
"api": {
"enabled": true,
"hashrateWindow": 600,
"updateInterval": 5,
"port": 8117,
"blocks": 30,
"payments": 30,
"password": "<PASSWORD FOR ADMIN PAGE ACCESS>"
```
Set the host value for the daemon:
```
"daemon": {
"host": "daemon",
"port": 11898
},
```
Set the host value for the wallet, and set your container password (_you recorded it earlier, remember?_)
```
"wallet": {
"host": "services",
"port": 8079,
"password": "<SET ME TO YOUR WALLET RPC PASSWORD>"
},
```
Set the host value for Redis:
```
"redis": {
"host": "redis",
"port": 6379
},
```
That's it! The above config files mean each element of the pool will be able to communicate with the other elements within the docker swarm, by name.
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
daemon:
image: funkypenguin/athena
volumes:
- /var/data/runtime/athena-pool/daemon/:/root/.athena
- /etc/localtime:/etc/localtime:ro
networks:
- internal
services:
image: funkypenguin/athena
volumes:
- /var/data/athena-pool/services/config:/config:ro
- /var/data/athena-pool/services/services:/services
- /var/data/athena-pool/services/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
entrypoint: |
services --config /config/services.conf | tee /logs/athena-services.log
pool:
image: funkypenguin/cryptonote-nodejs-pool
volumes:
- /var/data/athena-pool/pool/config:/config:ro
- /var/data/athena-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
- traefik_public
ports:
- 3335:3335
- 5557:5557
- 7779:7779
entrypoint: |
node init.js -config=/config/athena.json
deploy:
labels:
- traefik.frontend.rule=Host:api.athx.heigh-ho.funkypenguin.co.nz
- traefik.docker.network=traefik_public
- traefik.port=8117
redis:
volumes:
- /var/data/athena-pool/redis/config:/config:ro
- /var/data/athena-pool/redis/data:/data
- /var/data/athena-pool/redis/logs:/logs
- /etc/localtime:/etc/localtime:ro
image: redis
entrypoint: |
redis-server /config/redis.conf
networks:
- internal
nginx:
volumes:
- /var/data/athena-pool/nginx/website:/usr/share/nginx/html:ro
- /etc/localtime:/etc/localtime:ro
image: nginx
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:athx.heigh-ho.funkypenguin.co.nz
- traefik.docker.network=traefik_public
- traefik.port=80
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.26.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Athena is a go!
Launch the Athena pool stack by running ```docker stack deploy athena-pool -c <path -to-docker-compose.yml>```, and then run ```"```docker stack ps athena-pool``` to ensure the stack has come up properly. (_See [troubleshooting](/reference/troubleshooting) if not_)
The first thing that'll happen is that Athena will start syncing the blockchain. You can watch this happening with ```docker service logs athena-pool_daemon -f```. While the daemon is syncing, it won't respond to requests, so services, the pool, etc will be non-functional.
You can watch the various elements of the pool doing their thing, by running ```tail -f /var/data/athena-pool/pool/logs/*.log```
### So how do I mine to it?
That.. is another recipe. Start with the "[CryptoMiner](/recipes/cryptominer/)" uber-recipe for GPU/rig details, grab a copy of [xmr-stack](https://github.com/fireice-uk/xmr-stak), use the same parameters as TurtleCoin, and follow your nose. Jump into the Athena discord (_below_) #mining channel for help.
### What to do if it breaks?
Athena is a newborn cryptocurrency, and the [destiny of the coin is not yet clear](https://github.com/athena-network/athx-research/issues/1).
Jump into the [Athena Discord server](http://chat.athx.org) to ask questions, contribute.
## Chef's Notes
1. Because Docker Swarm performs ingress NAT for its load-balanced "routing mesh", the source address of inbound miner traffic is rewritten to a (_common_) docker node IP address. This means it's [not possible to determine the actual source IP address](https://github.com/moby/moby/issues/25526) of a miner. Which, in turn, means that any **one** misconfigured miner could trigger an IP ban, and lock out all other miners for 5 minutes at a time.
Two possible solutions to this are (1) disable banning, or (2) update the pool banning code to ban based on a combination of IP address and miner wallet address. I'll be working on a change to implement #2 if this becomes a concern.
2. The traefik labels in the docker-compose are to permit automatic LetsEncrypt SSL-protected proxying of your pool UI and API addresses.
3. Astute readers will note that although I set permissions to the "athena-pool" user above, those permissions are not **actually** enforced in the .yml file (yet)
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
Also, you could send me some ATHX ❤️ to _athena2i8SmWUGQffz6sXEdvCDawkDQvz2gdf9iBnepU999j3fUzuschJiKrow2GCTEsd5cWnk3sz2iz59WSr6NVdpvDXPbX6qj4g_
### Your comments? 💬
manuscript/recipes/cryptonote-mining-pool/masari.md
+462
View File
@@ -0,0 +1,462 @@
# Masari Mining Pool
[Cryptocurrency miners](/recipes/cryptominer) will "pool" their GPU resources ("_hashpower_") into aggregate "_mining pools_", so that by the combined effort of all the miners, the pool will receive a reward for the blocks "mined" into the blockchain, and this reward will be distributed among the miners.
![Masari Pool Screenshot](../images/masari-pool.png)
This recipe illustrates how to build a mining pool for [Masari](https://getmasari.org), one of many [CryptoNote](https://cryptonote.org/) [currencies](https://cryptonote.org/coins) (_which include [Monero](https://www.coingecko.com/en/coins/monero)_), but the principles can be applied to most mineable coins.
The end result is a mining pool which looks like this: https://msr.heigh-ho.funkypenguin.co.nz/
!!! question "Isn't this just a copy/paste of your Masari Pool Recipe?"
Why yes. Yes it is :) But it's adjusted for Masari, which uses different containers and wallet binary names, and it's running the improved [cryptonote-nodejs-pool software](https://github.com/dvandal/cryptonote-nodejs-pool), which is common to all the [cryptonote-mining-pool](/recipes/criptonote-mining-pool/) recipes!
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostnames (_pool and api_) you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
4. At least 16GB disk space (12GB used, 4GB for future growth)
## Preparation
### Create user account
The Masari pool elements won't (_and shouldn't_) run as root, but they'll need access to write data to some parts of the filesystem (_like logs, etc_).
To manage access control, we'll want to create a local user on **each docker node** with the same UID.
```
useradd -u 3507 masari-pool
```
### Setup Redis
The pool uses Redis for in-memory and persistent storage. This comes in handy for the Docker Swarm deployment, since while the various pool modules weren't _designed_ to run as microservices, the fact that they all rely on Redis for data storage makes this possible.
!!! warning "Playing it safe"
Be aware that by default, Redis stores some data **only** in memory, and writes to the filesystem at default intervals (_can be up to 5 minutes by default_). Given we don't **want** to loose 5 minutes of miner's data if we restart Redis (_what happens if we found a block during those 5 minutes but haven't paid any miners yet?_), we want to ensure that Redis runs in "appendonly" mode, which ensures that every change is immediately written to disk.
We also want to make sure that we retain all Redis logs persistently (_We're dealing with people's cryptocurrency here, it's a good idea to keep persistent logs for debugging/auditing_)
Create directories to hold Redis data. We use separate directories for future flexibility - One day, we may want to backup the data but not the logs, or move the data to an SSD partition but leave the logs on slower, cheaper disk.
```
mkdir -p /var/data/masari-pool/redis/config
mkdir -p /var/data/masari-pool/redis/data
mkdir -p /var/data/masari-pool/redis/logs
chown masari-pool /var/data/masari-pool/redis/data
chown masari-pool /var/data/masari-pool/redis/logs
```
Create **/var/data/masari-pool/redis/config/redis.conf** using http://download.redis.io/redis-stable/redis.conf as a guide. The following are the values I changed from default on my deployment (_but I'm not a Redis expert!_):
```
appendonly yes
appendfilename "appendonly.aof"
loglevel notice
logfile "/logs/redis.log"
protected-mode no
```
I also had to **disable** the following line, by commenting it out (_thus ensuring Redis container will respond to the other containers_):
```
bind 127.0.0.1
```
### Setup Nginx
We'll run a simple Nginx container to serve the static front-end of the web UI.
The simplest way to get the frontend is just to clone the upstream masari-pool repo, and mount the "/website" subdirectory into Nginx.
```
git clone https://github.com/turtlecoin/turtle-pool.git /var/data/masari-pool/nginx/
```
Edit **/var/data/masari-pool/nginx/website/config.js**, and change at least the following:
```
var api = "https://<CHOOSE A FQDN TO USE FOR YOUR API>";
var poolHost = "<SET TO THE PUBLIC DNS NAME FOR YOUR POOL SERVER";
```
### Setup Masari daemon
The first thing we'll need to participate in the great and powerful Masari network is a **node**. The node is responsible for communicating with the rest of the nodes in the blockchain, allowing our miners to receive new blocks to try to find.
Create directories to hold the blockchain data for all 3 daemons:
```
mkdir -p /var/data/runtime/masari-pool/daemon/
```
### Setup Masari wallet
Our pool needs a wallet to be able to (a) receive rewards for blocks discovered, and (b) pay out our miners for their share of the reward.
Create directories to hold wallet data
```
mkdir -p /var/data/masari-pool/wallet/config
mkdir -p /var/data/masari-pool/wallet/container
mkdir -p /var/data/masari-pool/wallet/logs
chown -R masari-pool /var/data/masari-pool/wallet/container
chown -R masari-pool /var/data/masari-pool/wallet/logs
```
Now create the initial wallet. You'll want to secure your wallet password, so the command below will **prompt** you for the key (no output from the prompt), and insert it into an environment variable. This means that the key won't be stored in plaintext in your bash history!
```
read PASS
```
After having entered your password (you can confirm by running ```env | grep PASS```), run the following to run the wallet daemon _once_, with the instruction to create a new wallet container:
```
docker run \
-v /var/data/masari-pool/wallet/wallet:/wallet \
--rm -ti --entrypoint /usr/local/bin/masari-wallet-cli funkypenguin/masari \
--password $PASS \
--generate-new-wallet /wallet/wallet \
--mnemonic-language English \
--command do-nothing-and-exit
```
You'll get a lot of output. The following are relevant lines from a successful run with the extra stuff stripped out:
```
[root@ds3 ~]# docker run \
> -v /var/data/masari-pool/wallet/wallet:/wallet \
> --rm -ti --entrypoint /usr/local/bin/masari-wallet-cli funkypenguin/masari \
> --password $PASS \
> --generate-new-wallet /wallet/wallet \
> --mnemonic-language English \
> --command do-nothing-and-exit
This is the command line masari wallet. It needs to connect to a masari daemon to work correctly.
Masari 'Classy Caiman' (v0.2.4.1-release)
Logging to /usr/local/bin/masari-wallet-cli.log
Generated new wallet: 5oUpENoBjxvjEq9Rq18cQHVhvBNGwJi9vfr36Uf5cVjx5JZNUWdHDPuFxt5sVBzuMsJcsNNEmQqvnV6UfiGuBgg4HogmEcZ
View key: d7b9d73856ba7b8e43010e3e8201d17e3aee90d2a3792439c179553229e9780f
**********************************************************************
Your wallet has been generated!
To start synchronizing with the daemon, use the "refresh" command.
Use the "help" command to see the list of available commands.
Use "help <command>" to see a command's documentation.
Always use the "exit" command when closing masari-wallet-cli to save
your current session's state. Otherwise, you might need to synchronize
your wallet again (your wallet keys are NOT at risk in any case).
NOTE: the following 25 words can be used to recover access to your wallet. Write them down and store them somewhere safe and secure. Please do not store them in your email or on file storage services outside of your immediate control.
ecstatic boil cycling bowling jeopardy fawns loudly baby
nuance token withdrawn nifty ramped taken donuts irritate
sack tedious fishing kangaroo toffee video lyrics mohawk kangaroo
**********************************************************************
Error: Unknown command: do-nothing-and-exit
[root@ds3 ~]#
```
Take careful note of your wallet password, public view key, and wallet address
Create **/var/data/masari-pool/wallet/config/wallet.conf**, containing the following:
```
wallet-file = /wallet/wallet
password = <ENTER YOUR CONTAINER PASSWORD HERE>
rpc-password = <CHOOSE A PASSWORD TO ALLOW POOL TO TALK TO WALLET>
log-file = /dev/stdout
log-level = 1
daemon-host = daemon
rpc-bind-port = 38082
```
Set permissions appropriately:
```
chown masari-pool /var/data/masari-pool/wallet/ -R
```
### Setup Masari mining pool
Following the convention we've set above, create directories to hold pool data:
```
mkdir -p /var/data/masari-pool/pool/config
mkdir -p /var/data/masari-pool/pool/logs
chown -R masari-pool /var/data/masari-pool/pool/logs
```
Now create **/var/data/masari-pool/pool/config/config.json**, using https://github.com/masaricoin/masari-pool/blob/master/config.json as a guide, and adjusting at least the following:
Send logs to /logs/, so that they can potentially be stored / backed up separately from the config:
```
"logging": {
"files": {
"level": "debug",
"directory": "/logs",
"flushInterval": 5
},
```
Set the "poolAddress" field to your wallet address
```
"poolServer": {
"enabled": true,
"clusterForks": "auto",
"poolAddress": "<SET THIS TO YOUR WALLET ADDRESS GENERATED ABOVE>",
```
Add the "host" value to the api section, since our API will run on its own container, and choose a password you'll use for the webUI admin page
```
"api": {
"enabled": true,
"hashrateWindow": 600,
"updateInterval": 5,
"host": "pool-api",
"port": 8117,
"blocks": 30,
"payments": 30,
"password": "<PASSWORD FOR ADMIN PAGE ACCESS>"
```
Set the host value for the daemon:
```
"daemon": {
"host": "daemon",
"port": 38081
},
```
Set the host value for the wallet, and set your container password (_you recorded it earlier, remember?_)
```
"wallet": {
"host": "wallet",
"port": 38082,
"password": "<SET ME TO YOUR WALLET RPC PASSWORD>"
},
```
Set the host value for Redis:
```
"redis": {
"host": "redis",
"port": 6379
},
```
That's it! The above config files mean each element of the pool will be able to communicate with the other elements within the docker swarm, by name.
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
daemon:
image: funkypenguin/masaricoind
volumes:
- /var/data/runtime/masari-pool/daemon/1:/var/lib/masaricoind/
- /etc/localtime:/etc/localtime:ro
networks:
- internal
- traefik_public
ports:
- 11897:11897
labels:
- traefik.frontend.rule=Host:explorer.trtl.heigh-ho.funkypenguin.co.nz
- traefik.docker.network=traefik_public
- traefik.port=11898
daemon-failsafe1:
image: funkypenguin/masaricoind
volumes:
- /var/data/runtime/masari-pool/daemon/failsafe1:/var/lib/masaricoind/
- /etc/localtime:/etc/localtime:ro
networks:
- internal
daemon-failsafe2:
image: funkypenguin/masaricoind
volumes:
- /var/data/runtime/masari-pool/daemon/failsafe2:/var/lib/masaricoind/
- /etc/localtime:/etc/localtime:ro
networks:
- internal
pool-pool:
image: funkypenguin/turtle-pool
volumes:
- /var/data/masari-pool/pool/config:/config:ro
- /var/data/masari-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
ports:
- 3333:3333
- 5555:5555
- 7777:7777
entrypoint: |
node init.js -module=pool -config=/config/config.json
pool-api:
image: funkypenguin/turtle-pool
volumes:
- /var/data/masari-pool/pool/config:/config:ro
- /var/data/masari-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:api.trtl.heigh-ho.funkypenguin.co.nz
- traefik.docker.network=traefik_public
- traefik.port=8117
entrypoint: |
node init.js -module=api -config=/config/config.json
pool-unlocker:
image: funkypenguin/turtle-pool
volumes:
- /var/data/masari-pool/pool/config:/config:ro
- /var/data/masari-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
entrypoint: |
node init.js -module=unlocker -config=/config/config.json
pool-payments:
image: funkypenguin/turtle-pool
volumes:
- /var/data/masari-pool/pool/config:/config:ro
- /var/data/masari-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
entrypoint: |
node init.js -module=payments -config=/config/config.json
pool-charts:
image: funkypenguin/turtle-pool
volumes:
- /var/data/masari-pool/pool/config:/config:ro
- /var/data/masari-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
entrypoint: |
node init.js -module=chartsDataCollector -config=/config/config.json
wallet:
image: funkypenguin/masari
volumes:
- /var/data/masari-pool/wallet/config:/config:ro
- /var/data/masari-pool/wallet/container:/container
- /var/data/masari-pool/wallet/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
entrypoint: |
walletd --config /config/wallet.conf | tee /logs/walletd.log
redis:
volumes:
- /var/data/masari-pool/redis/config:/config:ro
- /var/data/masari-pool/redis/data:/data
- /var/data/masari-pool/redis/logs:/logs
- /etc/localtime:/etc/localtime:ro
image: redis
entrypoint: |
redis-server /config/redis.conf
networks:
- internal
nginx:
volumes:
- /var/data/masari-pool/nginx/website:/usr/share/nginx/html:ro
- /etc/localtime:/etc/localtime:ro
image: nginx
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:trtl.heigh-ho.funkypenguin.co.nz
- traefik.docker.network=traefik_public
- traefik.port=80
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.21.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch the Turtle! 🐢
Launch the Turtle pool stack by running ```docker stack deploy masari-pool -c <path -to-docker-compose.yml>```, and then run ```"```docker stack ps masari-pool``` to ensure the stack has come up properly. (_See [troubleshooting](/reference/troubleshooting) if not_)
The first thing that'll happen is that Masarid will start syncing the blockchain from the bootstrap data. You can watch this happening with ```docker service logs masari-pool_daemon -f```. While the daemon is syncing, it won't respond to requests, so walletd, the pool, etc will be non-functional.
You can watch the various elements of the pool doing their thing, by running ```tail -f /var/data/masari-pool/pool/logs/*.log```
### So how do I mine to it?
That.. is another recipe. Start with the "[CryptoMiner](/recipes/cryptominer/)" uber-recipe for GPU/rig details, grab a copy of xmr-stack (_patched for the forked Masari_) from https://github.com/masaricoin/trtl-stak/releases, and follow your nose. Jump into the Masari discord (_below_) #mining channel for help.
### What to do if it breaks?
Masari is a baby cryptocurrency. There are scaling issues to solve, and large amounts components of this recipe are under rapid development. So, elements may break/change in time, and this recipe itself is a work-in-progress.
Jump into the [Masari Discord server](http://chat.masaricoin.lol/) to ask questions, contribute, and send/receive some TRTL tips!
## Chef's Notes
1. Because Docker Swarm performs ingress NAT for its load-balanced "routing mesh", the source address of inbound miner traffic is rewritten to a (_common_) docker node IP address. This means it's [not possible to determine the actual source IP address](https://github.com/moby/moby/issues/25526) of a miner. Which, in turn, means that any **one** misconfigured miner could trigger an IP ban, and lock out all other miners for 5 minutes at a time.
Two possible solutions to this are (1) disable banning, or (2) update the pool banning code to ban based on a combination of IP address and miner wallet address. I'll be working on a change to implement #2 if this becomes a concern.
2. The traefik labels in the docker-compose are to permit automatic LetsEncrypt SSL-protected proxying of your pool UI and API addresses.
3. After a [power fault in my datacenter caused daemon DB corruption](https://www.reddit.com/r/TRTL/comments/8jftzt/funky_penguin_nz_mining_pool_down_with_daemon/), I added a second daemon, running in parallel to the first. The failsafe daemon runs once an hour, syncs with the running daemons, and shuts down again, providing a safely halted version of the daemon DB for recovery.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
Also, you could send me some :masari: ❤️ to _TRTLv2qCKYChMbU5sNkc85hzq2VcGpQidaowbnV2N6LAYrFNebMLepKKPrdif75x5hAizwfc1pX4gi5VsR9WQbjQgYcJm21zec4_
### Your comments? 💬
manuscript/recipes/duplicity.md
+172
View File
@@ -0,0 +1,172 @@
hero: Duplicity - A boring recipe to backup your exciting stuff. Boring is good.
# Duplicity
Intro
![Duplicity Screenshot](../images/duplicity.png)
[Duplicity](http://duplicity.nongnu.org/) backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.
So what does this mean for our stack? It means we can leverage Duplicity to backup all our data-at-rest to a wide variety of cloud providers, including, but not limited to:
- acd_cli
- Amazon S3
- Backblaze B2
- DropBox
- ftp
- Google Docs
- Google Drive
- Microsoft Azure
- Microsoft Onedrive
- Rackspace Cloudfiles
- rsync
- ssh/scp
- SwiftStack
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. Credentials for one of the Duplicity's supported upload destinations
## Preparation
### Setup data locations
We'll need a folder to store a docker-compose .yml file, and an associated .env file. If you're following my filesystem layout, create /var/data/config/duplicity (for the config), and /var/data/duplicity (for the metadata) as follows:
```
mkdir /var/data/config/duplicity
mkdir /var/data/duplicity
cd /var/data/config/duplicity
```
### (Optional) Create Google Cloud Storage bucket
I didn't already have an archival/backup provider, so I chose Google Cloud "cloud" storage for the low price-point - 0.7 cents per GB/month (_Plus you [start with $300 credit](https://cloud.google.com/free/) even when signing up for the free tier_). You can use any destination supported by [Duplicity's URL scheme though](http://duplicity.nongnu.org/duplicity.1.html#sect7), just make sure you specify the necessary [environment variables](http://duplicity.nongnu.org/duplicity.1.html#sect6).
1. [Sign up](https://cloud.google.com/storage/docs/getting-started-console), create an empty project, enable billing, and create a bucket. Give your bucket a unique name, example "**jack-and-jills-bucket**" (_it's unique across the entire Google Cloud_)
2. Under "Storage" section > "[Settings](https://console.cloud.google.com/project/_/storage/settings)" > "Interoperability" tab > click "Enable interoperable access" and then "Create a new key" button and note both Access Key and Secret.
### Prepare environment
1. Generate a random passphrase to use to encrypt your data. **Save this somewhere safe**, without it you won't be able to restore!
2. Seriously, **save**. **it**. **somewhere**. **safe**.
3. Create duplicity.env, and populate with the following variables
```
SRC=/var/data/
DST=gs://jack-and-jills-bucket/yes-you-can-have-subdirectories
TMPDIR=/tmp
GS_ACCESS_KEY_ID=<YOUR GS ACCESS KEY>
GS_SECRET_ACCESS_KEY=<YOUR GS SECRET ACCESS KEY>
OPTIONS=--allow-source-mismatch --exclude /var/data/runtime --exclude /var/data/registry --exclude /var/data/duplicity --archive-dir=/archive
PASSPHRASE=<YOUR CHOSEN PASSPHRASE>
```
!!! note
See the [data layout reference](/reference/data_layout/) for an explanation of the included/excluded paths above.
### Run a test backup
Before we launch the automated daily backups, let's run a test backup, as follows:
```
docker run --env-file duplicity.env -it --rm -v \
/var/data:/var/data:ro -v /var/data/duplicity/tmp:/tmp -v \
/var/data/duplicity/archive:/archive tecnativa/duplicity \
/etc/periodic/daily/jobrunner
```
You should see some activity, with a summary of bytes transferred at the end.
### Run a test restore
Repeat after me: "If you don't verify your backup, **it's not a backup**".
!!! warning
Depending on what tier of storage you chose from your provider (_i.e., Google Coldline, or Amazon S3_), you may be charged for downloading data.
Run a variation of the following to confirm a file you expect to be backed up, **is** backed up. (_I used traefik.yml from the [traefik recipie](/recipie/traefik/), since this is likely to exist for every reader_).
```
docker run --env-file duplicity.env -it --rm \
-v /var/data:/var/data:ro \
-v /var/data/duplicity/tmp:/tmp \
-v /var/data/duplicity/archive:/archive tecnativa/duplicity \
duplicity list-current-files \
\$DST | grep traefik.yml
```
Once you've identified a file to test-restore, use a variation of the following to restore it to /tmp (_from the perspective of the container - it's actually /var/data/duplicity/tmp_)
```
docker run --env-file duplicity.env -it --rm \
-v /var/data:/var/data:ro \
-v /var/data/duplicity/tmp:/tmp \
-v /var/data/duplicity/archive:/archive \
tecnativa/duplicity duplicity restore \
--file-to-restore config/traefik/traefik.yml \
\$DST /tmp/traefik-restored.yml
```
Examine the contents of /var/data/duplicity/tmp/traefik-restored.yml to confirm it contains valid data.
### Setup Docker Swarm
Now that we have confidence in our backup/restore process, let's automate it by creating a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: "3"
services:
backup:
image: tecnativa/duplicity
env_file: /var/data/config/duplicity/duplicity.env
networks:
- internal
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/data:/var/data:ro
- /var/data/duplicity/tmp:/tmp
- /var/data/duplicity/archive:/archive
networks:
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.10.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Duplicity stack
Launch Duplicity stack by running ```docker stack deploy duplicity -c <path -to-docker-compose.yml>```
Nothing will happen. Very boring. But when the cron script fires (daily), duplicity will do its thing, and backup everything in /var/data to your cloud destination.
## Chef's Notes
1. Automatic backup can still fail if nobody checks that it's running successfully. I'll be working on an upcoming recipe to monitor the elements of the stack, including the success/failure of duplicity jobs.
2. The container provides the facility to specify an SMTP host and port, but not credentials, which makes it close to useless. As a result, I've left SMTP out of this recipe. To enable email notifications (if your SMTP server doesn't require auth), add ```SMTP_HOST```, ```SMTP_PORT```, ```EMAIL_FROM``` and ```EMAIL_TO``` variables to duplicity.env
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/emby.md
+96
View File
@@ -0,0 +1,96 @@
# Emby
[Emby](https://emby.media/) (_think "M.B." or "Media Browser"_) is best described as "_like [Plex](/recipes/plex/) but different_" 😁 - It's a bit geekier and less polished than Plex, but it allows for more flexibility and customization.
![Emby Screenshot](../images/emby.png)
I've started experimenting with Emby as an alternative to Plex, because of the advanced [parental controls](https://github.com/MediaBrowser/Wiki/wiki/Parental-Controls) it offers. Based on my experimentation thus far, I have a "**kid-safe**" profile which automatically logs in, and only displays kid-safe content, based on ratings.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need a location to store Emby's library data, config files, logs and temporary transcoding space, so create /var/data/emby, and make sure it's owned by the user and group who also own your media data.
```
mkdir /var/data/emby
```
### Prepare environment
Create emby.env, and populate with PUID/GUID for the user who owns the /var/data/emby directory (_above_) and your actual media content (_in this example, the media content is at **/srv/data**_)
```
PUID=
GUID=
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: "3.0"
services:
emby:
image: emby/emby-server
env_file: /var/data/config/emby/emby.env
volumes:
- /var/data/emby/emby:/config
- /srv/data/:/data
deploy:
labels:
- traefik.frontend.rule=Host:emby.example.com
- traefik.docker.network=traefik_public
- traefik.port=8096
networks:
- traefik_public
- internal
ports:
- 8096:8096
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.17.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Emby stack
Launch the stack by running ```docker stack deploy emby -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, and complete the wizard-based setup to complete deploying your Emby.
## Chef's Notes
1. I didn't use an [oauth2_proxy](/reference/oauth_proxy/) for this stack, because it would interfere with mobile client support.
2. Got an NVIDIA GPU? See [this blog post](https://www.funkypenguin.co.nz/note/gpu-transcoding-with-emby-plex-using-docker-nvidia/) re how to use your GPU to transcode your media!
3. We don't bother exposing the HTTPS port for Emby, since [Traefik](/ha-docker-swarm/traefik/) is doing the SSL termination for us already.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/flightairmap.md
+60
View File
@@ -0,0 +1,60 @@
version: '3'
services:
flightairmap:
image: richarvey/nginx-php-fpm
volumes:
- "/var/data/flightairmap/conf:/var/www/html/conf"
- "/var/data/flightairmap/scripts:/var/www/html/scripts"
- "/var/data/flightairmap/html:/var/www/flightairmap/"
env_file:
- "/var/data/config/flightairmap/flightairmap.env"
environment:
- PHP_MEM_LIMIT=256
- RUN_SCRIPTS=1
- MYSQL_HOST=${MYSQL_HOST}
- MYSQL_DATABASE=${MYSQL_DATABASE}
- MYSQL_USER=${MYSQL_USER}
- MYSQL_PASSWORD=${MYSQL_PASSWORD}
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:www.observe.global
- traefik.docker.network=traefik_public
- traefik.port=80
db:
image: mariadb:10
env_file: /var/data/config/flightairmap/flightairmap.env
networks:
- internal
volumes:
- /var/data/runtime/flightairmap/db:/var/lib/mysql
db-backup:
image: mariadb:10
env_file: /var/data/config/flightairmap/flightairmap.env
volumes:
- /var/data/flightairmap/database-dump:/dump
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
sleep 2m
while /bin/true; do
mysqldump -h db --all-databases | gzip -c > /dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.sql.gz
(ls -t /dump/dump*.sql.gz|head -n $$BACKUP_NUM_KEEP;ls /dump/dump*.sql.gz)|sort|uniq -u|xargs rm -- {}
sleep $$BACKUP_FREQUENCY
done
EOF'
networks:
- internal
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.44.0/24
manuscript/recipes/ghost.md
+77
View File
@@ -0,0 +1,77 @@
hero: Ghost - A recipe for beautiful online publication.
# Ghost
[Ghost](https://ghost.org) is "a fully open source, hackable platform for building and running a modern online publication."
![](/images/ghost.png)
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
## Preparation
### Setup data locations
Create the location for the bind-mount of the application data, so that it's persistent:
```
mkdir -p /var/data/ghost
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
ghost:
image: ghost:1-alpine
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/data/ghost/:/var/lib/ghost/content
networks:
- traefik
deploy:
labels:
- traefik.frontend.rule=Host:ghost.example.com
- traefik.docker.network=traefik
- traefik.port=2368
networks:
traefik:
external: true
```
## Serving
### Launch Ghost stack
Launch the Ghost stack by running ```docker stack deploy ghost -c <path -to-docker-compose.yml>```
Create your first administrative account at https://**YOUR-FQDN**/admin/
## Chef's Notes
1. If I wasn't committed to a [static-site-generated blog](https://www.funkypenguin.co.nz/blog/), Ghost is the platform I'd use for my blog.
2. A default using the SQlite database takes 548k of space:
```
[root@ds1 ghost]# du -sh /var/data/ghost/
548K /var/data/ghost/
[root@ds1 ghost]#
```
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/gitlab-runner.md
+102
View File
@@ -0,0 +1,102 @@
# Gitlab Runner
Some features of GitLab require a "[runner](https://docs.gitlab.com/runner/)" (_in the sense of a "gopher" or a "minion"_). A runner "registers" itself with a GitLab instance, and is given tasks to run. Tasks include running Continuous Integration (CI) builds, and building container images.
While a runner isn't strictly required to use GitLab, if you want to do CI, you'll need at least one. There are many ways to deploy a runner - this recipe focuses on the docker container model.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [GitLab](/ha-docker-swarm/gitlab) installation (see previous recipe)
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our runner containers, so create them in /var/data/gitlab:
```
cd /var/data
mkdir gitlab
cd gitlab
mkdir -p {runners/1,runners/2}
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
1:
image: gitlab/gitlab-runner
volumes:
- /var/data/gitlab-runner/1:/var/data/gitlab/runners/1
networks:
- internal
2:
image: gitlab/gitlab-runner
volumes:
- /var/data/gitlab-runner/:/var/data/gitlab/runners/2
networks:
- internal
networks:
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.23.0/24
```
### Configure runners
From your GitLab UI, you can retrieve a "token" necessary to register a new runner. To register the runner, you can either create config.toml in each runner's bind-mounted folder (example below), or just "docker exec" into each runner container and execute ```gitlab-container register``` to interactively generate config.toml.
Sample runner config.toml:
```
concurrent = 1
check_interval = 0
[[runners]]
name = "myrunner1"
url = "https://gitlab.example.com"
token = "<long string here>"
executor = "docker"
[runners.docker]
tls_verify = false
image = "ruby:2.1"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
```
## Serving
### Launch runners
Launch the mail server stack by running ```docker stack deploy gitlab-runner -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user "root" and the password you specified in gitlab.env.
## Chef's Notes
1. You'll note that I setup 2 runners. One is locked to a single project (this cookbook build), and the other is a shared runner. I wanted to ensure that one runner was always available to run CI for this project, even if I'd tied up another runner on something heavy-duty, like a container build. Customize this to your use case.
2. Originally I deployed runners in the same stack as GitLab, but I found that they would frequently fail to start properly when I launched the stack. I think that this was because the runners started so quickly (and GitLab starts so slowly!), that they always started up reporting that the GitLab instance was invalid or unavailable. I had issues with CI builds stuck permanently in a "pending" state, which were only resolved by restarting the runner. Having the runners deployed in a separate stack to GitLab avoids this problem.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/gitlab.md
+143
View File
@@ -0,0 +1,143 @@
hero: Gitlab - A recipe for a self-hosted GitHub alternative
# GitLab
GitLab is a self-hosted [alternative to GitHub](https://about.gitlab.com/comparison/). The most common use case is (a set of) developers with the desire for the rich feature-set of GitHub, but with unlimited private repositories.
Docker does maintain an [official "Omnibus" container](https://docs.gitlab.com/omnibus/docker/README.html), but for this recipe I prefer the "[dockerized gitlab](https://github.com/sameersbn/docker-gitlab)" project, since it allows distribution of the various Gitlab components across multiple swarm nodes.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/gitlab:
```
cd /var/data
mkdir gitlab
cd gitlab
mkdir -p {postgresql,redis,gitlab}
```
### Prepare environment
You'll need to know the following:
1. Choose a password for postgresql, you'll need it for DB_PASS in the compose file (below)
2. Generate 3 passwords using ```pwgen -Bsv1 64```. You'll use these for the XXX_KEY_BASE environment variables below
2. Create gitlab.env, and populate with **at least** the following variables (the full set is available at https://github.com/sameersbn/docker-gitlab#available-configuration-parameters):
```
DB_USER=gitlab
DB_PASS=gitlabdbpass
DB_NAME=gitlabhq_production
DB_EXTENSION=pg_trgm
DB_ADAPTER=postgresql
DB_HOST=postgresql
TZ=Pacific/Auckland
REDIS_HOST=redis
REDIS_PORT=6379
GITLAB_TIMEZONE=Auckland
GITLAB_HTTPS=true
SSL_SELF_SIGNED=false
GITLAB_HOST=gitlab.example.com
GITLAB_PORT=443
GITLAB_SSH_PORT=2222
GITLAB_SECRETS_DB_KEY_BASE=CFf7sS3kV2nGXBtMHDsTcjkRX8PWLlKTPJMc3lRc6GCzJDdVljZ85NkkzJ8mZbM5
GITLAB_SECRETS_SECRET_KEY_BASE=h2LBVffktDgb6BxM3B97mDSjhnSNwLc5VL2Hqzq9cdrvBtVw48WSp5wKj5HZrJM5
GITLAB_SECRETS_OTP_KEY_BASE=t9LPjnLzbkJ7Nt6LZJj6hptdpgG58MPJPwnMMMDdx27KSwLWHDrz9bMWXQMjq5mp
GITLAB_ROOT_PASSWORD=changeme
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
````
version: '3'
services:
redis:
image: sameersbn/redis:latest
command:
- --loglevel warning
volumes:
- /var/data/gitlab/redis:/var/lib/redis:Z
networks:
- internal
postgresql:
image: sameersbn/postgresql:9.6-2
env_file: /var/data/config/gitlab/gitlab.env
volumes:
- /var/data/gitlab/postgresql:/var/lib/postgresql:Z
networks:
- internal
gitlab:
image: sameersbn/gitlab:latest
env_file: /var/data/config/gitlab/gitlab.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:gitlab.example.com
- traefik.docker.network=traefik
- traefik.port=80
restart_policy:
delay: 10s
max_attempts: 10
window: 60s
ports:
- "2222:22"
volumes:
- /var/data/gitlab/gitlab:/home/git/data:Z
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.2.0/24
````
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch gitlab
Launch the mail server stack by running ```docker stack deploy gitlab -c <path -to-docker-compose.yml>```
Log into your new instance at https://[your FQDN], with user "root" and the password you specified in gitlab.env.
## Chef's Notes
A few comments on decisions taken in this design:
1. I use the **sameersbn/gitlab:latest** image, rather than a specific version. This lets me execute updates simply by redeploying the stack (and why **wouldn't** I want the latest version?)
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/gollum.md
+136
View File
@@ -0,0 +1,136 @@
hero: Gollum - A recipe for your own git-based wiki
# Gollum
Gollum is a simple wiki system built on top of Git. A Gollum Wiki is simply a git repository (_either bare or regular_) of a specific nature:
* A Gollum repository's contents are human-editable, unless the repository is bare.
* Pages are unique text files which may be organized into directories any way you choose.
* Other content can also be included, for example images, PDFs and headers/footers for your pages.
Gollum pages:
* May be written in a variety of markups.
* Can be edited with your favourite system editor or IDE (_changes will be visible after committing_) or with the built-in web interface.
* Can be displayed in all versions (_commits_).
![Gollum Screenshot](../images/gollum.png)
As you'll note in the (_real world_) screenshot above, my requirements for a personal wiki are:
* Portable across my devices
* Supports images
* Full-text search
* Supports inter-note links
* Revision control
Gollum meets all these requirements, and as an added bonus, is extremely fast and lightweight.
!!! note
Since Gollum itself offers no user authentication, this design secures gollum behind an [oauth2 proxy](/reference/oauth_proxy/), so that in order to gain access to the Gollum UI at all, oauth2 authentication (_to GitHub, GitLab, Google, etc_) must have already occurred.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
## Preparation
### Setup data locations
We'll need an empty git repository in /var/data/gollum for our data:
```
mkdir /var/data/gollum
cd /var/data/gollum
git init
```
### Prepare environment
1. Choose an oauth provider, and obtain a client ID and secret
2. Create gollum.env, and populate with the following variables (_you can make the cookie secret whatever you like_)
```
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
app:
image: dakue/gollum
volumes:
- /var/data/gollum:/gollum
networks:
- internal
command: |
--allow-uploads
--emoji
--user-icons gravatar
proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/gollum/gollum.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:gollum.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/gollum/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://app:4567
-redirect-url=https://gollum.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.9.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Gollum stack
Launch the Gollum stack by running ```docker stack deploy gollum -c <path-to-docker-compose.yml>```
Authenticate against your OAuth provider, and then start editing your wiki!
## Chef's Notes
1. In the current implementation, Gollum is a "single user" tool only. The contents of the wiki are saved as markdown files under /var/data/gollum, and all the git commits are currently "Anonymous"
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/homeassistant.md
+139
View File
@@ -0,0 +1,139 @@
# Home Assistant
Home Assistant is a home automation platform written in Python, with extensive support for 3rd-party home-automation platforms including Xaomi, Phillips Hue, and a [bazillion](https://home-assistant.io/components/) others.
![Home Assistant Screenshot](../images/homeassistant.png)
This recipie combines the [extensibility](https://home-assistant.io/components/) of [Home Assistant](https://home-assistant.io/) with the flexibility of [InfluxDB](https://docs.influxdata.com/influxdb/v1.4/) (_for time series data store_) and [Grafana](https://grafana.com/) (_for **beautiful** visualisation of that data_).
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/homeassistant:
```
mkdir /var/data/homeassistant
cd /var/data/homeassistant
mkdir -p {homeassistant,grafana,influxdb-backup}
```
Now create a directory for the influxdb realtime data:
```
mkdir /var/data/runtime/homeassistant/influxdb
```
### Prepare environment
Create /var/data/config/homeassistant/grafana.env, and populate with the following - this is to enable grafana to work with oauth2_proxy without requiring an additional level of authentication:
```
GF_AUTH_BASIC_ENABLED=false
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: "3"
services:
influxdb:
image: influxdb
networks:
- internal
volumes:
- /var/data/homeassistant/influxdb:/var/lib/influxdb
- /etc/localtime:/etc/localtime:ro
homeassistant:
image: homeassistant/home-assistant
dns_search: hq.example.com
volumes:
- /var/data/homeassistant/homeassistant:/config
- /etc/localtime:/etc/localtime:ro
deploy:
labels:
- traefik.frontend.rule=Host:homeassistant.example.com
- traefik.docker.network=traefik_public
- traefik.port=8123
networks:
- traefik_public
- internal
ports:
- 8123:8123
grafana-app:
image: grafana/grafana
env_file : /var/data/config/homeassistant/grafana.env
volumes:
- /var/data/homeassistant/grafana:/var/lib/grafana
- /etc/localtime:/etc/localtime:ro
networks:
- internal
grafana-proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/homeassistant/grafana.env
dns_search: hq.example.com
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:grafana.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/homeassistant/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://grafana-app:3000
-redirect-url=https://grafana.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.13.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Home Assistant stack
Launch the Home Assistant stack by running ```docker stack deploy homeassistant -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, the password you created in configuration.yml as "frontend - api_key". Then setup a bunch of sensors, and log into https://grafana.**YOUR FQDN** and create some beautiful graphs :)
## Chef's Notes
1. I **tried** to protect Home Assistant using [oauth2_proxy](/reference/oauth_proxy), but HA is incompatible with the websockets implementation used by Home Assistant. Until this can be fixed, I suggest that geeks set frontend: api_key to a long and complex string, and rely on this to prevent malevolent internet miscreants from turning their lights on at 2am!
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/huginn.md
+153
View File
@@ -0,0 +1,153 @@
hero: Huginn - A recipe for self-hosted, hackable version of IFFTT / Zapier
# Huginn
Huginn is a system for building agents that perform automated tasks for you online. They can read the web, watch for events, and take actions on your behalf. Huginn's Agents create and consume events, propagating them along a directed graph. Think of it as a hackable version of IFTTT or Zapier on your own server.
<iframe src="https://player.vimeo.com/video/61976251" width="640" height="433" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
## Preparation
### Setup data locations
Create the location for the bind-mount of the database, so that it's persistent:
```
mkdir -p /var/data/huginn/database
```
### Create email address
Strictly speaking, you don't **have** to integrate Huginn with email. However, since we created our own mailserver stack earlier, it's worth using it to enable emails within Huginn.
```
cd /var/data/docker-mailserver/
./setup.sh email add huginn@huginn.example.com my-password-here
# Setup MX and DKIM if they don't already exist:
./setup.sh config dkim
cat config/opendkim/keys/huginn.example.com/mail.txt
```
### Prepare environment
Create /var/data/huginn/huginn.env, and populate with the following variables. Set the "INVITATION_CODE" variable if you want to require users to enter a code to sign up (protects the UI from abuse) (The full list of Huginn environment variables is available [here](https://github.com/huginn/huginn/blob/master/.env.example))
```
# For huginn/huginn - essential
SMTP_DOMAIN=your-domain-here.com
SMTP_USER_NAME=you@gmail.com
SMTP_PASSWORD=somepassword
SMTP_SERVER=your-mailserver-here.com
SMTP_PORT=587
SMTP_AUTHENTICATION=plain
SMTP_ENABLE_STARTTLS_AUTO=true
INVITATION_CODE=<set an invitation code here>
POSTGRES_PORT_5432_TCP_ADDR=db
POSTGRES_PORT_5432_TCP_PORT=5432
DATABASE_USERNAME=huginn
DATABASE_PASSWORD=<database password>
DATABASE_ADAPTER=postgresql
# Optional extras for huginn/huginn, customize or append based on .env.example lined above
TWITTER_OAUTH_KEY=
TWITTER_OAUTH_SECRET=
# For postgres/postgres
POSTGRES_USER=huginn
POSTGRES_PASSWORD=<database password>
BACKUP_NUM_KEEP=7
BACKUP_FREQUENCY=1d
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
huginn:
image: huginn/huginn
env_file: /var/data/config/huginn/huginn.env
volumes:
- /etc/localtime:/etc/localtime:ro
networks:
- internal
- traefik
deploy:
labels:
- traefik.frontend.rule=Host:huginn.funkypenguin.co.nz
- traefik.docker.network=traefik
- traefik.port=3000
db:
env_file: /var/data/config/huginn/huginn.env
image: postgres:latest
volumes:
- /var/data/runtime/huginn/database:/var/lib/postgresql/data
- /etc/localtime:/etc/localtime:ro
networks:
- internal
db-backup:
image: postgres:latest
env_file: /var/data/config/huginn/huginn.env
volumes:
- /var/data/huginn/database-dump:/dump
- /etc/localtime:/etc/localtime:ro
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
sleep 2m
while /bin/true; do
pg_dump -Fc > /dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.psql
(ls -t /dump/dump*.psql|head -n $$BACKUP_NUM_KEEP;ls /dump/dump*.psql)|sort|uniq -u|xargs rm -- {}
sleep $$BACKUP_FREQUENCY
done
EOF'
networks:
- internal
networks:
traefik:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.6.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Huginn stack
Launch the Huginn stack by running ```docker stack deploy huginn -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**. You'll need to use the "Sign Up" button, and (optionally) enter your invitation code in order to create your account.
## Chef's Notes
1. I initially considered putting an oauth proxy in front of Huginn, but since the invitation code logic prevents untrusted access, and since using a proxy would break oauth for sevices like Twitter integration, I left it out.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/instapy.md
+136
View File
@@ -0,0 +1,136 @@
hero: Automate your social life! (what could go wrong?)
# InstaPy
[InstaPy](https://github.com/timgrossmann/InstaPy) is an Instagram bot, developed by [Tim Grossman](https://github.com/timgrossmann). Tim describes his motivation and experiences developing the bot [here](https://medium.freecodecamp.org/my-open-source-instagram-bot-got-me-2-500-real-followers-for-5-in-server-costs-e40491358340).
What's an Instagram bot? Basically, you feed the bot your Instagram user/password, and it executes follows/unfollows/likes/comments on your behalf based on rules you set. (_I set my bot to like one photo tagged with "[#penguin](https://www.instagram.com/explore/tags/penguin/?hl=en)" per-run_)
![InstaPy Screenshot](../images/instapy.png)
Great power, right? A client (_yes, you can [hire](https://www.funkypenguin.co.nz/contact/) me!_) asked me to integrate InstaPy into their swarm, and this recipe is the result.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
## Preparation
### Setup data locations
We need a data location to store InstaPy's config, as well as its log files. Create /var/data/instapy per below
```
mkdir -p /var/data/instapy/logs
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
web:
command: ["./wait-for", "selenium:4444", "--", "python", "docker_quickstart.py"]
environment:
- PYTHONUNBUFFERED=0
# Modify the image to whatever Tim's image tag ends up as. I used funkypenguin/instapy for my build
image: funkypenguin/instapy:latest
# When using swarm, you can't use relative paths, so the following needs to be set to the full filesystem path to your logs and docker_quickstart.py
# Bind-mount docker_quickstart.py, since now that we're using a public image, we can't "bake" our credentials into the image anymore
volumes:
- /var/data/instapy/logs:/code/logs
- var/data/instapy/instapy.py:/code/docker_quickstart.py:ro
# This section allows docker to restart the container when it exits (either normally or abnormally), which ensures that
# InstaPy keeps re-running. Tweak the delay to avoid being banned for excessive activity
deploy:
restart_policy:
condition: any
delay: 3600s
selenium:
image: selenium/standalone-chrome-debug
ports:
- "5900:5900"
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
### Command your bot
Create a variation of https://github.com/timgrossmann/InstaPy/blob/master/docker_quickstart.py at /var/data/instapy/instapy.py (the file we bind-mounted in the swarm config above)
Change at least the following:
````
insta_username = ''
insta_password = ''
````
Here's an example of my config, set to like a single penguin-pic per run:
```
insta_username = 'funkypenguin'
insta_password = 'followmemypersonalbrandisawesome'
dont_like = ['food','girl','batman','gotham','dead','nsfw','porn','slut','baby','tv','athlete','nhl','hockey','estate','music','band','clothes']
friend_list = ['therock','ruinporn']
# If you want to enter your Instagram Credentials directly just enter
# username=<your-username-here> and password=<your-password> into InstaPy
# e.g like so InstaPy(username="instagram", password="test1234")
bot = InstaPy(username='insta_username', password='insta_password', selenium_local_session=False)
bot.set_selenium_remote_session(selenium_url='http://selenium:4444/wd/hub')
bot.login()
bot.set_upper_follower_count(limit=10000)
bot.set_lower_follower_count(limit=10)
bot.set_comments([u'Cool :penguin:!', u'Awesome :penguin:!!', u'Nice :penguin:!!'])
bot.set_dont_include(friend_list)
bot.set_dont_like(dont_like)
#bot.set_ignore_if_contains(ignore_words)
# OK, so go through my feed and like stuff, interacting with people I follow
# bot.like_by_feed(amount=3, randomize=True, unfollow=True, interact=True)
# Now find posts tagged as #penguin, and like 'em, commenting 50% of the time
bot.set_do_comment(True, percentage=50)
bot.set_comments([u'Cool :penguin:!', u'Awesome :penguin:!!', u'Nice :penguin:!!'])
bot.like_by_tags(['#penguin'], amount=1)
# goodnight, sweet bot
bot.end()
```
## Serving
### Destroy all humans
Launch the bot by running ```docker stack deploy instapy -c <path -to-docker-compose.yml>```
While you're waiting for Docker to pull down the images, educate yourself on the risk of a robotic uprising:
<iframe width="560" height="315" src="https://www.youtube.com/embed/B1BdQcJ2ZYY" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
After swarm deploys, you won't see much, but you can monitor what InstaPy is doing, by running ```docker service logs instapy_web```.
You can **also** watch the bot at work by VNCing to your docker swarm, password "secret". You'll see Selenium browser window cycling away, interacting with all your real/fake friends on Instagram :)
## Chef's Notes
1. Amazingly, my bot has ended up tagging more _non-penguins_ than actual penguins. I don't understand how Instagrammers come up with their hashtags!
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/kanboard.md
+128
View File
@@ -0,0 +1,128 @@
hero: Kanboard - A recipe to get your personal kanban on
# Kanboard
Kanboard is a Kanban tool, developed by [Frédéric Guillot](https://github.com/fguillot). (_Who also happens to be the developer of my favorite RSS reader, [Miniflux](/recipes/miniflux/)_)
!!! tip "Sponsored Project"
Kanboard is one of my [sponsored projects](/sponsored-projects/) - a project I financially support on a regular basis because of its utility to me. I use it both in my DayJob(tm), and to manage my overflowing, overly-optimistic personal commitments! 😓
Features include:
* Visualize your work
* Limit your work in progress to be more efficient
* Customize your boards according to your business activities
* Multiple projects with the ability to drag and drop tasks
* Reports and analytics
* Fast and simple to use
* Access from anywhere with a modern browser
* Plugins and integrations with external services
* Free, open source and self-hosted
* Super simple installation
![](/images/kanboard.png)
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry pointing your NextCloud url (_kanboard.example.com_) to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
Create the location for the bind-mount of the application data, so that it's persistent:
```
mkdir -p /var/data/kanboard
```
### Setup Environment
If you intend to use an [OAuth proxy](/reference/oauth_proxy/) to further secure public access to your instance, create a ```kanboard.env``` file to hold your environment variables, and populate with your OAuth provider's details (_the cookie secret you can just make up_):
```
# If you decide to protect kanboard with an oauth_proxy, complete these
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
kanboard:
image: kanboard/kanboard
volumes:
- /var/data/kanboard/data:/var/www/app/data
- /var/data/kanboard/plugins:/var/www/app/plugins
networks:
- internal
deploy:
labels:
- traefik.frontend.rule=Host:kanboard.example.com
- traefik.docker.network=traefik_public
- traefik.port=80
proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/kanboard/kanboard.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:kanboard.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/kanboard/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://app
-redirect-url=https://kanboard.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.8.0/24
```
## Serving
### Launch Kanboard stack
Launch the Kanboard stack by running ```docker stack deploy kanboard -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**. Default credentials are admin/admin, after which you can change (_under 'profile'_) and add more users.
## Chef's Notes
1. The default theme can be significantly improved by applying the [ThemePlus](https://github.com/phsteffen/kanboard-themeplus) plugin.
2. Kanboard becomes more useful when you integrate in/outbound email with [MailGun](https://github.com/kanboard/plugin-mailgun), [SendGrid](https://github.com/kanboard/plugin-sendgrid), or [Postmark](https://github.com/kanboard/plugin-postmark).
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/mail.md
+191
View File
@@ -0,0 +1,191 @@
hero: Docker-mailserver - A recipe for a self-contained mailserver and friends ✉️
# Mail Server
Many of the recipes that follow require email access of some kind. It's normally possible to use a hosted service such as SendGrid, or just a gmail account. If (like me) you'd like to self-host email for your stacks, then the following recipe provides a full-stack mail server running on the docker HA swarm.
Of value to me in choosing docker-mailserver were:
1. Automatically renews LetsEncrypt certificates
2. Creation of email accounts across multiple domains (i.e., the same container gives me mailbox wekan@wekan.example.com, and gitlab@gitlab.example.com)
3. The entire configuration is based on flat files, so there's no database or persistence to worry about
docker-mailserver doesn't include a webmail client, and one is not strictly needed. Rainloop can be added either as another service within the stack, or as a standalone service. Rainloop will be covered in a future recipe.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. LetsEncrypt authorized email address for domain
4. Access to manage DNS records for domains
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/docker-mailserver:
```
cd /var/data
mkdir docker-mailserver
cd docker-mailserver
mkdir {maildata,mailstate,config,letsencrypt,rainloop}
```
### Get LetsEncrypt certificate
Decide on the FQDN to assign to your mailserver. You can service multiple domains from a single mailserver - i.e., bob@dev.example.com and daphne@prod.example.com can both be served by **mail.example.com**.
The docker-mailserver container can _renew_ our LetsEncrypt certs for us, but it can't generate them. To do this, we need to run certbot (from a container) to request the initial certs and create the appropriate directory structure.
In the example below, since I'm already using Traefik to manage the LE certs for my web platforms, I opted to use the DNS challenge to prove my ownership of the domain. The certbot client will prompt you to add a DNS record for domain verification.
```
docker run -ti --rm -v \
"$(pwd)"/letsencrypt:/etc/letsencrypt certbot/certbot \
--manual --preferred-challenges dns certonly \
-d mail.example.com
```
### Get setup.sh
docker-mailserver comes with a handy bash script for managing the stack (which is just really a wrapper around the container.) It'll make our setup easier, so download it into the root of your configuration/data directory, and make it executable:
```
curl -o setup.sh \
https://raw.githubusercontent.com/tomav/docker-mailserver/master/setup.sh \
chmod a+x ./setup.sh
```
### Create email accounts
For every email address required, run ```./setup.sh email add <email> <password>``` to create the account. The command returns no output.
You can run ```./setup.sh email list``` to confirm all of your addresses have been created.
### Create DKIM DNS entries
Run ```./setup.sh config dkim``` to create the necessary DKIM entries. The command returns no output.
Examine the keys created by opendkim to identify the DNS TXT records required:
```
for i in `find config/opendkim/keys/ -name mail.txt`; do \
echo $i; \
cat $i; \
done
```
You'll end up with something like this:
```
config/opendkim/keys/gitlab.example.com/mail.txt
mail._domainkey IN TXT ( "v=DKIM1; k=rsa; "
"p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB" ) ; ----- DKIM key mail for gitlab.example.com
[root@ds1 mail]#
```
Create the necessary DNS TXT entries for your domain(s). Note that although opendkim splits the record across two lines, the actual record should be concatenated on creation. I.e., the DNS TXT record above should read:
```
"v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCYuQqDg2ZG8ZOfI1PvarF1Gcr5cJnCR8BeCj5HYgeRohSrxKL5utPEF/AWAxXYwnKpgYN837fu74GfqsIuOhu70lPhGV+O2gFVgpXYWHELvIiTqqO0QgarIN63WE2gzE4s0FckfLrMuxMoXr882wuzuJhXywGxOavybmjpnNHhbQIDAQAB"
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (_v3.2 - because we need to expose mail ports in "host mode"_), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3.2'
services:
mail:
image: tvial/docker-mailserver:latest
ports:
- target: 25
published: 25
protocol: tcp
mode: host
- target: 587
published: 587
protocol: tcp
mode: host
- target: 993
published: 993
protocol: tcp
mode: host
- target: 995
published: 995
protocol: tcp
mode: host
volumes:
- /var/data/docker-mailserver/maildata:/var/mail
- /var/data/docker-mailserver/mailstate:/var/mail-state
- /var/data/docker-mailserver/config:/tmp/docker-mailserver
- /var/data/docker-mailserver/letsencrypt:/etc/letsencrypt
env_file: /var/data/docker-mailserver/docker-mailserver.env
networks:
- internal
deploy:
replicas: 1
rainloop:
image: hardware/rainloop
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:rainloop.example.com
- traefik.docker.network=traefik_public
- traefik.port=8888
volumes:
- /var/data/mailserver/rainloop:/rainloop/data
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.2.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot.
A sample docker-mailserver.env file looks like this:
```
ENABLE_SPAMASSASSIN=1
ENABLE_CLAMAV=1
ENABLE_POSTGREY=1
ONE_DIR=1
OVERRIDE_HOSTNAME=mail.example.com
OVERRIDE_DOMAINNAME=mail.example.com
POSTMASTER_ADDRESS=admin@example.com
PERMIT_DOCKER=network
SSL_TYPE=letsencrypt
```
## Serving
### Launch mailserver
Launch the mail server stack by running ```docker stack deploy docker-mailserver -c <path-to-docker-mailserver.yml>```
## Chef's Notes
1. One of the elements of this design which I didn't appreciate at first is that since the config is entirely file-based, **setup.sh** can be run on any container host, provided it has the shared data mounted. This means that even though docker-mailserver was not designed with docker swarm in mind, it works perfectl with swarm. I.e., from any node, regardless of where the container is actually running, you're able to add/delete email addresses, view logs, etc.
2. If you're using sieve with Rainloop, take note of the [workaround](https://discourse.geek-kitchen.funkypenguin.co.nz/t/mail-server-funky-penguins-geek-cookbook/70/15) identified by [ggilley](https://discourse.geek-kitchen.funkypenguin.co.nz/u/ggilley)
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/mattermost.md
+127
View File
@@ -0,0 +1,127 @@
# MatterMost
Intro
![NAME Screenshot](../images/mattermost.jpg)
Details
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik_public) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/wekan:
```
mkdir -p /var/data/mattermost/{cert,config,data,logs,plugins,database-dump}
mkdir -p /var/data/realtime/mattermost/database
```
### Prepare environment
Create mattermost.env, and populate with the following variables
```
POSTGRES_USER=mmuser
POSTGRES_PASSWORD=mmuser_password
POSTGRES_DB=mattermost
MM_USERNAME=mmuser
MM_PASSWORD=mmuser_password
MM_DBNAME=mattermost
```
Now create mattermost-backup.env, and populate with the following variables:
```
PGHOST=db
PGUSER=mmuser
PGPASSWORD=mmuser_password
BACKUP_NUM_KEEP=7
BACKUP_FREQUENCY=1d
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
db:
image: mattermost/mattermost-prod-db
env_file: /var/data/config/mattermost/mattermost.env
volumes:
- /var/data/realtime/mattermost/database:/var/lib/postgresql/data
networks:
- internal
app:
image: mattermost/mattermost-team-edition
env_file: /var/data/config/mattermost/mattermost.env
volumes:
- /var/data/mattermost/config:/mattermost/config:rw
- /var/data/mattermost/data:/mattermost/data:rw
- /var/data/mattermost/logs:/mattermost/logs:rw
- /var/data/mattermost/plugins:/mattermost/plugins:rw
db-backup:
image: mattermost/mattermost-prod-db
env_file: /var/data/config/mattermost/mattermost-backup.env
volumes:
- /var/data/mattermost/database-dump:/dump
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
sleep 2m
while /bin/true; do
pg_dump -Fc > /dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.psql
(ls -t /dump/dump*.psql|head -n $$BACKUP_NUM_KEEP;ls /dump/dump*.psql)|sort|uniq -u|xargs rm -- {}
sleep $$BACKUP_FREQUENCY
done
EOF'
networks:
- internal
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.40.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Wekan stack
Launch the Wekan stack by running ```docker stack deploy wekan -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user "root" and the password you specified in gitlab.env.
## Chef's Notes
1. If you wanted to expose the Wekan UI directly, you could remove the oauth2_proxy from the design, and move the traefik_public-related labels directly to the wekan container. You'd also need to add the traefik_public network to the wekan container.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/miniflux.md
+147
View File
@@ -0,0 +1,147 @@
hero: Miniflux - A recipe for a lightweight minimalist RSS reader
# Miniflux
Miniflux is a lightweight RSS reader, developed by [Frédéric Guillot](https://github.com/fguillot). (_Who also happens to be the developer of the favorite Open Source Kanban app, [Kanboard](/recipes/kanboard/)_)
![Miniflux Screenshot](../../images/miniflux.png)
!!! tip "Sponsored Project"
Miniflux is one of my [sponsored projects](/sponsored-projects/) - a project I financially support on a regular basis because of its utility to me. Although I get to process my RSS feeds less frequently than I'd like to!
I've [reviewed Miniflux in detail on my blog](https://www.funkypenguin.co.nz/review/miniflux-lightweight-self-hosted-rss-reader/), but features (among many) that I appreciate:
* Compatible with the Fever API, read your feeds through existing mobile and desktop clients (_This is the killer feature for me. I hardly ever read RSS on my desktop, I typically read on my iPhone or iPad, using [Fiery Feeds](http://cocoacake.net/apps/fiery/) or my new squeeze, [Unread](https://www.goldenhillsoftware.com/unread/)_)
* Send your bookmarks to Pinboard, Wallabag, Shaarli or Instapaper (_I use this to automatically pin my bookmarks for collection on my [blog](https://www.funkypenguin.co.nz/blog/)_)
* Feeds can be configured to download a "full" version of the content (_rather than an excerpt_)
* Use the Bookmarklet to subscribe to a website directly from any browsers
!!! abstract "2.0+ is a bit different"
[Some things changed](https://docs.miniflux.net/en/latest/migration.html) when Miniflux 2.0 was released. For one thing, the only supported database is now postgresql (_no more SQLite_). External themes are gone, as is PHP (_in favor of golang_). It's been a controversial change, but I'm keen on minimal and single-purpose, so I'm still very happy with the direction of development. The developer has laid out his [opinions](https://docs.miniflux.net/en/latest/opinionated.html) re the decisions he's made in the course of development.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry pointing your Miniflux url (i.e. _miniflux.example.com_) to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
Create the location for the bind-mount of the application data, so that it's persistent:
```
mkdir -p /var/data/miniflux/database-dump
mkdir -p /var/data/runtime/miniflux/database
```
### Setup environment
Create ```/var/data/config/miniflux/miniflux.env``` something like this:
```
DATABASE_URL=postgres://miniflux:secret@db/miniflux?sslmode=disable
POSTGRES_USER=miniflux
POSTGRES_PASSWORD=secret
# This is necessary for the miniflux to update the db schema, even on an empty DB
RUN_MIGRATIONS=1
# Uncomment this on first run, else leave it commented out after adding your own user account
CREATE_ADMIN=1
ADMIN_USERNAME=admin
ADMIN_PASSWORD=test1234
# For backups
PGUSER=miniflux
PGPASSWORD=secret
PGHOST=db
BACKUP_NUM_KEEP=7
BACKUP_FREQUENCY=1d
```
The entire application is configured using environment variables, including the initial username. Once you've successfully deployed once, comment out ```CREATE_ADMIN``` and the two successive lines.
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
miniflux:
image: miniflux/miniflux:2.0.7
env_file: /var/data/config/miniflux/miniflux.env
volumes:
- /etc/localtime:/etc/localtime:ro
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:miniflux.example.com
- traefik.port=8080
- traefik.docker.network=traefik_public
db:
env_file: /var/data/config/miniflux/miniflux.env
image: postgres:10.1
volumes:
- /var/data/runtime/miniflux/database:/var/lib/postgresql/data
- /etc/localtime:/etc/localtime:ro
networks:
- internal
db-backup:
image: postgres:10.1
env_file: /var/data/config/miniflux/miniflux.env
volumes:
- /var/data/miniflux/database-dump:/dump
- /etc/localtime:/etc/localtime:ro
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
sleep 2m
while /bin/true; do
pg_dump -Fc > /dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.psql
(ls -t /dump/dump*.psql|head -n $$BACKUP_NUM_KEEP;ls /dump/dump*.psql)|sort|uniq -u|xargs rm -- {}
sleep $$BACKUP_FREQUENCY
done
EOF'
networks:
- internal
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.22.0/24
```
## Serving
### Launch Miniflux stack
Launch the Miniflux stack by running ```docker stack deploy miniflux -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, using the credentials you setup in the environment flie. After this, change your user/password as you see fit, and comment out the ```CREATE_ADMIN``` line in the env file (_if you don't, then an **additional** admin will be created the next time you deploy_)
## Chef's Notes
1. Find the bookmarklet under the **Settings -> Integration** page.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/munin.md
+145
View File
@@ -0,0 +1,145 @@
# Munin
Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.
![Munin Screenshot](../images/munin.png)
Using Munin you can easily monitor the performance of your computers, networks, SANs, applications, weather measurements and whatever comes to mind. It makes it easy to determine "what's different today" when a performance problem crops up. It makes it easy to see how you're doing capacity-wise on any resources.
Munin uses the excellent ​RRDTool (written by Tobi Oetiker) and the framework is written in Perl, while plugins may be written in any language. Munin has a master/node architecture in which the master connects to all the nodes at regular intervals and asks them for data. It then stores the data in RRD files, and (if needed) updates the graphs. One of the main goals has been ease of creating new plugins (graphs).
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Prepare target nodes
Depending on what you want to monitor, you'll want to install munin-node. On Ubuntu/Debian, you'll use ```apt-get install munin-node```, and on RHEL/CentOS, run ```yum install munin-node```. Remember to edit ```/etc/munin/munin-node.conf```, and set your node to allow the server to poll it, by adding ```cidr_allow x.x.x.x/x```.
On CentOS Atomic, of course, you can't install munin-node directly, but you can run it as a containerized instance. In this case, you can't use swarm since you need the container running in privileged mode, so launch a munin-node container on each atomic host using:
```
docker run -d --name munin-node --restart=always \
--privileged --net=host \
-v /:/rootfs:ro \
-v /sys:/sys:ro \
-e ALLOW="cidr_allow 0.0.0.0/0" \
-p 4949:4949 \
--restart=always \
funkypenguin/munin-node
```
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/munin:
```
mkdir /var/data/munin
cd /var/data/munin
mkdir -p {log,lib,run,cache}
```
### Prepare environment
Create /var/data/config/munin/munin.env, and populate with the following variables. Use the OAUTH2 variables if you plan to use an [oauth2_proxy](/reference/oauth_proxy/) to protect munin, and set at a **minimum** the ```MUNIN_USER```, ```MUNIN_PASSWORD```, and ```NODES``` values:
```
# Use these if you plan to protect the webUI with an oauth_proxy
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
MUNIN_USER=odin
MUNIN_PASSWORD=lokiisadopted
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USERNAME=smtp-username
SMTP_PASSWORD=smtp-password
SMTP_USE_TLS=false
SMTP_ALWAYS_SEND=false
SMTP_MESSAGE='[${var:group};${var:host}] -> ${var:graph_title} -> warnings: ${loop<,>:wfields ${var:label}=${var:value}} / criticals: ${loop<,>:cfields ${var:label}=${var:value}}'
ALERT_RECIPIENT=monitoring@example.com
ALERT_SENDER=alerts@example.com
NODES="node1:192.168.1.1 node2:192.168.1.2 node3:192.168.1.3"
SNMP_NODES="router1:10.0.0.254:9999"
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
munin:
image: funkypenguin/munin-server
env_file: /var/data/config/munin/munin.env
networks:
- internal
volumes:
- /var/data/munin/log:/var/log/munin
- /var/data/munin/lib:/var/lib/munin
- /var/data/munin/run:/var/run/munin
- /var/data/munin/cache:/var/cache/munin
proxy:
image: a5huynh/oauth2_proxy
env_file: /var/data/config/munin/munin.env
networks:
- traefik_public
- internal
deploy:
labels:
- traefik.frontend.rule=Host:munin.example.com
- traefik.docker.network=traefik
- traefik.port=4180
command: |
-cookie-secure=false
-upstream=http://munin:8080
-redirect-url=https://munin.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.24.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Munin stack
Launch the Munin stack by running ```docker stack deploy munin -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user and password password you specified in munin.env above.
## Chef's Notes
1. If you wanted to expose the Munin UI directly, you could remove the oauth2_proxy from the design, and move the traefik-related labels directly to the munin container. You'd also need to add the traefik_public network to the munin container.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/nextcloud.md
+186
View File
@@ -0,0 +1,186 @@
hero: Backup all your stuff. Share it. Privately.
# NextCloud
[NextCloud](https://www.nextcloud.org/) (_a [fork of OwnCloud](https://owncloud.org/blog/owncloud-statement-concerning-the-formation-of-nextcloud-by-frank-karlitschek/), led by original developer Frank Karlitschek_) is a suite of client-server software for creating and using file hosting services. It is functionally similar to Dropbox, although Nextcloud is free and open-source, allowing anyone to install and operate it on a private server.
- https://en.wikipedia.org/wiki/Nextcloud
![NextCloud Screenshot](../images/nextcloud.png)
This recipe is based on the official NextCloud docker image, but includes seprate containers ofor the database (_MariaDB_), Redis (_for transactional locking_), Apache Solr (_for full-text searching_), automated database backup, (_you *do* backup the stuff you care about, right?_) and a separate cron container for running NextCloud's 15-min crons.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry pointing your NextCloud url (_nextcloud.example.com_) to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need several directories for [static data](/reference/data_layout/#static-data) to bind-mount into our container, so create them in /var/data/nextcloud (_so that they can be [backed up](/recipes/duplicity/)_)
```
mkdir /var/data/nextcloud
cd /var/data/nextcloud
mkdir -p {html,apps,config,data,database-dump}
```
Now make **more** directories for [runtime data](/reference/data_layout/#runtime-data) (_so that they can be **not** backed-up_):
```
mkdir /var/data/runtime/nextcloud
cd /var/data/runtime/nextcloud
mkdir -p {db,redis}
```
### Prepare environment
Create nextcloud.env, and populate with the following variables
```
NEXTCLOUD_ADMIN_USER=admin
NEXTCLOUD_ADMIN_PASSWORD=FVuojphozxMVyaYCUWomiP9b
MYSQL_HOST=db
# For mysql
MYSQL_ROOT_PASSWORD=<set to something secure>
MYSQL_DATABASE=nextcloud
MYSQL_USER=nextcloud
MYSQL_PASSWORD=set to something secure>
```
Now create a **separate** nextcloud-db-backup.env file, to capture the environment variables necessary to perform the backup. (_If the same variables are shared with the mariadb container, they [cause issues](https://discourse.geek-kitchen.funkypenguin.co.nz/t/nextcloud-funky-penguins-geek-cookbook/254/3?u=funkypenguin) with database access_)
````
# For database backup (keep 7 days daily backups)
MYSQL_PWD=<set to something secure, same as MYSQL_ROOT_PASSWORD above>
MYSQL_USER=root
BACKUP_NUM_KEEP=7
BACKUP_FREQUENCY=1d
````
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: "3.0"
services:
nextcloud:
image: nextcloud
env_file: /var/data/config/nextcloud/nextcloud.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:nextcloud.example.com
- traefik.docker.network=traefik_public
- traefik.port=80
volumes:
- /var/data/nextcloud/html:/var/www/html
- /var/data/nextcloud/apps:/var/www/html/custom_apps
- /var/data/nextcloud/config:/var/www/html/config
- /var/data/nextcloud/data:/var/www/html/data
db:
image: mariadb:10
env_file: /var/data/config/nextcloud/nextcloud.env
networks:
- internal
volumes:
- /var/data/runtime/nextcloud/db:/var/lib/mysql
db-backup:
image: mariadb:10
env_file: /var/data/config/nextcloud/nextcloud.env
volumes:
- /var/data/nextcloud/database-dump:/dump
- /etc/localtime:/etc/localtime:ro
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
sleep 2m
while /bin/true; do
mysqldump -h db --all-databases | gzip -c > /dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.sql.gz
(ls -t /dump/dump*.sql.gz|head -n $$BACKUP_NUM_KEEP;ls /dump/dump*.sql.gz)|sort|uniq -u|xargs rm -- {}
sleep $$BACKUP_FREQUENCY
done
EOF'
networks:
- internal
redis:
image: redis:alpine
networks:
- internal
volumes:
- /var/data/runtime/nextcloud/redis:/data
cron:
image: nextcloud
volumes:
- /var/data/nextcloud/:/var/www/html
user: www-data
networks:
- internal
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
while [ ! -f /var/www/html/config/config.php ]; do
sleep 1
done
while true; do
php -f /var/www/html/cron.php
sleep 15m
done
EOF'
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.12.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch NextCloud stack
Launch the NextCloud stack by running ```docker stack deploy nextcloud -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user "admin" and the password you specified in nextcloud.env.
### Adding full-text search support (optional)
Once logged in as an admin user, navigate to https://<your FQDN>/index.php/settings/apps, and install the "**nextant**" app for full-text search
Then navigate to https://<your FQDN>/index.php/settings/admin/additional, scroll down to **Nextant (Full Text Search)**, and enter the following:
* Address of your solr servlet : **http://solr:8983/solr/**
* Core: **nextant**
## Chef's Notes
1. Since many of my other recipes use PostgreSQL, I'd have preferred to use Postgres over MariaDB, but MariaDB seems to be the [preferred database type](https://github.com/nextcloud/server/issues/5912).
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/owntracks.md
+122
View File
@@ -0,0 +1,122 @@
# OwnTracks
[OwnTracks](https://owntracks.org/) allows you to keep track of your own location. You can build your private location diary or share it with your family and friends. OwnTracks is open-source and uses open protocols for communication so you can be sure your data stays secure and private.
![OwnTracks Screenshot](../images/owntracks.png)
Using a smartphone app, OwnTracks allows you to collect and analyse your own location data **without** sharing this data with a cloud provider (_i.e. Apple, Google_). Potential use cases are:
* Sharing family locations without relying on Apple Find-My-friends
* Performing automated actions in [HomeAssistant](/recipes/homeassistant/) when you arrive/leave home
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need a directory so store OwnTracks' data , so create ```/var/data/owntracks```:
```
mkdir /var/data/owntracks
```
### Prepare environment
Create owntracks.env, and populate with the following variables
```
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
OTR_USER=recorder
OTR_PASSWD=yourpassword
MQTTHOSTNAME=owntracks.example.com
HOSTLIST=owntracks.example.com
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: "3.0"
services:
owntracks-app:
image: funkypenguin/owntracks
env_file : /var/data/config/owntracks/owntracks.env
volumes:
- /var/data/owntracks:/owntracks
networks:
- internal
ports:
- 1883:1883
- 8883:8883
- 8083:8083
owntracks-proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/owntracks/owntracks.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:owntracks.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/owntracks/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://owntracks-app:8083
-redirect-url=https://owntracks.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.15.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch OwnTracks stack
Launch the OwnTracks stack by running ```docker stack deploy owntracks -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user "root" and the password you specified in gitlab.env.
## Chef's Notes
1. If you wanted to expose the OwnTracks Web UI directly, you could remove the oauth2_proxy from the design, and move the traefik-related labels directly to the wekan container. You'd also need to add the traefik network to the owntracks container.
2. I'm using my own image rather than owntracks/recorderd, because of a [potentially swarm-breaking bug](https://github.com/owntracks/recorderd/issues/14) I found in the official container. If this gets resolved (_or if I was mistaken_) I'll update the recipe accordingly.
3. By default, you'll get a fully accessible, unprotected MQTT broker. This may not be suitable for public exposure, so you'll want to look into securing mosquitto with TLS and ACLs.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/piwik.md
+99
View File
@@ -0,0 +1,99 @@
# Piwik
[Piwik](http://www.piwik.org) is a rich open-source web analytics platform, which can be coupled with commercial plugins for additional features. It's most simply described as "_self-hosted Google Analytics_".
![Piwik Screenshot](../images/piwik.png)
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
## Preparation
### Limitation of docker-swarm
The docker-swarm load-balancer is a problem for deploying piwik, since it rewrites the source address of every incoming packet to whichever docker node received the packet into the swarm. Which is a PITA for analytics, since the original source IP of the request is obscured.
The issue is tracked at #[25526](https://github.com/moby/moby/issues/25526), and there is a workaround, but it requires running the piwik "app" container on **every** swarm node...
### Prepare environment
Create piwik.env, and populate with the following variables
```
MYSQL_ROOT_PASSWORD=set-me-and-use-me-when-setting-up-piwik
```
### Setup docker swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
db:
image: mysql
volumes:
- /var/data/piwik/mysql/runtime:/var/lib/mysql
env_file: /var/data/piwik/piwik.env
networks:
- internal
app:
image: piwik:apache
volumes:
- /var/data/piwik/config:/var/www/html/config
networks:
- internal
- traefik
deploy:
mode: global
labels:
- traefik.frontend.rule=Host:piwik.example.com
- traefik.docker.network=traefik
- traefik.port=80
cron:
image: piwik:apache
volumes:
- /var/data/piwik/config:/var/www/html/config
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
while /bin/true; do
su -s "/bin/bash" -c "/usr/local/bin/php /var/www/html/console core:archive" www-data
sleep 3600
done
EOF'
networks:
- internal
networks:
traefik:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.4.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
Launch the Piwik stack by running ```docker stack deploy piwik -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, and follow the wizard to complete the setup.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/plex.md
+106
View File
@@ -0,0 +1,106 @@
hero: A recipe to manage your Media 🎥 📺 🎵
# Plex
[Plex](https://www.plex.tv/) is a client-server media player system and software suite comprising two main components (a media server and client applications)
![Plex Screenshot](../images/plex.jpg)
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. A DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need a directories to bind-mount into our container for Plex to store its library, so create /var/data/plex:
```
mkdir /var/data/plex
```
### Prepare environment
Create plex.env, and populate with the following variables. Set PUID and GUID to the UID and GID of the user who owns your media files, on the local filesystem
```
EDGE=1
VERSION=latest
PUID=42
PGID=42
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: "3.0"
services:
plex:
image: linuxserver/plex
env_file: plex.env
volumes:
- /var/data/config/plex:/config
- /var/data/media:/media
deploy:
labels:
- traefik.frontend.rule=Host:plex.example.com
- traefik.docker.network=traefik_public
- traefik.port=32400
networks:
- traefik_public
- internal
ports:
- 32469:32469
- 32400:32400
- 32401:32401
- 3005:3005
- 8324:8324
- 1900:1900/udp
- 32410:32410/udp
- 32412:32412/udp
- 32413:32413/udp
- 32414:32414/udp
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.16.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Plex stack
Launch the Plex stack by running ```docker stack deploy plex -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN** (You'll need to setup a plex.tv login for remote access / discovery to work from certain clients)
## Chef's Notes
1. Plex uses port 32400 for remote access, using your plex.tv user/password to authenticate you. The inclusion of the traefik proxy in this recipe is simply to allow you to use the web client (as opposed to a client app) by connecting directly to your instance, as opposed to browsing your media via https://plex.tv/web
2. Got an NVIDIA GPU? See [this blog post](https://www.funkypenguin.co.nz/note/gpu-transcoding-with-emby-plex-using-docker-nvidia/) re how to use your GPU to transcode your media!
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/portainer.md
+75
View File
@@ -0,0 +1,75 @@
hero: A recipe for a sexy view of your Docker Swarm
# Portainer
[Portainer](https://portainer.io/) is a lightweight sexy UI for visualizing your docker environment. It also happens to integrate well with Docker Swarm clusters, which makes it a great fit for our stack.
![Portainer Screenshot](../images/portainer.png)
This is a "lightweight" recipe, because Portainer is so "lightweight". But it **is** shiny...
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
Create a folder to store portainer's persistent data:
```
mkdir /var/data/portainer
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: "3"
services:
app:
image: portainer/portainer
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/data/portainer:/data
networks:
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:portainer.funkypenguin.co.nz
- traefik.port=9000
placement:
constraints: [node.role == manager]
command: -H unix:///var/run/docker.sock
networks:
traefik_public:
external: true
```
## Serving
### Launch Portainer stack
Launch the Portainer stack by running ```docker stack deploy portainer -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**. You'll be prompted to set your admin user/password.
## Chef's Notes
1. I wanted to use oauth2_proxy to provide an additional layer of security for Portainer, but the proxy seems to break the authentication mechanism, effectively making the stack **so** secure, that it can't be logged into!
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/privatebin.md
+70
View File
@@ -0,0 +1,70 @@
# PrivateBin
PrivateBin is a minimalist, open source online pastebin where the server (can) has zero knowledge of pasted data. We all need to paste data / log files somewhere when it doesn't make sense to paste it inline. With PasteBin, you can own the hosting, access, and eventual deletion of this data.
![PrivateBin Screenshot](../images/privatebin.png)
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik_public) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need a single location to bind-mount into our container, so create /var/data/privatebin, and make it world-writable (_there might be a more secure way to do this!_)
```
mkdir /var/data/privatebin
chmod 777 /var/data/privatebin/
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
app:
image: privatebin/nginx-fpm-alpine
volumes:
- /var/data/privatebin:/srv/data
networks:
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:privatebin.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
networks:
traefik_public:
external: true
```
## Serving
### Launch PrivateBin stack
Launch the PrivateBin stack by running ```docker stack deploy privatebin -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user "root" and the password you specified in gitlab.env.
## Chef's Notes
1. The [PrivateBin repo](https://github.com/PrivateBin/PrivateBin/blob/master/INSTALL.md) explains how to tweak configuration options, or to use a database instead of file storage, if your volume justifies it :)
2. The inclusion of PrivateBin was due to the efforts of @gkoerk in our [Discord server](http://chat.funkypenguin.co.nz). Thanks Jerry!!
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/realms.md
+120
View File
@@ -0,0 +1,120 @@
# Realms
Realms is a git-based wiki (_like [Gollum](/recipes/gollum/), but with basic authentication and registration_)
![Realms Screenshot](../images/realms.png)
Features include:
* Built with Bootstrap 3.
* Markdown (w/ HTML Support).
* Syntax highlighting (Ace Editor).
* Live preview.
* Collaboration (TogetherJS / Firepad).
* Drafts saved to local storage.
* Handlebars for templates and logic.
!!! warning "Project likely abandoned"
In my limited trial, Realms seems _less_ useful than [Gollum](/recipes/gollum/) for my particular use-case (_i.e., you're limited to markdown syntax only_), but other users may enjoy the basic user authentication and registration features, which Gollum lacks.
Also of note is that the docker image is 1.17GB in size, and the handful of commits to the [source GitHub repo](https://github.com/scragg0x/realms-wiki/commits/master) in the past year has listed TravisCI build failures. This has many of the hallmarks of an abandoned project, to my mind.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik_public) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
Since we'll start with a basic Realms install, let's just create a single directory to hold the realms (SQLite) data:
```
mkdir /var/data/realms/
```
Create realms.env, and populate with the following variables (_if you intend to use an [oauth_proxy](/reference/oauth_proxy) to double-secure your installation, which I recommend_)
```
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: "3"
services:
realms:
image: realms/realms-wiki:latest
env_file: /var/data/config/realms/realms.env
volumes:
- /var/data/realms:/home/wiki/data
networks:
- internal
realms_proxy:
image: funkypenguin/oauth2_proxy:latest
env_file : /var/data/config/realms/realms.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:realms.funkypenguin.co.nz
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/realms/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://realms:5000
-redirect-url=https://realms.funkypenguin.co.nz
-http-address=http://0.0.0.0:4180
-email-domain=funkypenguin.co.nz
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.35.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Realms stack
Launch the Wekan stack by running ```docker stack deploy realms -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, authenticate against oauth_proxy, and you're immediately presented with Realms wiki, waiting for a fresh edit ;)
## Chef's Notes
1. If you wanted to expose the Realms UI directly, you could remove the oauth2_proxy from the design, and move the traefik_public-related labels directly to the realms container. You'd also need to add the traefik_public network to the realms container.
2. The inclusion of Realms was due to the efforts of @gkoerk in our [Discord server](http://chat.funkypenguin.co.nz). Thanks gkoerk!
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/sso-stack.md
+44
View File
@@ -0,0 +1,44 @@
hero: SSO for all your stack elements 🎁
# SSO Stack
Most of the recipes in the cookbook are stand-alone - you can deploy and use them in isolation. I was approached recently by an anonymous sponsor, who needed a stack which would allow the combination of several collaborative tools, in a manner which permits "single signon (SSO)". I.e., the goal of the design was that a user would be provisioned _once_, and thereafter have transparent access to multiple separate applications.
The SSO Stack "uber-recipe" is the result of this design.
![SSO Stark Screenshot](../images/sso-stack.png)
This recipe presents a method to combine multiple tools into a single swarm deployment, and make them available securely.
## Menu
Tools included in the SSO stack are:
* **[OpenLDAP](https://www.openldap.org/)** : Provides Authentication backend
* **[LDAP Account Manager ](https://www.ldap-account-manager.org)** (LAM) : A Web_UI to manage LDAP accounts
* **[KeyCloak](https://www.keycloak.org/)** is an open source identity and access management solution, providing SSO and 2FA capabilities backed into authentication provides (like OpenLDAP)
* **[docker-mailserver](https://github.com/tomav/docker-mailserver)** : A fullstack, simple mail platform including SMTP, IMAPS, and spam filtering components
* **[RainLoop](https://www.rainloop.net/)** : A fast, modern webmail client
* **[GitLab](https://gitlab.org)** : A powerful collaborative git-based developmenet platform
* **[NextCloud](https://www.nextcloud.org)** : A file share and communication platform
This is a complex recipe, and should be deployed in a sequential manner (_i.e. you need OpenLDAP with LDAP Account Manager, to enable KeyCloak, in order to get SSO available for NextCloud, etc.._)
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. Access to NZB indexers and Usenet servers
4. DNS entries configured for each of the NZB tools in this recipe that you want to use
## Preparation
Now work your way through the list of tools below, adding whichever tools your want to use, and finishing with the **end** section:
* [OpenLDAP](/recipes/sso-stack/openldap.md)
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/sso-stack/collabora.md
+100
View File
@@ -0,0 +1,100 @@
don't use special characetrs in your password
perl -pi -e "s/<termination (.*)>.*<\/termination>/<termination \1>${termination}<\/termination>/" /etc/loolwsd/loolwsd.xml
Cretaed /var/data/collabora/loolwsd.xml and bind-mounted it for editing ssl bool = false
docker-compose.yml
```
version: "3.0"
services:
local-collabora:
image: funkypenguin/collabora
# the funkypenguin version has a patch to include "termination" behind SSL-terminating reverse proxy (traefik)
#image: collabora/code
env_file: /var/data/config/collabora/collabora.env
volumes:
- /var/data/collabora/loolwsd.xml:/etc/loolwsd/loolwsd.xml
cap_add:
- MKNOD
ports:
- 9980:9980
```
nginx.conf
```
upstream collabora-upstream {
# Run collabora under docker-compose, since it needs MKNOD cap, which can't be provided by Docker
server 172.17.0.1:9980;
}
server {
listen 80;
server_name collabora.observe.global;
# static files
location ^~ /loleaflet {
proxy_pass http://collabora-upstream;
proxy_set_header Host $http_host;
}
# WOPI discovery URL
location ^~ /hosting/discovery {
proxy_pass http://collabora-upstream;
proxy_set_header Host $http_host;
}
# Main websocket
location ~ /lool/(.*)/ws$ {
proxy_pass http://collabora-upstream;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $http_host;
proxy_read_timeout 36000s;
}
# Admin Console websocket
location ^~ /lool/adminws {
proxy_buffering off;
proxy_pass http://collabora-upstream;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $http_host;
proxy_read_timeout 36000s;
}
# download, presentation and image upload
location ~ /lool {
proxy_pass https://collabora-upstream;
proxy_set_header Host $http_host;
}
}
```
collabora.yml
```
version: "3.0"
services:
nginx:
image: nginx:latest
networks:
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:collabora.observe.global
- traefik.docker.network=traefik_public
- traefik.port=80
- traefik.frontend.passHostHeader=true
volumes:
- /var/data/collabora/nginx.conf:/etc/nginx/conf.d/default.conf:ro
networks:
traefik_public:
external: true
```
manuscript/recipes/sso-stack/docker-mailserver.md
+62
View File
@@ -0,0 +1,62 @@
docker run -ti --rm \
-v "$(pwd)"/letsencrypt:/etc/letsencrypt \
-v "$(pwd)"/cloudflare.ini:/cloudflare.ini \
certbot/dns-cloudflare \
certonly \
--dns-cloudflare \
--dns-cloudflare-credentials=/cloudflare.ini \
-d mail.observe.global
```
root@cloud:/var/data/docker-mailserver# docker run -ti --rm -v "$(pwd)"/letsencrypt:/etc/letsencrypt -v "$(pwd)"/cloudflare.ini:/cloudflare.ini certbot/dns-cloudflare certonly --dns-cloudflare --dns-cloudflare-credentials=/cloudflare.ini -d mail.observe.global
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-cloudflare, Installer None
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): cam@0sum.club
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
https://acme-v02.api.letsencrypt.org/directory
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(A)gree/(C)ancel: A
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit
organization that develops Certbot? We'd like to send you email about our work
encrypting the web, EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: N
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for mail.observe.global
Unsafe permissions on credentials configuration file: /cloudflare.ini
Waiting 10 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/mail.observe.global/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/mail.observe.global/privkey.pem
Your cert will expire on 2019-01-30. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
"certbot renew"
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
root@cloud:/var/data/docker-mailserver#
```
manuscript/recipes/sso-stack/gitlab.md
+122
View File
@@ -0,0 +1,122 @@
https://edenmal.moe/post/2018/GitLab-Keycloak-SAML-2-0-OmniAuth-Provider/
OAUTH_SAML_ASSERTION_CONSUMER_SERVICE_URL
OAUTH_SAML_IDP_CERT_FINGERPRINT
OAUTH_SAML_IDP_SSO_TARGET_URL
OAUTH_SAML_ISSUER
OAUTH_SAML_NAME_IDENTIFIER_FORMAT
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
gitlab_rails['omniauth_block_auto_created_users'] = false
gitlab_rails['omniauth_auto_link_saml_user'] = true
gitlab_rails['omniauth_providers'] = [
{
name: 'saml',
label: 'SAML',
args: {
attribute_statements: { username: ['username'] }
}
}
]
OAUTH_BLOCK_AUTO_CREATED_USERS=false
OAUTH_AUTO_SIGN_IN_WITH_PROVIDER=saml
OAUTH_ALLOW_SSO=saml
OAUTH_SAML_ASSERTION_CONSUMER_SERVICE_URL=https://gitlab.observe.global/users/auth/saml/callback
OAUTH_SAML_IDP_CERT_FINGERPRINT=41f1c588c928291c5dc30d11161d685231509ab8
OAUTH_SAML_IDP_SSO_TARGET_URL=https://keycloak.observe.global/auth/realms/observe/protocol/sam
OAUTH_SAML_ISSUER=https://gitlab.observe.global
OAUTH_SAML_NAME_IDENTIFIER_FORMAT=urn:oasis:names:tc:SAML:2.0:nameid-format:persistent
DISBALED_OAUTH_SAML_ATTRIBUTE_STATEMENTS_EMAIL=mail
DISBALEDOAUTH_SAML_ATTRIBUTE_STATEMENTS_NAME=cnam
DISBALEDOAUTH_SAML_ATTRIBUTE_STATEMENTS_FIRST_NAME=cname
DISBALEDOAUTH_SAML_ATTRIBUTE_STATEMENTS_LAST_NAME=sn
```
{
"clients": [
{
"clientId": "https://gitlab.observe.global",
"rootUrl": "https://gitlab.observe.global",
"enabled": true,
"redirectUris": [
"https://gitlab.observe.global/*"
],
"protocol": "saml",
"attributes": {
"saml.assertion.signature": "false",
"saml.force.post.binding": "true",
"saml.multivalued.roles": "false",
"saml.encrypt": "false",
"saml.server.signature": "true",
"saml.server.signature.keyinfo.ext": "false",
"saml.signature.algorithm": "RSA_SHA256",
"saml_force_name_id_format": "false",
"saml.client.signature": "false",
"saml.authnstatement": "true",
"saml_name_id_format": "username",
"saml.onetimeuse.condition": "false",
"saml_signature_canonicalization_method": "http://www.w3.org/2001/10/xml-exc-c14n#"
},
"protocolMappers": [
{
"name": "email",
"protocol": "saml",
"protocolMapper": "saml-user-property-mapper",
"consentRequired": false,
"config": {
"user.attribute": "email",
"attribute.name": "email"
}
},
{
"name": "first_name",
"protocol": "saml",
"protocolMapper": "saml-user-property-mapper",
"consentRequired": false,
"config": {
"user.attribute": "firstName",
"attribute.name": "first_name"
}
},
{
"name": "last_name",
"protocol": "saml",
"protocolMapper": "saml-user-property-mapper",
"consentRequired": false,
"config": {
"user.attribute": "lastName",
"attribute.name": "last_name"
}
},
{
"name": "username",
"protocol": "saml",
"protocolMapper": "saml-user-property-mapper",
"consentRequired": false,
"config": {
"user.attribute": "username",
"attribute.name": "username"
}
}
]
}
]
}
```
-----BEGIN CERTIFICATE-----
MIICnTCCAYUCBgFmyRcGiTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDDAdvYnNlcnZlMB4XDTE4MTAzMTA3NDUyMVoXDTI4MTAzMTA3NDcwMVowEjEQMA4GA1UEAwwHb2JzZXJ2ZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAI/quQQfWBuQgvxpkcqzkPmXmiO/XE9KmSLcoIOJuDMXXmev9WFtXYbKfozjrZgC4P0uPQLAXJU+2hO7U5fkaG2IuORCK/fKp+cD3GXVO38mxpGFdk3k2eTLUOFfVAUXpPT9dYPSs3EpiB/llslErBBG7bkfwHr06xjU2sMqo/pRbKDLvrqAaBMuHlgOHhAVWWxyzQuUF0kxHxsAbpOnzpiMMOZxhKKZiNEpozIOESplIKFsEYiS4w60z5ROmYEBVqMKP5rsEop9XgS+JNaqfjDbW/NOgT13bvxiA6HxEB0UGt2tWxVsnxpzp3v5rxnXzo1wZQO5KQYWNxJT9Wb9sy8CAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAIyrT5q/TC3CVF7YOAIrPCq+TEANnoaTUuq2BCG6ory+cnnI9T/qoW9c2GVYSrmdQraY8H3o4J+Trjz2OCmuk4Xdp326Lz7hGPuF6i2p9Dmbu696WDlwZHMm+Dn6lMegGb1WKJGAIB9JBss5lHqGbrAxUav9pWukKBZaNsFVnycOMGLQJuROf3jh/MNd7tcIhxAXQxWf//ZfYH7JfeK973L27oGyK+CrxGwsHIsuwSrkJVAPvWPADiPQFqExK/bC1DPGQO4YAV5rCJPTIwXL5I6l5Al2hw1FcAMND2bTA3MgYzg1aQvAJGO7+wQNQYOPkuhR1Hhb1JWGYj1YOdPnG+g==
-----END CERTIFICATE-----
https://edenmal.moe/post/2018/GitLab-Keycloak-SAML-2-0-OmniAuth-Provider/
manuscript/recipes/sso-stack/keycloak.md
+170
View File
@@ -0,0 +1,170 @@
# KeyCloak
!!! warning
While this could stand on its own as a standalone recipe, it's a component of the [sso-stack](/recipes/sso-stack/) "_uber-recipe_", and is written in the expectation that the entire SSO stack is being deployed.
![KeyCloak Screenshot](../images/keycloak.png)
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik_public) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container for both runtime and backup data, so create them as follows
```
mkdir /var/data/runtime/keycloak/database
mkdir /var/data/keycloak/database-dump
```
### Prepare environment
Create /var/data/keycloak/keycloak.env, and populate with the following variables, customized for your own domain struction. Take care with LDAP_DOMAIN, this is core to the rest of the [sso-stack](/recipes/sso-stack/), and can't easily be changed later.
```
# Technically, this could be auto-detected, but we prefer to be prescriptive
DB_VENDOR=postgres
DB_DATABASE=keycloak
DB_ADDR=db
DB_USER=keycloak
DB_PASSWORD=myuberpassword
KEYCLOAK_USER=admin
KEYCLOAK_PASSWORD=ilovepasswords
# This is required to run keycloak behind traefik
PROXY_ADDRESS_FORWARDING=true
# What's our hostname?
KEYCLOAK_HOSTNAME=cloud.example.com
# Tell Postgress what user/password to create
POSTGRES_USER=keycloak
POSTGRES_PASSWORD=myuberpassword
```
Create /var/data/keycloak/keycloak-backup.env, and populate with the following, so that your database can be backed up to the filesystem, daily:
```
PGHOST=db
PGUSER=keycloak
PGPASSWORD=myuberpassword
BACKUP_NUM_KEEP=7
BACKUP_FREQUENCY=1d
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
keycloak:
image: jboss/keycloak
env_file: /var/data/config/keycloak/keycloak.env
networks:
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:keycloak.cloud.example.com
- traefik.port=8080
- traefik.docker.network=traefik_public
db:
env_file: /var/data/config/keycloak/keycloak.env
image: postgres:10.1
volumes:
- /var/data/runtime/keycloak/database:/var/lib/postgresql/data
networks:
- traefik_public
db-backup:
image: postgres:10.1
env_file: /var/data/config/keycloak/keycloak-backup.env
volumes:
- /var/data/keycloak/database-dump:/dump
# - /etc/localtime:/etc/localtime:ro
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
sleep 2m
while /bin/true; do
pg_dump -Fc > /dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.psql
(ls -t /dump/dump*.psql|head -n $$BACKUP_NUM_KEEP;ls /dump/dump*.psql)|sort|uniq -u|xargs rm -- {}
sleep $$BACKUP_FREQUENCY
done
EOF'
networks:
- traefik_public
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.39.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch OpenLDAP stack
Launch the OpenLDAP stack by running ```docker stack deploy keycloak -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, and login with the user/password you defined in keycloak.env.
You start in the "Master" realm - but mouseover the realm name, to a dropdown box allowing you add an new realm:
![KeyCloak Add Realm Screenshot](/images/sso-stack-keycloak-1.png)
Enter a name for your new realm, and click "_Create_":
![KeyCloak Add Realm Screenshot](/images/sso-stack-keycloak-2.png)
Once in the desired realm, click on **User Federation**, and click **Add Provider**. On the next page ("_Required Settings_"), set the following:
* **Edit Mode** : Writeable
* **Vendor** : Other
* **Connection URL** : ldap://openldap
* **Users DN** : ou=People,<your base DN>
* **Authentication Type** : simple
* **Bind DN** : cn=admin,<your base DN>
* **Bind Credential** : <your chosen admin password>
Save your changes, and then navigate back to "User Federation" > Your LDAP name > Mappers:
![KeyCloak Add Realm Screenshot](/images/sso-stack-keycloak-3.png)
For each of the following names, click the name, and set the "_Read Only_" flag to "_Off_" (_this enables 2-way sync between KeyCloak and OpenLD_AP)
* last name
* username
* email
* first name
![KeyCloak Add Realm Screenshot](/images/sso-stack-keycloak-4.png)
Proceed to setting up [Email](/recipes/sso-stack/docker-mailserver/)...
## Chef's Notes
1. I wanted to be able to add multiple networks to KeyCloak (i.e., a dedicated overlay network for LDAP authentication), but the entrypoint used by the container produces an error when more than one network is configured. This could theoretically be corrected in future, with a PR.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/sso-stack/nextcloud.md
View File
Whitespace-only changes.
manuscript/recipes/sso-stack/openldap.md
+403
View File
@@ -0,0 +1,403 @@
# OpenLDAP with LAM
!!! warning
While this could stand on its own as a standalone recipe, it's a component of the [sso-stack](/recipes/sso-stack/) "_uber-recipe_", and is written in the expectation that the entire SSO stack is being deployed.
![OpenLDAP Screenshot](../images/openldap.png)
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik_public) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/openldap:
```
mkdir /var/data/openldap/openldap
mkdir /var/data/runtime/openldap/
```
### Prepare environment
Create /var/data/openldap/openldap.env, and populate with the following variables, customized for your own domain struction. Take care with LDAP_DOMAIN, this is core to the rest of the [sso-stack](/recipes/sso-stack/), and can't easily be changed later.
```
LDAP_DOMAIN=batcave.gotham
LDAP_ORGANISATION=BatCave Inc
LDAP_ADMIN_PASSWORD=supermansucks
LDAP_TLS=false
# Setup for github
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
```
Create ```authenticated-emails.txt```, and populate with the email addresses (matched to GitHub user accounts, in my case) which you want to grant access, using OAuth2.
### Create config.cfg
```
# password to add/delete/rename configuration profiles (default: lam)
password: {SSHA}54haBZN/kfgNVJ+W3YJrI2dCic4= iCXkNA==
# default profile, without ".conf"
default: observeglobal
# log level
logLevel: 4
# log destination
logDestination: SYSLOG
# session timeout in minutes
sessionTimeout: 30
# list of hosts which may access LAM
allowedHosts:
# list of hosts which may access LAM Pro self service
allowedHostsSelfService:
# encrypt session data
encryptSession: true
# Password: minimum password length
passwordMinLength: 0
# Password: minimum uppercase characters
passwordMinUpper: 0
# Password: minimum lowercase characters
passwordMinLower: 0
# Password: minimum numeric characters
passwordMinNumeric: 0
# Password: minimum symbolic characters
passwordMinSymbol: 0
# Password: minimum character classes (0-4)
passwordMinClasses: 0
# Password: checked rules
checkedRulesCount: -1
# Password: must not contain part of user name
passwordMustNotContain3Chars: false
# Password: must not contain user name
passwordMustNotContainUser: false
# Email format (default/unix)
mailEOL: default
# PHP error reporting (default/system)
errorReporting: default
# License
license:
```
### Create <profile>.cfg
```
# LDAP Account Manager configuration
#
# Please do not modify this file manually. The configuration can be done completely by the LAM GUI.
#
###################################################################################################
# server address (e.g. ldap://localhost:389 or ldaps://localhost:636)
ServerURL: ldap://openldap:389
# list of users who are allowed to use LDAP Account Manager
# names have to be seperated by semicolons
# e.g. admins: cn=admin,dc=yourdomain,dc=org;cn=root,dc=yourdomain,dc=org
Admins: cn=admin,dc=observe,dc=global
# password to change these preferences via webfrontend (default: lam)
Passwd: {SSHA}h39N9+gg/Qf1K/986VkKrjWlkcI= S/IAUQ==
# suffix of tree view
# e.g. dc=yourdomain,dc=org
treesuffix: dc=observe,dc=global
# default language (a line from config/language)
defaultLanguage: en_GB.utf8
# Path to external Script
scriptPath:
# Server of external Script
scriptServer:
# Access rights for home directories
scriptRights: 750
# Number of minutes LAM caches LDAP searches.
cachetimeout: 5
# LDAP search limit.
searchLimit: 0
# Module settings
modules: posixAccount_user_minUID: 10000
modules: posixAccount_user_maxUID: 30000
modules: posixAccount_host_minMachine: 50000
modules: posixAccount_host_maxMachine: 60000
modules: posixGroup_group_minGID: 10000
modules: posixGroup_group_maxGID: 20000
modules: posixGroup_pwdHash: SSHA
modules: posixAccount_pwdHash: SSHA
# List of active account types.
activeTypes: user,group
types: suffix_user: ou=People,dc=observe,dc=global
types: attr_user: #uid;#givenName;#sn;#uidNumber;#gidNumber
types: modules_user: inetOrgPerson,posixAccount,shadowAccount
types: suffix_group: ou=Groups,dc=observe,dc=global
types: attr_group: #cn;#gidNumber;#memberUID;#description
types: modules_group: posixGroup
# Password mail subject
lamProMailSubject: Your password was reset
# Password mail text
lamProMailText: Dear @@givenName@@ @@sn@@,+::++::+your password was reset to: @@newPassword@@+::++::++::+Best regards+::++::+deskside support+::+
serverDisplayName:
# enable TLS encryption
useTLS: no
# follow referrals
followReferrals: false
# paged results
pagedResults: false
referentialIntegrityOverlay: false
# time zone
timeZone: Europe/London
scriptUserName:
scriptSSHKey:
scriptSSHKeyPassword:
# Access level for this profile.
accessLevel: 100
# Login method.
loginMethod: list
# Search suffix for LAM login.
loginSearchSuffix: dc=yourdomain,dc=org
# Search filter for LAM login.
loginSearchFilter: uid=%USER%
# Bind DN for login search.
loginSearchDN:
# Bind password for login search.
loginSearchPassword:
# HTTP authentication for LAM login.
httpAuthentication: false
# Password mail from
lamProMailFrom:
# Password mail reply-to
lamProMailReplyTo:
# Password mail is HTML
lamProMailIsHTML: false
# Allow alternate address
lamProMailAllowAlternateAddress: true
jobsBindPassword:
jobsBindUser:
jobsDatabase:
jobsDBHost:
jobsDBPort:
jobsDBUser:
jobsDBPassword:
jobsDBName:
jobToken: 190339140545
pwdResetAllowSpecificPassword: true
pwdResetAllowScreenPassword: true
pwdResetForcePasswordChange: true
pwdResetDefaultPasswordOutput: 2
twoFactorAuthentication: none
twoFactorAuthenticationURL: https://localhost
twoFactorAuthenticationInsecure:
twoFactorAuthenticationLabel:
twoFactorAuthenticationOptional:
twoFactorAuthenticationCaption:
tools: tool_hide_toolOUEditor: false
tools: tool_hide_toolProfileEditor: false
tools: tool_hide_toolSchemaBrowser: false
tools: tool_hide_toolServerInformation: false
tools: tool_hide_toolTests: false
tools: tool_hide_toolPDFEditor: false
tools: tool_hide_toolFileUpload: false
tools: tool_hide_toolMultiEdit: false
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
openldap:
image: osixia/openldap
env_file: /var/data/config/openldap/openldap.env
networks:
- traefik_public
volumes:
- /var/data/openldap/openldap/:/var/lib/ldap
- /var/data/runtime/openldap/:/etc/ldap/slapd.d
lam:
image: jacksgt/ldap-account-manager
networks:
- traefik_public
#volumes:
#- /var/data/openldap/lam/config/lam.conf:/var/www/html/config/lam.conf
proxy:
image: funkypenguin/oauth2_proxy
env_file: /var/data/config/openldap/openldap.env
networks:
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:lam.example.com
- traefik.port=4180
volumes:
- /var/data/config/openldap/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://lam:8080
-redirect-url=https://lam.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
networks:
traefik_public:
external: true
```
## Serving
### Launch OpenLDAP stack
Launch the OpenLDAP stack by running ```docker stack deploy openldap -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**. You'll hit the LDAP Account Manager login page, which will look like this:
![LAM Landing Page Screenshot](/images/sso-stack-lam-1.png)
Click on "LAM Configuration" to add a profile.
![LAM Edit Profiles Screenshot](/images/sso-stack-lam-2.png)
Enter a profile name, and a profile password (twice). Leave the template at "_unix_":
![LAM Edit Profiles Screenshot](/images/sso-stack-lam-3.png)
When prompted to save your new profile, enter the "master password" ("lam")
You've created a "profile". Now to configure your profile... Start with "Server Settings", and change your **server address** to ```ldap://openldap:389```, and your **tree suffix** to the base DN you setup in the openldap.env file (above).
![LAM Edit Profiles Screenshot](/images/sso-stack-lam-4.png)
Under **Security Settings**, alter the list of valid users to "**cn=admin\<your-base-dn\>**"
![LAM Edit Profiles Screenshot](/images/sso-stack-lam-5.png)
After you save, you'll be redirected to the profile login page, where you'll need to enter the profile details and password you create above.
Once logged in, click on the "**Account Types"** tab...
![LAM Edit Profiles Screenshot](/images/sso-stack-lam-6.png)
And for both "_Users_" and "_Groups_", edit the "LDAP Suffix" to match your chosen Base DN, and save your changes:
![LAM Edit Profiles Screenshot](/images/sso-stack-lam-7.png)
After saving changes to your LAM profile, you'll be redirected to the LAM admin page. Enter your credentials (default admin/admin) to login. On first login, you'll be prompted to create the "ou=People" and "ou=Group" elements. Proceed to create these.
You've now setup your OpenLDAP directory structure, and hopefully won't have to interact with the "special" LDAP Account Manager interface much again!
Proceed to setting up [KeyCloak](/recipes/sso-stack/keycloak/)...
## Chef's Notes
1. What's not yet documented here is how to make the LAM "profile" configuration persistent. I.e., after each container reload, it's currently necessary to repeat the steps above.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/swarmprom.md
+402
View File
@@ -0,0 +1,402 @@
# Swarmprom
[Swarmprom](https://github.com/stefanprodan/swarmprom) is a starter kit for Docker Swarm monitoring with [Prometheus](https://prometheus.io/), [Grafana](http://grafana.org/), [cAdvisor](https://github.com/google/cadvisor), [Node Exporter](https://github.com/prometheus/node_exporter), [Alert Manager](https://github.com/prometheus/alertmanager) and [Unsee](https://github.com/cloudflare/unsee). And it's **damn** sexy. See for yourself:
![Swarmprom Screenshot](../images/swarmprom.png)
So what do all these components do?
* [Prometheus](https://prometheus.io/docs/introduction/overview/) is an open-source systems monitoring and alerting toolkit originally built at SoundCloud.
* [Grafana](http://grafana.org/) is a tool to make data beautiful.
* [cAdvisor](https://github.com/google/cadvisor)
cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running containers. It is a running daemon that collects, aggregates, processes, and exports information about running containers.
* [Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus exporter for hardware and OS metrics
* [Alert Manager](https://github.com/prometheus/alertmanager) Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, Slack, etc.
* [Unsee](https://github.com/cloudflare/unsee) is an alert dashboard for Alert Manager
## How does this magic work?
I'd encourage you to spend some time reading https://github.com/stefanprodan/swarmprom. Stefan has included detailed explanations about which elements perform which functions, as well as how to customize your stack. (_This is only a starting point, after all_)
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) on **17.09.0 or newer** (_doesn't work with CentOS Atomic, unfortunately_) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik_public) configured per design
3. DNS entry for the hostnames you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
This is basically a rehash of stefanprodan's [instructions](https://github.com/stefanprodan/swarmprom) to match the way I've configured other recipes.
### Setup oauth provider
Grafana includes decent login protections, but from what I can see, Prometheus, AlertManager, and Unsee do no authentication. In order to expose these publicly for your own consumption (my assumption for the rest of this recipe), you'll want to prepare to run [oauth_proxy](/reference/oauth_proxy/) containers in front of each of the 4 web UIs in this recipe.
### Setup metrics
Edit (_or create, depending on your OS_) /etc/docker/daemon.json, and add the following, to enable the experimental export of metrics to Prometheus:
```
{
"metrics-addr" : "0.0.0.0:9323",
"experimental" : true
}
```
Restart docker with ```systemctl restart docker```
### Setup and populate data locations
We'll need several files to bind-mount into our containers, so create directories for them and get the latest copies:
```
mkdir -p /var/data/swarmprom/dockerd-exporter/
cd /var/data/swarmprom/dockerd-exporter/
wget https://raw.githubusercontent.com/stefanprodan/swarmprom/master/dockerd-exporter/Caddyfile
mkdir -p /var/data/swarmprom/prometheus/rules/
cd /var/data/swarmprom/prometheus/rules/
wget https://raw.githubusercontent.com/stefanprodan/swarmprom/master/prometheus/rules/swarm_task.rules.yml
wget https://raw.githubusercontent.com/stefanprodan/swarmprom/master/prometheus/rules/swarm_node.rules.yml
# Directories for holding runtime data
mkdir /var/data/runtime/swarmprom/grafana/
mkdir /var/data/runtime/swarmprom/alertmanager/
mkdir /var/data/runtime/prometheus
chown nobody:nogroup /var/data/runtime/prometheus
```
### Prepare Grafana
Grafana will make all the data we collect from our swarm beautiful.
Create /var/data/swarmprom/grafana.env, and populate with the following variables
```
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
# Disable basic auth (it conflicts with oauth_proxy)
GF_AUTH_BASIC_ENABLED=false
# Set this to the real-world URL to your grafana install (else you get screwy CSS thanks to oauth_proxy)
GF_SERVER_ROOT_URL=https://grafana.example.com
GF_SERVER_DOMAIN=grafana.example.com
# Set your default admin/pass here
GF_SECURITY_ADMIN_USER=admin
GF_SECURITY_ADMIN_PASSWORD=ilovemybatmanunderpants
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), based on the original swarmprom [docker-compose.yml](https://github.com/stefanprodan/swarmprom/blob/master/docker-compose.yml) file
???+ note "This example is 274 lines long. Click here to collapse it for better readability"
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: "3.3"
networks:
net:
driver: overlay
attachable: true
volumes:
prometheus: {}
grafana: {}
alertmanager: {}
configs:
dockerd_config:
file: /var/data/swarmprom/dockerd-exporter/Caddyfile
node_rules:
file: /var/data/swarmprom/prometheus/rules/swarm_node.rules.yml
task_rules:
file: /var/data/swarmprom/prometheus/rules/swarm_task.rules.yml
services:
dockerd-exporter:
image: stefanprodan/caddy
networks:
- internal
environment:
- DOCKER_GWBRIDGE_IP=172.18.0.1
configs:
- source: dockerd_config
target: /etc/caddy/Caddyfile
deploy:
mode: global
resources:
limits:
memory: 128M
reservations:
memory: 64M
cadvisor:
image: google/cadvisor
networks:
- internal
command: -logtostderr -docker_only
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /:/rootfs:ro
- /var/run:/var/run
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
deploy:
mode: global
resources:
limits:
memory: 128M
reservations:
memory: 64M
grafana:
image: stefanprodan/swarmprom-grafana:5.0.1
networks:
- internal
env_file: /var/data/config/swarmprom/grafana.env
environment:
- GF_USERS_ALLOW_SIGN_UP=false
- GF_SMTP_ENABLED=${GF_SMTP_ENABLED:-false}
- GF_SMTP_FROM_ADDRESS=${GF_SMTP_FROM_ADDRESS:-grafana@test.com}
- GF_SMTP_FROM_NAME=${GF_SMTP_FROM_NAME:-Grafana}
- GF_SMTP_HOST=${GF_SMTP_HOST:-smtp:25}
- GF_SMTP_USER=${GF_SMTP_USER}
- GF_SMTP_PASSWORD=${GF_SMTP_PASSWORD}
volumes:
- /var/data/runtime/swarmprom/grafana:/var/lib/grafana
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.role == manager
resources:
limits:
memory: 128M
reservations:
memory: 64M
grafana-proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/swarmprom/grafana.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:grafana.swarmprom.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/swarmprom/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://grafana:3000
-redirect-url=https://grafana.swarmprom.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
alertmanager:
image: stefanprodan/swarmprom-alertmanager:v0.14.0
networks:
- internal
environment:
- SLACK_URL=${SLACK_URL:-https://hooks.slack.com/services/TOKEN}
- SLACK_CHANNEL=${SLACK_CHANNEL:-general}
- SLACK_USER=${SLACK_USER:-alertmanager}
command:
- '--config.file=/etc/alertmanager/alertmanager.yml'
- '--storage.path=/alertmanager'
volumes:
- /var/data/runtime/swarmprom/alertmanager:/alertmanager
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.role == manager
resources:
limits:
memory: 128M
reservations:
memory: 64M
alertmanager-proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/swarmprom/alertmanager.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:alertmanager.swarmprom.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/swarmprom/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://alertmanager:9093
-redirect-url=https://alertmanager.swarmprom.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
unsee:
image: cloudflare/unsee:v0.8.0
networks:
- internal
environment:
- "ALERTMANAGER_URIS=default:http://alertmanager:9093"
deploy:
mode: replicated
replicas: 1
unsee-proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/swarmprom/unsee.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:unsee.swarmprom.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/swarmprom/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://unsee:8080
-redirect-url=https://unsee.swarmprom.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
node-exporter:
image: stefanprodan/swarmprom-node-exporter:v0.15.2
networks:
- internal
environment:
- NODE_ID={{.Node.ID}}
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
- /etc/hostname:/etc/nodename
command:
- '--path.sysfs=/host/sys'
- '--path.procfs=/host/proc'
- '--collector.textfile.directory=/etc/node-exporter/'
- '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
# no collectors are explicitely enabled here, because the defaults are just fine,
# see https://github.com/prometheus/node_exporter
# disable ipvs collector because it barfs the node-exporter logs full with errors on my centos 7 vm's
- '--no-collector.ipvs'
deploy:
mode: global
resources:
limits:
memory: 128M
reservations:
memory: 64M
prometheus:
image: stefanprodan/swarmprom-prometheus:v2.2.0-rc.0
networks:
- internal
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention=24h'
volumes:
- /var/data/runtime/swarmprom/prometheus:/prometheus
configs:
- source: node_rules
target: /etc/prometheus/swarm_node.rules.yml
- source: task_rules
target: /etc/prometheus/swarm_task.rules.yml
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.role == manager
resources:
limits:
memory: 2048M
reservations:
memory: 128M
prometheus-proxy:
image: a5huynh/oauth2_proxy
env_file : /var/data/config/swarmprom/prometheus.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:prometheus.swarmprom.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/swarmprom/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://prometheus:9090
-redirect-url=https://prometheus.swarmprom.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.29.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Swarmprom stack
Launch the Swarm stack by running ```docker stack deploy swarmprom -c <path -to-docker-compose.yml>```
Log into your new grafana instance, check out your beautiful graphs. Move onto drooling over Prometheus, AlertManager, and Unsee.
## Chef's Notes
1. Pay close attention to the ```grafana.env``` config. If you encounter errors about ```basic auth failed```, or failed CSS, it's likely due to misconfiguration of one of the grafana environment variables.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/template.md
+120
View File
@@ -0,0 +1,120 @@
hero: Heroic Hero
# NAME
Intro
![NAME Screenshot](../images/name.jpg)
Details
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik_public) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/wekan:
```
mkdir /var/data/wekan
cd /var/data/wekan
mkdir -p {wekan-db,wekan-db-dump}
```
### Prepare environment
Create wekan.env, and populate with the following variables
```
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
MONGO_URL=mongodb://wekandb:27017/wekan
ROOT_URL=https://wekan.example.com
MAIL_URL=smtp://wekan@wekan.example.com:password@mail.example.com:587/
MAIL_FROM="Wekan <wekan@wekan.example.com>"
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
wekandb:
image: mongo:3.2.15
command: mongod --smallfiles --oplogSize 128
networks:
- internal
volumes:
- /var/data/wekan/wekan-db:/data/db
- /var/data/wekan/wekan-db-dump:/dump
proxy:
image: a5huynh/oauth2_proxy
env_file: /var/data/wekan/wekan.env
networks:
- traefik_public
- internal
deploy:
labels:
- traefik_public.frontend.rule=Host:wekan.example.com
- traefik_public.docker.network=traefik_public
- traefik_public.port=4180
command: |
-cookie-secure=false
-upstream=http://wekan:80
-redirect-url=https://wekan.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
wekan:
image: wekanteam/wekan:latest
networks:
- internal
env_file: /var/data/wekan/wekan.env
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.3.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Wekan stack
Launch the Wekan stack by running ```docker stack deploy wekan -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user "root" and the password you specified in gitlab.env.
## Chef's Notes
1. If you wanted to expose the Wekan UI directly, you could remove the oauth2_proxy from the design, and move the traefik_public-related labels directly to the wekan container. You'd also need to add the traefik_public network to the wekan container.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/tiny-tiny-rss.md
+142
View File
@@ -0,0 +1,142 @@
# Tiny Tiny RSS
[Tiny Tiny RSS](https://tt-rss.org/) is a self-hosted, AJAX-based RSS reader, which rose to popularity as a replacement for Google Reader. It supports ~~geeky~~ advanced features, such as:
* Plugins and themeing in a drop-in fashion
* Filtering (discard all articles with title matching "trump")
* Sharing articles via a unique public URL/feed
![Tiny Tiny RSS Screenshot](../images/tiny-tiny-rss.png)
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/ttrss:
```
mkdir /var/data/ttrss
cd /var/data/ttrss
mkdir -p {database,database-dump}
```
### Prepare environment
Create ttrss.env, and populate with the following variables, customizing at least the database password (POSTGRES_PASSWORD **and** DB_PASS) and the TTRSS_SELF_URL to point to your installation.
```
# Variables for postgres:latest
POSTGRES_USER=ttrss
POSTGRES_PASSWORD=mypassword
DB_EXTENSION=pg_trgm
# Variables for pg_dump running in postgres/latest (used for db-backup)
PGUSER=ttrss
PGPASSWORD=mypassword
PGHOST=db
BACKUP_NUM_KEEP=3
BACKUP_FREQUENCY=1d
# Variables for funkypenguin/docker-ttrss
DB_USER=ttrss
DB_PASS=mypassword
DB_PORT=5432
DB_PORT_5432_TCP_ADDR=db
DB_PORT_5432_TCP_PORT=5432
TTRSS_SELF_URL=https://ttrss.example.com
TTRSS_REPO=https://github.com/funkypenguin/tt-rss.git
S6_BEHAVIOUR_IF_STAGE2_FAILS=2
```
### Setup docker swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
db:
image: postgres:latest
env_file: /var/data/ttrss/ttrss.env
volumes:
- /var/data/ttrss/database:/var/lib/postgresql/data
networks:
- internal
app:
image: funkypenguin/docker-ttrss
env_file: /var/data/ttrss/ttrss.env
deploy:
labels:
- traefik.frontend.rule=Host:ttrss.funkypenguin.co.nz
- traefik.docker.network=traefik
- traefik.port=8080
networks:
- internal
- traefik
db-backup:
image: postgres:latest
env_file: /var/data/ttrss/ttrss.env
volumes:
- /var/data/ttrss/database-dump:/dump
- /etc/localtime:/etc/localtime:ro
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
sleep 2m
while /bin/true; do
pg_dump -Fc > /dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.psql
(ls -t /dump/dump*.psql|head -n $$BACKUP_NUM_KEEP;ls /dump/dump*.psql)|sort|uniq -u|xargs rm -- {}
sleep $$BACKUP_FREQUENCY
done
EOF'
networks:
- internal
networks:
traefik:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.5.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch TTRSS stack
Launch the TTRSS stack by running ```docker stack deploy ttrss -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN** - the first user you create will be an administrative user.
## Chef's Notes
There are several TTRSS containers available on docker hub, none of them "official". I chose [x86dev's container](https://github.com/x86dev/docker-ttrss) for its features - such as my favorite skins and plugins, and the daily automatic updates from the "rolling release" master. Some of the features of the container I use are due to a [PR](https://github.com/x86dev/docker-ttrss/pull/12) I submitted:
1. Docker swarm looses the docker-compose concept of "dependencies" between containers. In the case of this stack, the application server typically starts up before the database container, which causes the database autoconfiguration scripts to fail, and brings up the app in a broken state. To prevent this, I include "[wait-for](https://github.com/Eficode/wait-for/)", which (combined with "S6_BEHAVIOUR_IF_STAGE2_FAILS=2"), will cause the app container to restart (and attempt to auto-configure itself) until the database is ready.
2. The upstream git URL [changed recently](https://discourse.tt-rss.org/t/gitlab-is-overbloated-shit-garbage/325/6), but my experience of the new repository is that it's **SO** slow, that the initial "git clone" on setup of the container times out. To work around this, I created [my own repo](https://github.com/funkypenguin/tt-rss.git), cloned upstream, pushed it into my repo, and pointed the container at my own repo with TTRSS_REPO. I don't get the _latest_ code changes, but at least the app container starts up. When upstream git is performing properly, I'll remove TTRSS_REPO to revert back to the "rolling release".
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/turtle-pool.md
+457
View File
@@ -0,0 +1,457 @@
hero: How to setup a TurtleCoin Mining Pool
# TurtleCoin Mining Pool
[Cryptocurrency miners](/recipes/cryptominer) will "pool" their GPU resources ("_hashpower_") into aggregate "_mining pools_", so that by the combined effort of all the miners, the pool will receive a reward for the blocks "mined" into the blockchain, and this reward will be distributed among the miners.
![Turtle Pool Screenshot](../images/turtle-pool.png)
This recipe illustrates how to build a mining pool for [TurtleCoin](https://turtlecoin.lol), one of many [CryptoNote](https://cryptonote.org/) [currencies](https://cryptonote.org/coins) (_which include [Monero](https://www.coingecko.com/en/coins/monero)_), but the principles can be applied to most mineable coins.
The end result is a mining pool which looks like this: https://trtl.heigh-ho.funkypenguin.co.nz/
!!! question "WTF is a TurtleCoin and why do I want it?""
In my opinion - because it's a fun, no-BS project with a [silly origin story](https://turtlecoin.lol/#story), a [friendly, welcoming community](http://chat.turtlecoin.lol/), and you'll learn more about cryptocurrency/blockchain than you expect.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostnames (_pool and api_) you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
4. At least 16GB disk space (12GB used, 4GB for future growth)
## Preparation
### Create user account
The TurtleCoin pool elements won't (_and shouldn't_) run as root, but they'll need access to write data to some parts of the filesystem (_like logs, etc_).
To manage access control, we'll want to create a local user on **each docker node** with the same UID.
```
useradd -u 3506 turtle-pool
```
!!! question "Why 3506?"
I'm glad you asked. [TurtleCoin hard-forked at block 350K](https://medium.com/@turtlecoin/take-your-baikal-and-shove-it-up-your-asic-b05c96187790) to avoid ASIC miners. The Ninja Turtles' human friend [April O'Neil](http://turtlepedia.wikia.com/wiki/April_O'Neil) works at [Channel 6 News](http://turtlepedia.wikia.com/wiki/Channel_6). 350 + 6 = 3506 😁. Aren't **you** glad you asked?
### Setup Redis
The pool uses Redis for in-memory and persistent storage. This comes in handy for the Docker Swarm deployment, since while the various pool modules weren't _designed_ to run as microservices, the fact that they all rely on Redis for data storage makes this possible.
!!! warning "Playing it safe"
Be aware that by default, Redis stores some data **only** in memory, and writes to the filesystem at default intervals (_can be up to 5 minutes by default_). Given we don't **want** to loose 5 minutes of miner's data if we restart Redis (_what happens if we found a block during those 5 minutes but haven't paid any miners yet?_), we want to ensure that Redis runs in "appendonly" mode, which ensures that every change is immediately written to disk.
We also want to make sure that we retain all Redis logs persistently (_We're dealing with people's cryptocurrency here, it's a good idea to keep persistent logs for debugging/auditing_)
Create directories to hold Redis data. We use separate directories for future flexibility - One day, we may want to backup the data but not the logs, or move the data to an SSD partition but leave the logs on slower, cheaper disk.
```
mkdir -p /var/data/turtle-pool/redis/config
mkdir -p /var/data/turtle-pool/redis/data
mkdir -p /var/data/turtle-pool/redis/logs
chown turtle-pool /var/data/turtle-pool/redis/data
chown turtle-pool /var/data/turtle-pool/redis/logs
```
Create **/var/data/turtle-pool/redis/config/redis.conf** using http://download.redis.io/redis-stable/redis.conf as a guide. The following are the values I changed from default on my deployment (_but I'm not a Redis expert!_):
```
appendonly yes
appendfilename "appendonly.aof"
loglevel notice
logfile "/logs/redis.log"
protected-mode no
```
I also had to **disable** the following line, by commenting it out (_thus ensuring Redis container will respond to the other containers_):
```
bind 127.0.0.1
```
### Setup Nginx
We'll run a simple Nginx container to serve the static front-end of the web UI.
The simplest way to get the frontend is just to clone the upstream turtle-pool repo, and mount the "/website" subdirectory into Nginx.
```
git clone https://github.com/turtlecoin/turtle-pool.git /var/data/turtle-pool/nginx/
```
Edit **/var/data/turtle-pool/nginx/website/config.js**, and change at least the following:
```
var api = "https://<CHOOSE A FQDN TO USE FOR YOUR API>";
var poolHost = "<SET TO THE PUBLIC DNS NAME FOR YOUR POOL SERVER";
```
### Setup TurtleCoin daemon
The first thing we'll need to participate in the great and powerful TurtleCoin network is a **node**. The node is responsible for communicating with the rest of the nodes in the blockchain, allowing our miners to receive new blocks to try to find.
As of June 2018, the TurtleCoin daemon is not what you'd call "**stable**". Certain known circumstances (_like unusual blocks_) can cause the daemon to crash, and **permanently corrupt** its stored blockchain data. When (_not if_) this happens, the only solution is to delete all the stored blockchain data, and allow the daemon to resync (_from checkpoints and then from other nodes_). This resyncing process takes time, during which your miners can't mine. To partially mitigate this risk, we run 2 **extra** copies of the daemon, which we'll then start/stop at alternate intervals, so that in the event of a corruption of the daemon's data, we have hopefully-safe recent copies of the blockchain, allowing us to recover and restore quickly.
Create directories to hold the blockchain data for all 3 daemons:
```
mkdir -p /var/data/runtime/turtle-pool/daemon/1
mkdir -p /var/data/runtime/turtle-pool/daemon/failsafe1
mkdir -p /var/data/runtime/turtle-pool/daemon/failsafe2
```
Create cron entries (in /etc/cron.d/turtle-pool-failsafe) to start/stop the failsafe daemons:
```
# Each TurtleCoin failsafe daemon gets to run ~50% of every hour, but they're never running at the same time
# (to prevent a doomsday block from killing them both)
0 * * * * root docker service scale turtle-pool_daemon-failsafe1=0
1 * * * * root docker service scale turtle-pool_daemon-failsafe2=1
30 * * * * root docker service scale turtle-pool_daemon-failsafe2=0
31 * * * * root docker service scale turtle-pool_daemon-failsafe1=1
```
### Setup TurtleCoin wallet
Our pool needs a wallet to be able to (a) receive rewards for blocks discovered, and (b) pay out our miners for their share of the reward.
Create directories to hold wallet data
```
mkdir -p /var/data/turtle-pool/wallet/config
mkdir -p /var/data/turtle-pool/wallet/container
mkdir -p /var/data/turtle-pool/wallet/logs
chown -R turtle-pool /var/data/turtle-pool/wallet/container
chown -R turtle-pool /var/data/turtle-pool/wallet/logs
```
Now create the initial wallet. You'll want to secure your wallet password, so the command below will **prompt** you for the key (no output from the prompt), and insert it into an environment variable. This means that the key won't be stored in plaintext in your bash history!
```
read PASS
```
After having entered your password (you can confirm by running ```env | grep PASS```), run the following to run the wallet daemon _once_, with the instruction to create a new wallet container:
```
docker run \
-v /var/data/turtle-pool/wallet/container:/container \
--rm -ti --entrypoint /usr/local/bin/walletd funkypenguin/turtlecoind \
--container-file /container/wallet.container \
--container-password $PASS \
--generate-container
```
You'll get a lot of output. The following are relevant lines from a successful run with the extra stuff stripped out:
```
2018-May-01 11:14:57.662664 INFO walled v0.4.3.1257 ()
2018-May-01 11:14:59.868087 INFO Generating new wallet
2018-May-01 11:14:59.919178 INFO Container initialized with view secret key, public view key <your view public key will be here>
2018-May-01 11:14:59.920932 INFO New wallet added TRTL<your wallet's public address>, creation timestamp 0
2018-May-01 11:14:59.932367 INFO Container shut down
2018-May-01 11:14:59.932419 INFO Loading container...
2018-May-01 11:14:59.961814 INFO Consumer added, consumer 0x55b0fb5bc070, count 1
2018-May-01 11:14:59.961996 INFO Starting...
2018-May-01 11:14:59.962173 INFO Container loaded, view public key <your view public key will be here>, wallet count 1, actual balance 0.00, pending balance 0.00
2018-May-01 11:14:59.962508 INFO New wallet is generated. Address: TRTL<your wallet's public address>
2018-May-01 11:14:59.962581 INFO Saving container...
2018-May-01 11:14:59.962683 INFO Stopping...
2018-May-01 11:14:59.962862 INFO Stopped
```
Take careful note of your wallet password, public view key, and wallet address (which starts with TRTL)
Create **/var/data/turtle-pool/wallet/config/wallet.conf**, containing the following:
```
bind-address = 0.0.0.0
container-file = /container/wallet.container
container-password = <ENTER YOUR CONTAINER PASSWORD HERE>
rpc-password = <CHOOSE A PASSWORD TO ALLOW POOL TO TALK TO WALLET>
log-file = /dev/stdout
log-level = 3
daemon-address = daemon
```
### Setup TurtleCoin mining pool
Following the convention we've set above, create directories to hold pool data:
```
mkdir -p /var/data/turtle-pool/pool/config
mkdir -p /var/data/turtle-pool/pool/logs
chown -R turtle-pool /var/data/turtle-pool/pool/logs
```
Now create **/var/data/turtle-pool/pool/config/config.json**, using https://github.com/turtlecoin/turtle-pool/blob/master/config.json as a guide, and adjusting at least the following:
Send logs to /logs/, so that they can potentially be stored / backed up separately from the config:
```
"logging": {
"files": {
"level": "debug",
"directory": "/logs",
"flushInterval": 5
},
```
Set the "poolAddress" field to your wallet address
```
"poolServer": {
"enabled": true,
"clusterForks": "auto",
"poolAddress": "<SET THIS TO YOUR WALLET ADDRESS GENERATED ABOVE>",
```
Add the "host" value to the api section, since our API will run on its own container, and choose a password you'll use for the webUI admin page
```
"api": {
"enabled": true,
"hashrateWindow": 600,
"updateInterval": 5,
"host": "pool-api",
"port": 8117,
"blocks": 30,
"payments": 30,
"password": "<PASSWORD FOR ADMIN PAGE ACCESS>"
```
Set the host value for the daemon:
```
"daemon": {
"host": "daemon",
"port": 11898
},
```
Set the host value for the wallet, and set your container password (_you recorded it earlier, remember?_)
```
"wallet": {
"host": "wallet",
"port": 8070,
"password": "<SET ME TO YOUR WALLET RPC PASSWORD>"
},
```
Set the host value for Redis:
```
"redis": {
"host": "redis",
"port": 6379
},
```
That's it! The above config files mean each element of the pool will be able to communicate with the other elements within the docker swarm, by name.
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
daemon:
image: funkypenguin/turtlecoind
volumes:
- /var/data/runtime/turtle-pool/daemon/1:/var/lib/turtlecoind/
- /etc/localtime:/etc/localtime:ro
networks:
- internal
- traefik_public
ports:
- 11897:11897
labels:
- traefik.frontend.rule=Host:explorer.trtl.heigh-ho.funkypenguin.co.nz
- traefik.docker.network=traefik_public
- traefik.port=11898
daemon-failsafe1:
image: funkypenguin/turtlecoind
volumes:
- /var/data/runtime/turtle-pool/daemon/failsafe1:/var/lib/turtlecoind/
- /etc/localtime:/etc/localtime:ro
networks:
- internal
daemon-failsafe2:
image: funkypenguin/turtlecoind
volumes:
- /var/data/runtime/turtle-pool/daemon/failsafe2:/var/lib/turtlecoind/
- /etc/localtime:/etc/localtime:ro
networks:
- internal
pool-pool:
image: funkypenguin/turtle-pool
volumes:
- /var/data/turtle-pool/pool/config:/config:ro
- /var/data/turtle-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
ports:
- 3333:3333
- 5555:5555
- 7777:7777
entrypoint: |
node init.js -module=pool -config=/config/config.json
pool-api:
image: funkypenguin/turtle-pool
volumes:
- /var/data/turtle-pool/pool/config:/config:ro
- /var/data/turtle-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:api.trtl.heigh-ho.funkypenguin.co.nz
- traefik.docker.network=traefik_public
- traefik.port=8117
entrypoint: |
node init.js -module=api -config=/config/config.json
pool-unlocker:
image: funkypenguin/turtle-pool
volumes:
- /var/data/turtle-pool/pool/config:/config:ro
- /var/data/turtle-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
entrypoint: |
node init.js -module=unlocker -config=/config/config.json
pool-payments:
image: funkypenguin/turtle-pool
volumes:
- /var/data/turtle-pool/pool/config:/config:ro
- /var/data/turtle-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
entrypoint: |
node init.js -module=payments -config=/config/config.json
pool-charts:
image: funkypenguin/turtle-pool
volumes:
- /var/data/turtle-pool/pool/config:/config:ro
- /var/data/turtle-pool/pool/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
entrypoint: |
node init.js -module=chartsDataCollector -config=/config/config.json
wallet:
image: funkypenguin/turtlecoind
volumes:
- /var/data/turtle-pool/wallet/config:/config:ro
- /var/data/turtle-pool/wallet/container:/container
- /var/data/turtle-pool/wallet/logs:/logs
- /etc/localtime:/etc/localtime:ro
networks:
- internal
entrypoint: |
walletd --config /config/wallet.conf | tee /logs/walletd.log
redis:
volumes:
- /var/data/turtle-pool/redis/config:/config:ro
- /var/data/turtle-pool/redis/data:/data
- /var/data/turtle-pool/redis/logs:/logs
- /etc/localtime:/etc/localtime:ro
image: redis
entrypoint: |
redis-server /config/redis.conf
networks:
- internal
nginx:
volumes:
- /var/data/turtle-pool/nginx/website:/usr/share/nginx/html:ro
- /etc/localtime:/etc/localtime:ro
image: nginx
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:trtl.heigh-ho.funkypenguin.co.nz
- traefik.docker.network=traefik_public
- traefik.port=80
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.21.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch the Turtle! 🐢
Launch the Turtle pool stack by running ```docker stack deploy turtle-pool -c <path -to-docker-compose.yml>```, and then run ```"```docker stack ps turtle-pool``` to ensure the stack has come up properly. (_See [troubleshooting](/reference/troubleshooting) if not_)
The first thing that'll happen is that TurtleCoind will start syncing the blockchain from the bootstrap data. You can watch this happening with ```docker service logs turtle-pool_daemon -f```. While the daemon is syncing, it won't respond to requests, so walletd, the pool, etc will be non-functional.
You can watch the various elements of the pool doing their thing, by running ```tail -f /var/data/turtle-pool/pool/logs/*.log```
### So how do I mine to it?
That.. is another recipe. Start with the "[CryptoMiner](/recipes/cryptominer/)" uber-recipe for GPU/rig details, grab a copy of xmr-stack (_patched for the forked TurtleCoin_) from https://github.com/turtlecoin/trtl-stak/releases, and follow your nose. Jump into the TurtleCoin discord (_below_) #mining channel for help.
### What to do if it breaks?
TurtleCoin is a baby cryptocurrency. There are scaling issues to solve, and large amounts components of this recipe are under rapid development. So, elements may break/change in time, and this recipe itself is a work-in-progress.
Jump into the [TurtleCoin Discord server](http://chat.turtlecoin.lol/) to ask questions, contribute, and send/receive some TRTL tips!
## Chef's Notes
1. Because Docker Swarm performs ingress NAT for its load-balanced "routing mesh", the source address of inbound miner traffic is rewritten to a (_common_) docker node IP address. This means it's [not possible to determine the actual source IP address](https://github.com/moby/moby/issues/25526) of a miner. Which, in turn, means that any **one** misconfigured miner could trigger an IP ban, and lock out all other miners for 5 minutes at a time.
Two possible solutions to this are (1) disable banning, or (2) update the pool banning code to ban based on a combination of IP address and miner wallet address. I'll be working on a change to implement #2 if this becomes a concern.
2. The traefik labels in the docker-compose are to permit automatic LetsEncrypt SSL-protected proxying of your pool UI and API addresses.
3. After a [power fault in my datacenter caused daemon DB corruption](https://www.reddit.com/r/TRTL/comments/8jftzt/funky_penguin_nz_mining_pool_down_with_daemon/), I added a second daemon, running in parallel to the first. The failsafe daemon runs once an hour, syncs with the running daemons, and shuts down again, providing a safely halted version of the daemon DB for recovery.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
Also, you could send me some :turtle: ❤️ to _TRTLv2qCKYChMbU5sNkc85hzq2VcGpQidaowbnV2N6LAYrFNebMLepKKPrdif75x5hAizwfc1pX4gi5VsR9WQbjQgYcJm21zec4_
### Your comments? 💬
manuscript/recipes/wallabag.md
+208
View File
@@ -0,0 +1,208 @@
hero: Read-it-later, mate!
# Wallabag
Wallabag is a self-hosted webapp which allows you to save URLs to "read later", similar to [Instapaper](https://www.instapaper.com/u) or [Pocket](https://getpocket.com/a/). Like Instapaper (_but **not** Pocket, sadly_), Wallabag allows you to **annotate** any pages you grab for your own reference.
All saved data (_pages, annotations, images, tags, etc_) are stored on your own server, and can be shared/exported in a variety of formats, including ePub and PDF.
![Wallabag Screenshot](../images/wallabag.png)
There are plugins for [Chrome](https://chrome.google.com/webstore/detail/wallabagger/gbmgphmejlcoihgedabhgjdkcahacjlj) and [Firefox](https://addons.mozilla.org/firefox/addon/wallabagger/), as well as apps for [iOS](https://appsto.re/fr/YeqYfb.i), [Android](https://play.google.com/store/apps/details?id=fr.gaulupeau.apps.InThePoche), etc. Wallabag will also integrate nicely with my favorite RSS reader, [Miniflux](https://miniflux.net/) (_for which there is an [existing recipe](/recipes/miniflux)_).
[Here's a video](https://player.vimeo.com/video/167435064) which shows off the UI a bit more.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Setup data locations
We need a filesystem location to store images that Wallabag downloads from the original sources, to re-display when you read your articles, as well as nightly database dumps (_which you **should [backup](/recipes/duplicity/)**_), so create something like this:
```
mkdir -p /var/data/wallabag
cd /var/data/wallabag
mkdir -p {images,db-dump}
```
### Prepare environment
Create wallabag.env, and populate with the following variables. The only variable you **have** to change is SYMFONY__ENV__DOMAIN_NAME - this **must** be the URL that your Wallabag instance will be available at (_else you'll have no CSS_)
```
# For the DB container
POSTGRES_PASSWORD=wallabag
POSTGRES_USER=wallabag
# For the wallabag container
SYMFONY__ENV__DATABASE_DRIVER=pdo_pgsql
SYMFONY__ENV__DATABASE_HOST=db
SYMFONY__ENV__DATABASE_PORT=5432
SYMFONY__ENV__DATABASE_NAME=wallabag
SYMFONY__ENV__DATABASE_USER=wallabag
SYMFONY__ENV__DATABASE_PASSWORD=wallabag
SYMFONY__ENV__DOMAIN_NAME=https://wallabag.example.com
SYMFONY__ENV__DATABASE_DRIVER_CLASS=Wallabag\CoreBundle\Doctrine\DBAL\Driver\CustomPostgreSQLDriver
SYMFONY__ENV__MAILER_HOST=127.0.0.1
SYMFONY__ENV__MAILER_USER=~
SYMFONY__ENV__MAILER_PASSWORD=~
SYMFONY__ENV__FROM_EMAIL=wallabag@example.com
SYMFONY__ENV__FOSUSER_REGISTRATION=false
# If you decide to protect wallabag with an oauth_proxy, complete these
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
```
Now create wallabag-backup.env in the same folder, with the following contents. (_This is necessary to prevent environment variables required for backup from breaking the DB container_)
```
# For database backups
PGUSER=wallabag
PGPASSWORD=wallabag
PGHOST=db
BACKUP_NUM_KEEP=7
BACKUP_FREQUENCY=1d
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
wallabag:
image: wallabag/wallabag
env_file: /var/data/config/wallabag/wallabag.env
networks:
- internal
volumes:
- /var/data/wallabag/images:/var/www/wallabag/web/assets/images
wallabag_proxy:
image: a5huynh/oauth2_proxy
env_file: /var/data/config/wallabag/wallabag.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:wallabag.example.com
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /var/data/config/wallabag/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://wallabag:80
-redirect-url=https://wallabag.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
db:
image: postgres
env_file: /var/data/config/wallabag/wallabag.env
dns_search:
- hq.example.com
volumes:
- /var/data/runtime/wallabag/data:/var/lib/postgresql/data
networks:
- internal
db-backup:
image: postgres:latest
env_file: /var/data/config/wallabag/wallabag-backup.env
volumes:
- /var/data/wallabag/database-dump:/dump
- /etc/localtime:/etc/localtime:ro
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
sleep 2m
while /bin/true; do
pg_dump -Fc > /dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.psql
(ls -t /dump/dump*.psql|head -n $$BACKUP_NUM_KEEP;ls /dump/dump*.psql)|sort|uniq -u|xargs rm -- {}
sleep $$BACKUP_FREQUENCY
done
EOF'
networks:
- internal
redis:
image: redis:alpine
networks:
- internal
import-instapaper:
image: wallabag/wallabag
env_file: /var/data/config/wallabag/wallabag.env
networks:
- internal
command: |
import instapaper
import-pocket:
image: wallabag/wallabag
env_file: /var/data/config/wallabag/wallabag.env
networks:
- internal
command: |
import pocket
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.21.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Wallabag stack
Launch the Wallabag stack by running ```docker stack deploy wallabag -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user "wallabag" and default password "wallabag".
### Enable asynchronous imports
You'll have noticed redis, plus the pocket/instapaper-importing containers included in the .yml above. Redis is there to allow [asynchronous](https://github.com/wallabag/doc/blob/master/en/admin/asynchronous.md) imports, and pocket and instapaper are there since they're likely the most popular platform you'd _want_ to import from. Other possibilities (_you'll need to adjust the .yml_) are **readability**, **firefox**, **chrome**, and **wallabag_v1** and **wallabag_v2**.
Even with all these elements in place, you still need to enable Redis under Internal Settings -> Import, via the **admin** user in the webUI. Here's a screenshot to help you find it:
![Wallabag Imports](../images/wallabag_imports.png)
## Chef's Notes
1. If you wanted to expose the Wallabag UI directly (_required for the iOS/Android apps_), you could remove the oauth2_proxy from the design, and move the traefik-related labels directly to the wallabag container. You'd also need to add the traefik_public network to the wallabag container. I found the iOS app to be unreliable and clunky, so elected to leave my oauth_proxy enabled, and to simply use the webUI on my mobile devices instead. YMMMV.
2. I've not tested the email integration, but you'd need an SMTP server listening on port 25 (_since we can't change the port_) to use it
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/wekan.md
+150
View File
@@ -0,0 +1,150 @@
# Wekan
Wekan is an open-source kanban board which allows a card-based task and to-do management, similar to tools like WorkFlowy or Trello.
![Wekan Screenshot](../images/wekan.jpg)
Wekan allows to create Boards, on which Cards can be moved around between a number of Columns. Boards can have many members, allowing for easy collaboration, just add everyone that should be able to work with you on the board to it, and you are good to go! You can assign colored Labels to cards to facilitate grouping and filtering, additionally you can add members to a card, for example to assign a task to someone.
There's a [video](https://www.youtube.com/watch?v=N3iMLwCNOro) of the developer showing off the app, as well as a f[unctional demo](https://wekan.indie.host/b/t2YaGmyXgNkppcFBq/wekan-fork-roadmap).
!!! note
For added privacy, this design secures wekan behind an [oauth2 proxy](/reference/oauth_proxy/), so that in order to gain access to the wekan UI at all, oauth2 authentication (_to GitHub, GitLab, Google, etc_) must have already occurred.
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik) configured per design
## Preparation
### Setup data locations
We'll need several directories to bind-mount into our container, so create them in /var/data/wekan:
```
mkdir /var/data/wekan
cd /var/data/wekan
mkdir -p {wekan-db,wekan-db-dump}
```
### Prepare environment
You'll need to know the following:
1. Choose an oauth provider, and obtain a client ID and secret
2. Create wekan.env, and populate with the following variables
```
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
MONGO_URL=mongodb://wekandb:27017/wekan
ROOT_URL=https://wekan.example.com
MAIL_URL=smtp://wekan@wekan.example.com:password@mail.example.com:587/
MAIL_FROM="Wekan <wekan@wekan.example.com>"
# Mongodb specific database dump details
BACKUP_NUM_KEEP=7
BACKUP_FREQUENCY=1d
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: '3'
services:
wekandb:
image: mongo:latest
command: mongod --smallfiles --oplogSize 128
networks:
- internal
volumes:
- /var/data/runtime/wekan/database:/data/db
- /var/data/wekan/database-dump:/dump
proxy:
image: a5huynh/oauth2_proxy
env_file: /var/data/config/wekan/wekan.env
networks:
- traefik
- internal
volumes:
- /var/data/oauth_proxy/authenticated-emails.txt:/authenticated-emails.txt
deploy:
labels:
- traefik.frontend.rule=Host:wekan.example.com
- traefik.docker.network=traefik
- traefik.port=4180
command: |
-cookie-secure=false
-upstream=http://wekan:80
-redirect-url=https://wekan.example.com
-http-address=http://0.0.0.0:4180
-email-domain=example.com
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
wekan:
image: wekanteam/wekan:latest
networks:
- internal
env_file: /var/data/config/wekan/wekan.env
db-backup:
image: mongo:latest
env_file : /var/data/config/wekan/wekan.env
volumes:
- /var/data/wekan/database-dump:/dump
- /etc/localtime:/etc/localtime:ro
entrypoint: |
bash -c 'bash -s <<EOF
trap "break;exit" SIGHUP SIGINT SIGTERM
sleep 2m
while /bin/true; do
mongodump -h db --gzip --archive=/dump/dump_\`date +%d-%m-%Y"_"%H_%M_%S\`.mongo.gz
(ls -t /dump/dump*.mongo.gz|head -n $$BACKUP_NUM_KEEP;ls /dump/dump*.mongo.gz)|sort|uniq -u|xargs rm -- {}
sleep $$BACKUP_FREQUENCY
done
EOF'
networks:
- internal
networks:
traefik:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.3.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Wekan stack
Launch the Wekan stack by running ```docker stack deploy wekan -c <path -to-docker-compose.yml>```
Log into your new instance at https://**YOUR-FQDN**, with user "root" and the password you specified in gitlab.env.
## Chef's Notes
1. If you wanted to expose the Wekan UI directly, you could remove the oauth2_proxy from the design, and move the traefik-related labels directly to the wekan container. You'd also need to add the traefik network to the wekan container.
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/wetty.md
+110
View File
@@ -0,0 +1,110 @@
hero: Terminal in a browser, baby! 💻
# Wetty
[Wetty](https://github.com/krishnasrinivas/wetty) is a responsive, modern terminal, in your web browser. Yes, your browser. When combined with secure authentication and SSL encryption, it becomes a useful tool for quick and easy remote access.
![Wetty Screenshot](../images/wetty.png)
## Why would you need SSH in a browser window?
Need shell access to a node with no external access? Deploy Wetty behind an [oauth_proxy](/reference/oauth_proxy/) with a SSL-terminating reverse proxy ([traefik](/ha-docker-swarm/traefik/)), and suddenly you have the means to SSH to your private host from any web browser (_protected by your [oauth_proxy](/reference/oauth_proxy/) of course, and your OAuth provider's 2FA_)
Here are some other possible use cases:
1. Access to SSH / CLI from an environment where outgoing SSH is locked down, or SSH client isn't / can't be installed. (_i.e., a corporate network_)
2. Access to long-running processes inside a tmux session (_like [irrsi](https://irssi.org/)_)
3. Remote access to a VM / [container running Kali linux](https://github.com/offensive-security/kali-linux-docker), for penetration testing
## Ingredients
1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md)
2. [Traefik](/ha-docker-swarm/traefik_public) configured per design
3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP
## Preparation
### Prepare environment
Create wetty.env, and populate with the following variables per the [oauth_proxy](/reference/oauth_proxy/) instructions:
```
OAUTH2_PROXY_CLIENT_ID=
OAUTH2_PROXY_CLIENT_SECRET=
OAUTH2_PROXY_COOKIE_SECRET=
# To use WeTTY to SSH to a host besides the (mostly useless) alpine container it comes with
SSHHOST=batcomputer.batcave.com
SSHUSER=batman
```
### Setup Docker Swarm
Create a docker swarm config file in docker-compose syntax (v3), something like this:
!!! tip
I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍
```
version: "3"
services:
wetty:
image: krishnasrinivas/wetty
env_file : /var/data/config/wetty/wetty.env
networks:
- internal
proxy:
image: funkypenguin/oauth2_proxy:latest
env_file: /var/data/config/wetty/wetty.env
networks:
- internal
- traefik_public
deploy:
labels:
- traefik.frontend.rule=Host:wetty.funkypenguin.co.nz
- traefik.docker.network=traefik_public
- traefik.port=4180
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/data/config/wetty/authenticated-emails.txt:/authenticated-emails.txt
command: |
-cookie-secure=false
-upstream=http://wetty:3000
-redirect-url=https://wetty.funkypenguin.co.nz
-http-address=http://0.0.0.0:4180
-provider=github
-authenticated-emails-file=/authenticated-emails.txt
networks:
traefik_public:
external: true
internal:
driver: overlay
ipam:
config:
- subnet: 172.16.45.0/24
```
!!! note
Setup unique static subnets for every stack you deploy. This avoids IP/gateway conflicts which can otherwise occur when you're creating/removing stacks a lot. See [my list](/reference/networks/) here.
## Serving
### Launch Wetty stack
Launch the Wetty stack by running ```docker stack deploy wetty -c <path -to-docker-compose.yml>```
Browse to your new browser-cli-terminal at https://**YOUR-FQDN**. Authenticate with your OAuth provider, and then proceed to login, either to the remote host you specified (_batcomputer.batcave.com, in the example above_), or using user and password "term" to log directly into the Wetty alpine container (_from which you can establish egress SSH_)
## Chef's Notes
1. You could set SSHHOST to the IP of the "docker0" interface on your host, which is normally 172.17.0.1. (_Or run ```/sbin/ip route|awk '/default/ { print $3 }'``` in the container_) This would then provide you the ability to remote-manage your swarm with only web access to Wetty.
2. The inclusion of Wetty was due to the efforts of @gpulido in our [Discord server](http://chat.funkypenguin.co.nz). Thanks Gabriel!
### Tip your waiter (donate) 👏
Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏
### Your comments? 💬
manuscript/recipes/workflow.md
+8
View File
@@ -0,0 +1,8 @@
The workflow for creating a recipe
1. In my gitlab repo for the cookbook, I create an issue to track the new recipe I want to create
2. From the issue, I create a branch
3. I pull the branch locally, "git pull" followed by "git co <branch>"
4. I "git add" my changes, commit, and push the branch
5. I create a merge request in the branch to track merging into produced
6. I set request to merge when pipeline succeeds