diff --git a/manuscript/CHANGELOG.md b/manuscript/CHANGELOG.md index ed5edf4..005469c 100644 --- a/manuscript/CHANGELOG.md +++ b/manuscript/CHANGELOG.md @@ -13,14 +13,13 @@ ## Recently added recipes +* Added [Minio](/recipes/minio/), a high performance distributed object storage server, designed for large-scale private cloud infrastructure, but perfect for simple use cases where emulating AWS S3 is useful. (_27 Jan 2019_) * Added the beginning of the **Kubernetes** design, including a getting started on using [Digital Ocean,](/kubernetes/digitalocean/) and a WIP recipe for an [MQTT](/recipes/mqtt/) broker (_21 Jan 2019_) * [ElkarBackup](/recipes/elkarbackup/), a beautiful GUI-based backup solution built on rsync/rsnapshot (_1 Jan 2019_) * Added [Collabora Online](/recipes/collabora-online), an rich document editor within [NextCloud](/recipes/nextcloud/) (_think "headless LibreOffice"_) * Added [phpIPAM](/recipes/phpipam), an IP address management tool (_18 Dec 2018_) * Added [KeyCloak](/recipes/keycloak), an open source identity and access management solution which backends neatly into [OpenLDAP](/recipes/openldap/) (_among other providers_), providing true SSO (_13 Dec 2018_) * Added [OpenLDAP](/recipes/openldap/), a 20-year old project which [refuses to die](https://www.youtube.com/watch?v=cnQEo4bazIo), underpinning many of today's authentication platforms, and providing a single authentication backend for multiple recipes (_9 Dec 2018_) -* Added [Wetty](/recipes/wetty/), a remote terminal client in your web browser (_22 Nov 2018_) - ## Recent improvements diff --git a/manuscript/book.txt b/manuscript/book.txt index ec5eb43..efd13aa 100644 --- a/manuscript/book.txt +++ b/manuscript/book.txt @@ -14,7 +14,6 @@ ha-docker-swarm/traefik.md ha-docker-swarm/docker-swarm-mode.md ha-docker-swarm/duplicity.md -<<<<<<< HEAD sections/recipes.md recipes/autopirate.md recipes/elkarbackup.md @@ -28,6 +27,7 @@ recipes/huginn.md recipes/kanboard.md recipes/mail.md recipes/miniflux.md +recipes/minio.md recipes/munin.md recipes/nextcloud.md recipes/owntracks.md @@ -39,23 +39,6 @@ recipes/privatebin.md recipes/swarmprom.md recipes/wekan.md -======= -sections/recipies.md -recipies/mail.md -recipies/gitlab.md -recipies/gitlab-runner.md -recipies/wekan.md -recipies/huginn.md -recipies/kanboard.md -recipies/miniflux.md -recipies/ghost.md -recipies/piwik.md -recipies/autopirate.md -recipies/nextcloud.md -recipies/portainer.md -recipies/turtle-pool.md -recipies/tiny-tiny-rss.md ->>>>>>> markdown-to-markua sections/reference.md reference/oauth_proxy.md diff --git a/manuscript/images/minio.png b/manuscript/images/minio.png new file mode 100644 index 0000000..2cd283f Binary files /dev/null and b/manuscript/images/minio.png differ diff --git a/manuscript/recipes/minio.md b/manuscript/recipes/minio.md new file mode 100644 index 0000000..87d0043 --- /dev/null +++ b/manuscript/recipes/minio.md @@ -0,0 +1,183 @@ +# Minio + +Minio is a high performance distributed object storage server, designed for +large-scale private cloud infrastructure. + +However, at its simplest, Minio allows you to expose a local filestructure via the [Amazon S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html). You could, for example, use it to provide access to "buckets" (folders) of data on your filestore, secured by access/secret keys, just like AWS S3. You can further interact with your "buckets" with common tools, just as if they were hosted on S3. + +Under a more advanced configuration, Minio runs in distributed mode, with [features](https://www.minio.io/features.html) including high-availability, mirroring, erasure-coding, and "bitrot detection". + +![Minio Screenshot](../images/minio.png) + +Possible use-cases: + +1. Sharing files (_protected by user accounts with secrets_) via HTTPS, either as read-only or read-write, in such a way that the bucket could be mounted to a remote filesystem using common S3-compatible tools, like [goofys](https://github.com/kahing/goofys). Ever wanted to share a folder with friends, but didn't want to open additional firewall ports etc? +2. Simulating S3 in a dev environment +3. Mirroring an S3 bucket locally + +## Ingredients + +1. [Docker swarm cluster](/ha-docker-swarm/design/) with [persistent shared storage](/ha-docker-swarm/shared-storage-ceph.md) +2. [Traefik](/ha-docker-swarm/traefik_public) configured per design +3. DNS entry for the hostname you intend to use, pointed to your [keepalived](ha-docker-swarm/keepalived/) IP + +## Preparation + +### Setup data locations + +We'll need a directory to hold our minio file store, as well as our minio client config, so create a structure at /var/data/minio: + +``` +mkdir /var/data/minio +cd /var/data/minio +mkdir -p {mc,data} +``` + +### Prepare environment + +Create minio.env, and populate with the following variables +``` +MINIO_ACCESS_KEY= +MINIO_SECRET_KEY= +``` + +### Setup Docker Swarm + +Create a docker swarm config file in docker-compose syntax (v3), something like this: + +!!! tip + I share (_with my [patreon patrons](https://www.patreon.com/funkypenguin)_) a private "_premix_" git repository, which includes necessary docker-compose and env files for all published recipes. This means that patrons can launch any recipe with just a ```git pull``` and a ```docker stack deploy``` 👍 + + +``` +version: '3.1' + +services: + app: + image: minio/minio + env_file: /var/data/config/minio/minio.env + volumes: + - /var/data/minio/data:/data + networks: + - traefik_public + deploy: + labels: + - traefik.frontend.rule=Host:minio.example.com + - traefik.port=9000 + command: minio server /data + +networks: + traefik_public: + external: true +``` + +## Serving + +### Launch Minio stack + +Launch the Minio stack by running ```docker stack deploy minio -c ``` + +Log into your new instance at https://**YOUR-FQDN**, with the access key and secret key you specified in minio.env. + +If you created ```/var/data/minio```, you'll see nothing. If you referenced existing data, you should see all subdirectories in your existing folder represented as buckets. + +If all you need is single-user access to your data, you're done! 🎉 + +If, however, you want to expose data to multiple users, at different privilege levels, you'll need the minio client to create some users and (_potentially_) policies... + +### Setup minio client + +To administer the Minio server, we need the Minio client. While it's possible to download the minio client and run it locally, it's just as easy to do it within a small (5Mb) container. + +I created an alias on my docker nodes, allowing me to run mc quickly: + +``` +alias mc='docker run -it -v /docker/minio/mc/:/root/.mc/ --network traefik_public minio/mc' +``` + +Now I use the alias to launch the client shell, and connect to my minio instance (_I could also use the external, traefik-provided URL_) + +``` +root@ds1:~# mc config host add minio http://app:9000 admin iambatman +mc: Configuration written to `/root/.mc/config.json`. Please update your access credentials. +mc: Successfully created `/root/.mc/share`. +mc: Initialized share uploads `/root/.mc/share/uploads.json` file. +mc: Initialized share downloads `/root/.mc/share/downloads.json` file. +Added `minio` successfully. +root@ds1:~# +``` + +### Add (readonly) user + +Use mc to add a (readonly or readwrite) user, by running ``` mc admin user add minio ``` + +Example: + +``` +root@ds1:~# mc admin user add minio spiderman peterparker readonly +Added user `spiderman` successfully. +root@ds1:~# +``` + +Confirm by listing your users (_admin is excluded from the list_): + +``` +root@node1:~# mc admin user list minio +enabled spiderman readonly +root@node1:~# +``` + +### Make a bucket accessible to users + +By default, all buckets have no "policies" attached to them, and so can only be accessed by the administrative user. Having created some readonly/read-write users above, you'll be wanting to grant them access to buckets. + +The simplest permission scheme is "on or off". Either a bucket has a policy, or it doesn't. (_I believe you can apply policies to subdirectories of buckets in a more advanced configuration_) + +After **no** policy, the most restrictive policy you can attach to a bucket is "download". This policy will allow authenticated users to download contents from the bucket. Apply the "download" policy to a bucket by running ```mc policy download minio/```, i.e.: + +``` +root@ds1:# mc policy download minio/comics +Access permission for `minio/comics` is set to `download` +root@ds1:# +``` + +### Advanced bucketing + +There are some clever complexities you can achieve with user/bucket policies, including: + +* A public bucket, which requires no authentication to read or even write (_for a public dropbox, for example_) +* A special bucket, hidden from most users, but available to VIP users by application of a custom "[canned policy](https://docs.minio.io/docs/minio-multi-user-quickstart-guide.html)" + +### Mount a minio share remotely + +Having setup your buckets, users, and policies - you can give out your minio external URL, and user access keys to your remote users, and they can S3-mount your buckets, interacting with them based on their user policy (_read-only or read/write_) + +I tested the S3 mount using [goofys](https://github.com/kahing/goofys), "a high-performance, POSIX-ish Amazon S3 file system written in Go". + +First, I created ~/.aws/credentials, as follows: + +``` +[default] +aws_access_key_id=spiderman +aws_secret_access_key=peterparker +``` + +And then I ran (_in the foreground, for debugging_), ```goofys --f -debug_s3 --debug_fuse --endpoint=https://traefik.example.com ``` + +To permanently mount an S3 bucket using goofys, I'd add something like this to /etc/fstab: + +``` +goofys#bucket /mnt/mountpoint fuse _netdev,allow_other,--file-mode=0666 0 0 +``` + +## Chef's Notes + +1. There are many S3-filesystem-mounting tools available, I just picked Goofys because it's simple. Google is your friend :) +2. Some applications (_like [NextCloud](/recipes/nextcloud/)_) can natively mount S3 buckets +3. Some backup tools (_like [Duplicity](/recipes/duplicity/)_) can backup directly to S3 buckets + +### Tip your waiter (donate) 👏 + +Did you receive excellent service? Want to make your waiter happy? (_..and support development of current and future recipes!_) See the [support](/support/) page for (_free or paid)_ ways to say thank you! 👏 + +### Your comments? 💬 diff --git a/mkdocs.yml b/mkdocs.yml index fe0b0b0..bba8a36 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -93,6 +93,7 @@ pages: - Gollum: recipes/gollum.md - InstaPy: recipes/instapy.md - KeyCloak: recipes/keycloak.md + - Minio: recipes/minio.md - OpenLDAP: recipes/openldap.md - Piwik: recipes/piwik.md - Portainer: recipes/portainer.md @@ -112,7 +113,7 @@ pages: - Work-in-Progress: # - MatterMost: recipes/mattermost.md - IPFS Cluster: recipes/ipfs-cluster.md - - MQTT: recipes/mqtt.md + - MQTT: recipes/mqtt.md # - HackMD: recipes/hackmd.md # - Mastodon: recipes/mastodon.md # - Mayan EDMS: recipes/mayan-edms.md