public/geek-cookbook
1
0
Fork 0
mirror of https://github.com/funkypenguin/geek-cookbook/ synced 2025-12-13 01:36:23 +00:00
Code Issues Releases Wiki Activity
Files
1e31251c63bbee9c9e07774aaad7f702bb2506d8
geek-cookbook/print-preview.html
David Young bb2b358ec6 Improve title on recipes
2022-07-10 20:44:18 +12:00

529 lines
90 KiB
HTML
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<html><head><meta charset="utf-8"/><meta content="width=device-width,initial-scale=1" name="viewport"/><meta content="My collection of how-to guides and tutorials for establishing your own container-based awesome selfhosted platform, using either Docker or Kubernetes." name="description"/><meta content="David Young" name="author"/><link href="https://geek-cookbook.funkypenguin.co.nz/" rel="canonical"/><link href="file:///Users/davidy/Documents/Personal/Projects/geek-cookbook/site/images/site-logo.svg" rel="icon"/><meta content="mkdocs-1.3.0, mkdocs-material-8.3.9" name="generator"/><title>How I do "awesome selfhosted" |・∀・</title><link href="file:///Users/davidy/Documents/Personal/Projects/geek-cookbook/site/assets/stylesheets/main.1d29e8d0.min.css" rel="stylesheet"/><link href="file:///Users/davidy/Documents/Personal/Projects/geek-cookbook/site/assets/stylesheets/palette.cbb835fc.min.css" rel="stylesheet"/><script>__md_scope=new URL(".",location),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script><!-- <script src="https://cdn.jsdelivr.net/npm/@widgetbot/crate@3" async defer></script> --><script src="/js/i-am-groot.js"></script><script>window.plausible = window.plausible || function() { (window.plausible.q = window.plausible.q || []).push(arguments) }</script><!-- RightMessage --><script type="text/javascript">
(function(p, a, n, d, o, b, c) {
o = n.createElement('script'); o.type = 'text/javascript'; o.async = true; o.src = 'https://tb.rightmessage.com/'+p+'.js';
b = n.getElementsByTagName('script')[0]; d = function(h, u, i) { var c = n.createElement('style'); c.id = 'rmcloak'+i;
c.type = 'text/css'; c.appendChild(n.createTextNode('.rmcloak'+h+'{visibility:hidden}.rmcloak'+u+'{display:none}'));
b.parentNode.insertBefore(c, b); return c; }; c = d('', '-hidden', ''); d('-stay-invisible', '-stay-hidden', '-stay');
setTimeout(o.onerror = function() { c.parentNode && c.parentNode.removeChild(c); }, a); b.parentNode.insertBefore(o, b);
})('1802694484', 20000, document);
</script><link href="file:///Users/davidy/Documents/Personal/Projects/geek-cookbook/site/assets/stylesheets/extra-style.w7lhf_wh.min.css" rel="stylesheet"/><style>@charset "UTF-8";
:root {
string-set: author 'David Young', copyright 'Copyright © 2016 - 2021 David Young, Funky Penguin Limited', title 'Funky Penguin\27s Geek Cookbook'; }
h1, h2, h3 {
string-set: chapter content(); }
@page {
size: a4 portrait;
margin: 25mm 10mm 25mm 10mm;
counter-increment: page;
white-space: pre;
color: grey;
@top-right {
font-size: 7pt;
content: string(chapter); }
@bottom-center {
font-size: 8pt;
content: "- " counter(page) "/" counter(pages) " -"; }
@bottom-right {
font-size: 6pt;
content: string(copyright); } }
@media print {
body {
background: none;
background-color: #fff; } }
article,
section + section {
page-break-before: always;
min-height: 100vh;
/* Defining all page breaks */ }
article,
article h1,
article h2,
article h3,
article h4,
article h5,
article h6,
article div.tabbed-content--wrap > .tabbed-content,
section + section,
section + section h1,
section + section h2,
section + section h3,
section + section h4,
section + section h5,
section + section h6,
section + section div.tabbed-content--wrap > .tabbed-content {
clear: both; }
article h1,
article h2,
article h3,
article h4,
article h5,
article h6,
article img,
section + section h1,
section + section h2,
section + section h3,
section + section h4,
section + section h5,
section + section h6,
section + section img {
page-break-after: avoid;
page-break-inside: avoid; }
article a,
article blockquote,
article table,
article ul,
article ol,
section + section a,
section + section blockquote,
section + section table,
section + section ul,
section + section ol {
page-break-before: avoid;
page-break-inside: avoid; }
article pre,
article dt + dd,
section + section pre,
section + section dt + dd {
page-break-before: avoid; }
article h1 + ul,
article h2 + ul,
article h3 + ul,
article h4 + ul,
article h5 + ul,
article h6 + ul,
article h1 + ol,
article h2 + ol,
article h3 + ol,
article h4 + ol,
article h5 + ol,
article h6 + ol,
section + section h1 + ul,
section + section h2 + ul,
section + section h3 + ul,
section + section h4 + ul,
section + section h5 + ul,
section + section h6 + ul,
section + section h1 + ol,
section + section h2 + ol,
section + section h3 + ol,
section + section h4 + ol,
section + section h5 + ol,
section + section h6 + ol {
page-break-inside: auto; }
article div.admonition,
section + section div.admonition {
page-break-inside: avoid; }
article div.tabbed-content--wrap,
section + section div.tabbed-content--wrap {
page-break-inside: avoid; }
article * + div.highlight,
article * + div.tabbed-set,
section + section * + div.highlight,
section + section * + div.tabbed-set {
page-break-before: avoid; }
section.two-columns {
columns: 2;
column-gap: 1cm; }
article h1 {
border-bottom: 2px solid #f00; }
article h2 {
border-bottom: 1px solid #f99; }
article h3 {
border-bottom: 0.5px solid #eee; }
article h1 > .pdf-order,
article h2 > .pdf-order,
article h3 > .pdf-order {
padding-left: 6px; }
article#doc-toc {
margin: 0 .8rem 1.2rem;
padding-top: .6rem;
font-size: 12pt;
line-height: 1.6; }
article#doc-toc > h1 {
margin: 0 0 2rem;
color: rgba(0, 0, 0, 0.54);
font-size: 1.2rem;
font-weight: 300;
letter-spacing: -.01em;
page-break-after: auto;
page-break-inside: auto;
bookmark-level: none; }
article#doc-toc > h1::before {
display: block;
margin-top: -9px;
padding-top: 9px;
content: ""; }
article#doc-toc ul {
list-style: none;
padding-left: 0;
margin-left: 1em;
font-size: 8pt;
page-break-inside: auto !important; }
article#doc-toc li {
list-style: none; }
article#doc-toc li > a {
color: inherit;
text-decoration: none;
padding-top: 0.4rem;
border-bottom: 1px dotted #ddd;
display: inline-block;
width: 100%; }
article#doc-toc li > a > .pdf-order {
padding-right: 0.5em; }
article#doc-toc li > a::after {
float: right;
content: target-counter(attr(href), page); }
@media print {
html {
font-size: 95%;
font-family: 'Kelmscott Roman NF', 'Noto Serif JP', serif; }
body {
font-size: 10.5pt;
font-family: 'Classico URW T OT', 'Noto Serif JP', serif; }
article {
font-size: 0.8rem;
line-height: 1.6; }
h1,
h2,
h3,
h4,
h5,
h6,
label {
font-family: 'Gentium Book Basic', 'Helvetica Neue', Helvetica, Arial, 'Noto Sans JP', 'Roboto', sans-serif; }
pre,
code,
var,
samp,
kbd,
tt {
font-family: 'Noto Sans Mono CJK JP', 'Courier New', monospace, serif;
font-size: 0.8em; }
pre code,
pre var,
pre samp,
pre kbd,
pre tt {
font-size: 100%; } }
@page :first {
@top-right {
content: normal; }
@bottom-center {
content: normal; }
@bottom-right {
content: normal; } }
article#doc-cover {
display: flex;
flex-direction: column;
height: 100vh; }
article#doc-cover > .wrapper {
flex: 1 0 auto;
text-align: center;
padding: 0 3rem; }
article#doc-cover > .wrapper.upper {
flex: 1 1 auto;
max-height: 30vh; }
article#doc-cover > .wrapper.upper > .logo {
width: 100%;
height: 100%;
background-position: center bottom;
background-repeat: no-repeat;
background-size: contain; }
article#doc-cover > .wrapper h1 {
font-size: 36pt;
line-height: 1.4; }
article#doc-cover > .wrapper h2 {
border: none;
bookmark-level: none; }
article#doc-cover > .properties {
flex: 0 0 auto;
text-align: right;
padding: 0 2rem; }
article#doc-cover > .properties p#copyright {
font-size: 80%; }
article#doc-back_cover {
page: back_cover;
height: 100vh;
display: flex;
flex-direction: column;
text-align: center; }
article#doc-back_cover .wrapper {
flex: 1 0 auto;
display: inline-block;
background: #eee; }
article#doc-back_cover .qrcode {
display: inline-block;
flex: 0 0 auto;
background: #eee; }
article#doc-back_cover .qrcode > a {
display: inline-block;
padding: 0.5em 2em;
border-radius: 1em;
background-color: #fff; }
article#doc-back_cover .qrcode > a img {
width: 100%;
height: auto; }
@page back_cover {
background-color: #eee;
@top-right {
content: normal; }
@bottom-center {
content: normal; }
@bottom-left {
font-size: 8pt;
content: string(title); }
@bottom-right {
font-size: 8pt;
content: string(copyright); } }
@media print {
html {
font-size: 95%;
font-family: 'Ubuntu'; }
body {
font-size: 11.5pt;
font-family: 'Ubuntu'; }
article {
font-size: 0.8rem;
line-height: 1.6; }
h1,
h2,
h3,
h4,
h5,
h6,
label {
font-family: 'Gentium Book Basic', 'Helvetica Neue', Helvetica, Arial, 'Noto Sans JP', 'Roboto', sans-serif; }
pre,
code,
var,
samp,
kbd,
tt {
font-family: 'Noto Sans Mono CJK JP', 'Courier New', monospace, serif;
font-size: 0.8em; }
pre code,
pre var,
pre samp,
pre kbd,
pre tt {
font-size: 100%; } }
</style><style>@media print {
.md-typeset .tabbed-set {
display: block; }
.md-typeset .tabbed-set .tabbed-content--wrap {
display: inline-block;
width: 100%;
margin-bottom: 0.25rem; }
.md-typeset .tabbed-set .tabbed-content--wrap > input {
display: none; }
.md-typeset .tabbed-set .tabbed-content--wrap > label {
background-color: rgba(170, 170, 170, 0.2);
padding: 0.2rem 2rem;
margin: 0;
font-weight: 600;
color: #666;
border-radius: 3px 3px 0 0;
border-bottom-color: transparent; }
.md-typeset .tabbed-set .tabbed-content--wrap > .tabbed-content {
display: block;
box-shadow: none; }
.md-typeset .admonition-title + .tabbed-set:last-child,
.md-typeset summary + .tabbed-set:last-child {
margin-top: 0.5rem; }
.md-typeset > details summary::after {
display: none; }
.md-typeset > div, .md-typeset > div.admonition, .md-typeset > details, .md-typeset > div.tabbed-set, .md-typeset > blockquote, .md-typeset > mark, .md-typeset > table, .md-typeset > p {
margin-left: 0.7rem; }
.md-typeset > table.highlighttable {
margin: 0 0 0 0.7rem; }
.md-typeset table.highlighttable,
.md-typeset table.highlighttable > tbody {
width: 100%; }
.md-typeset table.highlighttable {
padding-bottom: 0.5em; }
.md-typeset table.highlighttable .linenos {
padding-bottom: 0; }
.md-typeset > ul {
margin-left: 1.5rem; }
.md-typeset h4,
.md-typeset h5,
.md-typeset h6 {
margin-left: 0.25rem; }
.md-typeset *:not(h3) + h4 {
margin-top: 1.5rem; }
.md-typeset hr {
border-bottom: 1px dotted #999; }
.md-typeset .footnote {
line-height: 1.1; }
.md-typeset table:not([class]) th, .md-typeset table:not([class]) td {
padding: 0.5em 1.25rem 0.4em; }
.md-typeset table:not([class]) td {
border-bottom: 1px solid #eee; }
.md-typeset span.twemoji {
vertical-align: middle;
height: 1.125rem;
display: inline-block; }
.md-typeset span.twemoji > img {
height: 1.125rem;
vertical-align: middle;
display: inline-block; } }
.md-typeset kbd {
vertical-align: middle;
border-color: #b8b8b8;
border-style: outset;
border-width: .05rem .075rem .1rem .075rem;
border-radius: .25rem;
box-shadow: none; }
:root {
--print-md-admonition-icon--note: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%23448aff"><path d="M20.71 7.04c.39-.39.39-1.04 0-1.41l-2.34-2.34c-.37-.39-1.02-.39-1.41 0l-1.84 1.83 3.75 3.75M3 17.25V21h3.75L17.81 9.93l-3.75-3.75L3 17.25z"/></svg>');
--print-md-admonition-icon--abstract: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%2300b0ff"><path d="M4 5h16v2H4V5m0 4h16v2H4V9m0 4h16v2H4v-2m0 4h10v2H4v-2z"/></svg>');
--print-md-admonition-icon--info: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%2300b8d4"><path d="M13 9h-2V7h2m0 10h-2v-6h2m-1-9A10 10 0 002 12a10 10 0 0010 10 10 10 0 0010-10A10 10 0 0012 2z"/></svg>');
--print-md-admonition-icon--tip: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%2300bfa5"><path d="M17.66 11.2c-.23-.3-.51-.56-.77-.82-.67-.6-1.43-1.03-2.07-1.66C13.33 7.26 13 4.85 13.95 3c-.95.23-1.78.75-2.49 1.32-2.59 2.08-3.61 5.75-2.39 8.9.04.1.08.2.08.33 0 .22-.15.42-.35.5-.23.1-.47.04-.66-.12a.58.58 0 01-.14-.17c-1.13-1.43-1.31-3.48-.55-5.12C5.78 10 4.87 12.3 5 14.47c.06.5.12 1 .29 1.5.14.6.41 1.2.71 1.73 1.08 1.73 2.95 2.97 4.96 3.22 2.14.27 4.43-.12 6.07-1.6 1.83-1.66 2.47-4.32 1.53-6.6l-.13-.26c-.21-.46-.77-1.26-.77-1.26m-3.16 6.3c-.28.24-.74.5-1.1.6-1.12.4-2.24-.16-2.9-.82 1.19-.28 1.9-1.16 2.11-2.05.17-.8-.15-1.46-.28-2.23-.12-.74-.1-1.37.17-2.06.19.38.39.76.63 1.06.77 1 1.98 1.44 2.24 2.8.04.14.06.28.06.43.03.82-.33 1.72-.93 2.27z"/></svg>');
--print-md-admonition-icon--success: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%2300c853"><path d="M12 2C6.5 2 2 6.5 2 12s4.5 10 10 10 10-4.5 10-10S17.5 2 12 2m-2 15l-5-5 1.41-1.41L10 14.17l7.59-7.59L19 8l-9 9z"/></svg>');
--print-md-admonition-icon--question: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%2364dd17"><path d="M15.07 11.25l-.9.92C13.45 12.89 13 13.5 13 15h-2v-.5c0-1.11.45-2.11 1.17-2.83l1.24-1.26c.37-.36.59-.86.59-1.41a2 2 0 00-2-2 2 2 0 00-2 2H8a4 4 0 014-4 4 4 0 014 4 3.2 3.2 0 01-.93 2.25M13 19h-2v-2h2M12 2A10 10 0 002 12a10 10 0 0010 10 10 10 0 0010-10c0-5.53-4.5-10-10-10z"/></svg>');
--print-md-admonition-icon--warning: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%23ff9100"><path d="M13 14h-2V9h2m0 9h-2v-2h2M1 21h22L12 2 1 21z"/></svg>');
--print-md-admonition-icon--failure: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%23ff5252"><path d="M12 2c5.53 0 10 4.47 10 10s-4.47 10-10 10S2 17.53 2 12 6.47 2 12 2m3.59 5L12 10.59 8.41 7 7 8.41 10.59 12 7 15.59 8.41 17 12 13.41 15.59 17 17 15.59 13.41 12 17 8.41 15.59 7z"/></svg>');
--print-md-admonition-icon--danger: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%23ff1744"><path d="M11.5 20l4.86-9.73H13V4l-5 9.73h3.5V20M12 2c2.75 0 5.1 1 7.05 2.95C21 6.9 22 9.25 22 12s-1 5.1-2.95 7.05C17.1 21 14.75 22 12 22s-5.1-1-7.05-2.95C3 17.1 2 14.75 2 12s1-5.1 2.95-7.05C6.9 3 9.25 2 12 2z"/></svg>');
--print-md-admonition-icon--bug: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%23f50057"><path d="M14 12h-4v-2h4m0 6h-4v-2h4m6-6h-2.81a5.985 5.985 0 00-1.82-1.96L17 4.41 15.59 3l-2.17 2.17a6.002 6.002 0 00-2.83 0L8.41 3 7 4.41l1.62 1.63C7.88 6.55 7.26 7.22 6.81 8H4v2h2.09c-.05.33-.09.66-.09 1v1H4v2h2v1c0 .34.04.67.09 1H4v2h2.81c1.04 1.79 2.97 3 5.19 3s4.15-1.21 5.19-3H20v-2h-2.09c.05-.33.09-.66.09-1v-1h2v-2h-2v-1c0-.34-.04-.67-.09-1H20V8z"/></svg>');
--print-md-admonition-icon--example: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%23651fff"><path d="M7 13v-2h14v2H7m0 6v-2h14v2H7M7 7V5h14v2H7M3 8V5H2V4h2v4H3m-1 9v-1h3v4H2v-1h2v-.5H3v-1h1V17H2m2.25-7a.75.75 0 01.75.75c0 .2-.08.39-.21.52L3.12 13H5v1H2v-.92L4 11H2v-1h2.25z"/></svg>');
--print-md-admonition-icon--quote: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%239e9e9e"><path d="M14 17h3l2-4V7h-6v6h3M6 17h3l2-4V7H5v6h3l-2 4z"/></svg>'); }
@media print {
.md-typeset .admonition-title {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .admonition-title::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--note); }
.md-typeset .note > .admonition-title, .md-typeset .note > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .note > .admonition-title::before, .md-typeset .note > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--note); }
.md-typeset .abstract > .admonition-title, .md-typeset .tldr > .admonition-title, .md-typeset .summary > .admonition-title, .md-typeset .abstract > summary, .md-typeset .tldr > summary, .md-typeset .summary > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .abstract > .admonition-title::before, .md-typeset .tldr > .admonition-title::before, .md-typeset .summary > .admonition-title::before, .md-typeset .abstract > summary::before, .md-typeset .tldr > summary::before, .md-typeset .summary > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--abstract); }
.md-typeset .info > .admonition-title, .md-typeset .todo > .admonition-title, .md-typeset .info > summary, .md-typeset .todo > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .info > .admonition-title::before, .md-typeset .todo > .admonition-title::before, .md-typeset .info > summary::before, .md-typeset .todo > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--info); }
.md-typeset .tip > .admonition-title, .md-typeset .important > .admonition-title, .md-typeset .hint > .admonition-title, .md-typeset .tip > summary, .md-typeset .important > summary, .md-typeset .hint > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .tip > .admonition-title::before, .md-typeset .important > .admonition-title::before, .md-typeset .hint > .admonition-title::before, .md-typeset .tip > summary::before, .md-typeset .important > summary::before, .md-typeset .hint > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--tip); }
.md-typeset .success > .admonition-title, .md-typeset .done > .admonition-title, .md-typeset .check > .admonition-title, .md-typeset .success > summary, .md-typeset .done > summary, .md-typeset .check > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .success > .admonition-title::before, .md-typeset .done > .admonition-title::before, .md-typeset .check > .admonition-title::before, .md-typeset .success > summary::before, .md-typeset .done > summary::before, .md-typeset .check > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--success); }
.md-typeset .question > .admonition-title, .md-typeset .faq > .admonition-title, .md-typeset .help > .admonition-title, .md-typeset .question > summary, .md-typeset .faq > summary, .md-typeset .help > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .question > .admonition-title::before, .md-typeset .faq > .admonition-title::before, .md-typeset .help > .admonition-title::before, .md-typeset .question > summary::before, .md-typeset .faq > summary::before, .md-typeset .help > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--question); }
.md-typeset .warning > .admonition-title, .md-typeset .attention > .admonition-title, .md-typeset .caution > .admonition-title, .md-typeset .warning > summary, .md-typeset .attention > summary, .md-typeset .caution > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .warning > .admonition-title::before, .md-typeset .attention > .admonition-title::before, .md-typeset .caution > .admonition-title::before, .md-typeset .warning > summary::before, .md-typeset .attention > summary::before, .md-typeset .caution > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--warning); }
.md-typeset .failure > .admonition-title, .md-typeset .missing > .admonition-title, .md-typeset .fail > .admonition-title, .md-typeset .failure > summary, .md-typeset .missing > summary, .md-typeset .fail > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .failure > .admonition-title::before, .md-typeset .missing > .admonition-title::before, .md-typeset .fail > .admonition-title::before, .md-typeset .failure > summary::before, .md-typeset .missing > summary::before, .md-typeset .fail > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--failure); }
.md-typeset .danger > .admonition-title, .md-typeset .error > .admonition-title, .md-typeset .danger > summary, .md-typeset .error > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .danger > .admonition-title::before, .md-typeset .error > .admonition-title::before, .md-typeset .danger > summary::before, .md-typeset .error > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--danger); }
.md-typeset .bug > .admonition-title, .md-typeset .bug > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .bug > .admonition-title::before, .md-typeset .bug > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--bug); }
.md-typeset .example > .admonition-title, .md-typeset .example > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .example > .admonition-title::before, .md-typeset .example > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--example); }
.md-typeset .quote > .admonition-title, .md-typeset .cite > .admonition-title, .md-typeset .quote > summary, .md-typeset .cite > summary {
padding: .6rem .6rem .4rem 2.2rem; }
.md-typeset .quote > .admonition-title::before, .md-typeset .cite > .admonition-title::before, .md-typeset .quote > summary::before, .md-typeset .cite > summary::before {
margin-top: -0.15rem;
background-color: transparent;
background-image: var(--print-md-admonition-icon--quote); } }
:root {
--print-md-footnotes-icon: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%23000000de"><path d="M19 7v4H5.83l3.58-3.59L8 6l-6 6 6 6 1.41-1.42L5.83 13H21V7h-2z"/></svg>'); }
@media print {
.md-typeset .footnote-backref::before {
background-color: transparent;
background-image: var(--print-md-footnotes-icon); } }
:root {
--print-md-details-icon: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="hsla(0, 0%, 0%, 0.87)"><path d="M8.59 16.58L13.17 12 8.59 7.41 10 6l6 6-6 6-1.41-1.42z"/></svg>'); }
@media print {
.md-typeset summary::after {
background-color: transparent;
background-image: var(--print-md-details-icon);
-webkit-mask-image: none;
mask-image: none; } }
:root {
--print-md-tasklist-icon: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%23ddd"><path d="M12 2C6.5 2 2 6.5 2 12s4.5 10 10 10 10-4.5 10-10S17.5 2 12 2m-2 15l-5-5 1.41-1.41L10 14.17l7.59-7.59L19 8l-9 9z"/></svg>');
--print-md-tasklist-icon--checked: url('data:image/svg+xml;charset=utf-8,<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="%2300e676"><path d="M12 2C6.5 2 2 6.5 2 12s4.5 10 10 10 10-4.5 10-10S17.5 2 12 2m-2 15l-5-5 1.41-1.41L10 14.17l7.59-7.59L19 8l-9 9z"/></svg>'); }
@media print {
.md-typeset .task-list-control .task-list-indicator::before {
background-color: transparent;
background-image: var(--print-md-tasklist-icon); }
.md-typeset .task-list-control [type=checkbox]:checked + .task-list-indicator::before {
background-color: transparent;
background-image: var(--print-md-tasklist-icon--checked); } }
/*# sourceMappingURL=material-polyfills.css.map*/
</style></head><body>I am a cover!<article id="doc-toc"><h1>Table of contents</h1><ul><li><a href="#.:lets-build-your-awesome-selfhosted-platform-together">Let's build your awesome selfhosted platform together!</a><ul><li><a href="#.:what-to-expect">What to expect</a></li><li><a href="#.:how-will-this-benefit-me">How will this benefit me?</a></li><li><a href="#.:testimonials">Testimonials</a></li><li><a href="#.:who-made-this">Who made this?</a><ul><li><a href="#.:hi-im-david">👋 Hi, I'm David</a></li><li><a href="#.:what-do-you-want-from-me">What do you want from me?</a></li><li><a href="#.:sponsored-projects">Sponsored Projects</a></li></ul></li></ul></li><li><a href="#IRXWG23FOIQFG53BOJWQ/">Docker Swarm</a><ul><li><a href="#docker-swarm/:highly-available-docker-swarm-design">Highly Available Docker Swarm Design</a><ul><li><a href="#docker-swarm/:design-decisions">Design Decisions</a></li><li><a href="#docker-swarm/:security">Security</a></li><li><a href="#docker-swarm/:high-availability">High availability</a></li></ul></li><li><a href="#IRXWG23FOIQFG53BOJWQ/KBZGK4DBOJQXI2LPNY/">Preparation</a><ul><li><a href="#docker-swarm/design/:highly-available-docker-swarm-design">Highly Available Docker Swarm Design</a></li><li><a href="#docker-swarm/nodes/:nodes">Nodes</a></li></ul></li></ul></li><li><a href="#4KN3IICLOVRGK4TOMV2GK4Y/">⛴ Kubernetes</a><ul><li><a href="#4KN3IICLOVRGK4TOMV2GK4Y/KBZGK4DBOJQXI2LPNY/">Preparation</a><ul><li><a href="#kubernetes/">Introduction</a></li></ul></li></ul></li></ul></article><article class="md-content__inner md-typeset" data-url="/." id=".:"> <h1 id=".:lets-build-your-awesome-selfhosted-platform-together">Let's build your awesome selfhosted platform together!</h1> <p>Welcome, fellow geek <img alt="👋" class="twemoji" src="https://twemoji.maxcdn.com/v/latest/svg/1f44b.svg" title=":wave:"/> If you're impatient, just start here <img alt="👇" class="twemoji" src="https://twemoji.maxcdn.com/v/latest/svg/1f447.svg" title=":point_down:"/></p> <div class="grid cards"> <ul> <li> <p><strong>Dive into <span class="twemoji docker lg middle"><img class="converted-twemoji" src="data:image/svg+xml;charset=utf-8;base64,PHN2ZyBoZWlnaHQ9IjI0IiBzdHlsZT0iZmlsbDogY3VycmVudENvbG9yOyIgdmlld2JveD0iMCAwIDI0IDI0IiB3aWR0aD0iMjQiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+PHBhdGggZD0iTTIxLjgxIDEwLjI1Yy0uMDYtLjA0LS41Ni0uNDMtMS42NC0uNDMtLjI4IDAtLjU2LjAzLS44NC4wOC0uMjEtMS40LTEuMzgtMi4xMS0xLjQzLTIuMTRsLS4yOS0uMTctLjE4LjI3Yy0uMjQuMzYtLjQzLjc3LS41MSAxLjE5LS4yLjgtLjA4IDEuNTYuMzMgMi4yMS0uNDkuMjgtMS4yOS4zNS0xLjQ2LjM1SDIuNjJjLS4zNCAwLS42Mi4yOC0uNjIuNjMgMCAxLjE1LjE4IDIuMy41OCAzLjM4LjQ1IDEuMTkgMS4xMyAyLjA3IDIgMi42MS45OC42IDIuNTkuOTQgNC40Mi45NC43OSAwIDEuNjEtLjA3IDIuNDItLjIyIDEuMTItLjIgMi4yLS41OSAzLjE5LTEuMTZBOC4zIDguMyAwIDAgMCAxNi43OCAxNmMxLjA1LTEuMTcgMS42Ny0yLjUgMi4xMi0zLjY1aC4xOWMxLjE0IDAgMS44NS0uNDYgMi4yNC0uODUuMjYtLjI0LjQ1LS41My41OS0uODdsLjA4LS4yNC0uMTktLjE0bS0xNy45Ni45OWgxLjc2Yy4wOCAwIC4xNi0uMDcuMTYtLjE2VjkuNWMwLS4wOC0uMDctLjE2LS4xNi0uMTZIMy44NWMtLjA5IDAtLjE2LjA3LS4xNi4xNnYxLjU4Yy4wMS4wOS4wNy4xNi4xNi4xNm0yLjQzIDBoMS43NmMuMDggMCAuMTYtLjA3LjE2LS4xNlY5LjVjMC0uMDgtLjA3LS4xNi0uMTYtLjE2SDYuMjhjLS4wOSAwLS4xNi4wNy0uMTYuMTZ2MS41OGMuMDEuMDkuMDcuMTYuMTYuMTZtMi40NyAwaDEuNzVjLjEgMCAuMTctLjA3LjE3LS4xNlY5LjVjMC0uMDgtLjA2LS4xNi0uMTctLjE2SDguNzVjLS4wOCAwLS4xNS4wNy0uMTUuMTZ2MS41OGMwIC4wOS4wNi4xNi4xNS4xNm0yLjQ0IDBoMS43N2MuMDggMCAuMTUtLjA3LjE1LS4xNlY5LjVjMC0uMDgtLjA2LS4xNi0uMTUtLjE2aC0xLjc3Yy0uMDggMC0uMTUuMDctLjE1LjE2djEuNThjMCAuMDkuMDcuMTYuMTUuMTZNNi4yOCA5aDEuNzZjLjA4IDAgLjE2LS4wOS4xNi0uMThWNy4yNWMwLS4wOS0uMDctLjE2LS4xNi0uMTZINi4yOGMtLjA5IDAtLjE2LjA2LS4xNi4xNnYxLjU3Yy4wMS4wOS4wNy4xOC4xNi4xOG0yLjQ3IDBoMS43NWMuMSAwIC4xNy0uMDkuMTctLjE4VjcuMjVjMC0uMDktLjA2LS4xNi0uMTctLjE2SDguNzVjLS4wOCAwLS4xNS4wNi0uMTUuMTZ2MS41N2MwIC4wOS4wNi4xOC4xNS4xOG0yLjQ0IDBoMS43N2MuMDggMCAuMTUtLjA5LjE1LS4xOFY3LjI1YzAtLjA5LS4wNy0uMTYtLjE1LS4xNmgtMS43N2MtLjA4IDAtLjE1LjA2LS4xNS4xNnYxLjU3YzAgLjA5LjA3LjE4LjE1LjE4bTAtMi4yOGgxLjc3Yy4wOCAwIC4xNS0uMDcuMTUtLjE2VjVjMC0uMS0uMDctLjE3LS4xNS0uMTdoLTEuNzdjLS4wOCAwLS4xNS4wNi0uMTUuMTd2MS41NmMwIC4wOC4wNy4xNi4xNS4xNm0yLjQ2IDQuNTJoMS43NmMuMDkgMCAuMTYtLjA3LjE2LS4xNlY5LjVjMC0uMDgtLjA3LS4xNi0uMTYtLjE2aC0xLjc2Yy0uMDggMC0uMTUuMDctLjE1LjE2djEuNThjMCAuMDkuMDcuMTYuMTUuMTYiPjwvcGF0aD48L3N2Zz4="/></span> <a href="/docker-swarm/design/">Docker Swarm</a></strong></p> <hr/> <p>The quickest way to get started, and to get your head around the basics.</p> </li> <li> <p><strong>Kick it with <span class="twemoji kubernetes lg middle"><img class="converted-twemoji" src="data:image/svg+xml;charset=utf-8;base64,PHN2ZyBoZWlnaHQ9IjI0IiBzdHlsZT0iZmlsbDogY3VycmVudENvbG9yOyIgdmlld2JveD0iMCAwIDI0IDI0IiB3aWR0aD0iMjQiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+PHBhdGggZD0iTTEzLjk1IDEzLjVoLS4yM2MtLjE4LjExLS4yNi4zMi0uMTguNWwuODYgMi4xMWMuODMtLjUzIDEuNDYtMS4zMiAxLjc5LTIuMjVsLTIuMjMtLjM2aC0uMDFtLTMuNDUuMjlhLjQxNS40MTUgMCAwIDAtLjM4LS4yOWgtLjA4bC0yLjIyLjM3Yy4zMy45Mi45NiAxLjcgMS43OSAyLjIzbC44NS0yLjA3VjE0Yy4wNC0uMDUuMDQtLjE0LjA0LS4yMW0xLjgzLjgxYS4zNzguMzc4IDAgMCAwLS41MS0uMTVjLS4wNy4wNS0uMTIuMDgtLjE1LjE1aC0uMDFsLTEuMDkgMS45N2MuNzguMjYgMS42Mi4zMSAyLjQzLjEyLjE0LS4wMy4yOS0uMDcuNDMtLjEybC0xLjA5LTEuOTdoLS4wMW0zLjQ1LTQuNTdMMTQuMSAxMS41bC4wMS4wM2EuMzcuMzcgMCAwIDAtLjA0LjUzYy4wNS4wNi4xMS4xLjE4LjEybC4wMS4wMSAyLjE3LjYyYy4wNy0uOTctLjE0LTEuOTUtLjY1LTIuNzhtLTMuMTEuMTZjLjAxLjIxLjE4LjM3LjM5LjM2LjA4IDAgLjE1LS4wMi4yMS0uMDVoLjAxbDEuODMtMS4zMWE0LjQ1IDQuNDUgMCAwIDAtMi41Ny0xLjI0bC4xMyAyLjI0bS0xLjk0LjMxYy4xNy4xMS40LjA4LjUyLS4wOS4wNS0uMDYuMDctLjEzLjA4LS4yMWguMDFsLjEyLTIuMjVjLS4xNS4wMi0uMy4wNS0uNDYuMDgtLjguMTgtMS41NC41OC0yLjEyIDEuMTZsMS44NCAxLjMxaC4wMW0tLjk5IDEuNjljLjItLjA1LjMyLS4yNi4yNi0uNDYgMC0uMDgtLjA1LS4xNC0uMTEtLjE5di0uMDFMOC4yMSAxMGMtLjUyLjg2LS43NCAxLjg0LS42MyAyLjgybDIuMTYtLjYydi0uMDFtMS42NC42Ni42Mi4zLjYyLS4zLjE1LS42Ny0uNDMtLjUzaC0uNjlsLS40My41My4xNi42N20xMC44OSAxLjMyTDIwLjUgNi41Yy0uMDktLjQyLS4zNy0uNzYtLjc0LS45NGwtNy4xNy0zLjQzYy0uMzctLjE3LS44MS0uMTctMS4xOSAwTDQuMjQgNS41NmMtLjM3LjE4LS42NS41Mi0uNzQuOTRsLTEuNzcgNy42N2MtLjA1LjItLjA1LjQgMCAuNTkuMDEuMDYuMDMuMTIuMDUuMTguMDMuMDkuMDguMTkuMTMuMjcuMDMuMDQuMDUuMDguMDkuMTFsNC45NSA2LjE4Yy4wMiAwIC4wNS4wNC4wNS4wNi4xLjA5LjE5LjE2LjI4LjIyLjEyLjA4LjI2LjE0LjQuMTcuMTEuMDUuMjMuMDUuMzIuMDVoOC4xMmMuMDcgMCAuMTQtLjAzLjItLjA1LjA1LS4wMS4xLS4wMy4xNC0uMDQuMDQtLjAyLjA3LS4wMy4xMS0uMDUuMDUtLjAyLjEtLjA1LjE1LS4wOC4xMi0uMDguMjMtLjE4LjMzLS4yOGwuMTUtLjIgNC44LTUuOThjLjEtLjEyLjE3LS4yNS4yMi0uMzguMDItLjA2LjA0LS4xMi4wNS0uMTguMDUtLjE5LjA1LS40IDAtLjU5bS03LjQzIDIuOTljLjAyLjA2LjA0LjEyLjA3LjE3LS4wNC4wOC0uMDYuMTctLjAzLjI2LjEyLjI0LjIzLjQ2LjM4LjY4LjA4LjExLjE2LjIzLjI0LjM0IDAgLjAzLjAzLjA4LjA0LjEyLjEyLjIuMDYuNDYtLjE1LjU5cy0uNDcuMDUtLjU5LS4xNWMtLjAxLS4wMy0uMDItLjA1LS4wMy0uMDgtLjAyLS4wMy0uMDQtLjA5LS4wNi0uMDktLjA1LS4xNS0uMDktLjI4LS4xMi0uNDEtLjA5LS4yNS0uMTctLjQ5LS4zLS43MmEuMzc1LjM3NSAwIDAgMC0uMjEtLjE0bC0uMDgtLjE2Yy0xLjI5LjQ4LTIuNy40OC0zLjk3LS4wMWwtLjEuMThjLS4wNy4wMS0uMTQuMDQtLjE5LjA5LS4xNC4yNC0uMjQuNDktLjMzLjc3LS4wMy4xMy0uMDcuMjYtLjEyLjQtLjAyIDAtLjA0LjA3LS4wNi4xYS40My40MyAwIDAgMS0uODEtLjI5Yy4wMS0uMDMuMDMtLjA1LjA0LS4wOC4wNC0uMDMuMDQtLjA4LjA0LS4xMS4wOS0uMTIuMTYtLjIzLjI0LS4zNS4xNi0uMjEuMjktLjQ1LjM5LS42OWEuNTQuNTQgMCAwIDAtLjAzLS4yNWwuMDctLjE4YTUuNjExIDUuNjExIDAgMCAxLTIuNDctMy4wOWwtLjIuMDNhLjM4OC4zODggMCAwIDAtLjIzLS4wOWMtLjI3LjA1LS41MS4xMy0uNzcuMjItLjExLjA2LS4yNC4xMS0uMzcuMTUtLjAzLjAxLS4wNy4wMi0uMTMuMDNhLjQzOC40MzggMCAwIDEtLjU0LS4yN2MtLjA3LS4yMy4wNC0uNDcuMjgtLjU1LjAyIDAgLjA1LS4wMS4wOC0uMDF2LS4wMWguMDFsLjExLS4wMmMuMTQtLjA0LjI4LS4wNC40MS0uMDQuMjYgMCAuNTItLjA2Ljc3LS4xMi4wOC0uMDUuMTQtLjExLjE5LS4xOWwuMTktLjA1Yy0uMjEtMS4zNi4xLTIuNzMuODYtMy44N2wtLjE0LS4xMmMwLS4wOS0uMDMtLjE4LS4wOC0uMjUtLjItLjE3LS40MS0uMzItLjY0LS40NS0uMTItLjA2LS4yNC0uMTMtLjM2LS4yMS0uMDItLjAyLS4wNi0uMDUtLjA4LS4wN2wtLjAxLS4wMWMtLjItLjE2LS4yNS0uNDItLjExLS42My4wOS0uMS4yMS0uMTUuMzUtLjE1LjExLjAxLjIxLjA1LjMuMTJsLjA5LjA3Yy4xLjA5LjE5LjIuMjguMy4xOC4xOS4zNy4zNy41OC41Mi4wOC4wNC4xNy4wNS4yNi4wM2wuMTUuMTFjLjc1LS44IDEuNzMtMS4zNiAyLjgtMS42LjI1LS4wNi41Mi0uMS43OC0uMTJsLjAxLS4xOGEuNDUuNDUgMCAwIDAgLjE0LS4yM2MuMDEtLjI2LS4wMS0uNTItLjA1LS43Ny0uMDMtLjEzLS4wNS0uMjctLjA2LS40MVY1LjFjLS4wMi0uMjQuMTUtLjQ1LjM5LS40OHMuNDQuMTUuNDcuMzh2LjIyYy0uMDEuMTQtLjAzLjI4LS4wNi40MS0uMDQuMjUtLjA2LjUxLS4wNS43Ny4wMi4xLjA3LjE3LjE0LjIybC4wMS4xOWMxLjM2LjEyIDIuNjIuNzMgMy41NiAxLjcybC4xNi0uMTJjLjA5LjAyLjE4LjAxLjI2LS4wMy4yMS0uMTUuNDEtLjMzLjU4LS41Mi4wOS0uMS4xOC0uMi4yOC0uMy4wMy0uMDIuMDctLjA2LjEtLjA2LjE3LS4xOC40NC0uMTguNTkgMCAuMTkuMTYuMTguNDMgMCAuNiAwIC4wMi0uMDMuMDQtLjA2LjA2YTIuNDk1IDIuNDk1IDAgMCAxLS40NC4yOGMtLjIzLjEzLS40NS4yOC0uNjQuNDUtLjA2LjA3LS4wOS4xNS0uMDguMjRsLS4xNi4xNGE1LjQ0IDUuNDQgMCAwIDEgLjg4IDMuODZsLjE5LjA1Yy4wNC4wOC4xMS4xNC4xOS4xOC4yNS4wNy41MS4xMS43Ny4xNGguNDFjLjAzLjAzLjA4LjA0LjEyLjA1LjI0LjAzLjQuMjUuMzcuNDktLjA1LjIzLS4yNC40LS40OC4zNy0uMDMtLjAxLS4wNy0uMDEtLjA3LS4wMnYtLjAxYy0uMDYgMC0uMS0uMDEtLjE0LS4wMi0uMTMtLjA0LS4yNS0uMDktLjM2LS4xNS0uMjYtLjEtLjUtLjE3LS43Ny0uMjEtLjA5IDAtLjE3IDAtLjIzLjA4LS4wNy0uMDEtLjEzLS4wMi0uMTktLjAzLS40MSAxLjMxLTEuMzEgMi40MS0yLjQ3IDMuMTFaIj48L3BhdGg+PC9zdmc+"/></span> <a href="/kubernetes/">Kubernetes</a></strong></p> <hr/> <p>Been around for a while? Got a high pain threshold? Jump in!</p> </li> <li> <p><strong>Geek out in <span class="twemoji discord lg middle"><img class="converted-twemoji" src="data:image/svg+xml;charset=utf-8;base64,PHN2ZyBoZWlnaHQ9IjUxMiIgc3R5bGU9ImZpbGw6IGN1cnJlbnRDb2xvcjsiIHZpZXdib3g9IjAgMCA2NDAgNTEyIiB3aWR0aD0iNjQwIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwhLS0gRm9udCBBd2Vzb21lIEZyZWUgNi4xLjEgYnkgQGZvbnRhd2Vzb21lIC0gaHR0cHM6Ly9mb250YXdlc29tZS5jb20gTGljZW5zZSAtIGh0dHBzOi8vZm9udGF3ZXNvbWUuY29tL2xpY2Vuc2UvZnJlZSAoSWNvbnM6IENDIEJZIDQuMCwgRm9udHM6IFNJTCBPRkwgMS4xLCBDb2RlOiBNSVQgTGljZW5zZSkgQ29weXJpZ2h0IDIwMjIgRm9udGljb25zLCBJbmMuLS0+PHBhdGggZD0iTTUyNC41MzEgNjkuODM2YTEuNSAxLjUgMCAwIDAtLjc2NC0uN0E0ODUuMDY1IDQ4NS4wNjUgMCAwIDAgNDA0LjA4MSAzMi4wM2ExLjgxNiAxLjgxNiAwIDAgMC0xLjkyMy45MSAzMzcuNDYxIDMzNy40NjEgMCAwIDAtMTQuOSAzMC42IDQ0Ny44NDggNDQ3Ljg0OCAwIDAgMC0xMzQuNDI2IDAgMzA5LjU0MSAzMDkuNTQxIDAgMCAwLTE1LjEzNS0zMC42IDEuODkgMS44OSAwIDAgMC0xLjkyNC0uOTEgNDgzLjY4OSA0ODMuNjg5IDAgMCAwLTExOS42ODggMzcuMTA3IDEuNzEyIDEuNzEyIDAgMCAwLS43ODguNjc2QzM5LjA2OCAxODMuNjUxIDE4LjE4NiAyOTQuNjkgMjguNDMgNDA0LjM1NGEyLjAxNiAyLjAxNiAwIDAgMCAuNzY1IDEuMzc1IDQ4Ny42NjYgNDg3LjY2NiAwIDAgMCAxNDYuODI1IDc0LjE4OSAxLjkgMS45IDAgMCAwIDIuMDYzLS42NzZBMzQ4LjIgMzQ4LjIgMCAwIDAgMjA4LjEyIDQzMC40YTEuODYgMS44NiAwIDAgMC0xLjAxOS0yLjU4OCAzMjEuMTczIDMyMS4xNzMgMCAwIDEtNDUuODY4LTIxLjg1MyAxLjg4NSAxLjg4NSAwIDAgMS0uMTg1LTMuMTI2IDI1MS4wNDcgMjUxLjA0NyAwIDAgMCA5LjEwOS03LjEzNyAxLjgxOSAxLjgxOSAwIDAgMSAxLjktLjI1NmM5Ni4yMjkgNDMuOTE3IDIwMC40MSA0My45MTcgMjk1LjUgMGExLjgxMiAxLjgxMiAwIDAgMSAxLjkyNC4yMzMgMjM0LjUzMyAyMzQuNTMzIDAgMCAwIDkuMTMyIDcuMTYgMS44ODQgMS44ODQgMCAwIDEtLjE2MiAzLjEyNiAzMDEuNDA3IDMwMS40MDcgMCAwIDEtNDUuODkgMjEuODMgMS44NzUgMS44NzUgMCAwIDAtMSAyLjYxMSAzOTEuMDU1IDM5MS4wNTUgMCAwIDAgMzAuMDE0IDQ4LjgxNSAxLjg2NCAxLjg2NCAwIDAgMCAyLjA2My43QTQ4Ni4wNDggNDg2LjA0OCAwIDAgMCA2MTAuNyA0MDUuNzI5YTEuODgyIDEuODgyIDAgMCAwIC43NjUtMS4zNTJjMTIuMjY0LTEyNi43ODMtMjAuNTMyLTIzNi45MTItODYuOTM0LTMzNC41NDFaTTIyMi40OTEgMzM3LjU4Yy0yOC45NzIgMC01Mi44NDQtMjYuNTg3LTUyLjg0NC01OS4yMzlzMjMuNDA5LTU5LjI0MSA1Mi44NDQtNTkuMjQxYzI5LjY2NSAwIDUzLjMwNiAyNi44MiA1Mi44NDMgNTkuMjM5IDAgMzIuNjU0LTIzLjQxIDU5LjI0MS01Mi44NDMgNTkuMjQxWm0xOTUuMzggMGMtMjguOTcxIDAtNTIuODQzLTI2LjU4Ny01Mi44NDMtNTkuMjM5czIzLjQwOS01OS4yNDEgNTIuODQzLTU5LjI0MWMyOS42NjcgMCA1My4zMDcgMjYuODIgNTIuODQ0IDU5LjIzOSAwIDMyLjY1NC0yMy4xNzcgNTkuMjQxLTUyLjg0NCA1OS4yNDFaIj48L3BhdGg+PC9zdmc+"/></span> <a href="http://chat.funkypenguin.co.nz">Discord</a></strong></p> <hr/> <p>Join the fun, chat with fellow geeks in realtime!</p> </li> <li> <p><strong>Fast-track with 🚀 <a href="/premix">Premix</a>!</strong></p> <hr/> <p>Life's too short? Fast-track your stack with Premix!</p> </li> </ul> </div> <h2 id=".:what-to-expect">What to expect</h2> <p>The "<em>Geek Cookbook</em>" is a collection of how-to guides for establishing your own container-based awesome selfhosted platform, using either <a href="/docker-swarm/design/">Docker Swarm</a> or <a href="/kubernetes/">Kubernetes</a>.</p> <p>Running such a platform enables you to run selfhosted services such as the <a href="/recipes/autopirate/">AutoPirate</a> (<em>Radarr, Sonarr, NZBGet and friends</em>) stack, <a href="https://www.plex.tv/">Plex</a>, <a href="https://nextcloud.com/">NextCloud</a> etc, and includes elements such as:</p> <ul> <li><a href="/docker-swarm/traefik/">Automatic SSL-secured access</a> to all services (<em>with LetsEncrypt</em>)</li> <li><a href="/docker-swarm/traefik-forward-auth/">SSO / authentication layer</a> to protect unsecured / vulnerable services</li> <li><a href="/recipes/elkarbackup/">Automated backup</a> of configuration and data</li> <li><a href="/recipes/swarmprom/">Monitoring and metrics</a> collection, graphing and alerting</li> </ul> <p>Recent updates and additions are posted on the <a href="/CHANGELOG/">CHANGELOG</a>, and there's a friendly community of like-minded geeks in the <a href="http://chat.funkypenguin.co.nz">Discord server</a>.</p> <h2 id=".:how-will-this-benefit-me">How will this benefit me?</h2> <p>You already have a familiarity with concepts such as virtual machines, <a href="https://www.docker.com/">Docker</a> containers, <a href="https://letsencrypt.org/">LetsEncrypt SSL certificates</a>, databases, and command-line interfaces.</p> <p>You've probably played with self-hosting some mainstream apps yourself, like <a href="https://www.plex.tv/">Plex</a>, <a href="https://nextcloud.com/">NextCloud</a>, <a href="https://wordpress.org/">Wordpress</a> or <a href="https://ghost.io/">Ghost</a>.</p> <p>So if you're familiar enough with the concepts above, and you've done self-hosting before, why would you read any further?</p> <ol> <li>You want to upskill. You want to work with container orchestration, Prometheus and Grafana, Kubernetes</li> <li>You want to play. You want a safe sandbox to test new tools, keeping the ones you want and tossing the ones you don't.</li> <li>You want reliability. Once you go from <strong>playing</strong> with a tool to actually <strong>using</strong> it, you want it to be available when you need it. Having to "<em>quickly ssh into the basement server and restart plex</em>" doesn't cut it when you finally convince your wife to sit down with you to watch sci-fi <img alt="🤖" class="twemoji" src="https://twemoji.maxcdn.com/v/latest/svg/1f916.svg" title=":robot:"/></li> </ol> <h2 id=".:testimonials">Testimonials</h2> <div class="admonition quote"> <p class="admonition-title">...how useful the recipes are for people just getting started with containers...</p> <p>"One of the surprising realizations from following Funky Penguins cookbooks for so long is how useful the recipes are for people just getting started with containers and how it gives them real, interesting usecases to attach to their learning" - <a href="https://twitter.com/DanielSHouston/status/1213419203379773442">DevOps Daniel (@DanielSHouston)</a></p> </div> <div class="admonition quote"> <p class="admonition-title">He unblocked me on all the technical hurdles to launching my SaaS in GKE!</p> <p>By the time I had enlisted Funky Penguin's help, I'd architected myself into a bit of a nightmare with Kubernetes. I knew what I wanted to achieve, but I'd made a mess of it. Funky Penguin (David) was able to jump right in and offer a vital second-think on everything I'd done, pointing out where things could be simplified and streamlined, and better alternatives. </p> <p>He unblocked me on all the technical hurdles to launching my SaaS in GKE! </p> <p>With him delivering the container/Kubernetes architecture and helm CI/CD workflow, I was freed up to focus on coding and design, which fast-tracked me to launching on time. And now I have a simple deployment process that is easy for me to execute and maintain as a solo founder. </p> <p>I have no hesitation in recommending him for your project, and I'll certainly be calling on him again in the future.</p> <p>-- John McDowall, Founder, <a href="https://kiso.io">kiso.io</a> </p> </div> <h2 id=".:who-made-this">Who made this?</h2> <h3 id=".:hi-im-david">👋 Hi, I'm David</h3> <p>I’ve spent 20+ years working with technology. I’m a solution architect, with a broad range of experience and skills. I'm a full-time <a href="https://www.credly.com/badges/a0c4a196-55ab-4472-b46b-b610b44dc00f">AWS Certified Solution Architect (Professional)</a>, a <a href="https://www.credly.com/badges/cd307d51-544b-4bc6-97b0-9015e40df40d">CNCF-Certified Kubernetes Administrator</a>, <a href="https://www.credly.com/badges/9ed9280a-fb92-46ca-b307-8f74a2cccf1d">Application Developer</a> and <a href="https://www.credly.com/badges/93fa53da-1f38-47a9-b6ee-dce6a8fad9fc">Security Specialist</a>.</p> <h3 id=".:what-do-you-want-from-me">What do you want from me?</h3> <p>I want your <a href="https://github.com/sponsors/funkypenguin">support</a>, either in the <a href="https://github.com/sponsors/funkypenguin">financial</a> sense, or as a member of our <a href="http://chat.funkypenguin.co.nz">friendly geek community</a> (<em>or both!</em>)</p> <h4 id=".:get-in-touch">Get in touch 💬</h4> <ul> <li>Come and say hi to me and the friendly geeks in the <a href="http://chat.funkypenguin.co.nz">Discord</a> chat or the <a href="https://forum.funkypenguin.co.nz/">Discourse</a> forums - say hi, ask a question, or suggest a new recipe!</li> <li>Tweet me up, I'm <a href="https://twitter.com/funkypenguin">@funkypenguin</a>! 🐦</li> <li><a href="https://www.funkypenguin.co.nz/contact/">Contact me</a> by a variety of channels</li> </ul> <h4 id=".:sponsor-me"><a href="https://github.com/sponsors/funkypenguin">Sponsor</a> me ❤️</h4> <p>The best way to support this work is to become a <a href="https://github.com/sponsors/funkypenguin">GitHub Sponsor</a> / <a href="https://www.patreon.com/bePatron?u=6982506">Patreon patron</a>. You get:</p> <ul> <li>warm fuzzies,</li> <li>access to the pre-mix repo,</li> <li>an anonymous plug you can pull at any time,</li> <li>and a bunch more loot based on tier</li> </ul> <p>.. and I get some pocket money every month to buy wine, cheese, and cryptocurrency! 🍷 💰</p> <p>Impulsively <strong><a href="https://github.com/sponsors/funkypenguin">click here (NOW quick do it!)</a></strong> to <a href="https://github.com/sponsors/funkypenguin">sponsor me</a> via GitHub, or <a href="https://www.patreon.com/bePatron?u=6982506">patronize me via Patreon</a>!</p> <h4 id=".:work-with-me">Work with me 🤝</h4> <p>Need some Cloud / Microservices / DevOps / Infrastructure design work done? This stuff is my bread and butter! <img alt="🍞" class="twemoji" src="https://twemoji.maxcdn.com/v/latest/svg/1f35e.svg" title=":bread:"/> <img alt="🍴" class="twemoji" src="https://twemoji.maxcdn.com/v/latest/svg/1f374.svg" title=":fork_and_knife:"/> <a href="https://www.funkypenguin.co.nz/contact">Get in touch</a>, and let's talk!</p> <h4 id=".:buy-me-a-coffee">Buy me a coffee ☕️</h4> <p>A sponsorship is too much commitment, and a book is TL;DR? Hit me up with a <a href="https://www.buymeacoffee.com/funkypenguin">one-time caffine shot</a>!</p> <h3 id=".:sponsored-projects">Sponsored Projects</h3> <p>I'm supported and motivated by <a href="https://github.com/sponsors/funkypenguin">GitHub Sponsors</a> and <a href="https://www.patreon.com/funkypenguin">Patreon patrons</a> who have generously sponsored me.</p> <p>I regularly donate to / sponsor the following projects. <strong>Join me</strong> in supporting these geeks, and encouraging them to continue building the ingredients for your favourite recipes!</p> <table> <thead> <tr> <th>Project</th> <th>Donate via..</th> </tr> </thead> <tbody> <tr> <td><a href="/recipes/komga/">Komga</a></td> <td><a href="https://github.com/sponsors/gotson">GitHub Sponsors</a></td> </tr> <tr> <td><a href="https://squidfunk.github.io/mkdocs-material/">Material for MKDocs</a></td> <td><a href="https://github.com/sponsors/squidfunk">GitHub Sponsors</a></td> </tr> <tr> <td><a href="https://calibre-ebook.com/">Calibre</a></td> <td><a href="https://calibre-ebook.com/donate">Credit Card</a> / <a href="https://www.patreon.com/kovidgoyal">Patreon</a> / <a href="https://liberapay.com/kovidgoyal/donate">LibrePay</a></td> </tr> <tr> <td><a href="https://www.linuxserver.io">LinuxServer.io</a></td> <td><a href="https://www.linuxserver.io/donate">PayPal</a></td> </tr> <tr> <td><a href="https://widgetbot.io/">WidgetBot's Discord Widget</a></td> <td><a href="https://www.patreon.com/widgetbot/overview">Patreon</a></td> </tr> <tr> <td><a href="https://carl.gg/">Carl-bot</a></td> <td><a href="https://www.patreon.com/carlbot">Patreon</a></td> </tr> </tbody> </table> <hr/> <div class="md-source-file"> <small> Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 9, 2022</span> </small> </div> </article><article class="md-content__inner md-typeset" data-url="/IRXWG23FOIQFG53BOJWQ/" id="IRXWG23FOIQFG53BOJWQ/:"><h1 id="IRXWG23FOIQFG53BOJWQ/">Docker Swarm</h1><section class="md-typeset" data-url="/docker-swarm/" id="docker-swarm/:"> <h2 id="docker-swarm/:highly-available-docker-swarm-design">Highly Available Docker Swarm Design</h2> <p>In the design described below, our "private cloud" platform is:</p> <ul> <li><strong>Highly-available</strong> (<em>can tolerate the failure of a single component</em>)</li> <li><strong>Scalable</strong> (<em>can add resource or capacity as required</em>)</li> <li><strong>Portable</strong> (<em>run it on your garage server today, run it in AWS tomorrow</em>)</li> <li><strong>Secure</strong> (<em>access protected with <a href="/docker-swarm/traefik/">LetsEncrypt certificates</a> and optional <a href="/docker-swarm/traefik-forward-auth/">OIDC with 2FA</a></em>)</li> <li><strong>Automated</strong> (<em>requires minimal care and feeding</em>)</li> </ul> <h3 id="docker-swarm/:design-decisions">Design Decisions</h3> <h4 id="docker-swarm/:where-possible-services-will-be-highly-available">Where possible, services will be highly available.**</h4> <p>This means that:</p> <ul> <li>At least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure.</li> <li><a href="/docker-swarm/shared-storage-ceph/">Ceph</a> is employed for share storage, because it too can be made tolerant of a single failure.</li> </ul> <div class="admonition note"> <p class="admonition-title">Note</p> <p>An exception to the 3-nodes decision is running a single-node configuration. If you only <strong>have</strong> one node, then obviously your swarm is only as resilient as that node. It's still a perfectly valid swarm configuration, ideal for starting your self-hosting journey. In fact, under the single-node configuration, you don't need ceph either, and you can simply use the local volume on your host for storage. You'll be able to migrate to ceph/more nodes if/when you expand.</p> </div> <p><strong>Where multiple solutions to a requirement exist, preference will be given to the most portable solution.</strong></p> <p>This means that:</p> <ul> <li>Services are defined using docker-compose v3 YAML syntax</li> <li>Services are portable, meaning a particular stack could be shut down and moved to a new provider with minimal effort.</li> </ul> <h3 id="docker-swarm/:security">Security</h3> <p>Under this design, the only inbound connections we're permitting to our docker swarm in a <strong>minimal</strong> configuration (<em>you may add custom services later, like UniFi Controller</em>) are:</p> <h4 id="docker-swarm/:network-flows">Network Flows</h4> <ul> <li><strong>HTTP (TCP 80)</strong> : Redirects to https</li> <li><strong>HTTPS (TCP 443)</strong> : Serves individual docker containers via SSL-encrypted reverse proxy</li> </ul> <h4 id="docker-swarm/:authentication">Authentication</h4> <ul> <li>Where the hosted application provides a trusted level of authentication (<em>i.e., <a href="/recipes/nextcloud/">NextCloud</a></em>), or where the application requires public exposure (<em>i.e. <a href="/recipes/privatebin/">Privatebin</a></em>), no additional layer of authentication will be required.</li> <li>Where the hosted application provides inadequate (<em>i.e. <a href="/recipes/autopirate/nzbget/">NZBGet</a></em>) or no authentication (<em>i.e. <a href="/recipes/gollum/">Gollum</a></em>), a further authentication against an OAuth provider will be required.</li> </ul> <h3 id="docker-swarm/:high-availability">High availability</h3> <h4 id="docker-swarm/:normal-function">Normal function</h4> <p>Assuming a 3-node configuration, under normal circumstances the following is illustrated:</p> <ul> <li>All 3 nodes provide shared storage via Ceph, which is provided by a docker container on each node.</li> <li>All 3 nodes participate in the Docker Swarm as managers.</li> <li>The various containers belonging to the application "stacks" deployed within Docker Swarm are automatically distributed amongst the swarm nodes.</li> <li>Persistent storage for the containers is provide via cephfs mount.</li> <li>The <strong>traefik</strong> service (<em>in swarm mode</em>) receives incoming requests (<em>on HTTP and HTTPS</em>), and forwards them to individual containers. Traefik knows the containers names because it's able to read the docker socket.</li> <li>All 3 nodes run keepalived, at varying priorities. Since traefik is running as a swarm service and listening on TCP 80/443, requests made to the keepalived VIP and arriving at <strong>any</strong> of the swarm nodes will be forwarded to the traefik container (<em>no matter which node it's on</em>), and then onto the target backend.</li> </ul> <p><img alt="HA function" src="file:///Users/davidy/Documents/Personal/Projects/geek-cookbook/site/images/docker-swarm-ha-function.png" style=""/></p> <h4 id="docker-swarm/:node-failure">Node failure</h4> <p>In the case of a failure (or scheduled maintenance) of one of the nodes, the following is illustrated:</p> <ul> <li>The failed node no longer participates in Ceph, but the remaining nodes provide enough fault-tolerance for the cluster to operate.</li> <li>The remaining two nodes in Docker Swarm achieve a quorum and agree that the failed node is to be removed.</li> <li>The (<em>possibly new</em>) leader manager node reschedules the containers known to be running on the failed node, onto other nodes.</li> <li>The <strong>traefik</strong> service is either restarted or unaffected, and as the backend containers stop/start and change IP, traefik is aware and updates accordingly.</li> <li>The keepalived VIP continues to function on the remaining nodes, and docker swarm continues to forward any traffic received on TCP 80/443 to the appropriate node.</li> </ul> <p><img alt="HA function" src="file:///Users/davidy/Documents/Personal/Projects/geek-cookbook/site/images/docker-swarm-node-failure.png" style=""/></p> <h4 id="docker-swarm/:node-restore">Node restore</h4> <p>When the failed (<em>or upgraded</em>) host is restored to service, the following is illustrated:</p> <ul> <li>Ceph regains full redundancy</li> <li>Docker Swarm managers become aware of the recovered node, and will use it for scheduling <strong>new</strong> containers</li> <li>Existing containers which were migrated off the node are not migrated backend</li> <li>Keepalived VIP regains full redundancy</li> </ul> <p><img alt="HA function" src="file:///Users/davidy/Documents/Personal/Projects/geek-cookbook/site/images/docker-swarm-node-restore.png" style=""/></p> <h4 id="docker-swarm/:total-cluster-failure">Total cluster failure</h4> <p>A day after writing this, my environment suffered a fault whereby all 3 VMs were unexpectedly and simultaneously powered off.</p> <p>Upon restore, docker failed to start on one of the VMs due to local disk space issue<sup id="docker-swarm/:fnref:1"><a class="footnote-ref" href="#docker-swarm/:fn:1">1</a></sup>. However, the other two VMs started, established the swarm, mounted their shared storage, and started up all the containers (services) which were managed by the swarm.</p> <p>In summary, although I suffered an <strong>unplanned power outage to all of my infrastructure</strong>, followed by a <strong>failure of a third of my hosts</strong>... <mark>all my platforms are 100% available<sup id="docker-swarm/:fnref2:1"><a class="footnote-ref" href="#docker-swarm/:fn:1">1</a></sup> with <strong>absolutely no manual intervention</strong></mark>.</p> <div class="footnote"> <hr/> <ol> <li id="docker-swarm/:fn:1"> <p>Since there's no impact to availability, I can fix (or just reinstall) the failed node whenever convenient. <a class="footnote-backref" href="#docker-swarm/:fnref:1" title="Jump back to footnote 1 in the text">↩</a><a class="footnote-backref" href="#docker-swarm/:fnref2:1" title="Jump back to footnote 1 in the text">↩</a></p> </li> </ol> </div> <hr/> <div class="md-source-file"> <small> Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 9, 2022</span> </small> </div> </section><section class="md-typeset" data-url="/IRXWG23FOIQFG53BOJWQ/KBZGK4DBOJQXI2LPNY/" id="IRXWG23FOIQFG53BOJWQ/KBZGK4DBOJQXI2LPNY/:"><h2 id="IRXWG23FOIQFG53BOJWQ/KBZGK4DBOJQXI2LPNY/">Preparation</h2><section class="md-typeset" data-url="/docker-swarm/design/" id="docker-swarm/design/:"> <h3 id="docker-swarm/design/:highly-available-docker-swarm-design">Highly Available Docker Swarm Design</h3> <p>In the design described below, our "private cloud" platform is:</p> <ul> <li><strong>Highly-available</strong> (<em>can tolerate the failure of a single component</em>)</li> <li><strong>Scalable</strong> (<em>can add resource or capacity as required</em>)</li> <li><strong>Portable</strong> (<em>run it on your garage server today, run it in AWS tomorrow</em>)</li> <li><strong>Secure</strong> (<em>access protected with <a href="/docker-swarm/traefik/">LetsEncrypt certificates</a> and optional <a href="/docker-swarm/traefik-forward-auth/">OIDC with 2FA</a></em>)</li> <li><strong>Automated</strong> (<em>requires minimal care and feeding</em>)</li> </ul> <h4 id="docker-swarm/design/:design-decisions">Design Decisions</h4> <h5 id="docker-swarm/design/:where-possible-services-will-be-highly-available">Where possible, services will be highly available.**</h5> <p>This means that:</p> <ul> <li>At least 3 docker swarm manager nodes are required, to provide fault-tolerance of a single failure.</li> <li><a href="/docker-swarm/shared-storage-ceph/">Ceph</a> is employed for share storage, because it too can be made tolerant of a single failure.</li> </ul> <div class="admonition note"> <p class="admonition-title">Note</p> <p>An exception to the 3-nodes decision is running a single-node configuration. If you only <strong>have</strong> one node, then obviously your swarm is only as resilient as that node. It's still a perfectly valid swarm configuration, ideal for starting your self-hosting journey. In fact, under the single-node configuration, you don't need ceph either, and you can simply use the local volume on your host for storage. You'll be able to migrate to ceph/more nodes if/when you expand.</p> </div> <p><strong>Where multiple solutions to a requirement exist, preference will be given to the most portable solution.</strong></p> <p>This means that:</p> <ul> <li>Services are defined using docker-compose v3 YAML syntax</li> <li>Services are portable, meaning a particular stack could be shut down and moved to a new provider with minimal effort.</li> </ul> <h4 id="docker-swarm/design/:security">Security</h4> <p>Under this design, the only inbound connections we're permitting to our docker swarm in a <strong>minimal</strong> configuration (<em>you may add custom services later, like UniFi Controller</em>) are:</p> <h5 id="docker-swarm/design/:network-flows">Network Flows</h5> <ul> <li><strong>HTTP (TCP 80)</strong> : Redirects to https</li> <li><strong>HTTPS (TCP 443)</strong> : Serves individual docker containers via SSL-encrypted reverse proxy</li> </ul> <h5 id="docker-swarm/design/:authentication">Authentication</h5> <ul> <li>Where the hosted application provides a trusted level of authentication (<em>i.e., <a href="/recipes/nextcloud/">NextCloud</a></em>), or where the application requires public exposure (<em>i.e. <a href="/recipes/privatebin/">Privatebin</a></em>), no additional layer of authentication will be required.</li> <li>Where the hosted application provides inadequate (<em>i.e. <a href="/recipes/autopirate/nzbget/">NZBGet</a></em>) or no authentication (<em>i.e. <a href="/recipes/gollum/">Gollum</a></em>), a further authentication against an OAuth provider will be required.</li> </ul> <h4 id="docker-swarm/design/:high-availability">High availability</h4> <h5 id="docker-swarm/design/:normal-function">Normal function</h5> <p>Assuming a 3-node configuration, under normal circumstances the following is illustrated:</p> <ul> <li>All 3 nodes provide shared storage via Ceph, which is provided by a docker container on each node.</li> <li>All 3 nodes participate in the Docker Swarm as managers.</li> <li>The various containers belonging to the application "stacks" deployed within Docker Swarm are automatically distributed amongst the swarm nodes.</li> <li>Persistent storage for the containers is provide via cephfs mount.</li> <li>The <strong>traefik</strong> service (<em>in swarm mode</em>) receives incoming requests (<em>on HTTP and HTTPS</em>), and forwards them to individual containers. Traefik knows the containers names because it's able to read the docker socket.</li> <li>All 3 nodes run keepalived, at varying priorities. Since traefik is running as a swarm service and listening on TCP 80/443, requests made to the keepalived VIP and arriving at <strong>any</strong> of the swarm nodes will be forwarded to the traefik container (<em>no matter which node it's on</em>), and then onto the target backend.</li> </ul> <p><img alt="HA function" src="file:///Users/davidy/Documents/Personal/Projects/geek-cookbook/site/images/docker-swarm-ha-function.png" style=""/></p> <h5 id="docker-swarm/design/:node-failure">Node failure</h5> <p>In the case of a failure (or scheduled maintenance) of one of the nodes, the following is illustrated:</p> <ul> <li>The failed node no longer participates in Ceph, but the remaining nodes provide enough fault-tolerance for the cluster to operate.</li> <li>The remaining two nodes in Docker Swarm achieve a quorum and agree that the failed node is to be removed.</li> <li>The (<em>possibly new</em>) leader manager node reschedules the containers known to be running on the failed node, onto other nodes.</li> <li>The <strong>traefik</strong> service is either restarted or unaffected, and as the backend containers stop/start and change IP, traefik is aware and updates accordingly.</li> <li>The keepalived VIP continues to function on the remaining nodes, and docker swarm continues to forward any traffic received on TCP 80/443 to the appropriate node.</li> </ul> <p><img alt="HA function" src="file:///Users/davidy/Documents/Personal/Projects/geek-cookbook/site/images/docker-swarm-node-failure.png" style=""/></p> <h5 id="docker-swarm/design/:node-restore">Node restore</h5> <p>When the failed (<em>or upgraded</em>) host is restored to service, the following is illustrated:</p> <ul> <li>Ceph regains full redundancy</li> <li>Docker Swarm managers become aware of the recovered node, and will use it for scheduling <strong>new</strong> containers</li> <li>Existing containers which were migrated off the node are not migrated backend</li> <li>Keepalived VIP regains full redundancy</li> </ul> <p><img alt="HA function" src="file:///Users/davidy/Documents/Personal/Projects/geek-cookbook/site/images/docker-swarm-node-restore.png" style=""/></p> <h5 id="docker-swarm/design/:total-cluster-failure">Total cluster failure</h5> <p>A day after writing this, my environment suffered a fault whereby all 3 VMs were unexpectedly and simultaneously powered off.</p> <p>Upon restore, docker failed to start on one of the VMs due to local disk space issue<sup id="docker-swarm/design/:fnref:1"><a class="footnote-ref" href="#docker-swarm/design/:fn:1">1</a></sup>. However, the other two VMs started, established the swarm, mounted their shared storage, and started up all the containers (services) which were managed by the swarm.</p> <p>In summary, although I suffered an <strong>unplanned power outage to all of my infrastructure</strong>, followed by a <strong>failure of a third of my hosts</strong>... <mark>all my platforms are 100% available<sup id="docker-swarm/design/:fnref2:1"><a class="footnote-ref" href="#docker-swarm/design/:fn:1">1</a></sup> with <strong>absolutely no manual intervention</strong></mark>.</p> <div class="footnote"> <hr/> <ol> <li id="docker-swarm/design/:fn:1"> <p>Since there's no impact to availability, I can fix (or just reinstall) the failed node whenever convenient. <a class="footnote-backref" href="#docker-swarm/design/:fnref:1" title="Jump back to footnote 1 in the text">↩</a><a class="footnote-backref" href="#docker-swarm/design/:fnref2:1" title="Jump back to footnote 1 in the text">↩</a></p> </li> </ol> </div> <hr/> <div class="md-source-file"> <small> Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 9, 2022</span> </small> </div> </section><section class="md-typeset" data-url="/docker-swarm/nodes/" id="docker-swarm/nodes/:"> <h3 id="docker-swarm/nodes/:nodes">Nodes</h3> <p>Let's start building our cluster. You can use either bare-metal machines or virtual machines - the configuration would be the same. To avoid confusion, I'll be referring to these as "nodes" from now on.</p> <div class="admonition note"> <p class="admonition-title">Note</p> <p>In 2017, I <strong>initially</strong> chose the "<a href="https://www.projectatomic.io/">Atomic</a>" CentOS/Fedora image for the swarm hosts, but later found its outdated version of Docker to be problematic with advanced features like GPU transcoding (in <a href="/recipes/plex/">Plex</a>), <a href="/recipes/swarmprom/">Swarmprom</a>, etc. In the end, I went mainstream and simply preferred a modern Ubuntu installation.</p> </div> <h4 id="docker-swarm/nodes/:ingredients">Ingredients</h4> <div class="admonition summary"> <p class="admonition-title">Ingredients</p> <p>New in this recipe:</p> <ul class="task-list"> <li class="task-list-item"><label class="task-list-control"><input disabled="" type="checkbox"/><span class="task-list-indicator"></span></label> 3 x nodes (<em>bare-metal or VMs</em>), each with:<ul> <li>A mainstream Linux OS (<em>tested on either <a href="https://www.centos.org">CentOS</a> 7+ or <a href="http://releases.ubuntu.com">Ubuntu</a> 16.04+</em>)</li> <li>At least 2GB RAM</li> <li>At least 20GB disk space (<em>but it'll be tight</em>)</li> </ul> </li> <li class="task-list-item"><label class="task-list-control"><input disabled="" type="checkbox"/><span class="task-list-indicator"></span></label> Connectivity to each other within the same subnet, and on a low-latency link (<em>i.e., no WAN links</em>)</li> </ul> </div> <h4 id="docker-swarm/nodes/:preparation">Preparation</h4> <h5 id="docker-swarm/nodes/:permit-connectivity">Permit connectivity</h5> <p>Most modern Linux distributions include firewall rules which only only permit minimal required incoming connections (like SSH). We'll want to allow all traffic between our nodes. The steps to achieve this in CentOS/Ubuntu are a little different...</p> <h6 id="docker-swarm/nodes/:centos">CentOS</h6> <p>Add something like this to <code>/etc/sysconfig/iptables</code>:</p> <div class="highlight"><pre><span></span><code><span class="c1"># Allow all inter-node communication</span>
-A INPUT -s <span class="m">192</span>.168.31.0/24 -j ACCEPT
</code></pre></div> <p>And restart iptables with <code>systemctl restart iptables</code></p> <h6 id="docker-swarm/nodes/:ubuntu">Ubuntu</h6> <p>Install the (<em>non-default</em>) persistent iptables tools, by running <code>apt-get install iptables-persistent</code>, establishing some default rules (<em>dkpg will prompt you to save current ruleset</em>), and then add something like this to <code>/etc/iptables/rules.v4</code>:</p> <div class="highlight"><pre><span></span><code><span class="c1"># Allow all inter-node communication</span>
-A INPUT -s <span class="m">192</span>.168.31.0/24 -j ACCEPT
</code></pre></div> <p>And refresh your running iptables rules with <code>iptables-restore &lt; /etc/iptables/rules.v4</code></p> <h5 id="docker-swarm/nodes/:enable-hostname-resolution">Enable hostname resolution</h5> <p>Depending on your hosting environment, you may have DNS automatically setup for your VMs. If not, it's useful to set up static entries in /etc/hosts for the nodes. For example, I setup the following:</p> <ul> <li>192.168.31.11 ds1 ds1.funkypenguin.co.nz</li> <li>192.168.31.12 ds2 ds2.funkypenguin.co.nz</li> <li>192.168.31.13 ds3 ds3.funkypenguin.co.nz</li> </ul> <h5 id="docker-swarm/nodes/:set-timezone">Set timezone</h5> <p>Set your local timezone, by running:</p> <div class="highlight"><pre><span></span><code>ln -sf /usr/share/zoneinfo/&lt;your timezone&gt; /etc/localtime
</code></pre></div> <h4 id="docker-swarm/nodes/:serving">Serving</h4> <p>After completing the above, you should have:</p> <div class="admonition summary"> <p class="admonition-title">Summary</p> <p>Deployed in this recipe:</p> <ul class="task-list"> <li class="task-list-item"><label class="task-list-control"><input checked="" disabled="" type="checkbox"/><span class="task-list-indicator"></span></label> 3 x nodes (<em>bare-metal or VMs</em>), each with:<ul> <li>A mainstream Linux OS (<em>tested on either <a href="https://www.centos.org">CentOS</a> 7+ or <a href="http://releases.ubuntu.com">Ubuntu</a> 16.04+</em>)</li> <li>At least 2GB RAM</li> <li>At least 20GB disk space (<em>but it'll be tight</em>)</li> </ul> </li> <li class="task-list-item"><label class="task-list-control"><input checked="" disabled="" type="checkbox"/><span class="task-list-indicator"></span></label> Connectivity to each other within the same subnet, and on a low-latency link (<em>i.e., no WAN links</em>)</li> </ul> </div> <hr/> <div class="md-source-file"> <small> Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 9, 2022</span> </small> </div> </section></section></article><article class="md-content__inner md-typeset" data-url="/4KN3IICLOVRGK4TOMV2GK4Y/" id="4KN3IICLOVRGK4TOMV2GK4Y/:"><h1 id="4KN3IICLOVRGK4TOMV2GK4Y/">⛴ Kubernetes</h1><section class="md-typeset" data-url="/4KN3IICLOVRGK4TOMV2GK4Y/KBZGK4DBOJQXI2LPNY/" id="4KN3IICLOVRGK4TOMV2GK4Y/KBZGK4DBOJQXI2LPNY/:"><h2 id="4KN3IICLOVRGK4TOMV2GK4Y/KBZGK4DBOJQXI2LPNY/">Preparation</h2><section class="md-typeset" data-url="/kubernetes/" id="kubernetes/:"> <h3 id="kubernetes/">Introduction</h3> <p>My first introduction to Kubernetes was a children's story:</p> <!-- markdownlint-disable MD033 --> <iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/R9-SOzep73w" width="560"></iframe> <h4 id="kubernetes/:why-kubernetes">Why Kubernetes?</h4> <p>Why would you want to Kubernetes for your self-hosted recipes, over simple <a href="/docker-swarm/">Docker Swarm</a>? Here's my personal take..</p> <h5 id="kubernetes/:docker-swarm-is-dead">Docker Swarm is dead</h5> <p>Sorry to say, but from where I sit, there's no innovation or development happening in docker swarm.</p> <p>Yes, I know, after Docker Inc <a href="https://www.mirantis.com/blog/mirantis-acquires-docker-enterprise-platform-business/">sold its platform business to Mirantis in Nov 2019</a>, in Feb 2020 Mirantis <a href="https://www.mirantis.com/blog/mirantis-will-continue-to-support-and-develop-docker-swarm/">back-tracked</a> on their original plan to sunset swarm after 2 years, and stated that they'd continue to invest in swarm. But seriously, look around. Nobody is interested in swarm right now...</p> <p>... Not even Mirantis! As of Nov 2021, the Mirantis blog tag "<a href="https://www.mirantis.com/tag/kubernetes/">kubernetes</a>" had 8 posts within the past month. The tag "<a href="https://www.mirantis.com/tag/docker/">docker</a>" has 8 posts in the past <strong>2 years</strong>, the 8<sup>th</sup> being the original announcement of the Docker aquisition. The tag "<a href="https://www.mirantis.com/tag/docker-swarm/">docker swarm</a>" has only 2 posts, <strong>ever</strong>.</p> <p>Dead. <a href="https://youtu.be/NxnZC9L_YXE?t=47">Extinct. Like the doodoo</a>.</p> <h5 id="kubernetes/:once-you-go-kubernetes-you-cant-go-back">Once you go Kubernetes, you can't go back</h5> <p>For years now, <a href="https://www.funkypenguin.co.nz/work-with-me/">I've provided Kubernetes design consulting</a> to small clients and large enterprises. The implementation details in each case vary widely, but there are some primitives which I've come to take for granted, and I wouldn't easily do without. A few examples:</p> <ul> <li><strong>CLI drives API from anywhere</strong>. From my laptop, I can use my credentials to manage any number of Kubernetes clusters, simply by switching kubectl "context". Each interaction is an API call against an HTTPS endpoint. No SSHing to hosts and manually running docker command as root!</li> <li><strong>GitOps is magic</strong>. There are multiple ways to achieve it, but having changes you commit to a repo automatically applied to a cluster, "Just Works™". The process removes so much friction from making changes that it makes you more productive, and a better "gitizen" ;P</li> <li><strong>Controllers are trustworthy</strong>. I've come to trust that when I tell Kubernetes to run 3 replicas on separate hosts, to scale up a set of replicas based on CPU load metrics, or provision a blob of storage for a given workloa, that this will be done in a consistent and visible way. I'll be able to see logs / details for each action taken by the controller, and adjust my own instructions/configuration accordingly if necessary.</li> </ul> <h4 id="kubernetes/:uggh-its-so-complicated">Uggh, it's so complicated!</h4> <p>Yes, it's more complex than Docker Swarm. And that complexity can definately be a barrier, although with improved tooling, it's continually becoming less-so. However, you don't need to be a mechanic to drive a car or to use a chainsaw. You just need a basic understanding of some core primitives, and then you get on with using the tool to achieve your goals, without needing to know every detail about how it works!</p> <p>Your end-goal is probably "<em>I want to reliably self-host services I care about</em>", and not "<em>I want to fully understand a complex, scalable, and highly sophisticated container orchestrator</em>". <sup id="kubernetes/:fnref:1"><a class="footnote-ref" href="#kubernetes/:fn:1">1</a></sup></p> <p>So let's get on with learning how to use the tool...</p> <h4 id="kubernetes/:mm-maaaaybe-how-do-i-start">Mm.. maaaaybe, how do I start?</h4> <p>Primarily you need 2 things:</p> <ol> <li>A <a href="/kubernetes/cluster/">cluster</a></li> <li>A way to <a href="/kubernetes/deployment/">deploy workloads</a> into the cluster</li> </ol> <p>Practically, you need some extras too, but you can mix-and-match these.</p> <div class="footnote"> <hr/> <ol> <li id="kubernetes/:fn:1"> <p>Of course, if you <strong>do</strong> enjoy understanding the intricacies of how your tools work, you're in good company! <a class="footnote-backref" href="#kubernetes/:fnref:1" title="Jump back to footnote 1 in the text">↩</a></p> </li> </ol> </div> <hr/> <div class="md-source-file"> <small> Last update: <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">July 9, 2022</span> </small> </div> </section></section></article><article id="doc-back_cover">
<div class="wrapper"></div><div class="qrcode">
<a href="https://geek-cookbook.funkypenguin.co.nz/">
<img src="data:image/svg+xml;charset=utf-8;base64,PD94bWwgdmVyc2lvbj0nMS4wJyBlbmNvZGluZz0nVVRGLTgnPz4KPHN2ZyB3aWR0aD0iMzdtbSIgaGVpZ2h0PSIzN21tIiB2ZXJzaW9uPSIxLjEiIHZpZXdCb3g9IjAgMCAzNyAzNyIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj48cGF0aCBkPSJNIDMwIDI1IEwgMzAgMjYgTCAzMSAyNiBMIDMxIDI1IHogTSAyMyA0IEwgMjMgNSBMIDI0IDUgTCAyNCA0IHogTSAxMiAxMyBMIDEyIDE0IEwgMTMgMTQgTCAxMyAxMyB6IE0gMjMgMTMgTCAyMyAxNCBMIDI0IDE0IEwgMjQgMTMgeiBNIDQgOSBMIDQgMTAgTCA1IDEwIEwgNSA5IHogTSAxNSA5IEwgMTUgMTAgTCAxNiAxMCBMIDE2IDkgeiBNIDQgMjcgTCA0IDI4IEwgNSAyOCBMIDUgMjcgeiBNIDE1IDI3IEwgMTUgMjggTCAxNiAyOCBMIDE2IDI3IHogTSAxNCAzMSBMIDE0IDMyIEwgMTUgMzIgTCAxNSAzMSB6IE0gMjkgMjkgTCAyOSAzMCBMIDMwIDMwIEwgMzAgMjkgeiBNIDggNiBMIDggNyBMIDkgNyBMIDkgNiB6IE0gMjYgMTQgTCAyNiAxNSBMIDI3IDE1IEwgMjcgMTQgeiBNIDcgMTAgTCA3IDExIEwgOCAxMSBMIDggMTAgeiBNIDI2IDIzIEwgMjYgMjQgTCAyNyAyNCBMIDI3IDIzIHogTSAyNiAzMiBMIDI2IDMzIEwgMjcgMzMgTCAyNyAzMiB6IE0gNyAyOCBMIDcgMjkgTCA4IDI5IEwgOCAyOCB6IE0gMTggMjggTCAxOCAyOSBMIDE5IDI5IEwgMTkgMjggeiBNIDMwIDIwIEwgMzAgMjEgTCAzMSAyMSBMIDMxIDIwIHogTSAzMCAyOSBMIDMwIDMwIEwgMzEgMzAgTCAzMSAyOSB6IE0gMTIgOCBMIDEyIDkgTCAxMyA5IEwgMTMgOCB6IE0gMjMgOCBMIDIzIDkgTCAyNCA5IEwgMjQgOCB6IE0gNCA0IEwgNCA1IEwgNSA1IEwgNSA0IHogTSA0IDEzIEwgNCAxNCBMIDUgMTQgTCA1IDEzIHogTSAyOSAxNSBMIDI5IDE2IEwgMzAgMTYgTCAzMCAxNSB6IE0gMjYgOSBMIDI2IDEwIEwgMjcgMTAgTCAyNyA5IHogTSAxOCA1IEwgMTggNiBMIDE5IDYgTCAxOSA1IHogTSA3IDE0IEwgNyAxNSBMIDggMTUgTCA4IDE0IHogTSAxOCAxNCBMIDE4IDE1IEwgMTkgMTUgTCAxOSAxNCB6IE0gNyAyMyBMIDcgMjQgTCA4IDI0IEwgOCAyMyB6IE0gMTggMjMgTCAxOCAyNCBMIDE5IDI0IEwgMTkgMjMgeiBNIDcgMzIgTCA3IDMzIEwgOCAzMyBMIDggMzIgeiBNIDE4IDMyIEwgMTggMzMgTCAxOSAzMyBMIDE5IDMyIHogTSAzMCA2IEwgMzAgNyBMIDMxIDcgTCAzMSA2IHogTSAzMCAyNCBMIDMwIDI1IEwgMzEgMjUgTCAzMSAyNCB6IE0gMTUgOCBMIDE1IDkgTCAxNiA5IEwgMTYgOCB6IE0gMjkgMTAgTCAyOSAxMSBMIDMwIDExIEwgMzAgMTAgeiBNIDE0IDIxIEwgMTQgMjIgTCAxNSAyMiBMIDE1IDIxIHogTSAxNCAzMCBMIDE0IDMxIEwgMTUgMzEgTCAxNSAzMCB6IE0gNiAyNiBMIDYgMjcgTCA3IDI3IEwgNyAyNiB6IE0gMjYgNCBMIDI2IDUgTCAyNyA1IEwgMjcgNCB6IE0gMjYgMTMgTCAyNiAxNCBMIDI3IDE0IEwgMjcgMTMgeiBNIDE4IDkgTCAxOCAxMCBMIDE5IDEwIEwgMTkgOSB6IE0gMjYgMjIgTCAyNiAyMyBMIDI3IDIzIEwgMjcgMjIgeiBNIDcgMTggTCA3IDE5IEwgOCAxOSBMIDggMTggeiBNIDI2IDMxIEwgMjYgMzIgTCAyNyAzMiBMIDI3IDMxIHogTSAzMCAxMCBMIDMwIDExIEwgMzEgMTEgTCAzMSAxMCB6IE0gMzIgMjkgTCAzMiAzMCBMIDMzIDMwIEwgMzMgMjkgeiBNIDIyIDYgTCAyMiA3IEwgMjMgNyBMIDIzIDYgeiBNIDExIDE1IEwgMTEgMTYgTCAxMiAxNiBMIDEyIDE1IHogTSAyMiAxNSBMIDIyIDE2IEwgMjMgMTYgTCAyMyAxNSB6IE0gMjUgMTcgTCAyNSAxOCBMIDI2IDE4IEwgMjYgMTcgeiBNIDEwIDI4IEwgMTAgMjkgTCAxMSAyOSBMIDExIDI4IHogTSAxNCA3IEwgMTQgOCBMIDE1IDggTCAxNSA3IHogTSA2IDEyIEwgNiAxMyBMIDcgMTMgTCA3IDEyIHogTSA2IDMwIEwgNiAzMSBMIDcgMzEgTCA3IDMwIHogTSAyNiA4IEwgMjYgOSBMIDI3IDkgTCAyNyA4IHogTSA3IDQgTCA3IDUgTCA4IDUgTCA4IDQgeiBNIDI2IDE3IEwgMjYgMTggTCAyNyAxOCBMIDI3IDE3IHogTSAxOCAyMiBMIDE4IDIzIEwgMTkgMjMgTCAxOSAyMiB6IE0gMjEgMjQgTCAyMSAyNSBMIDIyIDI1IEwgMjIgMjQgeiBNIDMyIDI0IEwgMzIgMjUgTCAzMyAyNSBMIDMzIDI0IHogTSAxMyAyOSBMIDEzIDMwIEwgMTQgMzAgTCAxNCAyOSB6IE0gMTAgMTQgTCAxMCAxNSBMIDExIDE1IEwgMTEgMTQgeiBNIDI1IDIxIEwgMjUgMjIgTCAyNiAyMiBMIDI2IDIxIHogTSAxMCAzMiBMIDEwIDMzIEwgMTEgMzMgTCAxMSAzMiB6IE0gMjUgMzAgTCAyNSAzMSBMIDI2IDMxIEwgMjYgMzAgeiBNIDE0IDExIEwgMTQgMTIgTCAxNSAxMiBMIDE1IDExIHogTSA2IDcgTCA2IDggTCA3IDggTCA3IDcgeiBNIDYgMTYgTCA2IDE3IEwgNyAxNyBMIDcgMTYgeiBNIDI5IDE4IEwgMjkgMTkgTCAzMCAxOSBMIDMwIDE4IHogTSAyNiAxMiBMIDI2IDEzIEwgMjcgMTMgTCAyNyAxMiB6IE0gMTcgMzEgTCAxNyAzMiBMIDE4IDMyIEwgMTggMzEgeiBNIDcgOCBMIDcgOSBMIDggOSBMIDggOCB6IE0gMzIgMTAgTCAzMiAxMSBMIDMzIDExIEwgMzMgMTAgeiBNIDEzIDE1IEwgMTMgMTYgTCAxNCAxNiBMIDE0IDE1IHogTSA5IDI2IEwgOSAyNyBMIDEwIDI3IEwgMTAgMjYgeiBNIDIxIDI4IEwgMjEgMjkgTCAyMiAyOSBMIDIyIDI4IHogTSAzMiAyOCBMIDMyIDI5IEwgMzMgMjkgTCAzMyAyOCB6IE0gMTAgOSBMIDEwIDEwIEwgMTEgMTAgTCAxMSA5IHogTSAxMCAxOCBMIDEwIDE5IEwgMTEgMTkgTCAxMSAxOCB6IE0gMTAgMjcgTCAxMCAyOCBMIDExIDI4IEwgMTEgMjcgeiBNIDI5IDQgTCAyOSA1IEwgMzAgNSBMIDMwIDQgeiBNIDE0IDE1IEwgMTQgMTYgTCAxNSAxNiBMIDE1IDE1IHogTSAxNyAxNyBMIDE3IDE4IEwgMTggMTggTCAxOCAxNyB6IE0gMjggMTcgTCAyOCAxOCBMIDI5IDE4IEwgMjkgMTcgeiBNIDE3IDI2IEwgMTcgMjcgTCAxOCAyNyBMIDE4IDI2IHogTSAyOCAyNiBMIDI4IDI3IEwgMjkgMjcgTCAyOSAyNiB6IE0gMjAgMjIgTCAyMCAyMyBMIDIxIDIzIEwgMjEgMjIgeiBNIDIwIDMxIEwgMjAgMzIgTCAyMSAzMiBMIDIxIDMxIHogTSAyMSA1IEwgMjEgNiBMIDIyIDYgTCAyMiA1IHogTSAzMiA1IEwgMzIgNiBMIDMzIDYgTCAzMyA1IHogTSA5IDEyIEwgOSAxMyBMIDEwIDEzIEwgMTAgMTIgeiBNIDIxIDE0IEwgMjEgMTUgTCAyMiAxNSBMIDIyIDE0IHogTSA5IDIxIEwgOSAyMiBMIDEwIDIyIEwgMTAgMjEgeiBNIDEzIDE5IEwgMTMgMjAgTCAxNCAyMCBMIDE0IDE5IHogTSAyMSAzMiBMIDIxIDMzIEwgMjIgMzMgTCAyMiAzMiB6IE0gMTAgNCBMIDEwIDUgTCAxMSA1IEwgMTEgNCB6IE0gMTAgMjIgTCAxMCAyMyBMIDExIDIzIEwgMTEgMjIgeiBNIDI0IDI0IEwgMjQgMjUgTCAyNSAyNSBMIDI1IDI0IHogTSAxNCAxMCBMIDE0IDExIEwgMTUgMTEgTCAxNSAxMCB6IE0gMTYgMjkgTCAxNiAzMCBMIDE3IDMwIEwgMTcgMjkgeiBNIDYgNiBMIDYgNyBMIDcgNyBMIDcgNiB6IE0gNiAxNSBMIDYgMTYgTCA3IDE2IEwgNyAxNSB6IE0gMTcgMTIgTCAxNyAxMyBMIDE4IDEzIEwgMTggMTIgeiBNIDI4IDEyIEwgMjggMTMgTCAyOSAxMyBMIDI5IDEyIHogTSA1IDE5IEwgNSAyMCBMIDYgMjAgTCA2IDE5IHogTSAxNyAyMSBMIDE3IDIyIEwgMTggMjIgTCAxOCAyMSB6IE0gMjggMjEgTCAyOCAyMiBMIDI5IDIyIEwgMjkgMjEgeiBNIDIwIDE3IEwgMjAgMTggTCAyMSAxOCBMIDIxIDE3IHogTSAxNyAzMCBMIDE3IDMxIEwgMTggMzEgTCAxOCAzMCB6IE0gMjggMzAgTCAyOCAzMSBMIDI5IDMxIEwgMjkgMzAgeiBNIDIwIDI2IEwgMjAgMjcgTCAyMSAyNyBMIDIxIDI2IHogTSAzMiA5IEwgMzIgMTAgTCAzMyAxMCBMIDMzIDkgeiBNIDkgMTYgTCA5IDE3IEwgMTAgMTcgTCAxMCAxNiB6IE0gMjEgMTggTCAyMSAxOSBMIDIyIDE5IEwgMjIgMTggeiBNIDEzIDE0IEwgMTMgMTUgTCAxNCAxNSBMIDE0IDE0IHogTSAxMCA4IEwgMTAgOSBMIDExIDkgTCAxMSA4IHogTSAyNCAxOSBMIDI0IDIwIEwgMjUgMjAgTCAyNSAxOSB6IE0gMjQgMjggTCAyNCAyOSBMIDI1IDI5IEwgMjUgMjggeiBNIDE2IDI0IEwgMTYgMjUgTCAxNyAyNSBMIDE3IDI0IHogTSAyNyAyNCBMIDI3IDI1IEwgMjggMjUgTCAyOCAyNCB6IE0gOCAyOSBMIDggMzAgTCA5IDMwIEwgOSAyOSB6IE0gMTcgNyBMIDE3IDggTCAxOCA4IEwgMTggNyB6IE0gMjggNyBMIDI4IDggTCAyOSA4IEwgMjkgNyB6IE0gNSAxNCBMIDUgMTUgTCA2IDE1IEwgNiAxNCB6IE0gMTcgMTYgTCAxNyAxNyBMIDE4IDE3IEwgMTggMTYgeiBNIDE3IDI1IEwgMTcgMjYgTCAxOCAyNiBMIDE4IDI1IHogTSAyOCAyNSBMIDI4IDI2IEwgMjkgMjYgTCAyOSAyNSB6IE0gMjAgMjEgTCAyMCAyMiBMIDIxIDIyIEwgMjEgMjEgeiBNIDUgMzIgTCA1IDMzIEwgNiAzMyBMIDYgMzIgeiBNIDMyIDQgTCAzMiA1IEwgMzMgNSBMIDMzIDQgeiBNIDIxIDEzIEwgMjEgMTQgTCAyMiAxNCBMIDIyIDEzIHogTSAzMiAxMyBMIDMyIDE0IEwgMzMgMTQgTCAzMyAxMyB6IE0gMTMgMTggTCAxMyAxOSBMIDE0IDE5IEwgMTQgMTggeiBNIDI0IDE0IEwgMjQgMTUgTCAyNSAxNSBMIDI1IDE0IHogTSAxNiAxMCBMIDE2IDExIEwgMTcgMTEgTCAxNyAxMCB6IE0gMjcgMTAgTCAyNyAxMSBMIDI4IDExIEwgMjggMTAgeiBNIDE2IDE5IEwgMTYgMjAgTCAxNyAyMCBMIDE3IDE5IHogTSAyNyAxOSBMIDI3IDIwIEwgMjggMjAgTCAyOCAxOSB6IE0gMTYgMjggTCAxNiAyOSBMIDE3IDI5IEwgMTcgMjggeiBNIDI3IDI4IEwgMjcgMjkgTCAyOCAyOSBMIDI4IDI4IHogTSA4IDI0IEwgOCAyNSBMIDkgMjUgTCA5IDI0IHogTSAxNyAyMCBMIDE3IDIxIEwgMTggMjEgTCAxOCAyMCB6IE0gMjAgMjUgTCAyMCAyNiBMIDIxIDI2IEwgMjEgMjUgeiBNIDMyIDggTCAzMiA5IEwgMzMgOSBMIDMzIDggeiBNIDEzIDQgTCAxMyA1IEwgMTQgNSBMIDE0IDQgeiBNIDkgMTUgTCA5IDE2IEwgMTAgMTYgTCAxMCAxNSB6IE0gMjMgMTcgTCAyMyAxOCBMIDI0IDE4IEwgMjQgMTcgeiBNIDMxIDMwIEwgMzEgMzEgTCAzMiAzMSBMIDMyIDMwIHogTSAyMyAyNiBMIDIzIDI3IEwgMjQgMjcgTCAyNCAyNiB6IE0gNCAyMiBMIDQgMjMgTCA1IDIzIEwgNSAyMiB6IE0gNCAzMSBMIDQgMzIgTCA1IDMyIEwgNSAzMSB6IE0gMjQgOSBMIDI0IDEwIEwgMjUgMTAgTCAyNSA5IHogTSAyNCAxOCBMIDI0IDE5IEwgMjUgMTkgTCAyNSAxOCB6IE0gMTYgMTQgTCAxNiAxNSBMIDE3IDE1IEwgMTcgMTQgeiBNIDI3IDE0IEwgMjcgMTUgTCAyOCAxNSBMIDI4IDE0IHogTSAyNCAyNyBMIDI0IDI4IEwgMjUgMjggTCAyNSAyNyB6IE0gOCAxMCBMIDggMTEgTCA5IDExIEwgOSAxMCB6IE0gMjcgMjMgTCAyNyAyNCBMIDI4IDI0IEwgMjggMjMgeiBNIDggMTkgTCA4IDIwIEwgOSAyMCBMIDkgMTkgeiBNIDE2IDMyIEwgMTYgMzMgTCAxNyAzMyBMIDE3IDMyIHogTSAyNyAzMiBMIDI3IDMzIEwgMjggMzMgTCAyOCAzMiB6IE0gOCAyOCBMIDggMjkgTCA5IDI5IEwgOSAyOCB6IE0gNSA0IEwgNSA1IEwgNiA1IEwgNiA0IHogTSAyOCA2IEwgMjggNyBMIDI5IDcgTCAyOSA2IHogTSA1IDEzIEwgNSAxNCBMIDYgMTQgTCA2IDEzIHogTSAyMCAxMSBMIDIwIDEyIEwgMjEgMTIgTCAyMSAxMSB6IE0gMTkgMjQgTCAxOSAyNSBMIDIwIDI1IEwgMjAgMjQgeiBNIDkgMTAgTCA5IDExIEwgMTAgMTEgTCAxMCAxMCB6IE0gMjIgMjkgTCAyMiAzMCBMIDIzIDMwIEwgMjMgMjkgeiBNIDIzIDEyIEwgMjMgMTMgTCAyNCAxMyBMIDI0IDEyIHogTSAzMSAyNSBMIDMxIDI2IEwgMzIgMjYgTCAzMiAyNSB6IE0gNCA4IEwgNCA5IEwgNSA5IEwgNSA4IHogTSAxMiAyMSBMIDEyIDIyIEwgMTMgMjIgTCAxMyAyMSB6IE0gMjMgMjEgTCAyMyAyMiBMIDI0IDIyIEwgMjQgMjEgeiBNIDQgMTcgTCA0IDE4IEwgNSAxOCBMIDUgMTcgeiBNIDEyIDMwIEwgMTIgMzEgTCAxMyAzMSBMIDEzIDMwIHogTSA0IDI2IEwgNCAyNyBMIDUgMjcgTCA1IDI2IHogTSAxNSAyNiBMIDE1IDI3IEwgMTYgMjcgTCAxNiAyNiB6IE0gMjQgNCBMIDI0IDUgTCAyNSA1IEwgMjUgNCB6IE0gMjQgMTMgTCAyNCAxNCBMIDI1IDE0IEwgMjUgMTMgeiBNIDE2IDE4IEwgMTYgMTkgTCAxNyAxOSBMIDE3IDE4IHogTSAyNyAxOCBMIDI3IDE5IEwgMjggMTkgTCAyOCAxOCB6IE0gOCAzMiBMIDggMzMgTCA5IDMzIEwgOSAzMiB6IE0gMjggMTAgTCAyOCAxMSBMIDI5IDExIEwgMjkgMTAgeiBNIDUgMTcgTCA1IDE4IEwgNiAxOCBMIDYgMTcgeiBNIDE5IDE5IEwgMTkgMjAgTCAyMCAyMCBMIDIwIDE5IHogTSAxOSAyOCBMIDE5IDI5IEwgMjAgMjkgTCAyMCAyOCB6IE0gMzAgMjggTCAzMCAyOSBMIDMxIDI5IEwgMzEgMjggeiBNIDEyIDcgTCAxMiA4IEwgMTMgOCBMIDEzIDcgeiBNIDIzIDcgTCAyMyA4IEwgMjQgOCBMIDI0IDcgeiBNIDIzIDE2IEwgMjMgMTcgTCAyNCAxNyBMIDI0IDE2IHogTSAzMSAyOSBMIDMxIDMwIEwgMzIgMzAgTCAzMiAyOSB6IE0gNCAxMiBMIDQgMTMgTCA1IDEzIEwgNSAxMiB6IE0gMTIgMjUgTCAxMiAyNiBMIDEzIDI2IEwgMTMgMjUgeiBNIDQgMjEgTCA0IDIyIEwgNSAyMiBMIDUgMjEgeiBNIDQgMzAgTCA0IDMxIEwgNSAzMSBMIDUgMzAgeiBNIDE2IDQgTCAxNiA1IEwgMTcgNSBMIDE3IDQgeiBNIDI3IDQgTCAyNyA1IEwgMjggNSBMIDI4IDQgeiBNIDI0IDE3IEwgMjQgMTggTCAyNSAxOCBMIDI1IDE3IHogTSAyNyAyMiBMIDI3IDIzIEwgMjggMjMgTCAyOCAyMiB6IE0gOCAxOCBMIDggMTkgTCA5IDE5IEwgOSAxOCB6IE0gMjYgMjYgTCAyNiAyNyBMIDI3IDI3IEwgMjcgMjYgeiBNIDE4IDMxIEwgMTggMzIgTCAxOSAzMiBMIDE5IDMxIHogTSAxOSA1IEwgMTkgNiBMIDIwIDYgTCAyMCA1IHogTSAxOSAxNCBMIDE5IDE1IEwgMjAgMTUgTCAyMCAxNCB6IE0gMjIgMTAgTCAyMiAxMSBMIDIzIDExIEwgMjMgMTAgeiBNIDE5IDIzIEwgMTkgMjQgTCAyMCAyNCBMIDIwIDIzIHogTSAxMSAxOSBMIDExIDIwIEwgMTIgMjAgTCAxMiAxOSB6IE0gMzAgMzIgTCAzMCAzMyBMIDMxIDMzIEwgMzEgMzIgeiBNIDMxIDE1IEwgMzEgMTYgTCAzMiAxNiBMIDMyIDE1IHogTSAxMiAxMSBMIDEyIDEyIEwgMTMgMTIgTCAxMyAxMSB6IE0gMjMgMTEgTCAyMyAxMiBMIDI0IDEyIEwgMjQgMTEgeiBNIDMxIDI0IEwgMzEgMjUgTCAzMiAyNSBMIDMyIDI0IHogTSA0IDcgTCA0IDggTCA1IDggTCA1IDcgeiBNIDE1IDcgTCAxNSA4IEwgMTYgOCBMIDE2IDcgeiBNIDEyIDIwIEwgMTIgMjEgTCAxMyAyMSBMIDEzIDIwIHogTSAyMyAyMCBMIDIzIDIxIEwgMjQgMjEgTCAyNCAyMCB6IE0gMTUgMTYgTCAxNSAxNyBMIDE2IDE3IEwgMTYgMTYgeiBNIDIzIDI5IEwgMjMgMzAgTCAyNCAzMCBMIDI0IDI5IHogTSAxNSAyNSBMIDE1IDI2IEwgMTYgMjYgTCAxNiAyNSB6IE0gMTYgOCBMIDE2IDkgTCAxNyA5IEwgMTcgOCB6IE0gOCA0IEwgOCA1IEwgOSA1IEwgOSA0IHogTSA4IDEzIEwgOCAxNCBMIDkgMTQgTCA5IDEzIHogTSAyNiAyMSBMIDI2IDIyIEwgMjcgMjIgTCAyNyAyMSB6IE0gNyAxNyBMIDcgMTggTCA4IDE4IEwgOCAxNyB6IE0gMTggMTcgTCAxOCAxOCBMIDE5IDE4IEwgMTkgMTcgeiBNIDI2IDMwIEwgMjYgMzEgTCAyNyAzMSBMIDI3IDMwIHogTSA3IDI2IEwgNyAyNyBMIDggMjcgTCA4IDI2IHogTSAxOCAyNiBMIDE4IDI3IEwgMTkgMjcgTCAxOSAyNiB6IE0gMTkgOSBMIDE5IDEwIEwgMjAgMTAgTCAyMCA5IHogTSAxOSAxOCBMIDE5IDE5IEwgMjAgMTkgTCAyMCAxOCB6IE0gMzAgMTggTCAzMCAxOSBMIDMxIDE5IEwgMzEgMTggeiBNIDMxIDEwIEwgMzEgMTEgTCAzMiAxMSBMIDMyIDEwIHogTSAxMiA2IEwgMTIgNyBMIDEzIDcgTCAxMyA2IHogTSAyMyA2IEwgMjMgNyBMIDI0IDcgTCAyNCA2IHogTSAxMiAxNSBMIDEyIDE2IEwgMTMgMTYgTCAxMyAxNSB6IE0gMjMgMTUgTCAyMyAxNiBMIDI0IDE2IEwgMjQgMTUgeiBNIDE1IDExIEwgMTUgMTIgTCAxNiAxMiBMIDE2IDExIHogTSAxNCAyNCBMIDE0IDI1IEwgMTUgMjUgTCAxNSAyNCB6IE0gNiAyOSBMIDYgMzAgTCA3IDMwIEwgNyAyOSB6IE0gMjkgMzEgTCAyOSAzMiBMIDMwIDMyIEwgMzAgMzEgeiBNIDI2IDcgTCAyNiA4IEwgMjcgOCBMIDI3IDcgeiBNIDcgMTIgTCA3IDEzIEwgOCAxMyBMIDggMTIgeiBNIDE4IDEyIEwgMTggMTMgTCAxOSAxMyBMIDE5IDEyIHogTSA3IDMwIEwgNyAzMSBMIDggMzEgTCA4IDMwIHogTSAzMCA0IEwgMzAgNSBMIDMxIDUgTCAzMSA0IHogTSAxOSAyMiBMIDE5IDIzIEwgMjAgMjMgTCAyMCAyMiB6IE0gMzAgMjIgTCAzMCAyMyBMIDMxIDIzIEwgMzEgMjIgeiBNIDIyIDE4IEwgMjIgMTkgTCAyMyAxOSBMIDIzIDE4IHogTSAyMiAyNyBMIDIyIDI4IEwgMjMgMjggTCAyMyAyNyB6IE0gMTAgMzEgTCAxMCAzMiBMIDExIDMyIEwgMTEgMzEgeiBNIDQgNiBMIDQgNyBMIDUgNyBMIDUgNiB6IE0gMTUgNiBMIDE1IDcgTCAxNiA3IEwgMTYgNiB6IE0gMjkgOCBMIDI5IDkgTCAzMCA5IEwgMzAgOCB6IE0gMTQgMjggTCAxNCAyOSBMIDE1IDI5IEwgMTUgMjggeiBNIDYgMjQgTCA2IDI1IEwgNyAyNSBMIDcgMjQgeiBNIDI5IDI2IEwgMjkgMjcgTCAzMCAyNyBMIDMwIDI2IHogTSA3IDcgTCA3IDggTCA4IDggTCA4IDcgeiBNIDE4IDE2IEwgMTggMTcgTCAxOSAxNyBMIDE5IDE2IHogTSAzMCA4IEwgMzAgOSBMIDMxIDkgTCAzMSA4IHogTSAyMiA0IEwgMjIgNSBMIDIzIDUgTCAyMyA0IHogTSAxMSAxMyBMIDExIDE0IEwgMTIgMTQgTCAxMiAxMyB6IE0gMjIgMTMgTCAyMiAxNCBMIDIzIDE0IEwgMjMgMTMgeiBNIDIyIDIyIEwgMjIgMjMgTCAyMyAyMyBMIDIzIDIyIHogTSAxMCAyNiBMIDEwIDI3IEwgMTEgMjcgTCAxMSAyNiB6IE0gMjUgMjQgTCAyNSAyNSBMIDI2IDI1IEwgMjYgMjQgeiBNIDE0IDUgTCAxNCA2IEwgMTUgNiBMIDE1IDUgeiBNIDYgMTAgTCA2IDExIEwgNyAxMSBMIDcgMTAgeiBNIDI5IDEyIEwgMjkgMTMgTCAzMCAxMyBMIDMwIDEyIHogTSA2IDE5IEwgNiAyMCBMIDcgMjAgTCA3IDE5IHogTSAxNCAzMiBMIDE0IDMzIEwgMTUgMzMgTCAxNSAzMiB6IE0gNiAyOCBMIDYgMjkgTCA3IDI5IEwgNyAyOCB6IE0gMjkgMzAgTCAyOSAzMSBMIDMwIDMxIEwgMzAgMzAgeiBNIDI2IDYgTCAyNiA3IEwgMjcgNyBMIDI3IDYgeiBNIDI2IDI0IEwgMjYgMjUgTCAyNyAyNSBMIDI3IDI0IHogTSA3IDIwIEwgNyAyMSBMIDggMjEgTCA4IDIwIHogTSA3IDI5IEwgNyAzMCBMIDggMzAgTCA4IDI5IHogTSAxMyAyNyBMIDEzIDI4IEwgMTQgMjggTCAxNCAyNyB6IE0gMTAgMTIgTCAxMCAxMyBMIDExIDEzIEwgMTEgMTIgeiBNIDEwIDMwIEwgMTAgMzEgTCAxMSAzMSBMIDExIDMwIHogTSAyNSAyOCBMIDI1IDI5IEwgMjYgMjkgTCAyNiAyOCB6IE0gMjkgNyBMIDI5IDggTCAzMCA4IEwgMzAgNyB6IE0gNiAxNCBMIDYgMTUgTCA3IDE1IEwgNyAxNCB6IE0gMjkgMTYgTCAyOSAxNyBMIDMwIDE3IEwgMzAgMTYgeiBNIDYgMjMgTCA2IDI0IEwgNyAyNCBMIDcgMjMgeiBNIDI5IDI1IEwgMjkgMjYgTCAzMCAyNiBMIDMwIDI1IHogTSA2IDMyIEwgNiAzMyBMIDcgMzMgTCA3IDMyIHogTSAyNiAxMCBMIDI2IDExIEwgMjcgMTEgTCAyNyAxMCB6IE0gNyA2IEwgNyA3IEwgOCA3IEwgOCA2IHogTSAxOCA2IEwgMTggNyBMIDE5IDcgTCAxOSA2IHogTSA3IDE1IEwgNyAxNiBMIDggMTYgTCA4IDE1IHogTSAyMSAxNyBMIDIxIDE4IEwgMjIgMTggTCAyMiAxNyB6IE0gMzIgMTcgTCAzMiAxOCBMIDMzIDE4IEwgMzMgMTcgeiBNIDkgMjQgTCA5IDI1IEwgMTAgMjUgTCAxMCAyNCB6IE0gMTMgMjIgTCAxMyAyMyBMIDE0IDIzIEwgMTQgMjIgeiBNIDEzIDMxIEwgMTMgMzIgTCAxNCAzMiBMIDE0IDMxIHogTSAxMCA3IEwgMTAgOCBMIDExIDggTCAxMSA3IHogTSAxMCAxNiBMIDEwIDE3IEwgMTEgMTcgTCAxMSAxNiB6IE0gMTQgMjIgTCAxNCAyMyBMIDE1IDIzIEwgMTUgMjIgeiBNIDYgMTggTCA2IDE5IEwgNyAxOSBMIDcgMTggeiBNIDI5IDIwIEwgMjkgMjEgTCAzMCAyMSBMIDMwIDIwIHogTSAyNiA1IEwgMjYgNiBMIDI3IDYgTCAyNyA1IHogTSAyOCAyNCBMIDI4IDI1IEwgMjkgMjUgTCAyOSAyNCB6IE0gMTggMTAgTCAxOCAxMSBMIDE5IDExIEwgMTkgMTAgeiBNIDIwIDI5IEwgMjAgMzAgTCAyMSAzMCBMIDIxIDI5IHogTSAyMSAxMiBMIDIxIDEzIEwgMjIgMTMgTCAyMiAxMiB6IE0gMTMgOCBMIDEzIDkgTCAxNCA5IEwgMTQgOCB6IE0gOSAxOSBMIDkgMjAgTCAxMCAyMCBMIDEwIDE5IHogTSAyMSAyMSBMIDIxIDIyIEwgMjIgMjIgTCAyMiAyMSB6IE0gMzIgMjEgTCAzMiAyMiBMIDMzIDIyIEwgMzMgMjEgeiBNIDEzIDI2IEwgMTMgMjcgTCAxNCAyNyBMIDE0IDI2IHogTSAxMCAyMCBMIDEwIDIxIEwgMTEgMjEgTCAxMSAyMCB6IE0gMjUgMTggTCAyNSAxOSBMIDI2IDE5IEwgMjYgMTggeiBNIDEwIDI5IEwgMTAgMzAgTCAxMSAzMCBMIDExIDI5IHogTSAyNCAzMSBMIDI0IDMyIEwgMjUgMzIgTCAyNSAzMSB6IE0gNiA0IEwgNiA1IEwgNyA1IEwgNyA0IHogTSAyOSA2IEwgMjkgNyBMIDMwIDcgTCAzMCA2IHogTSAxNCAxNyBMIDE0IDE4IEwgMTUgMTggTCAxNSAxNyB6IE0gNiAxMyBMIDYgMTQgTCA3IDE0IEwgNyAxMyB6IE0gMjggMTkgTCAyOCAyMCBMIDI5IDIwIEwgMjkgMTkgeiBNIDIwIDE1IEwgMjAgMTYgTCAyMSAxNiBMIDIxIDE1IHogTSA1IDI2IEwgNSAyNyBMIDYgMjcgTCA2IDI2IHogTSAyOCAyOCBMIDI4IDI5IEwgMjkgMjkgTCAyOSAyOCB6IE0gMjAgMjQgTCAyMCAyNSBMIDIxIDI1IEwgMjEgMjQgeiBNIDIxIDcgTCAyMSA4IEwgMjIgOCBMIDIyIDcgeiBNIDMyIDcgTCAzMiA4IEwgMzMgOCBMIDMzIDcgeiBNIDkgMTQgTCA5IDE1IEwgMTAgMTUgTCAxMCAxNCB6IE0gMjEgMTYgTCAyMSAxNyBMIDIyIDE3IEwgMjIgMTYgeiBNIDMyIDI1IEwgMzIgMjYgTCAzMyAyNiBMIDMzIDI1IHogTSA5IDMyIEwgOSAzMyBMIDEwIDMzIEwgMTAgMzIgeiBNIDEzIDMwIEwgMTMgMzEgTCAxNCAzMSBMIDE0IDMwIHogTSAxMCA2IEwgMTAgNyBMIDExIDcgTCAxMSA2IHogTSAyNSAxMyBMIDI1IDE0IEwgMjYgMTQgTCAyNiAxMyB6IE0gMTAgMjQgTCAxMCAyNSBMIDExIDI1IEwgMTEgMjQgeiBNIDI0IDI2IEwgMjQgMjcgTCAyNSAyNyBMIDI1IDI2IHogTSAyNyAzMSBMIDI3IDMyIEwgMjggMzIgTCAyOCAzMSB6IE0gNiA4IEwgNiA5IEwgNyA5IEwgNyA4IHogTSAxNyA1IEwgMTcgNiBMIDE4IDYgTCAxOCA1IHogTSAxNyAxNCBMIDE3IDE1IEwgMTggMTUgTCAxOCAxNCB6IE0gMjAgMTAgTCAyMCAxMSBMIDIxIDExIEwgMjEgMTAgeiBNIDI4IDIzIEwgMjggMjQgTCAyOSAyNCBMIDI5IDIzIHogTSAxNyAzMiBMIDE3IDMzIEwgMTggMzMgTCAxOCAzMiB6IE0gMjggMzIgTCAyOCAzMyBMIDI5IDMzIEwgMjkgMzIgeiBNIDkgMTggTCA5IDE5IEwgMTAgMTkgTCAxMCAxOCB6IE0gMjEgMjAgTCAyMSAyMSBMIDIyIDIxIEwgMjIgMjAgeiBNIDEzIDE2IEwgMTMgMTcgTCAxNCAxNyBMIDE0IDE2IHogTSAxMyAyNSBMIDEzIDI2IEwgMTQgMjYgTCAxNCAyNSB6IE0gMTAgMTAgTCAxMCAxMSBMIDExIDExIEwgMTEgMTAgeiBNIDEyIDI5IEwgMTIgMzAgTCAxMyAzMCBMIDEzIDI5IHogTSAyNCAyMSBMIDI0IDIyIEwgMjUgMjIgTCAyNSAyMSB6IE0gMjcgMTcgTCAyNyAxOCBMIDI4IDE4IEwgMjggMTcgeiBNIDI0IDMwIEwgMjQgMzEgTCAyNSAzMSBMIDI1IDMwIHogTSAxNiAyNiBMIDE2IDI3IEwgMTcgMjcgTCAxNyAyNiB6IE0gMTcgOSBMIDE3IDEwIEwgMTggMTAgTCAxOCA5IHogTSAyMCA1IEwgMjAgNiBMIDIxIDYgTCAyMSA1IHogTSA1IDE2IEwgNSAxNyBMIDYgMTcgTCA2IDE2IHogTSAxNyAxOCBMIDE3IDE5IEwgMTggMTkgTCAxOCAxOCB6IE0gMjggMTggTCAyOCAxOSBMIDI5IDE5IEwgMjkgMTggeiBNIDE3IDI3IEwgMTcgMjggTCAxOCAyOCBMIDE4IDI3IHogTSAyOCAyNyBMIDI4IDI4IEwgMjkgMjggTCAyOSAyNyB6IE0gMjAgMjMgTCAyMCAyNCBMIDIxIDI0IEwgMjEgMjMgeiBNIDkgNCBMIDkgNSBMIDEwIDUgTCAxMCA0IHogTSAzMiA2IEwgMzIgNyBMIDMzIDcgTCAzMyA2IHogTSAxMyAxMSBMIDEzIDEyIEwgMTQgMTIgTCAxNCAxMSB6IE0gOSAyMiBMIDkgMjMgTCAxMCAyMyBMIDEwIDIyIHogTSAzMSAxOSBMIDMxIDIwIEwgMzIgMjAgTCAzMiAxOSB6IE0gMzEgMjggTCAzMSAyOSBMIDMyIDI5IEwgMzIgMjggeiBNIDEwIDUgTCAxMCA2IEwgMTEgNiBMIDExIDUgeiBNIDEyIDI0IEwgMTIgMjUgTCAxMyAyNSBMIDEzIDI0IHogTSA0IDI5IEwgNCAzMCBMIDUgMzAgTCA1IDI5IHogTSAxNiAxMiBMIDE2IDEzIEwgMTcgMTMgTCAxNyAxMiB6IE0gMjcgMTIgTCAyNyAxMyBMIDI4IDEzIEwgMjggMTIgeiBNIDI0IDI1IEwgMjQgMjYgTCAyNSAyNiBMIDI1IDI1IHogTSA4IDggTCA4IDkgTCA5IDkgTCA5IDggeiBNIDI3IDIxIEwgMjcgMjIgTCAyOCAyMiBMIDI4IDIxIHogTSAxNiAzMCBMIDE2IDMxIEwgMTcgMzEgTCAxNyAzMCB6IE0gMjcgMzAgTCAyNyAzMSBMIDI4IDMxIEwgMjggMzAgeiBNIDggMjYgTCA4IDI3IEwgOSAyNyBMIDkgMjYgeiBNIDE3IDQgTCAxNyA1IEwgMTggNSBMIDE4IDQgeiBNIDI4IDQgTCAyOCA1IEwgMjkgNSBMIDI5IDQgeiBNIDE3IDEzIEwgMTcgMTQgTCAxOCAxNCBMIDE4IDEzIHogTSAyOCAxMyBMIDI4IDE0IEwgMjkgMTQgTCAyOSAxMyB6IE0gNSAyMCBMIDUgMjEgTCA2IDIxIEwgNiAyMCB6IE0gMTcgMjIgTCAxNyAyMyBMIDE4IDIzIEwgMTggMjIgeiBNIDI4IDIyIEwgMjggMjMgTCAyOSAyMyBMIDI5IDIyIHogTSAyMCAxOCBMIDIwIDE5IEwgMjEgMTkgTCAyMSAxOCB6IE0gMjAgMjcgTCAyMCAyOCBMIDIxIDI4IEwgMjEgMjcgeiBNIDkgMTcgTCA5IDE4IEwgMTAgMTggTCAxMCAxNyB6IE0gMTIgMTAgTCAxMiAxMSBMIDEzIDExIEwgMTMgMTAgeiBNIDMxIDIzIEwgMzEgMjQgTCAzMiAyNCBMIDMyIDIzIHogTSAyMyAxOSBMIDIzIDIwIEwgMjQgMjAgTCAyNCAxOSB6IE0gNCAxNSBMIDQgMTYgTCA1IDE2IEwgNSAxNSB6IE0gMTIgMjggTCAxMiAyOSBMIDEzIDI5IEwgMTMgMjggeiBNIDQgMjQgTCA0IDI1IEwgNSAyNSBMIDUgMjQgeiBNIDE1IDI0IEwgMTUgMjUgTCAxNiAyNSBMIDE2IDI0IHogTSAyNCAyOSBMIDI0IDMwIEwgMjUgMzAgTCAyNSAyOSB6IE0gOCAxMiBMIDggMTMgTCA5IDEzIEwgOSAxMiB6IE0gMTYgMjUgTCAxNiAyNiBMIDE3IDI2IEwgMTcgMjUgeiBNIDggMjEgTCA4IDIyIEwgOSAyMiBMIDkgMjEgeiBNIDggMzAgTCA4IDMxIEwgOSAzMSBMIDkgMzAgeiBNIDE3IDggTCAxNyA5IEwgMTggOSBMIDE4IDggeiBNIDI4IDggTCAyOCA5IEwgMjkgOSBMIDI5IDggeiBNIDIwIDQgTCAyMCA1IEwgMjEgNSBMIDIxIDQgeiBNIDUgMTUgTCA1IDE2IEwgNiAxNiBMIDYgMTUgeiBNIDE5IDI2IEwgMTkgMjcgTCAyMCAyNyBMIDIwIDI2IHogTSAzMCAyNiBMIDMwIDI3IEwgMzEgMjcgTCAzMSAyNiB6IE0gMTEgMjIgTCAxMSAyMyBMIDEyIDIzIEwgMTIgMjIgeiBNIDEyIDE0IEwgMTIgMTUgTCAxMyAxNSBMIDEzIDE0IHogTSA0IDEwIEwgNCAxMSBMIDUgMTEgTCA1IDEwIHogTSAyMyAyMyBMIDIzIDI0IEwgMjQgMjQgTCAyNCAyMyB6IE0gMTUgMTkgTCAxNSAyMCBMIDE2IDIwIEwgMTYgMTkgeiBNIDEyIDMyIEwgMTIgMzMgTCAxMyAzMyBMIDEzIDMyIHogTSAyMyAzMiBMIDIzIDMzIEwgMjQgMzMgTCAyNCAzMiB6IE0gNCAyOCBMIDQgMjkgTCA1IDI5IEwgNSAyOCB6IE0gMTUgMjggTCAxNSAyOSBMIDE2IDI5IEwgMTYgMjggeiBNIDI0IDYgTCAyNCA3IEwgMjUgNyBMIDI1IDYgeiBNIDI0IDE1IEwgMjQgMTYgTCAyNSAxNiBMIDI1IDE1IHogTSAxNiAxMSBMIDE2IDEyIEwgMTcgMTIgTCAxNyAxMSB6IE0gOCA3IEwgOCA4IEwgOSA4IEwgOSA3IHogTSA4IDE2IEwgOCAxNyBMIDkgMTcgTCA5IDE2IHogTSA1IDEwIEwgNSAxMSBMIDYgMTEgTCA2IDEwIHogTSAyMCA4IEwgMjAgOSBMIDIxIDkgTCAyMSA4IHogTSAzMCAxMiBMIDMwIDEzIEwgMzEgMTMgTCAzMSAxMiB6IE0gMzAgMjEgTCAzMCAyMiBMIDMxIDIyIEwgMzEgMjEgeiBNIDIyIDE3IEwgMjIgMTggTCAyMyAxOCBMIDIzIDE3IHogTSAxOSAzMCBMIDE5IDMxIEwgMjAgMzEgTCAyMCAzMCB6IE0gMzAgMzAgTCAzMCAzMSBMIDMxIDMxIEwgMzEgMzAgeiBNIDIyIDI2IEwgMjIgMjcgTCAyMyAyNyBMIDIzIDI2IHogTSAzMSA0IEwgMzEgNSBMIDMyIDUgTCAzMiA0IHogTSAxMiA5IEwgMTIgMTAgTCAxMyAxMCBMIDEzIDkgeiBNIDQgNSBMIDQgNiBMIDUgNiBMIDUgNSB6IE0gMTUgNSBMIDE1IDYgTCAxNiA2IEwgMTYgNSB6IE0gMTIgMTggTCAxMiAxOSBMIDEzIDE5IEwgMTMgMTggeiBNIDMxIDMxIEwgMzEgMzIgTCAzMiAzMiBMIDMyIDMxIHogTSA0IDE0IEwgNCAxNSBMIDUgMTUgTCA1IDE0IHogTSAxMiAyNyBMIDEyIDI4IEwgMTMgMjggTCAxMyAyNyB6IE0gMjMgMjcgTCAyMyAyOCBMIDI0IDI4IEwgMjQgMjcgeiBNIDQgMjMgTCA0IDI0IEwgNSAyNCBMIDUgMjMgeiBNIDE1IDIzIEwgMTUgMjQgTCAxNiAyNCBMIDE2IDIzIHogTSA0IDMyIEwgNCAzMyBMIDUgMzMgTCA1IDMyIHogTSAyNCAxMCBMIDI0IDExIEwgMjUgMTEgTCAyNSAxMCB6IE0gOCAyMCBMIDggMjEgTCA5IDIxIEwgOSAyMCB6IE0gMjYgMTkgTCAyNiAyMCBMIDI3IDIwIEwgMjcgMTkgeiBNIDI2IDI4IEwgMjYgMjkgTCAyNyAyOSBMIDI3IDI4IHogTSA3IDI0IEwgNyAyNSBMIDggMjUgTCA4IDI0IHogTSAxOSA3IEwgMTkgOCBMIDIwIDggTCAyMCA3IHogTSAzMCA3IEwgMzAgOCBMIDMxIDggTCAzMSA3IHogTSAzMCAxNiBMIDMwIDE3IEwgMzEgMTcgTCAzMSAxNiB6IiBpZD0icXItcGF0aCIgZmlsbD0iIzAwMDAwMCIgZmlsbC1vcGFjaXR5PSIxIiBmaWxsLXJ1bGU9Im5vbnplcm8iIHN0cm9rZT0ibm9uZSIvPjwvc3ZnPg==" style=""/>
<div>https://geek-cookbook.funkypenguin.co.nz/</div>
</a>
</div> <div class="wrapper"></div>
</article></body></html>
Reference in New Issue View Git Blame Copy Permalink