From d5a8837c95a90ed450b36196b0c942229cb43915 Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Tue, 20 Dec 2016 10:53:26 -0800 Subject: [PATCH 1/7] design-proposal: add self-hosted kubernetes proposal This had been a design doc on Google Docs but it is constantly referenced so move it to the Kubernetes repo. Old doc: https://docs.google.com/document/d/1VNp4CMjPPHevh2_JQGMl-hpz9JSLq3s7HlI87CTjl-8/edit# --- .../self-hosted-final-cluster.png | Bin 0 -> 22888 bytes .../self-hosted-kubernetes.md | 88 ++++++++++++++++++ .../design-proposals/self-hosted-layers.png | Bin 0 -> 17774 bytes .../self-hosted-moving-parts.png | Bin 0 -> 36201 bytes 4 files changed, 88 insertions(+) create mode 100644 contributors/design-proposals/self-hosted-final-cluster.png create mode 100644 contributors/design-proposals/self-hosted-kubernetes.md create mode 100644 contributors/design-proposals/self-hosted-layers.png create mode 100644 contributors/design-proposals/self-hosted-moving-parts.png diff --git a/contributors/design-proposals/self-hosted-final-cluster.png b/contributors/design-proposals/self-hosted-final-cluster.png new file mode 100644 index 0000000000000000000000000000000000000000..e5302b075abace095415e8e42d6244ee004b4b45 GIT binary patch literal 22888 zcmZU)1yojBv^ER~5=xgKNK1EjcXxM7cQ=THbSl!NG)PHGH%NC#iF7yo>&0{LcgOh8 z;0)p2d#}CL%x6Axz7a|alE_H-NKjBv$kI|`Do{|+)!=^{gy-Os?;YecCAFmkf9=h4UW{u zBluZGOxlfv5=_N~Bh(YDRf!BEQ>73mEUXXzA~&}hRjXsKTg?j1~%~?4LIn7zkr&gpc?{KG1CJ z%dDTrZ!RAnZTe0zsvDq$NT_;Gs~dKxvV>1bHK_)kLkVO4DeuEkaH1Y?MV}`g=q=*1 zey>}^<^TRraTF2tT+Hhg6iiqrIt8@QL*R5e_aeDkxQOJONc;K=r3;a{u+KwlIT_(6 z&1YxP)B+LJ%6Ec0sZy^IxXB*bg7Qduiwj?YTRfNAyi0DM zx69~&0?Os*T@synRK}l!x;nUGQrjkU{Am^cw)O7#8x&j4tw%AALpe{=Kou5R<|yK{?p>QT<(| zowl7xrtg?Orj?}mq(Ki+ZSkf=s`q{}A*`~sW#d3>g83Xv|5L+s)TF}Lr&4q_mWv=J zh_!2O{mzlO0c-KyBFCb?-m`M-nXuPgAJ>jOv74!v@ra_gS8I;1uUPJ!?kIdY{Sh7` zgx*@Amm&y+Y6S;YMktAJNi<0PdD~1Zhqn3QMoiL#DPB@fd{LNB#Fk{40tKB?>V#Z0 zft5NVNn~D%hGHtQJpN7Ma$>(3Vf8aF^mN5#AqI*p@)cs#I6+k`s<&8*0~nEM)nA;G z#$~*Oyc7#mkc;gnwcc^K@z$nzhutLINWXt;%Fvmz)?b?XHO@L|C&i0QHfbu=Emi;R zXIfoSADxTxJsa-ajG&An0!jNXdSQ#)jRK9#UZ0u^d@Fr_+;Lo+o^Wo4V;>_}WJ zikG3r?MAHQuXtY-y=sYOl);xdkU^BuP5ztwBe^GeDrqSRGwDUr>m;9~wqhprW_1;H z_pfYU`INhhKNT~m=;wF~oqzgk!~Ww_BEDvs(0PQ#cHtakE@6GmWSIv4$$`azC4kk?!GG^g|8!wHpuVzJs)ozt zxKHKh_=*$;LoAL|0^UfQ%tNoK*Q=7`hJ)Mi< zww6bo#}eqIz{I)V$%{al3wE2;?lX>k`*6Y z3G?tfW36MldnI}_dSmD>44N&E$1oNsu4TPb@fq|R&$d{m%&i9*=u-7ZwEcNdUgKhW z&>hfHrm6PhMOpS9MiFg#J6}2*9xr-!H^;+nhvtUjMvO#=M|4V$NH--)B>|y93arxRngO$^X>~?tKj#30kJ`if zOLY=H!#(WJY?fx;7{HfB8P3!hc=#>fm)KpHD^J8BX=mR*(YFxC zc@=Vq>UPa?oqN|xub_+a@ZmcC(X>}%x6E9(4fhDi9$N=11lb)w= zr0yi|)7|Q1>Mg6TDgUZe^jn$sTfguXy*hTqTw1j6 z*Pcs=u1Pzp_?~PsI<=i;B5_T*e{%cr5dDzuXK8Ku0zaom{bQbdzkI@0+E%&C!}h#R zhR(>tyNBIhr}Nvi3{(cQPVuhHhrc%p7G+J#I;Lk*6B$tJ=pB4qDQ*VtOQOo%^)s?T`x9420YH_*XTv!=#pLJtfu5DXb+!>o2ug%^y zskgb@^f9%Z!`mjj-vcdE=6`Q!)XvFO0Zt1l|6vnpyGrdCoM>+)tM3E@=*9{VT7>IB-}l&P;E5BaVX=X|mo4 z)CklEMBJR1m~?C2&5S$rtmw4D`v&l(3vg3!eBheU0H{kr4tACBFJ>__)EEymasQC# z#!jGx?>$@4V;8G{H;>Y`$`q2&)pIDAUsW#&k;jtUo)NdKxH#V1?YvFC}~ zi==yA98Am^Jnb9-w1tA=^W*`)+L^l=6MNd(+Pm<0 z@{>N@!2^DWe9TBn{B(<}4L_-tyb`gfgR?m?I|Ca76R7|aF)=ZpvzY~tikQTIkAuJX zNv&L69eEfTJv=-ZJXjeVoGlrdxw*L+nOGQESm?nW^e$fZuEw79_AX@q7Wtob#LQhx zovj^RtsU%%A?q5OIJmj;lafMS^uPc8+o!pw_5XX5y~}^c0taM-Tw!EpU}F5=wZWr& zkWYD(tUb+bwZyFL%Fk=gNOi@-adV z@P7{UPg+l(0&xi-@iG3d=mn7Q-|p~2K?y-giwUcFLhpZpPg51geUCbxf--KrBaViK zrW8l2QoxBGN7@~XB@8F*%^2*OY>-6La`~26T{tq3_-G=&j!LRGpjG=YA#5*UZ+T;x ze1#A7^Tsl@<#OZ3#!v<~=DJ~WXauZOAmqn4soh0@hO_%EGzKx`M?Nqz8WjdH4hk_Y zP!Thm^mJz}?}%Io2K=a*`9A;ep-)iQ>58aCVU$l#Ws|P6KHWjSIsseU=j(k%#rL!V z6gFPgf9qzw6%yZ?=#*TOetKsE+pP#`E-X=dl8~i z{Po$9vggyzg0$Qv9WVB@8$FM6A3WAN;C@=J>nt`nZT_7q!JyTkLqtJI2}Q(|x!9Yd zkWHbxVRu+=c0YWxN%pjX4{w8Bx%2jH{tSVyHXp>%Zg9*Y;PX^cDUd0^#xzhZQYaH# z_v?wq>vX$6SZZ=USmfsRI?2xv@Y9JQ5@ZNRB}x&Cz>uyo>E0OTTuFj~LkgwsMExh0 zR>gLl$i1!M&(-!zJld5;L3MSJxQqd0;i+z;9>0k$jc?(ud2S{UjKU&Er5Xh%}@=X9`h)4D+pA9E9^Rvo0TRXr~P8w#s0!zNkuEo7haFLlC-3#f5!+4?4_?>>kNNU zh-n=5({6c$S~`Iotcn#u$XhpAVQ5&H(ku349Yzi?Z&b~cU=CYv%k`Sv<$=b?W}EkD zzXG|;hg4e(e+n*x{%;o-6sz%H#l?PGjps)+UoQK23#R!s5jC1=F?L=Ac1Pjv7L&wp zZ|VOtds>9F^1g$`Ml$tpYVrQ}H`3tuvOkGm?L^Xzy@4as3J{Q zOlmBK$AhALE)VPsNvQE4_76)8J(0nmLa+N)GKSz-mg4<3%G^W8DUp_qpx9P|hDkFY;KPUK6k+dFOjbHpR4AS+syhB-15_*~Oj zzpOO8J2Yt`zc1V_g?Lj{DF6J(3gg4YBx&p9iuZ0=ppuD^ex;GHUk_f)7h}>VSENTr?gVh{{H41ddgv!NK@v?)fF*6q=>)2im;pij@ZEH5?U>S zKuaLr`JbBTgQ#d(#13a$eCkm)&FpYXtqB!~xD-DgzbacB)qvPkJ29HM_;Z_=nUO)C z9AS|aY9AhM51(N}-^xQ2T}O;YA?^;m{bx`Tg-lYVIAtkDuG4xCk4~%le$9V6i>IQl zPNvoT%cK^Bg66Y@gv&s;Q15{1inS*Q5v-O3u}tJk8&Papjqh?+yzJ4nEE&c<|7{Q% zx*i*EWcx?JV=K8-TswY*-82p99g(2FUL|(We{OOMyQpFI;j31T2;8CUyLBAdWEu+H zdi&a{P^^znW{?`V5c%?R7I)dJhoHcT!TDO7PQwk#e+E^7<|rF)JUR(BNDe&N0DVLg zKd2Mlgo!2oDeQS>1?ooHj%M;FP%6GTl-+qo-w}eo8~&f|E>oivPQ}PTbleejMW|HoJMXlkHK)sc-5p7P7*=3KLjF zB&IjH?&MN`;c^nu`EG@t;xYV*-T2I?#W(=>VfYI#=iZN^WwW8Q_>ni6!GqoqV<5(W zv8S$1V>XpgQ9pAaL31jQNrwH!g7*AHxaRLgMlBQIuBjlH?g(`wA? znDmM#ibRuf7|RwOuXixKzS*tVY&vX>YIS6o3@k(Z-wmS;HVum}-1X^vkDRD! z;6-uLBi|6UBmVZEjt?;^lSz4fuPXOut3(yC@n0Ev2|Mwpk7Nll!XaTHc;NMZBPQf^ zr_;mZGVvti*ys#L?d<=bw2?^(RNUlqSuRNtT>Vu!Pa;j+g6%o&pmuPAi<}l~h_w8$;=;8f7}0=w5`hVWfj`Bq8e*t^dU&n4njW-!|Uy=1Im*A6$a`R0`yI%31~zf=5m8XRta)qd&_U@>7nL5;vmgoc7XkcBBz84ag3VlTzcKBgFvh!~p; zB>v~mKCp{AWFf4b{fU$Y={RZx+FS7d?YOAfXeF#Bx4d z=HH(_fV;L`ffX?~4j`<9W%}Q60b&iQNzE}hVhdv#vaVj0Ntk<6eU25@)A3+}g_5xF z3URCE(xhQ`O*0pX6?*<<<9N&u38aIW{7o@&&<-s^3N|z4$sn-*T{nYCe|DGZ`91)XSDY^%QE= zw}fTO?ky+5LI^WqWe92}27$wGA!Y^qS-GxH$x=${gS#uj_j49lM*rLWdN*PdZ(4~H z*JB)rNIZjtlIU6bp4;ThrL$DK?x-++c$f9|twIJBk?#?-{=q&gc~U4?D-cm^kn~wS zug8HDNVDc!L+NjLNxK!+L((E4fy`}pdQO^dKm@WkXjx(%9K2$Z2b0H#`vPEUZv?~U z{wn`)+fDI3`z6U@G4!fL-1zpNTprOs)QMZfb-6B7nf7i(F}5->Trb%y)Y%mPXPk#Q zZyKo*h1|KC2b*gbk9Y5X*mCJQOXGQZZHmufn}$WB_CaQd2tL&hq|GC1m_lk6rB^(z z#dJE~r7jMZb|;iFvEIDSQp^&d-(RQ?IKdHwWO9&}PL&_>=ql8e|Q`| zK}>)_Ze0!k;q5D0=m~5*q0ND0Sw&`(Z@)BWh9qL&&`04iPN&SiU~DEvNP0OK{hv1& z!1CqxjfrbnXfg779%a8I;2JY%i4nu^Qo7qnR3vxbn`HqW*3v?(TW9-23?M>Rt#6+> z9VAqqIs9jmiGjUXcsxNz@?;XxaT(k$GKp{HVQ(w|Pp2^z)CO}PJ}G5S%l_}1*l@d8H!jMcW=2TmcCnye2k{tEJwUv!EN%xjx9SM>NkbCt zHjyv+#kfiUGTvcr&Sv|>>0m#HwFE$1zH#S!=d(Y`W$glYN1akjyQdrdm>g20{FfPg z-e*#)Xu({G|D~O2^rQL7+!}`8jw1k2TJIKu0IBj#jXL5rXL(I&p z!rB(~5j2I@jDWg~43cDkHU13%x=v=b9Qkuz7w~wmY0~u}CpK34Ge?mG>+btsz9=aAYD;wIV*w6vGJEnE_qiW#B~b-w zPoPqY_y??2ym)H1sF!Kk%bdzO+qvJ4;ST@GFh23X$RV?8E^hqK`Hjl+W|gHekiC!+ z3yI@EzcTn&!yh5T=#ia%bjnTo7w6E?vPq+$qVZDE8;Mju{%4$-WbI$YQLDn%;TMn5 z^*q;mKE8PuHuWzZjgg*?u?EVI1GgS4QXNfa?G-Jn)cNORI%aSDp95@AQH1LRWvQeT z7p`l14tWonC)l=^=a6LVn=-|^gO#?na9xvsSJD6{z{hvc=)9T0VK?A=3c*BRiDtf-EZ>ue=+~ShgJsp+ zko_DP0v#B z85->#;fm)+C=qq}G$!GL<(A!AjyW`lUP1WPF?-;t-ou6b1&aV(9uL2@_maEn(gqc<){{t*QbRA(Y?1+)Y!F!0IS4f4ThMup7{ke+5 zp-%u|enAL7@~QM{6$fT4-&3MhWrBf$K?;vTfR@29{LejB@xdO-olZQGD_13%e zAUhR;;h<<|XZeisvin~oa{LISXp_;L`J(&^xb%g0lWiQt7JWZL72(+Nn1F&{B8CV; zoa9R<&a25pibEEe#>^JCxPQf!40Ukj`0$F)vvT6LS?WnGXfnXI3l&8`oG1W11dHn* zC2s$_2Kj<~EyxCp_@Nc(bp)TI%Q&8H$@j;Tg=0}E2?3miXDqM`5m=DWTOhEWyeuYT z5jH^&0`BQ6jTm}@#zygKBMr;Cw8JQueAqGCHN zFsrvhXByZLV86krT6Jz0TuEKL5;eya1YXn^zu*qphzy@%46d0eM}TzsXRBV{0`&m;qTcd_Ny zZ~>6!?Cdvys9x^ry;LA*-jrrs|t zPtM;)0+iH-C*ux)Qy?r%BG6H6>@CO@J3s%dn@Sx6N6VEARE_$zZs}!B(C)R-htBP| z3PmVLM}&+7kun_Eq=ph_)o`odEmDSH>&;(UOdE4`hz3@9fyk~$w8W{XMKno%_d2xQ zzr|xQ`@lrx@FY<|48RtKD2LJdk7RH|QuE3E<;tya?Cev1cKibDLf_JSA%j{aKQ8DQ z{6Iz91C7VQVpUlN))TA3CkzxsS*)W}e_0qlg$Ptc$9KO_ z;j#5^E>{BXzgzA-0m)E^w`YbM`hEgL8KAFv8E-liAX{p$K~Re4kBUQG?rp6GS#U>w zT!2bI{7F*v2#z0z==ECT@iGDQyUH~iOp%{1*WZ0DqACN?G@x&E%#>j*w@rkFh35$a zmVUUgI4tw=M#(&73*vc5lzyG#M_a6v6HLJCE{8%SFvJyoLHJfi0z^Z3m_=7C`9OCD z0oz$7?N;B$FXH*O5GkVB!F=z_{TPS+pW#Ud+L&pKTP{GZ3BmjVeu!fi+NuoAA+?UC z@RUxDIDySQaO9$&6{(kKK#F;06BdcMkEu^0*#<|vUW^l$Uwx1KdH~Ry=D)illuH5& zAW^iv5cbcW@kG#ZsFwUEw)g_s)PV?0N^($EWm*NhJsIr`B8a*?&b#;oOh$p3I(y>< zQEnoY<`5Aur33Tn7uwg;cXX_DJy_JX7)m3-WizU`vKXKdt(**sN7{7V0Y?xZM$f@+s?IkwUH#uw$A-&>3emg}6!4Z8?ZN>Xsdy4Q+M%9VKwAnR;O?>EbHtgC=SxFK;MY2QIbp`xnX@R+)%@Rr{LYx;LwAzuau?eP8+% zk%u6$okLl$81>*vvs^F!;T~^u0rh>NJ~4C~@SLqn(LH~7Ck^dz6hboq&Yk~GQeV;< zAwGpWQs@J+Q&L;Qci#Y>PT+Jf=+EG;VT_R12DS$i2Ve{ofve_^Xs&4F*Bq`Cw5PLy z?S`5T15>9YCx6!Rn5;)0{FN^rJ=GX8u2}az9Y4R0Bih{iL~Phs(6&eOX0sX{$oVz$ zb8(WOVvMwTo1imqpYNe}csJc`-t!uGi@w2fe{MqLh#EG&zKch_RvuBczO8PlwmW82 zE`M`^?7&Jtu&K@D2wVVeL`$-<=<;wS6#!JRM!px`S?v$>lCF6~vF}KUko|ghVfM6L zge*{WENr4ica>(ddm#b9)at(Iw}9VGAK;!g>R0>|zgth-DE8V;C*FDx^<8YQuXGt4bgLWt3gKayr{y2v~_H!rhA>+%A_MPm0*ArGeYHc9qWh+g;e$v~e=GdvkN# z@|fs}xGvK{ew;O6smIm4jyhL=UD@>!S5ju0hK`z*&tVZ`k+4RcM9sD%4L4WoRyNmv$!}Z?^PZ>=QakSiXp*}upPdZeh*taU7+l-x zv3d#Ed79Tb9VmPu8ZXbN)?QL1{mDJfh5c|A|038uu~3FiOZC=kiDZYpqD(_hw#PoM zcB0<88#_OVU}NsbAIs~F&mOhaL422o{7Og+WqM7nqAN|`3|QOJ3@#kFzb)_Z)s9Q<$_=!u@R3wcYa-jqK>XqUkZAceVv1Ttqg}zo z#$~OD@R#iveHEM~_Oh;Qa;Jdy>d}0$VUc+!*j)#uz`k3Nc)IZhANd`&9ls#k%7N@O ze7Cal&&7*(ZuGlrUjnQsH1bUZc3AdB?->iLn^P2Li%*4(NsaNpuZLc4d>K(cw|pRh z)&x-4B@E+Ks>yvH*6Eet{h2BaD=I%~W0b`N4qgTuhG3QwZG=i~q}A^BMo!I-sk|n| zJ}-F2F2Rc|C+SaLTX6gOMYmm-&8RV@F$;X-jP8wO(4}?V2TX#K&u>SI&T2)>Pp!0% za$J`*zfYa_K1`?_FJUBbd5O#w6`~z8x)ow2T-Fkt_5&2Kwh zSuu#ybxbYqzqXF3E~iSOiY)!bq~46@lmDYeTKdH^B3Lis+4RJD6iJTPl-f$oxC^r= ztMJ^b{$oGB+MX%t{#w5sf!g;MO&Q9OR`;d*-ZE9ad(#r}MFyC~wLGE^)yR=yOOi`5 zy6_$LN63f`LI6y{NeyAhKsmiq%-|xKHSIBD2>Q(V5TBeFfF)PovhrxB&pln9J;adF zTuMk^c>oI^$8sL|*Sbgk)ZIF209%>~dne5ZPdue^BrLSHgylSnCOn?_=h<78x>S|{ zYV##K{kxjFY=PUl#=76$Q-ozj@y^ay z3BK04x$mVu?(kGS2i0sj$N(1@LI22+CIrU zD{W{IkQSD3zo^Ems;LtbJb0uc9c}@lEy%cUY!I^X(}A z&mXv0^mJ05YQOzSHfuk?ZlmWT56JMA^Wm;lVq~y>%>2aXvzI&IBU?`W0zDEBaeDKL zLC`83jj9ENv~YQ^zc24ZVdBv~%*hr=9h~AIib+AiJNA57>)=xlWrtn<7{S^*AWLEB zIDecv<*in&CcUw|bc;djE?3ZAE!)ZX&3fjVOSlRy2BXTPH(}d^zX)5Xj01ZktV3I5 zSEUnbw2Oeza{NvBt5iC#;Lhoc%y&wgv4oU$BZq%}rAd^z8Gk`iP0>2-;Q#*dyhdd6 zWPta}HyNeih8v>kQIp<}^r(}c@XM1+wY9#r8tRzv;fku{%NCXnrX`4#^_uskYdGT! zl=9<-B$6?^soz%OyCI(cA;VZ9tK9tD z$zs`~w}QDiO5xv}u)9rYemO+p_-Bj+kIgEMPPcAoF;e?Aq`YVVAbm*QXy}a$6m%G* z0}Lr-0XG(|^_1ONfD@(UT;9JAM}A#6O553J{#2zKVT0^g(swJ|6!ro)1ypU)fXdnU z-5upWM&=#_@B%L4*!?oDMJH8NU=i};j4+Yhbih=tV!E~g<~=>wM>k&UP$Er z)j0>+Tc6%?L-eJXZO4cvi)E-2-pA$}^~MiB?**%ky4FrkpCNjJl+J?MfgJixHD8JZ z^b~NqOwA(SKvPv?Cvpx>MR6{cBiuG9;RN~RC1iIgJuDu}j?%>pFU*#<{MmII9U6Q| zzKRSEvz$G|Aw^6Bn>4lFz}<7)qWq=CrXhS+;qcyLX@@^gWt}~^8TX;#j*F*d0JIDN zw42PGB3rfGUupHz1qa-=Y&r4wK;t$YqMUGpjpiZJ9D{Z%DxS!AkupNhuA4dm^OJPG zL_~^xS6z2WF}s4qU@aqu_Is`{)9L`dLF#XIgXznMAKt1K$OdmEFEb{USq;iE48PUW z<%M7f&WUPyx=5Ce3rC~F9~Y=FO6g-K6RD>4d>u!?@f8>EcX*Vapo}_YMf6Mt(x~`- zao7Xs$+y}y7RkFaqNr|Xhs)mO9@%Q3H&fiunxzhR$6hMhdjk4R%K$oV+5PrkGytr7 zfoM1i&+lG%B46T0m+gynAO#yPIDEtj_k+U8^n`GE?80FTe5W@PL0s{8))k*n_;L$z z;_a!U4j%5|PwGE8yULNIK28&%ueHuBxOZ)2_L8!{-Vb;REqHn9*&GRX-(aOobHOJd8HOU62qdQ-PVmqh=sSo2jk~3sRN>Y=eYF(XXRt z+QGrWW?<$5l!o3A3XbTk?^d5mz3Ic`J9P!xtn0k#Kj&phJBil93Pdf$0hx`3u|7`j zJZzLQK8D6{yar1KW#vfd@09jGK_R>N?j1jPT#P$t|Uv~oi0Q;jc9(Egnh1Cj15OS z{9y?CP;xSior5cy_$Td2=t*#wm#KDgFdNEX7ypK!(`q!iY$DrUlBM3|WtXYm9(RYD z*#!Zv%w`cRsj8X(ym{MS9PD)d@I)KP$8GUQlFdVz{0j~@oiQN7&c*_?d*9PlLH>!^ z^gWx^m=J_G)x0;c1QB*m6og*^)3wxu=G%iN2*14_GfH*s&)v0$g{Y+fE}oDaIY66q zM^fZTd`tlBQ-4>)z_XYqMo$5{uI=%m36Z@D3GyU@VjzyT z=bO9#h&xl@0lv2FcjAiy`aHR_hTenz;9Q;w;9QhD)%HcOpd&*9hKkT^Q=XauL4N># zh++^<@%Sv%5sGL&Q7F&id1O>SSpCFk@qv`HCrx^{=LF+*x|Iy(Oa4CNQp|kGzwvAR zV>N_eY@-0qT`y{L7<0bApou{t1DR8(s?SaWoO1gG*p02A*aNi@0M_QK?LjP-BLsd3 zMFAHHEs%cKb4X$M+`Mir3F^)T z{z$3SIoNkoWh7vR!_QrUs4s{f>yl_OwM4UgAS?AT>GtwK_J`}v#^hJk1X6J%=`4Yk zu2>>NlddD8k12fCGS}WiGXhLJK@_Ce+D0EaePm$@(LL|aXEsv}{o_GbN(PV~=gRz! zzrFTrXUt|RjWIxHgxa8)t0+dEY$TGDw4J^BL*LPl8AL$U4DH%g|13riWPe^lt6C^` z;|8eXYR_YnIWVnqA=T~5jHUfE7%uGdN?uP|fgYqp=Aqp>BMt%$^g+NCQ32-k+=#Z6 z=Lc+dQg-D)3Oy}oGQ{6r9`5o7K}c@t#JAR)p!-V)nKCmQ{-lmUE=3IKrF#{R<;rGl zl99H5?S|||6A^s-MwYRK926~E*5d{ghdQQ8BH%dFmV=w4S%zSsgz|{5y8(@2Uw9A)ZYO@E+rXv|CPHGVbgEmExDlWs zRY24EC*#e{`EHYXN;%8NTOCagiOIa zfB7oAj4MxohOMaB^Orox>Gm+D?xPsRdX3ctT#iD3?V`J4A2maUredPX1dXx3PJu9y zazS`15`}%ye8sr0*O>RMCS}7c#@1}q5FClL)rza4dlA|fkX561XDU+h+247V5^C&a z1m?778@&WSTO<=ex1Sfd`Pw zoxPM_n`pV5@wGqW{zM4ai47%6!@sq4qc7(7n< z4EL8#L!Fo8tI@!PS!tRrUcdnQ<=3(^>KnewFun&;*bD!U1l%gwbdLEbN$|@H6zsOM zN`tWV{(G9AIqdK`)fdbiZE4`K)8*mcn?ziE0-Mc>#$$cYB}WVe_q-qwd0_V0b2tml z@K2xZpYlGp8+--F74r?OU}ecmUioBj+KE4fz_2h!RqgaXWplh<%Q1rb(O})Mh@X0*iMPVI6jl?c&_vI5OTT4u{ifnOh`QHRA}(`veSF zO0{^OL#jwME5%@>%F&A~U4%+#V~?!2M{|~0hF7j1sH61p)9qr?@-Ec6tI9B0UgxVEKMdFC? zGwB^@$hanH_W1f-YC;eIz~@nGoXoov^{ zTE1TL@Lyjr$~LAz@8~R?gqC4~9M(fcq7fTyxlmLSZh2kpo``6KL~S{efl*u}O_qU$ z^@;=86!jG!`q!>Z|30)F*xQZv=Wk+Wp<<-o$~{0T3c=nf2cNL+yC#oAq~O8lDjlgz ztrC&jbSw zE73nb`<#Ak9?fXpB(Yw(tMy7*B>Rq;BC1j|8pCAfSL7!>H8bV)9u*vn_U}5N($5Ej zQNJskUR9xRnD)H9FYp1aGJg5@Rs?d&lu%*sk=L%616wSWtiX`j)|>CeWVQ0rG%O1* z;ozLY+l!U+Hev-I#?Z%?s4Xv>m5eHJjwqtC`>)2k*AiRN6b24#k*_P62`CLs^RB=! z${9X>hG=_cKPRFiQMyCS>jDkfBO81J>j$(d7{gphu-p!Y1WR3%G?}wnvnheIsv<_O24Fw6wZ3?E_= zh=pOwW|i#9c_ z6iBDntz(i+M;aDb-=IRoPA5Zdy_AOe5NY^Tg#rw31w*D)>-L^urB&g$?2J>~clUi9 zsw4Qul*1&z)r1^F3zV~t|MOvXz=`~bNep8nP<00jj zh~jxPu~_W6PQ=fNF9O%(=xLZ^voE9yW<60*C8O~=447p;NHR&5sL)hft(h7(p$Tnx z*t+k7-mSjaI7vJ)YU`cjJbRwgtc6uBIf8#E@t%4l7WJ9+0%#YTCdE<3v$dgNr(ery z=fS~NmjB8Tg@-iW2)LcQU=ZJMB35wTy^~}^5zFgMC0b^^hKIK8>rFFRfWNpU*~2c3 zNiPL7x&@(6oh=LW9`pd1I4}=)#9zxt%g6C)KQixiRC$Y9E&q+R!@#(c#{%X^G|af7 z^fdtu%kFHAc_L!_M?Wy+^Lt-O;fg(mk=ZwWO%@piCNAg~m)skp*BbC8FO4f+GD_Zk zFit`{M8(2Rzf7+YN5wd@NJ0e)wKn!f>>V^herfn_Iq7Ezm3HlGSfijzFagMWvKsyp z3qKQ4f%^wEGrsnYKM}Gg-8%=ED~zA5M9n6Id0;%KP)oYl0gRRy!|Qb94dZr7&-tAB zs`Bu`gF(R;yp6AMcydF*0Acv&Z~R>O-~0FwQq^@OSHZ~2$q4VERc|a9;(8%MNn#@E zih=dY-+Wz;iHcDysQtYKP1PTz^U>-L&x5V#Jq6HK@kfbKLb_SI%mhF?YKztUBL{Kd zP77%04T3)4=3(38GzN)|f4N@cVhW0(fKssIu_YTy{V|*?wC99K&Kf*gM|^=|R;IED z-;tP7ycZNr$@k25FaduiP!2lvhz}%3=!@9h^GatzlLM#W$bFxoBNafRA0 z{ZOxDenUw>EW}po^}fY9b!+1KYhR{C2d#ht)N6Gx7=%}j3&!GHw80w2c6ngd3Cza) zs;sSa#6Ee3C>A6HhFHF40d>85uYWhR~J{0Z~5PBS*8@l0W?Le6e)icUtp3 zuv?>9jF8S2h=6}<%ZAcf2SMkwlf38o)@+sO?r6Qk%CrrhmZoT+mc1ED=f=rvF0Meg zFeGd<(4u5WeQ_3j70(WdV5oxGtNCVpsWGEg*)EZ#kKg-9^E!P5Z;LF zEa-qBwz9<;c_lL%8uSi?*a&d2nzq2i;}>_ak=8$w%Opve^h8s0yKK2O*%IS2<%YK3 z^d@m!P?&`9`|JI54q-8k1fvJq8@`@1y@>7H&)%4ViJ1w0=0aTN`v%ZR{tB=aAcZ6q zPd#}_Kg8zw$=m_r;T@{-U7(S378Djf$K-nNBxv!S+jaoNb@m{6Y4f5K1I$pA9@jP3 zYm{ZRV}zVO&mA)alkqyW?+VBqiA$$I!>@UuTdV$5EKlQD-0&zQ8#0Bp+77MgI$XfR z-;V?1ndkuU6S+bLmoxZU1s<<^>GF#m3|Lf)6SkG&NCj~mb!WP59B5x?BPXBD0d?oA z`c4c%1DFUx%5q(|G|&?MvQX`H`rJ#v%eg;^BoU1B0a|YRIwAs~dVua3rDHPY&QtyN z=WEZMh%Nk>U-_mo>XJw%6`9{`((I%MUP}%x;V&J8bwKu_E zd*F3FK+&H`*3^6=gU)|010g~XoX6a{rL;(9o(kAzJ}}Ys4E|-8CD(V#dqfNq{) ztZZD5{Uxf2)~w>ZO-c21`A7dBHwjQuP-#wO9}0r?Uop16@p|)liJ& zQ0f+(;gv*9q~ru|2ll$XUPU#tOV{-MCeFSy?Zn*ky-(C+*y-tP+L~N2pg*MQa*Jsu zH2Q0;{It^)M#+NL!y)-s-jd~a_kA8b)_0{Z@s5Vqk**n{+n4W~Hk#5+)r+gmjc4@y zo>feLQ~xHN8uJWhy2B5Lma2x|r(S$#&QD|MVEOXycVH6+bDJSrpo=(|Ak6nE`sx3R zNYU+gaIq#P_H&CL9 z{L?5MQYwshK1J}J=p@%0eDRo?yg;4L8&}7xq+kCP5>Vjy)1({+an=`mP{ZSP?i`km zhDA}y{7Z0lOGRq26b1uYU-CM|T&+$03vMDy?PkviH7ecBW4Klq87KvQw|AUl%B&d{ zj?U=2KhBrm{oO!el6yiyu^l%7o!JG-GhI{#4XMduB>l0 z_KZ)=lV(S^6e2MfZhg8I<3|;|Fe&6{I31QL!UcLfe} zg935+=g+BCxH0{1Fb5gG2?e3&ekf?DodKMzrX1hMYBKaRNJ4JJ3#z=j*BD6u5fUhX zefF?oX8boVvyY&K={Vy4iV6+!iW9oGc;5n!#yxhnjmGEYWh4_PVuKXmDWkH%5Swpu zd#0hgRO*}kf_@*jv)!N&pS-DLL4~^m_iM)R^Y~qonT_ohUh@vK%TA}pCE^#bSb#za z#J6|n7(*D+WfJ9av!kVmN>yGfeO`izV352;G@Td{ej$!0Ud!Xqpzl8Qso-Q(uzoV0 zylA7>?)4(WC>X221D!w#iSSFQ_HaZYrgDhrLpFw0g)U(w;}%;yB)pe)d{DPg7%m6LW88TiqRh3LmgFmD34c>B zFI#w6*-1Q^F(j$JoZ(Mk7PX+(|K^+TcJvmETtL1s=P$FG>emFOF}6FC(tZuUKmR;< zJG}5WHp|1BEGkJQnDpl^?mHw5>-!${lV82^ldh}k_l?FA_|BUHSvM`~SqCiwi4K%6 zQ(d-4-$j)i@H+hUrA{nTNm=c!cD?rP-CJ88&Ak2c5^q|w9=UsNue1K#uWw1E*6;U* zQ={jONCV5A5X=aq(GtYLA}f319F=xb?|TtMu!qz5wNKjAq~>dPN)OFPAA)_C z3u@I=&v|6YBv6yW1WPB(Z8Usz-%Zg{Hn~6M*KGNdRGg`(=E#w+A^}^|q+&)zyds8{ z8wB4D+oJAYb9K7E!8YgCS1uy7MCE0|fObE=9!MVi5xp+_h1s;*_uHFwn50+ZRqK-E zs_*G~%=^D8!1S`^)PzpFCy~svU?*D9i1|3Bft*nPx^Dr3t^sKedIy*Du1?(wJ?k_ZA*&e5cEIRU=U%8I^Op zjJ=k;5(V_%JE{g1DP)aJr_}Id7o(-v91N+S_*QT53@Y;aCoMLf3=|U0@X(qp*u;z$ zuh-zkxBv39(5F6>$^5W!+Pw?LGrN^fMbq0pGYfy4qNevaTt=R0s!ooSY83SAcN*Q*VMRQKecfQa+t{zcYCQBrFtkF)julX71ahb}!`3I|uj&d=8BB zAGJRk4k$jzxcl?Fc9zq&nf-t5ocBB1Z`l4(t!jyu+BHhl-XEK$rS>dE5K_A}OKUVV zYPYtkm0GP$>?(>7u}AGuvuLO-i5csOZ@>Sao@*vANO^h=j+N$jcmV& zk`l_U7K%lIaZ&rluh&A7y)Aem0b8kgJs;P61i^+{%Pr}GCIxSW8TfP6dp{R46 z#mT)=T%m6)aa!<7_Q^h5fii8KRyf`?Z%dOSmh>zvD8kpQ!o{uX+&^8@`M|niug@M- zB-#v7AP8JFgQl*Xs|O_?Dvh6mKNN;=52jUvx%5Mo<3`2;!b>b$`X zQQ84e5x+t;b_7JvT@K^f<=;%NFN7DNZ$4dkC3OXNq&OeG@X$l~vp_tIN7u;i=)xAb zI*8mDYuCQvsShkTu1ce7ha-E*A@S1^-sV3 zE4>JcGH)3*f9D@s4jU_O7%Gt*>axYj*K^acd7JE>>l;&*p9P-KyK6slFMM6>Z95M? zY76lRZSrq%IEG*moIu(QnIv~pt(A}k_(9I07lwTUEOzZ8QFdNR?w>)XbZ(x_DaYa-Uo*bR^P}L zvF})|S=Th%X2xrc0DE2&rj~~44$W=9LhAhu?i(CiT4jXaK+&(r6L!&R*fj-JVKhar zLVmENXek#3r!JKuBOn=q=BUbp6{XnWMhzY}ZkVuzd zxtAD>ca~iJnJqS6iprPWLVcM8XU!s`p-G(}fv=^LV!E9%>Aa6a1i$h1t*nWUt?a;o zTS`q8%^SO1%{%YJh#+|5M5g0jo!TBs;OT$+C%9hnrIIEK6vEe{i#ak8#R}3J$qzTJ z0T<|V@e4%58V{*6nK1CO#ayal@1A@}x{ih8bDbbiv`w zBWktdriFEmE#*)y*`q`fySd1-sL*~u>1MQ07Zl%bC9ekcEX*1OOWLnJn0VUsHtU)Iho2s zy&9A)#yg2RK8YwdVZ#P)B}es7YV;5O2|qbtlA$#MaWx$FBes0UX*2 zLN*2}{NB(CQdu%G+;p98a?E^?4_i&)g_YbG1y}YJj$#62^8|Zd@Yqsyo=9$a8TdL*5TS zg1@P=$vnTeI!>4UI2#c`DuAF@)WBW=GE{~pJxIv{; zG*z1a?Fyj4+>6$1(r!N&i9N~wTWkgHtXzWVPqDv0uq(N+G78Lp8kQQalg>&#D1vnL z8^lQW&R~;GqJ3FEkR@My~Pd?ew*lG}p=NbJF4BMcF~W1f31C zN3iy2Yis=h?vB+*sR%XhvtWDOFBBDFpIy$r1}JgpR(p3KT$;rlSpdo=AL#F==(>dZ zogU#Z<;xAhp$z4`i;+)aE+#tXjX;QmXE-Gf&F-R!kFgGV(rnq;fgKU}QD=jaf$cA@ z+5XgC7gA(oG%c6209I}`pZX(MK_jR4o+KV)1c)!Wm_)L23ytrJru)s8l_w|E>@%4E zh=P9id&*K!aV>t_M49XR3#9drsvahOVRlzl_8d$_g~Nf zTPA<15Z%>CBRwsnHo^J@?V9WDph~w})1^LrgM0CkdO>PE5aT$esrPZ+!0Y4@3D6df z2iztoE@2;6nb%P_7SX$yqE}t>ihr1UV^4{?XMwh&EE`Uhn?s#wwU%{Qr9K6yJSQXP zG-Hdk9f7?fGb53+^K#sSrkQbh?Be{TP^dUGkB5LACV2g-V)C^)(+sl@w=c0O{Gx{- zmSsh~8Cf2ZbSVG{}Ni1)!^C@9}OwJqU5~K9ilGB z2w%fesDIOB!v_vE#R4+->JWgpIR&vRMa8{6%c_M;nLS^GSGZdEc8?2H_CHX(3?D4a zkT^Gra=i$KI!w18!`LeQt0waY_y_VxEuLcv%nTt51F16Y!rINk>BfGrWh7J{?Ef8B zQxP8i?x@MAh|GjEz$^eo7t*QcG_~^*G9O4S>y|)^wtoI<_d$pU#-CF%+@i(Hly%pU zPLyE+UbRor)13=CI?86z!CZJEXWJBlaZtJcGyy-z%@ zzrH8V3%z^TbR<<~lV1+N-wdC(YJk}5FyCn0X%$;fJy&n$yb{R_WUfTMD zufT567k;?S(L=CtH4dR*3J?qh~auMTT7?oh^?xN6hae7&x{S2A%Zc6`x0O@@{6eqdc z52>@wpYME!e>vOX!n4A*My?^ z#OEWgCP4d_l;^sSHnRC~N>2a!e?{ElMWM1)aU?V@T_T^7?{8X+4Cg29FS6w9!{FWZ z`upkfUVq#bCwdNnyFgOg_dq)=TbsM}I`_2N#RCks*~$G+c!s!;>spC7J6%P>-NaUM zs)^a5M+-X3e)BhPriUqq-(VU)+4flrR2M)Q6t}nyx z0{J;P{?oO}9vSuYcjU6dX6wrfIRc90qvhw`bgXwM&VICgv6%2v#x+%UE+QGITi}N8 zL(REtVCsJ8r~z`}sE?!rDe9i(>pTViR-<&%v%Y&@v<>(O$Bk~Dsh>1ge-s>WsDgb- zKIw*pc0855SSo4l##{^Eu2D_zn1sfGVh+B-2#4Pf*sWv^L7zRF!7D13% zkOrD?E0c#Ewt?=`?=q3J?s=X8-^*~^J4$6$(jW1HEsCoir2rwk@7|hi=P7xt_!W8t z^#`mP{EcuoC$4OR>x~ORs|}?Fh)TsLi54mp@aglii=U_X(e3nkQQqE&QpN=o2U{{` z{4_XY2b~+R6vuP2QIYVl6YIMWOE{Yqdu~T+LK$qUOS&kRy5jkR7EU)vQcrXO+3eJ- zRbTPWXH7?s6cKO2lJOcW!D=mANfcK(1Hch;?|f?Kk0vw`IKD@jsav2krR;qz@s8A^-io^K3n?>ETrkQV5B4FjD!0bovlUZdSAIQg&(2O%y1+Iloa7K&2B^(4l z!l|PIqR6u5pzt|Ck`5iPo>eDKatJ>Azx&2~=MIpo&Q%y1yPgZ{ud&LcRa&}z@Rv}$ zx3e(#YfGyl*t79q%i;HsjS$zdVyMs3m?W!;?|T{+iPp=nu`H4hBF*2tWpxdNxh-Ip=1WFHp_62zahsZi7V+g9c2ElW$MxcH7TD>*E|qb$#MnP^#h_@kHUJFI&SjWwuJ%wtOM9d!J37)AXCM| z=eY)y(33IkU*+RV*$um!zMCUDY%(4>S#ux1|8?lM5&ES{e`7I~QsEl}N@O8{ebQl$nB1$gC6-=YkVMVUsZENa@KUk#%#pB>?itU?WQAAGy?KK6 zGLzq4jA*+t`U>Al!fnP8oe~0v^K-{a7@M6Z^&%w4Dwu!jrfGljQQ#vxTwNP2>_d!i z{Hb6M(pF~lmo1Mf=M-4u!irt&4y)~VAhcY@koP^C_fr5!SF;QBVdIO1K7`E1ACe)I za)i6HKz59NsmmewYUkYA{nD>6SeG;EY6t4OFj-m;lG3D(2S|nL7o$!MZwLiIowUd5 zXLK>Ehbb^$u3~4o+gWq#<9XXND-Rl-Xh_Bx>m3WKe +Last Updated: 2016-12-20 + +## Motivations + +> Running in our components in pods would solve many problems, which we'll otherwise need to implement other, less portable, more brittle solutions to, and doesn't require much that we don't need to do for other reasons. Full self-hosting is the eventual goal. + +- Brian Grant ([ref](https://github.com/kubernetes/kubernetes/issues/4090#issuecomment-74890508)) + +### What is self-hosted? + +Self-hosted Kubernetes runs all required and optional components of a Kubernetes cluster on top of Kubernetes itself. + +The advantages of a self-hosted Kubernetes cluster are: + +1. **Small Dependencies:** self-hosted should reduce the number of components required, on host, for a Kubernetes cluster to be deployed to a Kubelet (ideally running in a container). This should greatly simplify the perceived complexity of Kubernetes installation. +2. **Deployment consistency:** self-hosted reduces the number of files that are written to disk or managed via configuration management or manual installation via SSH. Our hope is to reduce the number of moving parts relying on the host OS to make deployments consistent in all environments. +3. **Introspection:** internal components can be debugged and inspected by users using existing Kubernetes APIs like `kubectl logs` +4. **Cluster Upgrades:** Related to introspection the components of a Kubernetes cluster are now subject to control via Kubernetes APIs. Upgrades of Kubelet's are possible via new daemon sets, API servers can be upgraded using daemon sets and potentially deployments in the future, and flags of add-ons can be changed by updating deployments, etc. (An example script is in progress.) + +However, there is a spectrum of ways that a cluster can be self-hosted. To do this we are going to divide the Kubernetes cluster into a variety of layers beginning with the Kubelet (level 0) and going up to the add-ons (Level 4). A cluster can self-host all of these levels 0-4 or only partially self-host. + +![](self-hosted-layers.png) + +For example, a 0-4 self-hosted cluster means that the kubelet is a daemon set, the API server runs as a pod and is exposed as a service, and so on. While a 1-4 self-hosted cluster would have a system installed Kubelet. + +## Practical Implementation Overview + +This document outlines the current implementation of "self-hosted Kubernetes" installation and upgrade of Kubernetes clusters based on the work that the teams at CoreOS and Google have been doing. The work is motivated by the upstream "Self-hosted Proposal". + +The entire system is working today and is used by Bootkube, a Kubernetes Incubator project, and all Tectonic clusters created since July 2016. This document outlines the implementation, not the experience. The experience goal is that users not know all of these details and instead get a working Kubernetes cluster out the other end that can be upgraded using the Kubernetes APIs. + +The target audience of this document are others, like [kubeadm](https://github.com/kubernetes/kubernetes/pull/38407), thinking about and building the way forward for install and upgrade of Kubernetes. If you want a higher level demonstration of "Self-Hosted" and the value see this [video and blog](https://coreos.com/blog/self-hosted-kubernetes.html). + + +### Bootkube + +Today, the first component of the installation of a self-hosted cluster is [`bootkube`](https://github.com/kubernetes-incubator/bootkube). A kubelet connects to the temporary Kubernetes API server provided by bootkube and is told to deploy the required Kubernetes components, as pods. This diagram shows all of the moving parts: + +![](self-hosted-moving-parts.png) + +At the end of this process the bootkube can be shut down and the system kubelet will coordinate, through a POSIX lock, to let the self-hosted kubelet take over lifecycle and management of the control plane components. The final cluster state looks like this: + +![](self-hosted-final-cluster.png) + + +### Bootkube Challenges + +This process has a number of moving parts. Most notably the hand off of control from the "host system" to the Kubernetes self-hosted system. And things can go wrong: + +1) The self-hosted Kubelet is in a precarious position as there is no one around to restart the process if it crashes. The high level is that the system init system will watch for the Kubelet POSIX lock and start the system Kubelet if the lock is missing. Once the system Kubelet starts it will launch the self-hosted Kubelet. + +2) Recovering from reboots of single-master installations is a challenge as the Kubelet won't have an API server to talk to to restart the self-hosted components. We are solving this today with "[user space checkpointing](https://github.com/kubernetes-incubator/bootkube/tree/master/cmd/checkpoint#checkpoint)" container in the Kubelet pod that will periodically check the pod manifests and persist them to the static pod manifest directory. Longer term we would like for the kubelet to be able to checkpoint itself without external code. + +## Long Term Goals + +Ideally bootkube disappears over time and is replaced by a [Kubelet pod API](https://github.com/kubernetes/kubernetes/issues/28138). The write API would enable an external installation program to setup the control plane of a self-hosted Kubernetes cluster without requiring an existing API server. + +[Checkpointing](https://github.com/kubernetes/kubernetes/issues/489) is also required to make for a reliable system that can survive a number of normal operations like full down scenarios of the control plane. Today, we can sufficiently do checkpointing external of the Kubelet process, but checkpointing inside of the Kubelet would be ideal. + +A simple updater can take care of helping users update from v1.3.0 to v1.3.1, etc over time. + +### Self-hosted Cluster Upgrades + +#### Kubelet upgrades + +The kubelet could be upgraded in a very similar process to that outlined in the self-hosted proposal. + +However, because of the challenges around the self-hosted Kubelet (see above) Tectonic has implemented an alternative scheme that side-steps the self-hosted Kubelet challenges. First, a kubelet system service is launched that uses the [chrooted kubelet](https://github.com/kubernetes/community/pull/131) implemented by the [kubelet-wrapper](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html) then when an update is required a daemonset updates the kubelet-wrapper configuration based on version annotations and kills the kubelet PID triggering a restart. + +#### API Server, Scheduler, and Controller Manager + +Upgrading these components is fairly straightforward. They are stateless, easily run in containers, and can be modeled as pods and services. Upgrades are simply a matter of deploying new versions, health checking them, and changing the service label selectors. + +#### etcd self-hosted + +As the primary data store of Kubernetes etcd plays an important role. Today, etcd does not run on top of the self-hosted cluster. However, progress is being made with the introduction of the [etcd Operator](https://coreos.com/blog/introducing-the-etcd-operator.html) and integration into [bootkube](https://github.com/kubernetes-incubator/bootkube/blob/848cf581451425293031647b5754b528ec5bf2a0/cmd/bootkube/start.go#L37). + +### Conclusions + +Kubernetes self-hosted is working today. Bootkube is an implementation of the "temporary control plane" and this entire process has been used by [`bootkube`](https://github.com/kubernetes-incubator/bootkube) users and Tectonic since the Kubernetes v1.4 release. We are excited to give users a simpler installation flow and sustainable cluster lifecycle upgrade/management. + +## Known Issues + +- [Health check endpoints for components don't work correctly](https://github.com/kubernetes-incubator/bootkube/issues/64#issuecomment-228144345) +- [kubeadm doesn't do self-hosted yet](https://github.com/kubernetes/kubernetes/pull/38407) diff --git a/contributors/design-proposals/self-hosted-layers.png b/contributors/design-proposals/self-hosted-layers.png new file mode 100644 index 0000000000000000000000000000000000000000..1dc3e06a847403b8e1bb8e26f0ff305cb24ab42b GIT binary patch literal 17774 zcmeIZbySp5xHk%j5(+4(Akv^BB`KZKp(rtwbaxEhDGDgvB`w`CpdcXK3KEL07_MR{mC22fda$GbtG(1@u$@gez=-a@5 zUF>_no1awilxS$UrREY6DzXw1)G7|PCgxVgXlOEFv2j=*R5i&vw>{V(gM{KCgUQsX zXv3Dz{hi7l5z0NM!Sw&E^W^F8a%$RkTzvoMe59fs{u*U?7;@!GQDR~`SogECDhM=M z`w=jBrC$FrV%xKR8`TV+g{+*TXB;>NwsL9U`#Oo$JRgMyJfS4&`6G{p^Zd?h6$}l0 zUisB$goN0G36v;uDi3rn?itSH;j?2D%JT0%VMQI9D9!WE{ffG^=jmemwCc~h@1cni zt(N>HRkDB4?eu7py1O%v&pcQwkIy@JTX_J7;84=-F&ait+am^a(M#X)#5dFQA40^X zVdB3Q?yDS$!-A4~=Q2}5_L>e3zPu0)t9W-VvPOR)#i?7^HAgS&wycvI@UMMqQrBi;9y`hNW6`(j#hPvGdk{n#=pSE6R=)6Kn=?^_pqrmAwnD zcY7=NxLdvA-elE#oWeGDG)avEvQ{a22WmZY*d&iquHaM*usGt{=Q{lD&l-^>tt3Lo2V5YE`^>+N*l0$ zlh&4+784Y=qM2d9f5arSM=$Z6^F>OG_@vBBhOy|9Z_lDrW^Wo1aG zpQR><5>X|7&Pc4>P53#XBGn;gSk6tN3e1(?|6y;QY%XmEHIs093Xkf~%^SSG z#6jYyXoS>~Y8)|`Hpnp=HVFGMlCdut|D$JEag=4;dW3RpWcdEDRhBCljF-ixx5FAIzO*ADi~RR^pzq zoi@ivo=Bok{*W-u9i+IWaGHx$Y+Tk~dDBbOTa*Gdeaqp<+rv@BNx^QKfR$wX+V-^% zr=G3%=DE)P)QV4SS+z_hpTX|m_rHf{9iolHmjsxJm`gqejd@uXR!1JXZFyTW>cHHy1XKVaMgGKOc_sjN@Vb3z@^KF1US0eS5NR(sYEZTgbQlBZBPsHshZjbN=s zEg7N%@dD8apLNx64HfKn|Kh>#UUF!4DBk1=!ENGiB59&)>Vc%6t)9@Ij-Gxwn>;Hy zvOa7(yL+;=w_#{mR=){4%5Q<7K&T@07~eAAJzrBlM!&3IanyW&Ma;m13qfc1@7_=L z7h@N21@A;ZY>VoR(j%KA4SH3maUeMQ+DXyO<1ves+eCR;Fd+1^JW$$E&R*S({CulPPL6Pc6mTEgRmMa){#*@6>F+ z%Y-x6Om)lD^x7b7xH@CQpw{wu$wT+JWY%=H1zuT}SHzoD+hQx5{v#bKa!$SVdn)7& z0n;;U*Dqe$JCX?5F4e=l*cJ-Z40U<5_TwG$y8Tz1uxv5Jzao)}%M<>XTJrl;-(t^_q>A`nrpgr4X%#q6)Qyt%ET&H!Bw_JFPG- zH8r)MgOLgUd&#%A!+|FuS~DjnJAO7cFc{1V=47>XFlFO-^X3g3`)jt>uUUW|ERJqA zPWrAaHjZ>Rll<>_B#j*n9n9^V%x!I`ujbV^uyuA4qNTkO^dJ9j^fY!g|4&Iaj<;q3 z2C`kXuyL@mv;D{1z)-=fxBM#RuEtgxlIGUNHjaP{VXimag4g~3Pc8o`@xP4J{Lhh` z-0c5#&U^`sRuIE!u8LIc|2{O=t8! z$=g?hfSws0KtFCZePXhkiF{r6DX%9)#~yu*{-?3y<(GMJU^?L4L)+#%x5CKjd|ja$ zLGQG$Mfsx5_knNvP^%bXKHMfRV*h;qMvc{UM*jlUJdmTKW0GcW67=skI_t?QbK@nq zgFj(42ev-U%=msIOl55=BDSXKKHg_fqUo;8Va5LOa)uD`gww?7-51Z4BtbU^7#Z!{ z<@EWfaVMR41Ez_{v3A3g?-W;^2{BiB*wC@qZnP04#i5?j{7KS{d85K}F|=LQ|H-Wf z0|uFApj_S=*EW;>ltlWBF~8RQs;HjRG*8`uXCd`pMM2E!ie5!QNs-5}a}iKj1q2DyN;3(;fTY zgxTr2EycRPouRPr4Dy=-v#0pZI@y@II}RH(3lMS^k@jGqzX~2ky`|f%%ak>b-9ORf z*T!*eE4FE*h_UdM2L$mwS}!ax=37Tx$2 z4qs7%&sAt5N}OvW8g~)T9P$L|M^hE&eSpSk*+qsE5pamu)mI}5#T*^%u0GtRVzpoD zj;(L*Z`Ph35Ul9PjCABJ=?%o+*+Di$KUa1<&KLNqJiDgEvg2cy7iJPLEt|2@xM2lG zhT*|e|0yX)%gGa9Shy82_3662l9*d#0TZS~uz z)k66TZt%J|RBa`6VZl z{ajkB(OKHx`-32?Ve4s4G;hdNGFgxFC6}(RxHy^`7@ch1X}!NSEYGGGsJ&8#z}Z~T zCA6#dEw#mCM2|C1(ZcYG#BV4|^j3-n05~p_^MNc)MnXvgy1SE-rbqNIEMe> zVw_n-d<5@Jjl!#oOHR7>@I>+|K3ylSsJ6Icm}$DT<8p8Lh>`$HfnN2W0ik2<^#&}A zg=sy1WJV(nq6Q2OZ^Qznp3`@yDz-jdlM6MJ*D@Tl(AOJT_cU~uiK^=)ZRqbJUSUG8 zj`=00JtuI^JBVop?JC!g7}ejO*1t9@Rn&fgk?d6!*JftOvkK&Pi-@`32A<_QPp2A# zT~qpSa!-~Q96Sorok|Vdw*p&i6;kLF8-^js5k?-3Q|+mk7ouPbeamx!?`pg!iHOmc z>PhPu@pe7Lh^Jse<3!NZtYcmo`F7l=POKme{8Q(#)a^Wu>u{B(2G19hk1ke)5iyPH z7{_`1PHk5)Aiw+3++2@k&+MIw^Q(8U(Xt6k{K=p}!u9^#a=S2(^0n!Nc_Jy~OH=zd zpPMFvu0FST%md`->j=GilDNw2g|9y2-XztBR~Ul+{|<(THl5;MK^hP)7GKcjbqq<~ z+TX%lDDUizKE9$~ABi?u{%n2DdZQHD?y|rw6y|eago3|*eB6Fz7Vr)LMy&rUz<8is z*~F~&v#cwcDKF8u&<&5AD}VER{E5Igf}AsV(^XO?{?}Uy>lmBh#e0Aj8JK>-{#rOb zO}BGD{QDTap^sCIh8XKwcj`eR08p94Pb6+^c@2PazebSC(_7e520WaGJx6e35HR?U zfFX&v{q1Z*fO#IYfg(57QD4FAmyg)(em5o^NTTf`!X92;hdG+x7ohWhCzyH+oD6MC z)}SCK?=GBo|MZ?T>H!vOtqrH=3t~Kshxzsx1!SrVR2a^!ajSmqhn=X0ta1foyUHpd7rX~ zHRtlc084a}_Jq;pDGS@jE@x3{GR(*O{ZiZxkFIrAxw789(7J@=+E)$$X8(_DbYPjO zG##jM+g~12_41~Mf>tF;2wbNdkq^`JuM^8RQJZ$;ay^#I0t< z$(71MpnCxU4>^&%{(d7d9hhdA)k~^+SQ~pI1o!^B{~RdPsRTiXNK75RPzg@2t9uFu zkJVI4we+vNxk6YpzcIQffn7Cu1EgD4H73JuGzi4iSuh{s2ahXMMkmBIS z*AX8ho#Es5yOQcGlrBOB_l@QF&rPzLdvdOs@#QeMPh;{w1QZeBa$Gd-rN|d8M8@M1 zY>QRzF$(_#j&Jao1h5^+4A)sed;;&ot=DeohuzL_BC=J135fw=CQ+U=z z9kYFWfivZn=IlE%B+stohP{Hkf8-6syJ0%;F6+!J5j8MjG}hi5d|==A$_EBlhF8~x z1K=hjUnK(I`kG#YM_+13HQ3=eqcLHep&3;o;G8W#ah=#7K93YFX>ZIFqioX9!%mt1 z)>$tw1`kGzda=&Uz*4PZ8gmde4tWfQWB?9v;ao5xOd14w;gVoEc^12{BM_qBK99rJ zqKoF*f^O=lDVzP>h0ZA*Fx?8JuejFyK|l9gdYBNu5}}TM)2O`wG2jpyK`$= z6O&)3f%QZn-49d`R9~k|QAI)^#aB!|=eq7Z$_b>?G|P0$kn7M*q6hXQM%C}kuG4A0 zDUkDppu$l6YdPGqzy_$xB75*A*$iAIn-Z@Q${SGm4oHa}dz*O;Dj8{j3WwTAZGjsp z^net`qOamNDhLsYHf_`&)w|qCkp-kEs!;^pd_C@rfC`RVj$1cUq5&yMKY4KRu2ncx z0906+T$#L)@_#f`6jjC6tn&Ovs`2uEfXs4HD8|*c>o2k2hxW#toc3IW#P2WAVqYm3 z8XCg7s=qunR}Cg$6L|J$m|>95$?QLOLgY;h3T>sB)c+fWH0gdNfX6KY_iho?o!Zy+ z7!BL@7Iek|G}J75)P1uY4_E!r=HYj!n{MG%6WHgO7QcN?aJxEKfVl*|JJ;5tV;g-% zn@_DK&lJ2i7tQY(FqaToE5!}kMR5b2|2x4y&TRJGU5P9uBqTJQwFwluF}~0JS{kYN{b;xJ&ScqlyO zkM#31$xCIu0thuFkxcaxe0ti7)1>z=3j)CJcrWWdHX##E{6sw$LE0bkpY|Ya!+lZH zSKt~$4R{EL&-Iq-RLB6(V^FNZcNRNt<>@JI?stWm*VwmR?>ZDsv-~5ep&XJ z46t!%oqBJNP%~b~#)mjj#2;iqURuRg?d%K>+(~uHAe_bOV0DPc<7l({eNpzP5@g2e z0Nh8~tSe2{mvbH6e14Zt*z{oS0=s{RtP#`oJYHu`6oRg0sV*6Ofru(=)Ebh70&KU;_e}3)i$G(j_9=%ifdwZ_Ci_3 z!TfVfE+D1Gykv{nV0=fB!_@ z1ZAHdqGiJVY0!OxMd>@%@RMv(9VercItUHdC_*^)dJx>n99>r5JHM`-AJ|mp^z=r6?VB%NR5DzC+O6N66BBI} zx$x=jM|!%3P8@oYA=D-6efCwJL(}`QPnlDmp~bm-a-%*9Usn+z4_D)%_hap-?0%uw zxb7MVTpWyS)bx0^H;M@|WF4PJ-$&Gne-$|w?@U51>zeo+^?w$*lsDr}Ut*c+Kl-m>zrrHp4bX?ZRSt~hJ?7Lv0R4@(O;b& zEfUfoWHS;`*@iFBtMsNlIoR1^eNy;S3ak`m<}#R*8Ls)9zNQ_Biq@f;(a#G zTObx7o~L_1+F9l<$7`F7Mb764Hj0%5v`hauBDX_WO{z=Ag1lCjGOM+p+O>BdWj~ME zGHsoCyp69O4lxw}Ji8Y2)$6 z-{(Li^dAh=Z1?z-f%grsB60njD59ZHS?KUXwB2@D8@|h+7#2g%A4D_KMyuRFN@%tw)T1KM85N%kq1+{6!%1<@s2EN#}V8th34TTxq-4 zb+h+h(awUGoL@uJSh4qBUhk!`k1Yq1uMu=&K?4R3pq_PSmD?$P2xl?RVT!g(KYdaO zsl_MSkjo)4*uepymKufWFt(>#VHBq0eY!M0IQS`2a4%Fbg|EE$Ub+XOz9GfBqy&M^ zTDZQPwx1Fy?Ll@h$@-6Q+F+!v67-8)TJ5Bgz~b#tNSK$&=!Tq~4o4ZmAKC7H&oSQfcddZl1(OY*^M(7zfA6c5FHl0RsC9T)5@732sYFXZ7)4UyUa=68hYS^CB)^RFxB#tk59$#1zL zqi?7-ftNEaKOq7Q-_4^*Z-65NAYYUvKW^&^`Q!lIvM`f1;+*6vX?O$Usv@I=7QV$b>USJoE$`W%U(YG ze|#gSeWH`&6IF3i4u{c22BVce%^ z-HnQ)D&71NI>?0am5ygs)syy$kml3--Lu6R*(subH|&2!gVC`+B%&G4e7vWyck2{6 zfUz{@h|$@B1AqHuo}aXvIMuQ zVGOKBEPAxy?P?4It070kAauJLu>f)Y?*vCpHui1IvHZlt!!uS_7qWXnA@`O}rR7O+ z0xofQLowQ)2TViBXMH>xLDWZ|Di@K z+C1bXQU5Iqe+{_3-|0|=@22;!Lco(B6YcKZQfG6(&$&e*t+$j^jltKov^$XVc10C{ z73I_sBF4YzUG@_FkH<&Gl-p8}Q4+w)*VZ%bZhLyB4rQ?xSAl!t9)dw9S|b3ahY6g9^Faa3v$`+~Iq(0q)3 zr-Q--xoyj1mjQ!=kA?H<9s5lBUTc-p{>bzKus#1NI%p==P*(1t%vWDfXDNgf^XPO% zz~x=h!|i6vqd|$^<|32bu_m~vvyCnNG_M+=OI78>zCK*O%nDoSlkh&2Y z#fs_E@4XCLTGX`BcRtw*JZ!YB-z%cX21Wpx!#J=eX`&krZlsoIRY$HxnZ9Y|BDkU=MNKd zziKCFH@C~BWqgoZeJ)ddo3kjP&_i^rz^V_yioD#aN#ze{OrYuRb}C6!r!q3i_J*f1 zr6ntM3;w81VUzLR*-S9r_e$rhuV`a^W#00e@1Ol{=00JhW@RJdb+RCA(+36J9F}i8 z+50Aa?@=m8+h&qw>!p53Vqf~KcQsVUl*h|GRK&FmrtD*p%ep_JbYH(_W>R4{Tj;zD z#xs?W>Qw{#jfIujyv0*l>e;?|G$o;qG>%V9_3{2Pba6DQ+J|Ul`*GMQl7vTTakRu1 za&Gu4?6iI*>0&3uB;a0<8U(blRE01ES9qZp!Q7B0raMn1>xVBZD2=f~z|T zOnbdA_o=IJ^Im^>Le$@dTA<>cGNY?3DtV1WfDONt)VkXZ-f1go3`Xo;9Ni}%0MD4$ z%mvf*`(Y)eI>=2xPfititMhU;$lp|4BnV-T(gCUFnPsYFi8+RSuV!?&5n<~gl-JVC zY}-K1k_SZ0-*+CtIL87tNFua#j*NUfl;+<}{addr<3bzk@p*RE$1PXEoPfjP1G4$( zbI`xj83-pS`TFf+Hn6w>obE@t|$!at$``8%<~YmI_Ww1l5%HG;Mrn)5IxKX#MF7v&Trn zDqUqbm3-ac~r-d_GR zR<2T@MJ^AjBdjS>8kf(QRSc=>#_}7HB$C&3XeUW&3%fI!7 zi;u`!PJ&R_cn61h;3@4smjcz1GflztV=6*js_bADpE%kx#o@9mhE@sJ|2qa@_5{)z@r;NYpn-?TepJ#)@xW1L} zS(*Ao$P*1ha*$$ege@otEJTg}>aiO14l;fqwIRIj4@X}MA3B*O z$VX#W_v_i$OtJnvR_D9Gf4=3mFFx^&PNM$61|1&W2tdG$_|8&g7N+MHK9XcE^Y<@L zpGU{*dDrh`Q(fc_w)mi0y^&k-a?KD(vut#S>#6p3;XmyMseU9b(?hs34fe9t{@_eT zH^{+vlJj8Z6Fkt4nnuMT)scUTRrWu_0$B|*KmTyI`O-RX2}XObl*)e#5weeSinz|D zYybJDouC#(?uoSgjy_@iE31LhmpzY-+lzn-db#f$n-+gOzX-|BaCi~vipP2;Tc@E! z)uVWRfmN8X!vUAGVN&ED9buVeR3u;gR;bE2T>Jp4A`K~+l1(hqf@|%14_@RMlFj?K zS(5ysm%vi%#+qE!=QqcNsnm2I}6^G%0wB%UOq^V@BdL6)U>$mZ{v zUjw}9h+9C%`nJvcbG$X@XMOHpyz%+LF43jTZ(vxPqi4blCA?kmk8#ZzCo#oLpM-=q z%rzFift2WxGsmsH|9p_VmFjxj$S!5^9!>r10%PQ2vb_oZTP@KMjdo79>#j+RtNb5h zchNx!Uq;?8lRjLy{UmC7MI4;cp~8K?p_X+2J+xu(b5H8LTTaoF{+Ao;y-o=0H=$U5 zAR6l3QHcfwq|`?!hJPlTKZ%`YnmMVuG~|<_-1#zZn-D9;qH92HgihNg@_#sU z)t~Cl>vcCI8~^0vU=uoJW=q^7fAikscfvHWY*~<~G=>h`qy1RENiV!jHpX~pps{cY zAptLPkMycKp{4Fi-L*MbF&xcyO{1$rQj|TP@5V@#-?C z6u+}Kr?AAx`Hf{3?BY12OV^zo`kK93*O2*RsFP#iMvwXggS~r0!sBDkbe!$_agUlq z$&z?eGlOu2wRHlH#d(U~{>rWf@&9_1-8!owLoKcp(1Vu*&OI9;Vi0uqPM0e8gbdQs zSQboo+KgxH7vH`hiYMGzz1I}&&t0ixT@QCk_*%}5u`@{<6y(vz)-M;C0zxg#vk*Fo z-uDkK(c0CU*X=hDnId7oq}#C<<(ia5E}YVX1b=D83!G9hSgmswlpeE9GhJY{IA6%M zml=!DI6D@#iCH>}piaw3H7ffL!I6Y5K9?)(h99otbLmPL96x-m?rZa?2 zPn=aOGB=H#8+-IUR5%Xb54iCgt;9-np-6Ci*2};`qP01jn|fgz+dLOgFT5d`iedDe z45xtchs=jVM&`uJMX^@qk&(UZ$lzUOXCL{|!(oE3(m?04gPs0O`ZAx@+!S`2(A+>( zbJV7({`b!|-ql}Xa}^8?K*J2^(DxiHTf<3n6EC5!X<3NgH=nIAL&&t`d)`JUWHGr( z+mA;>_TDG0SB%^V&=t(WiYn=T@;-k+xjM>kVa3_DN0|VJKd-L2l|GRyJAUiai$^k+{O3PpqldfK?{wwWLe=|n z3@-1izOsw`6m^(mAk6jc)pq@%8uJO}Z=)CKahs;HchTsCpKsFP2A_419mYUU;}fCIT~BABn0Zn9xt4U%0(#Q$*(T)@aMHW z`sxEK&?W7onQ)MopQ+2W?>CIVIqj(8Qpn`ZH|LYOwD{c>6WpNatQ$>Z+J{@p_fcK} zp69Vm8An`D^s%_UKve=Z>X=pkcn|s#yM0VeK8o~}N;gf)OmlAho|(Ua_R>So)6Fp7 zbxl@0RhS8yF4eTmJkaOerg<4+ zGy8DKQyvGSQ+sy`x(M(wkMQmpP>n%Qh!{vvk6<3$nSzb(eN?YL_dXjb`=hKgQDg~v z`Gd6bvR=XGdlo~Ox(ROo*BXuXl3!vjKH+QFY8SXXo!% zB`9y4{KMiByXv1~hsCJ7kt_>i^%m7746WpJP!CM{#UZYJjoi3u-JBQtY~B=UCSpAY z9c;xMLMMp2p3d?a(bMWphS4E|6W6D;@YKjl zD;hLE(F`i;>&nw-UrDdCgF%v#ax3R>+uGDqEdu7)J8MAYfL65U`nr(HI}A`MCDclt zOP>%ZIZa{@?%7EE#=os6@c`w#>UYX9gg1qaK_Eo_pK7HZi?1U7j5?_;#@_U0n4Lvm z=WRKZhcDOKRC*g1gn{Md!%^8$s2mlg;_~uxb@lP#icClI)sNb=Ino*BLn!_tGT>Dr zv=lXk6{m>-X#pQ5FSA*9zE;dAx8GNYFRcx-+!?pato;^tTLJPFJp*pMD5bx7b6bx* zySlSOBz#@R1Ip>cOunvp11V2$iaM~;;`P1St0LpoO&j4ky4N>Or}n}qfJ|w}s}a~5 zSerfH7Fs&|MSoH1?CkKW)@EZ#K#y{AEq5}xL9sS)@t$7<=6kQ&)6oJ|YgmObrF>L& zc5W`?)tQ>6`oSSli-|4dLQT^iJ4M*9T+7OMBu|-5V4*Z^hs4sxhV|;Onl1x*G`*c3 zuN%CG>4*CeIHoP8DhDq4e#n@BDH!Z6b{Ugxc9Bt4+Dj8KUmtGo^!6zqzibl`_9)#t zinVqpiI-V>Sa&dpo zGouK*`Fr1*{Z3B*yj{TRu{S_~Q-X<+$P{I{E4VeLvPzZ+nplAGOoNBzFdW5*7`lt}*O`QcyRQT`Pg$j&G`c$CDa| zB{|^eryWZ>$_QooLa6HOE%$43x_4e1>QALQIT_7$XNs)WqmWGJRdgWS`WKNKxLHtg>sk_0T9de z^T7&=*I7e6GX{3aR06R~8E2rAKbAEP?H)DD4R@^eBWk7hlNy|m7IXRT9d?pPOBW6>LH7>bHe z)L*cVsEw<_zKePS`3z3@H^(*IZ8`C%^*-AKP-T#6Jess@gt!W-rHk&*#!qc4rCU#e z0$>lxmJO1@1K~%5&ik7ZRL?BJ(?G*Kq{6#_Th>KFyp>D{jZeqgrwMo(0-j4VI?cl! zRD2ySLT7aCU1fSS814zc?|3}6Qh7n&&xWT1X_E0PBv8)E6zpX!eQCR<9-hnlVR|jx z)Oh8yZqHmuk!n9>GC(^4v;l0G3tr7 z8@HRMSnK}z{{b#RgT2O5SdxFy@d`#`ANNaJ8BS4FQg)Z0p}I_~x7FWM zDc`OISFyNEj;7W7WJ1mlh9wUkZS(YQie}LJ9SK)`|83ExYEX>WgM(+`O8jd-YQ#Ag zn((Ab!MbV3$Mhv2HH(6RvFTXrYNSa`9C(Pwnyg{$zC~g^9;|AJ%f@{ShjS~As#iA9 z*uw|=SE0nG@KoOF3+0&?`j@{-(^lCZDe6pC5NyMZHXi8qY#8WJKo)jt{F{dPtID^8 z>4<#7PL`$G=-shZ!ab&&v#p|8Lg-mGDG8?d!%*jK;Zws;YL@QR?5_UqI@V7JiC#mp zPk5jiarzuYT^tEiT6OvJS|{TucD0Gm|HcBU|8#!ZVaJNRaHIRAugZW6hOmip!>##n zHYK#D&ZF;Kx)|aQumjNGL@FB!xvc7%Pa75R8&0Brkxk7oS;b>FHvmMf$?Cir zk_PASI~975FbIc=S~y0Yoo}gj(R1}lh1Tq$T7Oi{^J|Z2EesjxkRgk-1ScDXgIPx; zKvj=L)@UP{!fPxaXk>565Rj59z2iNbC2KZbkF7^E9#4!2ANT_&9!1jB>hnTQt145L zr4Xy>I(r_A(dQI`dvKd9ZdFfUoS$>C%-l1Ho5|szXyajW6kao59QR{GhqEW!dbx5u z#1^9eI=FogLtkWtlss z1H()zo7?%xE|1%u5e1KxhACS_4}AvJ!3E0R)MNf=MbXm#_-xW8H%D@46g1^N&}R)% zY;~YXh3-ZX1nu_SQG&loT+A(Z>_@^!u) zfyTPWYzO7*$I)-@C*||3h@>l=k9k2BGPR~W$@ZP^5hw&p;tM;1kpYphY>q>~T-kqK zG@oeg(0;}YkQ*=>#X^22| zb0h`#}J|8rV1d^+3KsYq~x)=kCnZ+YocBAJg(S|E^Bcg&0WcG$#( z#!>Yl+fVaI?dpu1`|~G50sy`9yBN>$_K;$Q$Vl)`@IiY20uH7f{Kul7)Jozn`}%e^ zE^PMY)O*Hz09n4?ZbSiR)nWJHoDeo9Qp;KtQ&O=YE& KBum6U`Ts9W_->2< literal 0 HcmV?d00001 diff --git a/contributors/design-proposals/self-hosted-moving-parts.png b/contributors/design-proposals/self-hosted-moving-parts.png new file mode 100644 index 0000000000000000000000000000000000000000..423add2e196bd712d3ff76ed1a9d214037fd650a GIT binary patch literal 36201 zcmZs?1z43$*9Hokjg)jrw{&-Rr*wx%cXxw;v~+_KDjgEijgr#c-Q95>MBo3r&N*Hi zgePX!ti0DfL%5QHBqBU6JOl&;qO_ElG6Vz&3jzYt4fX|a02SyG4kG20NTW@Q6#~1@}tPtq18=XW_) z>hWo)#3sUb?Mqr4rv;mePL)%PZ#Gr@lM`RQOB|Q#jMJxH$(!hH zEvF_2Zl|GN3P_jfyCm8($c$}$I@;J`Qkw`>{CR2Z)v@&tqu%-F#TiW7P19R1($_Jf z{dzML!km5lv5#H^Y{kHl`eT+uCT+39`tJoNs(9hLo5E^@UrGO*mUBBBBWIchm0t4| zB!=V@K&Y76Z=kmvW((+`1Tkr#f4V{1Y8L;tVR`%wf-Uz}B!o+2lo#ILnhhcfukP)e zK{-TLm$pxZI#|O2@WbFAt*|M%nbVMzOYdFxkOPIe!XWz~u*rj*i9xijxaFv8K_cbo zdywl(#9u({OJFYOv{ny$99@`NVahGY7=Q4~01ZP7Z!kIvc$tJoEJ&4@UW$k%q?j0M z4dfvjphSfM)g>$)3xOtTl{273q6$tF4b9cw!rsDn#`yu~B8;2^X%6;;@d-dp0gW0V zPC;ZLR~fo-K_G=Dw@cf<&Oq2~n_X6}LlZ#aY)@XEwMV84FlaM|73zhB`Yq9qg0&J3 zCaaJlB^8+ zQi@W%QXu=OHh7ahsP&}1C8)5rX5)Zs0Hw#!x2YQsy)85HDi@uK;lhs&VC|eDV!3m;qwwbRg?$VcdTowc3@Z?< z8R%agt|Y=GQ75(ax{+88Wj*XhO!6&LoTRS!oG_n=HOT@65-O$C3AyM;R_crdkr^o( zit+f;IKuda_@D0xDxo}3QxzA47$~yI7m1N$1ywMpUSlZ!Li>`_!KZ}#U##$t7CwY*`CX6S$B%Pz7bdkvHDxu|uWVoW-gJFQ z`@*2Cm+L8Xp0;Df{v$0OSEEE|KVe4HMRQ(^U)ATkXW|XPIWwX>q5)#AL~1u?)MVOA zdCmchfwuz@X=K`L^w{*x>J<|dCCPH)1-z_LTE7~9;g73NUW{T{5ltKnXALutg%3~X zjpm$+CFk{x$dA!a*o+d4kB%UW=zP{Fyj05mGBlz%u~@WNDrjqHj$%^YmGjeZN@+UQ zCdEqA>}o2^a?LWqLe;`~@s(LWivs(C-JAs|eJYJST{UHnJw$$A?z-SszG1^?le-_i zzbIqUOoG{ovyZum6_3d*1t!go#g4^~)xgeo?@sS@cGIu6{JT^Y*W2S><+hPU$9R*7 zbzWL@+ETTUaUZL1-($`_cTIO|ISM%7IAE-AY%FSoYjmt^S#ZxCbl;2183?z2w>Da>_vF zmDNpQa9{}4B`=*R#h;X!G_(C-%U5ezi+#|2Kz-0Nzv!;%9>zE58R^C2S$b}LF4E}T z1mDQhh}lTi*w>VOvvoy&J$4;=GjmgVVRPPb1AVoBvTJNrUcaY(@uj8dQTwq7GQq#x z|HR)c;B~;K)+A!CUO{uW;nk2E1ZafxKw&0ccekT>l#bZ`SOe^3tPr|ynrD1tEY9-g zUa#n_J*FzkW0nb*dE1G^Ux|m2&5)XqC5uf+L`!;zV~R_E+LSDdY9{!M+Yw_C-PI$} zt=l;ygM^2T=|*=zOPQkb6DPu~ z=P-h3-P7^X(cpN_y{j<}yczr{7(0A0Ts*u(dQiF{UgD!(yzEDVQ~vAqyIN{rY8aJn zl^~Uv0)+zceD*x_k=2nQI1KRDz#^P999q^N!3FK!x#XkFPKp~}sDxBAp-t!qEC&ld zn{Gx=N!GoB^KM~(v9JDd2ff=}{99FjJ+KSz6KV>z#a4v}51S*}0b zHPb8TAU%X#$2}VNsPC4T=(J!T!P{bLV;PXfMOr>Ffrjp5=ZiO;y&H2c9!mf z%JQ4v<%&Lw6F#fA9=8q1bp&4ozc`d1EhUTIiQb3$AaknYswSx}e{=knMHJ3A^1!;g zIb%7#)4pRT2`x#%p3AObwQpq=^Ob1!4a{Wtu0%8=_ku_Ju9#Dk*8TDe3DIR~du8vF zbw-Dlvn+V7ao04L9}kfasXk^FW(c^s-D**v<$ub5+(_9db$ZyG(az8we6W1j{e3#K zNy|W`KjjeT%zU`AmOm$JT+%)%J(W)KwRu&UN+2&^hh?!G5wW z*CLnS=5=a_yKif)F4w*EFH0BA z7F*`4$_tA)Kh?I_NoVI}PYT`<@3+l{awAg`+PFvh=pKn3NdHthFHCH~dt4x{h}V5} zgj;t?w>SM+_+Y6>u=3%GC*d;Vrg1G~B4Wm(fGm>{y9M8q!khh==h5!OZ}zf# zYamr{{-Kc-m!IcM{mk`bzUGqVFvhq1dYA)?<>JivrYHP3P?09vQlLtp)-U|#a{SgQFQ=yDae7B2zJoFFhdOUKobA#cVFA~H}$?}3wZ2g z74YOy+EgBg7rJ@@0s39>k^pfy!3ByKJYdfWp`#;vN)(r9Zo~dr+*YJS`sdo3*BIVP z6y((eCpY(RHty>ZT$>9Lh!P^R#%tBy+=oY{(nLhsqk|&3qq1W+fhOpJ0WOFL0LNxp zsA@WE%FFQ>+u1M}y|a63!r*RW4?tT82tIcn;71!1XCq>F8*5u99(R7yKTq%gKR;b& zBqjdyh_e+xsiwRVv8bJ+2{AhZ8v_%m06Z}pEtGs?@d-t_WynJfA0K!laKLf0sph0XKnqt3aCo} zo{#ZA)eFEkryT%>5`vHx6IOMH+{=W^7&yOtRJ;XP28fC~OiFkQYm`u+N?U(4W?uG<0K#^9XSCPUffs>7yl(&c6opJ4LP zb%xWzCf|vmN^l5R%Kz_2SBN+qVa5=hTnO~{lLMyGJPQ5q*CiGZ2A%HvkBD&)aIt?r zO65y-|6Z4BH31&4dr5&u3W-Mi=aXaDw)pSSSpPm)rE+^584j_3!wa3ID*5KBY>Mzf%)K!eiYb zK1&IFuZsj8PtJWO`rmjMbXR!KQUc%Yae&7eM1Q{jZ#-b(od3Jo;k0*e%dK-krv*{fw7nG3|e~9nWf}@H2rro?J0& zZel+K0efvQlh^TJj{Dy5@#=Wh@o<5k`pu_c)lyxpz`#H%^Qkb0pg$^W)qqnfckj)d zZE!0qHfYhWv0G$N6uhq<`|?Kgo9b86?a@NN-b7?UM5EQM-=7_qS|O(^j6-V{t1VRa z$;3ZkAlvz~|6PCUldZxw-nT|FyY6XXzRr2`MX#YIfkBVwBNk~|ADZ_TATAmf)@z*_ z>$oI(9hp!h0_%+gFz4Vui?l)k&M+nXei00YH>dKv*rPt*ovh{Et(P}vQ7_e{M!=$r zO=Gu`k&=QPAkXR;%6gs|O~}7UkJE-e|3 zrvM7k`ISgZDVszm-DJ|zDII0ep;)I@VPhb@kXDN2T`$U_sXM{nu&9d^ z5;CVdQa7lz)w{l2^fTEW$@e8uUBXr>)92rwtu14aZt}hV(dUctU!bkX5)-pwG2x7T zRguKyuzqErpzkx^>~l-b#-?hxVf27n)~tq)|1|y;KV%Cw-TRj~q%nLaVU6b#I=()v zlcjpJ*}gYl)5IGs>Wd9TH`kt5`U&O8N~QOccmz1S<4*5OtDd+JV=%$t%y+AV^!|Tw zy+jT2rPbV9Xwumj%4QrbkUz!f6FV$)+8hjA|I(L4FMC#D5Akf@0O?jnd6F=d-v#n` zh(Y@w=<=G+M*R(ce|bZe>DBCg?QOAzPy0s;{zNE8ylo&_p>OIX-_kw+;dt9d@8}EP z`|-m#sn^Ax5d{6_@iF|LOc4MEo>l4X3`ft`Yj8#TA_DD;5CV@mvSct?sHoVg7?k`R zy0QMJ;5!*(hwTxHD2aNHQ`1f9t7**MsSgx2Md~FV_ol0Er}wxBp0ke~#gax~*KS!$ z>9n+g@3o2D`u=YR0v>0n*rvq648pF#!6y|F=fGV>QIqrW={{VoBIha{FI`5i7us*| zx$aE`1J_{8{Ns*tMB@u_iOO(?%q5Hw?l*Z;SUILcn4Q zGSs4w-2vDh)( zRk5MN{tP>ipjB4T&ni%btd0jb7q6ziaO*xxYCy776$q^!ytF^7^XstXk?=5=IJ>7< zt7^rki4_}*K{sUs#mv8g;n`DFo`hAj*56 z8@Wt+FOqhNHhYxrbd#4GCU|F}g!cAgf3!DSN;rI z5#>?-)%m2scY1T~nHn2fYURAx-=8HVw@3A0h&9XqmT&(C*emOqGZvFyy#?~=b6ph5 z4ZT$XFei;3r`A2UmxpD6pl%3AsKAaWJkB%kLWNA#DkH(>EPk(V@1Yd3oHqL77@My| zK2k&|y~=&a6%Ac0Dr>I%emEHQO|3}A-Mx;t!Ep9D$3V)on#q%7H5#~YiQ+N)-nr}4 z4s&&$*}s4P8SvqnA5$qjF5eDnt*4dcNr8$K1S=aY!1v}U-AiZke5d^Jmgw&|Qmtmt z2q`EB^9_{JAL|y=f^DY)%*$BLYY)R$ytx&Px9nv5L1{d5YNrHC_-svRpW%D@f1&Wq@M;xZ9l0BwQZ0-FC3UVBodS9>r<|2X^1JTWoHk(<~=e z*3g6mMTxC;N3F*z3RJ+MFr~|-FbRJrq}Q%i%8mpOglbtAw_;bhP-x5DNyNLs4DOM- zdeOgyvy^nH3KxM7|2XzcET8}c5(1JZ+YDuUc*jc?ixKjDLRkq`(|#%-wM-h@%v8UL zf>wV|yvMcEu?gxiQf_n6`{5i%BKrOd@ySk6nI~N)_lnIh#!9#W;_VYS3uq-@swxsu zFVUuu5_^w{nME_KDr5zz5^B5$ccmueFVz15sGZxf#WzEIr;+f47puXhP<@Id30yb2 zQqM+Fz0mr%Aq7NSSA1C>p2*4SuM)+H&it*4R zBY=Kx>7Jh*)2vHpnx*_B#_Odj0W1L9MXV6xOx3d3R)wli`_Qd)fB60!3>lP9QBN4n z#NL|MipB|AI=@VSCN`ZfL! zKCDakS46a?&6QSCApN zN51pX@E>AuLp`!+11ZHSSzR8?AFQl$?gOYn#d7@HI_K%K*t4*Wf3kL%V?BU}4)b=! zKYzmMjIezX`Ued8%C&AnM*wB}SH07wIa2t#r03;X< zx=~VQ+j=cDdi?4wq-<+@UdR;~G8=K*MiJX?EiCLB$Y~OAnKlNDD7Ai%_ipkZ0AwoA zT?X4p1hFai>nyiYdDh_2AW6!(lTm znM9<$+Uldf0X!D*e4~dGwSJ?!5WTQ?@du#JrLtQkX*^7EQ9W0kN6IBP0j%iSmv6Xb z6KN=b(6#4V*SN)I(2aH7pZN(u#c}*Kjri4f&!c+3_g}Q6E9PZ#JJ8i?_O4&DSup5Z z^n0Wy;B`#^S`OteN0aIO9#4Iqe+EzQn z)Y?7)jgCW0Ec-JIk!nQ*-|3e_uoh!H^%wX6?0NC>6|rXYHZt_7$Nl*vJLK$ltI2me zV_zSb=m!J-7^5{54jgP4a2ro54zE>h9=B9NXm1ZfQK5S+^IErFpx<`C)C)?{+hLGDS3_D)0Quvt`77dCcJkOI(8OPcC(PS|4JPUEV$SZ zvgW9v70L_1zdKV}O`a}g+DV>3#Yh$K`N3v6CZ+GWO~LJbRi40TAkdvH4kTo<%%IOS zRZS8i?SBy{CpxToTl-rC^mebqCQ7S`;y6WtYcT-bQ%S}Uepm3i!}G<^#6^&YXG2Ay zT0%c5U#Xiri<2aL(`NDWBL#9p!sb6?Qh-5&9;_UeT9xlm--+KJG#tJ=T51Ejs1_*z zh6ya4uKh%x&1c{aUH@#~J6G>iXlmZY0gk9k!q#^e1|TDe^hfeIuge4Jjs8@Zby#od z|KI~U7&PI*%3#0jS7AHPbDtsLqa6|ngq=B!qXw5(`&fn95MkTL-{_PI<_((g5#*Lk zqg*;DMi^;Py7i9R-{Jt!Xx?ya(Kp?!mHjuOh5Q4>@%>jzo|;&vY8W8$RI1g^VfCk2>{CA_#K?nPata7dx-oJXf;Z|l-r;0 zsI_fg?9bxPqYEH?RVhpYkm^@0Pyw8GIVrRVT=TpZjYcWm^769dMnBH}WTsDr&EF9r zfB|i92$ll$ft14eov_x_NM9I&DMeD?X6tjEUgy6;5(@#+#rz22&a4L?1wB$t>8gNM zjJ7?T8)UuMtcM~L@sHA(lLsgfJXkWH>**>pc8>RWX>Q+s4i77Fdz|E{6v<}n_$U4| zax}n>@V+XY3GN_=2iZ}$2E>H2%BrJ#HSRrYwzU?Z7VMAjXKU@Ud7M9K*&(4HtXNcW z|CLp9;@wdy@_L$P!$1w5A2jcxdE@fdcHQ5Yi(L8JO57M@4*{mzFB%B(D?+h@3Nqz_^XsoPpu8%Wnc zX#o*|^F`p`;C|n~#9^uXk{rw#6#PyTDS`_Ra8o(=AnWIJ<`3S3WwQA}5E*PunwsL1 z%Ij8|hKnCFJ1JdkllMl@@1YT>SUeT)TYaA`aF+lWK(MI-Ieo$(5{ghg>#1bE?- z;boRWT+T~W(Tc#3uj5Ud&GzI^x0i|lyIk#LI{mIg?Rg6{18Jk30z=^uX=bd?Z$kpS znz;Y`(N+N-1_sbMvBklmzh;V&a`tzR{9`oYL12VVoJ2p3mxzcags6|rT^WDK2=`!M zO)-WS4+@T(gY-VP7l{C?1K;lZ&lW=7uuMYg-e0b_**{A}os6bB{B`gwQlY4jLaC~M zNg5UcSRiQWwbO@x`IZ{Ez$ZS_bMwT-5o7+>zj28I?!#IQ`f`xMY|M6l3t)Ya`9ucI zb3ZBq*j|cCSg^@=s@y2>OGcB|<;5tDB5a@@2jTsq ze$u^vD8Psvp_m)&6##sXa-F@=0NKf;f#3BC2AAn`7$Bnq1g6na^H`=2Au5mK%}C~| z=977ekDnI0QgIoc6({))Xo!?6-il|xVuX?Y1|@NYVUBYJa*RFRyC4fBG+K_JDSjN_96txMw3(UE&?2f|<9jR)R;Y%H_Q90$-gTZ|L{Jm+VS@t%M) zB-#-_DAIV@>yqxbWF#LcK!>4RY0WixX`tlB|I1TC{)*rSq!sZTz(L1cG|FX~ieoy*dyqqMfnyoa67>*ky`!WspWNLSQV8Qx+5siNS zfVSfS-V*`*66=qV0$N`KYAio&ku?=GY=sb39R#R7=9n9@3h}>G&H3bqQMYUHpZT1(V()O&9`VdhH)up zPmhamzo4RGP^DA4$I=vjp@84kf=lpEW1&#Qn$2GZ_#)#eE*oEj9}vMn6|B z;unBOZy`E54oD!(2KQ%c`#w_0&W%FqQ2cREDOKP$8m8Zp%ffdn4Zs~@V^KCx;TmU$ zaDZL{{8W-5?cxdV>NUB49#cr8EJm+?}(6s6bY4sfS?i9ePZR_ao zZ_Y>60a&~Je&W9BXMZYd-%?wU$zrolKiNulZOW6puqYs1+nx7#QTc@cC-D);TMpu3 z=-$hnT6t2aCaz~&ZzxOOsXhA{z_Z)6eR57^aXkN}W|tZ|8MT`2_Yl`Z=+(d5y-p8B zqg5GPtXV<&%O0f!!R))z7P#LCeI? z;73>D$p-)L&84MraH)QN{Xmf&3d|oi6baB77sPVuC5rh}d9G@ay8P1E&%e{)J!OGz zANbxk92O&DD2Yx$B8TMmQ}`3t-cJeKpwSAFWdqnAq1O9R-Cb))7za_modM9J{zFe` z^0aoig_6l;hD}&E7NA?MF2%Tc|KvM#!2j>oByM)uWD=-yJ;XK!Gik#d+WyWv5B}(J zdBnwoh9XRe%4N@Dj^&5=$FuJNf!rVY;jd6hV|oLj5kOf`kblY~BT5c+C-wna&vQl) z0to={BP3>y3h7_h?H%wWZk;Ut=cv}})6IOKtOVLUXJ^PmG9e;_!v;$HH+0~oP9V9; zxNJl+@LJL(pZv|BFtAF-%aFZ$?Ji(|!=?mEhT8Jmo@jzns}%ZY7pZ%~9m$68$g;?p zfE3>*6Zf3rDch(5Ch9kuJFCaN$1Pd_uBUqDRsoAfP7r zNCCWi`SaLm_|KZlz*~*mWo{f08{Yx&_7l(}M1O+HpE9{f0zj6-Iojy40EiHH7~;7< z*@f{bUJz_S|LveYJn5ok?k7Zc53n1y4(upFDg=!FVVF88g@__LTVU`55Mal?smYe= zH_@04X2fAoE7#?Sb3DUd6e2*p715|kcY*+bCY2pXV>jtfVNNrh-+l3_&tE2)x5jlFK{&M6=s>_ra*;Ct-9tHF7|yO)l?tXMe6Ijj&fC)Z}?q z>F)Ab*w;jmYp4PKkhd()=*oW+u1>G~YBO7h#ZCK5Yqio+{pR|2S38*kx@l|HCie#z zt@?M(8T|Jp@7!rVtCqe|q*05pjtI6Ov%U`1rgU2{%X958RW5#qW3#$?eK|LVq+1J>E%>!P`jEzRiX)VE1KJc9D#(%r>Atj5+&v0V8nM7_xAXPsN zT$@bFm{hH>ouZz4oXvZiIasE(%Smo^kx_FxcSQFjbIoLG+pjl->+S%UdHdamgDSO(%jkt)ehdMbv)KhZE-Au$GKQt;`m#f zz#vG#Ei{i#Fx}N_LbDV14Kl_tCs;&tY>V^{3||KMFox z>4`|5Y2?0bFYeMd8$5~n$fm{OK$*{*^?aC42e)Be1J^qTYi=qdI0W zHut)Cil)3k7-;mINr=G0n1WV7#StnSG)oT za*GoBm%h0{w%ByIHRdIO5dFw5xEzzIKBwnze5N{Hf=)$dIlXorU`fu?YsU5o(kFU1 zZ(VnTa0^0nX3Dl4%rR&*%8HP$ns_-RS&D?LVp-H^NdQgn9?o6z>*g4WfTj;Te+ zUYOq;M?x_b-FQAlVJe1b0|g`UaU%-GXDIC~df)LzV@iqq=0ygV-9jVve1luEUIW!0 z#SP*z1W=_wf&R1XSUoLwb1zQP046Rll(8C}YSQmoi|Fo$@k7S#1Mhr)W|{xHpIFO` zkOL6G;397;wii@G%}1YHsAlZAB1#voLoQ$zqgM?G!&QgSZ0&~^hB}%@I4{4WahXOT z4i=`en5j`J;TzOk`=^PhTi@z-%Nu;T>fQr#BYR3KIJh z_hRj|&JCBH?*2f7>MY|lr2TNwWaFBm4adB9%$>e*cHCoIhj5ASKbrUSdv|nqfB>v)ZFQ@04 z@)+)HdsH$xTcowl=T;cydnZw)cetdfN-Y9`EJB%&PcfEvApkIA&^Y6WDk2hm`$ zhovI77S)pIGw8V;H{&ih%9L7O@HJ8nXnpJBU9M$1D;fMDvpT(LFt(Z`^oHb1K@F$H z8F9d3h;e`&_>PU3T!<^68{AAZ9GGKJJ^Moixu?&Wh*o;|^-d<|OoH+{jclPlY8-(V zGVW)juMIWD$oDwry9#qJeL7sm-0&ZFJ9!CX3(Kv63kz1^nCB3qB_z zXh&us+tZ3yoIU|Ssgaa_p@{{*4>Ji1`YHiJ<*6`On%-2yGaMJbNA7Ul z5xu_|nslC~l*7NEaNdxV;Nqrni$bR;m891H;Z6P+g+o-T-VzOTT)Wq~tA*()xO!tR zq4r_!C<7y7jWhj9e6YVxvT2SK$TISayf5l_Jt%Bn4$nrbnGXseE2z+Nxz2P4$4cf2 z2fYB1bR_qU_f7VX>_7I~ojZ7t4y-CDuK)0WU93MXyLP)s$@5~#+`q9Z(97tHf3%oo zwpxu@d}MpHePBBuP1{x04khz>bh{L+_k@1rAc{l~=_qM=jI-vhl=!wgNO0%pbiZj} zj9=*iK@eWOMv{{Q;bQ-g%kj?`9wHVE@(F{siXNj zux395zDUozhlAtRb54N|1oRR~z%nToAkcySfnwH)s;CX7smD^JAEPkg>Zx z(jlje3^t8j!$u~_;mpmeDdX~qOzvx_ul9br>vFe6$z^eEc8HV5t^IFmMMMK)QIkhL z`w-cn8wX(-dv3|i_tdHt`~Hsc3E+ml|E$=LtPNdcC=gY|Svx?qI$yzUYNij1VUzA$ zJq{C;C@V0&!IY$!EPlpMf9A74v=h1rXbu%T_gWK$2>`rSurGH7WkP~-R~M2z`pl%S zY1ZFOPViT51$8M`U!8tzF%<55N=ECEdjJa(*jmEec7)^2v9Moz^U zXHPxt8oONFU$F;4>zY>@o``J3W(t|Wi7U;(ZxvXu^g0tZiS6Rq@%4SCY0P3}1mRy% z`SupKehdG&-2If+7e{gk3#LdYE%Ev`4LT)c<7prI=GCykQ?oU*J86=Gf3+8@MQ9+q z8Bp!ePiOg~S8!l9$VK4r`B};88dOMYb~k>&$!dMX zjS}AF0@Y_DSxZ(kHp>N}ZscrYJas#kKr!lR&qbUQ%u`}<{>FgMx<8KMF&Ne4Uc5Xc zvx_;fYj}t{*nf98=wk_wbGu+(((u-y6YplsBc9x8LQeU!zS=mr75Vcx9%_ZDQiVld z_sDNuWW=5o4^-_yhXO=c*s#Mun7i>ljqd{~4>^pmJWT2R`h4>ixuAU!^L8$!oc)x` z2N4%|A2xAE%x}crAL6!}anbKs zOI1FrNSJ5eCccr*-!xhHan}Ou$V&j$-{FAq0{}|G+&cmQ_P3pLI&PL^A{u`(@yqqrrYO~hfyJacS`V}HSneXN zku7;>VV(KtWJcEQ)dp+A6ra;N1cu%u=nFZ;K1cCp|CaczWLDj$r=C~8&j;wLVV_zK zSy(QRrYCSrP74km9*vy-^a5?JG^Fv&QQDYh>UAhGeq335OVWP2>$lf$uL~$F+rQW^ z@>RNy-@qm|p}wNa&2K;|rJUjyK^DtBPtux1N5DsVGj(Ha3!-gjUFdu%Y==-VDAs@0hYm#pB;A6K@KzFk zs%rkY{EhKLiL6Kk-Pl7A2g>mXjXM7tA;58%!OEp`$g^WX!AQ{Ixo&7HPKIa`aXh-c zh)H(U)9#Zj+Uz@#k8pXF9I5h24HtJ27Gq2_KoAmM4#U3SCG%j?Iqu|}#}NJ=(uaAi z*Jvf?51O(Z6)T-5WUU>wMpYW*egWR!Q0tF(gJ9}h^h8>`koOowyPHcuBOkhoKGCh- zZLcWKKx=BwAod6()8C10ez=>tpY<#ym!jC*wJ_HeC%WOpMIy)R&4eEj&(L<=)Xlqi2H7(Y)dfW60?Bx_c> z<8pN&R)4wJ1a=9!Hp)t2l%0dEQ!@_eO@bv1${dc0I3@F3OTF7OCirFGb z)7T|sbOuqG&by>Y$unsA7yT$GNOV=f#EvWdwcY`R1WFFn_S{gQ@ zK#YHNyFstTXIjFSN>PGQqAFF`xR&|-r0XK=itzE$oaO8A7cmTso=aa!yv@V7nSC0W zi@Vg%8E)eN>4J|WMu&FEBKapWU$q%K8>YuuYx?XoY-Juk)stTvzhQSiww;P$bS^Ti zSh|;1u2KF@)hDNsZk`OwcHR_j16cxIk>*OOeGyZU&(_1ec56Oc_d|}5Rnyd`aiW6f zQo5BIl$)CioP6Ct_Yf->O#ebli?qjWzAzOdqp+$Gp!RA&r^zy%m2i-KS8X&A|0ugW z7+TeTQQWBN2NbffE^<$2v#6(R1Ewv)Ihn-%ARTsJ zIeU7aTsEtaTw1x7w##^}+4sN}(-AnU<(||^_Q*;hptteNhM}(JGpVW4#%XMn&EZmC zfIsO?P^k^Ot01!nRQM*?;I+RyFJ-jeC-!#BOwNvvi#HWVqf;`vSgT+$n`uCM=s-mL z7UDUqNsJ}#Yyxlym40r)WV5HQ2gUba44es4jdgKxyr-W;S^6oT0iPjC_Yx!0=yKv* zxmwI(6;8k)i-VbsDf`8Mj`**FA8wmiWJkKSQRo{4eS0eP5_&qpZ_I54vM!s6L0h4( z_qaqp=-}vm?|`@)Ef+!Mc4KI-V5lgFk;({yK~4g`WWv$jEEe;z182IOe$|lSp^p)a z-Ih(jPf#Sp3&OIuY^K3k#$Y#O34Q3uNXq$n}@E3UH!C_AA?8<23eW-s09tpSs#; z#o;y*7*5T*T#y(Fb+#&^ zp26dY*+MQLw7!JJ$vw?rzb3fu4}K8}pX5vOf$sO?+1|7YN@A7y?@!?ZNx?56AWIc+ zB;~|@26K>pQC|;c>#>3gh5&!?2-4VrVpG#LcDy}j3@{un*J1xU(pY_i(KvZ8dTQA6 z$z{p7+-w}xg1=fgA(S~*ShE~{rrrb`v;{^e8pP&p{%v&xAvt^7-W;3T*YFEnv0wt` zdaX19yz#Qg+w_J(?H1bN+SMmuv2%?fV(k~QpzKVfVsBH!<=RnqS#=a70?#)?qicJi zSbr3dlT-?`)t;uO5<9hY6bH1ANSHZ$UVv+r@1X5;B~?2m%4)j@7Qj-;m-~;tt_DY& zoR5SA&JiBvw!&x!u15%jk!ns+qKo3ePB*!Bp;e%L;I7%NYx(_-)n=x$$$5uhIY*;e zMt)o=hmZ}SSF!SiHQ@_=>5p;7wsXK@)+8YNb|S3+*pjnvB1fkY-}2TR9|c=COGGV`}go=p8kGP z#8@DAl+V=!zopnA-KFewk5WBWV>EG2rwf;9R53WT1EY-%60HRFlfyHydMk&cl*w`B z!3$gTMmdwWHjx)3K6Fj#CM^;vug)zL1C*d3U(BGJqY*`SJ$cF)QPL+ z{EEC!ix%7yO(u}w&}iLdX}QqK?L{d^e!5JxWk(XxpJGss%6Et1(~jtP?#Wx=Zm zRpD0}2lEC+cRwV__S*61n+k?Nq2sPxpA+SJ8P6}O(_wbId@a(gnBbe22kRe9rzHGQ z{ih-Ac8Olzi8*5RlU_%xC*^FIkY75XYR`;H`YMsuXZJBI`uV)6C7}}vIv!26+JMZY zL1RvVqs#u_tjN#sbu>?P0Fv#6I<-n|%6ICn$PUVCM5-H!uXbP_f zxsyIcPG?@TwqhPmShh!8VVE?Oa(|@&<-rt@iNU#C;}NJkbV`sdI^A%?y(RKqncj6p z8cFxdz3_P{=*_)*B>YTHGPURqg~i-ew$Pr7C?l4!VmaZjb{{{GT)$#NWa<%udTT%& zImd4LQIaSMg^rFDc1OtHpSZ%|+|i?1Q+nE(4I^kV^;@ziRiH;#1wSltl16ilQ7{?S zA%#6~1o~$tZ|x;gEEyRjHrki*(!ey>@()nRF5XAo`jrx>?1-)2{*}IiLD+mnw~M6< zk))cMn|m!J{+em1_@VFHXp$fd_jMLEZQf$SQmR+5QNO|lf8>nBf-XQv;bVgFR##kP zm@lYSU2gp`c+>1e`?h|oRwAsmr+Myf9r=3pDm6zF1Xn{sCH%QPzaR(oa)qM}6yC6e zZe%dwTd5iQzVWc!;Ud(;zAU`>hr?O2Y|BrT>FhD>7TpiSESg9OI8Fu0uAgz!%b}&$`YVXWi9qeBE9OGG zOPYo^LKSyx<%mSlYt%$)^f0C3*XKKjH&KEpZ};x`tSReIX(4M%pxb3W^kFkj@q`kA zRy4-GV6#U$2AP(u%_L^`Yu^*p+`!FmO?-m;r*xm}O@Skx$=RO-&19QA&+~v?tV*Mf7vrAX zUOu`zLw?CgD1POr^P$^l&sLg6zUaW5RV#!+HbC8}ety<3v& zl!?L~vja7wC&eD+NQ}{&!|4~}Rwx@UA1S(w{~9z2eOK!M+f^?kZ}5>ZYIU=BgY{j( zaf5$66&R~%U)5C`sTwH&l9oS<@iJ2XG5@!9s{>|Q^rfda`Tc;mJz+a+FfOcIAEkA| zs6oak7Xch%Ii}0|2ft1m+z5Czt2A;!gmEzlgP?Mg2)3+{?|Gwy^l~UtUz<-p*R!-< z8Pt7hjWHIQ#1^=kH1%GQXEDR;4hbcL8cvTRMXsmE-@fLj2 zEGa*a>>OM*&Vo|1@>|T{?`R+q@hac%=ZMYk&Pg44?lLTyi2ZYxylJawz^0LY58fzn z+y_nFC9l1y4ABD0pw&xrTE!qGRbXka46R`%2ojHNHm7hyzUFK9CA&OYtm3^b!4RI$ z?fdb|ip0gF-a+8|j98Z=;)mI!iW-~+Dbq`Xf*Isu*ovD<^^_@|6^UZ-999oV0LJAP zR+6xB;SS!tgR#ClpFD0lPP>EZu&M6jby_KT0QHK6@yBwv4d-IyQRtoJ+=Hz?qw$c# zjta8SwTY6JJf*m|HJEr49$YDef}Ua;HVI6)h(&@|20|#|ws3Un5v~28N4OqPNngd0 zsApZ+aYt_&f1$08aTg6vN?37V;%lZ+OCc#e^FeRkzbQ8q#$?g1wWS9RC&oAu!mOb6yv>38 z83m#6o7<9fW&>T3&**eBAmo?JUq zyUK-i>ooP=-oI7n0y?*Als|Ql!xy;H?a}W8kc$I}f#n+jhC2CwXgaH?IHF|@!(fBE zyL)g8?iwUWu;A_*T!Xv21_=arcMa|y971q+xy?EEJn+D*#n7{-dspqM|Nn}MpyBFz za)`@^AFrR&cgR|>;Vjyb$-wS@>wW;7=gn9pnM6z z9w>kEr&G50gJ!(uSV!xIZ?--UeTml+$l2q@;HvT@*WkEgbxpbv%0p$`5Q>CUO+bD`WM^*hoEb95$+!O3W&HEg=$mGsP?N4 z|U$(KKc1b)(~Rol<=jEjYD)j|XWYZ!_vnPI6i1u@X(2e?CqcIDcPGI8s~J7Ogc zy17UHhzPU^LD6djSrr`%Gw_838hD1?`)BW4M3f zZ&y%TGX;Ni2D4f;ob5!KMRR53VDmQ{JS%Pp^K0YrkUS(|qe%?1iZZRRM8HCVD|aeA zu)oJ+d@ho$81|WSO+kMJX`a5Fq>6Tg{!aT%37y|=c=%qfS5Od=d_+B?OuMR|x-(sQ z{*9qlSBpSiVjm0{8@|j!^XDNyQAX2Uy0!7N6n+h-Jd2Vgnc1>8(T_;B`o5n^zPFFVJigfNxf>_E+kSLKRkb^%EHSyO;`7s~pg@57;NmmV44Oqg?*p-#gXZ4Gm}U z)vY~kbZGwk{)~U_=)mMoDLtr|^cZ~vX8+!+W|Yq7Fb;Fo9TFEnrRyH1(^Kz2d>OKN zLJ|lgH@(eLf);QhH_uQqxHDE7<__Ww0oj5sUGp{x&7!HVQcJ%14#|Ca(O7DvG8`Jay_S*eP@guRoU2k3dFI)y)k&W$&@2u)exB?~2euz&y zF{?6z>J{IV3;R+aoCi&a$IsO~1qpFqRY0T z+W*G_7-ym3uvDOSPEm+|mSaMmRKoTB91W9DSE1Lb0fXpG`H+_E&Q#3p0ceA}J>FZ} zS)>?MU@VZP;Pg)=(wA-te{8a0-UuGgGT=-iar(#WTuQ&r>{HW6d+nExPsEwA!PIJG z_c*}LAI|}3{1jas7uzrRQvg@QN0JEFx2`veCP-$n+Cum@7KjUeqSOO1%3zG_JY(cq zs|p70=wMf*4Se=uH(3QVX|v(_kLgD+gum7BB8h7B&L&@xZUt`b9IIHW;Hm4-@P2N_k$NRGM1F`M$~Z6L-rnOYW@*@ ze>;A(CfiH*jstar^5@Dl+@+GzEr1u9UsFwe8u=tcU_azTlMgn}!hCUqd=y#%Fb z{OzpYlN=jCOZqG)ZHnIVk#-8RQ)9iacT5&ij_m=A}T)^M_ln+&{cx^CNGRQ8H4Ncz(78P_SLS z)JWSnlswq8wux)^j=DpZpuypORDS0dcfi5(3s=-pnWu5#uG z2s#q|BOt;}+6Gf4oy|0GCWdThGO^!MNW^|@G#p0b>3 zJ{f!A7x*64WHfz#VPI!49K-%)L7bFU8Lo_1xtblih5~0&hv6)9p30$Jx7n&PiC$lB z^6&3l>8_KvCH>f|*q`AtcreHaM9e%vwip)TWY7li2UwKQE(oW;R?r2erfDO%xStxg zzSlSm5cMR?jQ8lcmL8Qc6UsMrqQ|RG^Gp1@G-rLq*I3^{75Z#xq@k6eqBo$2#B`bJjxTg*yD;J|4G;q^FanRwjt ztnY7n1>#knN;dK;zo~95@Qp1@3Fs8iZxsETLyu!&3N1N-jmlci&V{R`t)^ zJ9WBvU`Wd}**MC5p3V*iL#_ZPYiH#P_k>@&UHnTvUejngb<=$wbdxo@aR;!~E@IQA zI9on{MEoE!S#C{xv|H`kp^ldu}5*mM2lK5 zd^|B}$3Mb~)46wc_Pyirpe(is=H8r!nl3|S170N_=Pr$=>IkY4_6e^80{yi^d3=gI z6pk@N|H;0}s)1zJ=AT$G*FU&1(vg4NJ3^e1S7= zBBqz$pO|9fYwYn^L=U6)EIRTGTbTiTpClR5{mBFoC6qutt6(s{`i?xTk|b7A2|~0r zWlaB{qGR*cF;BypTQzr>KrHjycdAWmPNJZ6-Srr+jk9dNW|D=cw~TVTE>G}O6SVKUO#FTAu8 zGmaK!L&0pIA#jf^6WijJZERFd%aUq3G$gKV8VC~EBK?*a7X%1(`Ai{@JOiB^90A_6r%1Od&ujXV!)tu_iV9V2T_ z6hgG1xV}PMij~Q+m%>^gtIpnFf>)6V;|Y^rjAEK4?E~ zQX`l!T8#SV&wG>o`bKM}9%x^TTeFCNg1D=P29z|a;KA>;-Dze-qvN@64g&o?EL59~ zZ}bPtvW?5;aP=;tSotO0Mrd=TUMRs+V+(C2iv}^iJ|6w-0OanVn@S60GRLY#1C1w3@UI-2_ zd(IC*6Uqi?wAJ7{9tWb{fhA-gn{*BGLe2C~eh>dbcBt3x*g~sOM zxRbOUSZI1)pW1s`NQrb^piro_>j7}ZNd^6hPm$pb!r=v*uHk zSEm>xHc6Z(k?HL7q)Wq4;{aTteZvL2`6jLB+J#d7jK|GXs}?`kz=L^ii2PedK6}w< zuZ(&P1N1(+d>{HhDmRQ|e;rO7_gVY-(x2BSb2DHpt{c%#w;`Ik^%rI?)$Wb$?o->D z5;;eD^~wbAryCVyc%)nyFmercn!v|f)^yBO?r4kEAgWiJA+vzGpAA{qf7_ZYt)6)g zM<(ya@ij81IOi;v6b4cL8Ebgj%qG*-z>MJYIi~!+v`G0k&%g+i!~COHX+*8G$a;fZ zBT=O@KRe)UCBdg7)nSMB7dj`8XK~jfXD7Q(YZS>9v$05X0VfI<>;^S(cTx6 z?l7TNH7peg{Yqu+buP{Kg^Pa{VWxoX`{^0XX zO@yv2eId3oA*K_lVqN^g37gUe?y z7>J5ckB!`_F?;f}wn%jrnMh>g?5gHEb5)e73%R0OI9zcPJ#lE$?*d;kMz_WEPbX%L zUPROF&JYFF8il0%3dwUbSR28D6KxA4CTH)v6|k2nX7Iv>ei~gfAZAo4Zm$pqlQ}F% zgsuOemK&{`?6TkwMZWf7dd@Q$&ymmDyT3hR8F=;V&*gnNam2=vOZxD0AG1se4C4UN z;LSB9eAt&h>sx*RXY3>JNqbgF@z^S05L6~BgtTf&yxjy95P}_H5oR6QCsY%uO9Lq0 z#$hOeXy)=czMN*&p2(QY=Fo~D9kXe+UBLAG5E-P%pjg+QhP0Jb4kxH}I%3xuR8wM= zmm!^(TZku}d`pX?$!g{Hny|sBt@nvq>>h#~<;ittPZmKF=}DN#?0UB3hfN%~==gcF z#4+G}#I9z+T5esOMMZ%%OGD26kccj|+JhkosgJ|NTEJz!0Pg)3OCqQZiyX1cg!6JtN$Qhk z5I}_2r_gM}ND2p%CU?iK4er$D*UW(BiNU%*6vwn8^bj#)%J@|rT7;NteRvM259K1c z$Dw;F*NKHxD3y(<{^ckz@^>x$An*Gbm03)`fX)IXS{N5X{F2|z2`E(Fg#cU4=qFAR z2g(IE_o_5eIpr#BWeW%ij%U#`2brGne@L&xXZ+krd+@6CI?d|SWydjAly)ah{G#8~ zCFe!Ru%A+HR!!oyq+%enWPD;={%Sf!vTik#6zQi15L9{%FEzPttiHKt8f=}k^L93) zpG)=hG7I52_HA`F)S8TzDwe(Si%%Q_ysdZu((=n->(nCeyBF##+(GN4P;zxPBbHD- zvOC?*R+o{ii63geHJxb?B$#4$-A8{=0lzd*9FVJg>WsU)c2C@tA{f9V?;ZNX1GvlY zW6WN%1fITD#KQRi6$PtTRF}s{bnZdaOuFQdpS#P--`W$M~efjS;oVkhZh$ zihk>Tgx3Y=(lBxXKMKzp6t9oA84(ZXBuz)qyU}uvC=jXSl<7;At}3#~etG)b5ClO$ zKmnXZs{0hhzlGSp9}a{l6!co}6SWq;4naFTX$8k3DHSc67z`%{RBsI5rqjO7(!|Pm zzKBo;zzISJi5iQki9?>iI{Ao**YtxKGyLcKP#8p}5LxBn>WH`DgHyLL~WAo}+s$20R z1DPSCeiIGtP{X1Gb@x(z#1i8}8UfL0x~lmc)@9FJORdALbmOdMxZ_>do6XZC)|mOK zVg}WA2eep-Pr#d+?#PakKxJU8I*(3_jTBu(TLtgeM;A}}AOrb1FyN7NNh7jqZJmu>)3Z#@S z3TR{t$GY^nq)0pyLvYdQMqL*=_jC!Egb>oY7+055R?w7n07u{qFokYB}6oo*rgmq11epr zx$J#hyT2*U;3qor>U)jWr+nrj;-Xn+<&Ft}2BrHeA>g1v$8F6av=HYK-e0ue!DuO~ zbaKn99<@K~dS}2NSXUR`_3_PuG6HsXYp~|$_s#<-FDEuH2*?mX1B2D$ht7uZP@2pE z^LQMzK4dqU3(+Qe-B!ru)!nY=`n(?7Nw&#w^~Ag(25J^iLYS1|>+!(zXZ>OtsWE)N zh56&_*481PP`O|fYG9I&NR{eFmIfwxoD8*K=!$An3nWCZsr$BPKJM_Cn|-oax3okE zu0NU4!5K@ln0m?DS8dEfst$mXDQMN|K~5(?^;f?x8o>s3(XWUYZa3w-s7eAfFi^A2O|ZQbogew{vx z+?VNEj_g1+y3yJI=a&`>g+;;%8PB`jH``xlM^mJ(+HDADd?TPFdDzJcn^6P@^p(6B zYeEaM)Zj<6-RF4o@%T~QE-kzD`}uZL^_lV?7S4+c$QVYi1^DL2W0DccYWy6Bl*;p` z6-UvuUVO+7rZ^@qK_b zfgJFJk9P5%5x4OS);MHd(fm;Q1~`g@cyoQ$nY1$b#dA0|RB_9v;4~LCkoPVJ5I?ti zn=V@f%6{T*5d*Y%8qUs}TReV)R6hP&kP(IY!_@4)k7D%Nwe?nv4wZ zQI=3Qr)i<<%LE&8v5=I7&JI4<$|D-Tep$|nOCwd^ps!AvgnFC<7~$_yhe4aHfK&-t zdcD-Gf0Zr>Er{Xm(LzE^1?=weAdLZ${18`uYl(q<+oh?;>k=h zwaOC&|G8S6EQ92u+*7V$ZU2wFj$aLg7Q6-Xc8ut;(=%ye65fH$5vJI3hxd`Y6)NN7%F8Poank~(pY@|yaUwydjKQc)$H~~N?=V$v>r|HKKi+tL=1QoIiWQ_fa9_wn0|AyD{$Z7zag%A0 z$p(*I@}*(nJDhwmlh#d-b&=7eVLga$I0b=wOL%Qcmm!UEOgxq4rQ|Ppw?4L)|szL4;%<9sP;^O1PiM^7Hx;snbT%nE_O?)&#c16hHST6;d5WT{W$(h zZ&IC)aoP!iNlCDXAv`v3z*Er6Sr@-$8SXHl&*OEx_rs+u&{-QGj>(F#sqr+PtT|x^ zx;GDkG_C?CsZZwvOZ9#gp1SRNPD)*J{*M?v#34Wz>xGBVtTz0g|C4*BnR1$aeeoM9$D zH^3geLA43AD0yEW{%TM$oM#UkW+7aJplm=rfo8!7HqU@m(!h^FZxxNA8BK+>rZkZ01BOBH^!j$*jrk2;8lc~jkf@uz#q&{y>1eKVGc9kI)Xw`!Ga zwO(wCw!8(YA4Nq5V|O2h1uH4{shlN5$_2+Ed#`}e3th6)KImiNV>c>ZFqZS&HO%9S7xO#f~ zbZhDBGn%L9M0#jqu#D#P=QhzGOyexre-cHP>Q*IJdpg8ACGb!&aVa6{miFYxVG&d`h|Z%MB- zwdIV-OukpwQftbNnJLmEB?5?v-deUR?X zx@G%khH5h({8!vM7|ITl2+6PGwT>AFHV8Tm4Q&9MIN_(>O8V!-PEQ@v7KS<`vZ*ah?opy#dfP`?rlF+}@C z*@yZtp+0>pf5_`H*LPPTv+1^{IO7PW(ho3K*D>1oIUnkZG}dUZP5G8F9J}C-E>7ta z^3$e_i&d#VUnZTQVQ`o(o;JJt*)SU3mJ9{F5+b7hhqrHgJTc6d z5S9E=WspNt3)!ABsh#56g8C9H!48mhsdd!K(1<~g9{>maJ7?z|y9Qu2Tyc^&=7hf; zr+EVqRHySzQT&fCCGV8B?!Rv@U16c5DF-1TA&=wUeCQZpHISY2I+9x6`o}ZJS=`}! ze8hPV?EM1jqdO{@l5PP~r9>&uAf&qmdar?IgV!`}+l+UnlT;Mm7PhfmlqleElm-wZ zWlR7tgc>LfB%u(0EHSw7d%N??sM`gQ*aVu#faEEhP2v_{3vE(>0E8u&D$_AWB+r70 zi22`nk=Xwj3@w!^;#ShUipexzF5ac3- zboQAxH+w72<1NQb==LNCE*w2i<7Xeufa03&mwrMoz~}PvvT*pk$qzY;sxdH@#i#li zK=iOWO8oSwj-mWa&)%N`Ko4M%39!O3dkaVb?jS(PC?G&R7B7HIlmj?|4KFgd4xs>$ zX&T_*$Z%c<{Ba}$$Vg5@#+Yc^y$~SyAl5V~vN(w^$1u?VSfaEl#Pnl+9y&1_2_@v@ zJK2YC;rRD|QE9&Q082?LK=}9hR$q}~_Q#8yGWJ#_9RrXwF>E#-VsJ#bU5Mt7)wrCSSP|I#Qx+n_{#CFr$mSNcAK)e4 zK}zhXrygMn@Bcy}^a{~h!t9Yb`1=DPUJ835tk{Imd;J!)&@>vBPzFYd(kC(%B9 z?Q(fg4ZtKp=O@i^#S9R^W%B3x;32UVB-}Gj(Fx6o4L~Nbco-BIJun+MPBMpvUz2ZoZMJ`tcDkX0Q_uf$%(d!C`8Cag;oOKPoNL8m3BQ;t7Ucx zH%;zRQXt5iLXipgcjr&bP8ET5ld@&8Amc#*d8te}gpeIQzVmusU3h#fJ^+NNzh>l2 zJPN48ax2Y_##MyPKc7Ejr8iSs*sQl}ky#kPe*jh)TwBTV&TbiXn+zRN?CIsPBLOKO z6Y(lv`H1LjRSe9ONTizg6AmH?6O2lK4-Da1Lyzgn2Cl3*@xy8i0ELMxv1qheeur+w zxgITeND{%A7=3NDN&jg1C>J!8|F&qKFbQrb^V9#I(z5Dfg%S0tUirzOUUtD07Bb{^N`yr4;D}WNr zqv%^oUI6v5=1AmK%D?_hF3Iqo7#~kzNGqoHHnT=rpNv{inuQdYl0vxxEV4{;tg-goVh0nG_i)T?+>h{AWFBfp<8k zvb{BZA0`n(F%pt2y$gVYGMXslV0ehmQ%p1M(lYKNHi9{vV`ZmAaFzhH&Bs}+qXhp^ z;4dBcK6B%}-!ewUK}8V&R0yLG@eH8mk70jxF#|X(R=!%TE~SI|4k~qe^HIWxiV(mp z(2ZnLHJz8Yr7k?+kEhTM6Is4oE1yt!{el#xA5KV_L&zt84Q;p4d2*Xf&WcU8 z0Y@U{;CX##5#M?0hW^4r{aAp-`QvYILC!m-sF$P9m<2d+oRT>JFnZUSV}Se!4F}g& zfLLWSn+x1Wklw)7^p0Nb#>WET)Z?YZ!g3Kz#F_kQMgyKYT)A>_kgo(pnd0UUoAbVc zs7R>@)9ZoT)|gL_gCXDd&%oVEbTk+Y+&=Go7z`N+kc?VL+|)uQHR#KHsVc4Bs0uE# zHr0D1#%9#$KYEnt-@L4R`E%BbZ*Ut3J4Sa)fSfj>DbtM)tW;M1Kr7XzAt;P7o>Ce` zT_=`+OU{P^ZLxb9n-csjcxuf8H6sGhgt)uYtS=LjlmPKaU0SQW#2}nZMD%-CP!Mp* z@o1jyDbNBU5M|&US5UZC6-aP;?rW`i{m|2bpPvbn7 zGD}8(;9a^RHMe-Pg5sE$4yeTgjP zh-@2_j1{;0xk2=%<%9WB_P!Y>^nn{XauP3*6BZ{xQMaA7xR(R9@hT*$Z_5A>H6qa( z0(kcfh-V5kqxY}|9DL+}q$LG~-1Ttoos<3v4QDouMlq{I9#Q!5fKoP-+g4%TU4Q^I zcPRli@LL>M=F6O=Ohq~iUiewKGS%wyp3FgVIGlisouRV*t!Zge^({~!Ov2}E0goA2 zkQf-GOJ0o=sAq`%d#T3wMd%v-)#*V93t{uVPXJoSdyOJsGkd=s2Hx z0MB%o96JcO>weVG#eIQ3^m0GwE5*FE#aTqJe*S=_iDbXr_U**!>yC|V@ZJG*bOtat z1c)m|fPLX|-Rtba?L@o_8v#eirH)q(!@r{y$jSoqw8Bg;9{%PY{fY-tnqQ=_>m^#c zN@T#uki)Wf@+$I&g=$0bMYmp@Wd9B+V_*|=dc3yu*bb*H1W>OGyd1ERcW=u7fn05a zWyvrgpN9~CDz=aJ#BRSexshzDU|W;=y#SSVJ!LEy7)fPiQ8_SOST1>R0)WncKBylvF3mic$i zpEm)eTCa)pZ@wJnll*7xS16p=OJ`8I0iE$_$YBTq{wD|;%iVs%KNr0*gn|R?C zB^hQ57#c7a*e{9uRgGTVKYM7tp(@^LzO2l4J(G?wNI5(@k{OpU=;aCf>6%H9MSuAi znbuVER!6HRq+>;ysobZeKN<50>IspGgF~LXd_#Hbw=NG&l={uew_8Y91E zo&8L6O;D*X>6?LW%k|Sp=NF~Xp9Wj>e8q?aAQ%`pQ;A+WKBwIh3F*O+uWoD1$b_5- z8BZ)nQjtnTskyf&$yHs;^ha0DR7~a3-7n27Qr8B;?2bv=vLd%JD?J8p^NakZ;;-74 zgg6W=9lM0LOOS37+ZObXC*Y?JsQbO&>!OauA0>Nw)y4-#G&lp=qaUnjrz9T!)~Oaf zWHU3ix~rx%(eS9wd?~UJCe$Q2vK*eAOx@~}qPt~D?1)7BQZ*hIRjx)_y+H-vhiUq( zS?h-0;hmQKAzjNdA-jX-paB;E-=1(q;krNr56}-On*u5i#b4}E<@hdlXXD6!?I)%w zEqD(usSIFXrADvJf5vrWLh#s)q_j^uH4LfqBpx1QynqK@k!#&jHy+!$YAr{HNb200 z1pK1)JraXn<$`V~ASEGS6T-tOO{8XYJC_6w(CELM|TLZo5Ng!gb+!PuN#NT`9cWdb9>&PZp$SW(7?reeHv1PO2P#ZTzfKaO3|< zMh!#}nTp?D7x)2D*XlD|4D03(Fve1!GC9ADvNyH)9O3^<#~>CJMz{N3C+9G9!fdj- zV5?!S`;?L1BKPBCVuy7-(NJrhCF1fSq}NbO`V|ulqH`mm_16h#6cVO5!XqBKzn?+; zPb*gRhbj#TgMa*!$xO~ZaIM~Fyi6N6(thP-;O+K30~LQRNUDD8nBxubjSrN}Yhpc{ zqy*6_Zq=5SAXbFMUTFVzEW4=FbJG-PKewk^(DdKDq+VyZ=FQb`JqB+~d*J{r~WR0E73uIZY?`?!P2> z0FVc~&^siO4nT871~BdyrSUHfa>16~=ERoMjrmcTykapgr=4#|N7}iU(*R2?jdh{{ z{}G^>@{%+jny_UzUcidIyx&^{@c}I2*Y6HJBA_0rJge$ItY@jf0f`kX!pP=z>f-UY z!gQwCmET8oX_eY?R(>Ow_ERK2y^%R8ub)&g{aGfG%F{!>ZZq|O~u!TFo^LBu9ge8j~|^vV>UWQ zzk^~Ui-Ldr=(`N~1$SvX`zyO>V#DVx8bfU2_r%xLy##r&Lbpk2xzUhCneH=qW#g^8 zPyGl0T$Goh`YO`)bAjilQN)!n=Lgd-!wE2DOk$E%uV0~A?7_sAjJUtk9Q$16vfCPe zAJmAO*j+L)V*zvLQsEP&j;zr@d9L?+sUV-{wIGS~1WvtM+~e6uuJh zd{tF(gilrrY)O!q&=`8#>XZKt(aOrmgk`TtM90@MPFBAVtyPs=Z`dVzRkLRZd>45 zxrHJ~41JUyZE=nnb;yX%{o0+dCd(2%-$;M(b_+@^IP5JpS-qVk`s#ETmDi#G2%G-iPN%{RFJ|DdEE&HK~3M=1^I?1&A*P%Z} zzowz^G8db;eUGsOgB|_U7;X8d8~aXbPp&3+|LkIG>!4qVe6htv$99g*330b^Q1Wox zcCj7S(%u3P-SaHJW44$-xN8|j>_w-a4Xc^9!wX|i#TZEbE_lgt$k^+a4z&uPU?}dW z9I6|^o0}@({j~cRmrQvqBa!Onhzkv@Luy(z^NsTMW#g5b>>xk79XZXjw@y`~dxYFv z)y%YL&vg>)z|hZOryvR9hGz9_y~}yJVF;YJ-dL2!s-Q$7Ni(SPgw&4Og{fl+GtVBp zkD12t#ba9uw%Yh8k{19;p&_sYcdy)hIHHYPSQBf$i%Kco&RGGK;-XRdS+aVjmlVo| zrdY{Gl_n;W0dDSBR9WS(BK^$;*Wh!=zENP?DY3p`7BI*Q7p63@JZwFlydghzGd{)j zgaMD-Z)8CR{788;bFda30r7w5}fRqK50FWeDfuNKm3+ESbCR7Zq)o+^a2| z$aBcaq?;q2xO}CkIe9v^z)3N8qIEpHJTCw$p=Y0$^> z8|aWRl~m7||L#qo9^?%-Co>$zBQzxWyQOS9J!4{;=o~qI6%+u;N#=FyU1eiGG5qtW z^FzZ)Jnnbr_oHFm7B+@MIE3#x>r6|N4=rDjl%eX+PgBla%BX}r+NL@#yF4dL^tH}B zJGFgEQRNA^?c}S%FDlkZn{o8^vmco?$Fgn`aIn_Zg@?gtB4*TJ=!DW<^oGW@Zgw>7 zR`Tn}S(M#!50+N@$huoa;nYs2=8qrD7xdF?XU!m(6z*A6jHg}nnV6bIh|~o=e=Mc$ z2*H8tz0z%ny8DOzHs|B~T&D^>t1~r6F6;)%Q3YdbF$zuhq8{6lkl@i8=HDIKbgIe6 zZG2hGdtv;P=kjyRAX>hMLQ^N_?t}w$p6n2+Z3pI=0*~9-e;A#`)sZsi4V|7cP5l~E zr9H6PVU3C};Lt4e?G{zo4i6m_?>TNk0N^7-Jr3bs9-q{&)eiAhI zzQ&a}>?uscb1$%Eu0JS^T=Ne`7qnTxsDo-#(N@~v7Zj;#on#erep3f)JPu~^c(LXij-py)KLCON)hGr#*>JXlW@X~2cNcAz&y;A8H+kW zf|044;~Fr5LzVpH38{A1C@B&XFTO_#3g^1yi?@P3F{%d{-a7S#py2_Ye75zkR7fZw zJMizUe_E|7l=jxQtgOer|1O0Dn|)6ASijN+(}PArv}fpt@`65za{M7rOT7CU`;qan z)yJ=1xN+jF{rf!kFw}$2otprMYh}wRi%tf?g4TH(E|6-uL&nO=LX4iU4k1Bye>9lD zE%?w^_v41^OPBbc%<}!9e2PI--u>JfImo|klvk0D*xx(M};$pS62Fa=P3w>quZbVO~8Rr>o5447FSg_1mGBR9|qQwod z=nB>%s(+dCHbfI)qXz^}KK`>HymV?P26?r2pM|;l3jZoaGLFTQ7`%dfLYzasIEJeC z6J{VpDCnYGpR=4i!WO)B^f-Q>U9x^*d`L!H&F)=~4{p&NW3wNg#C8vNF{|<)FGE0iwZSja!bNB}XK2`l zE?|p7ivGic4tHq5bn@L?_DF>!iuLG^L*lp$#)YB$^zm)mbqUH4xPo;pL0szXJ(5$M3kQX;<1F{MMTuJ%(PxYq^AYXz_dyQ4Y*T+pQ16O20=FlM{dkGKjDQNyF~h@uJ;Z&H*CT*tqGDI|J4879Aa6t8|-}j z>?h&QLoISL`wiCJ@pFsDP<0!mBwJ8I1344Oyp`qv&15#9%? zP5387XSOqZ&vj1w)tY@Ov^nVNQrvb1BgvktorHQ(#zvYUI_$w(JbGi72v*=2@(A-f zlEjnr7NasEf5?cFiKPJ9lVW-wXI^OT?N<>o1jZ@Yl!hHRGEI`fEjWgfiuU+S_$aE} zqw$w`WuxlLo56_Hbb?7X?BZrT<_byScYucG@hrisd|v0oCnURAq#8Q0N27ModmjtU z5DsC>!~ic42~fSyb0DH%wss(L(~d&-y7I#h@&xk582wq<4YSrNiVKj;jh24THHRFr z9(?};@wzVO`w1agoOY?XdupB)O~CCm>yp-~1&72yZRxo(1!rgZOL6;$ia87eXa(km zc78RlAN_3g*baGSHuET=P_AZV0V34Ec~My>SyjUyok@ebdkYf@6J5LBwnVt*5aPpRG(;DogHAMr_`p-rlhu+E9^@^aLwAsm z37!nu6#Ud6#gtCkois>P5n`fevo6oKBa;&z&%5`lhgN=bv*uY3NEGs)VHEs091)YG zbpLQAi`xM=tc#DkR&$JGijXn$EQwLY7(HIzLFW>@4b~~E0~wO=&~|{Y=^w}n6WL7k4iAl zN#UXKAV2URSHtd|+=uv=$&>h~YH#ju|E}fds2-Xdd_Snh`zjp__KhT!q}^T)q~zMz z2r<-|3K~xC6?a;)DJ?w}o4gGN6Cl1>z%2vr z6kc=|2c&+dB@+Mj((#ezJWOqk80P5=4kj|<$eQW51axKdcCT&fCpDk05(aM}UHv=H zrX#1|@61(yCa8un{eAu%2^jjB#wA|8-uAq<9vyn-#)mrthtKUn3!dxq^uWdyqfZtsUe^^T$hfi|*uR536y3p3ybGYv>T-IBO{=W{k62KvK-W~c!{FY6r725)n! zeT6{pKF1V|tQ08M$cSHZdn;`Y{GJ@$wV`q-uIOA9 zw`pm;UJ3`Tn8iz9!*+qOP+V|*B-m|#8*sjz!f_ zli@}q*K+E-JM5G35URXYhCSe17?ihQtaVEUDsRCFtv03qx0srZ<$w3B`E;gWt`xWN z7NOS>nJjv{-vmX@-W>Jsw|)AO`wbGF=Y5`L3GTjT5%OcTj$dW;FaDm1w`bkcIUT;F zQa5BlhqwOTkWJUuP3vAY2N?6ujxB8c+wktvlW%RCl9s-|vHoP(8-2Ugw@jz~y~mLE zHFLvz_HWzo>S!BJ3taJ|o8MF{Otv-NbAbsH@Xql`Qnv0KslZtRt{kOZrSH{%=krX_Uauo>&oQgED`~#-w-+I7zpYBwO?{Vee|OEz z<;#|No{QSFJ^eoL3YeFdzb`C${XOk37zI zf~o>Ixp>$uaVl`xFbEx318gVG*!nagAEH6%gEO$(^Jha~gep|PKpU8K({e4Z!ESQ; z1G?4TK1U?16Kp~Q%NbA+5EMpUXO@geCy&VOV|u literal 0 HcmV?d00001 From 3d5eb3e4a3b64b9656ad9b9e59be64803b89f97f Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Wed, 21 Dec 2016 15:05:31 -0800 Subject: [PATCH 2/7] design-proposals: self-hosted-kubernetes updates Address questions around the kubelet node agent and load balancing and control plane nodes. --- contributors/design-proposals/self-hosted-kubernetes.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/contributors/design-proposals/self-hosted-kubernetes.md b/contributors/design-proposals/self-hosted-kubernetes.md index 9cca176229b..a4d1fae0a4e 100644 --- a/contributors/design-proposals/self-hosted-kubernetes.md +++ b/contributors/design-proposals/self-hosted-kubernetes.md @@ -45,6 +45,9 @@ At the end of this process the bootkube can be shut down and the system kubelet ![](self-hosted-final-cluster.png) +There are a few things to note. First, generally, the control components like the API server, etc will be pinned to a set of dedicated control nodes. For security policy, service discovery, and scaling reasons it is easiest to assume that control nodes will always exist on N nodes. + +Another challenge is load balancing the API server. Bedrock for the API server will be DNS, TLS, and a load balancer that live off cluster and that load balancer will want to only healthcheck a handful of servers for the API server port liveness probe. ### Bootkube Challenges @@ -68,7 +71,7 @@ A simple updater can take care of helping users update from v1.3.0 to v1.3.1, et The kubelet could be upgraded in a very similar process to that outlined in the self-hosted proposal. -However, because of the challenges around the self-hosted Kubelet (see above) Tectonic has implemented an alternative scheme that side-steps the self-hosted Kubelet challenges. First, a kubelet system service is launched that uses the [chrooted kubelet](https://github.com/kubernetes/community/pull/131) implemented by the [kubelet-wrapper](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html) then when an update is required a daemonset updates the kubelet-wrapper configuration based on version annotations and kills the kubelet PID triggering a restart. +However, because of the challenges around the self-hosted Kubelet (see above) Tectonic currently has a 1-4 self-hosted cluster with an alternative Kubelet update scheme which side-steps the self-hosted Kubelet issues. First, a kubelet system service is launched that uses the [chrooted kubelet](https://github.com/kubernetes/community/pull/131) implemented by the [kubelet-wrapper](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html) then when an update is required a node annotation is made which is read by a long-running daemonset that updates the kubelet-wrapper configuration. This makes Kubelet versions updateable from the cluster API. #### API Server, Scheduler, and Controller Manager From 71d92d83db1a73038f39b0164f5a77835b5cc60b Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Mon, 16 Jan 2017 20:43:20 -0800 Subject: [PATCH 3/7] design-proposals: add sections on HA address some feedback requesting HA notes --- .../design-proposals/self-hosted-kubernetes.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/contributors/design-proposals/self-hosted-kubernetes.md b/contributors/design-proposals/self-hosted-kubernetes.md index a4d1fae0a4e..dbd2dc143a8 100644 --- a/contributors/design-proposals/self-hosted-kubernetes.md +++ b/contributors/design-proposals/self-hosted-kubernetes.md @@ -5,9 +5,9 @@ Last Updated: 2016-12-20 ## Motivations -> Running in our components in pods would solve many problems, which we'll otherwise need to implement other, less portable, more brittle solutions to, and doesn't require much that we don't need to do for other reasons. Full self-hosting is the eventual goal. - -- Brian Grant ([ref](https://github.com/kubernetes/kubernetes/issues/4090#issuecomment-74890508)) +> Running our components in pods would solve many problems, which we'll otherwise need to implement other, less portable, more brittle solutions to, and doesn't require much that we don't need to do for other reasons. Full self-hosting is the eventual goal. +> +> - Brian Grant ([ref](https://github.com/kubernetes/kubernetes/issues/4090#issuecomment-74890508)) ### What is self-hosted? @@ -77,9 +77,17 @@ However, because of the challenges around the self-hosted Kubelet (see above) Te Upgrading these components is fairly straightforward. They are stateless, easily run in containers, and can be modeled as pods and services. Upgrades are simply a matter of deploying new versions, health checking them, and changing the service label selectors. +In HA configurations the API servers should be able to be upgraded in-place one-by-one and rely on external load balancing or client retries to recover from the temporary downtime. This relies on Kubernetes [versioning policy](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/versioning.md). + #### etcd self-hosted -As the primary data store of Kubernetes etcd plays an important role. Today, etcd does not run on top of the self-hosted cluster. However, progress is being made with the introduction of the [etcd Operator](https://coreos.com/blog/introducing-the-etcd-operator.html) and integration into [bootkube](https://github.com/kubernetes-incubator/bootkube/blob/848cf581451425293031647b5754b528ec5bf2a0/cmd/bootkube/start.go#L37). +As the primary data store of Kubernetes etcd plays an important role. Today, etcd does not run on top of the self-hosted cluster. However, progress is being made with the introduction of the [etcd Operator](https://coreos.com/blog/introducing-the-etcd-operator.html) and integration into [bootkube](https://github.com/kubernetes-incubator/bootkube/blob/848cf581451425293031647b5754b528ec5bf2a0/cmd/bootkube/start.go#L37). + +### Highly-available Clusters + +Self-hosted will make operating highly-available clusters even easier. For internal critical components like the scheduler and controller manager, which already know how to leader elect themselves, creating HA instances will be a simple matter of `kubectl scale` for most administrators. For the data store, etcd, the etcd Operator will ease much of the scaling concern. + +However, the API server will be a slightly trickier matter for most deployments as the API server relies on either external load balancing or external DNS in most common HA configurations. But, with the addition of Kubernetes label metadata on the [Node API](https://github.com/kubernetes/kubernetes/pull/39112) self-hosted may make it easier for systems administrators to create glue code that finds the appropriate Node IPs and adds them to these external systems. ### Conclusions From de711ac20424e5a847227c28ee0675eb7e3213a1 Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Tue, 17 Jan 2017 09:25:59 -0800 Subject: [PATCH 4/7] design-proposals: fixup nits from pires Good feedback on HA setup notes and some grammar fixups --- .../design-proposals/self-hosted-kubernetes.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/contributors/design-proposals/self-hosted-kubernetes.md b/contributors/design-proposals/self-hosted-kubernetes.md index dbd2dc143a8..bc80f58a2b6 100644 --- a/contributors/design-proposals/self-hosted-kubernetes.md +++ b/contributors/design-proposals/self-hosted-kubernetes.md @@ -1,7 +1,6 @@ # Proposal: Self-hosted Control Plane Author: Brandon Philips -Last Updated: 2016-12-20 ## Motivations @@ -18,7 +17,8 @@ The advantages of a self-hosted Kubernetes cluster are: 1. **Small Dependencies:** self-hosted should reduce the number of components required, on host, for a Kubernetes cluster to be deployed to a Kubelet (ideally running in a container). This should greatly simplify the perceived complexity of Kubernetes installation. 2. **Deployment consistency:** self-hosted reduces the number of files that are written to disk or managed via configuration management or manual installation via SSH. Our hope is to reduce the number of moving parts relying on the host OS to make deployments consistent in all environments. 3. **Introspection:** internal components can be debugged and inspected by users using existing Kubernetes APIs like `kubectl logs` -4. **Cluster Upgrades:** Related to introspection the components of a Kubernetes cluster are now subject to control via Kubernetes APIs. Upgrades of Kubelet's are possible via new daemon sets, API servers can be upgraded using daemon sets and potentially deployments in the future, and flags of add-ons can be changed by updating deployments, etc. (An example script is in progress.) +4. **Cluster Upgrades:** Related to introspection the components of a Kubernetes cluster are now subject to control via Kubernetes APIs. Upgrades of Kubelet's are possible via new daemon sets, API servers can be upgraded using daemon sets and potentially deployments in the future, and flags of add-ons can be changed by updating deployments, etc. +5. **Easier Highly-Available Configurations:** Using Kubernetes APIs will make it easier to scale up and monitor an HA environment without complex external tooling. Because of the complexity of these configurations tools that create them without self-hosted often implement significant complex logic. However, there is a spectrum of ways that a cluster can be self-hosted. To do this we are going to divide the Kubernetes cluster into a variety of layers beginning with the Kubelet (level 0) and going up to the add-ons (Level 4). A cluster can self-host all of these levels 0-4 or only partially self-host. @@ -37,10 +37,12 @@ The target audience of this document are others, like [kubeadm](https://github.c ### Bootkube -Today, the first component of the installation of a self-hosted cluster is [`bootkube`](https://github.com/kubernetes-incubator/bootkube). A kubelet connects to the temporary Kubernetes API server provided by bootkube and is told to deploy the required Kubernetes components, as pods. This diagram shows all of the moving parts: +Today, the first component of the installation of a self-hosted cluster is [`bootkube`](https://github.com/kubernetes-incubator/bootkube). Bootkube provides a temporary Kubernetes control plane that kicks tells a kubelet to execute all of the components necessary to run a full blown Kubernetes control plane. When the kubelet connects to this temporary API server it will deploy the required Kubernetes components, as pods. This diagram shows all of the moving parts: ![](self-hosted-moving-parts.png) +Note: In the future this temporary control plane may be replaced with a kubelet API that will enable injection of this state directly into the kubelet without a temporary Kubernetes API server. + At the end of this process the bootkube can be shut down and the system kubelet will coordinate, through a POSIX lock, to let the self-hosted kubelet take over lifecycle and management of the control plane components. The final cluster state looks like this: ![](self-hosted-final-cluster.png) @@ -71,7 +73,7 @@ A simple updater can take care of helping users update from v1.3.0 to v1.3.1, et The kubelet could be upgraded in a very similar process to that outlined in the self-hosted proposal. -However, because of the challenges around the self-hosted Kubelet (see above) Tectonic currently has a 1-4 self-hosted cluster with an alternative Kubelet update scheme which side-steps the self-hosted Kubelet issues. First, a kubelet system service is launched that uses the [chrooted kubelet](https://github.com/kubernetes/community/pull/131) implemented by the [kubelet-wrapper](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html) then when an update is required a node annotation is made which is read by a long-running daemonset that updates the kubelet-wrapper configuration. This makes Kubelet versions updateable from the cluster API. +However, because of the challenges around the self-hosted Kubelet (see above) Tectonic currently has a 1-4 self-hosted cluster with an alternative Kubelet update scheme which side-steps the self-hosted Kubelet issues. First, a kubelet system service is launched that uses the [chrooted kubelet](https://github.com/kubernetes/community/pull/131) implemented by the [kubelet-wrapper](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html). Then, when an update is required, a node annotation is made which is read by a long-running daemonset that updates the kubelet-wrapper configuration. This makes Kubelet versions updateable from the cluster API. #### API Server, Scheduler, and Controller Manager @@ -85,7 +87,7 @@ As the primary data store of Kubernetes etcd plays an important role. Today, etc ### Highly-available Clusters -Self-hosted will make operating highly-available clusters even easier. For internal critical components like the scheduler and controller manager, which already know how to leader elect themselves, creating HA instances will be a simple matter of `kubectl scale` for most administrators. For the data store, etcd, the etcd Operator will ease much of the scaling concern. +Self-hosted will make operating highly-available clusters even easier. For internal critical components like the scheduler and controller manager, which already know how to leader elect using the Kubernetes leader election API, creating HA instances will be a simple matter of `kubectl scale` for most administrators. For the data store, etcd, the etcd Operator will ease much of the scaling concern. However, the API server will be a slightly trickier matter for most deployments as the API server relies on either external load balancing or external DNS in most common HA configurations. But, with the addition of Kubernetes label metadata on the [Node API](https://github.com/kubernetes/kubernetes/pull/39112) self-hosted may make it easier for systems administrators to create glue code that finds the appropriate Node IPs and adds them to these external systems. From 916dfa89f67a2057218c24a3df632d5e1b2811f0 Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Tue, 17 Jan 2017 09:32:35 -0800 Subject: [PATCH 5/7] design-proposals: note on kubelet exit-on-lock-contention From https://kubernetes.io/docs/admin/kubelet/ --- contributors/design-proposals/self-hosted-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributors/design-proposals/self-hosted-kubernetes.md b/contributors/design-proposals/self-hosted-kubernetes.md index bc80f58a2b6..c10d1a2ecfd 100644 --- a/contributors/design-proposals/self-hosted-kubernetes.md +++ b/contributors/design-proposals/self-hosted-kubernetes.md @@ -43,7 +43,7 @@ Today, the first component of the installation of a self-hosted cluster is [`boo Note: In the future this temporary control plane may be replaced with a kubelet API that will enable injection of this state directly into the kubelet without a temporary Kubernetes API server. -At the end of this process the bootkube can be shut down and the system kubelet will coordinate, through a POSIX lock, to let the self-hosted kubelet take over lifecycle and management of the control plane components. The final cluster state looks like this: +At the end of this process the bootkube can be shut down and the system kubelet will coordinate, through a POSIX lock (see `kubelet --exit-on-lock-contention`), to let the self-hosted kubelet take over lifecycle and management of the control plane components. The final cluster state looks like this: ![](self-hosted-final-cluster.png) From a06518ffcec09edd13691db6fd71737f2f52ab55 Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Thu, 19 Jan 2017 20:46:53 -0800 Subject: [PATCH 6/7] design-proposals: add feedback from roberthbailey - fixup some grammar mistakes - add skew of control plane to known issues - add links dropped from original proposal --- .../design-proposals/self-hosted-kubernetes.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/contributors/design-proposals/self-hosted-kubernetes.md b/contributors/design-proposals/self-hosted-kubernetes.md index c10d1a2ecfd..81fddf0a3bb 100644 --- a/contributors/design-proposals/self-hosted-kubernetes.md +++ b/contributors/design-proposals/self-hosted-kubernetes.md @@ -24,20 +24,19 @@ However, there is a spectrum of ways that a cluster can be self-hosted. To do th ![](self-hosted-layers.png) -For example, a 0-4 self-hosted cluster means that the kubelet is a daemon set, the API server runs as a pod and is exposed as a service, and so on. While a 1-4 self-hosted cluster would have a system installed Kubelet. +For example, a 0-4 self-hosted cluster means that the kubelet is a daemon set, the API server runs as a pod and is exposed as a service, and so on. While a 1-4 self-hosted cluster would have a system installed Kubelet. And a 02-4 system would have everything except etcd self-hosted. ## Practical Implementation Overview -This document outlines the current implementation of "self-hosted Kubernetes" installation and upgrade of Kubernetes clusters based on the work that the teams at CoreOS and Google have been doing. The work is motivated by the upstream "Self-hosted Proposal". +This document outlines the current implementation of "self-hosted Kubernetes" installation and upgrade of Kubernetes clusters based on the work that the teams at CoreOS and Google have been doing. The work is motivated by the early ["Self-hosted Proposal"](https://github.com/kubernetes/kubernetes/issues/246#issuecomment-64533959) by Brian Grant. -The entire system is working today and is used by Bootkube, a Kubernetes Incubator project, and all Tectonic clusters created since July 2016. This document outlines the implementation, not the experience. The experience goal is that users not know all of these details and instead get a working Kubernetes cluster out the other end that can be upgraded using the Kubernetes APIs. - -The target audience of this document are others, like [kubeadm](https://github.com/kubernetes/kubernetes/pull/38407), thinking about and building the way forward for install and upgrade of Kubernetes. If you want a higher level demonstration of "Self-Hosted" and the value see this [video and blog](https://coreos.com/blog/self-hosted-kubernetes.html). +The entire system is working today and is used by Bootkube, a Kubernetes Incubator project, to create 2-4 and 1-4 self-hosted clusters. All Tectonic clusters created since July 2016 are 2-4 self-hosted and will be moving to 1-4 early in 2017 as the self-hosted etcd work becomes stable in bootkube. This document outlines the implementation, not the experience. The experience goal is that users not know all of these details and instead get a working Kubernetes cluster out the other end that can be upgraded using the Kubernetes APIs. +The target audience are projects in SIG Cluster Lifecycle thinking about and building the way forward for install and upgrade of Kubernetes. We hope to inspire direction in various Kubernetes components like kubelet and [kubeadm](https://github.com/kubernetes/kubernetes/pull/38407) to make self-hosted a compelling mainstream installation method. If you want a higher level demonstration of "Self-Hosted" and the value see this [video and blog](https://coreos.com/blog/self-hosted-kubernetes.html). ### Bootkube -Today, the first component of the installation of a self-hosted cluster is [`bootkube`](https://github.com/kubernetes-incubator/bootkube). Bootkube provides a temporary Kubernetes control plane that kicks tells a kubelet to execute all of the components necessary to run a full blown Kubernetes control plane. When the kubelet connects to this temporary API server it will deploy the required Kubernetes components, as pods. This diagram shows all of the moving parts: +Today, the first component of the installation of a self-hosted cluster is [`bootkube`](https://github.com/kubernetes-incubator/bootkube). Bootkube provides a temporary Kubernetes control plane that tells a kubelet to execute all of the components necessary to run a full blown Kubernetes control plane. When the kubelet connects to this temporary API server it will deploy the required Kubernetes components as pods. This diagram shows all of the moving parts: ![](self-hosted-moving-parts.png) @@ -79,7 +78,7 @@ However, because of the challenges around the self-hosted Kubelet (see above) Te Upgrading these components is fairly straightforward. They are stateless, easily run in containers, and can be modeled as pods and services. Upgrades are simply a matter of deploying new versions, health checking them, and changing the service label selectors. -In HA configurations the API servers should be able to be upgraded in-place one-by-one and rely on external load balancing or client retries to recover from the temporary downtime. This relies on Kubernetes [versioning policy](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/versioning.md). +In HA configurations the API servers should be able to be upgraded in-place one-by-one and rely on external load balancing or client retries to recover from the temporary downtime. This is not well tested upstream and something we need to fix (see known issues). #### etcd self-hosted @@ -99,3 +98,4 @@ Kubernetes self-hosted is working today. Bootkube is an implementation of the "t - [Health check endpoints for components don't work correctly](https://github.com/kubernetes-incubator/bootkube/issues/64#issuecomment-228144345) - [kubeadm doesn't do self-hosted yet](https://github.com/kubernetes/kubernetes/pull/38407) +- The Kubernetes [versioning policy](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/versioning.md) allows for version skew of kubelet and control plane but not skew between control plane components themselves. We must add testing and validation to Kubernetes that this skew works. Otherwise the work to make Kubernetes HA is rather pointless if it can't be upgraded in an HA manner as well. From d08603423ffc23b1796f53848c43ee1c152a351f Mon Sep 17 00:00:00 2001 From: Brandon Philips Date: Fri, 20 Jan 2017 07:07:41 -0800 Subject: [PATCH 7/7] design-proposals: add note about self-hosted not replacing existing methods Thanks to @timothysc for pointing out that I didn't make a note about the intentions of this proposal with existing install/config methods. --- contributors/design-proposals/self-hosted-kubernetes.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/contributors/design-proposals/self-hosted-kubernetes.md b/contributors/design-proposals/self-hosted-kubernetes.md index 81fddf0a3bb..cafbc5962c9 100644 --- a/contributors/design-proposals/self-hosted-kubernetes.md +++ b/contributors/design-proposals/self-hosted-kubernetes.md @@ -26,6 +26,8 @@ However, there is a spectrum of ways that a cluster can be self-hosted. To do th For example, a 0-4 self-hosted cluster means that the kubelet is a daemon set, the API server runs as a pod and is exposed as a service, and so on. While a 1-4 self-hosted cluster would have a system installed Kubelet. And a 02-4 system would have everything except etcd self-hosted. +It is also important to point out that self-hosted stands alongside other methods to install and configure Kubernetes, including scripts like kube-up.sh, configuration management tools, and deb/rpm/etc packages. A non-goal of this self-hosted proposal is replacing or introducing anything that might impede these installation and management methods. In fact it is likely that by dogfooding Kubernetes APIs via self-hosted improvements will be made to Kubernetes components that will simplify other installation and management methods. + ## Practical Implementation Overview This document outlines the current implementation of "self-hosted Kubernetes" installation and upgrade of Kubernetes clusters based on the work that the teams at CoreOS and Google have been doing. The work is motivated by the early ["Self-hosted Proposal"](https://github.com/kubernetes/kubernetes/issues/246#issuecomment-64533959) by Brian Grant.