From 6b3e5f9051a8a6b809d885c32136a47419fdcd07 Mon Sep 17 00:00:00 2001 From: codebien <2103732+codebien@users.noreply.github.com> Date: Mon, 5 Feb 2024 19:52:00 +0100 Subject: [PATCH 1/2] Added more sections and details --- docs/design/020-architecture.png | Bin 0 -> 45240 bytes ...0-distributed-execution-and-test-suites.md | 154 ++++++++++++++++-- 2 files changed, 138 insertions(+), 16 deletions(-) create mode 100644 docs/design/020-architecture.png diff --git a/docs/design/020-architecture.png b/docs/design/020-architecture.png new file mode 100644 index 0000000000000000000000000000000000000000..8a14b447608b19e484bfdf28cbb638adefb47099 GIT binary patch literal 45240 zcmZ5{WmFu?8ZFKQcL^@Rf+x7Uy9amo;7)L7aCZnE+%33U@Zf{{;Qr>^bMJa@y885As?YY`6{)N!g^KhM2?`1dRYqD|6$%RG5ef<#4uAl;Q?LD*5^{ldRh1Hjs+l4_ zf`TH4k`WhC_cS=whu2Ql!S}Djtz)ahuET&mHDO!S6t|Oz(Z}62?9f%I!)0@4Iz9hp ztDr10pbcBcx(Ee}``45SWo$uUG3A%*718Fl;e)tw@%KL4_U`rh{$lGlfE6YN zimLbjUL=165+?Z;>Bw=0U>d;@KuZSx?}7nf9X*Kse;)nMiv)x)^59rIkrm?q`{cjx z$p~KjKX3o%UhfEo5zx^Ld2IjxA66dhh5rBFlMI7O@}<&M<_!PeC(Izi98Bjejk2~M zE^TZW(7I}z{rUYHhLe+%mnJ!1dYS_HH(;;Srrt%0rwQ`@seQop73J)2^&1)62k0NUDSA{le$gBazi z&>S|rfr?T$&Z32K9Wh;wG>=2wzwOBWHFsbirSQPU__$nxSL8C$shO8aj4&}kz1jpl zE$(c#MxSqJxc47x86p3kH~x1aSpDIsU3%P37P(`aCUl_MTy+XMn?!*s-7x|aU(7}7 zey=Y0hIau5uhXK`V43NOpsj1I=A(MITnqVDsf=&DF{4hnrgsM`m)Yxly{cKwMvS;Q z+|7>SIq@kz__2*D!Fc{n{-5QP@q;ufHH~)$BI`0&l55Zq`Dl|ro7TFq*Pau%2~s6s zz+kPQ0|dj7-@>o9On6{d|L#tD zte_Uh&EJEA@X1|aNJ-B)Z87{u;uz#!(^0qBDP}zq4esgee#GM*KKLtfhZ;&^dePyei!L}DOQjoDT3Qe# zgR!Hhy2OSC?|0nQlKZpS*VFR4?UC#GouWeq z63!U@Z#-cogp-aotkh&Oc5Gu`V6z8V{5#r?nDE^G!K7&@GX6?dZG6seYC|%*Yl8pU zgyu(Y0)^Nw`!!DE3vw={?#V?n&0JIk0CYQ@?G1`VY55n+seB`mIF;GkIGHQzJO@k^ zv^THV`WmPrT_k$3V7AnH^KnYKRK_8zZ`qtHYin6C)pm3cUpyoW%H*)!fw-}W5nAhl z&>Mf@HswsAt}$C%#=OAx&KQtHzgE3I;pL~}=H;y|PGMp9HY9fb9+FFArhL)ocGytI zxFhQ0!oVaAy1ex8rnqWH$b7bhy9o;XCz42nlK!=`vlB*lUZ~VMXf%J)h9d0;VUiE4 zg(4xHIVT1$YP0U;kDC65Vuitip(^8{I$VTrZNKTg$-pKcAfd{~2a-KLA|KvzFvuqO zFtj+V58ws}>2-hqK>IFOVEU!a2eQ%0>y^p~6crVBKI23Fnh}Jl&~fv3JB7ZVk~eFN zXTMKZ#Z0I_O=eKRlZqpeF1IL7_d1CI1w7xOk_k%gn1U=zTKqS2*SVc{xL;nGsrdQ% z6%*_}uk}HKYl&Q%$Q#UKB8#DaGNT5sj0=``enf<2XK__>Mn*MhR@Y}dN=nKdds%}wgD&p^sd&=UZ6y>V z|2#}(*dVX#{XR(eY}~5iwSgMeP=JBf(D#>?j*wZ-&X%;v`Db<{+ZRHB=`Qt%zbuB` z>IvuzKrTZ;_2$ap3XRG_#q}o{O7Q3h%J@kD0VRx5EGc`5U>a-o4et#~MF;LtJ5BhL|1HGW02@MVX5+GG(1^eyOqxk8j zgfB*}4<*6#-Fe^8?Mbr1^F+p%BE_z!;6K=H(|J9Cag)2}|HK%K{CsC#k)otVwSc^c zy}h)Nw_nC$gshPDlZJu88f}3@_|FG#Ng=td^i)RK0O`J#?(S|UCnr`#D%1gn>cp1_ z^t{XMU0RXzt~3@{uZt2<(q}=x=k29^Ce#7DTm&kkb^ylq{rWFbBCe`viMx}9+W0ot zeM-wI;!yL`Yd6j0ba-_^MhWYr21`^HZ-JSmob}1@WdAabe0w=4$?} zRc}_Yen|Ld!$@F1{Y7hduS+#b{r(!RjpRHO_52d_Uo zA{1tPN6eDUtR`7z>U=Qz11iz)!d$Ly)Z>Q>4E1IPr#-^A#s>~Bn!zG=7fz%>y;r0``N#bCkfX680H0Gkb4&AToT?cwB%NpesLZ3aUb11er zJkS&vZ!yIWgsqz+HK&Vt^seWd zg-1_k)XUowExVP@4?QnCA=zy%pT>#h&6Q=j_?-W0|7~;CO_wKqZ8}-(OD*9mt%qNp zqJ|VPLXl7|u0>M+tVXJ~De(PpRPE+)5fzdGu^Q~3Y+dEEze)srQKFzyDJ_^UQ`}jc z%#uzMB&jjD8g_(5C=ibX(*F*2uhylc20`<-ejoTHBuPDYVx4M&CiV@IaT2Jc+|`1E?9Ji6isoCyW_sQ{U>=Mt0R$Y(kRTP{ z(k6qgIQVFP_3kW3LROFg$LQp&K{lX4k%Fqr_X-A^Hm3j4b$0$|TdVu;8RGzgPF#>k zt$q>jp>23v&EZBDEduHbYIAeSt?WadpkFZFWM*%HqynRM`^u4d%kazoC|dpXcF0L zGJpHY2q}bSEz)M63g4I}8jh<1OfMcBq%o}Gf2R6Sh&)`f5r0SK)1M$PtgEjV?F)vN zAd!-ix)Q*@#h8fzp%ZWeIy}yBs8qZsZi%=W#Ay-E9XC2fYR~nrE{y>;3pGDz6tX_V zzxBaBCsK$ZoGv%vw=ps;=rRc``Qv|@cjZ-SSVa7B>I`aX0Zz1C;X##MtC`@DCui=mk)gjU(dTB)cf-$ ztJ;sCGjq5EFYqgBYDAyq&B+Z}th*1Xm8{?Ks4>JNET8#QJsPEvHbD0*S zi4wkh%4gGf{!}4>;z6vdn4~fxc?CxyNBnhiVRU=8E)bhkp<0|tsQvdZ(5T*%N#jV6 z3Rl8bhE@cVY$M2_KSeS0@Uz$R?&rI^yUWWlV@R2ZJNZcesk81_>GCATiP%HX5K#W5X$%; zoW_anb1t`(%ZM2BWwip z7ZNfdfAL2oT;(dy5Au1ugoAR)y+aDR*e}oSSHHbp+AKI=U||wy6l`!9^x#m5I7HXn zr$&V=u-Y8A{~$B@A4N`7v2YFATjOxfNB*f1`y2|AZ%Sy+V_q~VB=gBw zpnUv+h(2_8wkE+c-s#7U?@vsY_46m)pGtjL_^PU{4zH|JbUaL-F`0Mp9`bCd)vU<_ z_%<4tjg3hv=z^nKrpQDJ16tI7Ui!#*nHkQ$xR}oAhX=-|3xqg^$a;rbdF_%@u zKGV0Icw@1vH@^^u%2y*oJwtBQ`2Yf(oWJ1pM^-4f`FO&yVoLXUMNN&;Dxjv9?bXY_4d%9Da`TQ=| z?>&;r#K-BlhDlp`=C%?ca ztw>4=+ryy*fi8GDkv=af;kecsHZXa6e4GXuoTY#e!lHziqm@PO?n!m3vnk<^fKr=7 zLr%pqVS#yez-a?tuWI)tG&o=u$Z?qsy_*jg$P4Kw zfB%>c#blL)lb-tG=I>Y%6CUyQ_WJx7kThm8PVtD2k+x|3|)$CdU(@L4EqcqHu|@U~-!=?A_pNhpI!4vX zOs7sW^1F|W1NnFCzi|$CcsSPCOn!s(4zf4TkJpx8A4>|jN%Gm;PdD%#Q%mtCKOzGI zKo6sEXTO@6qj4)2BM^}num}hObJyD=^qvQ!aEBYqlMnX&aRh2_{>tse^Nf)xq9^Pji>#x zd>8rd?3SqC0!eVVpqG~t?gLhb2|D8C9`s?QTus4*nLRb}2zutg#-rhQ65J@YOJ!Y+xvlzd&`3s@gt3`u-?yS`wPV6V8?(FA&J zh&k<4iBc*&`641B4oQ#zSzfn~&o=PS{hrx8KadN5yiDv5-<~*{|84uX%rHPCA@W6) zLPqKMkF|KWZ~1rkGxB0)+ok8I2{u&<$Cl}+J?*YJslzQ=#oQ?796hrI5u7!V-=X7juIfpWE+6vzpIB)eri^(et&T$u9ONCAR_s zVtUh6sIG2L@YxXby>T0q-SLHb#&^U;f5);ugCCJO?bp$cKtYNk78BqYyb(Ec8i>Q= zpnt%TbJ-bS3!=@@oB4nMssDZ&mX(e)iVYW51*oI*)!6?9_(sD-;mPHW|Dpz}73Xu5 z#{M1n=)1dNW2cV-cSzweF@e^KkA>nc`U)qNN-IQ12T;gjm}7G3t?r!0gWvLIF^q~m zS=!wlMh|}Zwm%hnU~DXEBL+^DQ7etgS7HDu(AM&rXP0YqGy`x@k$q^`mlDGB_>Zag zWU@G`Puss+YcPe|6u#T@xfjjEXAx*HEw`IvTPWty?2GmN{yvK{>Y%>I*I*&ve`l{- z=kZQqHk~Lo05?0f=O|08FnahkK9fktBGcvR%9mNemvuhp zXlD33X1zxB;;K&1t6;!b2@g7$fU+LFymUC5-?_6jaMHGqQoYh4zN^FI!)UDdpAQ#k zXRGAYpde4_3hg?qLl3UtVDV#RUWIf4%NnZ=tDv-zUa>h!UhC~bQW)yp!te^`-Jt?4 z7QqiH=mItN69C^gxMara0H9J(1Zc2qhr_x@WKV)?|JGs+1sVE@7$sNuzC9$GT#uHP zwm;K|eL@m%bhF+ZISf_Jlb_|$P%g~6^e+W{(nP94cdPd;66xQb`1^~^BC9%+pBj~% zyzTB$GwCMd|C_{&AyG9zn$V-)Sf+0B-$r@$S^&swya?)}l`}V68 z(jWasvy4`8+owi)2ttwSx#j#)!sgxHM&W9KyvMEX0Tv&`Y-pKgz+C=}OZgyvb2J6b zi&-#_)ee1MdrKq{$dm#BGK}}4_F6r+gA=Z1eCgi(l4vGp>fLNhh$H7B+uON|v@SMg zL1OeCX-6LWK(p`~!uzYM*^4dyk5opz0AB!N)%Z~!NY3EJra6vaw=Yiyc7u|igM`ZI z@{cT|4$p^Vp@8{P1zyt0%Z@Y&a7q3e@ROWze<@R#V+!U(Lg$Q2LkG2bS|=E1IaB*> z8U13DN@2B8Rz}4|M7pnOvs+>ih^La8-F@x=Xm!VhN5;h($YFm{?sd=$JuOW;mN2Wh zJ^UKFI`<#P@dSg|=oWv-LZBD#*fvfoqX9t3J7RLf0?R0EMzHNW63hYwAXM4NY#Nd5 z99(V`$54Ld|HWx&u~m)peXVa0xBrb`Dk}z48V@d_QLRF0)HRPWX-fniMm3e>jcSZ2 z81o^r_k=ZHmE#rYOK(nNeT6T-T};qPMi^9b628^DZEv;K%(Oq1V>=f(0tU&6v}E(n z7vCchO#|pK34|O_Ak8hTV##RMfBw<iHnlz5;kZ&D4cb4 zUjN1HxNOoU(HE_KZyE!VVK896f4M}uF#l=L#mniqF_Hfd@EQ0EQMjr7iR!UpE(AB% zGGVx$ycQPnFqGA-!WGZg+j$F`#~0^+(9R~zgkZ-EP$en}7pnEXQKT=G`LZBA%L8>i}M<5`? z0+_dx21K2qgo$^6X!ZYT=q;B3^Z1pjoceJF zV@k6_(4;aHiJ+h$ne%u)Be)popzuE7vmwX?NS;&hx&FFnl9(q6b=d8LWM{#0BQ*s% z{3fJkM}SApv{2?vdZD(baF_t3GsCFX_^Q3Z+O$_rrw)hquG~&CW=#BxnnMEE<3SO;|AJ4g$`Do6{h*I>{=BX!_mOmile*Fg@gCK?@nH-g2T~1#ldp$8)eQ>R2 zRE1nfWE>}`H$Csrbqn<&X8SU_bRM^}K>&>{8rX1>K4a2QUXL+jxy6*^t0VV2MV8hk5S*6Y(9W^I^x{9 z$nS>$#b*CdTA+p9wiB2jL=oZjQpTlO_~yT~RQb#>MWuYZPm{XTt)1S{c~@CI`1Bx$_N06 z$vBD$B$5aB&^8jtgHequ!Ndg6{i0io)mevs>wcuGLt=)pLY!qkz=|+vJdFieHQ?>V z>*^N5O^#uz-@otj3CBvxgPTxQA-&5hQ?2efGJ;9IvCZgQ5IO}Px@SCXW@$ZE zru)bNx~S=8cR2p?`nn%NAq<=xFH}av#Kc&zm?u@JmLTcr>lb-}^VSh7mgWUUd+@Q~ z;++)+v$;FsW*sCr86yEPdAl*6p&{Q11SOSS`iX}O$?2`sM4dJpPIm9E-Csno!*)kr z1l3w>4lG?*eE3Z@^gv^834yN+TFRUAgB+J{o! zlSPq|K{!W^rVxymrD7O(DAbf;lKp2h=#ND?#@z#3u7F>K+rS~X6w-KEcNzM*Tc!1G zXj*R4nor4tlY&WGIH=(^y8S-$-+-#il5BxHBlK-9dsyhO6?=PBy)~uhGX6%FJTbWi z3A2iX+tEmiXfCA_ZUoRtqDH^@EguZcfq8b+@6jtBot)3X;qheKFQaOJ&hLaQdHntT zx*jMFR2GwYh&sri+2_tV-E^v2w@C3}zt=;;4S^4Gjkz@p{GB-Yl{IaK9GJf*^3WGf zmO||p7_7dvV9xn+sWpTTRDKy%BBrV_P6vd9hJF~GTxqa`uj}!9ZeGQn1b#y1^*E2& z?hlu;eda~^sjkpNohR^U!;)7RL}*T`;v4F8Nbj#$cp=+0I+OlqXE@)RM7JxWbYvC# zfUkc#Po{fA=575CnbYl&AF+N5p*ULJW;?zutodjfpNcApw@&WxCy8^#+{RIggbsC; z;%T6ceMjDAXZT1&-bdpFw8QO2>vvMCmCjE{{^Jqfzfr!(gBJ{F91`RKf4h9NMjRrR zA-E`JJgI=H4-Gsrp7=tAMoKLtUk7MehT{SS8COw5fNn#gs~mo-)8C`2oyT8p3>Yg z>&BjR#L)%Gk^wv8Xo{Dhm5tYkO>*x5W2f?mL6M=Gdb*AUuRF1ADSt41W z>f+O<$FTEJ+Cm8yN-)+D5S~WM1P%o%h`+5`xvfxOwb?#AEDlXtVs#Hk{YDa;C($`X zFkRl-{7l=%vQ&sE%TX1nifUU-KH^ysBRnR&bypYQ^t`O2x9O+Ovn@&WRN!CM{PUHA zmfe3T`xvrs2z`2&EoGqVmTL5|Nxdm=ixagbeW`{aP|JO4wCK$Fn1sLeqw zm;g`nTbqzX)Kr2YNsF4*{h6b=u5d1ltG8-TazBiZ#MSW#O{)d9S%_mRcp+Y#O&v<9 z2;LNrOOEiSppaQl_mdh{;D}%KW;;O$;#VEz0}(=K0ZL(7?KMRC-s`pd!JnZOnt_!| za9^ji3HB+AzrBL*BX=J*XZi{$%4XEbgd7S8VZ^sSt)OE1P(I`d1!z__*sat{ByA+K z7?N7gmughThcMtXR~Z5!Of_sxHEy1S7#)Oob-KC1`xhKs`1}r;l)Z9yOxMF$D5mrW zfV5e_Qb7u!-A#mNPsAbDYOz%SDHQ?BlbHj6?WPF6KVJ#e$!)k~2gm{>kpGbmEO;EKWE%xY1yqXw5<|Gu8QSh zpuxdO-2z6#1=`R%6@+Q30nf~H0a(VRb|0c@Hxr+T5}uwtLPEQG=MSbihGOa&CBoiO zYD)61lj+f-KG5k^QzB{7vz(aOodjOVcKJ{G=ggKl3TaD~vnUt=Yp+yY`r$ADsHhs= z$C1mM9r%3oVxmx`vMI9VWwfe}j*jTEZspS^QMv!a3<0H|rUeq8@f5pmKJdj40uJGW zOv_p#x)eh}e573m|Yb)Z~#JbTL=iTtzUC#tBL(`D7P`hkF6P-V^UZCP;UZgN~%?W1%xKDbeE+ z)gP~J;7CR3L`4l3Z_wF`))UR`^QPLkpDZ)%c6;inG)ZdE9)>FLhG9=9Xz>7&p~EgA zWitV?SPr(vF~F#(sI>aOwzEs8Hr_(G6&2*-)0KwLC0saka^6p0t-nTpRZDP7@~U(5 zav6@Y$KpIf}^I=?}`$hSh}d17Ui=g_7#3=%9NPCb6X9FhK4_{xx+$w zN35qri}Lu=a64(vq=J6ys1#>YOeQdUKEDa{{px6QJurEDd9+DkP$z|fyVZ_=aWwl+)BXM^Th@qR(fBb@iv7l z{rWJHA7&OV;~%w*=bc0SIthuV*ccdr5DI)G_eXJA8bCl_yvR|Y>vWVxxvoyl6mAnG zjQM&=4T{{nP6ha4Hd6Ct+6SPRy-u^gwa**J@vyqg|5qwXG*Gdge~1Iu^=+XJ@h_iJ zpqm@HzGOcel?qVE@?4Km`3FdiGY3M&d_Z1_cH~B${31gTL^Yq*>-fs7QZDZ2b8|-= ze|JwM&RSiAY>s@@<3}X*`r>qdWs0#vpG}MY?N3#t%V5+fsUUNq65NKf**o`hLa@{5 zNsq%ODd;uV;7(f}Y%XTrhT!``P1lO@nnsDA7+T7KSo=bCaAW*mirfs?BI= zX^Cl>&jK`@T*NFA@D=av=|(c2I})f$C5Rg$+ubgzE$||i?9InWAay^A3y1V@$`ibw z!K3dj)vmf&w2!X)CVeoExxOAPxve`Fv^3{e)0|VKMc1D|(7-4t_!VZ?W19BF-~qlK z7)GJ;vUvzA{N-tV<4DxL&*VYDoUh1V=SMm854^&O?$ z(BQ-H$`Q()XJUkN{un>g{fO?g)3vyY$DP!>0S@jY-xWENuee8oaP-2d-GZK%!c1$@ zN-QOH#kFBS%e2PxJsISv^IWXCs@4sN^oNec7Lt_kL>nM zBMAqp+5+ujqmqSszH9Vnw4wlo5Rvg1F}|c?d#>#)(Rer~hcDH-Y>w9! zo_81Vcef|wx#&$-tMw8(JuArtbeqR%+Gq6haays>JNNo;TmV?UyJP_b$mEnuMxq+u z9oE}%nXV^YIn;WAxdthJ%HRaLIPoeiCLx6LVIug!fXHv?lnQT@tVsvIe_(~y`f4;< zsjqnlS}fl5WKx4PUhr6=;Z=x$D>ZBA z^2>Ktv0^0$nXr{%`D(@hhr~gIdpKIZ)oLDVDlLjmdN4;YN{Di%G6JDmCo?|{oop0; z_?yWgR%3Uy8L#iSC2Tg8j_G4Ll^hE@>TdIPmo=BAzdW<;6}akMk2ryAu4{ALYjVJx;1W5bXGsZ+oa`MQTN=(s2Mi6Fp}tS% z6qiTyv>1tLFy{2vsIQ6+Dhu;!`F49_DY2qTEK6@8$1>D;$ER;PRcPB)J4t{>MPMm6|sKK?0s@qOgBr{7P=FcB4~4y1j`DNjp@f>o8g& z^C&5jhKA>D+s}C6f#`Q9zk8J{Z!fK@_`;F{I&H3+N`Fg)xgF~nl9!cj54Nnn0}pb) zefie*TPQR|WwXe9AdiP4PPK73C0dHgKp);d&(598rhwlEC0Iua+aUgj8)o8n#?6Rq z3FMvKEDz83V-7q0%%bz&^T=;C6>=V%odo@=2a?_-vR7ic0%EzRD~S|cFXT(&u@OZu z*4J5Y1tXzK<^#84&hFUm*UXwNYT2!y?jAMI^5b=t9^NCmjDfYINJlNG>-v=NBXe3~ zT1p!}ZJ(s#h$)4-k{?U;+FTIQ@(gO2+H1;W!EPm5n|@ZGd^zQ@N)N#U1as2{_XgxSS*`_5TLhCYjOM7FLg@5 z4u3zb7w4M#=~z>0!NFdob#)qXh$WAbt;d4TH}-|N)?CwlZ{|@+LfKk0l)0;Ly!LR8 zb90?Y`BCG=k7hBA=A!5Q0n?F^Qn`1I{oA$cQY~_Z;^PkZhp}m#VUeK_zkm5i4jI{K z**$gJ+<)P(*1(X{e{5uPQj2093L3LXrs-JpAlBoRt zFWTCLm4AR0eg03>j}Yyfc7-WiR*J&RM~D#4J?klZm(}UCbJ`{(s0q0KKPNAMAxNyJ@?i`bv8oz zdEbZhh(bTl4Q)Y2vgOtZydejWv$OO0K zl-H~oC!)+A0F3?z%Hv}Eu(f92)b;DHVU&dMZ9|>KqyHbrstx$0c6NS9nzx0R4uVSI zzU31ijeqp$@1UI?Bp9^Okl2fxRH<^;NMu>pj@ler=Vo`SW2KeC{M>q>U;g#%PoWUJ ziX)HZ*Q%Uj@h=(0T$ZUd{kWT5BZjjd&(h>2hpO;qk0)vuNUgtE2nV1t8zf2lw;^#Z zbGZj^Y~QVP;3K!#_ctw6n#TI=-Pt!1aiM*AzLV(izWtM%RXkqSm@m-665Y~;PLS`TEUNy^Wzkzx!mEg-gg7p`3;rUT9Q_S^rfy4tL(`H66 zLW0i8(e0Y2|1)Vl{scO(-Rc^e>gDlsmDP4wr()+(E{2savNzgTKad_!eiFq~b!nhGLdk6<^iD#PC^u``y_ka?o|3@&<=| zCkhMq4>D5f7Su9$)BbLI+GM*_Xx?40X1<3vH$Oa~R69DF@b^l-Ya^9Om(^~W^%F|_ zNE%rZ=;U+96r{BMrw=VV5>fDb|Mg0b51rP%ce!oK?d^8m+bZ(|^>@O6%v;rpcBeM! zsMcgnKegf8lVFFllv$1!N#Tq$UX8&A2oEN5#k*i+f%i%(7O*M6&2NtRG43$BB1cz> z2sD!%+Uj#mnW;cXb93x9o*L*ib^36_Nz50fIJh=E9YQ=*p{irvF>+7lWVVW>x9KMS z$Olg6W66j?VdLKiGesG=o&HnZDoo2A%7op`r7WEKv8j9R&KKOkGwOX|RkrWJCYikZELZH|E1TW5+V2nN_B{fQgm0EfAZz0@$4ndcMCpnN`94uI4}9}^V2#?Fj;C@xs6c&(G#{<< z=?QeBhF5Z2o!(%11>h#~n}3?Iu`xw3cbgS@P&}CsL&Ju0!YM>Bj7dmH?*oPOk}NH7 zbR1poY+HXS(rVHP$ma7vV`h#Tcm4@c<;Po71&m=gLF>DWx={mbE45+mjr#pqVMap( zZyq7I08=WX)_XuR^L`H9XdaI(XX*&`BSeHC+CxVF6+%w5=&2ZXZ|!e?Y2E2SjIV)< z9HO=@U#w{GdUC@2zQ!vpq zFz7*+Dq|7(bYNk{??BF9*(gkUf^}H0c?F%{yVkbwZEy`$z-mTJCP>yC9*wQxZ05s2 z9y9EWm05uL(I57@qXBQBHfjH9!Y@vz;iUpKGV@QMsZ`Wvh?3@KG_veBopSpuJs=DG zM#b*v#aajc%M0@laDn4SRZq0%9|Q{cub$WYu`~1Y(b9g($_WWqqx2Qt-jp|uq*Y~U zKeEq{XI^1&8N`(mSlvQc#&r3t>}Np0lkcZm2719I_CAdWC>{dHZ_p=Y6o{i5KZi$$<^50_Q!g^<<3y| z?#^?nt5pAQcG?XAKQwSvnsY@g-WMq=2e`mt=sr;-V1qWr_jOh>>$$T2>3W9#l$ck;^MU@XwEa_? z^xKazrK{7PcZ}OJqAJkHH#d(>aJjp?Lu?wG`^M1T*0w#8NVQWOU97QJea7Y%a@YCD zCiB*)l7;y?p~cnQd{{by{Il{;etQPTJ)~Gigrk*`0F`iV&YPU+GYm~ zUuSvD8V`Qg)?g0jai4?+KKZ`K=?x+m%^SL6xLkd7N6Td_7TnW)3zoRm6Ys59V|JuC zNkqcDsCopGH<+Wv#*!0GR=B@PAY`9s;ZsN%f2I(lc=aVc*%75u%y6l3;0ye*+x7Ks zz|f!v8?v(p9f!GkDLr=5Px87nB38K<4-_MdwVHmX^y9HtfiG`!}t*?{l)SF=j z+(#VRCK1r4(Hc@dwzq>m6Q;$4TfhNh0*_0y!+>tgDoXxq&1b^w#5~jiN$)bA?WktJ z+96FZyzhwRUy6U2U$r_a=Ko0;&$@%skK1#0|Ge0WGBC{LxD9D|OAIz0`yGfgnzgMS z=$|LDhNpi`lxL;GC*CukbR|qOun30sgr|uSVYYtjPNDMs)=s6gf^}G@lXxqs8%oOC zd`A}UW64}#r7$?Cxjk*Lb=8zb>6wb;b=%>n6O)Cgv;#rMAgTdM*~JDeA0#PD#=$)| zKDIxO=yRn?W$r#=XsT%_MR zkhy*pr%@sYe#`Z}H@NJW$b5mwDUd0Zs1=_orc6>;3=KSwA_2hC+}oSue9L9G67@_? zO-<2eF>57Vj6N=#yIJeM-f(#_tpBN z@-&ntbR&^Jb4AOQOZ((?MWoEUsQC&%O=MhJ4vYNGx^Q;rXaz+(bac5re66^5k$mj~ zgU1Bhy}J#EXezf)4n$ET_-5|z;jTF@wNA`-ffnrG1 z8`BmeJ^gbuPXX44v*UX~DyxG{rcy&7l^rBhD|>sn^Rqh=!s}{g5es9?2vvLsy)1aT zFmbO|9ciuxyYGpQDIBG)3{3E{`_aEFUWglH0i6s4Q8RW%_Achz(G51d^7)MuI^Ps% z(M3WR6(dn8HpeI7zrtgmsTr2AJNoYb#%a+`e>eG`#_v0!j_ExIrjp<~b>&Cu$|=mMl9Cct zIVF0gBTbztB^FyBLWNIb$$uKH)r{kdii!Y3C3EdMN;giVwRJh8B4F$KDo7w5Z7ql- zH)^DQ|Dl5;=`{_>PQ$VCnP>Cgbx*;uhc{)kNUJGJo;PXtq)3JD+*>pTf4k#7lzu0C z3LyP{lR**eiv4-pn5C}Om^Rs34^e>z1<&BQb*!|z$oYN3yfK{v%RFll+nWF4eK(#x zt_h1s^ifbIYdPjCryM>Om`-i?pR@q}y1_M1LC|V7Nk2ulsX9d#gUY`RqcY%<(Q1M308?LI zBS+I8E3Z!l%DH*?0p-Z4;^1<9UpDr4=GB<$am@+^@N?Te*2kOVY}!2L^mdoMC@_sq zKtMpkkbxp(8mMC_pUbFkL&TcxRzP3(cxfX3Iqa7V>elaY&p+j(Ua$K#60E7`GQg%VBE^#KJ1&b&WSjU94|NoCvwi zG$vh5GkwY~LEPvGotb(W7IpfI2ncUyMnQw=K=W9?wIdr7IE5*lLEWRE>)h}uB_!@T zuQ+Oek>-(iQ@jH9MZodXRPEmw!Dp@LULEFRDkEv?_pMOm9n~r>Ow66aDtyO{e8=%r zIt}FpQ)iys7awFU(*^|U9-F-=7(1~$Pvu*!RJma! z6~JU=Q3H06ZL(GfK^8+01aaUrV*7qb_tpZzq>dpwUDz zQ~Pd6dN$g|dtknoxuFo;ZzrN0?SY8sRK3O8y+`F3ET>Z$b=W_av;5rtk&<248fQ%< zO!@*ja2(Y5gtejsNc`JYCCozkI(M%g@aG1tFvBzljbO$$)j-(1O%cn8Q;k~q>5b%r zg(%hqt@4C_=liQvfO^VmME4Qm9+yn|Nqp%Jp7~%-X5q|h=GO?;?P1*UNO`VR>f*N$ zR8_+{UyW{AP6mV&lxO3-RC{?|Ss|YUht87CVq%4EdB}oS#HTm-Vvf5twwYobo@^%J z|GI5)w&LzYuW+|6DpQGdOb1av)^2o5)Mk89X_6<5IbX#Ue;9_iWOQF&>-$YPRMwm| zX%3eX)z)6Z%f!Kjag46eY!7cx%cuevf)Gaq211pj}xbTk|n|PXL+8g=l|0J9Q>HcTiyupguM{z zOtQR2$9g{XqRZ`x@n5B^g{?6^1kB{14@NKlFI9+_CoUQ$BlXVHMDGt6C43_k_LAs5?S!NOF^;?6KAx+3Na2 z_D^H+_<2OmCp~04rrFvB__EjGZoJy=w#;sgRiUH%gZ*?uyN9$~tt8mI*={69fUhE% znW^(J6UUVD5pwPWC#YN1-y13Y_2wvvQTs5SVXruieG{Y4QlN|Ruoak#LH=9mo<*72 zFA`&B=KE--4ZU)HAdp-7qOB1eXlpwd>fX-G>^HvMKkP(cvQ;g9Si8v|AGqb3U#mwD zlpF|!=?77dCi=;(m5qF(dQ?)AFrgWDyJ^mQI|fmx8lZ7Eec0#Wt_&~jrKKIEl~J7e zff&41i6^@!otF8jX@ugDaC&plv*pc%P7Dl!tkkyzQhF(YlICQeBOFYM1@+Vt2xGiB zp(dj*J`9XePmg?ba#mD9;535){$i-q*%OTWRt_b=EI+>vQ@loXR{3msnD2V)HESbKD`WA#an#0$U zU8)z3DLo62rZ&toa1*0=6i@w!0UYtuQ~f_oeFJ-5(Y9`5+qTnKjg2O0lE$_g+eTx2 zQDY}fV_S`F+qQjI_de%5H-F$;Yi6!F$9OgHhk14l4uTy`Pfg7kzD}hF&}46;=5RDQ zFFTK4pfc$x>40dkh8yN;&6yGLj^j;Q>m;u?3Z4VuM%pzh5_YNL7rGO=w0U@q0fU1e z){@J1tM^wuyHHhz*$>|P=zH<}_&3G8Z!VeN#^a>l9fX8#Q`uoU8*QUkVu`nYWs}5= zrUnJ#?+>e|>^_{&;B*koGX#N|Po)0zyxaMAxscI2xZ*+`hRa{uOWm{%YtFz;327+L$g$A-zu>!)h^SiW2@R><{}9 zN_5j3RQkhmdl5M@WnARAE=@a^6GReIiQ8&KoZvmSAnG~-^w^7wPCTPdrcwlz4gg-) zblUI-E8M!hg#>~zhtXX83o?1@*E0?I-im^W-|_4ERA65u7E>fAgO5^Eff-5#z_QHSEdSM)%zB-8 z47Ge&Fx>d{1^npIc2VwWB(4qHJNz{&seBdf{S~X_g(=u{h#X(R;-jeEpwT9(7`E&Sa^ES#YKfevDEzuWt8_!p9E#%f7eQD#eKy4|62-M$9NbZE)u=$ZZxPI-)Y8iRihA6r=@0l%(l zW0%a!88KHU_LLk$t2aSxX&iSVfuw_W^_pY>qL;_tt<->)AfnH=K;ql|tWH1WuQW`0 zbFp!NhV8A6`k>e;JWH_3F#a>?b7yqVS^~}{hcBx23isUXDC7zr{7`XZ)w8?jc1u?1D^!J)rj%cZePt43>UBGd8cbJJk{Ih(wjQ)Q!pJhiERLRF!$plZXtK zim$u9ZE-nCK4h*2@w)to|1Nhun~)`FKAsWIx$^+{(uMb+*Z!j_&uuypUHFFr0s5-V zdZS;{{54DY4}{5kho0TdA0}c354lN~ODr;91fdq)4!47?e}raOChamedt#P)A!8*R zSG$Dwvw=*+IH3^R{@4h$c|8-pAy>+SkcD3R7_1#hLih_KDfFMd4`%SC96rbhvz%w} z#Qq+f7>t_($6Dc%%miDD<1H7(bbYv`vL;{w+vxC(4pB23A^g2s4P~8~PG;P&=rLe9 z`ewYp`L5Y%VN>dC8zM=8*}SYea-;nqe7{exhuiMTv>+3T2^?rBS%MO40T5H8-C&V0 zDA*0QUmU_5CdC*muaE{(1?yR!Db#rP4mvaVNSLjiTXh9S?eWn5vMMoe~5j z2BcqNGC*|7)Qf;3N8m1Tp3S&F$}!E62v81(1$-)8c6M+;(=-N{=o}nxP|)s%7cOLM zwNNI%ZN<*y-;xx)(>)Q~fL(Ea=Vw6L)r(16+&QtzUR9co1%N-KtxCy2Wbtgm~$gG|K^BL*#v=cHUD* zEH0%;R8H{U*vbdPP+%7wP!H5?x?gZUa{bGZGE(G4Je~T+)710JFF(0y3ZbMrnhXRv zy*GWtk^uTajLv^#)Mo*yeD|~LR87G)kDv9K{Fl7DtPu<4F^-*FU74NCn*j1Coj@k= zOsIi`FtVt(S`I7vA4+T1+c4rQ*W?I86)?d)?h$fA%X`?FU_}3@gR8%l+*V@7v)L#J=H_AeV zUSB$QrRmtf^fX&q)_Q+G^ltmXzrZ3HDgWtVE}K83ovj~;GM+!#JSVw_iWzx{chcyC ztpCRb#%4CGiVL&i3-{Ee$x0dUY^MNjDnAWjYdzWRZvTPyE0URx#=Pe(jFv zLpF1MT&;&GBg}6u9DRs2??>-)Mb@r&%OKdh+z2LG9fj!^zw<+@mJ>d`LvY>W$CxkJ z^{kge9DCgME=**z7XM(_rTzY?-K{)q{-N?DM7loG06x*Zv zvrC4Ieq_alml;#iEWOT@A)qWn8%ERNH6OTS|OHz5Qg%Hzutmn11R_-zi~ z4^J4hDB1Pbf0CFIHMK!`(n3{GXWs{=Su-6hpoluVn8$GPu{WHMp1e3CA6KwoXn;O z5!ca>CJTBUeaSIm_BSK{^&cF8Ozo}6gd24PO~r(8hmGaKnPJ_GPl!DAnkNArK*t6* z3INbyudjFjksP=77hE$Tju^!MMARRPV?}8?E@NjH!*w%|Z_n9Bx6;^n!CGoYaAjbq zT#<3PobaIcEtd{hd0W<;a@(*VdSTi^xamZa=Nz1L=p4n z-($;x(JzLO-!Z`xU5X#7w2a1`)8>lgy)ffbDa1=D{VG53?f zI%L`vH{0V7lOr`MXd203=c;dB>t8eUn>F1JJEeSjLguu4gClts$#sjO9$&X8vY^(` z&|`I|B6IR55PuzvB8PY3R4OAC5?PVynmLIFxA*#y#a?a%Nx8f3=zDt-Fg^b{nNl{{ zT6%tQN3&kB8o1T~YnZiNxxffc-Wac!$jH&vY9$&lW0JZ_r7)N&>re~e||3RZLs5I;+f!{6k*DJv+`%a7`b=y6mowo9fp~ic``af&}sZf^TXoWQNsWE zT5LZSPUQwqWHceetTLLrpG&0F9*HmzaOwnmoq1L6WJuvfN_UN@youi z9@g3=vEg#<+oLhR_rxE`+Mirn?rBrTla)lN9FmuhoVfn&Zl@@tPr?$b{Zl4 z+jBnIn$I>8d;(j7y#D5@|J0yp_jpA-X+cD@WD3{7DJC%P z^pgU}OvN%YES*;;0{T`C9Q{h`<;dgq3?)B%=2gCvpKE?{IC}0s7JP?Hc4>{U81kbr ztE}P|(4o$TDHKwT1~uDIYjEiO%buJl%`*rI`Vy8{$cNo;g zqHvK2RdA?(HKiMJ+5Ri?N4eNmfWz?>D#_X=UdS&1(!mT=IcGDNnFZCoQ9CuGMKV%G1Yu z1~%7IWohSr7sWOH-f!MuH^L6QCl*^15xC^N4R3-i)p~HXU@2=n+_~$aS>&<@*Lx=b zXJ~cY=30|m%HODI_-#ld?(-fPP|P*-)lvr1(E&nT+$S2n7zMA!WWLvWv}73_%6jB) zpI3|I8B*r%g^(Na+y3}J;Q~vmz7Sd?i|-tLnUEu7fYm&y@$>D~@*5X9c)QtnZ{Oyo zg%t?3n}!i#IT_=mt?)x*imBq~BcLhUTAC?-toLxH!%FZjw-q}d^+yc@{v>-FET(3g+Pq#dJt_h?{9dR@U3}5aEG=AG{$O zE`z&WX>N3EF~TQ*TcriQQ0+$BJSV9Tw^SM5?M*AKAQLXiMwx7?+RK;&<`1VroY^CP zl#g~1tIBHHHEFcpB70kVpW<<{VLgjG_emc&WghJok<#~(ro3CW!P)6Ctz|uCWsVc( z=$V&4r~RJ2`4>A?OV1?%@MeeA9@VEuvel4@WW`TqXu~QQj(j<*!)GfN zoSt?51?~ahgFkGMKUNx9>%?FXtrwW^RZx2Viu>ChiAD@OGAS2r0%EnVfh0K1#Xq|N z`$1Pkn9trJ8sEvvYv|<42RtbIOD)v5%vXZjuJebTA}joz#f{>8-NZgh{qyrZ&lX|f z7;c^Gr~!#-s@Uc~4;dXPUI#%;))7nQ0Kg~WCacJgxJI+JyBpkKvl3XNQ37Zn8VnX$ zBtkiKi+|1H43Tg&zyT>V-QS6PtT~7&`bdRSU3FdZ7GQ6`;_7?FUJ_#NQ&{! zRQ@pD!Dt%R8UeTDGKXdF#k%L(uG`mnK>lBXi%-1;slxq$ieI+ruI0y&`4oZaWVP!P8J~|fzuS9q1K*Wjr7?XMh zaNhU0oR2NQ45?GGDN8-(wbcjZGILMFqD>NjLkw_WA91|g;r>L!K<~ti0Zc>5k}nWM zX>E#!$0gEqOfsFm&d=ZXE{tR7O@B9c9MIm!WPIKg)a4$Hb#ioj7X&Cgn#dKa7a~oD znu9XyIjLEq{YDteX{m(`8WXZUj10{h?3DTQt`wAP@oa1l$E~$moCs@vGqY{+?tOKM zaaVdwec@^d=k2(5Mc<~g>lp4u4P|Gf@ZAKYf*v6P+_1PRmS91@7=>&hgqiQDsR7b{ z0Xof-IgX|rHGsKAF12cF5H0HNs(S%fWCm_4zE#fQSK+siN)n;1u^krgx7(Y3Etyv4 zc(0V0%sh|FLf_O~tCk&5CsDj5`DVdOk-GOrX@`^xZT%m5))W(ZkZ*=>vNUhBA#4c$ z=2iUZBP>t7ObU~39;HCV{g6vgZKCN$nsQ2^}0cdeS z{tqp1i(V$2r=>hbGhEr?>43!{w#k=vs=HZq8jYvf0BPLA4wL9^wE?qPmLbNGu4jCu z*pOU+I{v6V-z>5&Jm{MpHss6G*^9=`E8j8VA@qyh_|>=Iu=kFsEYTnmym#WrV>d|C zwIe_Ga><0^bi436p%zCX=8KT+n-`Qp)iP9CD)S_!@}%#?GTUN7la`|YVGE5kqxUcMVfam+T&|J5yI|+==$NWVC|&fkvn(U zJC&0G%2keJx}?cT#o0f1%2_f?+!0vGsnXg=IkzI$+^4@_HZt{Szfa;L6cMTzyXqvN z{R`>T71YE>2R+I!05(hm1ynR4L8~uXqr_AuoDNvib>_qISZmy$zjw&5->H;fDAavL zM;+=RIL3jETx}MICf3DniGP@nXRosTj&m=tOd;Em9bmRhsGhXzRc9C5J$@y0*-Pm5 zRH3_FL!T0gRR4vl{H}iMP$^2ILGCuC)ZOqE=NcDimR0Yw75g)rsHFZoUYvY*oKfYem`Mm-D z&uW`%lb|3v4n{;h?M9=M4Zndj0^I_&8eb%pvuXULvg?S41viq=$&mnuqRFVG?v#n_ zD>^pxyOUW=6cj9am9E(m<)mj@mqMEGA(a9-5oj1kLJUw%t~4YclbyRxugSF^{c?<~ccyau$E3qd4n=&FGis90f@r^B)jVqRO7-QU2 zlM&sNKU7Am)9PJUyLAE#ch)2vaJt+~Vac}aGC{a9#_}@0rrg{-Y9)cGJ_!#dyCVyg zQiE2)LSpttIil^?sbg1AX)5DywtHCo*y`LUsAXj~#Zt(!bPSnI5a#;Ph|LqYw zDcdy$xsb#*hEH^qiuZ?1ydDcx{CQJ@MH+`?A|0HuXX}xa`F zuZQ!`HD`dutlaf*ahPIA&u1v;w=6i5<+aNfNykdG*a=ulr~o=W+a~utSRi|n#r<*< zvUNI@CR^Ep+7yn{PIlKuWDS`-D0-$TpvuJ-nvQ2-C));JU$XEi3bD1Yh;pODPG7@p zB$P zjW&r5zB@&e_W^_iTj)|Joyt<`r<8mK&Q~iOd!hb+Hg7|aT^U1$zb>ZSi#iPoe{02H ztRA3tCF=u_D$(-Q`y%knB1*sh=R1uSCz`KVjA7WER`8%Vzy=b33<7trJtAwBsug-i z^|&$iTqqQ_)Wz_^r?FYC8%*c63naD=2QI+fKSHJGPSc41vx^&0Kz4b5rY&*{Bn+dL z<^#C_yI&GypL3TN`q z2DCHy-XB>5rY+expQU$;!&JGJn_*i1t_$c*%ptN|0NJ53NTF8OY4elX*+y)f`9xv@ z&BgnEIEE6VpKm*!N|VovhH;7K{gKz#^fW;7Trl=)>*Gup%3VbrL!b%Djrsg?M=L|r zfmv}muN$GeT-+Y%8!WnogWFdJct|%lB*+xVz>kgQtwLGr4O`a)dA6BYvs$ma)s)@Q zRkU8Q^b17fR?SKWz4ZZR2ZP~-UAwh@?cr0D%Sa*#N6Wc60fICg)z@e)(R}yKg-Z^I0!*VD^ z-bn2cy{=E{__zcEM4Zr_)Cw86nbhBUgDZ671x<%pY(j^7p3=FMYE|fZ;i=}1a&Hc* zuQo;a-JdbN&rS|-PTR_CGYU=6cFVp)A6@UM*fhjuh3`ZzFcf-QZwSTR$`anaR~t(? zTt7M}Itcl`;v&wN&yA)Ybc4;lqcqzfAj7GcQm;k}(+ z_)fRQc`)s~^WgR6d~GW4hpMyH>f-6=>?TJj8_8JcNFAitCv#`ZMa2h=l^8;>4;z5c zw_ATxADg%Lcb7ShEB&guD(DSg2_(gSF3uV}A;9@KUoMySF~Ad=!b&QAP0?d-`)+Z( zUdMq62?@3aJDGGanJX4?IWw)$>~^7tO7b2DjR~qG_+d#4t3en5OZ9Y5w7j(&zVkzQ zueGlKVa>Pt5!zH(r0YBEFWGBgMZzS-er}K6orDoFU>7|chmeds>N`B7aUrcwWEqy^ z=U13sLvPO3ThS&lXa7*x%N2b%r%hu>sTp=EBdVxq>1VUv^?2v^eQpU|yxQKLc|qxp zLHq8{8Wp{>I z!jjMj555&391MDe+vzuYs-XgV!L;&qCu|-yWCGs@y=D%?1~p$xt*rFe^g#E1s&ZTA zs7GwPJ7M+efzHU_R#>RvZ?FzX_z6?Ns2An#uw(STKZaXxfp#yK8fe<;T5`esdY4jB z4W96d{q5b4QnubEGx zG~XgGgJFhJA{yi09paHR-~wb91g19n^tX`R+%n|}7J8N(G`i;D=PP4c@}D&=OWDdu zQMn*Tsh$|zE+jWn*&PPQko-sl%cP&{3uV!0T&+h}+VG5VnER8cKImF=%N@@;e1@O> z4EhoGX7sD2P-f|M@hSLRdz+|aRqQVcB_ie4f9FzdFsZ^n3i?#^PbO%wJ8bs?ZtLz2 zCsYQ(f&W|IX$n$Gp=pv_Yf^Yp`@0I7XT)eykizTnTuI1J6k=P)%pZ?pnEq-F zsB@?+DZ5?*9)ZIGUU~B6S_Wv8=x+kb#q{okMe;pICkxe9*Cx95tOm&*4;~DgceiwA zWxA1XGsP6f$Fmk(*1zD7Is0^fcDIzdju@;CFzZ2TLX@ske@lRW_?9J=r!sVDE9jRU3z!{?CRyUc%=xM^!3`BnPE)p z3iqXq1F%{g5?7|_ve$4Ub#tYxhD&S(U(+z(v>L=Cf6UqMZcEfTyQ*!t*-i>`dm_#k ze!QJ<+bNbvqm2@=d>UZZm)yw}W6e=zaJHGHovr3D`k0u?Z+dvV;>^?t8o5H7vJ>o?(hK8&oVUa3E0kmL_YTcK<5SYP1X z+S|@|4zS0zc6NdQCT?pg=x1084_ca>iwfPVI3$?12q`0_KE<=LTDj=*?vN6xHrFYy z1Gq-Pm*7|9$vde{PkWe0S$74{I8trQ$8>E-=_klIy!N*B7p`!@vH60_HEar zCbES}C+fj9EI~9wRea?)F5%Eg}O+}u~JSr!nD{)LPTp9rZVMVCstmTTrE z>C?O#HTi?(;Bh`XIEtP}o^mWK^%MW6&ojv3zRR~q4rjwyE=-KWJ;$X+bFt&GB{rvP z=#(2;d{;mhu70s5mP`0WG}GwuZwNi}`93~UA|f2H@ba z-U3s(rATq!70t0?6aM}kMt!osPyFPAPt)!HOwZE8{t~?w@_7zM^LsCu-&$-aAO7Am z%2QX6-Rglcq1XPs`K*|a=U4qar`H3$m2XFxW;~^#C=1`WW0RpOui-?+0Y0BI{C+0Q zJV56h#@!Urm=>}@SToVFf6ERn`Y(P=6&4W-Qr>XWD~;UnvAO!ou9Ju`Y84{*mgMz9 zlhKmrM*-A)8*3FJJzE4R>O1VI5If2ab<6MY9)S=u`xV_56EtdhToQ708TJB+eNaez zv6hDklae$PbObDLhWnEZF=sN9s5L!)Y(ml?G}wmhygvQ8RBgZB6Or|DqM=bmjO=c_ z4Kkm!js{VUY4O7GDmzuI)=D*hni}vc3WDg6&0a6K(x6JQphW&oU<+ML)G5H&g z&2%`J+s^xHe~h5b=LWl@rO zo)G^C5ovDt&@+!#R0cIhD<-9WY{)!BaQvZ0SK+e#1ZGO1VSe5kGz5yOyBr3yv3vDk zo4{QsGQ{*e28t(Mm&I5m|Bbe5Rdkb3J6{j$trbYY|`b3ZxU%AGc-uy>WLUTq%3_kF20wh%IlIwn{|2l7wgoDXL;eO&K` zv2>cf;nXX1`jbXdf<{%#+#N=<*!9~L{wmbyKfO2}zkox*cTG(x(ZMYX`F`8;*MG{| zI_b*3x^cT$Ppub{b>6lK+z0vvE7ZsKu58PV83`G2)G*|KSt~eL^p6AE8Z&>HC^u>p zW{$^ds}44`5Z`@}r;G2H&9J8&HxIE3zGe}7IATX17K#r)ecU_u$#D}( zu+55G*<906K_~I|f_Bmx<|4p5Ai@Sq!H-dGR~T1IZS54tKGsXIK0bEzPU_t#{;_b0 zj3!DB;JvF125x%e<1iaTJnH59HP=BbcC#hgArx==x=Us0XZ=o3qDzNu%J6tTx69ol zAY!9ii*W4A)dZ_Q^RbZcou9m+=wGW8rtX7!l#bp@zlo2JZlAY%C#QMC#e?y^QumV` zf_cM&&hUR<8tDG~#v!t{X+`>2u7W z_9x2ojv&V|=}+#sr3_H}>^r-<16fP{1n5a&K&-fLzMP`Srq`=a-Wo41JyPgbm=z7# zx)sQEv=j2~O2A>Zzu%XC?C>eIk_n~bUiz1YJ{HLO0Aw4;HAkay8x&HG1vB6ynPYUdWFB-bk^?lVdt+W0!QfI=+QO%6p?zoaa z4iupkvIK-1x$@^e_lztkTpi6wnaE{B)R(@YhsNhiE%?PW89hf04gO-KKqltLJuWi# zhI>^PRCGG~nP%78JR#vg;rE&k(_1t0JJ7$GvQzG0oc?BP8I1y!&(%7zOR3JsD=GJ< zMBQNu2~fN3A>VfDkbM}9&(%=9u6Hh+UB~l#!0!&mL+lN!3(p;MUTy~9aM@_Yjs+N* z9@gqN`a;Z>s|zi*OGtt2;>D0Tqz$OZcnSlU)ByQ`5;roC5e@%1yzq*n*FBjA5|$8F zhUb71@nXJwNkh-2BMOax&{tkU4d;io@XnFw(X*HvY-2R)cV$TSHikWVdEL*9w8vE= z$nJc^`KrURNBf;pHovkqM0>5-59Hs2nZnDpM5NAkF^?9y*`8ABb!7Bt-N ze{*L5&Dv}U8i?d>T;J6M{9mEX8i`X1uF=eHt_D^1n?cCq37yrrSGdw_EN&F>+7YO_ z+_U(K%O1nG2c&i)u;MRTFGN?PBY(o|+brb?qKr=nR=pE1n*IzHGA@de&K88wz#Un6 zA&Vgq_@kr2K?ax3>1B)B=}q=T&2O?bnHb?;^r?!iBv-U+!CT>lzpCbPyD!pS!*Edf zGPz4p2KIqw2o7rF(uh4r==#;U{ zBwu!2BoCxJfx&M`iwI%+i!1iC?5zVpry13R1j)|6gL_Mh5}=!mxRwIG!EeEh?rE%J z4UdYzSOmC`7CU_S(5d7C3v0)<40?0KV24on$_N_9dRD2JfN(Gd+f~6>4D$!0yVcS# z44QN9pp6bUyw?ZaqP9U0-7cf8LbjLj5wAKS?|own-lKlN#ngUrw{!x1K7MTW#$8JU zK#G=jYVti){@tF#P^8t@yglZDBOa05J}k)`{+>-Er39G%fk>|Z7Q2><+=OG2s34Lt zWSs^pL2yH|CoOn+t<`^Cm|OSMa^rO&67mBAiB;ETaFYy zr;~?FAZ-8@9zcA|sMBz4S!7khu3(+8zXX;ravoi;NWW^-yTlDm=<*__ z$k=Q?oJW)IzP4Yy>|?PNjD+IlyZ&KKsmLr#{G@&`=6?199$Aur%S_EbbpeU|?j_@=zA3MNm_k0Z2ENQ??2Bk4ouECx}d zE|#%ehLbrx1xJf!9A4d5{azo_8`4ZSI+Sr1-Olp_z21(o+KAoh{(f9;L9BMVp#X}~ zULY|2vTFo^SwD2mCn#8o;xz#4ldpm!1vxCFBxzNi0d5fLG?J*n0;P7V63bp9#D*Yb zF;6h&Mov4)^jQd%EeT@jx~l~z;+<6kx2adlas-t=j$AjF=h+RobF|0V4PinCH-&MF z^<;)aw<`RNkfovue)rY#gDb{H67oiBOKfx}36C!6nI{#!mbKB~7b;Zl8 zC|VE9k4Kc<*bS5HXwh|gf}ROCuzHuv&xf@p%#Ds(hJdU7s;A#S{S?Tawqlz{ab?he-c*}6gt~SM(|QKl>E%Hk+dX3D^{L7J zrNQxIilWTL=3c!oTbpy%|GplA1 z8agGqCg(HAazOmo9Ld`pt}>?Wy35P(*A>YK4*&FbS>3d>}>}+A>h#a zc@Kxay-c~yGwSHoNxNlv-RG-jn|xe`Ac(kDsJ3t>le5?T3{9SnIa}F0O=YK7RtSPd zvcFgtxC4spm&@yGQH`>aSFw05o8?bbU6V**UxizokDSdhKtJ_SSzy4#Bm&?+>8>JT zLD3_Jc8@mvre(?UKNl*awJAZABPJ%ccXCqhZp5^DR6!WxJ_v_Wvq=jVQjbv;Z?lc& za({ZptJvU;WXD0--jVv)_LuW%G3c{s^S;9j@*I_1Lazs>&euj>vljIBj%5{7+Zxj* z*((wY{CGU_k(lQU$6{z8zn){@Qn!=Ont=gkVtubnpFIlPNPEGt&uBE^nYT+PpgeSI z84!N0HK(KIwGpqd4pG(aN49=67i?)Hh+M2@OsF2`r?lxziSu3&qR`BZCN*FhYIKGl zdov&3JlN`CE!G!$Rs54EdayB5JO%h9SKp7-?dVMaAWvk#VJA=(^%s6^kdJwi`2yjQ z^#(DJP>wJU5|)i=Y;^l{M02%(`RT0G5Sm2!pg2hgP&kTgf2vClNTgFv<1MJefn<#F zFQTS{5DNAcuC@y{3wWJ)iJT4g7_!^aq%Qz_vts~JbPJ3`$JO-glHZ!>LM=muspyM2VkW>;u1j*9@9cqKyHjhWn|hJ#kAfS#@3jrnGU z#8`XlHUxq;qmnQbMFahk+kB$i*PJaTmN?8IONf5=yvKOV~KK$oj_**F62we|_^kvP1 zUpp@){UZX^P((^c374ZqH=Cejn~CR2lwsJB0$+e6rr&Igg)&KSPBg?-w2VM49CR}o z?#JRQ$=w@`p-{+01vimi+C+W=;1JEzh(}*=3M1#b1{6qCC(q_}BLg$TnD~>1xw;-x z*RC$MqV115aX%Oo$z~;{$`4ix#q3?oSX|%z8-OcPJOwOiswN&ni|Z8Y^8FmbYV-Kq zzrUU@e@r3B{%V-j#VU@;Ms4qa8J!p0&#T3#XvL(?W(6A)a#DqnleIo!*QQ9}cLJEC4!@nMU0HIz9KlQI9d0x>X*Y$Wrsxy1~=n`!NYWc})W1 z_dO0lP>-SCBdwa8$WHZ5Tev{GoPxMmIpog)u?ah@Hu`pi5%D4GJbwm?w8A?GRC#S% zfx`&RTAF-7X=!f{+fJJq_iFJRH`ws@4{~2%F*%m|976s1!eu0h9Z&Ch;$SQTua?WY z3m5=0>tddiC!y3}Ad)b^c4{DfU4P0u5JS|Lu~cin{di^9UsT^|S~~w&V#<(j(&;pC z&*RJQ?v}~&S;PDQxM?9Bz@~KqenYY-qdIU%0i$Oy^3dZ1eLVB&PMZ*TX? z15;!{xUv!GhI&qt;kV0Ozop4Y28xXqqSE_QwNd}yAgM-&+gaqjzdi5`wf(1R5dn1D z)!vx%wyDvSu0u*4DnzrTQ^=`oyDX+*7AvkJVCqfH2t%6G%8G5DsKA5NI?`&S|lJtt^0?wrE=+NYR|zA z$?K(`*lyYw?*T?yiaq{-FPOB$OgOJZw9~Yo6<6y6NgtS~uslmv*iMQ{hsH9Rs^2K7 zz1WDnW+Oa7-8rf6RnCxHe&ePzJ1uy8L_oAMnUZDuB_L7<&q&1TX`cu|sZNn9={q}* zp=7<;y@%q(PgnYJQH%}lN>)QPx1?wqN;F85cG1r0ROOAuVdY`fo`nWsN-DBwaMH)>CP$<=G-RXVfl42)7Pz%TN-Of=%0+O zQi>HvDTA*3Gx0vqe~f%{bCSSnzD(`Jcptk+6EPBCAKtC%)nX_pg%~Z4FT4)kRh>z6 zh&mfdNa9!rs<_!NPQNrk$7Zdnf4Q3}8jk{RmMtNCynVre(}(1`aOykt5WVifaY_CB zog7sS)VZ_B8l_hUyKugtZp&M-)=FWE>)1xZ?}aT;EU&Eb?PLkV6h62-uRsq}Nkkm< zdVjtY{f}}OXJVyp`4(P7xv>Z#%P4*SbaB4c3RcM(vI)fO_lLq>f06N(Zv5n(ouncG z888?uJeF2(e>5#ozZ)aOM{i}X5$v&qH%JG(18wN~!&W}u@K}*_!1bT4+?^GcAAba> zd2EHxPbJE@tD~&ZAi2&R`Up|Tp0L4`p~IJf-m6|>f5hh+Hbtqf26^7!+455WRuTCN>Mcxx>oV4Ran{=k~lz)Nb?`r zk$_;XZVNxXm zFejOJSrQUV+ZF5&xr{J=%^Lr{GGDwe=nz`HHqTei`gTSlb~*}zI4Z3Hce5PPMcXC zytEyeIB%z`1m7vxf6on!o{Ily+}NBf!)xKN{*8LFe;kpZ zUb&qkGKDLksMp~H<6}M!+1c@ODFU^s#-N5Mkj5c$kBE1Rh-O4U_GB2y{FkrxSG?g5 z#h!&I43@%0v`Bxb4lou(bTc$ofL}$2wO(mJ8trg;yxg)29Z9AVdSsTG2R~+@av6@I zk5k5o-o+VI#?Oa&SwKRoMg}h)zZeMI%a5b;`G&i<8NLH#AU&DY>L9iTzB`eoiuMBXxY1(7 zX$8iTz6rA<9cJ>m1-2vQh<4=}wy{o9&?y%oD#7KXD`v1jCZ43~FN+I6S}f@U(+81( zLA(ZQ+zqm7W<*~KA%b@a#J&o~R=}3MuV5jhzz@Y%95BCdfdy2>d4r1eW{M1V0wDfa zV69mnYO(i7L-ArH7q9w9?2d-_!eOLMf6gh7^R6|B{GJH}Iu;s1SUfFs;F}>$w=Myu z){@&CIRs#~s@k1uD_#ydDLHI#=%?&XfYp+yozapR6ZwLGJ>0AwQpIBJuVPxxLBe64LVlN5|PE1T3YFX}aN5QZ- zL*$XikxRhm8%slo_`@O|3xLV0DPVw@V}nRh!NEtkTa2#g$)2wLVZkNQU>SxMs0~QZ z5lBc(|OgZo)V_=44IRx^u+;5NQZ>5bO%3L zp#L|>ThOa1pPSKO#uxjO85qv5$-Uz{15>u6xSS(yz&NpsI?Llb12;FqPH(&w>NK^h zc(4W`K-k*+_L6_o8*Xp0uG9Phpncdf@4*hn)!sm>XMQ!b@KhyJi;6ah`3c(m$A>#X z*FqVgwOk!O82-`0=TAWa^%qnui@1ReH&4IfD}L<(EQjkiaWx~We?(h7M*Q6(Be|Py9kEd}m8Y2ATytM`JG*iaNa(?b^+}%rtoUc{?UDDj^SumMA zUDBvAhWRJC@Z1|IZA8fsh*V=@TIBSoxn5adFve!?zt5MdcDgosYzJmm`aEyOG;)KM zmbPvezF&C`uvwdbb|WHt+Rq5Rt%e2%hSh@2`+v|qA%$3^s0X!D|B6AF`9A#sQ%&X zXL7k#y1=*17}4v`(1nO5rjP(Dhn6GSN&rY{x!7tjxlea$b4_vU_Rrq#`Sn%!c(!D09E1gzQ@02d8rq0o@n|E+ z0X0PgahdhVHS8hT6U6Fj)sG3{-dJG7?^e?PY5{Nr?KVPNVAYCD(MqF@`*ZAL2&q3m zuDv|)IBx~Q?{bbde)v3`TPW4jw{Y58PrF+)Qf`o4nhy&R{US+xC$-{@Nypc4Xg2JGq6KF7F za;TVu(rG?ZN-y zuDLzTr$tb!Sik`|M8f<9`NwR7&^v6g>sjEaVv{^U5{&*x4#qP2*QJ%7mnWwT{U%gb zYL%~GI(oPEB{ZaQj8LKl4qOQXjASFlVy?bwc@=m_#Z9Z} z*f>|KSeEQ6b5p{#lKXuCV>T`p3q`4!sy`R#xX30@@{3 z4;^J*cTdke2?RfM_WwZ}63?gAP{XESEkqhsW>JXwEu1SwucJjNk@DEU8^o_U^Mp+2 z=nH@xH>euZo(c?_I2u{(GCV=Tt5Mh%3=m0Dw%$@YDJmN{vtAqQ753(OyBGBUgav!G z1umSVeIpnU;s)n5#X?UD$b%JUWG>73w*tEwVWWPnf=PX|l>+jjSS&$TUCzL{2q|F& zX&t?L!=iDN-25~FOi2%1sxcHURg?ZxdFo+!FHek=a(h*}>1$5hqsPyzu_qi{gr<&{S?M+uwZc`t9f<|y$Dp}<@A*j@ z$EHgi?l+3dstAZZoy*iw`Wz3CF>dF{pF*3C!Y@%~sQ;ta6sLy_jD(rQ3?SgP4gP=K zefK+D-TO5%M08?=5C)@{=sYC4FiMCP1|hl-6214{OY~l%_ZBUBCrTnpbkTc_8s0t6 z^ZEW0uU}jj*PJ=~?6db?_qx};?xshR7~~?3?85ek75cHvZ%KxeEA^{)rHh%w$t>|9 z!lx`zvda4)gZ@CCc#>Z7_$&+lK8GD+S?Tz}Dxioy+zmXOTXJalBIy42=5W93ev8LB zf#vz0T2C!jo%MT7f%NwK3_uokXjg8yHS|55?&Qua9kQsebN%!^}>9xfYsS< zBgQZypcj2e?0k95FqptX2q>4H6dO9n@6X=RW?in-^?)klHN?z1L zKXBQnLJEl(eG4XL&R~lAr4%Q2t8#nErl&hR_XGxCwA#$e1rl0mE{xy34Tim?cdQ-? zA4{=CiIPR<06A2cD=65yOc#MXrnss0X!c@d`|mV1Zm>~hB!!d)c!RR$*R8FE-)op+ znO=ss1ro>rib}<4NqnuK&GOjGYgMsFLawza7~zHzC_y0@IwxE8ytYd`h$0KN`|zDg zRt!Pw%s!Ay0egFVirL-mUw}eu5O$9lF3ZcylWyXT0WxVdR^ZV1S~R zkHQ`mU;G5|_ViRq5N+G-JAqiq<8Og4gj=$80$^iT%!zhr#+^OT~$}2+>L0*SpOGo`e>)Ui>Nr7wXb?l_7@UAzIzq zJXsw~qz5U*#CD%M$P@nsdKEg4P}0JSw!7r7@c))~E;j}sCe=)S13&VR$ub>TBe<2( z4(@un|L?KKG}-|s#0HG%jS9LL>=*f~J%lYJadNlg4+}V&A5th0 z1sj0xyyanH`oux>F&;X)*z&-1`X9I7SSZ8Tz%TE03UcrBM_ z1Q$2BJYL&eY!kcsvlmAF+*R5adiSN98-N*=Y{|lhfpo6*4R5DDo^XLizDEfA`K_^c z0l#42F*A=TB2Q?d{- z@CMZ66lM>HYF3dAc%5xSi#2ZFa2$YU*l+dnfvi$eW(gGL3fZmWy-XJl0HTOo^YI)_ znecAU(TwL>wdTaFcMEs_5Qen?y`qTYZhMs2ZK3rJUVz(gG34U6cz(-CgN>2&2Dfug zPThvA;O7B$Ye1p$4YXPS3@%0>s-t%Ko3EjuyDPm+C~3?{-G(1crAs;!+0)b@dZxcM zcNUU@)Fii^mxDRt8Q+)pzbLPmMkdWh+3B@;h+-r79PkAXbg@s!^G7JC_w3e&qq-dt z6YBb{hhDe(KP^44wZ8^fC7*P0zCG-i3rAmWYwz4ptQbRj?d;1y#E2J zr5T=Ee6PHp2-ti#$?u%S(Q@Tt6oJoE3PaRNI)A^{;jm>C}|O={WXh^*Td{0%Pp1Soep0F1;~AhZr|56k-AwTOS(_EftX4pxu9^U+F? zDg3^@Agfj-4CtC@#?EdoPm;vET2+NWv*GN8jSVV=9n0^g^bJRS=k1qjPF1eoSa(G# zQI8bwqYN!^U9@ntcgzKlQ@Rpk?^E!u{X^GS5SCSB!6T)U8SVbHfN3uWj)FubB0@{s z&+qKDQ-t_zBHw%)CS(Udh=FF-yov>x0y?9W|3S8Sdc?0Wg7!ay%KVl5X`zgIR-o%q z!LPnhC@y%-)yUql;)FeViWTw)6yjE z(ib!CCo<712)M0iyHzDS2*jt%0lgnLrz&V#zJ84_!M*;0V=Xr=)fM;!8@>cT(PfH{ zt1{{{Jlp=6b(S^=xo7C&*8Yq46Ud&i^{jyoCdE%eT+NNfWFm`Bo$1@T!L!J*rqQaj zZTCh;AEL*C-v$w2I#5zPcjcR#n~MfeQ)StFqVN?_z(7I!0F zv2c0ff;~(B+ZUP+q|in0t*iI~{n7eWHprj^7ut_bXD# z#wgcsE7x@=_Z~FVw2#T=PyY^z0cD~Qorn)bu(1`Tb13o-NSgEdh2p(Yj_WyTs8kwT z0nd1inQ|TH0`&(fCnu-1s}o>lU3?4Vz8$`pxavRUNn970!6akq)LXsUm+I}!I&0M| zL%w5g!Wbva4DAL7jry)PBiI;J;F-67Iv(hy07qxwdq6%Xiy7Pi){n3FgcL{tOo*t> z#5};B1|?i+Vl-aXvw@05YDNp*=>31|+#Q;^jfeyttr|SOzWUJ&Ft=+=tYdv78iIF# ztEA$Gs}kJ^Um|>~Wt*{M{rc|aB#LefURG=VL$yM?!iD1{kKz<5vdf|_>WN5w^<12H z%^-)}O5AD96OXkmlCFwge;lk3K^x-fL^31I!6*FhNRL+sa+~79f$mqkENJSu?e?mq ziJ@Q?uV99jyrFv@KvZ{ z>JU6pE#i9}MV^h7S#8UZusiM3Rb!UQmkbJCmSTGag3)YZEtHsu^RMwau4@f`oCRuL zq8#24Cfzw3_w?hM_Ev9^v@uZe@<@Y$x-Y<+xuSv+{leCl~an z>~%#3j_z6n8sdX_$ALW?UB||IbdN(8bt>NzJEo!nT=oDt(%gTkPF;8r z{O>G_qh?u1b|xShdm3jzzut}stD$i@sab^VVA_YJM00s?r6>5IhybV(>L`>6``|D@ z66Z+q2A)Yad3eadXL0gfNFF9WBon|B zf=$8cWc}McE|=SzM!G@d7>Ay_dB8>ln#U1~{S+-NBf0=%ZD-uHo!+SvA!5@@`Lka~ zxjGIW?=Nynzw7*f3`;(BVe(3a@XccBM*A(2cax<>)wmy(h6SY0pRh{UyMXSgp{jwK z)t`9HUP3E}V5=a|SpqKZI7Y+(1cQ1EP&M;-gSnaOA-WFjWvR0cfXm@q69~*Xp6Ndm zpsCqw#iGWec66$bPJ$MR%QD9n(9c8upbP^aLOv{SDKf8Y9+K~nGD=x?4PSM?mFuoC z8Oq1_u$3eAn(7aI2h-)3KPQNie+C5ZeMpYWDCdK(4eyMOl8alF+Lkl)jvW^0i-w3i zc<$#?)NXDI#6iPJ+)_PQTn@~BK)56k=C${$uI!^fPd+LzAOa*GJ+^c5q`+VUO{ZH`A5RuRajSI!<^s;Z z6n-*aEeQs^A25oQEM;qW2_}q>k$rUr8f;2WxY(h%tn{Zo1w5Uz_f3zNIqvy+ zAotgI!=bZu@`vM=T8!F42`?1f&A4IDvgp*EW_v6hd@s@T zvtau7D*s;W7R+iQ6D^WHE9M5IdRFE-1BgR=cF%TVCPoN2iZ#@B4&Ba2N-?y4F_tPF z?}^lQ4Ls}vDauVX`Z!#0EB{`k0~CMhXM_X5{o+_*V<-s9$w?g8Q>BnNHat!Ys)ST- zEZ^)^a`yfL2DYDZSR10z-6xs*o=pB#@B(iZDn68e3DX;ZJ<={SY=yx$0U9-eHn>19 zh*2+jtbMrsx#FoNA&oV+I2g?5m9m~lc`Opthftpd;w8@5Q9hH}X^^uJy5%u|(1X~t zO)H3?@Kp|^*Kyy9Te=i+%s-dpmKqR@+N;{1K~Sw~LD$hD(gYyJ14GavLDz$4ZnbGa zE1VWR&1Z`17&a2CAU@bK?V56)$s_48!rDBgPq;e`p-g3n<8#n?-qV9hJ*5-MrYXHi zF}Uh0ZY<+1Q*^xkEsVzVLSn7_5i#lp+hehlq>n{@IR$!ba3-Yb_mV*9NKvZ?#j=+= z>_m&$nN9!$c9WU6uS%1+4YOX?n93yxRSgJf$E`Luf9s@j64Y4g>*(lMtpmiPI1$?Z zTx0I>W({%=0CP-||DHf7cmj)g7HU)?uB1>ljb7;~7BW3=JVtD30M#UqLpprRbTr=F z%o5UIiA(O>;+UBkk4ASGC;cyCKl-7}G49kZ9(Nry+3|5&#cHM&1TZqM;yZFEF+io! zt9aWXprBnFN|gPwH{^*4h6Oq{&|Ri9N7jXIYq>=6w7`hyFzOSSGI2fbmhWY>d6u;< zxLy#odU_UWG}R6i6vo_)yr0eXh{|9*CoJ>J8wK9#JX(A*OQT|U-6%!buR2Sk$x`ic zQp}lp)O6sYd%)YYd8wY;Lcn3uTmJRBtd5N0H4(NG#42Xh5>^Vj1${Le7N5butehAh zH?~S4qzvqO^we;|8@=#IjSV%FY|nX`BcNCXC%A+U1}3|3J?e=(g2?Umg;V%}JZ84n z-R-!^Y_&3Yir7pL`U50>-e}@Q5;FJXwX1%pX%1L4Q?>}<&%;_yBEnTfV-A@-FJYK_W%mg3FWTDhIY>lW(C=FOI;>^sN~e z$&`jDDdpYS=KBvB6~${zzYQZZ?ZG<4#Q?EaXXV?jiJFCC4YuwG%13qYeoBI4nA?7H z%=`V=NqQCnAQLVsQ~)LFF?<%P;ZH=!P;?CXZ#?H&#EYTO8;Io8E;} z^jig5h97C)T7K-CrOX9i0rc805^61=#OE&8WK89{9kWgcD&+$JfL8c@kL#*naw+DHL22FGYZ0AAp0rS&|L|NJ7PyioUZF~bzC(E|+PK=1h< zOcx$wXjWp5iZLsQ`R1S@X~n zQFzM!rLTyBb+I=YFHb2~`Qe`(=UqQIQV=&fvX($0lH#z*>vLzj8*EnloqS}&`A$&E zXL%X<7Cet5-hBChh}8NewK-?9*)UB=32?Vh;5caRG^(`sz5qU@fVwfT68OX_ zp5XcCwE6D(pbZ*ww^fh{jC)IU?FIgWv9t;a?C(T|5;)XWGxAy;b|#tC3yHwyUT|{) z4F5~m@0ebup?PJH7qST&)nAbIrvOP+n74K8XgJ#NTFJnbk`5phdi$I0v>Rat-r}kU z_sO3-gdQ6eaG@poG{hj}4t0b?*aV-*CocZV(o-JN%>Aiq*3{gb3wm(IJfPMtffqk$$*-E1T~L|XszMA#l}XJCwVaSrBucX_fgh#$0?`5rJ6MLauP9}+|T z82NIJHbQU!z(awO1W9qhfen9?p?JTkzt``q2da&V=%YVcq5C9diqg*fZ7!TWanfz^ z(BWWF+nxyB6)e-8Axj`%dA#Y@2INpeZqAU}fpq%&WEHwb;|>QiUvd9l2^CHx-zT6_ zk?^PU6}Ol5E*Bu-qnyvgm}!F#CS6E1;~m8wWHky*p_Ch@+Qi|-`UifM2jQBCe2C-gr~MYHFC`hiaX4ZGAFM`5*)H8D$Hn5y7%_Jm>|r;*x2p{&x$% z`xc3;EIB^ejpGNM1TZ$*uJuBu2FRQ~^OoYVYde%W4Dyu!`)t8ZY48t3k?t}r;Wh}5 zE+?3O(e^$q{R}mRO@?U4q=RME??sOW`@h2OtyfL63-Ugp@vOP>KZ^@>Dtt5&3bc0` zv@AI+;`S(f;;Jior{5xtsJYtOGdAy8tTw^LA!W|`Qc?@fqW^Fh085hX#<8OgfA;4o z0sljwRo$>BJ+?Ja>+2m)Y@o9Iv<%rkv?D5?$Qc1pla|X1iZ=kh0X~5520`R$EX=KK zX)oE%9y?0&E}z7Rn@DMHZUZ@=L!M6X;twnKGzP}stJ6+UA0KbIaJ)9u6KaDSL}y6^ zv4Z&2n2qL`3?%@{QZWhVVy55Gn8H{5lN7j?GHH@vri^$kOgGkBMj=t>+0xTNT24x^ z`#-=K0_B(fM@B#21>sW$f&)u7Hhzc)fE(B|B7;{?d$&OEwU?lsI~ZPjJrP&xsvTt@ z3;(6Z{+tWf@J!E2(X|yWk9Gu%=#03*^NbaFGok*z_C>_wBoQa;iN)Go6)-qO$t56i zDZY5bcCSD#rZcbgyB7FsX_R=9oM@t1()*xAwN#q*Ro_94Vx-p@j~@#9Q2FQxmgjmj z3uK9T_J&hbIqYa2HP#6Y@?^n?NPYfCjW4Qq)tOYbeQ8ERBN9MJOQ@%T^1{M72S70b zFtQThQUq$dAkh2y+hhQT`c%>rnZ}MXROX$7akgnk05HlL)rwlU(gYC0-f$nU+lA2a z+!w6APdzVUe$UYjv3huTRGE*{@xA{=3Pfh%>_Y@o4&N~+M@9%h42sKh5X+#Tk|#%; z8X7hO67QbZdxCQ`vYbL%?F#r^hB=j*yi5Q16P$eP%o)5ijWELRhnKetvL0a0B zc&GxjkI;C?Dh@Q#{<*pIEiEl51raHyuGZh{#k<=eKq4Cby*lep;V)6zwFDgW#ozNq zjqnVHRS-O1>^C0(Ye@`%s(MJI2FZQJf!N&;p2_@Oc5!Cr9~k*-srr{+uPP5FwEI14 zJtvG$>I20AB#&W!>z?-Pqb{^T#C|gxNvCEqlFTcsYj3sJp409scM3Q=nwzVgihl4w zNT**-f2lP6_Bfh-5!69H;?z~?&LH76RRkrPQYxfDI0}VMw8cUeSvjry9c-#S-{AAOIN)D zX?rsCuvqsT>4;X4T5oVZG9(%^pRt+QHbc}=n9wlIXc>#cUhYoICBdkvUBuK^J%1K! zpbuHk)|WNKKMX-)yI-9?5ET`bi-kWh><%VgSqE_|CNH`(!IJ87PJ_J~mYzKLDO7NC9Mkol>8pv7{f<2s`cu14hvh zjC0B5Fgc6p4oHQNC$HkGFn#1gyn7I$uMDU?ZVisN7r7E!1Bq%5R|n(nGbntL9$)F{ z>7-$yd7^D^ED3Wg)e3&}lpy2Tv8JY=vht`+|MMS9{|E`e0EZZ3p;N6%j~xwBO5&23 zA4X+L0QaidmS}N_$9{Ur*ywt9`}K+$9ADG57ICj~iXwQ8dhs+Ae2%mO;W1<@T2+Ri zoS7u_(?@sDku&$Zzvr-!3iYDccQwFiCr>F=v(b}hGj-JsaI>KhHMyLra=jAlGVZsT zG4!&ml7)!5hL2F%-MPjZx|UOPvnA#S^Z|A+Qw8vQ!X6QB{@&OqkQ1pI>sFyy*Ydji z`$biS+_F8h0`TH&IWIqVkl(_ay8R&@n`7DAW^t|i9Rr}nObko{<7^yIoXaaKB;NRo zxrXg_SwpzL--|$ zeApkrl&Tw;fym|_V-_VgLM;FXI*k_+*zJW*-2YwwBRVP z9B#Y2E%`)iC?)e*zs+---{x)M@DcA-t=ijyvulv>Q}OcKqV5q;{3Ov6Vz#`#p3lD3 zUGoZvKOfcR(1ERs`t#f?^tbGU+_^)9dQL7Gt7y89Mx)L88Y$vBrn(a{V-A1p1(JEi zx{F$pvU*wR1KYo|zhK@!U6VkC7Wh8S8zymG<-+iDxG^-DE+5z5e3r!xaFlrz!JSx! zi=ipuJ@cV<#Xei9UF-(r8@ma`ZeU)pGFy^*ZsxKqxCC6~nH|mp+Qrdi-h!t0Kl`1* z-WMN`WUOD)SwjQT^1@y=ai>4<{W9b1$MFLff5@hFWKsEXver^rN(xdTbtuvhlFGJi zCPQR0cp!^bA-4{;UO%)Kv3(mSU=7H-086(Dx)F6Qx$lTS5f2bBzt@P!EsWd{GBiaK zNi}ouXV)U46Ei{xylwikj z6S|-}rWb52)*Wf2UaAFLTP`;Qt>3EUE54G4+L)sHzV!bEXXH=5mdlM9o<-4sL8y%~ zvpvH~L~Qohqs&n>DDDCd(ZdDlyqD?7gX+)eFrxQF*n+V(!OUKTm$H6&nQgF5O{b$3 zOg#w0ue5M0b)rf%jK_`i2o|0CQaUcWaR9ajnyk$8QvE-zhVJ8^F6aMDf`oLUDK{OO z&&1;q=U{@-!|Uziv-nfDxGY#UgdUH>i}KeJB=Xd$&DYl5Pml7UX?dTlg~BhF4m?PZ zB>b^A?H@7VO)VZ-bmAokfNqE{A(&EdN*!ClsjX9!Kzw43*|uSZH`qVQ4oG)fNpg{wAZV{|o;T}@JkvRqjHC8zJwA^YYLEXb7h z+OCn-Vyb$2+$1>E{}1m7x49A%qF_ZvCYVzg-lofBGbjjJ$dXbv{UA5&r=-q!KJTBD zA18?J8mdElKjSI)`kZ>NGre@ch)VEN=p$AK@P2F1pZR3+HxUv$k=&rjMM?LL9_+)p z+pC?Y{EQpxye|S2Q2OOtDVU^OYt=o21oUrZ-mBeXI7X#U#Wwg^y(ng5FZ%9#yuyjN zQ@Hb{Jm=!;S!#V4ciyjBtXM^ zPvM?Wj_iNJ_F|YkI3y{U_~CX`=~dK0PjGhmpiwWHVPS=YAHCF-HrI{IvZ&m;3i=soCi~2)229p!FyT{>QHIM}su3sK-pvMD*Bqvo!yh)x>f(`;hEWu_*;#<#4$| z7N4nX4G*SC(4zX*^q>5t3J17$K4u2_i4*l`KpT@eOTjhCe7JMI`lJQ%Mg-mwytg|> zkkDNTgl**e>7n=ZJ~FqsWPeVFq+CyCVD?I=I4c0lNdqoNpHTEZ93}IC4PEw78QPBp z-(Rr1lytTA=-!Vf%V2`iHwngk`VkQD&t-BDQVgV>rFhy{aNgT@IUG=f)NJG z3vyWqo_mY2$-ZL0A2TO%>Z$(bA{5>50I~<$#cWEv0L>5k2sUz=PUQe^Yialme1-#A ztK#-~huQ{n)7sd)&TR6rY0+CZR^#0vATe|241P49w>fD%j2Sp63HZ>Tc3u3(ra?Qu zIiB0N6+RpL79t;;FT}&?z0aQ#+wW53f294p$zp9o`;aW~Bc*zixp{UckmDR(?jAY@ zG%zS8sbH?EF0U*UwuhjCUNMjoT0>M)K=@=N)UfyhI#+OSaGT%COHMVuBHvc|Ttz=w7 zOcF0FY0~5>F3w$5d+9yegAD5B_x=evuPiHK+>xaSE(Cmw%y9FuB!<#T|DqD!McxqZ z0PoSVnLXj#N%E#72(|ub6UOgRQF|l0aYpowGMp*_#*V#FEgH`eN;}iv*=$oF5+}`Q z`I<#XdSObJcQl>IB7+#j57@9I_kMptag;~bUIS04y;TElCQ<p=fb?DCrRpDAR3=DVh_2m=xfbn5aj|v|$MfeN z{abNLGQQo>)?)VOS#8fl9JKsbV>spPKXMPd(Ksst#l*dAhs&87v%#=vy{`3dV|K+j%N03Z{f3(&`$A1KD!2i-y|Ni>_=o3qK-E2Wm{6p;1(ZCNv M=B0GGq=E1M1J_%There are at least a couple of reasons why you would want to do this: +> +>- You run everything else in Kubernetes, or any other kind of similar infrastructure, and would like k6 to be executed in the same fashion as all your other infrastructure components. +>- You want to run your tests within your private network for security and/or privacy reasons. + +https://grafana.com/blog/2022/06/23/running-distributed-load-tests-on-kubernetes + ## Context ### Ancient History :t-rex: @@ -33,15 +46,16 @@ While the current distributed execution solution is much better than before, it' One big shortcoming is hidden behind the "_once the test fully starts_" caveat above. While execution segments allow the actual test execution to happen independently on multiple machines so that "the sum of the parts" is equal to the whole test run (as if it was executed on a single machine), they don't do anything for all of the initialization logic and actions that happen before the test run starts and after it ends, or for error handling, or for metric crunching and thresholds. Let's assume we have somehow fully partitioned a test run into multiple execution segments and we have initialized all of these planned "instances" (e.g. containers, k8s pods, VMs, etc.). Here is a list of (hopefully) all of the tasks that need to be handled so that distributed k6 execution has good UX and works nicely: -1. We _somehow_ need to distribute the script [.tar archive bundle](https://k6.io/docs/misc/archive-command/) among all of the "instances" that will run the test. Ideally without the user having to manually do anything except point the instances to a central node or registry of some sort. -2. We _somehow_ need to ensure that [`setup()`](https://k6.io/docs/using-k6/test-lifecycle/) is executed only once (i.e. on a single instance), but that any errors in it are correctly handled, and that its results are distributed to the other instances that didn't execute it. -3. _Something_ needs to ensure that all instances have finished initializing their VUs before we _somehow_ start the actual test run on all of them simultaneously. This simultaneous start is important because, without it, a distributed test run will behave very differently to the same test run on a single big machine. Everything will be out of whack and k6 will behave unpredictably, which is something we don't want in a tool that is used to measure other services. Predictable performance is key when you are the measuring stick! -4. While the test execution will be autonomously handled by each k6 instance for its own execution segment, _something_ needs to handle failures in a user-friendly way. For example, some of the k6 instances might suffer network problems or might die due to a bad test script or out-of-memory issues, so we somehow need to detect such problems and potentially stop the whole test. -5. While the test is executing, we _somehow_ need to collect all of the metrics from all instances and crunch them, because we need the aggregated metrics to produce the [end-of-test summary](https://k6.io/docs/results-output/end-of-test/) and to calculate whether the [script thresholds](https://k6.io/docs/using-k6/thresholds/) were crossed. For these calculations to be correct, we can't do them separately on every k6 instance in a distributed test, we need to first aggregate the results centrally and use that for the calculations. This is particularly important because some thresholds might be [configured with `abortOnFail`](https://k6.io/docs/using-k6/thresholds/#abort), which means that we need to be continuously crunching the data, since they can require us to stop the test run mid-test! -6. Besides thresholds with `abortOnFail` and errors, there are also a lot of other ways to prematurely stop a k6 test run, e.g. by calling [`test.abort()` from `k6/execution`](https://k6.io/docs/javascript-api/k6-execution/#test) or simply hitting Ctrl+C. _Something_ needs to handle such cases nicely. See https://github.com/grafana/k6/issues/2804 for more details, but ideally we should have the same nice UX in distributed k6 test that we have in regular local k6 tests, including the same exit codes that can be checked in a CI system. -7. Even if the test hasn't finished prematurely, _something_ needs to detect that all instances are done with their part of the test run. Because of executors like [`shared-iterations`](https://k6.io/docs/using-k6/scenarios/executors/shared-iterations/) and [`per-vu-iterations`](https://k6.io/docs/using-k6/scenarios/executors/per-vu-iterations/) and because iteration durations vary, only the maximum test duration is predictable and bounded, but the test might finish a lot sooner than that max possible duration and good UX would be to not force the user to wait needlessly. -8. Regardless of whether the test has finished nominally or prematurely, _something_ needs to detect that it _has_ finished and must run `teardown()` on only one of the available instances, even if there were errors during the test. This is important because `setup()` might have potentially allocated costly resources. That said, any errors during `teardown()` execution must also be handled nicely. -9. After `teardown()` has been executed, we _somehow_ need to produce the [end-of-test summary](https://k6.io/docs/results-output/end-of-test/) by executing the [`handleSummary()` function](https://k6.io/docs/results-output/end-of-test/custom-summary/) on a k6 instance _somewhere_. For the best UX, the differences between local and distributed k6 runs should be as minimal as possible, so the user should be able to see the end-of-test summary in their terminal or CI system, regardless of whether the k6 test was local or distributed. +1. **Pass the archive**: We _somehow_ need to distribute the script [.tar archive bundle](https://k6.io/docs/misc/archive-command/) among all of the "instances" that will run the test. Ideally without the user having to manually do anything except point the instances to a central node or registry of some sort. +2. **Run setup once**: We _somehow_ need to ensure that [`setup()`](https://k6.io/docs/using-k6/test-lifecycle/) is executed only once (i.e. on a single instance), but that any errors in it are correctly handled, and that its results are distributed to the other instances that didn't execute it. +3. **Ready status**: _Something_ needs to ensure that all instances have finished initializing their VUs before we _somehow_ start the actual test run on all of them simultaneously. This simultaneous start is important because, without it, a distributed test run will behave very differently to the same test run on a single big machine. Everything will be out of whack and k6 will behave unpredictably, which is something we don't want in a tool that is used to measure other services. Predictable performance is key when you are the measuring stick! +4. **Failure detection**: While the test execution will be autonomously handled by each k6 instance for its own execution segment, _something_ needs to handle failures in a user-friendly way. For example, some of the k6 instances might suffer network problems or might die due to a bad test script or out-of-memory issues, so we somehow need to detect such problems and potentially stop the whole test. +5. **Metric aggregation**: While the test is executing, we _somehow_ need to collect all of the metrics from all instances and crunch them, because we need the aggregated metrics to produce the [end-of-test summary](https://k6.io/docs/results-output/end-of-test/) and to calculate whether the [script thresholds](https://k6.io/docs/using-k6/thresholds/) were crossed. For these calculations to be correct, we can't do them separately on every k6 instance in a distributed test, we need to first aggregate the results centrally and use that for the calculations. This is particularly important because some thresholds might be [configured with `abortOnFail`](https://k6.io/docs/using-k6/thresholds/#abort), which means that we need to be continuously crunching the data, since they can require us to stop the test run mid-test! +6. **Abort the test-run**: Besides thresholds with `abortOnFail` and errors, there are also a lot of other ways to prematurely stop a k6 test run, e.g. by calling [`test.abort()` from `k6/execution`](https://k6.io/docs/javascript-api/k6-execution/#test) or simply hitting Ctrl+C. _Something_ needs to handle such cases nicely. See https://github.com/grafana/k6/issues/2804 for more details, but ideally we should have the same nice UX in distributed k6 test that we have in regular local k6 tests, including the same exit codes that can be checked in a CI system. +7. **Done status**: Even if the test hasn't finished prematurely, _something_ needs to detect that all instances are done with their part of the test run. Because of executors like [`shared-iterations`](https://k6.io/docs/using-k6/scenarios/executors/shared-iterations/) and [`per-vu-iterations`](https://k6.io/docs/using-k6/scenarios/executors/per-vu-iterations/) and because iteration durations vary, only the maximum test duration is predictable and bounded, but the test might finish a lot sooner than that max possible duration and good UX would be to not force the user to wait needlessly. +8. **Run teardown once**: Regardless of whether the test has finished nominally or prematurely, _something_ needs to detect that it _has_ finished and must run `teardown()` on only one of the available instances, even if there were errors during the test. This is important because `setup()` might have potentially allocated costly resources. That said, any errors during `teardown()` execution must also be handled nicely. +9. **End-of-test and handleSummary**: After `teardown()` has been executed, we _somehow_ need to produce the [end-of-test summary](https://k6.io/docs/results-output/end-of-test/) by executing the [`handleSummary()` function](https://k6.io/docs/results-output/end-of-test/custom-summary/) on a k6 instance _somewhere_. For the best UX, the differences between local and distributed k6 runs should be as minimal as possible, so the user should be able to see the end-of-test summary in their terminal or CI system, regardless of whether the k6 test was local or distributed. +10. **Cloud metrics output**: We need to support `k6 run -o cloud script.js` case, and unfortunately, it requires a [creation phase](https://github.com/grafana/k6/blob/b5a6febd56385326ea849bde25ba09ed6324c046/output/cloud/output.go#L184-L188) for the test for registering the test on the k6 Cloud platform. It means that in case of distributed test, we may end with registering the same test multiple times. Then, moving out from the Cloud metrics output the test creation phase sounds more or less a pre-requiste for being able to deliver and use a distributed execution integrated with k6 Cloud. In concrete, it means to address [#3282](https://github.com/grafana/k6/issues/3282). If we need to narrow down the scope then we may decide, for a fist experimental phase, to not support this use-case. It would error if the distributed execution runs with the Cloud metrics output (`-o cloud`) set. So, yeah, while execution segments handle most of the heavy lifting during the test execution, there are plenty of other peripheral things that need to be handled separately in order to have fully-featured distributed k6 execution... :sweat_smile: @@ -67,7 +81,7 @@ So, ideally, instead of figuring out how to open the limited cloud solution we c Let's start with the goals first: 1. Ideally, implement all 9 points from the list :arrow_up:, as well as anything else required, to get native distributed execution with great UX in k6 :sweat_smile: -2. Have a distributed execution solution that works well for all k6 OSS users, regardless of whether they want to use it directly or via something like k6-operator. At the same time, k6-operator and k6 cloud both should also be able to make use of some of the new features and become even better than they currently are! +2. Have a distributed execution solution that works well for all k6 open-source users, regardless of whether they want to use it directly or via something like k6-operator. At the same time, k6-operator and k6 cloud both should also be able to make use of some of the new features and become even better than they currently are! 3. As much as practically possible, do it in a backwards compatible manner. Some changes in k6 cloud and k6-operator are expected and even wanted, but if we keep `k6 run` backwards compatible, these changes don't _need_ to happen immediately, they can be incremental and non-breaking improvements over time. 4. As much as practically possible, have only one way to do a certain thing. We want to avoid re-implementing the same logic in multiple places whenever possible. @@ -75,9 +89,13 @@ Let's start with the goals first: The good news is that all of these goals seem achievable :tada: Or at least this is my opinion after making the proof-of-concept implementation (https://github.com/grafana/k6/pull/2816). -My proposal is to basically continue what [#1007](https://github.com/grafana/k6/pull/1007) started... :sweat_smile: That is, continue moving more and more of the logic out of the center and towards the edge, towards the individual k6 "worker" instances that actually execute the distributed test run. For the best UX we should probably still have some sort of a central coordinator node, but if we reverse the control flow,so that k6 "worker" instances connect to it and are the main actors instead of the central node, then the central coordinator node can be relatively "dumb". +#### Reverse the pattern -After all, a single k6 instance is already perfectly capable of executing all of the steps in a local `k6 run script.js` test run, from start to finish, completely on its own! And while every single k6 instance in a distributed test run is aware of its own execution segment, it is also already _somewhat_ aware of the existence of the other instances in the test run, from the execution segment sequence. It may not know their IPs or any other details, but it already knows that they exist and even their total number and segments! +My proposal is to basically continue what [#1007](https://github.com/grafana/k6/pull/1007) started... :sweat_smile: That is, continue moving more and more of the logic out of the center and towards the edge, towards the individual k6 "worker" instances that actually execute the distributed test run. For the best UX we should probably still have some sort of a central coordinator node, but if we reverse the control flow, so that k6 "worker" instances connect to it and are the main actors instead of the central node, then the central coordinator node can be relatively "dumb". + +After all, a single k6 instance is already perfectly capable of executing all the steps in a local `k6 run script.js` test run, from start to finish, completely on its own! And while every single k6 instance in a distributed test run is aware of its own execution segment, it is also already _somewhat_ aware of the existence of the other instances in the test run, from the execution segment sequence. It may not know their IPs or any other details, but it already knows that they exist and even their total number and segments! + +![reverse the pattern](020-architecture.png) #### Instance Synchronization API @@ -92,7 +110,7 @@ type Controller interface { // error are saved for the ID and returned for all other calls with it. // // This is an atomic function, so any calls to it while the callback is - // being executed the the same ID will wait for the first call to to finish + // being executed the the same ID will wait for the first call to finish // and receive its result. GetOrCreateData(id string, callback func() ([]byte, error)) ([]byte, error) @@ -155,6 +173,33 @@ The only missing piece is efficiently streaming the metrics from agents to the c One remaining prerequisite is efficient metric transmission over the network and efficient metric processing by the coordinator node. That can be easily implemented for `Counter`, `Gauge`, and `Rate` metrics because all of their raw `Sample` values are easily aggregated without a data loss (at least when it concerns the end-of-test summary and the current k6 thresholds), and aggregations of their aggregations will also be correct (we are basically just summing numbers or comparing times :sweat_smile:). However, for `Trend` values, we probably should implement HDR/sparse histograms (https://github.com/grafana/k6/issues/763) before we have distributed execution. Not just because of the more efficient network transmission, but mostly because we don't want to overwhelm the `k6 coordinator` node or have it run out of memory (`Trend` metric sinks currently keep all of the raw observation values in memory...). +As these days, most of the observability world is using OpenTelemetry protocol for emitting metrics, we may benefit from it as well. The coordinator node should expose an OpenTelemetry endpoint for ingesting metrics, with the following advantages: +- It supports an efficient binary format over gRPC +- It supports delta temporality as k6 +- It supports histogram aggregation +- An OpenTelemetry metrics output is something we want to add independently from the distributed execution feature + +In any case, when performance matters or a high cardinality of metrics is generated then we should encourage people to use a proper storage systems for metrics, via outputs. As k6 is not a metrics database, it is expected to be good enough at store a non-intensive metrics generation but in case of high volume then it may be better if we use the right tool for the job. +Otherwise, as explained below, the coordinator could be flooded and become unstable, potentially we may mitigate the issue in case of spikes having classic back pressure and retry mechanisms. + +#### Error handling + +TODO: define clearly how error handling is expected to work, as it is a tricky point so we have to explore it deeply in advance. + +#### Test abortion + +A test can be aborted in several ways, as described in [this summary issue](https://github.com/grafana/k6/issues/2804). For example, in the event the running script invokes the `test.abort()` function, below the expected flow: + +1. The agent will intercept the abortion as done by a non-distributed run, and it emits an event for aborting the distributed test +2. Then the agent will continue the abort flow for the single instance that will lead to stop the test on the instance and exit +3. TC reacts to the received event broadcasting it to all instances, as they should have a dedicated routine listening for any abortion event +4. All the instances executes the abortion flow as the point `3`, returning the same error received from the event. + +Instead, if the abortion process is started by the Coordinator, for example, if it has received a SIGTERM signal, the following is the expected flow: +1. Declare the test abort emitting a dedicated event for it +2. Close the connections with instances using the gRPC GOAWAY flow +3. Each Agent instances listening for abortion event will process it stopping the run and exiting + #### High Availability As you might have noticed, the central `k6 coordinator` node is a single point of failure in the proposed architecture. Because of its simple API, it can actually be made highly-available fairly easily (or maybe even replaced with already fault-tolerant/HA components), but I'd argue it is not worth it to spend the time on that, at least in the first version. @@ -167,6 +212,81 @@ Because of the potential complexity of k6 scripts (especially ones with multiple In summary, if we make the `k6 coordinator` node dumb enough and the metric transmission efficient enough, it'd be much more likely for a test to fail because of a bad script that eats up all of the RAM, or because of network issues, or due to all sorts of other potential problems on the `agent` side... So, a non-highly-available `k6 coordinator` is probably good enough, at least to start with. +#### Disruption resilience + +Based on the high availability assumption described, the architecture is intentionally not very highly resilient to disruption. Below, the process described for the components in the event of disruption. + +##### Coordinator + +For its nature of being a single point of failure, the `Coordinator` results to be the most impactful part of the architecture in case of disruption. + +In the event, Testcoordinator disappear the agent instances will not be able to complete the test as they will remain in a stuck status waiting for the connection to return reliable, after the gRPC connection's defined timeout expires, each agent instance will abort the ongoing test run and will shut down. + +In the event, a graceful shutdown (SIGTERM signal received) is initiated by Testcoordinator then it is expected to execute the flow described in test abortion section. + +##### Agent + +Eventually, Agent instances could be more resilient. + +If an instance at some point disappear, the system should tolerate this failure in some percentage requested by the user or by the default value. It means that a rendezvous point on the Sync API will have a timeout and if all the instances haven't reached the same point in a specified time frame then the Testcoordinator will remove those instances from the number of required instances for the sync consensus. + +However, in a first experimental version, is not expected to implement and support this described scenario. The flow will let the instances stuck on the rendezvous point that at some point will make the request timing out and the Agent instance will start the abortion flow. + +### An end-to-end example + +It shows the end-to-end flow for the simplest use-case in a distributed test-run. +The first step requires to startup the `cordinator`: +```sh +k6 coordinator script.js +``` + +that starts the gRPC server for synchronization API: +```sh +k6 coordinator --instance-count 2 script.js +INFO[0000] Starting gRPC server on localhost:6566 +``` + +At this point, the coordinator listens for a number of connected agent instances equals to the value defined. For this reason, at this point, we need to startup the agent instances: +``` +k6 agent localhost:6566 +``` +and again in a different terminal session: +``` +k6 agent localhost:6566 +``` + +At this point the coordinator reached the expected status and it will notify the agent instances to start the test-run. When the agents will have been finished the coordinator will close the gathering process for metrics and it will publish the summary. +``` +k6 coordinator --instance-count 2 script.js +INFO[0000] Starting gRPC server on localhost:6566 +INFO[0005] Instance 1 of 2 connected! +INFO[0017] Instance 2 of 2 connected! +INFO[0018] All instances ready! +INFO[0018] Test 1 (script.js) started... +INFO[0049] Test 1 (script.js) ended! +INFO[0049] Instances finished with the test suite +INFO[0049] Instance 1 disconnected +INFO[0049] Instance 2 disconnected +INFO[0049] All done! + + data_received..................: 3.1 MB 99 kB/s + data_sent......................: 19 kB 597 B/s + http_req_blocked...............: avg=18.43ms min=4.09µs med=8.67µs max=290ms p(90)=12.4µs p(95)=249.54ms + http_req_connecting............: avg=8.33ms min=0s med=0s max=130ms p(90)=0s p(95)=114.66ms + http_req_duration..............: avg=129.33ms min=110ms med=123.2ms max=260ms p(90)=135.66ms p(95)=227.74ms + { expected_response:true }...: avg=129.33ms min=110ms med=123.2ms max=260ms p(90)=135.66ms p(95)=227.74ms + http_req_failed................: 0.00% ✓ 0 ✗ 270 + http_req_receiving.............: avg=7.17ms min=63µs med=140.57µs max=130ms p(90)=209.36µs p(95)=111.86ms + http_req_sending...............: avg=36.95µs min=12µs med=36.13µs max=84µs p(90)=50.1µs p(95)=66.54µs + http_req_tls_handshaking.......: avg=8.61ms min=0s med=0s max=140ms p(90)=0s p(95)=117.16ms + http_req_waiting...............: avg=122.13ms min=110ms med=122.44ms max=150ms p(90)=129.48ms p(95)=133.08ms + http_reqs......................: 270 8.516465/s + iteration_duration.............: avg=1.17s min=1.1s med=1.15s max=1.5s p(90)=1.25s p(95)=1.35s + iterations.....................: 267 8.421837/s + vus............................: 5 min=5 max=5 + vus_max........................: 5 min=5 max=5 +``` + ### Alternative Solutions One proposal in the original posts of https://github.com/grafana/k6/issues/140 was to build the distributed execution on top of something like [etcd](https://github.com/etcd-io/etcd) (or Consul, Redis, etc.) and a central metrics storage like InfluxDB (or, more likely nowadays, Prometheus/Mimir :sweat_smile:) @@ -198,6 +318,8 @@ Because `setup()` will no longer be special and because we'll have a generic mec And maybe even more importantly, because k6 instances will now basically self-synchronize, test suites (https://github.com/grafana/k6/issues/1342) become much, much easier to implement :tada: See below for details, but we basically no longer need to have a super-smart coordinator node that knows the details for how tests are a part of the test suite, only k6 itself needs to know that and use the same `SignalAndWait()` API to synchronize even multiple tests simultaneously :tada: +A potential improvement for getting a simplified user experience, when the API will be defined as stable, could be to hide the coordinator interface just behind the classic `k6 run` command. When a dedicated flag like `k6 run --distributed` is passed then the command will act as a coordinator. In this way, we are not forcing to the users the requirement to know what a _coordinator_ is. Only users with advanced cases will may require to know it and its dedicated command. + # Test suites https://github.com/grafana/k6/issues/1342 is far behind https://github.com/grafana/k6/issues/140 when it comes to upvotes, but it is still one of our most requested features after it. @@ -242,7 +364,7 @@ First, how can we run this at all? One way is to just use the unique test names And we can also reuse the deconstructed `execution.Controller` API to model these dependencies between tests in the suite. In the above example, before test `B` and test `C` can start initializing their VUs, they can just call `Controller.Wait("test-A/test-finished")` (or whatever the actual convention is). Similarly, before test `D` can start, it can call `Controller.Wait("test-B/test-finished"); Controller.Wait("test-C/test-finished")`. And all of this would work in both local and distributed execution! :tada: -Similarly, if we split `GetOrCreateData(id, callback)` into separate `Once(callback)`, `SetData(id)` and `GetData(id)` functions, tests down the chain would be able to even reference data from other tests in the suite before them. For example, if we save the result of `handleSummary()` under a `test-X/summary-result` identifier, then, if we want to, other tests would be able to retrieve that data with a simple JS API directly in the test script! And again, all of this would work even in distributed execution, since it relies on the same primitives it does! :tada: +Similarly, if we split `GetOrCreateData(id, callback)` into separate `Once(id, callback)`, `SetData(id)` and `GetData(id)` functions, tests down the chain would be able to even reference data from other tests in the suite before them. For example, if we save the result of `handleSummary()` under a `test-X/summary-result` identifier, then, if we want to, other tests would be able to retrieve that data with a simple JS API directly in the test script! And again, all of this would work even in distributed execution, since it relies on the same primitives it does! :tada: ### Bundling Multiple Tests Together @@ -254,4 +376,4 @@ Ideally, we also want to be able to configure custom script entry points and cus Additionally, we probably want to support custom thresholds and end-of-test summaries (or disabling them!) both for individual tests in the suite, as well as for the whole test suite together. -Finally, how and if all of these things are packaged in a single (or more than one) .tar (or other) archive bundle is still an open question... :sweat_smile: There are other potential complications here as well, e.g. https://github.com/grafana/k6/issues/2974. \ No newline at end of file +Finally, how and if all of these things are packaged in a single (or more than one) .tar (or other) archive bundle is still an open question... :sweat_smile: There are other potential complications here as well, e.g. https://github.com/grafana/k6/issues/2974. From 48606fea9235ed0cb401bf0f007629de61b47b70 Mon Sep 17 00:00:00 2001 From: codebien <2103732+codebien@users.noreply.github.com> Date: Mon, 19 Feb 2024 12:44:24 +0100 Subject: [PATCH 2/2] Drop point regarding Cloud output and error handling --- docs/design/020-distributed-execution-and-test-suites.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/design/020-distributed-execution-and-test-suites.md b/docs/design/020-distributed-execution-and-test-suites.md index b7929790be5..58d370be368 100644 --- a/docs/design/020-distributed-execution-and-test-suites.md +++ b/docs/design/020-distributed-execution-and-test-suites.md @@ -55,7 +55,6 @@ Let's assume we have somehow fully partitioned a test run into multiple executio 7. **Done status**: Even if the test hasn't finished prematurely, _something_ needs to detect that all instances are done with their part of the test run. Because of executors like [`shared-iterations`](https://k6.io/docs/using-k6/scenarios/executors/shared-iterations/) and [`per-vu-iterations`](https://k6.io/docs/using-k6/scenarios/executors/per-vu-iterations/) and because iteration durations vary, only the maximum test duration is predictable and bounded, but the test might finish a lot sooner than that max possible duration and good UX would be to not force the user to wait needlessly. 8. **Run teardown once**: Regardless of whether the test has finished nominally or prematurely, _something_ needs to detect that it _has_ finished and must run `teardown()` on only one of the available instances, even if there were errors during the test. This is important because `setup()` might have potentially allocated costly resources. That said, any errors during `teardown()` execution must also be handled nicely. 9. **End-of-test and handleSummary**: After `teardown()` has been executed, we _somehow_ need to produce the [end-of-test summary](https://k6.io/docs/results-output/end-of-test/) by executing the [`handleSummary()` function](https://k6.io/docs/results-output/end-of-test/custom-summary/) on a k6 instance _somewhere_. For the best UX, the differences between local and distributed k6 runs should be as minimal as possible, so the user should be able to see the end-of-test summary in their terminal or CI system, regardless of whether the k6 test was local or distributed. -10. **Cloud metrics output**: We need to support `k6 run -o cloud script.js` case, and unfortunately, it requires a [creation phase](https://github.com/grafana/k6/blob/b5a6febd56385326ea849bde25ba09ed6324c046/output/cloud/output.go#L184-L188) for the test for registering the test on the k6 Cloud platform. It means that in case of distributed test, we may end with registering the same test multiple times. Then, moving out from the Cloud metrics output the test creation phase sounds more or less a pre-requiste for being able to deliver and use a distributed execution integrated with k6 Cloud. In concrete, it means to address [#3282](https://github.com/grafana/k6/issues/3282). If we need to narrow down the scope then we may decide, for a fist experimental phase, to not support this use-case. It would error if the distributed execution runs with the Cloud metrics output (`-o cloud`) set. So, yeah, while execution segments handle most of the heavy lifting during the test execution, there are plenty of other peripheral things that need to be handled separately in order to have fully-featured distributed k6 execution... :sweat_smile: @@ -182,10 +181,6 @@ As these days, most of the observability world is using OpenTelemetry protocol f In any case, when performance matters or a high cardinality of metrics is generated then we should encourage people to use a proper storage systems for metrics, via outputs. As k6 is not a metrics database, it is expected to be good enough at store a non-intensive metrics generation but in case of high volume then it may be better if we use the right tool for the job. Otherwise, as explained below, the coordinator could be flooded and become unstable, potentially we may mitigate the issue in case of spikes having classic back pressure and retry mechanisms. -#### Error handling - -TODO: define clearly how error handling is expected to work, as it is a tricky point so we have to explore it deeply in advance. - #### Test abortion A test can be aborted in several ways, as described in [this summary issue](https://github.com/grafana/k6/issues/2804). For example, in the event the running script invokes the `test.abort()` function, below the expected flow: