From 75f4c8ec19f31f26aaf33cae19529f2746004605 Mon Sep 17 00:00:00 2001 From: Dashamir Hoxha Date: Fri, 29 Nov 2019 14:34:09 +0100 Subject: [PATCH 01/11] Manage experiments --- src/Documentation/sidebar.json | 23 +++++++++++++++++++ .../docs/user-guide/experiments/branches.md | 5 ++++ static/docs/user-guide/experiments/dirs.md | 5 ++++ static/docs/user-guide/experiments/index.md | 6 +++++ static/docs/user-guide/experiments/mixed.md | 5 ++++ static/docs/user-guide/experiments/tags.md | 5 ++++ 6 files changed, 49 insertions(+) create mode 100644 static/docs/user-guide/experiments/branches.md create mode 100644 static/docs/user-guide/experiments/dirs.md create mode 100644 static/docs/user-guide/experiments/index.md create mode 100644 static/docs/user-guide/experiments/mixed.md create mode 100644 static/docs/user-guide/experiments/tags.md diff --git a/src/Documentation/sidebar.json b/src/Documentation/sidebar.json index 512a6018ef..d21fde9420 100644 --- a/src/Documentation/sidebar.json +++ b/src/Documentation/sidebar.json @@ -120,6 +120,29 @@ "label": "Managing External Data", "slug": "managing-external-data" }, + { + "label": "Manage Experiments", + "slug": "experiments", + "source": "experiments/index.md", + "children": [ + { + "label": "Tags", + "slug": "tags" + }, + { + "label": "Branches", + "slug": "branches" + }, + { + "label": "Directories", + "slug": "dirs" + }, + { + "label": "Mixed", + "slug": "mixed" + } + ] + }, { "label": "Contributing", "slug": "contributing", diff --git a/static/docs/user-guide/experiments/branches.md b/static/docs/user-guide/experiments/branches.md new file mode 100644 index 0000000000..2402641f32 --- /dev/null +++ b/static/docs/user-guide/experiments/branches.md @@ -0,0 +1,5 @@ +# Manage Experiments By Branches + +

+ +

diff --git a/static/docs/user-guide/experiments/dirs.md b/static/docs/user-guide/experiments/dirs.md new file mode 100644 index 0000000000..ad95f26e8a --- /dev/null +++ b/static/docs/user-guide/experiments/dirs.md @@ -0,0 +1,5 @@ +# Manage Experiments By Directories + +

+ +

diff --git a/static/docs/user-guide/experiments/index.md b/static/docs/user-guide/experiments/index.md new file mode 100644 index 0000000000..0d81aa6568 --- /dev/null +++ b/static/docs/user-guide/experiments/index.md @@ -0,0 +1,6 @@ +# Manage Experiments + +- [Manage Experiments by Tags](/doc/user-guide/experiments/tags) +- [Manage Experiments by Branches](/doc/user-guide/experiments/branches) +- [Manage Experiments by Directories](/doc/user-guide/experiments/dirs) +- [Manage Experiments by Tags and Branches and Directories](/doc/user-guide/experiments/mixed) diff --git a/static/docs/user-guide/experiments/mixed.md b/static/docs/user-guide/experiments/mixed.md new file mode 100644 index 0000000000..c8ec962928 --- /dev/null +++ b/static/docs/user-guide/experiments/mixed.md @@ -0,0 +1,5 @@ +# Manage Experiments by Tags, Branches and Dirs + +

+ +

diff --git a/static/docs/user-guide/experiments/tags.md b/static/docs/user-guide/experiments/tags.md new file mode 100644 index 0000000000..6ead626b31 --- /dev/null +++ b/static/docs/user-guide/experiments/tags.md @@ -0,0 +1,5 @@ +# Manage Experiments by Tags + +

+ +

From 69cf0b44cc597f09140bd805f74f05e770bbce4d Mon Sep 17 00:00:00 2001 From: Dashamir Hoxha Date: Fri, 29 Nov 2019 14:43:35 +0100 Subject: [PATCH 02/11] Add images --- .../img/user-guide/experiments/branches.png | Bin 0 -> 7499 bytes .../img/user-guide/experiments/branches.uxf | 241 ++++++++++++++++++ static/img/user-guide/experiments/dirs.png | Bin 0 -> 6630 bytes static/img/user-guide/experiments/dirs.uxf | 24 ++ static/img/user-guide/experiments/mixed.png | Bin 0 -> 17124 bytes static/img/user-guide/experiments/mixed.uxf | 104 ++++++++ static/img/user-guide/experiments/tags.png | Bin 0 -> 3015 bytes static/img/user-guide/experiments/tags.uxf | 152 +++++++++++ 8 files changed, 521 insertions(+) create mode 100644 static/img/user-guide/experiments/branches.png create mode 100644 static/img/user-guide/experiments/branches.uxf create mode 100644 static/img/user-guide/experiments/dirs.png create mode 100644 static/img/user-guide/experiments/dirs.uxf create mode 100644 static/img/user-guide/experiments/mixed.png create mode 100644 static/img/user-guide/experiments/mixed.uxf create mode 100644 static/img/user-guide/experiments/tags.png create mode 100644 static/img/user-guide/experiments/tags.uxf diff --git a/static/img/user-guide/experiments/branches.png b/static/img/user-guide/experiments/branches.png new file mode 100644 index 0000000000000000000000000000000000000000..0368ab6896ea33b52ff329ca92458aa51aa6dc15 GIT binary patch literal 7499 zcmbt(XH=8hwsvTW0R%UpN)>@9(tB?~2t5dhw16T=@4cgeC`~|7L`vuz1Qeu+fPnPg zn}A5~U25Q4xX(S`8ROpj>t+lFBQxt=Yp&UzIiL5ruC^)#lo1MnKq!!E2t5ddpa}fY zlM(_+Brzr%0%86OiBQz{F<+kycA#FwUv@cH(o>6xGRp#jODE2R}kx^QEv0~KMdTDh<^Vt=~ zK73#sAfJywBB`af52(>AUwSyX7hVq`kY{59onL$ewR}5Qv%_ zRtW+@utOjjR1mfhI5C8j4N3qZ1k#ZIlW2T=JU`!X%;D(L(3pd_p@BjF)sH7xQV4d1 zot>iqZJX%H)R95`882sY;uQm@y(}%Q*ok!#?c4;f@JV?Zb|^tS-F$kZ;K=GEALT94 z>+2znD0SP`jkFh37zpx(w#KrL?k>}-cOxh9ljeCdNkSP^Y)AvcB9AJ=)_cf;1vg0^ z9-grpeX|t?B7md95!g{TISk&}hNL9ExB01Mp|2t-$+o5@v+*xV zmecKfI)Mj+f7^gl+5T!msjU}Qu*BWi>`bz0yuZ3R{T#R>4j!Y_Uj3R!H3qsov9aMe zQvhtEAOOt}c;EL##^vWVYf<1mQm(hs4)oC@4YAP6m%LE@eHB~@+xUZm#{0@J!Vn9X zs-?wT`^zt{VAQ-1s=X8s?SNiXns-o3yF*D`94I3blfP;iK?D$ING?cbzb7a=p`;q9F!A1rsFgY5*kOnU0@ z@Nh=Pjei+#aM7(whJ`ZF6@TpcU!wcqgGTcQc1;(W9Q+(EguGN>W@a858d6roVGLKv zyPcNmD8=_KhalqqdYSQ(P%yyv@jT<3)m3={`zh0+!nX@oAb*8vM=WQ(HDiRmKP?SO zZj(n>Osu~>rH1@kJKkU2D{h%}tul33=h4m`n1S#6q2p~5-bK6h&iK&ZYamj1PeD!& zqoFA)FE7D3gP4RVc==o2OBh(6;-vhWxMJkTMvazMUDmAPIK?=F*SY>PIB_qH?V(7Z z;(7mwgYO*=V`Bq@$E9PnwVDXty{x@K##FipkUSnLjB;A47zctIN-)AD#Nn20`K5}d z;Nj<6GXqq-2=gj%j7@6OnK}=Zs?8E4;b4PaqpORRZ!fZ5afvh)Kpr zfSWsF>OR*%D^uA_7N>r;=V*j0xokYxBZD34*|4}fG&mS_bcu}Hz*fs~lThf|(zU|a znSp#o*b2hYk@qC=1$Nj*yq;kX+fUV)Yw#34NF^XX48$jhBcA+}+Aj03L- zgBE}OEHd6maYgn8ONUbo(GrTZLbjh3fg3$OK5l5qQHf;HFtHHNzqN<0w~Y0;f`XTK zyx}i(PJx$w{``68+3$$SCirr(>fSx;;4La=tkS0xxxjaU(;*wZI!X9?<*)3}rtnN9 zN&G&kD(1EV?1ffvnlet6{HlMJ23Bcid9yg--$!okiCa_1LR)P9=qvO1zR=Re8kh-_o`3gs41fxwPUBA3CW{pK_ceLOD`0m-jb< z*0>}x>sXerWax4inN|sTt_=F_F3cl>Bdk(yRWAzn?2T{)D!^9ifBv*UBCph7ip;8W zET+i2ajNNr2po&UM0xa-hH5Me1$uO3WMpb;YJ6P%eC1ITpC9q_3v0I}aCcen3irq~ z(n2g6s4x##mzR?y-51uzN?kPxsK|f?Z}afnyLV6i?C?=J1k$GfgKB=6Z{sQc+CU3o zQ^zWyswOu&J3C8CggRf6QAs0kFHeKcGgKl=N=gKj`4mjqpcs8LS|_(Tf|Avm43bWo zp)120bZDur-kzW%*G3CbxP`!3d^O+v@St?f%ir0V--Jll%*4dRVCZ%-ue`@tl$}Gl zbV2@Y+smVYdc-LB+m#i&(y@S*!CU2!biTBnMUUapbzdzl8mEJ1g>>`zyE@~8dzF*s zKRu)#)p?R+=-%A?x_cEb^rOn1+XSM^{67H!`V|$X;c%_Ar=+x0Lqp@yISDBhhZJx2 ztYVkDuATX&21w>8n|7>{AUOpEuh|z74vx&4nv_ClC7+&2`QEf0Iz5}T zE=`*3c)|wtJ!qms|L`0!6H6{ED2Oa1c=D_pHRqeE?)d)_xcBFlkoD=p6}UdT=Eai)c1b-EBtdjh$mR38sCko#U=Un=hWPzgb;BU0_UMG zA@I?DTJCb7y%586aGg|uXkDFC(+f$xWy)Y9-f8VQa0ME5g?2@C<#Nb3>LwID>0NX6 z7ta$vm{qd{l;x%#7rlBKakVhsd9EB>fz{)#=31^y0@rijMl2R#>F9`j zl2T~GDnfj#wu2}`SN`_x+x`9ha`?S_U|$BXuasg=Y1+ZmNz&(rGyJ^X)!`BOo56>~ z*Fh05<)g#jIoO=;p@ke~{s)wv9S<$PO77}S@az+219%b?P+vand& zL*7^U?b``>j|%jcvqLfB0s=16fd^V?a+9;$A4MVEYUdi)FjT=N#+TJd z{#pxrh|JMI+Swj{GZ?Mj_^p)!Z{y;nd>&YdtNR8*-tm(W2!C)331Jx-O<&=!?g4Z!eVz(N83(AKC2H9#-l;xs9lm{o0#7T%y-O%5MwgJfn;6S>)m6?dtA6 z1F1gB2oYadU8Q7}`p$=k;>B|yB@<-H@7_Bd@cF87R@N+dZ93W=b23x*JN0&dJfY)6 z;4XP>{6OY5U(^#2mTgUubhjwRvZoQb*#-s%V#(_Tk%&QM9d#Y%SzlLB=NxmjQ-qSc zKpWyf5OTfRd$ZPMhOk-(it6JY{Pe8vSeBR-;wVhmJ8oIl? zYy1y_7Nw`#y1R4I>Y2TNp!D?mM@Q8<#tcZjw|>^Yf6vs|NCTX1l~SJfW5lRJevwyjQ_RZZN` z<&h5j!8AW*y-QV9mHc5nrN>reUU_-qx(VX5T=f;)QgPkVwRx`U=UH?S@T^C&?e$wd zT+PCdnxl4h+;FiJ%co)ML&R9mR2uDIyh`j%nFBRySFDm0e@uW3ebL8{TSNEJkHy(Z zrLwx(+MWkoy`j1~kcp6C8b$~Aaa`$BDIXq2D=p{8=grN}&*Z%fzEJmOL}tTFqs*<)%Pbb?$%zMFeFWur=F%>d6O>qs z3x62h(A?bI*{S(kGzRNN6|6`b_MeIa7yx>#b;#8I`t?vsy17z`{lXC_2|53)*G8Ey zfwLp}qXUy#glzk>PB!aU?%P6uGj_~jy;jpe4X?3yudvNR^ zT^gL682GzWTw2=y?DVMAy6ekqj5M5<3<_ll*iw|D&k}x;4-J%|7pnJfL_VvWFtegw zE7qsUsH0ws+iMwH9?0E2Kk*Nf{cFi4tF}@FmsMI?`sq_#Nso1DU!*peB-(uG?3K5i z=hqsac}|=yEN!RuTo-dlFCHl}>yUe^LA@MjB+nt*BneGWpYy#5GC;MZ$?u)?4Y%NZ zDQYz{GiKg^*=X_6%kH)}WWxm6p|k10-!(NgWo56=D54*%udK9pbp>Vif z2vKhpg6htv{5Q)c*710}1I079?O#n%QRGrxO!V{?_UUAx)?^67qyz$v=HHAFVi%VB zGVf+yqrSOJ1Im+Op|L1vh70Z&myl3({a&9~rke;Qx;#pVr9{1TajNj&nQPOL*(NrD z^j*}yH*a*_i7W#R94Lq*K+%g~mR|Y+t!NtzEh;Jk@li0evh6dIf_~XyE1BM@r z>S+};(*kyoetyUCoEk0@JbmoYAug@EuD`xtDTh2f2Dqyu-@BL=oXQ)bONFt!PEY?9 zlx@%i3&NDe;g@ew+@c0nB|_ze&bGu3NUQcT=qko^$1F1?b-!8;NGc3vqr!~C`T2X= z+Z$TezbMMLH#9V4WMo_yclszwMUbI;{SBX)x|UXxgS4O9@3k>7M?LOHJi-T+U;3d8 zv$LLUj_LIGqsfTT|(RAXSWj=nn!z-p9m3p7pH zM3=AjDiTkL`3Xr#RQhcHx^*vS$u_*$Dg-{*4@!0z37K{Gd*9Vz-ttQgJ3cao1!a|$ zm48~}Vk!)yG{Bt{9e}qNTPF$IaB02MH#gU=TOVa~xq`~85w#!A%FR7rls~s5V-s2) zh+UtkC?)y%SB9=XEDVN$VSzjE-@B(AaSBGJ$7J7F5$tqJ#7R@a9wnW$3RXl!L}YY) zovp1vvl}KvV1%o3Ssg9}2k{wwgs}21cb3_Y0Ki-9Od1#%XdDjV#wwXJNxBhJYpm#* zm?Sqn{5vo2Tc?KemWO| zT~+RAj=FU0D+q*g$S;#>ufjGfDX%s9!1aB<>x_)Fe5OxC7%9-Wi;=x$P5hk#Y$vGQLLxweEB@_HA&vYebt3B-98#r4XjYc0H)<5ZcBzSC0f>jFgt(iJKJynUMoeAT`nbig) zMOVQ4Xh)-!z$9Pw`}c(Nfb>i+PT7Ir`vm}rO=KVdJpkPfhtDxGQzpK81%ttU!jPsC zw_$`CFTFhw79xz^Yoj3Ps185ql46wzWVXLkCvfX{RZjvlht{#}Qlj7?pd4wZNO=zB z8(auoF{jF~7c=v3&^g{J_Vzp`kZbVj@n_o6u4T{dz>K)Gxd|AigDL;w=}jAZHbT>< zostNgz*OLYnV<95qlIL!$+h(qTRS_u@bK{bf&ytNsZmEoa08W{US-Ppg#|B9PjLCr zp2bWLVT5~~5>)X;1CGtho$5YTDbBRgg`Jc0F_`N5E8a3FBRskbe)0{2E`5K0zf%dz z0jBQv+BSA}D>+)pdm_SjOS<-P>2-W>I%cb!b!o%ROkP>?uO{ShG zTpf#T21tpbq8Kyi3!qGT9gore2uGos@>M}{X)oY}rOjHdB$X1 zh-h2Td))zFBk?S)psy{I!Qm~v1=TAHcHe@9v{9N8Ev`;_(Tk-d_HU;zNWw*n z4>+r9bQ^^q16p8D9I6v6`^)f|<>>3zJZ42SJNo_kRi@qT?do3qLl*`*E9%l6=HOU3 z*0uU-IcjKcuG$j0qt>7A>9JN~TsY&O2atw-&%3GU%qxITqIB-M5ym@d-s@M98K85Dd!ljHP3*l9RPH zy&eHMPS#Xru1R24^= zSzTiP-#CVi(NA;UbtXqgiza!6rjmMENJ0E=+Bx?hl7U$FMt)>x2u=d}cg_AVNB^~7 zNy0tt_s=@eLAMQJnS4#nqk93R+^)5CL)Zcy({Of{((lT8K`KN)Na%i*ZD(n@5PjWb zKKaGPGQp~K6>GL!HfSS5bTiRqmMiGRp5P*KgWa8+Q~ah*TLpC$16cMc(j|_emBc-Q zB6HjL5H<7dTT$Q%Q+S;G_VqtRLfOpBBzR>|WR9=XFEi>~>@G|I6FT71zqk0%C568K z1t*U(Wzc&QFC;W*yFN~`^ST&4eTVtF-1(`}AAW=IJ4Lvqj_!Xl>^IO=rS6e141aB< z+CO*Nj<6h!yA{*Wd!?lBNx)`JT3>u>s^k3pxiWXhIs+ z3ZOwk4jTp5T44yw)>pR@`}HOOt)k4M%caM-Ft3H9wkhyr%$|`yyJmLv61e8`OpsXd zo%@zpF6fYLy;*;_wGMfDOkTJp_mC+ow!sC&LdZKV33BPb7mh*146HpT(D$0K9BmUD zxg1>KH2Kt|f-vu9gTR_q@@k`9O8poG{$M8j46?%DOz#EC59&cx^ZfC#&bN2sZ|v=r zGUbI3>^Cz(0PlJ?ydABQ3u7gPx$K61TlK1z#j%ni!r{-d66f3lK>|i(4z30w&j#(p z+7%;aXAtK}@RIMfy^!}BJK(ve5%2%a#!rX-CmT<5iBctTl#Wq>6e%~N>Z0cRy33j@ zX2+05_Lgi}_+k^g-PemvzMHb@U1yo%ojgfB-OD{_1&$~2zA_j>$7t~?3?3yg5-}LG zUl5`E{5gD%!Td=G>P4o2K({GtFn*O9ygLcK46VhlJ0RE>x0HyY^q4S@y7yh@+uc20 zN;r;ZfkhZeceiVp_l;x}0_uveaI&=3#??tV=P3_kpQYa0$F!(cd7wvTSXEGX+3rz4ASTadFqi1FC7?tlRdVl=SzlkJnU_iJ+9`}4A;&6}{RBVK zbTW%YHQS@Q_CquOy?ow1P=MlJyMr*m1{B3^0T5)pxC2TBCk1*k#*34wFm2@gBH!Q$ zOQ8!TU(}gY*h!A4mMFpqqZ&+E+sIL*0#NtBo5Be8Z}86jxK0o+=tfgkBpHfyOB4t` z1Hm2eA9S%02zdE$@s>~lf+B@bDZmIJ#1~S6|FcNpjP>LD@$`X#R0McQ3PGx9BZ`zP G!u}ruDnDlc literal 0 HcmV?d00001 diff --git a/static/img/user-guide/experiments/branches.uxf b/static/img/user-guide/experiments/branches.uxf new file mode 100644 index 0000000000..79bfeeeb37 --- /dev/null +++ b/static/img/user-guide/experiments/branches.uxf @@ -0,0 +1,241 @@ + + + 10 + + Relation + + 40 + 150 + 110 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;90.0;10.0 + + + UMLSpecialState + + 40 + 150 + 20 + 20 + + type=initial + + + + Relation + + 120 + 150 + 110 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;90.0;10.0 + + + Relation + + 120 + 80 + 80 + 100 + + lt=-() +bg=black +lw=1.5 + 10.0;80.0;60.0;10.0 + + + Relation + + 120 + 150 + 70 + 100 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;50.0;80.0 + + + Relation + + 200 + 150 + 100 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;80.0;10.0 + + + Relation + + 270 + 150 + 100 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;80.0;10.0 + + + Relation + + 170 + 80 + 100 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;80.0;10.0 + + + Relation + + 240 + 80 + 100 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;80.0;10.0 + + + Relation + + 160 + 220 + 100 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;80.0;10.0 + + + UMLState + + 360 + 40 + 50 + 30 + + *B1* +bg=yellow + + + + Relation + + 310 + 50 + 70 + 60 + + lt=.. +lw=1.5 + 10.0;40.0;50.0;10.0 + + + UMLState + + 380 + 110 + 50 + 30 + + *B2* +bg=yellow + + + + Relation + + 340 + 120 + 60 + 60 + + lt=.. +lw=1.5 + 10.0;40.0;40.0;10.0 + + + Relation + + 230 + 220 + 70 + 60 + + lt=.. +lw=1.5 + 10.0;10.0;50.0;40.0 + + + UMLState + + 280 + 250 + 50 + 30 + + *B4* +bg=yellow + + + + Relation + + 270 + 150 + 70 + 80 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;50.0;60.0 + + + Relation + + 310 + 200 + 70 + 60 + + lt=.. +lw=1.5 + 10.0;10.0;50.0;40.0 + + + UMLState + + 360 + 230 + 50 + 30 + + *B3* +bg=yellow + + + diff --git a/static/img/user-guide/experiments/dirs.png b/static/img/user-guide/experiments/dirs.png new file mode 100644 index 0000000000000000000000000000000000000000..83e5853bc4e54c71b6be19365ba8b0760cc7e982 GIT binary patch literal 6630 zcmdT}c|6qH-ygdWx3Xs`+9V|<6-JCKEhwbyvP9NUV;hCgf;Ng=rb)J#A=zatg-9rS zGh@jXlV!#}w%?f$rdPB=AVO$ubyWi|^Wiv)Tj!dEL-XRTJR{ZSth*2DKZLt`zyu%va|bzM;pww!;@6|64Ldg8J7gLT|J zB~841R_=8dr;fzdFGLV=#07NDR0q))$r;n`xYRMQ(BeO9hdnZozA2ntt?brk0p_i) zHoJn$x5A`&xrI)k+}`VP1j9nk?h=uURO`Nv;^heCO4K|2-|1q%OHwtTd+@@322N|2 ztp=rKrRC8k?0jyrnX>)-YAhmLd%_BW^%&Tg{%*4|+#uZZ7(2^Y_eCr8<_ml$*3Y=T z%4`Ww=SXDr%RhgfsBtd%EECjpPx@8KG__O)@xA;lg_2KR*dFrgkGb@Cuh~=V?r0|H z7SK*K+8|=65kekJM*0)cd7q6O`NkNmtk5PUce*fgn}upll(O9n&T(1T&4`|Fv%XhM zu94pNW~>~Y*WuH6TA_HnPH|~*&eeUPLcropcgSWj#Kf7aeKk?i!^6XQo<E#6l1c zH2+e4{0pr<#lGe=BbCLO9-#wvr=n%Mv2!DHU+?aFNJRRPtCgqm2d;d57b@oSZEm#0 zNWj8>wr}Rio#@W%OUkJiVjI|On87Y4)7Z1KgrQeSZ+TRde5ahQUtf~R!#~&9jfJ}4 z^D!6Nfz5j=i(aI!#|`y9 z`nbAg6fSo7j;4OKKuk8PEX|KM>Z%k|hSEq2WStA^5XUquX%0Cioz|^CYEqr&*otv> zatg~(A-&sFJ^A6uzBDO(;Z;X>+m`eT)w0`RsV=cf-ej=-SbUOQc217rdV3fwNSM^? z6Ub$|eS$$KVA-STA)`61<|na+qs~KOlXgj%l7%T;^<)qm&r9O;4e~Z zwcaB6!@i;1r7$`Fd9O+HG;HzQNYV5#E*U!m9R1MikCrD}Ep)MHk(!8zh+axY$>LMt zn(ZeaZ*xto*5l^lO2IWT@64-s*IG3nbtWWhBjrX|X}NKpKVQJPBo%D3E>g@QL!@S= z3J(6<`vTa+o~dAa#2bk%=#wsYZ)do+Wp2G{M*9A`9y{9~JD8Tk@q?RoihI_@DGFR~ zIO8EiwesK-&rx`XcQS=J0lM+(wQJ_1K@k)#&z^9Fsfme{JY2~lg*Xn*rz?o3X%gOv zFhYErr4Tdfkt$2zHL%uXqe6Z!SzVPX917hcnVH#wwmAMQs(CCYCr8FKU+;8nQ>s1= zhYJl2?IEV3-njSG+~JdUDH=YsfV|0zL`=m{AQ+b-BBrRrt0#QZPMxr@MeT11_n>D> zEOdT9d}Zo;I|+2WoF(o*)2;nzhuP?W?`c)>`x`(gwPZ*%)OgMaDor-&4k%|Mbg$(Gr#8zgHZ%lHGoQ?#FAu~LRMfwI0J9v0- z^0ORDi~2h@A3oZFh6<$VZk(GME^_M*q!=O4m=xlmN$^AKgMz$!es4$gzTY5!ePLmN zl&7s{cBZxYvU~DxkajlfQ<-jd(+{_w+^urUfVKH*&Qm{W=3`T3f}6k~0v@j2vv&INrN@v!}643=JI^eM|}onn(@;qNbOS4HvO zs2AY$AbhvH_Wbe-raRNX@YkB12z zF!vcu9L3JXBNSQCB0PN`6*|+giy%of z&!>J8QE4p7dRAl2F31J66M}QVqk866QUl_~$P8{^(@cZhM5>EgGd zy4pr^Da`}941+B^N04t1bdA$M@E)ckN9LOGdqv$Z`Tl`UB_^l^Z8oJ|&r|j4Z7bb6 zq?FfM;3w)=jm=IB3ysZQq)&4_d1BqNF^Tts(H#&Z4M3{Wy^a|yrJ|M8h1;|6Or;19 z9y@u|A-E^>_E06063>< z&r=Dp8Enp<*=gH6H)aW^o+N7j%wV*Jg8nbvPlaw&5pDct$FbV~RvvBzPLgAPHbkjk zFOWL*Db#7<0BUG8aEN^$;mwv4Juy$6TPj;yk_Ip1vLFHtU=E%~`$3pAS5#J3*4%9^ zIwJW?7gpV>NYuPI%~jN`jd|8K=?%HoX{kRw$hfv=ykRbT+csLzOM7J&>F1|8ZOGJK z+oqdQyM*N! zOSf`AqnIB87o;zNoYTtK#cfSRLbe)q${2#TqR;)l)tvK&Vi?^QvtJw;+(PM-{=G@M z%ESyMMcFyWSZFAt9;7+SVm;)aKl}DR%ue^ZaeCDr4H$kkuuyIplic*g#6&C>o0{qX z0zZ^TT&L4u{g)RT*8HX_jZ;RtFV4?`unh_dg27;DH2O-PezH#DXro%}k4Iu}>*K`f z54AmIVPPTCo@IN|=QQUkl$N2V_jN$Oa$;(#@z_^cl1GE2 z?t;seDSpiG%hnoE%=O8pIhPInE*TjaRfMyyE{BEZ8c3DxBeRjj){J|u#P8qJ-+W~M zWjOH6S;PaZ{GFpmk3PJ7;g;5q;TK#~!_N@=ZaE78Yw#!;#zVK)f@Ea33Gs^&!6vs63fT)WAHwU8jjaJa`0eN zM+X#CDGG%mrJn>!%%xV0*Rzbn+#9-r~t;}kM+UkD@qirHX zExsaUb;rtu@$FMEr@+g(=AH2Y z0Ra=Dt6<*4IbyyAP0hPVx&9ea&lQ_la^Il*>V$MdNw9VCx-o9#-s;0y7c1^eJmp`( z;fKH?0WE10B_tVN;NdY)M|gdyzWwzjHa0c^8Drk?cyIr%(tsK*!>LG8zk!+U~Zxkz#(Lj16_#1ecKvRru8q*kitDsy|L$5A>!qiK4(Se7gIu1Ln^ zvisNO+Hb#nz|B4d`8&k!6F@B^dwwx(MXwW#z8SIn=H^Sw)pk2UdDvmfnSY$#NTpjw z%3H||DAoyYXC^>e(r6}?NltV{?$^B)Hb^AieUj_c{?M3%^(V%7D%8DplZ3U8 zuWt_upMS;rvXtOAexT?IMrKm;cR1%T1MIllq;HbY)VL>puU^d6-v02l@8|o@YBk6J z`cLD*cq#k7{h<%ze>)KqpG$3Dn4E57C4Rvoh z0S)V#@**K2;eUnn{WUE9KcNii4mfmiF`~aVW^!^;Kj{?6dUY7HqTPQ)8MJSxIW5nH z;BzsOg_A!emN+!TEvDr4N0hbkf1Hg}gpY(h$#w0F|0BrAJ_GtMkv$|uTMv)%!Nz3i zCu)H%52z#{qI0+DN>x^)C%n&X*$$1AxjAW9ZY~;FYlnH^#h+@dxwk7L$zVmA_r%Qd z-B5)XD2a=U%K$IN>q+JK(Kx9Qp9CNyBD|rtwWOXZ&Cb0EeIK4AKMo9_PA2G~SYhT$ zWsW{j+w$s1J9`rOq|d$hxixFiuLk6~k0629$18)I9#Ihy(`@@Q2Yvcy1DM#(Uj50I zd-J#yRp<7fMPO6Y(j1+f3_J?1FD>3-q+>sx(3TYNW+N~R)I+B`A^V^rMczn(+e2z(L?$s&d=(4o5svQmomm~&@$QFZK zH1)m7>ihow{+gvJm@RIo(D*c#FCD_o%3q1^h?cb~pB*32RC)e_s>hFRu~cuP8Cvh1 zJi&EX5IE5n=D+q6ituHnUHKs-sA`4YL8DmmC9+9IM&_mSWCq2!LsFE1X5IVD=iHOb zfb9x;3md@5i#s<2Q#$f_bbPVW0aa+ZsXA8W`3be1Mu z^g})?kggT2WJPAEDD@59R+Y$!G0#&%Cpx|CaWoZj8yO)58{Ev}Tm-VP=dN&_D$_#G zR7&ovSFgy2R{PPU$R1|8jk3o5`mlBaLDVSC23+D)FD&!DtvlR3ISR=FR| zceuE6GuyjVLQWeRvBG|UpP#>Qe!@-_4E=R?ALg1#P*o)G;fXKTZc~+q^~jrD4!G@% zKx_QZcOGlF25x${?DXlOvDu8){0OK+tK^DnJHb$;`|~?nfXHq>m=K zPTCm`G!eS&W2%t2jnoxt#Njsc^i_s$ZggB7Pty1SCP;UiiLvpS-tmx-5bLUl#3wm4 z5fH6ti67=d|Ao#7D`2W5)KTF>8kfR-`SSHPKG)gAM}~<;1N*rJ;fgRo6jv06E>IZm zpJ)<7(1R_kC=Nv>xdr|U6fX%yMa3mo*^omH%pigodR#66MGsQ)zOR>~JUu`Y39F4ydTfE p=~?EM{A2&>6ULu@2&uZx@WR_zp7_-37WjSx(LS!Dj#0A={0|p;!6yI! literal 0 HcmV?d00001 diff --git a/static/img/user-guide/experiments/dirs.uxf b/static/img/user-guide/experiments/dirs.uxf new file mode 100644 index 0000000000..b553a8c070 --- /dev/null +++ b/static/img/user-guide/experiments/dirs.uxf @@ -0,0 +1,24 @@ + + + 10 + + UMLHierarchy + + 50 + 40 + 180 + 300 + + type=Package +project + *D1* + *D1.1* + *D1.2* + *D1.3* + *D2* + *D3* + *D3.1* + *D3.2* + + + diff --git a/static/img/user-guide/experiments/mixed.png b/static/img/user-guide/experiments/mixed.png new file mode 100644 index 0000000000000000000000000000000000000000..a3f7dad280b88335a09dfce15755c91326687b54 GIT binary patch literal 17124 zcmch9c|6qL_rF56iXuvOQudcpVw<%_ug~wJ@=e*&+~aaR=uvUkD8g9goI?@)hid( zNJw@+Cn4EYM!6fj>3`P9LP8>+boHX#O&5cyK7Hv6{)|XEseXuEih9aCR~q z#kiv=Lgr*$T7X=kC)f~>e3O0+M@dLH4AWdmNl1A1irbQqJbYA&+6TTk*;%lQgyi6! z7zq-R^JTZudq_yG9JcBQji`M7Z#4SyxwN#jz5PtrWR}5XMpsC7c6QqSS&!Y6eg|ki z_%|FiNk~ZO?&%n@*IHh1_jFl_X=#Z)#XgyE?%eXIeMeSKPG;lmI_a0gmj%!6j*+NI zb4>d%hH}UC400VBEtUG#`uRz+pF5sJ7Cp0;7sioeU+*Ec03B;+o#OSmL}-1#=z?|gK98#i}M#jy`M)ipKQ zO1$OE2Y>8-qJ57-+?MjDmX_`bE2gM-cUgVQ)Eyliw>OL%eLQi)F~lf}gufi>>$V(+ zpl_();q)K-_X<)zil=)<4vp_c?!_oD5L?-_cNyh+A^HvhIe_VGElX-g~YYo?`sq1A15GKJC)y`T zNojT;>R|Oe#m^ce-Zv)VWNaJ~iJX}#OiiB3zTl73H|1;1b75BGyi$46NGmxnH`R9W z*DuEQn!rO6X%k18g<-B47YdqTCyh$!fT$OBsn?B-2CqiMt1cS}`f?oT30MsA@)B9o zSen*97P(4sS@5ton1+!qb(=_)8s)+CU~T-b8Tf=&SJs34^1yb!l5yg3hR)1AnUS6wHLBE*m_Au&)PoRXZ}Rg6e>ksAY> za^YkehpLt0o7GCN9$eW6dfJ_0w0d)wr*yd^BiRq9-=^HdNEYbM*PCX@Cmmj^7jJJ@ zNeZS~A!Z~k6}hJM5qtc^=2zx?d-J{wdd1W67#&WRcJ^#9{*;M^S6r?jiE;Sxbld7?YNajG!;wCukqkrMQlyPg>&CntJjab6uqIvtYsYf(5&& z=e$z+K92%Sm7qocdFRK+@6z~A4L-P;{8rT^FD2#a%&`)3NJFgp=}v^_nwgk{f8o)X zIDBO6tBL4#U_&Rq#uM-*{vtVcI=M`4- z3z^wZ3+rh?3oY&UbM$fan_({j%2Rrvv*2elI9s~9x|v;8b3^ZpDs8^n1S8LRGd|J2 zat=(RaBj{H$M6t*%xRbAdSQXhbvVqlE^^L%mzD1Un#aJHJUPt9hEmz34F-Qk9wTwl z>i+$O$*xT7EfQVB3i*SEXup7S%e@XkOd_(*(=2e3+s8oHTu|4wwO6-w{oK#b&lA5k zfZphu%D3U@UkjPY^~vlWoAR~8SExa=6I&*X3*Bv*y12Mlh{vtD?|K*k};rZdb@R8*Um`y7pAh z#Uwpp=Y?OJ4R%8fYzyZ0fZnOR571l%rcvG%zrH;8PLPN4vfw*XV1`C&!nlC}@mOrr zA_^u2ELU1%+&SqpXDX58X@|gvB;vNOV_v<&PT3;rh_Cj5S8fx@IW~D?38WPJXUR!4 z%L05`fw|mp0A6?O?d@m!3ulVfgo(>eZ`FVPw77V-(^Pkq!Gp&lBxwgh-&9+kgCRUT zoRyXJ!i5XOH8o9ht*#)Qn3Hrk%Fdcbixj&@)yz&`f4YuSj)knk{z~XjDDK7Wgj@Tm z201t)FL<-~FI1C>HP~OFqu7JQ>99oEC(y(&I|gJ zN$wvDp)z|3e%Qa;NRp?}7{w#@*Sm{$U)(3>1bvB7qEV*~J0B`J|L_r??-h<3s5>sCh2b^MgOs8Q zY@sLP>8QxMpfQKVgp~L$?v_K-27*0yPw*wQV{tuS_FSHa&IFP^tm$+U z56<0N>|wgmLDZPwvfPG2_Gbj><02meZom zlNwZCKT~9*bG10^r&G24!}=^Gzxa~}hIfTU`5C5|iaya!qa#cqJj#zV+kLsu8Mg54 z!twwDU*=0Q+JwaadK(fa?VNHS7X-%+MrawhcIZ2&i-FyXk+3aV>b9Jin6NEgUy6cz z=C)qJti7O>UHP##fXD5V=soFxs*kZPa$TP8i$^zsqj_9*?(wZmGof1S%4f@9d0}bf_w}uZtAwvRZ-!1aru-?y1tuE5?q9c;3LMt+cAE^x4-5`O~uF zdIoS+%q9X>2n0+TzcLh(Sp|2Ws{~6t)20Pq|Eg6q2oG64^_YV;py2f0bkirDffeVy z)Pd@O{U^N19!_^Y6mOfKmnRzi1c4JnVwA?P6%}NixCV&pOxjwdrngKnOeQySYhE>>Yq96Et{Jp3ZN4 z9=b&gHhp}9A)kIg3rV%f|af!HVaD=pi$e2|tCwv--%8Lf+w$NPtRczDE%<@7&DTDfoq z?;kAfGG72ShzPDy*8acGXcj-A+o#wp+=8OOXZg0^2eoffQt(mwGWcVVOa== zt`-ufXss+2|4P=Q8^6e&&AbXmD;pb3#k*sN*kSG~vmewGrKF{OJlD;8vaPlB9oum0 zYh5vKigAmnu+{G&Md;X>G&E%F!Er&%1;K~Wm<|KCF0f3=upazgMNb!DbuX_ z*N=3hBWGRD562q@2pLys^6a7J5sgk#xGerS=1$hrurO=fjQ+C?6GS+DgU=Mw>_@IG z3H!rhOpIQ$9N%#|F6!%Bze&^Yy_n?4K&{f&);3qz>a|dPQMQFJfIpnNtA}C9xr!aX zQtdAH?6H^qD;W-8vnnD_83`ODB|lhsZ%+D-V@T+cy&6YaYtcL+`z}~X4y&C#dGaK1 zfoCk+m6VibO|SC)V;RA1gUdAzjPk-#(vHy7|v8>H7;(Whl( z>NN{}C}>v)BKb4KCS?SK^r;T0sf2R3l6oJ|Jv2_`Jw_KJA?-9%)G_c5?mB{xfQtjW zfMAPpe)(b4VtsXSq&QC9AS~8AKHip7ew@nw(S!3RjgsWg$(>|1LYql1jJJ&rxMtd_ z#6I|h6}9TWuG-Y{*QKDRERt%g*#%tEP1h6$Dh)`cR*{rDsoY(#?%W?9-qYD%UzD7j z?1~wos1qqiO_43i{Pa-Lc)#`aB3bGEfCjf!Y~mYh1YIrqmL zjnto&(?pf3M##n5#C5JkEKYQc+DmGNflaQGKK%AR{qqNTV6wr{Hr}rs$6NW9a%zpf ztW0I*Db(!s`?zWs6n)Q4XoqE-Z*IM}*)DeT z(J$R&ZK<*GI({NAuUpOh7ZUaykve_pyW9GFm&6aP;6{$Pmb+q6XU-9 z`%(5CBWQLUGOq31JcfPPUb6jIQIyn{2v8K1eN9zsNRBWCnZ&)?{@6ZwM44T^$>HW zNgwI&?+00o3;z2*2x_B*86WFAvPg0e#k*yxm#mZZF*Vip{{82LSY~#aGq@_9c zyL*vtt-nJK<=tDBrJ~Muxvx{Cf*}2NMA;l@6*bgbTwY$&03m*5R0QYeNt*GG++Sf2 zeMKC6TE;@?T@x_$J*ROg2dH*7hv5?DoP)2Z*5wV|b$9Q@SA(#~oE_%*N<{c=kNs6u zE!G>jU4l0DQtNhHO{a;2Wwif6{$x9i6AUk7V!A4`va;5eXGViW658y6lnJ_(w72cT zgT_*+=asm%ZkkUs$_ZHtfz=jfYSl0gBb6tvO*^xewY9Z@-DrP~Ejyg|+ud9}RV{VF zU!>Z-!+NVpzA`N}huWg^FR9iYV*0f2sCb&#pu!%%V(m++TCtZ1>Ikh)?%VWLUpt1Oh1{hM|5mSCay^_am6xraWPEXvbEJ1P)zec_i&Ty&yu5m((oi z)Jiu6UnsLjfO&)pjHOC#&cmnb?I;5zn_zqm%mawrHV9cZ?Lkb5O!FW2RcsZrO*Iltr77yN8L2TA#&|OXhfnjbJE< zy&ce>nwqk{b7wY?w$bB@E&K=4Fn*Fbg%Py9w7VI(pV0Pc%+2qEwq{9NjfbkVKd;1e zeVtFauB;5!pfTyQbrf2$D`N72J4p~@QVWuo%@J59Z^nw~3t$tu?xf2!PIRP-^t=@b zo*Dr27t&0GDa(ul)%k7GjT~ONxw+Lzm|YrdYLYAKUr2nS5DTozYjQhOC{QR$oln8& zXfbnUX69j-5K~)gYnsj{;Q2fHfpPWIdHq{7Np%tw zy9AXHtArCvsY7SJA6mcHmEC7WbhDL~t#d-!j2tm2j)(DMD(oRrKq2L52J1P&W*qq3 zd1N%xyeSgalpNG%Dl{7`e@r|97C{tTu#r^NkNnY6^z*~%b7>A0n%EL2A%zgO5jg)% zZwp5Hki5U3HSnbJ8tgCezv|_G;3$o(uEku;0M-xjL6s`umXgeP;AJo|8G(W7n_>RW z{Z5$<6@#(PRwo;b25)GAvk4rX1EfD>LQYjj@F>=!M_Z8B{=T^4jVsn@7wwn0g}L{T z?y_LT;|4O+LCFrtYNTzh>CWLk;Q6D_1i|sM$GHhD5{r- zMSXZmuumJ0sHn$H;vrszV=6}o3)fa_%PFF~Pr)ET%Kkh!*c*>^@PuhRe^tt83<=+s zZv5GEMHVZS;YS-JcBySC$jxfy#5;2mnHt`C%(R4F?TiI=unkrb0V%*CiJw005|Mom zzgM7ArOx7DDDIsx;;s9yH~yU`*D+0q#wfnodX*avIdr;g?Cb-X<|@XC>O>hN|?0wcv1MgBeJkg|JFj!5Ne$KMrjkTcbMEo;U4~3v>E9iU#*0aMGl| zBJpPMOemNf5qS90M*UXuC1-}=Q~Xe*8NL2 zmsw=pSH`)Pd=Y*!Em%HlW<*B-S=Og4hP(PQ`dEc)c=!qPrZ^Amie1s!^8}+FfD~+5 z$-m6=-t?`H>crr)*D2oIp}1e^Io*@{B;0x$`rV_hKO@9e8;%jA$iJwbI4vRJRyknW zSB!wcSp?rFHF6QHC0Ul04c%RRJ>mCBKPcb9jPjjl;!YV?Jv*id-N@$Ao9543r(K6S zOW4NJ65UY>Lc2LrwH+S9)B%zPIB;ljZggi`AC0*opq0XahM+Cd<&R zBGbQq&8ALHPYc6Rxpv6J;t6E>1l~b1F~|Gob|DYJ__w*KamAmX=PL;gX<~9xF!tt7 zzqubqnI)`wpjG1@Su?v=b9wJ%tOVR?*QO^(A1Nx5MU?V`z`A+tT2f4VMgA9m_u}D#I@!5;Avr=QDH9Gn4F8t5`*Mm=Xd_B3 zC3^>t88)<`f*kg5@8NwGWcoVp)e^-y@R_5L1v*f`mQS922aot#-g^*Ih2IW<+zuoc zQ!mc_xvS#Si+}D4fABLKaW21&b3o=$sOOQsKevF<{rvM7sZW+p=8!>q`rA{=y<+uW zK$po7*WtJGqcaCdOC~DJpIhMdSGG3dKV5|6>E7Wlgl+abKAaO8mEgxMqRvW)CpI7H zy*buLFD(*>`Lme@_YF$#1%*vn{_{tJ4_4oLywLR~oS;8C2L%IOGCQgDFxdab#w)bg z{vxg<6mc5W4Gj&sElpZjSuHSqO-T`+_O{ny3JDIj?k~V3#Mpo!A77&&JFkLYP{FSc z;2ZvK*L8gE-MMpTs46&D@9DlWEAgK0?gx{dr*MDK01RR;Q;Y5Nj?oGW3qdUFzn@j@ z7BD29gO86x8A<0r%~XSi6eO@YAQxK;XF9*bcYrb4K5BrF^6lHVR>lVDM&$WLZ{GYI zt_{aJUOJpk=jSjcvAVg~V$M74I3C-CUr54MBd`>3Mvfi!Md|P(Qc_aSh(fpJc~gLb zejCwTRo+^|`aW>^NR9{gY-~`4dwuy-5Tnp)F@6oUX@jAwD#5xs(}}LHuX{%5=R3`r zG68sWw|_HxIpM6Jnh-=g4SHH1MB=fd@mduJJ!XH%;R0`O*v@!{!<7SPxa5SAXypf1 zR=(hyhC#d}>xn%Jci(kcFf{iW&IO#7G}naa30>7X9Yx?s+1+PLYieq`vn;#7>s){J%zD`4jXrI(95)>bk|4yV5sSRV^P|8A~l2jP;W z0GBlRDlDTiVKnfV{y3IDM%rPKhptW-B6{Cb>88s#Pf@oRJmS~nI zr4~J?dU9k^Mt9$U%TTb|*fCdmEmm`f^Yx3$R$$dKQZb6>Y$w{24~YOs;$(9)?k+H= zhG-?R4hiDb!5@-<_&`pt_5#N*`>$U#N8-l+{R_1S+()2y|1wH*xaZLg9dPB{jN@X6 z=|U)~qvn0bvmOb|lGiqXNN=#kwhb)EcEstJ^m;a!(2mc%TWo*+{j)RcPC(`zpK-O= zPW}Bek#%t(_>RxYnr$(E|BP>a>F=L`Th()5ZMWlFisx797r-&z4uNGS!fpX4x*bz9 zcU+}kKAH1~qJ78#B`zyEv0_~#HRwwL&4A1d}2@z3w5 zSa0H=b*R{<#6K%hu`f6NR9$b7R;G0TG?I$fUt=WLP=GJ;O{1aVDB&I5O>1;jc`qQ3 zoIYUNfBgC@m!&Rq{7-hAVSj;Z+-L(5hO};Kg5hPP;hskbZ8?Ih&;ae#Q&6b>bSraw zd|Y^S(!D7L1TFvy!Ys6?WLnOigs3T3zFYN!Y;rD08C6>6_j;4lOc$+z6cH~iVdO`2 zrKpB}mCV@ugap5V6!Iv2rz81R9VzR)C>bgrqv391#Qg`s#HYGx+U2O};B)Ie$O{(KBUfW#x{;jAH!2`}4}34QuMi zRfb|o_}5^G{*@+RN8K+#qwr3m*26;F9}T<`F~wmq(Mo={l;C~qqrL+3`WL0PjZvbO z?VrFMbABJ#jfXGOVsh|T@9Jc93HC$8oKWAs0{k31J`A)U0jSz(97Mo*C~3T*v!PY6a|s>C!( zKij5Hia6m?l*e>$F2MaQQ*!+V-iv6$;l(Dk&jeDw7=gnNt{*>ifovpa4p{ZKRN^%z z;rK;3laLO>!f+>;ER^F+)yuWFva)J;d1i9&ni#SP4Nayb{$i%93ex}rhOQSwqZ+`9 z75wx&ifTeOiO)}@S-E*cm)-CK0x7 zkzEa&ui;i#S9d=Je!HpG&=6zjCf<`~pv5>h0Ewt$n z>RfPQ4S4B&E6-JB<5Q9)n|@is?L#XmMH=nErs$rPUk^Q2UU_+MaiZK$ zH`JNIy1To}q)L__^BfsdDOERNGke)bM7pm4jP$14TAtqYcwAf)EiJ;~L7o`Or|)y( zC{AxW58O~^i|ZoZEgWCv6Ws>33uik>oauYjgwXBP9Fozdc=<9ieLZFdneZqcma7tG z_#21+ym$ZpjOAX9f!uX&Btj-imtlT%v{n%zkK`)7*}I`fZ3%XXQB>tNGm#@;QHrt_ zsY7ywkJjch5YT;q(wEVy?gC(kABQEMU(kT8&(nm2ql(V5X3k zogSVwjw&i@d;30{-Hxf+{E~#(#B?qFVKy{|XkjgIcI<4~*b#rF&tVa&;%-Z()giWwBB3l0 zoRKl|=|P}UTw+l5)rysvOFseP-UlKagm~>{QFYam;W8i`^TEdHe0+289P4*!h6>!h<{4v6}^;Xm$u44M)^4a-&Q3?s{$JfRarM8$@Gwk~8e?X)#L(F_gO$9WH z9=IoFM2&l7#u(5jd}a{%HHqGPIpk@?l!tWL}1V ziJK|1Nn<86o8qJ##{rj#hEeb~O*A4dxN&-VTFkt0Ma!lu<5)~X(fn5dK>3-@w(jY? zwj5QAtrX#hQzIXYf6%qb9hv=Ej#%p%(Dt$Notl|3A7w{i?b=s@M2c3x))%ghC&4ja zwXhR&vb-B?lua%~=n29>+ds|Jj;j9v&lc<%z%L`(8ckC80aE8UVV}-d`_s1$Ecn%% z$y*CvA1Skx^4Eg3mR1xV){>4!V(QJWje?w0XZLa$jmYyxKt_s;OlwxF5sm><5J6z- znoX@t0lq@6%Lp<+4UFy0n1h18JDDGHmnLW20de91bXafo5suLH_a9|0@iv7M*&rT^ zNpK;Lx_;L*%TC>WPPn53E0@P)f(}X}7JhwiKC*OqIL>W)u2@*ew|K2@rT~rPa51p9 z7IHgYQB;UqomlUhuZH2$Zx)g%8gkW;azWVl5$v=pOG-)xD}%N{g7)^6c%pVz zM0P#v=!)5#k;Gh z&^pKE0xc*4j0iwHQr`P~h(Ipd+|D0X14V_InJsw0TFh4M6rO(FCPnR=H~}KH=~T9k z7WN*XcG0}hyfM=>#W;u61q*x!qVw)AqzIfkF{l`ao&%&6piPO_V~rd*esv&#zB#pN zXiW_iBMXnL0^Y~!TlV!|(gT?N()do}5Fx!UvrNS+mt>KMsjQA`#klhJmXW%sD9_rO zga05e*qWFOmr}6HXx)^GuGJG0&A1Bj7@jkhZB@zQgTtu7$1pxK!hm=>hG(c?J_r*>M>;r_Pp`_TxraV$BbJC z9I*~!#YkcOahOYa+Tuc^^h^PEzOL{gQcctM@QLc$?HW+B0^*N?E=B{-qWBT7X`3lp zIWcEIc>4VL^Ydj@re=h}-rwTc)pBd++8=X^DhEWdQ>stHx6^xGXJ$IAElpv&xR$ii zf!lY-mE&6gbS;bb0lDSX9?pHuc<0MtPA^e={Q^ zZxO|RNcrlFq-3tkoKfVRq8~=UnON1;k1#GSe5R$frO(kQuXgtLM^24l7~Ps0@($pT zcfvg5-T(tYAV}p#s_%hl9^e3#>(%l3oL7be9A}AqE!?AlNlO%>S9QN^Fe~g#kIB&W z$6B~jHN$TWtAx1%#@2C3HEhG{gJgl+w3M(KuF8SE{Bo}oMBGVe+(sLi&XSDLGEAwP z=!i{~c#%R|pJ7U~HZcss@J{5UrITOmsEBXContlRIuJhqsp;LEh$Yot5tTR_i&G4o z+}VI5@tj$Xy=0Jgv{{95o*)b$rn$?`Xp|9PxQnIg*Egvq zarWNLa`O=ODzMJ!>l&!!k??ow#_gLO?6l1e?#D5PpQPHap3N`Byj^36ZdWQ-&zx(y7E1GZxA%dguw=4h zVMJU};u?th`l74)Q$N9rWuxlS11D9M&h(caj*ofW+OiMuIqL7z-xR{QdsaVXb?&uO zP*x6h-nI5J)^Jn+H%|vagrd#1VwH(CQ2zIK!c~!Lu(zkD=i9fEJRJb2bmw~Dj%a>O z`FR(}b(4o&NBh2ya+4ABIPbNma2o|xa>$-?+FHJ)KwcYnZXO*=+*A>88mC zk{s>$6(D{<9dfk2ay)}^R6I$-y&>)4` zYg)9$wZPEp7Ufy!3mR$cdA*m0a!QP^@|^;}3MwMAE`MsT{J@evIkG0=Kl$tXTdbn* zwUsdHFoAQImIi}F=1r+(49`4%m8)B$%XVU#uD-v}4U-{CPfw3C<%h715v-1?&BKLa zIhW>l3T3@Q!$Ja2wUL9BhtppTU{MlKcP8A2xQ4GXky8=q44k&P#l-{26askl^-i*l z4pS8RlRLZmCANN78U}?=DszL+gS89LPlK)A$v=`A`-`KL)>;gpQWu-&p)_Wy$7UK+ z@3N(1OWv1P?~-h|C@MmVEwK9BTo$)HDy~oHy~;QcgaCa6kEt9Bfq>%;CWEj1o7~(k z0i5U9M)J~vFg_q}WNt!=fGZKKogCC6VKmo+n%ly!H*&JM$;l1KB;ZYk>d15m)A!CB z7bJp%fGDBEtvNBc5UedT%1W!)f|Mno!5=~DI8ey>{{0pDto;0ZEY@$|(au^WUUg4< zsoP)z+LtO*@}3Z>B{w#9e(pZAg5VT=d*|ZHqccK4fju^k7GcLBWNP%|3LHm}K4~XD z*&fiA*BSNoK+sO~7s4(@bqj>!(nhWO@*MM;$2I39w4#LdKf0v@io|wMiZF@$A=-U- zjVJm|JMdmG{038k2={_sb0fLUUA3t2NQnWmE4N~Cr#Eo!Ew02)js;g9aG1dUsCK`r z!Fr->$DVldu6;9>0ae}L6CxMu%{VmkK6&mL;7*(0AZIZ~) zN`Ioe05FoAeoIsWqktJoOc)DCC8ee|nPmb}Ch(L)DSSt^mT@PAFTRygn}wkq&AoskPG;9CZ%2<5a+0`gMx)EDqjCQ zxZq&Ub)lE0`|6@1q5(}!B;2-tgh5yM_{>M2O=C14nN<~$8{{#cB0ml|$poDO{Id-k z3Qr@l&#d|bg`&cp+vGOAhijeA(AVBdSywI2MZ&7QgvBcl%`hLh3c+$Xm>D1euC2@s zx!WD-YtStmv4575_n0NL`L$?ZvE@Y`U;zajn4+%aQ4616H8z4k3V@8z_{uvthzWp| zBmn*Hr;;0VlSZ~0|ENs0MN${-!DPw&kX{6=H}$bI;;HwxkVaQl6Ukh z2mt!cN&YXSfY#n1)}S{~5$QbLJC>>mSeAM>D%-4eT+xxAKgk2+kI?L~kYZItJ4b7t zwl8su38>X$q0GuUc-mK@acuACsD_Wb0+c7&CHLTC~30U+rxUIOAIM#T3!Ap~yY zfnc7X$%;MRAcMMqA3Z}a1rnY%jMwkP;{TxDXFDwN|GCuYzYNW$9r8O9tQhL~`Y1d5 z=tfzb8flCKose$!WS9cm&!*j-)ZC2CWZ>04T3U1>i6E$PweW5@`U@)fl9KxRySj@* z3g`>SH6Y!lC5aViyqyJJHum<bQyk$5C3DTx7vb1YK# z1Bhi|pf(Mp$f1H_!&q65)%0pMX_r|JVkOxwFmgM%V~2y#>U3PGP%cFCD;M|AcRm*pzz$kQBDX7B!#W--Al`IR2Fwj zX3Do(dYhaq^oi6RN#YGk5XV9FVDyI%AC{IJb9>v)okOy-;ZO_r&Fd-hEmWG;b9)^D z)$)_rwo+(OD^{z!d6H0h=#@&>wEd^g2pK1F4!|A)xZ9f>lt8|QDr@#9j8D8KR@Th6 zT-hjO1?A8}LP`1gF_ju0bQGMO-4qpR1{5Cx-a;uWn5rr<&42adR+^ve3kk8v8WfN) zQukDXI%^e;1m>LPgOu%o|0~R$i=yh09r4eNMl= z6|X6S^3Ri?a#)8+#kj)z6p%U~n<0{&loY;{+LiAdp{*mu%Nw@R_!s9kbL}Uf<%R017lTf16lF7* z@d<8}M_&U)MN2bq0RvS}Q1y9b$3QRQ(~_se5f*@xAFTcLADvt@poVIqFZJRNrW=;4 zIf;hn2IjLt;=~;|dLPtHZ>Q zy5UF*%F03Q>~smJxdwZ5>kmpm)St!okpA4?sXsvR8X$a$pm@y@wI(YDyY(JI{ejAV z9UZn`(~%wzZq@^hRoJPq-+uhK4eH=z@z!=4f%uwHL}=ZJcGR}kqTuu>E7#3qsw=bj z8hw&|l$D)#{^tQDW#x486H&UYEiJM#1Z_H3eb4)tVQyqXz_2{nsx$t7^ZyozhIqp$0c7)!|JebJ-!&l+o%?)& Ti5y^;Bv&t8znCI_+xve3jqX*C literal 0 HcmV?d00001 diff --git a/static/img/user-guide/experiments/mixed.uxf b/static/img/user-guide/experiments/mixed.uxf new file mode 100644 index 0000000000..0db951500c --- /dev/null +++ b/static/img/user-guide/experiments/mixed.uxf @@ -0,0 +1,104 @@ + + + 10 + + Relation + + 380 + 140 + 110 + 30 + + lt=[]-() +bg=black +lw=2 + 10.0;10.0;90.0;10.0 + + + UMLUseCase + + 220 + 160 + 20 + 20 + + +bg=black +fg=white + + + + UMLSpecialState + + 160 + 270 + 20 + 20 + + type=initial + + + + UMLSpecialState + + 240 + 270 + 20 + 20 + + type=initial + + + + Relation + + 170 + 270 + 90 + 30 + + lt=- + 70.0;10.0;10.0;10.0 + + + Relation + + 490 + 260 + 30 + 130 + + lt=[^]-[<] + 10.0;10.0;10.0;110.0 + + + UMLHierarchy + + 130 + 340 + 140 + 140 + + type=Package +root + name + name + name + + + + UMLPackage + + 700 + 220 + 100 + 70 + + Name +-- +- Text +- Text +halign=left + + + diff --git a/static/img/user-guide/experiments/tags.png b/static/img/user-guide/experiments/tags.png new file mode 100644 index 0000000000000000000000000000000000000000..a02c8036c0eb5143419720ed393c0ff63bdbdee7 GIT binary patch literal 3015 zcmb_ec{tQ-8=m5ntQkp`EMtUZD@&b8BTX2xd`>9`O$kFCBiqz zI3Zgz6O(0VEZLH=Gsgan>C)-@`#b;quHW^(&wD@j{XFmeTrbAl)PRp$j2i#|@WBmX z768D$9M*p@*M8PF^tI9!0KgLshh4ntb908~>U8zG*nxR}$MSE@#=et0jTY7=LbYS{1=vLa^Ja)>Kk{Ijm(qxr}2bcFMq31#y9UD zES0S->xVfXiuX8?e5Im7GE!fux+VyEezUTw>JLd(Mi|-~DiUQ`UQvN3&TnvppUJ`+ z9!kWWp{GSOOQt+$hHZt;yO;7c*{#q?qpJsxAYrO;Kn2Lbz78KoW#OVf=}#dcRfBB7 z`XEOci8|3v;~egG(xZuqyrQDx42W*`_wRRIP^bxSXDz>7XyHU0N2Yo6w{Pkj%xiBt zI}{sZ7kT|4gIusW3PGa7}w zdv9lnZ()+$LKkVP+P_^pGX@XRWyVWLp69az_B3@BKW)&0s7MKh2*Nat<7|SBw3oD<=LqR(+jA?tsK5xv`h4H*=aPo7h z2P7|gUqTPj^PM8l&N=r)twW0i)Rrh|Zplb+ zB{)|iWIEi>$(CYFQ~MsQxl?)(m8IREmyH<@gM(|y(4E39ZmXmYSi!Eqr*$*0R~iL& z-jcHDPh!<&@2#xvwE0~zBdcssPj!q(j>-AW)j@B>_)g*`>BzEE*dOfT+y*k~P4TSw zx-AB3TD`cQKnc{^%?ii#xlKQ_1zT*+RHhLuuQg*+EH8g>3BwB+e~6suLfk4RRIj(n zcFYaxpn`Oh#@+Sb3)O9)%Dlq!U->L&$ME;1rv+49$=U5i`1}Bh-$DLGq$sPS_&Y6k z(ORmmRBP9n3aIo3E0-w7BLBL9LCnnHmJ;LP`}dn>7uF)S)PIU=eK%u{wrhLd`G_Ot zV3*12L<+f|JYl1P1k1*i&(i!m{XQ;{T}#r5c$*UzzYt|c_77-Vy-eDqFj=wZJ2#EM z{jXa}4i1nVtyF~azjl}uf<`MYQ8998{o=YFr{hqs@7fHcvU^o(Tmwu%vi5TyGeQ9R zWryai8WmGtf+QDnOcm<|GN`vVgz;HO!aRm%&zCb0cmN@7v#%h>+5 zORoCM!2DpmM_zfk=F!6R=RVXxDW9r79g&Ewp8EQt8za1Kg0iym$o;d!=NMslX#dM}10T$;u*GBTB%ZRe zvemgEvs{bKL!S=pavKA-n^x!ND{dcl3f+xz7jTfLjWmX>&eA%he#5aRz-%TxCnYUS zQbJ-ZCfJl+msN8#Fe)VF@lGhbyS4%+wVs%1Z ze(h3*C|i$oBJK!XqQOOZ_CZ|*$U>1#)oC@ghCL$h zS%#|dTlr21*kER72(j_bCgKe9uW}<~df0;ze5lWM7{3~dVP5c2KqZ4;KJ5v!GbqDk>KLuqNAhplbUe5>BBGqsl_ne zGwh>(DM0o`OX~E0{@k?33!h2ne4d;vDkk=IuRv6DbWtR$K=1B9qM`S&#`qF(O5U?y zs8t^Fgl0IBji+snHeMb6n`5s7i@fb9OnjTYi3xGM1G^@a!MCfk)mWh>msK9gK>^Dj z5|FHOoxQ!iLqp^}9+H=+%FX5XdKq*7vvw}QD0g1ij>w+bMj7-}(PPKBLi8FN8~2Fi zaJG|5Gc#psanTd({?296k4Dh$%A2J%L7z-TvW>yzfF(9H!H~$wP(JY_W$U9%RmKjT z3qm3-nthT4P6=SZf;MOj?CX~*k9$3@6%`e4`x+S=_r9?fiD}tlZUjF-BDaz zT%3@%FO4oKqpNRHG(utPmq5Nxp3&&^zx@6E)sB56lP5vKA|fKf!d{-9B`x!26pPcz z=Y8gfH`Pv;lJExjmL=>bUZ70es;4nHkIuSY)8D_-H7m`1eb;*4SO+ZJi{(H5d(+$n zC5F#7@Wn+Ccrguy;Y>X4U-#h!Y6Ll&eAUuYOH1qS_Ih+(*2HQZ*I+F_LJW+Tj>BqP z0cX(rki}F4MfnH@MoK-`h|m+Khaj!2tQgm6X!-)(%8%f58N>thg_B{Z4wX1V&c_!6 zdy|Ea97#+}blR8`#^G?A(IQbg*!t81u}(7J#n3d&mn0gs3@NgMCs|lqi^sIA@l0hB z2!m4ygQJX744Z9tG2JX*!!c1Ep(^DXRu309h`D)rjvYzLn;{x%nwsU$i_bg0PwOaZ zKwMP>f0d3q_|{~sf^J!4)8?v$?ds~fR%p30{j#zWJ5+dy&b@j0nH~rNjwc3WrD(e@ zUX~3Wz$+;z412#YRy$@xcyKQ02teB9)~$!CVuD|ia4QMG=qHW0Yw=45fK-{dzh;G; z*V|EB>noe49@KYkvINITzJHQ_UGQ;pQ?DO-eSp$8=~wjABjI@-&38<_w3!S$1Zna&}f!A`}!9n51Zxig?nO zY|xfVy`D>+{!%Kdv0Rwim)4-$4oEl{xZsmoZhcmQKc_x7dJ8(3e?!PH2MUfixK9<# zZu6hyd9r(3;pXUw)0&!I(>-stw*wgn_|U>Pb#Rvy5EJq*JTw3AC+ocHSbrmLic<)Q Q^*{%}FPp-O^lk + + 10 + + UMLSpecialState + + 30 + 90 + 20 + 20 + + type=initial + + + + Relation + + 40 + 90 + 90 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;70.0;10.0 + + + Relation + + 110 + 90 + 90 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;70.0;10.0 + + + Relation + + 180 + 90 + 90 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;70.0;10.0 + + + Relation + + 250 + 90 + 90 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;70.0;10.0 + + + Relation + + 320 + 90 + 90 + 30 + + lt=-() +bg=black +lw=1.5 + 10.0;10.0;70.0;10.0 + + + UMLState + + 110 + 30 + 50 + 30 + + *T1* +bg=yellow + + + + Relation + + 100 + 50 + 50 + 60 + + lt=.. +lw=1.5 + 10.0;40.0;30.0;10.0 + + + Relation + + 240 + 50 + 50 + 60 + + lt=.. +lw=1.5 + 10.0;40.0;30.0;10.0 + + + UMLState + + 250 + 30 + 50 + 30 + + *T2* +bg=yellow + + + + Relation + + 310 + 50 + 50 + 60 + + lt=.. +lw=1.5 + 10.0;40.0;30.0;10.0 + + + UMLState + + 320 + 30 + 50 + 30 + + *T3* +bg=yellow + + + From 2492e6ead44d5ed200935ffff162cc3b520fe2b0 Mon Sep 17 00:00:00 2001 From: Dashamir Hoxha Date: Tue, 3 Dec 2019 09:44:48 +0100 Subject: [PATCH 03/11] Index page and Tags --- static/docs/user-guide/experiments/index.md | 16 +++ static/docs/user-guide/experiments/tags.md | 108 ++++++++++++++++++++ 2 files changed, 124 insertions(+) diff --git a/static/docs/user-guide/experiments/index.md b/static/docs/user-guide/experiments/index.md index 0d81aa6568..370157294f 100644 --- a/static/docs/user-guide/experiments/index.md +++ b/static/docs/user-guide/experiments/index.md @@ -1,5 +1,21 @@ # Manage Experiments +Data science process is inherently iterative and R&D like. Data scientist may +try many different approaches, different hyper-parameter values, and "fail" many +times before the required level of a metric is achieved. Even failed experiments +can be a useful source of information in ML. + +DVC makes it easy to iterate on your project, providing ways to try different +ideas, keep track of them, switch back and forth, compare their performance +through metrics, and find the best experiment. It stores all the context +necessary to reproduce easily and efficiently an experiment: data, pipeline +stages, parameters, models, etc. That way, someone else (or you yourself 3 +months from now) can check out and inspect all the details of an experiment. + +You can use several ways to manage experiments, which are described on this +section. Which one is more suitable for you depends on your preferences and also +on the kind and complexity of your project. + - [Manage Experiments by Tags](/doc/user-guide/experiments/tags) - [Manage Experiments by Branches](/doc/user-guide/experiments/branches) - [Manage Experiments by Directories](/doc/user-guide/experiments/dirs) diff --git a/static/docs/user-guide/experiments/tags.md b/static/docs/user-guide/experiments/tags.md index 6ead626b31..91bb8443be 100644 --- a/static/docs/user-guide/experiments/tags.md +++ b/static/docs/user-guide/experiments/tags.md @@ -1,5 +1,113 @@ # Manage Experiments by Tags +You can mark (or label) each experiment using Git tags. +

+ +Using tags to mark experiments is most suitable when you can easily create a new +experiment based on the last one. For example when you are trying different +values for only one aspect of the experiment, say feature extraction. + +## Examples + +Examples of managing experiments by tags can be seen on these sections of the +Get Started tutorial: + +- [Metrics](/doc/get-started/metrics) +- [Experiments](/doc/get-started/experiments) +- [Compare Experiments](/doc/get-started/compare-experiments) +- [Get Older Files](/doc/get-started/older-versions) + +They can also be tried interactively on Katacoda: + +- [Experiments and Metrics](https://katacoda.com/dvc/courses/get-started/experiments) +- [Data Versioning](https://katacoda.com/dvc/courses/get-started/versioning) + +## How it works + +### Commit and tag + +At the end of each experiment we commit everything to Git and add a tag, like +this: + +```dvc +$ git add . +$ git commit -m 'Baseline experiment' +$ dvc commit +$ git tag baseline-experiment + +# experiment with bigrams +... +$ git commit -am 'Evaluate bigrams model' +$ dvc commit +$ git tag bigrams-experiment -am 'Bigrams experiment' +``` + +> The command `dvc commit` might not be strictly required, since `dvc repro` +> usually commits everything, but we are using it just in case there is still +> some data that is not commited yet. + +### Compare the metrics + +To find out which experiment has the best performance (the best metrics) we use +the command `dvc metrics show` with the option `-T, --all-tags`: + +```dvc +$ dvc metrics show -T + +baseline-experiment: + auc.metric: 0.588426 +bigrams-experiment: + auc.metric: 0.602818 +``` + +### Check out a previous experiment + +Let's list first all the tags: + +```dvc +$ git tag +$ git log --oneline +``` + +To go to a previous experiment we can do: + +```dvc +$ git checkout baseline-experiment +$ dvc checkout +$ dvc repro evaluate.dvc +``` + +> If all the data and outputs of the previous experiments have been cached +> correctly, this `dvc repro` should not have to re-run anything and should +> finish quickly. + +Switch back to the latest version: + +```dvc +$ git checkout master +$ dvc checkout +$ dvc repro evaluate.dvc +``` + +### Move the best experiment on top + +Let's say that `baseline-experiment` has the best performance and we want to +move this experiment on top of the Git history. We can do it like this: + +```dvc +# make master identical to the tag by reverting all the changes +$ git revert -n baseline-experiment..HEAD +$ git status +$ git diff --cached +$ git revert --continue + +# delete the old tag and add it to the current version +$ git log --oneline +$ git tag -d baseline-experiment +$ git tag +$ git tag baseline-experiment +$ git log --oneline +``` From 6b734fa9ad96121886d8ed560ced84021f403833 Mon Sep 17 00:00:00 2001 From: Dashamir Hoxha Date: Tue, 3 Dec 2019 12:49:16 +0100 Subject: [PATCH 04/11] Manage by branches --- .../docs/user-guide/experiments/branches.md | 114 ++++++++++++++++++ static/docs/user-guide/experiments/tags.md | 2 +- 2 files changed, 115 insertions(+), 1 deletion(-) diff --git a/static/docs/user-guide/experiments/branches.md b/static/docs/user-guide/experiments/branches.md index 2402641f32..2dd44aab22 100644 --- a/static/docs/user-guide/experiments/branches.md +++ b/static/docs/user-guide/experiments/branches.md @@ -1,5 +1,119 @@ # Manage Experiments By Branches +You can use a different Git branch for each experiment. +

+ +This is usually more flexible than managing experiments by tags, since you can +easily base a new experiment on any of the previous experiments. + +## Examples + +An example of managing experiments by branches can be seen on the +[Deep Dive Tutorial](https://dvc.org/doc/tutorials/deep/reproducibility). + +These interactive tutorials also manage experiments by branches: + +- [Pipelines](https://katacoda.com/dvc/courses/tutorials/pipelines) - Using DVC + commands to build a simple ML pipeline. +- [MNIST](https://katacoda.com/dvc/courses/tutorials/mnist) - Classify images of + hand-written digits using the MNIST dataset. + +## How it works + +### Commit and branch + +Let's say that we are working on the branch `master` and at the end of the +experiment we want to save it on a branch named `unigrams`. We can do it like +this: + +```dvc +$ git commit -am 'Evaluate' +$ dvc commit # just to make sure all the data are committed +$ git checkout -b unigrams +$ git checkout master +``` + +Now we can continue working on `master` for another experiment. When we are done +we can create another branch for it same as above. + +### New experiment based on another one + +Suppose that we want to start a new experiment based on another one, instead of +starting from `master`. We can switch first to that branch and then start a new +experiment on top of it: + +```dvc +$ git checkout unigrams +$ dvc checkout +$ git checkout -b bigrams +``` + +Now we can continue to make the necessary changes for the bigrams experiment. + +### Compare the metrics + +To find out which experiment has the best performance (the best metrics) we use +the command `dvc metrics show` with the option `-a, --all-branches`: + +```dvc +$ dvc metrics show -a + +bigrams: + data/eval.txt: AUC: 0.624727 + +unigrams: + data/eval.txt: AUC: 0.624652 +``` + +### Check out an experiment + +Let's list first all the branches: + +```dvc +$ git branch -a +bigrams +unigrams +... +``` + +To switch to the experiment `unigrams` we can do: + +```dvc +$ git checkout unigrams +$ dvc checkout +``` + +Switching back to `master`: + +```dvc +$ git checkout master +$ dvc checkout +``` + +In any case the command `dvc repro` should not have to re-run anything and +should finish quickly, if all the data of the experiments have been committed +properly. + +### Move the best experiment to master + +> Usually it is not necessary to move the best experiment to master, since we +> can easily switch to any of the branches. + +What we usually want is to completely replace the master branch with the +experiment branch. Using `git merge` is not the best option in such a situation +since it will usually result into a mixture between the two branches (the master +branch and the experiment branch). Instead we should copy the branches, like +this: + +```dvc +$ git checkout bigrams +$ git branch -c master old-master +$ git branch -C bigrams master +$ git push -f origin master +$ git branch -D old-master +$ git checkout master +$ git diff bigrams +``` diff --git a/static/docs/user-guide/experiments/tags.md b/static/docs/user-guide/experiments/tags.md index 91bb8443be..6947f65fdd 100644 --- a/static/docs/user-guide/experiments/tags.md +++ b/static/docs/user-guide/experiments/tags.md @@ -47,7 +47,7 @@ $ git tag bigrams-experiment -am 'Bigrams experiment' > The command `dvc commit` might not be strictly required, since `dvc repro` > usually commits everything, but we are using it just in case there is still -> some data that is not commited yet. +> some data that is not committed yet. ### Compare the metrics From dff0f05478606ae03d93c82c2e63b8fe46051cc6 Mon Sep 17 00:00:00 2001 From: Dashamir Hoxha Date: Tue, 3 Dec 2019 22:43:26 +0100 Subject: [PATCH 05/11] Manage experiments by directories --- static/docs/user-guide/experiments/dirs.md | 92 ++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/static/docs/user-guide/experiments/dirs.md b/static/docs/user-guide/experiments/dirs.md index ad95f26e8a..238934b2b9 100644 --- a/static/docs/user-guide/experiments/dirs.md +++ b/static/docs/user-guide/experiments/dirs.md @@ -1,5 +1,97 @@ # Manage Experiments By Directories +Using a separate directory for each experiment is the most intuitive solution +for managing experiments and is the first thing that comes to mind. Most of +DS/ML practitioners are already familiar with this approach. +

+ +This approach is most suitable when the different experiments that are being +managed do not have significant differences in their code or the pipeline, but +maybe change on the input datasets, processing parameters, configuration +settings, etc. + +Often it is possible to generate these experiment directories automatically (or +almost automatically) from the code of the main project (using the parameters or +configuration settings), so keeping them in Git is not interesting or useful. +What we would like to track instead are just the parameters that were used to +generate the experiment directory and the results of the evaluation (metrics), +so that we can figure out which parameters give the best results. + +## Examples + +There is a very basic example of using directories for each experiment at the +end of +[this interactive tutorial](https://katacoda.com/dvc/courses/basics/pipelines). + +## How it works + +If we have a directory named `experiment1/` which contains the pipeline of the +first experiment, and we want to create another experiment on `experiment2/`, +which is based on the first one, often it is as easy as: + +```dvc +$ cp --reflink -R experiment1/ experiment2/ +``` + +Then we can continue with modifying `experiment2/`, and finally we can produce +its results with: + +```dvc +$ dvc repro -R experiment2/ +``` + +The most important DVC commands, like `dvc commit`, `dvc checkout`, `dvc repro`, +`dvc pull`, `dvc push`, etc. can take the option `-R, --recursive` which is very +convenient for experiment directories. + +The command `dvc metrics show` as well can take this option: + +```dvc +$ dvc metrics show -R experiment2/ +``` + +However, if we use just `dvc merics show`, without any options or targets, it +will show the metrics of all the experiments, so that we can compare them. + +Deleting an experiment is as easy as: + +```dvc +$ rm -rf experiment2/ +``` + +However we should make sure to save first the parameters that we used for this +experiment and its metrics (results). + +
+ +### Tip: Use a script to create experiments + +When we build a pipeline we have to use some long `dvc run` commands, with lots +of options, to define stages. Doing all this manually is long and tedious and +error-prone. The recommended Linux practice in such cases is to record all the +commands in a bash script, which can then be used to build the whole pipeline at +once. + +Some of the benefits of this approach are these: + +- Typing mistakes while building the pipeline are avoided. +- Modification of the pipeline becomes easier and consistent (for example using + find/replace). +- Building pipelines becomes flexible (for example bash variables can be used). +- Pipelines become reusable (other projects can copy/paste and customize them) + +Using a script to create a pipeline is also very convenient when we want to +manage experiments with directories, because it allows us to customize the +experiment based on some options and parameters that we pass to the script. + +This can further automate the process of creating a new experiment, producing +its results, saving them, and finally deleting the experiment directory. This +way we can automatically iterate for example over a large number of +hyper-parameters and save the corresponding results. + +The implementation details actually depend on the specifics of each project. + +
From ad6d7370002e19af91ace28e1f2a16d921f083f5 Mon Sep 17 00:00:00 2001 From: Dashamir Hoxha Date: Wed, 4 Dec 2019 11:20:57 +0100 Subject: [PATCH 06/11] Manage experiments by different methods --- static/docs/user-guide/experiments/mixed.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/static/docs/user-guide/experiments/mixed.md b/static/docs/user-guide/experiments/mixed.md index c8ec962928..a8eb3f8aed 100644 --- a/static/docs/user-guide/experiments/mixed.md +++ b/static/docs/user-guide/experiments/mixed.md @@ -1,5 +1,26 @@ # Manage Experiments by Tags, Branches and Dirs +On complex projects you can use a combination of the methods that we have seen +so far, in order to manage experiments. +

+ +If you want to change different aspects of your ML pipeline, like input +datasets, featurization, learning algorithm, hyper-parameters, etc. you can +manage these changes with different methods. For example let's say that you +create a different branch for each learning algorithm, and a tag for each input +dataset or featurization. Then you can create different experiment directories +for different hyper-parameters. + +There is no standard solution that fits all the cases. The way that you might +combine the different experiment management methods depends on the concrete +problem that you are trying to solve and the details of the project. + +In order to compare all the experiments, you can use the options +`-a, --all-branches` and `-T, --all-tags`, like this: + +```dvc +$ dvc metrics show -aT +``` From e7dfcbd7d1a09fa34b3ace99062d4fb60252c614 Mon Sep 17 00:00:00 2001 From: Dashamir Hoxha Date: Thu, 5 Dec 2019 06:53:31 +0100 Subject: [PATCH 07/11] Improve wording --- src/Documentation/sidebar.json | 2 +- static/docs/user-guide/experiments/branches.md | 2 +- static/docs/user-guide/experiments/dirs.md | 2 +- static/docs/user-guide/experiments/index.md | 10 +++++----- static/docs/user-guide/experiments/mixed.md | 2 +- static/docs/user-guide/experiments/tags.md | 2 +- 6 files changed, 10 insertions(+), 10 deletions(-) diff --git a/src/Documentation/sidebar.json b/src/Documentation/sidebar.json index d21fde9420..020f71688c 100644 --- a/src/Documentation/sidebar.json +++ b/src/Documentation/sidebar.json @@ -121,7 +121,7 @@ "slug": "managing-external-data" }, { - "label": "Manage Experiments", + "label": "Managing Experiments", "slug": "experiments", "source": "experiments/index.md", "children": [ diff --git a/static/docs/user-guide/experiments/branches.md b/static/docs/user-guide/experiments/branches.md index 2dd44aab22..bfb698832c 100644 --- a/static/docs/user-guide/experiments/branches.md +++ b/static/docs/user-guide/experiments/branches.md @@ -1,4 +1,4 @@ -# Manage Experiments By Branches +# How to Manage Experiments by Branches You can use a different Git branch for each experiment. diff --git a/static/docs/user-guide/experiments/dirs.md b/static/docs/user-guide/experiments/dirs.md index 238934b2b9..a72eaf3173 100644 --- a/static/docs/user-guide/experiments/dirs.md +++ b/static/docs/user-guide/experiments/dirs.md @@ -1,4 +1,4 @@ -# Manage Experiments By Directories +# How to Manage Experiments by Directories Using a separate directory for each experiment is the most intuitive solution for managing experiments and is the first thing that comes to mind. Most of diff --git a/static/docs/user-guide/experiments/index.md b/static/docs/user-guide/experiments/index.md index 370157294f..e96bb6fb3b 100644 --- a/static/docs/user-guide/experiments/index.md +++ b/static/docs/user-guide/experiments/index.md @@ -1,4 +1,4 @@ -# Manage Experiments +# Managing Experiments Data science process is inherently iterative and R&D like. Data scientist may try many different approaches, different hyper-parameter values, and "fail" many @@ -16,7 +16,7 @@ You can use several ways to manage experiments, which are described on this section. Which one is more suitable for you depends on your preferences and also on the kind and complexity of your project. -- [Manage Experiments by Tags](/doc/user-guide/experiments/tags) -- [Manage Experiments by Branches](/doc/user-guide/experiments/branches) -- [Manage Experiments by Directories](/doc/user-guide/experiments/dirs) -- [Manage Experiments by Tags and Branches and Directories](/doc/user-guide/experiments/mixed) +- [How to Manage Experiments by Tags](/doc/user-guide/experiments/tags) +- [How to Manage Experiments by Branches](/doc/user-guide/experiments/branches) +- [How to Manage Experiments by Directories](/doc/user-guide/experiments/dirs) +- [How to Manage Experiments by Several Methods](/doc/user-guide/experiments/mixed) diff --git a/static/docs/user-guide/experiments/mixed.md b/static/docs/user-guide/experiments/mixed.md index a8eb3f8aed..f8a811ec77 100644 --- a/static/docs/user-guide/experiments/mixed.md +++ b/static/docs/user-guide/experiments/mixed.md @@ -1,4 +1,4 @@ -# Manage Experiments by Tags, Branches and Dirs +# How to Manage Experiments by Several Methods On complex projects you can use a combination of the methods that we have seen so far, in order to manage experiments. diff --git a/static/docs/user-guide/experiments/tags.md b/static/docs/user-guide/experiments/tags.md index 6947f65fdd..d87d2ddb37 100644 --- a/static/docs/user-guide/experiments/tags.md +++ b/static/docs/user-guide/experiments/tags.md @@ -1,4 +1,4 @@ -# Manage Experiments by Tags +# How to Manage Experiments by Tags You can mark (or label) each experiment using Git tags. From f891f8a082bc157b802f978682949a0900715d0a Mon Sep 17 00:00:00 2001 From: Ivan Shcheklein Date: Thu, 5 Dec 2019 10:19:33 -0800 Subject: [PATCH 08/11] Update dirs.md --- static/docs/user-guide/experiments/dirs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/static/docs/user-guide/experiments/dirs.md b/static/docs/user-guide/experiments/dirs.md index a72eaf3173..f3714bd560 100644 --- a/static/docs/user-guide/experiments/dirs.md +++ b/static/docs/user-guide/experiments/dirs.md @@ -67,7 +67,7 @@ experiment and its metrics (results).
-### Tip: Use a script to create experiments +### 💡 Use a script to create experiments When we build a pipeline we have to use some long `dvc run` commands, with lots of options, to define stages. Doing all this manually is long and tedious and From ebdf3a8522946b61ac92ce59c81d4cc14db7c4f4 Mon Sep 17 00:00:00 2001 From: Ivan Shcheklein Date: Thu, 5 Dec 2019 10:25:14 -0800 Subject: [PATCH 09/11] Update tags.md --- static/docs/user-guide/experiments/tags.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/static/docs/user-guide/experiments/tags.md b/static/docs/user-guide/experiments/tags.md index d87d2ddb37..849b4c6443 100644 --- a/static/docs/user-guide/experiments/tags.md +++ b/static/docs/user-guide/experiments/tags.md @@ -37,9 +37,11 @@ $ git add . $ git commit -m 'Baseline experiment' $ dvc commit $ git tag baseline-experiment +``` + +Do some changes to run an experiment with bigrams and commit it: -# experiment with bigrams -... +```dvc $ git commit -am 'Evaluate bigrams model' $ dvc commit $ git tag bigrams-experiment -am 'Bigrams experiment' From 5ac45911e822e91df3bb437a12595305dc8a1ff0 Mon Sep 17 00:00:00 2001 From: Ivan Shcheklein Date: Thu, 5 Dec 2019 10:27:42 -0800 Subject: [PATCH 10/11] Update branches.md --- static/docs/user-guide/experiments/branches.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/static/docs/user-guide/experiments/branches.md b/static/docs/user-guide/experiments/branches.md index bfb698832c..0477f2be38 100644 --- a/static/docs/user-guide/experiments/branches.md +++ b/static/docs/user-guide/experiments/branches.md @@ -31,7 +31,7 @@ this: ```dvc $ git commit -am 'Evaluate' -$ dvc commit # just to make sure all the data are committed +$ dvc commit # just to make sure all the data is committed $ git checkout -b unigrams $ git checkout master ``` From c165839eec4fa6ffe6f6dabbdb52078226cd401e Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 17 Dec 2019 14:58:41 -0700 Subject: [PATCH 11/11] Update static/docs/user-guide/experiments/branches.md per https://github.com/iterative/dvc.org/pull/828#discussion_r359008195 Co-Authored-By: Mr. Outis --- static/docs/user-guide/experiments/branches.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/static/docs/user-guide/experiments/branches.md b/static/docs/user-guide/experiments/branches.md index 0477f2be38..c5722469c1 100644 --- a/static/docs/user-guide/experiments/branches.md +++ b/static/docs/user-guide/experiments/branches.md @@ -93,7 +93,7 @@ $ git checkout master $ dvc checkout ``` -In any case the command `dvc repro` should not have to re-run anything and +In any case, the command `dvc repro` should not have to re-run anything and should finish quickly, if all the data of the experiments have been committed properly.