From 2f939acefad0ec0c243c37869fc3d3bd136de713 Mon Sep 17 00:00:00 2001 From: Angelos Katharopoulos Date: Fri, 12 Dec 2025 04:36:46 -0800 Subject: [PATCH] Progress with the docs --- .../distributed/m3-ultra-mesh-broken.png | Bin 0 -> 16519 bytes .../src/_static/distributed/m3-ultra-mesh.png | Bin 0 -> 22069 bytes docs/src/usage/distributed.rst | 353 +++++++++++++----- 3 files changed, 261 insertions(+), 92 deletions(-) create mode 100644 docs/src/_static/distributed/m3-ultra-mesh-broken.png create mode 100644 docs/src/_static/distributed/m3-ultra-mesh.png diff --git a/docs/src/_static/distributed/m3-ultra-mesh-broken.png b/docs/src/_static/distributed/m3-ultra-mesh-broken.png new file mode 100644 index 0000000000000000000000000000000000000000..108ff58da45a6efe8d9326248ce9b2f73275c255 GIT binary patch literal 16519 zcmb_^cUaH={`Xr#q7)4pN_$IbNGd7qG?W$^ik3u?q=Ax_Hj&Uy8x0yLl?J7~w0CF- zMdNvx^&T;Fr}d_M2-dOe@d$9jG5=;^A_P_s}I1VN*ru3|tC z>!k5teyWZ56K-jjd-%sjYb`YuVvYRIld`lZg5V@HR1P2ai2wHGnnlRcU&YxDw`hTE z9a_<@ESr(yV7gdU6Mu#X=dJ=>;c+LYX{Vp2^>V7TRo-%H`VzJs_Af^Asv^HBO%F#K z1T|YdnEtKSKbH33zJpz#WXsy6jTdVe>JD)duuaa~}mu@B=BZQm>#Harq4 zBONxppVq|*f5`UenTx#FBHcZ)lhzUbneX2Et+zUVJ}27DYbkTLt(Z^fVI#^-n|2VZ zY05tD-n}zoj;fd)&Gq!Dvsh)}%I{_|-!QaE%jkv@r=`KYy*_Pk9hNc=mfsB>mJO|B zWvy{#x&5El>g$vZYmqI@cV$a+^{^S1)g7L#u6B*D9t{2e{DS_SYGXY;-zQHvMNYnE z@bmM#SW^2`ok!WjL-uIMPCxd8mp|A(aHnTw4W5`dXZO?SQL5OQ&)N#}Luuz5B_#)! zmzVX-%$_Z5rD6P0{cB;;aN+wGR<*mktKQw*w9mPFzlhmOb+H!j#o0Z(6}?0Hz5ix+ z-+NYEY$7JrH`Wj*xl_u4;b2{iHzrR-{9J)3zHWskD z@^`o`^8|HveT+?CrT(bDr#!OwNd;NpOZGx?v@Y4eRv-0si>f6N!5vxYiMcN zVVz&xC)H=y^yF}7k;`Lc$Ml0bCs#LZ+xTkIFgDkGG| z_j_%hi&Lg?wlaQW9kFra#tZK5{z{y_2?+@nskuppEBUlV=fB*zd-twI>Y8*Yide_NA^SWK667GHqx-%!o991x?kGg)WmL^S0Idni?BiJ#t0v z_3PIc+}u<=J>~FfW-GTTa`N$Qyz;&G-B4BV)6C4^*4E?ka_-c7PnEu{;qVFBXH)$_ z-sNRMf$i=SC+9{_o4&C7I504vH(7T1TlZ7*qVtOrNq74cBo8d5|LJ`eRFhjPBrYD_ z@bTkj!apFOIb@gQhpH-neBXZP{T;W}<*OGjU*=}I#1ST;H^_!cvrGEYD6cifD>!+k-Ug#8Ybv3Zc znnDSiDrQO2g(9hDrZ#A=PgitIOvKZtd;D+T?wGn%cZ#gKzR=LXbKklPD1EuOxp`+k zK9spjG(~W$@yukODtkcaojP?YrSXl|QsmgDM_Sksl)k-RzGM^^FFtZ-7BSJA4AxQE zf9==TBJceEhQDLmKD4wL&R6kqy}tHa>`s-ngtT-j4XdlZE@Xwk4wMSh-=R9q= zhw4u=)!gMsJEpLtlA5~0*Vj44qc8Ti^TB?53Sy_knZS`lj*gDR=)opox9`2Xh*|7% zhsXV8*JQ*jihb9u-)Q;w*Eg!oo3EZdDVjYsmr};RW&Qf~#M{;k!*#?XId}9BB{#SI zxaTc|y1F_+KlpRH!~NOQjKz7M5^NZ!zS3Q*tEA?Eq>47u{<{)CTTzOduPO|)i2^ zjGbM@=g;&?OP_i6?%kU)yldAkql2F#1NDrJ`3xUDe=d=cw*Gjr_ljI;SsDF~9fp$v zC#%*vUFwb9Eh1aG7HIrZMb?horajC_HxRtHJbj_dXLU*Kx#=gt01IhV;ZLbz0s`Ts zC-Ps-jjd8es2{hNk5Iqm>r0@o>q&fib!lAB#6+MgcTO66=7_87zM-KZB^4@K+UgXo zd*S!)-7YPa$1cdH-Ym-450l-s(G!9M0ipX zHx^mLq2d0%n=fC!{D2=r$39mXK$D!3qFNs-dhKkv);%$eFxtx#O@5bW`gBrKQY0G9 zx!M$k>hAft-`>W-M;NSi7rC6y)j}(1=jOhpo2yk*G zMn^NmdGEvpCDwc~Eso-UMRpuGaKOFJjQJVLDtPZ&(Qs(C?)JDDn;le?6HQN#lqsN) zuGBi3HhyhyYxB>^$<442Q9Ph>XaR$Gof<(3&*ZJ!I;j!yb>Kq|*Z1d25 z=F_Y3a+&%0xKTmLO=;G+XltCW`k@2~iL3Kx@L8U_tvt=c!(;i@cirso-!%)9 zEsSQtnUArE73lTH666mDGl`nitRql=ma_xz@(K%&9zPyE5^*!$fH$K`p^zNX7&s=y z7oJu)USFQ8xqtAA7`j(PTwKPir^T1e{>lqeZCbVuSO~YTo%HCk#a>Giy}iAhOu?pV z4&kkd`Pk%JM9p5leSc?1jKK`X#I5}NeM}-IPonXCDu3k=7O5cj5QjfMr*8)aQn4Mp ztSz{<>rz{WmyNK9$or>9Zj1Y@dd1D?pFSNB5HS1mr>cjCN6My=i_eBJCe^jH3~Q@5 z5*UbTdU|xfX8P|IICeY_YnjdSj&CUW`laOF!q1eTM6tSoV_ z#S4pn{|c8WoRGwpCq}nP8oRT~oPWPfaoM*e?bvY3F-2SUv_0wulTw9>tBW@YJ!NHz z7hCy{CGZdoyLQ!ztUqCDZ!d^Omif^v9NW>pB{kyCoy{r9$$4Qdbm_lX9%g0=pYJQZ z^Zva`>H*wdz>o39;a}f+&b4RlY7t?(Fj%pnv%8y1RFoz*Ha2Vc!w9-^UO_=s=84?! z@Ni!~odhJ6ES_Le%&joY_3O$)Hq-3>t99CpU4VeAid z=G%Pu^r<@jz$KHK&mKZOooE+NQc^aYJ$trYzp3=;L|qQM0gMo0g?Xc$P3W zCpI=VMnz4neBpvPZmvcy?5JUvj=r$4@K%=nR0;|TSxM@hJw3zi+2-n+nnMq_hnzWk zwz8pND;n2MMQ{0OS@)sQQU7f7BCb7q)?sp=WmnsgF_t<}70m4V_m{KR%Fm9hq~p)8 z%$yn-AGeuoPRUBzZ=(E^PEJm4r<|L>w1E4smEZkI#~v#HU#V(nY@U{7(coo$`1mmg z=DL4h*|is81{%O&1Jl!6PMG#fOh ztl@J!F^*3D{{B>3wp0RFnyN+Ap-jZY#O_NuGNCh6%jw=!CcAEk+4s|#&oJLQU%Lwz zI(6MZN6QaOay@>Yqvvyy+CIm25qZxAs?^lftYP&SwDjGFJfB>d>67(dmc<)9YHiI$ z-X(U8(2RtONuE-j3Gadzzj7Q7$r7uk5-L2JFs*m$T}mr3i6p>1tF1 z-L74CPW!CHxW{iiBEV45+RD)0-kz280T}ZwZZSMMI@w@T?S4N^sX}zYq25=Qgi3b( zNj;{x!mx!|?EQ@mTTtig+g(|11+~b(e0hAz(R}a8lFL=iDO!V*lXn0V+gccQTb#3*ufeul8H7nHgd49mQP9x2oLOu8%KE6=K@S z9y}=g?U%~Wwsd`NwgVR_nZ+#LetLA!wC3|UR3Mh@&e2mU=g*xBlyYo;i&2GHV8gD) zovv-8rKzcW@uCDUZJnHEHpvA5o=sj|$!MAcjNA=>Wnb_Duwka$UD}R5VWlw3BKXk~-*_r0Vi4)PWvCqSjB3ge+2$@b! zO;yy_Zvj{N@Zp0cyV|!5ZJRALH2yDMNK#U)n-(ZYO-fp?sHoUmc8wY9bcS7R_{^Cz zoKjLoNvy2o!zE!8b(UX0I4$6CjF+Vc+#(`2Hnsh(*L3eINpeqs4{!lpy0e8|{Ig)c zlZC|^Z8$4QAUZ6JmPyp?7AD>R;7`_Yx>U?LCy=>3EXOOYbO*gVR_3PK$^%>6p`r6fmiIbbV7Z(n_YQxQS%isT9ZLNl=%#nT`Ix2F`pL2Ay z0Wn%;pgTaKgloUDs$Bw%&vm~PEdMhl3=|M`n)CFw2G$<*j=7mPYu584wd58;BMRGn zi2A8sAa(@y^RLzAAIC;UN39knKI6teCaMO9$He%Dhtsv|w=(^xW}>I}BauDU6JP&0 z@b2auiqd9jeze$ic;mFJgN^`0BC5o}y{Pw(LrW?Ry3?)4KC5x=3O$y?|qARWvZsHrOj%7^ytRb z)xR#<_r?8exB53VH&^7oSq(xZ4QA$+TeGVTfj5#RtHKr0I(Yv4d2}~BgOX>!_r1lg zM{@plP*YRWF)-Z1{^b%EpUwVzc=XU-2g4GV{_?#1{6R-&_o|UWleM!wh0d*6rjju} z6VBJ;6|O%36q%SEtjtQ%Gd0~s!!8$1=wad&45DT1x2mAzD@fT)Ma2nXRaajhr4@j# zuU8YsUi;}&SV(B-Wh7)+y?O5*^XT~a4*jdc zQy{TY=X*^YhNFL%rt5s8wzaiw%{G_0=yr}`dg|&_nN*sFW@eh$ zf;{NiLLwq(BKHUi-a*M}VzX!EztVmr7xVnZi@4StO%W!;>Go_4DxHh<`UT&8R)26v z?BAcxsS}iP8oAZX+M)o7>yRwwdKTQnCBniN&L&IDAu9QA15_)8WI1J(d^u z#fq33WHyeCk5_hdFl)t%yvsRRGCS2KT#IeLKkHN(i*^3ajEs!VVpndj>(?_+=9*bp zgrFPRjDLD0wL2O81|*IiWFlvwySewZ`WohSJ6|yMvjA3JAQx{=6!J3+Wb&r;_JswQs#EE+n z5)x-v9@`WP{&}~Fqv^H#v`NfayVd2T)|d8%aleW}*$za8aVUyiWhc4ECX(AwC;^2< zx(3j&9)*|^U8(@&FKYT+W&U;Qn1pQ|LtSmHm4W#KborOft0%6G?BViUn&6AWO<`M zVR`l}|5AKROw5_(-d>C4g_dJ8UIW_R-U@Yfb(YrFiC8!WCZ?ggimTxkdj;>xGlHw$ z!0;cg-}~l$QNMTax1J($q~BM#{<>le_v)Akk%8O;001Th4vbJGxTmw6^6u|1EH0Ma z@7TT--NjZvK@c@}>(;F&R}a@NLIb;S=~CsL9sEL%w9*^8o8u2&p#=@txzGAm?|i8SC*2eDhQK>_ z{8CbQ6B848KC06HP;T#bk;J4Yi7>Q{DiF=S1Kvk!Mg|T8C}F8Kva|DDk9h@6f(yKK z-@bi73z@C13kwT`(iE@h-qYpL8M=UH`q=0XWn45@6BMpfKk*v(s-mG0&RMn8H)naO zM+FjJ`{m+jgc^&DjSZlm@#{{De_cE&H}@{q!Fp|V#RD+r(X{l=y1qqGEH5`VH=yj@ z=USQZ7>$@qOpowNyYu4X*-)mb0L~}6xBd9qCD~naWiLH__UShXgr2>9#+hOO#L3CY zHj@`q-Z@yqi*9ab3`+R0=aO+HA^C0e5uu?p&i&<&0;3P^ebyeUikggj_^<`5C;a0q zi@@mP@2;bDj3^*2X-SsnfO9r=(Sle|&jP8PAc9^$KFW|3!O7W~o{o;dfGsXAR!`Ux z5f!zb@L2g%Fmj12f(mR-Kuj!F@Q8{EB}(XiYb+l(_j+vH7~LP?YP!C2!JQx zAB=K)&t!;hDQ`F4Qg=QDs#+r6yiiG29iSGYskyl!5XK}nQ%nFO14B4q_Bu`iBc11f z95cu$6B84qFUB&-Ns}OCI`o$xTK5DWH$)>S@HFapfE#(%W@L;i_VW7B)RfmP`}5-^ zU{%kVvft{cYCi4zO22;ndI^ZG$*(Q!aO#7BxiO=o^UQ0uOh3cb$=x+ZyjuZu5H`4wd)^`x%fMu7pe(A|0Zc^={k{2rj4|0fhZ{}?8?O% zpS5~9+8y`yo<0g8=4Nq2jslC6!!4ILD@>qHhtHoEo0b*Q$O!!g=yCu z*|ybzE$nmZ6!%(P8ZY))Q<#?39lw6E%aN@y5r)U~XQLFPTW3m6*uIChrmM&JdIIvN{%83Fghh zO#f+9-Zo>jaC$~Ye@H$d9E!U)Y}k;+Eog!2#b_4CodD)k|DAcW4fppIG}b!*zB^k? zB7P-0nvzQy?JwHMAiKb!joQ`KmAqphHhp8`YPm>dl4$G{e^ULB;zF~iNgjcsBGsB{ z0U_0l(2qdvpgzXN8U-Csk8FyUzqb6vVHI@r7%xk9=h)r8bu$7KbV9QG z*Y`p>XtK+o_fditfC;yCmZYx&aJ7Apt*!;mY&!vx2Sk;FiLUVXt7OJ+so_2y40XKfYW%E}_vx&x=y@Gds|*SXYIk&n4r^%8wkW>fhb4Iu{m;uhbm)-VRIA3W z(ftpKie$>ay}VS z1SJLNJ~Py)x3@O$xB#WNJz`1~02BgYWXb47_wk8|z`Q*1j3OSFbJ5M6&CSi0Xn7$9 zG&iszp;cMg*xZB`IRHRx$DZ*CPu#yXW`~-Z8pN?d2;Nyq;%T_sgoID5596_(fN*Sp z444s=n4P9-&k9Jw1wdDtUt6ojah(#KyZiukUuWwBKfiT}NlBkmw3xTD%P|zW^y31? z*4EaR=gxh|<%ms!2f-~N5$V2ljG?ddRH=s_8g*YM!~3QtIsnXx=9Dd906E4Vfv2q@ zKQ(o9oMl(F3E8=i0wAbbac#L$j@ppg#mU~@o_NdpkYX1Li)MST!pednUg2%```tn4XdGbU!x$TQ z+ex$4v7j5MVe%^*G5#P?Y$VBQ_?{!jzb|bDQY*gvjm5=@x*1=jqu;(`hpC#?Q9-i6 zj(@f-77~)y`ionD?J?M>xORqJgG*YP38NoRhZo3QaQ+KRb#=8+nXvL_51STCblhqz z#Xh$Qy%4qo|J3BPI#DNc1mZ{IOl+nSyPR8id_)u7Ty^Ro#yH*f?KiMvESDGO0z*S> z**Bm46hHXzDRp}j>%%ZEeqEIhwY7tx*-X0?yaFLt)?R-NEU5zN)HuhY)-7!wZ4#O% zm8hsFz>9djJ}dAzNwTMR7BmALZ(L!c^p`HgUIg#kEM5DcM@uI;w8r5T`nZSv& zxIc)<%I-8w)wW_!GB$g8&X1(;`|j(Xv#I~|X`|liees;jfr0Lnw4+6L4AdY;(7etU zbmQo$zqfZ?v?71hu5(_ZALwp{A6`c8fE3< z{Sw{X-5{^aXAHv(pJN6opEyzUPT`pH)6i^n9$u;J+8$(FQppA%JYelauf zuvy<^+2}ncYb+rt$+|#u>MA@bZ~U5e_ZGVCfa3Qc?-tipBc_HCQ@f%FjBZd;_9vQCw1PjO30 zF;e&`S%rT4Pog@%>fDxENGWL2k#n-PzFzCI?F2SsYrc&dh!{6^KZOz$oM^f8R#tm- zKz%j7UD-@a%f~CiwE3!C?i6lr0XUkDnE&-FXMa|Dy1JN@16QsOGeJ#DYpkSPWm({rjnJg`$}K*zL#m zHGBV87C;$TCFYd5KG@m-v`X+@!0Ppi%d_i$eg7hSwc!+OQBvnYBith+Q`CHR`oSSo zBNQd*hf$@>@KxWwRZ>w=QSZz2)HvqlC0`xN%HOrd$igC2TWt4{1t!lPAt7pThnR+8 zbSzS4#Vx}#3(@{@>!*a(C?pRLm=!72#{Dxg1OqcOsjUG1oejNbFE~`h>-f!q;rYJh z0DzzH@^XbcDv2yL8X6i;=Z0&d^aPTjA>#qLw6wJ9K7Ql{)<#`$6K+HAH$RYd3=Am2?(WXRP0P7W&_#7WPyeP+i2o-cBg0QHV5CUeHE@!4EGPu1 zQZw$ZwyP@+cu8GN?e^ENUmfC&ou7}7k4J$cz*E!cy9VCxKGXLI_K)CISZ%q@3WaET zk-@=R);)nI2;x05GjmMTypV?=Id_@oPcDLigX0)BdNOLm@bYEps_N=UCC(@K%m;ZN zhrct=s2vLLiV6t@L8b!}4GA3iy>jaLna%jzcKXjy%#**58??G;#daI<2$klqER8>= z^u-!Fu${$xa$8<>t)0Gq|GwthTSj4HIwB|}BsxSASZ@jynlf4Xf7`Uh&9dmQPtk_? z3T*Z7-Q0vV%8qsIE4@nT8^(5k>#EDosWwXA`T6;g$m@{qVpv$|KLh>tcX#$R{IyG( zTUdzRFU^%$a%CpcM1&88+?{82^Gqo^yaA|Ne_Xc0T8>PHxs5p_PWQ=X)p@hs{WFqh zKQ&gqHly!|1*S%Taw&e!jq`&QI%4f}VZC2bq<80{0Ug5s>K zt$ciZnz`BPjN@BqcS)$sClnM&5thh4ME;h$@_j1-l1Y!bJ@x3MR_33Ct@bmui9_X9 zhH1JR2!OZEN=u(6#7@2@2%Av|2~o8SZqe}REUmyV^o z_TRz2gf)7!`G|4AL0NE@VC#5GI=U@ zY~Rk)@(r&Tz81rLPh>&ie$7{Bi|}?}z9!RVF6MtJc3uDY`Y+QIE0FrEACkshoIZYddzZ0bx$QaT$Yr4;cz+uW`KZbV-TZoczk?FQ>Y+*s{WD`9A9BO)ffy5A`WxXGVpOf@p_JoBrG0NiO&zsBYWNtaQ)s7K zGj_nx7(}arO`vH9Ig6>HvU2XDOn>XE^8g)e?Ck3ZL^XVsIMZKlPR;6`8vFPa?i@74 zd#!U=SUptobSRr1-gxNK(ojg2S=+umc9H9zDNa4=} zH;LfL7e>m4zxr^mb6>9X??FuD&!0ckf1PWg9#AWOSufz?R9jnX@L1E?+WLk|`O-!_ zWI%SdCMaZef<%*cX*y;J4q?2 z4SIdWVy8+2!3s6{z!bHr9l>3F=WY47-?=c>u#1P67u}p0N~Bc6(PZlSTmIPKM%|}* z@6Hs7F#PO!cCU!tEd1WWf*XYr9JXk=(kD-RL3B0~>o_qVYClKA#eANZ$4R_A_nGVU zl^K#%>Z8`6|DG<7hlQa7y_h(_!4es?Ha6fyLC#5gnu>@_za-KFcGD=1)?{7f~ISLUu5mahBF@+$Bs)`C7{bx`?(pyJRXaYeqXmdw#5iVk6 zVnQ8bB$9<}_&}%R*Z?9x+gcBX@%g!Jy+eahzfZ@*CwywE9EUtrDExeNA1j=kT>0bu z23RYz?V0Bp1AS~yhYqE=v}?!3Wn|JYw%@qN1E9Y4Uh#ZX?h7()U}!jrE^ly~pN~%> z^CW9sx$(>Lt}5$&UrMh=L8A{$U+(2K9kYU0)bQgE_{(tv1FowpV}cu8>Ur}v95BA@|-zK(J+nqg2vZX;x?>dosSC(n`^&UT` zVwAu8rGA21LT21Z!}|4Y<{DRbW{?lR^_vpG;j@y`8B(OB;SfQ^ICb&T;jrN;F-1iV z^q%|sY&L@4t7U}J5NH6dlg+#-y<+jh`%Upi-dpF_J+`U2(ahSDuAc;j_6C52K*|0Z zdj{+s*eiqCS6-b5fy1GXrgtoZoQT+NfXkH{fKv;f?KTwR_ z4h5kmQNAYQ5a7oBxHzMdj$J3hG;F59VK;8w=?ifwU!c*|0g1G&pv$m$B>OxSO9DbP zY}Wg3*uumqBC?fq0vY{=TMZCDTK)ZI%?h3u9#OH_H`56g3$sq*euR zJ<*xZji(Im^33|Txdv0w^5X-knBC9oVmRu?i=XY5_aGpEKrO;jv9!=EU_03$ zd8uPdf}*0TFp7AEDJ?>yui&Ey+EsRT?nH*nirs(sxEWqF0Q_9riPQOEiSzemq!F`1 zhQV^CuMFUB-Lx!Q+H>!vQD$rejd_nlId%=Kta{!R=OLmjO-a?X4M2N90LvGG3M{|F zyd?1wD_>i^wz#x32)Q9^_(FW$5^i>%J9Pqt&dNu}ldm)C>AG{eLOYa9!4SFHmOX#q35kBvqGQ3`_29J(mlscsHKy5TV1^oG|!~eW8bMV z&nh|Enti{AsTaQwYL2=w!?3fG2jePyiZ36F>LwS+o4 zNn(m!YK6C?uB&UuetY=7ppZ}n@K2uQJBppGtnmAckLTpCds}+-M-wb7c<-D3(b~dn zhR~Nh5Q{`$1nzFPer%MLe`=&7f?Gu&8JiypLO0(;u7mbogmXOB2Yl zR0G@&z|`(3a(NL}>8i=Nb7vsbAgh1k1nImn&Q8R@;NUvwG}y?AFJDHxvxON_b5(!` z4SxN)6(&i$Yip#Yj5IqtJ7OQ)2M+8)sDqnaF$PGHM_^KuOdj`TFp1iR##)UAosa$o z@)Sz`m;hG={$vo!6BYuFYT@y6JSG;D>(dDh7iCZH-M&=B$?GH}~4vTtF_BiCY5qkqkz4ZY7$iOlQ5 z>)RtN9E=Bu3WnJW{ItojJNcGMS;tgM(Gn)^0xe z+RchI@=v0oN$esg#{%r~9<1;q_D@?dA{xytElb=2qw1ot(%mMTH^3rX1Z~WB?vZYH zmCeybYs@?UMUJ%gA-TYvw@G)#ui>&bAY~@yUI$>pnA8jwoAp2(C94ad1J$M9`Wmly zezbmYYKj_|q8D?7e2Ojz9{F(R$kOcXjfW+l_zMLBe%VxV#RWZRBj^gb4%$fk=^Gf_ zu&s~9&$;eT%%7Z|hBUVwCcw@E7kMxVP1S~qi-0C7pm^n99MvpE3=Yu|w$Q6`w-B|1 zZ0O|VB#wy@f9M*M&V&6{?7>kL>K>Y?*w~{$#mSIZ4dJjLGZKqT+yObcI`;>VgejHg5zaZ`9FGb2K-}tv z#dMvSxiK&c#4|OdZ!T(oDBGiN)7~j$VrIq#j)~;lO*kGvTT{z#gwo{52q?CY4eCAe zH(WJC=TaoIL%;|%oK3HKcnl!x`##I`#R&b~r~r3IqiE>82B6Nh$mgnXk@pi;+R=IR z?rt3jkly1Sz(^^u*I%_|>o}V(qb!j`0G-wpYZ*?~Yz@4*shJc$Nh}PWkDzkZwPxK)G2V$aO{w8Vd09tIeCm74)YR7vC2V z6}3mWqu>wQ3H}FHW|ba54A}$SL04B-Kv2*R!4SnKPbgJYRRe>Ak2EG6BJDWhEh0Ki zNJ9eRQ4ok0tRpU8z8u+zH=t3STleV`t;^_TbiGh zKkR^%%@vVeJ?RYzIIpOv2F@Dq4BjmT1pDw(T^$`Gvu{_fBB3Dd+)a7q$`$Ym<1rwO zw0t;UyCm(l0$_$>7goG`w=Vbe8#{K5B%}RdXko`$jXy6;IUu4!W`#%GVd3Q) zz3iSr#%u$u1x(x4OfmkFjcvdwAHw%60yw*37lrIT!~(e}i`za6xJnr!IE=2C^HdL@ zIh)_)F;!?Fk!%?ufq|SDY!pgJekS6}`iM;yxb!CwO=z2wQ2!$9yS<++&WGElLuvGo6tWqZS{_RD>{Jsg@*h3!r%?IF@Y0X?ocEKtb~6eEw>+yiZw9v zT_G=Zc6R24CDUV2L6X?RON2jq6pB+5PoR>Di;JiKTu6|=N|A3O^3xzSIhn`A+`MDa zRJA<^oMSUKA>!I^JYD`=EQoEj3=3*c#%(}eCzDy^aDt2QPhQE!J|1q?n^RYM&b2)= z3a#vGZ)G@H!UvF133x!y%zR?}wZ#{y57(WHV22E$>`5$)O3uIb`+R$ZtW<1V+#S>z zd5{7u*ar3KFze{V=;r2@hm_4r2#RE24jUlPriON6V%GG1TNvzdM0u7Tx$YWz(=7zi z^E1Ra((wejB2T*jHk#H9wZ6x}4L_7Rr?@yBFcE71-O$5C10cN3U z)}fsrAx&J9cxpj7=*az(08+%FINY6$79Q4DWVeq3E`YLzB3U@pUEr9-?J8tFH&jK6 z!f?!%zU3XKba5&mJB!7uiyPqc<`oq9zj>oLtBA2;*so!}-HC-)~wNjKM)#}&zy*Aq}d%p@W@ z`n?=enhFBXpl?cOZU4dx#X=r7nYZb=j}G7t#U&+=fD7=9Ml4m5^ zU5l>_gz6lI(2A7jEks<&xN#LcF`mX3BdzsNeaNVEmU&TFe~`M)Y;~#<93OJv-ne;_ zo`od{SD6=PY7N1NAaj5oSFc(@xdiNy7z)+d40afC|9+rxwwVBW;2_QX z(_V8Ms_$~#MTGsH`^$~E3=KjKi&&Hk-+K>`83XL*pSX?GRJpqPB_!-7KLrABjgIB+x&446CK2HaxLvK`0F#;a z*s){qz^!o-O6pYq!3zUQ_etG>{R9Wa&@p~}!99ESzzeiyzj^HlGT1)x37#55j!r^B zYd%RVWPp{*f_?V-KPL+N{-;j){rATL0mS}KmQwG$%qs?ncH6eqKq(;`=PT_OIpE&D zI16@|XZya7|5P_kFPz@bfZl9}JQ>J(K!PI=a@i~Gzd*bs%JBaCO$KpYV-{lz3z2&) z&tdr8N3x5SUGDwk)i$Ns>h`xwi;HB%2Zn`J<3K}GlG--JJoM(pWB)+deT!dArv0(7 zaLYs_=L(G}8HE7a(afmI*x1*}QVa{~DQFHsfQ*CuDRa^(;f0H}bbQWOH3F@5{yCI$30&s-rChk6UoJ^x36+ndyF%}o? z40;OJQdW7ncZxltnE#F^dOG4noF%-diK!`aj1j(wB;Y|S-o7O;a*1b(TX_`}b^}#J zeeHlJN_H=R$ieY(q!}02L5D5AG)~L3_Y`@44}R7V+BV6L6;>B*z+;Z4E*bnQHp$J& zNx-(gHz^q(z^R)8G1SnRNz2lb6KSm-1T_s!3P4pjj2LJDEs)YDmzREweEq78w0m+= z(hudN6#X;~B9! z5a*5;teiz(u7Io{j$&m~^kzfw^st6Tt@$)yCIfoyvxQTT`ezYBD*bus=0+N2@cMti z@#5DPjRR9tLD;t%)vpj|{@zy>j7BilC>7(ZtrtZ$z zv=EFRM?S8Y8vTB>E}V@n-E&H{>MR&u@csKc|IGtYc50Yz1!+%mEnXQkJB_5IZrah) z**O4sL{17YrVT)c8a6qC<^~4a(MT{4c&{qrkOm3osTaO>?LeUV;>X4eroPT%O@JZz z9Ku};Qc!){DPa^peJM}M43I{U(fN$*Km2d4fVSvwC~R<~N+2%Y8AY5H;4;@9OM4e$554z~c9k8M+-ixZRy&TSUA#9&Uto%IG_3sPs#b-ko6NBae z1;B!5X9the$|6~O#LDWa_QMDNFDLOzT~C&J(1Tb-mNq_Cycg0@>}o^6-=)M)Lj1f5 zBs(%T#szO5M)W;*CRZ9fmjlXy7QBe?1vMnDROTm_O1<`7#t2DEq#^0XTabk&qCE zO-`!LWEKqK=idIfTKlc*BI0vA#zwe!cql*~H zDgcrZ4<1khRMqN#dVpr8%(K1hlKhk9*`fNA1D^v^I?-&wD{S9n^IW4)K40oynu||I>ra#XLFj z%vDW>8rNn7P7>h{n5YfN7F5kVsdtS})u;-lI){H#01x@^1sMAKe+|(Ig36Zt1`f@K bt*-cFJ$By_Ww@{&pC>d_byaedEpPlU7STE! literal 0 HcmV?d00001 diff --git a/docs/src/_static/distributed/m3-ultra-mesh.png b/docs/src/_static/distributed/m3-ultra-mesh.png new file mode 100644 index 0000000000000000000000000000000000000000..e049c024c45b2809af67771e7df0e2d0023168ae GIT binary patch literal 22069 zcmd741yEM)+AscKP>Q4|jg(4=ARt{LNQs0<2`U(XqI5_pB8`+F0wPE%t)v11N>~_# zpoD}S}uS*b}R5{>2& zH3JffOdS7Bp(Mw@i4I?1!~c+<(biBStrP#bU!5IGB5{*6)l`f;lE-^|EZ)wqD@``# zMrsL$oV`#I6+C=b)AmqXQ81vq>O1`y^>sRtEm$WeFcViaMewQbGx%4A>-cu>Hbnx4(S9RZyRc1Q+YlG59 zUHR>cfBO$g7i4-wIkIO@{qv`b$-(FCm-=da^33}!&p0?lerpJN>=5uM$X|3$X!kYc z=T)13^WVF7kCK-5d=L4FavMb+!!tb5l(7Q3XL@cnCP_Q;j4@Yq_S@UAH|5H6{(t!= z>3&7U&eNw)FARi<(y_8qo1T=eC@Eo}*v9_+=Z6RMOiX;SK2jRq-tuS8oVnBfV9+@$vDZv#*&eJNiElsx+SNDmlAfLShr?=h#?7XXlO$*GTzh zr7%*`+Z243E4#JHF|9YATnGj!wozdwY9+aba>YyRb!Tc1@|XO20krqMC(KKkNq(m{H5 z_VB`*C2B`U#|p1mLB;hIXUafTn{(%Qq@=!Y zl8%}A=F6}hgLMHEKfjcVraySFi#e>=u3Zr8c?Jvq`H3y#lhV>%3?Wy>KNoxLs1dcO z@DC5)$}WASztlA)B}L44#Ump-TRY(5i(vXIPEM*vk48!~TOL;1DdWU9_q%6{wY4>_ zW4yPTO*>w=Zm{Xrhw7{Dn>KI$)~d9ABPE5SzP>){?%mMfU`mTB7vlMuJU&MgFKqfT zr|%ZukwuZYwjHTv53lVt%<6w|OK}6~iCsJOQ}>C+dkR+OcS_$D|cF3=4O44ksF3vCFZ-I%kDcFUG9 zRzImr-F(xjX*Ich!SuXWriL_ilB8%@SXhdRilXlQ$ja%J9&YSxt}qLai1^mu&z)j+ zH!-oUwY9KOYISMW&_s!hjBH_fqL!YMbDM%fHS3sr-qpdljkMwgN4vVZ?x1Lp?%Q`3 zP;T9t=Pbv|`@n2>83X;phYwkgXXDRzpAPWtvwRkeW#iemub@%is3kZrt=F>0*tpNQ zKaLg;G9*2nhtzH*855<$Wo4=NFfT8!_~hBM{Hm&|ll^rYN2GQiO;*;_3~@R6aM#JF zZih`xSr`}?l-1}$)Q)IrsrvdVPEAkiA3uKl{ACVV7t%2fqinrYbz9rqCw=Ez?`f9X z_f$!exW&a8Zr{FrL`UZZPIq)tQrMd}I$vvimP-}1)Z;}vf3$6N)S@DVsHLl4V&me{ zlai9!B>^jT$3Bl5ib0($dy`fGrz~eM-iyr>9qilXdp|dBdk}zmDu- zih1-%sIs#1!!yt6oQ#YN{nD_ou*}l8Z{A$!EI2vyr->^zB!o)D>`Cy)PoHexCChB` z_nQB$8yy|3rsgu%AtHX@052mWBdP-1ufEz;tOO-VH(5G*pH+=;zSyO1iH{#Y@*FsD zp!=EU0o04oCr_SS_>g|M@R#V`y&>4!+N7MT^S_@s4yr%Q$+^(hD)4P&L=zQX*NZm$ znC1o^#a?4|6s>=1e#})7o_dstv`3G&I6F_Dt6qO~Td3&C6Sc99{OH8QkbNhsj@(l1 z$WBk+fR!uk*_NZIS$^qD8Hy0e|LfPUth;Al#oB~&aB>zzd0E8rPfLn#!fPCGkxw% zs@t#b)1R_*#rrNFIYN0ubyp;pQbAN|K+qkhL6s~uw~1aFb93{dA3u0yWLP>nI`Yiz z#5g!OMribGY+uFgy(3}EAg`d{xVq$;IFV{7BHtXRY-!1fiY#W`xOHK1@!P=5u%gn^ zvpw-Tl!5iKm&v~xV6DECT?)8;dpnBoEfo92&VC~bTJc8g?o-y*0qbk?wbE|mJFj2A zKKkrX#lx7+;H{*YnHf3`4jK^=5f2ZKwyIizY#EOzF#}`co1W9(>(K7DZQFK4TRY&v z1KzPRo1+>U8vA&~L*{;c;gywTJyBp$H#72SQP)F~iB_EacV8_niXlRNysf^zz6(>YMT^~jnccW?8b z;p>F45H%j7zutlpLH+1$aYKMkq{G*WYiWmq27dQEJ))^ucY})2q&(wFXtJ5s zoy5dlf`ZiN-zC?DZWm%QOR^52NKCeP=0T69fK$w2U|{e*TaT+lL@e!RZl>D$+G;Tn zg7^CBJPHzZVe4Mvf))mEDNXiXov3SUBo7G*@feIaXj1-SYZ!gv3-pG`mffkRI`T}t z|5&~K!yQ|kld~tB?cg>>0UhOw7lp&a!|~a;^v-6>+38j#?H%i>3RfHRXph!-M712V zvEdWYy?Z0^!1;msK0l2|#`*X#`=i{<8^`wCIKroj!nv}tvXHoL7fO*BPV_>+yLvqJ zVpIjnO&dBy_HK?h)TTleAy9-yBzs+z>)4&S`W9?Gmf*<9#<;zPL>s->RkGR4Y^77M z&Pz=|%;o1X@738(lM)-M+NlE80lz`BpKZB@cf`+9dtbL1q@z7kQcVm z#}XPE+CMrvH>0}atN~D#c890bMp}(hW=mWedl^2 zKN>dDvyd*>+MY&9-y$at@n0k9-KwdPbyoW!_fpe#(hFTVWKDcF4{y;p=)o!nt+qlV_yOLXQC|;t) zVC5tkX-m|B`zHVVQNc15U-}}|@hgYvEN8T49!>&^@yqe9$EM}=e?Le`O|7b>MXh@1 zkmUDQ1tsa@yyB$&`}Zqfxw0QHy2!qB9~SKG^MG+ladGh???tCPlVa6N4tF0Nc~ymr zq~}%df846ru(CMqIMJ(s&+77V`+7MCU>_?C4G(|AuR`q?@AFevCkMLS?UR?88Q2oL zyHmEVqqUV5AmA`Aa=YbIe&3~$-2imCYSMAHZ=ddeehqN<(3vwl>sdEPkB~dgQq$6c zk0|2=$D7koP*7~(UL71Cx3Eu&iD7VZbv%a?(gzv2x7 zU#=z4?A%Y%PS|%?S(!xaK~4p)B8L$BtsVLg^VTNT*KXl^4$*Qc#+gSr4mN&3X5GI}GB0;{Ss%Ei_;dp~)AsF}R!ZaDUUakyYRP@vgn^x%VAto*QJI0p#sZlEi*GSK8BFdo^x;Rh%wRA@A6^avEu}<@P-W= zC`pG8AJ!~)baBZ5O7Ezwy>n-$xTGXMiHVELYOqu@R^S6_MPgfg;QRN-0CHj$eYfvY zQc9I|9aVdLzV%!Wb?5#X>`_Vkj_0plQ3KN8HgDe!ym4dG=kD$StOenDwQDK?_J_ZJ|9-{U`H+@Y=uG9$pBE~ss>E}NuW#one)UT0_OnNE<>dzn zQquMK{2&m4lT1RCy-fSkZi9Dd2LS~qpPs=(n04S?d~MRXd69a{7IP!NuN96(B_%P6 zk)fdv>1{&uM1*2$7ao@Nua{W)E^fy2G^zDf7&ET$n&e(?LI*6qI`1HCUZx`(niB?a z0j@@m8a6fMG%_tCBl8*XEX<4GgYxq7Bob&K9pxrThsj4d-6edBCk2=1Mc6)qj|rQW zY!W(A@B(lSd%XP`MVFHA${!7v@z3;F8v849z^Mz%%e?&jn^9+pqKwVu?d{z^JiM9o zQEuj$#~;xdw|#}e-q64G9py?-=C7NDOk-}LcKK_Q_`$*(;Uu*riKrJrJEZ+WV&z{MnuPU;}^18A`llkZ7?yXsw(k42Jsi|q+ zmv8sKskrq&C!?bWel;ymFSAs9Y?b1@G?Gow-SR%)*HzoMv2xm^rKMeQbGunwEQZU8 zmpv*%rVK#+IzhB4Uc~$gAa=*E3-S>(B@k`~UxYAzWYm;dJl2$(`GV-bb6+kGCS9&M z-4ojQ=`;72^2=OiAZfpMC^9nm5`NJurKwSD{ky(2GH zv$8Q(N2J{m+XXn|8n(HWkhKnPz&HS%G{8&1iOPPPLn#_I%Cl!<4T+SZL^VaDJfDw^iFqvkOpGr|0Gxzg(`3;PP9Eb`@gP<_!=A@dm#a^h{Ga{@`8M zDnoa7x5<+W^iXkf84mIoYq0LHdHM3CwCgC7)zAq#*Hkkt6uph-&YhF|&G>3>WaDO2 zs8_FEzZP@&D&3KPVbe8b(xpq6N;#v++D}ZZjz4ZKLNf=Sc)8ZL8N(eRwQ$op+{JsB0OKN0fr1~aP&<8P}B^Pwsw1*Ev zUFBF0Ke#^LU25+!^OGJ;4v#52Lji&LS0d^{ZJrO4s%g9}DcDbkJ`cr_x2<^_!Q>r#u8?kHT9I+`q9 zrx$gcJAz7_r#{y(yZ`6UX8fWDnugk>ol3s)Z{EHwwr<*nFWnkJRZ_dQv;{JOeDlut zPv{fN%A~UNQX^C3-0h6~A7_ZExQ({$-O0w*(9sc(b=UZ$jt7>zQ!iEVCA3T2Hob&E zb4+Y30YCr3z{7(-QCGi?P@>ZARPd5e%GOcAN8a622s_XP_Iq<1$17ZK#@_RDwXfd3r9trz4h}xm6WaYLm>ad7okTo{ zaJdgUcV&4FNJ!iW3)=!k%A`Df&aP*A6|xvqhch3-zC!RKU4Hh598B?D)$Mbh^F8h# zGPQSfT#`u2ifr~D6>2_uSAq&_stP!O4P0io+0?**K|t@`oAe!(hV7}^d<>^QW@zSp zxge!0w>ofk-@bi-YtQi;{)}X&mN;l!YL_eJ<5|<7TFcp}n_{-1y*$F%&F%Y`jZ+pq zL|XH^@}-+d?(Xi%@}3MR?Rn<<=JL(bm7j*_S&k`GGf@XhyZofX*#sjw`dh*?Vk-$K zCjg9eawL=e5BK2E(NEd&%Cj@#yLORn-nI9J z2;PbU5Eu}!G1c-F)j$6x@9xv5a%Zfq{|vlDwa($-v`PMIyL6`Z28wNHSXgva)N>rN zhJf%`ImObvj!NhATRCLo^hAgKDOh&9d)qB4{r>%%xJ+6%7AUMO99+?>ttb|M#=8#S zwqo$?0{?z8V%!fr<)-iI(w~vcaN^=4^+fEQ+})c{N6}KSS)?F?J2h$8NP2pB^#2)b zLLnfoo&4|J|JkQSGtybee8XfckI*m8bXnOpc{y%jLDc708*>~i0>%a<>|Nma5ESoWRl zCj%h80leEz9bHl)p4gVt+WzRo2cWTw-A|}+h=jnh_Fw$GA1Jb=v_L$jv&}%5*?MVa z1UyM2W&W-ApMlLz&dwide69v|`lg`G)_?t)tS5T*hG*2J-e)WX_XasBDJgkhXr*+q zujZscf?RP?k$-u)drF~*RgFb~Mg%S@^2G}^qTQ8W?rR)siFF+L$Pbii<{)|d_A_-G zB}m8tCr(C$gj|EL^=3>eYi@q zdiyf%b`n9;I5|1R^S^xVj`;TbcjU~Bi;ay0h_llqgNHy-Gvl}HYUleGe6OH z3)m9`QY)1ah^W{7`x};(F{|~Mg4cmTLE2X>Ko22kURjtty)xNA`?1y=&2G6Hz0qf)rYwFDrt|rd~%jjRsL04| zGIDaAPYVH(VsU0eUJi72i0s*OAG9wD0_04k#w|{Lm`AzTg&%)FQl%p0pL`l09DFoY zQ5WC2$$uM{q7FWk2@zJmDUySVw3&i}ph8rn6dC86-`^%_pjTvo)ES`lQEl3=^!w=q z>*{he$_UTKjT>dJ^ee~qQuwR!=*N!jjRm@i!9|_5v3U>qPxVv|?pu5`G+G;{W zWnyR7K|9^#4{FlV&=6o@VL?gCx2P;>T*D$B`Sa(Gn%WGw5Cch2L?pjh7jplv&&7w9 zOrlGZ>RVfH0XcPy&pWHB#kCD^^YSb$jTKV;Wg)2qeIdIKV5^;K4&X#~A7_hoLR45d zL&o`gR85T{*?sJfW2UBE_G4hO*^<}8gMap|uek@P%sJJHQhc8pex0eJuIBQqTaqE{ zrOm~QI*?&v96Dh_ob#FF;o-UP^L<(b^+-bQp{QS9%5Rb0!z0+k!=t+7jk|~_qNJqE znU!(+hYV?P?G_OD2$Gn2YivvGznDkG5&~h7|U)w0=I@(4) zW@p=ekEL_g!W#b7M$)rq&qyR=V`I|89pU-;`D7%V!j1Hji}fC`mI%uQqG!**NU&f` zo}mCA-}8bT&4sSNS+N>?4+|lXFC9&mQID z$G5}FB9Qaq?v57dT^rd2WUyqoZe57XAPf^s97i5T_D16_=8_iK7QJM@GVy zELG(GPn;12-~r_&Gi$Fcs0pZs_AAhP#8DmvyhRJAKBD~muP z;JF#C3cf2lm1m=WmR!r)NW9-jB zx%&xC!@`0c^glWwf#RC-G3$|fP`h2FNn^uo3-B=>6W1m#v#RQ}^Jzte$x^b-(!!Ki zjLn^WT?O!UckSBcG1-63B?->teZm(pYy#I!2Q8$cqT=QO8R6sS2jAyfTwFx8c=-5W zORX1i3nQPh_07x#f6aq1fBXf8Ktl=+4_AZ8$-v0yxHxro==*mVB8+HP1?A`qr)-uU10RlF)(ng8`2|HQqn*!)#J82gh z!}{+mK!4pP|E!}aTDG>fE-MR;C66ETl7`^Iz-P!DAHQe^1(_;P6(Se^!Gj0y9=yzX zJ1$P*p_`*)@sqN|M3!IOrD2L|3)bRdVrhrNnDxE9WZxzp2qGPWaK0C8H#pd#r|K{^ zq0N~ys<_t&ApEhQKVe?-kKSrWoK=CXoD&i(dfHwekF$V?)>X$q8P?AI} zpWVSd+5cK@=M4gMGs5w+e~LbZbqc}uiYh2@$mdeu+mRh2yXYv163JX$3;FM3Q-p%EnL%(14YMaKrQggIbCNwNHBO@jPlUH%vZdiF5pS%<+ zJU4A0BRL*Gs|YzhZzfow(}RG;vEj{-ctLMyuuG+VmL$fEjkXckl8;whx4Ov2+SJ74 zJ}x0RH1teQ=&Nr8noTv6>Lt7iDEx^h`ArRxKp=1{TPp+sV8+3O=MFpsWd4(Tc94cu zg7RdQiwepSzM1-8`1;SL4Pf74pE{dOGzWoth7NlaAF za}od_B6+dH;j{nxT}kREPV7MS9scxMp#NXjlmeS_ZDEjhgvPt#=3gS9t-S?%##^mp z6Icj054(a_oU4%FH42t&x~YotOj)Rb9O~ga>+~8f@Zo}0pLt)_E^rjkx#N!|lvoOF zx|`SgdSU!axn8u-6+8G&K@_Hp5nM`~Vay`+DB^@?-tlWssaHU_6=O&&f7-F;?0DTa zh*8);-23)LQgqq3-8CZ^lUhccAF2=;DK|G)*Q+o6aM(K-lWr1BfFKIRZl3FxR&e zWVFf5%!F0*wu)*rYS*D-$BzdBX||*tC=gpX<2U_meQo(d`@`)}f^A(KNvbPi(8{4Gi}NDvGzwhghfOaXG+Kf z=?Qse0YDY@?w4_&tt{f3$jQm29S3fVvktOGh|ZP2K@Y-)Rvrl$2X;e7W&;p`*%Pj` zV-T;@C%LU}r22jkhY_Ix5gGL41@10mMy2e0D<`1>K2VRa`%}Zt&OX`1<@e;--hvif z@>#+w_&GB2tvQ+(tF5KQWrJha049-^pWid!BO@!zBP6t?v(PF*<-CRAq-06_2ZUH~ zE46ug3URW{28M=?v!lloeI}gW-&ff$CH0+)oP43JWSIEkWNF9iebXh|v^Z?Q*)*Vq z5tJx#91b&)^a-NnLinblanSH}{8b!=$)Og;mDx@!T!<`wgo2jy229{h8?L>X8yOh5 z9=lsF9S!;1@{r}NTg_$#e(P)d#l*giJ##(Q&HdDORd#J{`7&JAW|vxG5rAi4%-U30 z6`aN~aYD}|@+`C^vU@H?fgm)zd9!t*X1PFT{!7{+3SzktF%s_)C3I_1Ht02kCAe_h z+A8gdirZwrO2yNsuZJ!SKf>9jV`JM=Y$81E1^@R9gyHA2eCM)`w?OXm~M|}JzNG(XgEnMSchr_vh&PG9MulxMjL|}JaM<(he z@c{^D!zF$NQi?pWG{8Xm57hs-mYl$6Up6*k=`_89>^u=WQT6bU2Fe*W4p`=@Q5TTI zWfKb5x=)ST$n`u#wxDEOfBg8d#Te38yax{^a8(=8DVD}@z-NX;I{>{Ei_Hp<0@D-e znxNR&Sc|b|vmNF;6}_cXEd?o4&0a;8mP$g`f03Oas$az&77_6hXz4UK(7c(Yr6sDT zD#9LL?QLaou_84`jblGE%j^D;TzvX*9~ z5)aM@ZFaA4v#iTc5~3LwKNqR&nw4O&8c)avAx=m0@tc_9(%NnV@<3R!`yqY>wkIDr zzm3>-zrV@^wzsLtH`_eIsv+k!#L%@e(OGgoojISxEVqwR~DTEJ3E-X)da4hI1j7L z(PmPWz}HC6$qCks(oK=wj?@;kC!-5JXr(s-1HXld)+k@Na3NsFo9XH429yVQBbzyG zN)8g`t=ell=%jdG`70kOO--C@16YxX?N&vn$XtJaO|*RGkrw5Jde`>&yq=Srn*ah= z5XZm2edPS*-v*nxF<7!1&_=91jjl74>J7?J1cP+uA6-v<%a=(dE+;2l|3#s)yn?Myh=&pVIP?Rmop%QC%LkzsYyU@9_=m_!lUnesta#a z_s;%nk(>%z=wCM0)`ze!|Izd&G05k?~%9}``QO(vZbdd<;5At z{7#%yTaarVyVnL#qQIgnJKn27LElYs!BgV-__;O9Nk;^q&;~s^+e1!I-&5n0bVK!Y z!1NlFy<=)>H)b3OmbtzQ!7Z@o^z@7`(z3D!t@jk{$Ft#ie1DU82n)K&A9U%My_uZt zM?SB$6&WnG3q=17%8K(fPQfBq}P3gt*Lzj67=5PKgUqGp~Q3`;d+eJ~9TwFMYRY z`zeb?3OP6;$3U3Q0Ga!t59r`1D9`S%{ts`?o-of!Lqn-6Eov+bSy)-?VZ8vZH$9!G zc~qRL3snvJ3#fzD_k%39A(v|w$%%$@!pbKQo3q_R1mu83M@Q!mYo&8>t*Rywto7Ir z*69D4ZJI182_qs?P*8xSzdOW}=$)c7M^J1CR`xUJV;9f(LA>3%bt|Gd4OQFs5;mb( zDK(Boo@;CnOn)*G0#im~J|dMTvS5jzG1^%aO9=`-Yz#UaU3&J{Fg?p|bKE?F5S>3; z1*W;Rj~;C-I^B@L<#P7hj~_u~8_D}W=_%zqo-6(j*Un@xAqjj8=(a59oST~zzVzRg z9Y~IW$0g6GmGxm>@vbo>H30{2MRLg5jh~P8jLOW&IE9U$_a)~rpk3|C)E2O`036ei zX_%gq^Ya==3gGML7#Obi_n%r=Sa^wOx@6Dj~Vf=Z#`;Wl0B z0q5S{Jq&`70a!oc4C`cQY)qR(NkR}&^Hv9Bw;1dtvimp+1UW&jA{<*nopb@h-VC0w z{(qLQQ!{J8u>lvbpM4PAJr$LltV8+<2goV^LCA|UVx^b9#G}e(AJw%*Z6=Iu2#{EYHny zrg?Ip>*C5fmzU%wo_*?H9yT@SR}|L3Le;^>D56o_z`l-F(j#I3EX8PBc3W5#z*G25WM9J00DiV90K=>z#d<4p-bx-80{;<`l%)iM)=Lj0{a~dKC z#Kqs|n;#rG!DiF~ycm7&UL+A|Ll&ULhunyUE4sxBm*F`7`(($Z{dE`B;t^3KQXGPP zXZpOSX^_5yHF$Gq$Obe+0qg2_xq6oQFGZpe0L+cWfyVEdSVO)aTVv0 zlm~7ob_k&2v=O+}F|hzw07em+5&_6j;u7_Nf+%vgNf zw%XVjLU|@5-IMpMuV8$MUje>+4b-6fB%M=H;8zb3eu4BhrBY$P`%%iw*HS;^}^uYf8 z{VG|Mz&oV-`Up*;ro&UDq1?8u5t0g=iXdbQiXF616ZZ!I=n=K|y3Jbe*CfWAjqo{; zQlf`&ije1}KQubWk)a$?5jgLymcVXJh1J<2besC8K zB=;2Z9ur(xw!E|3%(F9tH^L(1IJ$sTE zJ%9#&h>zhL@OM#Jneo#JyQ$pF6)0D>LmV|sinW8hB{va2ria{Y0&EG?vLE3%A}0VI zaHhvBjGi9Pt>|)}Vn_ayJIHL@q~&^76=8;zq2!9Lz6U)EaE8b&dr!6O{@L^NAS^1= za{dEoCN@2)VV+)Ig2N$iQ8m2naw)NcL9C8o?^d9Az_h0(fu?iwtOfny!nSBk4PUYy z4c7UlVyv&97E|tT{qp&95-?8QmwuyQ)XSdh^?T);A(sRn`EXX|C@-87RXx31u75Yo zr?xcFroF1@XAm(V8(e{+b2I`Unc_A)qJmkM-3QN}J&VhTx=#F!16UBw3_ZS$Ry;B` z7FpK_`Rb5c#1BDotbg(1#=pL|vr_|fCH=KivO0DTHxP9L`o;E>@q-c@5SA5_lpI!R zbabi@&{2UN8w5IEa}@s^m5>mGzER?E2q!TS9+0Kwy>xBoWHan+kEtQ5H*em6{aw+? ziYG11#0=Vu!_OodG7iSyf3I#sA}*{HoOc8M-{Gk6W{E##zu9-|(Q?ZBD|5eZzS<}# z-wcNzRfM}*zc3xon)dOzw-4t9b|PQv@fTdb%J?6~a0DM~iyk*N9)Mv$nCpn(*_5uPYYx8ROdtKq9d>u|D z-*5uc5D5-|uW!G8MWDIAw&}wKK1b;xrbo;U@*tmsVV$Ms=^qB25diGQud1j?M6y*Gn`C}O)`*lQ>lS%u&z(z$1D5yY)OF1$Q!_J2&yV%LFf%co!f`aIxUx&?;^$+7 zZj(gT2kBwh(nOFRPOdfn-u0*{v#4b_B1vjbOn7mchO!ZKPc?9{34ULH(tjl?)+jDB3UAR+lqdF)qLb!3W}a zoZ|fG#5d##7dmU#_)pfxd`o>`w3}t3z#h``Z<_+NYzyXZKDJQ#+)Wy`~B=?RT zJ2ogIHkMbMimpKTfb{e}2qtdwN08C}A|orSmYm!8DTAkCq^Ki&c5P+2<~D>THYk7l z_DuqXV(QPIUC7DcyY7R_eMAEc9zdt2Fmr90gNh1S#7?p6 zh=0LG@tO%nh13SZj8jZVP1Pe^z){3h1(K4)camJ-+Y;AL2<)X>Qqd2_W9U8oGva!K9UPkD;@boQ7`xi zZH9OCn3Te;aQtl=s%#zoqSXak(7bdU46BDrV@0Dj`Rv6IDgjA z5AH&r7#SO5B$=TGql;CS{mxGHHREj#4qjWG@H@EEMs$AJsYnaz0q6$??rMs6a%|2ObP+(O1A+suY`4UsR8U816=r>B`S1mS-8P;eyS z(somGOTL3DW##1wit|uzNhD0sWwg$Ywr^3Mr5F7#ElSJ$oUzJA%(hXGJUl(gNSJ~l z8F9rGmz9zKcVkd?sc%;+<4%HZqsB>2jcJksK4{X`2j+FgC4ZPJ$8OBGtKQqZv8sHl)OF*6&O{MK+s)>ROR ztRVQb?VsKLZA<#y@akmJ2;X@PGiyX#0m-qy(^c)gcX~&cD~-Bt3!NZ_1|pti^P{?v zP9YNg@q_kBKEFDU#(&Jg{9<>ZgUHuKupNwwj~DRyjfzitcq>M;fPrrN_CWZ^2XgbN zK`=0+<&QKc$v-jCG@tHwXJKmn5)~!o^FF`z2oO9%eMJHP*m?Kkmq$tog5U?YBP%57 zKXKz6_8rsLCrdkKGUkdQNhGRx_wT2JkQ;LoWOUm$c4?BI-+E_XU0qK3I{R!w#9hE83(fav}<{>#Bc9Were!Sh& z>*foDk%QALk}z#m702MWyKpKBJEw~K&>Pr ziuV6-9fv1|%N%-Gm1oz>L7Vsl1O{OJgEYc`6L3I==Dd{%HenZ@4dIJI@xYO47~Ove zT5;mtyC1<|M|-mVNvl$k$hiMcy|wB8@Yb3$+zjti{cmi?|9@MJW=qc1I$e=w7 z6R~-{an;!=v!GxfSW?pL8oJAKq+THs5V;v7U#7bItPUUE44r~)_UgNq78R&anXefl zDqlEj`|YD@F*0%**k?}!gD_Eus7qbdcH@}Dd<0+N=_7YTQ*83Je;1m^m)FB#UrClv zl?7Y7UukjE-3MC{@$v(s&b-BTwXE{ko;>I3C;MV@bJVO04EsK&YMY`Tal$RP=^;0K z)6q z1Ot(1w21LL3Jxy(86?8s1y|B@<9M1C7y38;938Ehdtnkexb_1B&-d1H%rx#NZUUW z?**`;;uks_g@n_67z$WF3h>CrDvqP-*lR%;oH<0LVwBF#@zK7>*xN7;(vZB`DRpt@m~mXF0XYi^l^1SW7+2j;MFJGG`j;;= z>&(B~@?_GKl=pgVA%fwhcH1G?nUps;zUe`%4|hskNfz-?zngobKT+I4x{zTz(?i`P zh75%wvbffOGXHYSh;%CfSrKbZ=rLW6XE*ieWZR*UfUP8Kc?3K;8d_38fXGt2D`THGa23+hK8r0yut|G+`-P>Y^uI}I_TBDKvLVr3gXFQ^>$uTK zuo)nen2}lm)F$9IrXC!f=HYGj17NSnH5H~AsF0JeiyI-@7h`A!&FZm3(BKdv9VdxV zdhF-Fj`2QZf1O`ny!1Dl?MtS9^wRbnJBYEwnwlCctd}5*4Hz#WD9;=bhseEj zCEm)Q;pr)prISc()a)4ImwCnW|?AmXM7$`Bt*B${~t_wVG8Lpt*JDBPw&DjNV1 zq=!fffex17nrs9_L}>7=iM%$l1lP4FSV%;W6pgA0;o3znmTalAi(MNKwDSFJ!kgsqLO90`$FxXu6-T6fF}xyT{IahvweL|{w2>m}8^#$4ExBg-w;a2yOR%dPdnXT& z;-0;GiNR9bzIYoCZa--mYBM9UH3XB#GcD*Mej7+UVoaFf5?5Z82&35W2w4!BWW5fI|{Lajr z<0y`pU_e$5lHoQab}+bR@j$_IY7@2{D(sK272OOs6>=j#u4#gcLInH+pNUu|*0UI8 zL6!Z{$ze}kx)m!40fm@yL3HdCP9PS%g8aCV5y9?=r9}P+XaDwt4)?_sqK3eUC5jE= zTtx7FEMB_cD1Ij~6_11=);-1y>rT)&f^9=sLmJv2v?9S(J9#UrLO*EV$0o!sU~4+% zhtVx9E}jNGxzJZrU2lJ|5mOu>MP$y-&WW?x`bRbz85_p{>=3{LP!AUp!OswKq=QQa zX{P^afbOWA)J8tQ1x#jX_(jjNj~p^Q8Z?_)X~t|px0Jio!>X+CJ~vZ zVKdkm0bb8p?I!n+dzbrWR4Q|B%mOhf006{OEyz#W=JAy(*qJ6^-1-~Z4Gi~dF#nOM zeqF^Mas2<-w5zCUfpbIz{AWNbh^>K)h1L0#e^_-#MOFghvW>Tt)-b=k1-1F$RW5c3 zdo_VShy;v&%BG$D`5^$32KITxE|%#3>kO(vRUrs35v`pa`4oiCjj4n8sN?qSAS0RJ zphvibO6JDoeOK;aVEY2BS6yGx6Nh4;qS6C@GDBmGh6|4D-yKYCs4&<--%SH|7?~y} zJnDJ4x&2AVoW!}X!B~fWe?Daynd@CR^Ba`mT}xUfVcuN9Z_R>8yIP$e`ItG=$aeXd zrlw|WZ&Q0ZOW3L_g5kWFOo)UQSl8Rj2?=-Qeff31zd253==+;Vh?1X1#sX)TjI?v- zP7*0OB_(qLtUuaeT@d*dq7Z`T)nOlsoqMyB)Na@ef_LZ69TEwuACvL2E1Z{NyiY~g z^4U%piV zT6AQn)an(e|M@>~E#^;JF}qm@q`o8#-%UNL6uI-$rEJ7>Ji`0lD}S0+8hOOi|Dd$O z1Uqu%h(DAJgb3O*{7fseK6P}cLo0*wm)SZzG;|7yJCW`Qh&RM3BsIT&9iYtZPb8%Y z?iASOn}UPFvwJs%mG_^+VRlaI4FTg^m%5)2`7O+tP$Qa(mxqMAYTLc;>*LzHckh)e zSCsK?f$ISQsMH&AL`ct3p2$d|md~PUvXPR6db~$SC;*}w>Y(G#_q&2$o*_L4c_C)0 zi0L8{9Se);SR6S(PGFnE5rAIbg>MunL65t3zjtyyg-N+C4BSRz$Pl+4>bt(`j&b4b zE&8u0zuwo|(`012?#J%OIg}HXl@@Vf|K7{5_EO%><;XC8^}!)t@=8nG zO<`T*hNMkx%zQ_k-_vM+8rjdoaO;Jxj?K1^8`lmy-5OBNstg!8^1j9-J;>SDZ(&ep z^9{O+ivHDwR_B7G&UgL%oj<$_#-?lCSe6FA(lYGWasGo{X*`xoX@x?tduz|g4%#zP zNg_}lm?PA!LxO{k$-cYqxc+AC=1p>KoS|FUT*!R2oRMIa^DfNG8^5+ZH(V)aEuJWh zaj{|yOwImGPB%Q{zVtHSf&?o9pDZZgfEF(=NR+_+yLf4Ma8wtG8o!)ueQ6mP*QH)e zjbifiJfL;Xqep{?Hz_C<{=BntPVqVNea~a{6k9+I7lA5)sBJJCtkuLIXd{)>Rq4DN zI86LyHf{4Mza4}OiTLDRjE&>H30Q1F4$YinPEO)@-@qh8H#8n2oRwILUemlRwl@yz4a94UoQ4k-G&9s!Zvouz-B!f zM*>NdA^7VATmCug#mdSW7s*ea1MuJs08cdM%m-WT3~UshoOwx&b0_1sCJ!X+WFC6~ zFAKSj(O!fOyj(vpez?cfb!c*O@~;N`_H9IhrB?`oNtwJMWc3%NreT{MaQ?0Xo#BB? zvWi~l_%0b~>8Qj+O+@+8aiLATU%DtH=HjzBoB%(QamS9%2gBc9`0*a@*n(MJVw?i} z2BRRv>k6Q4;wqU?M(UyREDXi!*;E}?isl(RTv)p%i%KIcElqr`vW@-v-K3;H^YiDS z-Q5TZ65=@D9K|z6OkY90W?*I}8t%ZQ z*BW&kLC`n%6p|p)Wm<Kp^B(4!|`5Ka+gBy$C8PhS|`{{3@=D^ZrXVO}Egw}-=)_<{4R5jP>7V0x<($zP0- zyl8AZQ*06!DX2gT973(I(^`V{b|in* zH<*Hus{Qdp*z=hEHj@1qwfk@uH-r*Q)S;pdV(bCk`LW-+@8Z}+lD(asYGJ1{ELY#P z#q-GI-;U&q;$fpB-fRPM0Orq!q$kFv&|UIfgdJ0tmb|d8wsv+rqN2A@C~=xrl=ha8 zfiT{~HpDH6CF*#1c_Fx04_ytn;6FNQXX){6^P6)AFoJxl{st9}mU}c0+bPJ(d-v=) zg*}gW=HTz&@9>b~A|*4nGc$`L1r0FN+}hgI)nz(zWI)ltW@8i%3dTN&*>fCM7)GMLF`=3T&Q#RJssf-*+--sW&6C2A2B+oA6bOXHBxnEdoRzN#smgr++;jk>|?yvz)&{+=MCZak2Vxm&#^{CL=X%g(?fs;FEBfs>yo#g z_v8r^Fao^zJeRnAs~B=2UekeON@#+WZ(>_p+tproxF)^NJgGAo=RcVYJ@q~Q{(^*- z2AwB!E3$k?eJiJ?2Ul7l8abP34oZ0rg<6Is8Y7a@-V9A++#|lQ=QFdzCrhJ_u>DWkIK3XHqyXOdV$0%rYT2D4+rQ9 zFrr5i+vW4;&z3Uh^iJZf59r!D05vXqpDFW1jASCcXJ*u59KKRg# z^U>VW@(sLt{#hrIlR`g!ZhPGI21V>O4ixt5ukz{8nq7wly1Smbi^9Exa}$ecHb%tB z@X9RW{Z?E{pK($!((b;{iwd%IDKJO0X5l(s5aKJ7@bjvHO?rxMl9U*<213uq2#@lE z4@fmnTFTVqUs}%Of&PT=zFf#&D}+Xl*ATdv<*54D{E1c>#V+>-Z$q(h9)3k6i*XhR z#|bf!_^K*dgq>LskAwjvKljqF==-Xw!XsTGfetK$( zcpby@6!952NWkRSj$=}|o@9lq@yMnSbk4HyJoGhTi;5_WLdV>@*E{?Jk8|U5Jo2Zw zyfCINZb`egJtUN={KB`ua21B^m+tw@nUY4w#=iWj#k(3X26+c_AP^{zmm5=-7u&A$ zwtukiIeq>-#d8c(l03dAp2v|zc(4RmjD#p-k%{mMb}X&Dv7sR&@v2?9OObD@Pz{vyze$31R>yyVQcdC@lSa@Jpb~OgP>_PZoaQuPa{aNdU3LEa0nG zRX@zz%H(KeUuS(9>8>urw7JyfULEA9g8cHSXQRYQR327VvQ#)f(oZUT-So66w%eiA z_RBjPaYqEN!_^^A_@eYiQ;LR<-Y7)7s+q?*9U< zVB6Qd58)9w^ggqFdllVj%ngYM67(McN@?E}X-u#HA?TZ%zlOv8beTRxMGZ&j6)OL! z{Un$-hjn#nK>0d64Pk!X$Gbd;6u{zlw*GfI7)iBbWn)`JP~Q`?bEbH|5|LX%bQJF( zaBfNK}4pNq*n_Cv@1NJhkrx$nwp2uizw%2?E)5H74GaWt8 z)Aej6ShtcNdU|cV=2Wo>(5^M!<5*EqF>_%ZJxkzdaxfTsX=$mnoSb=W;P@NhLUh@a z4ixSJBgh{z=NoVNx@bK0TeI3H_ZNelj7+7~yWLeT293>Xa@bFB&J^0#z$`la^zO-5 znlIayn%RrFX8tmGqU(42&dxhS9}WKBbQ$z6&+uEJ7$Y|rq1>>aH3C;G#Bu2BVa~i9 zhLrq0d~uLig=mf0rHAT5?RKsAQGSqN;r$=`+W9Tb!I^ZbZ|&A-EduU5cl*k-8}ET7 OX&yeRR-k(7`hNo&ocYZF literal 0 HcmV?d00001 diff --git a/docs/src/usage/distributed.rst b/docs/src/usage/distributed.rst index 0b83709e2..73c40f8b9 100644 --- a/docs/src/usage/distributed.rst +++ b/docs/src/usage/distributed.rst @@ -7,22 +7,29 @@ Distributed Communication MLX supports distributed communication operations that allow the computational cost of training or inference to be shared across many physical machines. At the -moment we support three different communication backends: +moment we support several different communication backends introduced below. + +.. list-table:: + :widths: 20 80 + :header-rows: 1 + + * - Backend + - Description + * - :ref:`MPI ` + - A full featured and mature distributed communications library. + * - :ref:`RING ` + - Ring all reduce and all gather over TCP sockets. Always available and + usually faster than MPI. + * - :ref:`JACCL ` + - Low latency communication with RDMA over thunderbolt. Necessary for + things like tensor parallelism. + * - :ref:`NCCL ` + - The backend of choice for CUDA environments. -* `MPI `_ a - full-featured and mature distributed communications library -* A **ring** backend of our own that uses native TCP sockets. It should be - faster for thunderbolt connections, but it also works over Ethernet. -* `nccl `_, for use in CUDA environments. The list of all currently supported operations and their documentation can be seen in the :ref:`API docs`. -.. note:: - Some operations may not be supported or not as fast as they should be. - We are adding more and tuning the ones we have as we are figuring out the - best way to do distributed computing on Macs using MLX. - Getting Started --------------- @@ -85,7 +92,7 @@ Selecting Backend ^^^^^^^^^^^^^^^^^ You can select the backend you want to use when calling :func:`init` by passing -one of ``{'any', 'ring', 'mpi', 'nccl'}``. When passing ``any``, MLX will try all +one of ``{'any', 'ring', 'jaccl', 'mpi', 'nccl'}``. When passing ``any``, MLX will try all available backends. If they all fail then a singleton group is created. .. note:: @@ -192,16 +199,247 @@ almost identical to the example above: loss = step(model, x, y) mx.eval(loss, model.parameters()) +.. _ring_section: + +Getting Started with Ring +------------------------- + +The ring backend does not depend on any third party library so it is always +available. It uses TCP sockets so the nodes need to be reachable via a network. +As the name suggests the nodes are connected in a ring which means that rank 1 +can only communicate with rank 0 and rank 2, rank 2 only with rank 1 and rank 3 +and so on and so forth. As a result :func:`send` and :func:`recv` with +arbitrary sender and receiver is not supported in the ring backend. + +Defining a Ring +^^^^^^^^^^^^^^^ + +The easiest way to define and use a ring is via a JSON hostfile and the +``mlx.launch`` :doc:`helper script `. For each node one +defines a hostname to ssh into to run commands on this node and one or more IPs +that this node will listen to for connections. + +For example the hostfile below defines a 4 node ring. ``hostname1`` will be +rank 0, ``hostname2`` rank 1 etc. + +.. code:: json + + [ + {"ssh": "hostname1", "ips": ["123.123.123.1"]}, + {"ssh": "hostname2", "ips": ["123.123.123.2"]}, + {"ssh": "hostname3", "ips": ["123.123.123.3"]}, + {"ssh": "hostname4", "ips": ["123.123.123.4"]} + ] + +Running ``mlx.launch --hostfile ring-4.json my_script.py`` will ssh into each +node, run the script which will listen for connections in each of the provided +IPs. Specifically, ``hostname1`` will connect to ``123.123.123.2`` and accept a +connection from ``123.123.123.4`` and so on and so forth. + +Thunderbolt Ring +^^^^^^^^^^^^^^^^ + +Although the ring backend can have benefits over MPI even for Ethernet, its +main purpose is to use Thunderbolt rings for higher bandwidth communication. +Setting up such thunderbolt rings can be done manually, but is a relatively +tedious process. To simplify this, we provide the utility ``mlx.distributed_config``. + +To use ``mlx.distributed_config`` your computers need to be accessible by ssh via +Ethernet or Wi-Fi. Subsequently, connect them via thunderbolt cables and then call the +utility as follows: + +.. code:: shell + + mlx.distributed_config --verbose --hosts host1,host2,host3,host4 --backend ring + +By default the script will attempt to discover the thunderbolt ring and provide +you with the commands to configure each node as well as the ``hostfile.json`` +to use with ``mlx.launch``. If password-less ``sudo`` is available on the nodes +then ``--auto-setup`` can be used to configure them automatically. + +If you want to go through the process manually, the steps are as follows: + +* Disable the thunderbolt bridge interface +* For the cable connecting rank ``i`` to rank ``i + 1`` find the interfaces + corresponding to that cable in nodes ``i`` and ``i + 1``. +* Set up a unique subnetwork connecting the two nodes for the corresponding + interfaces. For instance if the cable corresponds to ``en2`` on node ``i`` + and ``en2`` also on node ``i + 1`` then we may assign IPs ``192.168.0.1`` and + ``192.168.0.2`` respectively to the two nodes. For more details you can see + the commands prepared by the utility script. + +.. _jaccl_section: + +Getting Started with RDMA over Thunderbolt +------------------------------------------ + +Starting from version 26.2 RDMA over thunderbolt is available in MacOS and +enables low-latency communication between Macs with thunderbolt 5. MLX provides +the JACCL backend that uses this functionality to achieve communication latency +an order of magnitude lower than the ring backend. + +.. note:: + + The name JACCL (pronounced Jackal) stands for *Jack and Angelos' Collective + Communication Library* and it is an obvious pun to Nvidia's NCCL but also + tribute to *Jack Beasley* who led the development of RDMA over Thunderbolt + at Apple. + +Enabling RDMA +^^^^^^^^^^^^^ + +Until the feature matures, enabling RDMA over thunderbolt is slightly more +involved and **cannot** be done remotely even with sudo. In fact it has to be +done in macOS recovery: + +1. `Start your computer in recovery `_. +2. Open the Terminal by going to Utilities -> Terminal. +3. Run ``rdma_ctl enable``. +4. Reboot. + +To verify that you have successfully enabled Thunderbolt RDMA you can run +``ibv_devices`` which should produce something like the following for an M3 Ultra. + +.. code-block:: bash + + ~ % ibv_devices + device node GUID + ------ ---------------- + rdma_en2 8096a9d9edbaac05 + rdma_en3 8196a9d9edbaac05 + rdma_en5 8396a9d9edbaac05 + rdma_en4 8296a9d9edbaac05 + rdma_en6 8496a9d9edbaac05 + rdma_en7 8596a9d9edbaac05 + +Defining a Mesh +^^^^^^^^^^^^^^^ + +The JACCL backend supports only fully connected topologies. Namely, there needs +to be a thunderbolt cable connecting all pairs of Macs directly. For example in +the following topology visualizations the left one is valid because there is a +connection from any node to any other node, while for the one on the right M3 +Ultra 1 is not connected to M3 Ultra 2. + +.. raw:: html + +
+
+ M3 Ultra thunderbolt mesh +

Fully connected mesh of four M3 Ultra.

+
+
+ M3 Ultra broken thunderbolt mesh +

Not a valid mesh (M3 Ultra 1 is not connected to M3 Ultra 2).

+
+
+ +Similar to the ring backend, the easiest way to use JACCL with MLX is to write +a JSON hostfile that will be used by ``mlx.launch``. The hostfile needs to contain + +- Hostnames to use for launching scripts via ssh +- An IP for rank 0 that is reachable by all nodes +- A list of rdma devices that connect each node to each other node + +The following JSON defines the valid 4-node mesh from the image above. + +.. code-block:: json + + [ + { + "ssh": "m3-ultra-1", + "ips": ["123.123.123.1"], + "rdma": [null, "rdma_en5", "rdma_en4", "rdma_en3"] + }, + { + "ssh": "m3-ultra-2", + "ips": [], + "rdma": ["rdma_en5", null, "rdma_en3", "rdma_en4"] + }, + { + "ssh": "m3-ultra-3", + "ips": [], + "rdma": ["rdma_en4", "rdma_en3", null, "rdma_en5"] + }, + { + "ssh": "m3-ultra-4", + "ips": [], + "rdma": ["rdma_en3", "rdma_en4", "rdma_en5", null] + } + ] + +Even though TCP/IP is not used when communicating with Thunderbolt RDMA, +disabling the thunderbolt bridge is still required as well as setting up +isolated local networks for each thunderbolt connection. + +All of the above can be done instead via ``mlx.distributed_config``. The helper +script will + +- ssh into each node +- extract the thunderbolt connectivity +- check for a valid mesh +- provide the commands to configure each node (or run them if sudo is available) +- generate the hostfile to be used with ``mlx.launch`` + +Putting it All Together +^^^^^^^^^^^^^^^^^^^^^^ + +For example to launch a distributed MLX script that uses JACCL is fairly simple +if the nodes are reachable via ssh and have password-less sudo. + +First, connect all the thunderbolt cables. Then we can verify the connections +by using the ``mlx.distributed_config`` script to visualize them. + +.. code-block:: bash + + mlx.distributed_config --verbose \ + --hosts m3-ultra-1,m3-ultra-2,m3-ultra-3,m3-ultra-4 \ + --over thunderbolt --dot | dot -Tpng | open -f -a Preview + +After making sure that everything looks right we can auto-configure the nodes +and save the hostfile to ``m3-ultra-jaccl.json`` by running: + +.. code-block:: bash + + mlx.distributed_config --verbose \ + --hosts m3-ultra-1,m3-ultra-2,m3-ultra-3,m3-ultra-4 \ + --over thunderbolt --backend jaccl \ + --auto-setup --output m3-ultra-jaccl.json + +And now we are ready to run a distributed MLX script such as distributed inference +of a gigantic model using MLX-LM. + +.. code-block:: bash + + mlx.launch --verbose --backend jaccl --hostfile m3-ultra-jaccl.json \ + --env MLX_METAL_FAST_SYNCH=1 -- \ # <--- important + /path/to/remote/python -m mlx_lm chat --model mlx-community/DeepSeek-V3.2-8bit --shard + +.. note:: + + Defining the environment variable ``MLX_METAL_FAST_SYNCH=1`` enables a + different, faster way of synchronizing between the GPU and the CPU. It is + not specific to the JACCL backend and can be used in all cases where the CPU + and GPU need to collaborate for some computation and is pretty critical for + low-latency communication since the communication is done by the CPU. + +.. _nccl_section: + +Getting Started with NCCL +------------------------- + +.. _mpi_section: Getting Started with MPI ------------------------ -MLX already comes with the ability to "talk" to MPI if it is installed on the -machine. Launching distributed MLX programs that use MPI can be done with -``mpirun`` as expected. However, in the following examples we will be using -``mlx.launch --backend mpi`` which takes care of some nuisances such as setting -absolute paths for the ``mpirun`` executable and the ``libmpi.dyld`` shared -library. +MLX already comes with the ability to "talk" to `MPI +`_ if it is installed +on the machine. Launching distributed MLX programs that use MPI can be done +with ``mpirun`` as expected. However, in the following examples we will be +using ``mlx.launch --backend mpi`` which takes care of some nuisances such as +setting absolute paths for the ``mpirun`` executable and the ``libmpi.dyld`` +shared library. The simplest possible usage is the following which, assuming the minimal example in the beginning of this page, should result in: @@ -269,78 +507,9 @@ Force MPI to use the most performant network interface by setting ``--mca btl_tcp_if_include `` where ```` should be the interface you want to use. -Getting Started with Ring +Distributed Without ``mlx.launch`` +---------------------------------- + + +Using the helper scripts ------------------------- - -The ring backend does not depend on any third party library so it is always -available. It uses TCP sockets so the nodes need to be reachable via a network. -As the name suggests the nodes are connected in a ring which means that rank 1 -can only communicate with rank 0 and rank 2, rank 2 only with rank 1 and rank 3 -and so on and so forth. As a result :func:`send` and :func:`recv` with -arbitrary sender and receiver is not supported in the ring backend. - -Defining a Ring -^^^^^^^^^^^^^^^ - -The easiest way to define and use a ring is via a JSON hostfile and the -``mlx.launch`` :doc:`helper script `. For each node one -defines a hostname to ssh into to run commands on this node and one or more IPs -that this node will listen to for connections. - -For example the hostfile below defines a 4 node ring. ``hostname1`` will be -rank 0, ``hostname2`` rank 1 etc. - -.. code:: json - - [ - {"ssh": "hostname1", "ips": ["123.123.123.1"]}, - {"ssh": "hostname2", "ips": ["123.123.123.2"]}, - {"ssh": "hostname3", "ips": ["123.123.123.3"]}, - {"ssh": "hostname4", "ips": ["123.123.123.4"]} - ] - -Running ``mlx.launch --hostfile ring-4.json my_script.py`` will ssh into each -node, run the script which will listen for connections in each of the provided -IPs. Specifically, ``hostname1`` will connect to ``123.123.123.2`` and accept a -connection from ``123.123.123.4`` and so on and so forth. - -Thunderbolt Ring -^^^^^^^^^^^^^^^^ - -Although the ring backend can have benefits over MPI even for Ethernet, its -main purpose is to use Thunderbolt rings for higher bandwidth communication. -Setting up such thunderbolt rings can be done manually, but is a relatively -tedious process. To simplify this, we provide the utility ``mlx.distributed_config``. - -To use ``mlx.distributed_config`` your computers need to be accessible by ssh via -Ethernet or Wi-Fi. Subsequently, connect them via thunderbolt cables and then call the -utility as follows: - -.. code:: shell - - mlx.distributed_config --verbose --hosts host1,host2,host3,host4 - -By default the script will attempt to discover the thunderbolt ring and provide -you with the commands to configure each node as well as the ``hostfile.json`` -to use with ``mlx.launch``. If password-less ``sudo`` is available on the nodes -then ``--auto-setup`` can be used to configure them automatically. - -To validate your connection without configuring anything -``mlx.distributed_config`` can also plot the ring using DOT format. - -.. code:: shell - - mlx.distributed_config --verbose --hosts host1,host2,host3,host4 --dot >ring.dot - dot -Tpng ring.dot >ring.png - open ring.png - -If you want to go through the process manually, the steps are as follows: - -* Disable the thunderbolt bridge interface -* For the cable connecting rank ``i`` to rank ``i + 1`` find the interfaces - corresponding to that cable in nodes ``i`` and ``i + 1``. -* Set up a unique subnetwork connecting the two nodes for the corresponding - interfaces. For instance if the cable corresponds to ``en2`` on node ``i`` - and ``en2`` also on node ``i + 1`` then we may assign IPs ``192.168.0.1`` and - ``192.168.0.2`` respectively to the two nodes. For more details you can see - the commands prepared by the utility script.