From bfcbfafc25297d5d73420fff454c7e20e44403b0 Mon Sep 17 00:00:00 2001 From: MDA2AV Date: Fri, 24 Jul 2026 20:03:10 +0100 Subject: [PATCH 1/2] Stop publishing runs whose responses were mostly errors rps counts a 500 exactly like a 200, and nothing between the load generator and the board ever looked at the status counts. They are parsed by every tool adapter, written into results/, carried through rebuild_site_data.py and listed in gen_new_leaderboard_data.py's BASE_FIELDS -- and then read by nobody. A framework that collapsed under load was ranked on the rate at which it produced failures. Twenty-one published rows were not measurements: php-fpm baseline-h2-1024 294,864 rps 99.2% 5xx symfony-spawn-franken baseline-h2-256 28,243 rps 72.9% 5xx symfony-spawn-franken static-h2-1024 52,582 rps 56.5% 5xx symfony-spawn-franken static-h2-256 63,518 rps 47.3% 5xx symfony-spawn-franken baseline-h2-1024 97,817 rps 36.5% 5xx phoenix-bandit upload-{32,256} ~700 rps 33.3% 4xx rage api-16-1024 9,825 rps 15.2% rage api-4-256 13,999 rps 8.8% apache/caddy/pingora pipelined-{512,4096} 0 rps 100.0% 4xx traefik pipelined-{512,4096} 0 rps 100.0% 5xx fasthttp crud-4096 0 rps 0 responses effinitive static-h2-1024 0 rps 0 responses trillium-tuned static-h2-{256,1024} 0 rps 0 responses php-fpm is the clearest case: 294k rps of 500s placed it mid-field on baseline-h2, ahead of frameworks answering every request correctly. The four zero rows are a different lie. Their logs show why: fasthttp crashed (a V backtrace in site/static/logs/crud/4096/fasthttp.log, and 14,506,257 reconnects as every connect was refused), while effinitive and trillium-tuned bound their listeners and then returned nothing at all while burning 1486-6187% CPU and 4.3-5.1 GiB. Publishing rps=0 states "we measured zero throughput", which ranks them last on a test that never ran. Absent is the honest value, and the composite already reads absent as 0. Two changes plus the purge: rebuild_site_data.py grows failure_reason() and applies it in merge_results() to the merged list, not just the incoming batch, so a bad row already in site/data is dropped on the next rebuild even when its framework isn't re-run. This matters because site/data is generated: hand -deleting these rows would have lasted exactly until the next --save, which would have restored them from the raw files still on the box. The threshold is 5% non-2xx/3xx, overridable with HTTPARENA_MAX_ERROR_PCT. benchmark.sh warns at save time when a run fails the same rule. The row and the container log are still written -- they are the post-mortem evidence -- but a clean-looking --save should not imply a result that will never reach the board. Composite impact is small, since these rows were mostly earning little: rage -24.4 (api-4 was 20.6 of it), phoenix-bandit -9.4, symfony-spawn -franken -5.6, php-fpm -1.0, against a ~2600 ceiling. The zero rows score 0 either way. The eight pipelined proxy rows change nothing at all: pipelined is reference-only, and #1017 removes those frameworks outright. The 1-5% band is deliberately left published -- roadrunner, sark, vanilla-epoll, swerver and others sit there, and at 16k connections a handful of failures is plausible rather than diagnostic. Verified by rebuilding a synthetic results tree through merge_results: 9.1% dropped, 4.8% kept, and a pre-existing failed row purged without being re-run. --- .../rebuild_site_data.cpython-312.pyc | Bin 0 -> 16484 bytes scripts/benchmark.sh | 16 ++++ scripts/rebuild_site_data.py | 37 +++++++++ site/content/docs/scoring/composite-score.md | 6 ++ site/data/api-16-1024.json | 26 ------ site/data/api-4-256.json | 26 ------ site/data/baseline-h2-1024.json | 39 --------- site/data/baseline-h2-256.json | 19 ----- site/data/crud-4096.json | 19 ----- site/data/pipelined-4096.json | 76 ------------------ site/data/pipelined-512.json | 76 ------------------ site/data/static-h2-1024.json | 57 ------------- site/data/static-h2-256.json | 38 --------- site/data/upload-256.json | 20 ----- site/data/upload-32.json | 20 ----- 15 files changed, 59 insertions(+), 416 deletions(-) create mode 100644 scripts/__pycache__/rebuild_site_data.cpython-312.pyc diff --git a/scripts/__pycache__/rebuild_site_data.cpython-312.pyc b/scripts/__pycache__/rebuild_site_data.cpython-312.pyc new file mode 100644 index 0000000000000000000000000000000000000000..8fc8a48c95503e5219c2f9bfa167adfed10627d7 GIT binary patch literal 16484 zcmd6OZBQInl3-PL_4oHD-GHD#fYgW&BMI9=9~J_vPe5O`R`STW=_;V1>26mQLZI$3 zGwZ!Uwr<_>%&pMcKA~O58J)Ff_RhB_Z%o|EyW<;q#I5a-RVP%^U423=-8CmP(p`H(3wJ7{8`huD zlZ1j`4Xk?5BR5sJ50(_la*0tf>h~4dr}xS1{e+n@F`8GY6BfqIXyI;UEQ}8BHh9*< z-OgAU1Kb^~lQ9mhCkRdptvcS%5haN+<(SfE8`%nk^@rKskzk0S`JliyGJe6|Slckj zN5XU;7a68GmLCZTeB-t#7wHRz*d2}A0+Dc-hkGCAA7)1*+|Z63K78gI^!j5QP~uzk z^b(SJTt16S)AawC_!&Jg*2@JMde|SOBYpH4Cdfxa{;{4g5N^Rou8w9{KEMT|LNF4p zqZ=9;>gZE`F6a*n{4??RLv7?)tk1ESYYH10X1RWrp`$EUpX*3Hma!|jF4BIE^N;Zl zQ4EZ592*w$EX!Q`IcC()vGfMI`v?FoG7@I!V7MRZ=gF)1Lc4K*0>B)}GGRKj`T=e7O5?&%w5%BH0|sDYB%GDu7CY zAn#?MaDj*sgG;|kD1Pypz98<^O zDW<^p;|AbMGY>(+_lDRy8V0K?5@zXOpEN2ssX+DUj#Df*Mu#GPhVEy>Eaw*h zF_>3|#!#@gQHVrBd?V10MmT{7b@K}&JRJZH75F+DhTM;hO5{cdB0NiPJbik@rqicU zJI7frLia}m8b)&fhrHJx7^3|G?T2y4S>&e!oSz@4gTjD6+%GZemuPWppQN@pq1XUw zLIZs;+a)9v1Xj2xzx@e`A(p2rM_HJYetra0RccqFA`HM(>K}-X2$gm8=s++qK*P)! zf++w~D!+u#AwMtBFbQDduuu{PfW%>x(64GhUl1R?(+vjAif@$*05T0PM}=@|NI#M6U3sG&|BW{p7kcS zP1{nswbSf-!OOu}<28S(W@l2jb7EHpzWdJl5(lO$Qu?atk@vZ5)E7@3N$P3*s6Th|mZmskcO?P|?^OMqW&NCXea2w9 z2h9_K{A7R!9sDbCf;T|_e`2{n1c|sp$WL+szGbh$rL}VJ1Y91ZD+Y^zH02Z$R3dyo zM`iiFojCcAI4yW;G3D~sXVmik5Dsj~=w-(n6)iRYyOCQNNUe(6*?Rh#eJ>x|F^Y{k*`i457 z=m)Jn31b;o$JC5V>Z2ihA9@O=9eK_YQ=eCyvr|Oe7*`AH=r+*D2hYBBto7N9KkEEXIqPA2-L$gBuEXa%qX))yeM)_n4L` zzFHzvlldYEp-9GIN@M1?h<6l)Z)2LZgn+AybX5^t9rUh%PsNmP5sWIPhFX-6w`vEA zQqnna2RgO`43srma7Y{am)Lx=qIOR~lLQr1YN*P|&1 zJ0KWoy=YK?y$QTTuB zi>!eFRye(VJH3>-I+_iE=>UcyY`^|c6wDS@2nPHidGjrZULX^h=wJ$zetD_8Z{y%j73>d%c4aO1Vl9(hCAz1iAvy< zD+6`oXau;daJXWK)Nq7hAF?@dqk$!uyk~?0+MEoD6Z#0@Q*?1vP`W|@QzXmd{z7xO zT0hWk3;Bn88UK!PpS&R#y4iyY*sv`W0U_`^ppZp90u_1mGAD@Hm#?}0^7I73x!@6q zD~H&zhj_wma1-*h84CY~+)t1aXD4AJ0b7=22;}grWQW3HV~ml1LS7(Yd9+c)82J~< zQPQW7SnuL+>G3H=MZ*&ut|+1=CsiJspy`Qhe{|c>h&&zTc8u3PzC$wZb3lSG1@O)P zCcc>0#F&_ zfv^BGnB#D>2V0Sar#?S;!J^t9MfXTl!9*K|mOy-hNHM{HAgYFkpdC>^iq3D3fFD%M z$Z(YRSwV3l5kb`D%{Ngk%|u?5CPt6Muexuv2MLhF(wK2*K8Q+izeEi#Ot5Z<=3JpC z*KbX3<>DZCfw;YcqFV5SpDS>U@P)x)K8hfiTWv!iT zUDOf|=Y{I?)l=pfb;{m2srl03&6Jd0YQES!y=GRCD%q5Ax-N8_@0hBYu1q zS~tBhrL9S8>yz608TPhz^Ws|Q!(t6lS_UR=Y2~E@7Z1GEb(heqN|IX7q$W|c;4Pol zBzw1jq!z$}*`9dn_2Eh79fKoNRx{a_(4Xzj6ctY@CwISQTy#O3iylH{ z{z>`3%-3T}flx}ig z;#6W_PG5eTqQANS3}N-&B@`fZqixPvHGO<~{|q-Se@cjj2gu4Vc8|)!%FP}jJ)|&K#QxE zv`Kx>HWep7$r_-KZBxBsn;M8+sK8uP#54ev?@*E{sRi>|w}cz@VBWkd@XJuLwB$RO z+)n~GdKjm?llGX=(aUw^d)yE+#B?#kAZ~8UuBD;RZSd{f(-hH9fDx@-Rd33F2Y$3N zAMzcjo50*R=FENIDsGG!AIDuxcZ!6O_QZ_f0F;0OV1`=eL9{@YT`}EiPK?Y2i4Kn? zW(EmA`~5hGTVvLk1-S4%f?ES)l+FH_B__FOEB0y$9>%etSG&Mrff~{X?4g!8wlT|U zePK-CcA4ciFe_nJXk&KCl_`~nZV+wCW!F`m+%s^=a($5;#P6#yic=`tg2vcD0!!(D z)BJs8QgCKt`M&lkKnp8C3mrhg0dls!;S4Ax?!eqd7ddW?J7Z30Uk6&_l<>_GKE8Kw zFEZA_y!%?{I6BfuPhQ*dvUM$yt9=!@y8an*MaSE?Qg1nQym8*4;mUhV1MhuMMhUby1+#?L*4w|CCe+W zIF9&F%5LtmI(5*$LVKE+UeZP5?E! zdK*V!_DD|nY9(dd8MiLQ^tt`B(Cud-EY12cc>Bo1BXB1S* zj>1 z1w#Q*5Ejdwf+L=+!Cp(Zv&={|1a39NJ;2S+laZRNqlZ{l@}QB}030b{C>&z!`RYf& zJ9@}D&qfO#CL>{U=y5(E>&I1+*EO!r9sr8EJT8YZJP!A04&yo;>S>p#gytmg?jb*x zf^T3Ek@t2%;rpEDthSqjN3QJRUc@$$1#odCr%V-!jDm*-C!4T}0?4}P98T@=a_OuO z!s61w<#&DZe~Xo{7Zb#_roU`IwBn% z!g)f7JxPx%>gb2Mi>8m{wr_;OQm0*7Xh}49VIo5u6d?HywZv&xNKU(QTtTmYK>B32 zoofI(;98>VsvI2{%Q@}29zKCQWoOI^J(QhvTyc1`KC(V~W!FdEYtC!@G&BAD%*L6( z%yY99^J_OJE4zNn<7oZZRYZ>4USPNgxVLat2nP%3Vsp*#jRO;p;GPEHbAxvbM;&Tu zYVh1n{LqR32U?^cp22f%_`C(5H83MN^k~&V9z9!*!A~s+a0t0}1RV&##37KXfUdRr zls`1W^3rjJNDW4UVXhOAx&QzNhTO196c1uW15^ky(vdMQD*6URL+*fAa;L#ZmX57O zEuKtCuT;|cwWz~`lpb7KL_?Sz?UB7~vQP58vyqc*XhI4zWlxYrO>g#iTI!Cd>I*_- zPgF_D0g;BR1UXjf5d(qAV2F3aLv~6phgoOb^pL)CuK+rVik}7NXo9$3Ost{b>X_J{ z(b?x*HA!6!oZM)1KN)rDEQ)He+;V>}%5Y%A~z= zQnLUx&ZNsXCd)Ub$~UJx&G(2+y1L0d8Ar*5XU;z}@2JXHJsD@|h0gPxlN}jP#q{WO zaOUM%*Q{oCWUgv^(zAVX@0T0uC-+0>U~12$&WoKh>rESI8NZaKFuYOoTx^Hnv#gvFD#lqW6a z^Og#bfUR<&eC?&Ni(^-vGr{-EQ{`I{dlrpEW$ov#V^gg&hdy}j%5$?@uPajZZ8x^w zs(=1X|D5aCJm4#dWFThIwP-qMBG=JGndKD!EH2O*IKJQ{=fY za(-m$g_+7!QGJ5SIEto5la891iVv!hp5`TGI{N zP1mQc8)xQz@M3Cj&s;xuYcDrVzNfmZnjV_nc$UOP5d-0@l4&3&KlyX}O=eV==dOuhV&1g@XCRe$(TZF8O@2^D}vJ<&-X zCO>x{zW(wQc}aCqH8nJ|G39Ca&Y{X7kpxyY2GWv@mnzulXrz zD)8o+Id|LS?xefzj>VnvuDRsB=!G@qdjEW7SIXO!P%l_ra~|J}?}LUb4RgLN*X-Ax zz4r8+yZy6GH(Ne!xov%3VzF)N=QkoHORlmM}Q3>U5?^h9? zin|1f%I5UWmDbH1nrX|$KZmWX z^0#+A1TDZ#O?Z#OO-~dZA%FXYyY07kwS>2w&rY@`c7u(CawbUo& z-tJ=UCtE0hH|*Z-a^sCUC%{ji_I6ike`Ti-mR5E9w7=HqyBqYswyObtR%Pwpto^K+ z0{Gvbbaa`Ne{UoaT6Ntj-QRmFx(VG)LIvfUT0?h{=BAqjxN00RomZADdK}H}3seCbCxT=y(=RpYjEWh&RFP>}W12qLK z`8znfu+8A<+DaiC$b!mo32uuEaBve@b}<@8D|5nV(OMbIhjj|5a(i)#B*G*G`B*Up zqw9yTlw221Gxc%p((u#&$8yP^F8+(jMVpli^gkci67bMMx(X+=OWx(fDC6K<%HtH* zfp%eVXm#y^v*v!q?@7PFUVNl}nPt8oEk{{%vbq(ATt3tv=%wLtz3e9#%b)$eX3<*g zC7j2rr4WObvE@ppm1j-9S9Y8+jAI;N+>q~^5&D)72Qp4sN-Kq6WJ;^fIwQ!-^bk$g z_oBJ_%CUQuk#)=TA3d^g%Kkg794qyY@yM;jO-t4lFH_Q=KTD6BajgMeDE%`sk8R1l zTeIYyl_`tczK2eg|K4;8j+j8FXj!LL*CY(Kt(ePNk?*x5W{)`-E$SL-UEG$H8DzES zVwxCaWW-D{Glm^wMyB#@eOwhYFeK(#K>maQa!l4@I1+-S&X^6}*kVqm3g0l*u#3Q2 zCHY>ixEo}vjOFz!UKA?=9k9iUV$PV9fpM+~@-Cb}!3i{LkOCY}tdoM`o|tE)EF1yd zQ0W?^bA=^0ptAm|Pu@r5-a;@4=4AF_YGa;<14(0L5SW4RR`!D65$sD+pr}(0%B=WZ zaW46Y?-1CDY2rF5u=BT%z{$3uIM?%-@A;gJ6?q`f{pJz%EtuyF`>K~I?pG|?Sxvwd zdG3!NQ7g+8d5)65`x-|63gpt}j|!0LMnl4)(Aa&X5wcr{Alq-njN5!FY&Q2c7x>u) z(UE$#55h5ml=LBmr=(mDaCbu?KAY+Y851v!gonbB(eO){e2JkINbP00&=|&>Ajx7F z6KBzn%Vur*Pl1PrIh&ZW2?2=+$J20Tn@fYxCzR^K0E#AEix zyw$Rr?X{ZSb%cPil``Q`13~{(m}L}zVC4}aKMsnnE2QM8Us1z}_((6f)d9$o6peua zHZar!eTJk>QPbY>eAjaa4v1>Vw}6}rbc`W)TZ#%`ret840Vg?on9hT#K+xH+21JrZ z!sF%Hv-4c)^sAG$k$?V}${ux*&~Vj1LH*aYcQY`$OQKdlkV3 z083+Le+Z^n4hMmcVosS&^&mBh*4uMA= z0%fb3_*kuLwJSfikpM+ihz~&E4cQ(ybb=~{q^M`h1G42Sv?oPX7*$Xto5meS2QVFK zJ@3Qxewd^j+XDGS9Lqw_2ga#RcDMx+aw&dWkcb@$X!CF z5v@Ht;SNIzcyO4F$Gd>CA{d2Cm|%DarH{jGo%uAOxa7I@}K{Hgp+_0v8F1`ocRvw<9_-(7;EA{iop@Wb@>U zfMClba4^O_fFo+<(nFs|^Plr2C9p7VT>9gW<)KiL`w?{6r{-QmA?qM>Uw8qQx|7`N z@XT$M#KEz!XN9E)7#K{(l*-(HhSCE_Pv_{El;!WUbAN;%{}qCtAeaOIt1QI+1sEcb zKPIz7Ha{Lgz>~R7^z$W5~fX^&c;u7A}R(^rNFDt%XMQJ zBdFXt1h@=v{|>=(03_XDIozVSD+u00fJ-3v9}xTn08ta+dn8RmyI<7jMjSMfyNWm; z;9C_Wc|nSoGRU0fKE$^u2bkTEFc=caQ``x7?>38+ZML7wbF#QORs2-Cczd#V`?Wx-cvnIr^8|Wewk6h0ZBCS@OcfK) z-my7vxjIv}u8DnjT%KFru9T~LqU%eOZQ|L7rF5%i*Jmy1{U?+APtKcPOqNqH zVi{p|WNdCQeln$X8H+t*c4h3IjJ-5dydL8CU%L!8HOv?fp|vIU-qNhelvF_?+J4A- z`_gKkXuG2^Wjtl@X3tp@*j`2IyM$I%a#jy-p8MKJm>h{sukW9PUxFDk-eO2)Q@yrt z(FPb_+X<~9G-#>Zn1tnp0LHJf>617IK8C0fk@4LVIN}qe> z$ykeCS0SF~+xyP%;(5ON%J|(^_S~zDci`tOqtjF(GCh{EH@&Vw!W+N6PcN_Xfd8+s z5dG^`Pq&HqxTXf+-+JL;4e@syYXJVjOLx~3zu4C5?S6*%Z*Dij##ZA&3-QZs6u=v# z7vY+92g`|1HCqoh62DS51N^nO)p$rxd{$Ej@TN)EU97z6B@wRCb=N9y`bdQJhVE_3 zn_EeQ&*%;sm2(CXp+$GFR5@2dA}rS(tW(ZyAQ3h?4{4N1HHlEKJ5;1hx=DZm=t2I% zI4)Q{Xo7} zxD(;E(r<27ddE-@yUuYBUJtQaS{)`~uaq*h8Xs`_t6Em=}eSd;BL{^n-s`!T|&>*CJCM;|Gr$B(P)>oFsF{E5Sm zP!CzjXzt<*dVS6}YoW8b+4!Se>F0{sU!wVZ%4}Bs>p)DT2G|f7k(kvCInZ#JDr$0v zvp$Qcg*;@S!Ly>dmFpiFhSSx9_{fQxo*u9idV0A3f<%mdhUo!Z2_tL3XnUkn*`6MW zULB|{@?SWv@$ zIAWfMm#v`b{9i+1f>ZhsV&viaowM6+B0w3n_Ik!l2lPfYyFw< z?$N93|NO;t_10wd)>QTObai{Ox;=G-W88 zXosK5y|Lx&mZfj3&cvznrfGH3S~IaXV|Pz^;V^b$|36YCQ|;5fd8#g>v8OfOq{ch1 zDOu3E5<}Bl=C$<;lsh3z9luSjlYUzuf&j5oJw0G{;`s-+9|4-390pUsYL#q1RQs#} zD3$QLNa4>;0Rbf?jm9zr?sFud$P$yvF77j|sE0ut!59q;EP6}auMo5#*ogo=gshp1 zNB)@mFATuXRkI_m>VuqoDbf8@kb|Ex;?WLgM&Lvs4RU^brLn?3qOSz4twR(S0z9pN zUunRPJs`_x2e$x^=qmCY09d$5@@vXKQvauzAlH6DSiT?((x2`N!uAEB{|CbJwUw~B z6TN8eT?n5K&sFcZu1vYxQ`U|N(>=9?wBD;~AS>?mk_Sk#?%s=v{R(*c0c9rFFIEzi WadP~Y(tC%}{?W6GN`fkuc=~_*3M23U literal 0 HcmV?d00001 diff --git a/scripts/benchmark.sh b/scripts/benchmark.sh index fadc448f6..72616569d 100755 --- a/scripts/benchmark.sh +++ b/scripts/benchmark.sh @@ -448,6 +448,22 @@ save_result() { EOF info "saved results/$profile/$CONNS/${FRAMEWORK}.json" + # rebuild_site_data.py refuses to publish a run whose responses were mostly + # errors — rps counts a 500 exactly like a 200. The row and the container + # log are still written above, since they're the post-mortem evidence, but + # say so here: otherwise a clean-looking --save implies a result that will + # never reach the board. Threshold shared via HTTPARENA_MAX_ERROR_PCT. + local ok_count err_count + ok_count=$(( ${BEST_M[status_2xx]:-0} + ${BEST_M[status_3xx]:-0} )) + err_count=$(( ${BEST_M[status_4xx]:-0} + ${BEST_M[status_5xx]:-0} )) + if [ "$ok_count" -eq 0 ]; then + warn "$profile/$CONNS: no successful responses ($err_count errors) — will not be published" + elif awk -v e="$err_count" -v n="$((ok_count + err_count))" \ + -v thr="${HTTPARENA_MAX_ERROR_PCT:-5}" 'BEGIN{exit !(e/n*100 > thr)}'; then + warn "$profile/$CONNS: $(awk -v e="$err_count" -v n="$((ok_count + err_count))" \ + 'BEGIN{printf "%.1f", e/n*100}')% non-2xx/3xx ($err_count of $((ok_count + err_count))) — will not be published" + fi + # Persist container logs alongside results for post-mortem. local log_dir="$ROOT_DIR/site/static/logs/$profile/$CONNS" mkdir -p "$log_dir" diff --git a/scripts/rebuild_site_data.py b/scripts/rebuild_site_data.py index 57598fda6..9842916dd 100755 --- a/scripts/rebuild_site_data.py +++ b/scripts/rebuild_site_data.py @@ -22,6 +22,32 @@ from pathlib import Path +# A run whose responses were overwhelmingly errors is not a measurement. rps +# counts a 500 exactly like a 200, so a framework that collapses under load +# can outrank one that serves every request — php-fpm published 294k rps on +# baseline-h2 with 99.2% 5xx. Rows above this error rate are dropped rather +# than ranked. Override for a one-off rebuild with HTTPARENA_MAX_ERROR_PCT. +MAX_ERROR_PCT = float(os.environ.get("HTTPARENA_MAX_ERROR_PCT", "5")) + + +def failure_reason(entry: dict) -> str | None: + """Why this result row isn't publishable, or None if it is. + + Every load generator in scripts/lib/tools/ reports status counts, so a row + whose 2xx+3xx is zero got nothing back at all — a crash, a hang, or a + refused connection. Publishing its rps=0 states "we measured zero + throughput", which ranks the framework last on a test that never ran. + """ + ok = entry.get("status_2xx", 0) + entry.get("status_3xx", 0) + err = entry.get("status_4xx", 0) + entry.get("status_5xx", 0) + if ok == 0: + return "no successful responses" + pct = err / (ok + err) * 100 + if pct > MAX_ERROR_PCT: + return f"{pct:.1f}% non-2xx/3xx" + return None + + def rebuild_frameworks_json(root: Path, site_data: Path) -> None: """Aggregate every frameworks/*/meta.json into site/data/frameworks.json. @@ -135,6 +161,17 @@ def merge_results(results_dir: Path, site_data: Path) -> None: if stale: print(f"[purged stale] {data_file.name}: {stale}", file=sys.stderr) final = [e for e in final if e.get("framework", "") in valid_names] + + # Applied to the merged list, not just this run's batch, so a bad + # row already sitting in site/data is dropped on the next rebuild + # rather than living on because its framework wasn't re-run. + dropped = [(e.get("framework", ""), failure_reason(e)) for e in final] + dropped = [(name, why) for name, why in dropped if why] + if dropped: + detail = ", ".join(f"{name} ({why})" for name, why in dropped) + print(f"[dropped failed] {data_file.name}: {detail}", file=sys.stderr) + final = [e for e in final if not failure_reason(e)] + data_file.write_text(json.dumps(final, indent=2)) print(f"[updated] {data_file}") diff --git a/site/content/docs/scoring/composite-score.md b/site/content/docs/scoring/composite-score.md index 07a5b0565..771e27015 100644 --- a/site/content/docs/scoring/composite-score.md +++ b/site/content/docs/scoring/composite-score.md @@ -34,6 +34,12 @@ Summing instead of averaging means the composite scales with the number of score Frameworks that don't participate in a scored profile receive 0 for that profile, which lowers their composite by the full 100-point ceiling of that profile. +### Failed runs are not scored + +A run whose responses were mostly errors is not a measurement: rps counts a 500 exactly like a 200, so a framework that collapses under load would otherwise outrank one that serves every request. When the site data is built, a result is dropped rather than ranked if it has **no successful responses at all** (a crash, a hang, or a refused connection — its rps of 0 would read as a measured zero) or if **more than 5% of its responses were non-2xx/3xx**. + +A dropped result is treated exactly like not participating in that profile: the framework scores 0 for it. The raw result and the container log are still kept under `results/` and `site/static/logs/` for post-mortem. + ## Scored vs reference-only profiles Not all profiles count toward the composite score. Profiles marked as **scored** contribute to the composite. Reference-only profiles (marked with **\***) are displayed for comparison but do not affect the ranking. diff --git a/site/data/api-16-1024.json b/site/data/api-16-1024.json index 1e4486c4c..94df3f2d3 100644 --- a/site/data/api-16-1024.json +++ b/site/data/api-16-1024.json @@ -1011,32 +1011,6 @@ "tpl_static": 0, "tpl_async_db": 388837 }, - { - "framework": "rage", - "language": "Ruby", - "rps": 9825, - "avg_latency": "84.56ms", - "p99_latency": "1.78s", - "cpu": "749.7%", - "memory": "6.0GiB", - "connections": 1024, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "47.17MB/s", - "input_bw": "566.09KB/s", - "reconnects": 34413, - "status_2xx": 147382, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 26433, - "tpl_baseline": 62443, - "tpl_json": 68252, - "tpl_db": 0, - "tpl_upload": 0, - "tpl_static": 0, - "tpl_async_db": 16687 - }, { "framework": "rails", "language": "Ruby", diff --git a/site/data/api-4-256.json b/site/data/api-4-256.json index 240b22219..f555bac6b 100644 --- a/site/data/api-4-256.json +++ b/site/data/api-4-256.json @@ -1011,32 +1011,6 @@ "tpl_static": 0, "tpl_async_db": 164332 }, - { - "framework": "rage", - "language": "Ruby", - "rps": 13999, - "avg_latency": "16.30ms", - "p99_latency": "266.00ms", - "cpu": "407.6%", - "memory": "5.9GiB", - "connections": 256, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "68.87MB/s", - "input_bw": "806.58KB/s", - "reconnects": 45990, - "status_2xx": 209997, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 20194, - "tpl_baseline": 85974, - "tpl_json": 86523, - "tpl_db": 0, - "tpl_upload": 0, - "tpl_static": 0, - "tpl_async_db": 37500 - }, { "framework": "rails", "language": "Ruby", diff --git a/site/data/baseline-h2-1024.json b/site/data/baseline-h2-1024.json index 91dd37df0..66b12825b 100644 --- a/site/data/baseline-h2-1024.json +++ b/site/data/baseline-h2-1024.json @@ -404,26 +404,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "php-fpm", - "language": "PHP", - "rps": 294864, - "avg_latency": "143.45ms", - "p99_latency": "143.45ms", - "cpu": "2169.8%", - "memory": "4.8GiB", - "connections": 1024, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "60.80MB/s", - "input_bw": "", - "reconnects": 0, - "status_2xx": 11817, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 1462506 - }, { "framework": "pyronova", "language": "Python", @@ -558,25 +538,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "symfony-spawn-franken", - "language": "PHP", - "rps": 97817, - "avg_latency": "465.99ms", - "p99_latency": "465.99ms", - "cpu": "4817.0%", - "memory": "7.1GiB", - "connections": 1024, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "57.95MB/s", - "reconnects": 0, - "status_2xx": 494959, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 284226 - }, { "framework": "symfony-spawn-tas", "language": "PHP", diff --git a/site/data/baseline-h2-256.json b/site/data/baseline-h2-256.json index 72a8ecba8..31c83802d 100644 --- a/site/data/baseline-h2-256.json +++ b/site/data/baseline-h2-256.json @@ -558,25 +558,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "symfony-spawn-franken", - "language": "PHP", - "rps": 28243, - "avg_latency": "229.68ms", - "p99_latency": "229.68ms", - "cpu": "5180.8%", - "memory": "4.3GiB", - "connections": 256, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "76.21MB/s", - "reconnects": 0, - "status_2xx": 142064, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 381252 - }, { "framework": "symfony-spawn-tas", "language": "PHP", diff --git a/site/data/crud-4096.json b/site/data/crud-4096.json index 1855e56ca..be770ba56 100644 --- a/site/data/crud-4096.json +++ b/site/data/crud-4096.json @@ -119,25 +119,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "fasthttp", - "language": "V", - "rps": 0, - "avg_latency": "0us", - "p99_latency": "0us", - "cpu": "0.0%", - "memory": "0MiB", - "connections": 4096, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "4B/s", - "reconnects": 14506257, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 0 - }, { "framework": "fishcake", "language": "Kotlin", diff --git a/site/data/pipelined-4096.json b/site/data/pipelined-4096.json index 52725ecf0..a217a0637 100644 --- a/site/data/pipelined-4096.json +++ b/site/data/pipelined-4096.json @@ -18,25 +18,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "apache", - "language": "C", - "rps": 0, - "avg_latency": "26.50ms", - "p99_latency": "250.40ms", - "cpu": "2043.6%", - "memory": "136MiB", - "connections": 4096, - "threads": 64, - "duration": "5s", - "pipeline": 16, - "bandwidth": "136.05MB/s", - "reconnects": 14, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 1858306, - "status_5xx": 0 - }, { "framework": "aspnet-minimal", "language": "C#", @@ -171,25 +152,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "caddy", - "language": "Go", - "rps": 0, - "avg_latency": "47.40ms", - "p99_latency": "1.56s", - "cpu": "4491.3%", - "memory": "362MiB", - "connections": 4096, - "threads": 64, - "duration": "5s", - "pipeline": 16, - "bandwidth": "105.93MB/s", - "reconnects": 0, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 3779515, - "status_5xx": 0 - }, { "framework": "dart-io", "language": "Dart", @@ -1059,25 +1021,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "pingora", - "language": "Rust", - "rps": 0, - "avg_latency": "54.31ms", - "p99_latency": "73.60ms", - "cpu": "561.1%", - "memory": "75MiB", - "connections": 4096, - "threads": 64, - "duration": "5s", - "pipeline": 16, - "bandwidth": "10.01MB/s", - "reconnects": 372335, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 372355, - "status_5xx": 0 - }, { "framework": "pyronova", "language": "Python", @@ -1556,25 +1499,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "traefik", - "language": "Go", - "rps": 0, - "avg_latency": "573.13ms", - "p99_latency": "1.29s", - "cpu": "5316.4%", - "memory": "517MiB", - "connections": 4096, - "threads": 64, - "duration": "5s", - "pipeline": 16, - "bandwidth": "9.78MB/s", - "reconnects": 0, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 534152 - }, { "framework": "trillium", "language": "Rust", diff --git a/site/data/pipelined-512.json b/site/data/pipelined-512.json index 03ac6b812..fab187253 100644 --- a/site/data/pipelined-512.json +++ b/site/data/pipelined-512.json @@ -18,25 +18,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "apache", - "language": "C", - "rps": 0, - "avg_latency": "21.35ms", - "p99_latency": "197.50ms", - "cpu": "2018.6%", - "memory": "82MiB", - "connections": 512, - "threads": 64, - "duration": "5s", - "pipeline": 16, - "bandwidth": "137.31MB/s", - "reconnects": 5, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 1875336, - "status_5xx": 0 - }, { "framework": "aspnet-minimal", "language": "C#", @@ -171,25 +152,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "caddy", - "language": "Go", - "rps": 0, - "avg_latency": "13.48ms", - "p99_latency": "196.80ms", - "cpu": "3539.8%", - "memory": "122MiB", - "connections": 512, - "threads": 64, - "duration": "5s", - "pipeline": 16, - "bandwidth": "84.14MB/s", - "reconnects": 0, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 3002243, - "status_5xx": 0 - }, { "framework": "dart-io", "language": "Dart", @@ -1059,25 +1021,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "pingora", - "language": "Rust", - "rps": 0, - "avg_latency": "6.77ms", - "p99_latency": "7.65ms", - "cpu": "576.5%", - "memory": "69MiB", - "connections": 512, - "threads": 64, - "duration": "5s", - "pipeline": 16, - "bandwidth": "10.05MB/s", - "reconnects": 373945, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 373972, - "status_5xx": 0 - }, { "framework": "pyronova", "language": "Python", @@ -1556,25 +1499,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "traefik", - "language": "Go", - "rps": 0, - "avg_latency": "71.62ms", - "p99_latency": "138.20ms", - "cpu": "5337.0%", - "memory": "181MiB", - "connections": 512, - "threads": 64, - "duration": "5s", - "pipeline": 16, - "bandwidth": "10.45MB/s", - "reconnects": 0, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 571145 - }, { "framework": "trillium", "language": "Rust", diff --git a/site/data/static-h2-1024.json b/site/data/static-h2-1024.json index b18360084..819e12783 100644 --- a/site/data/static-h2-1024.json +++ b/site/data/static-h2-1024.json @@ -95,25 +95,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "effinitive", - "language": "C#", - "rps": 0, - "avg_latency": "", - "p99_latency": "", - "cpu": "6187.1%", - "memory": "4.3GiB", - "connections": 1024, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "0", - "reconnects": 0, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 0 - }, { "framework": "fastendpoints", "language": "C#", @@ -558,25 +539,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "symfony-spawn-franken", - "language": "PHP", - "rps": 52582, - "avg_latency": "246.82ms", - "p99_latency": "246.82ms", - "cpu": "4915.1%", - "memory": "5.1GiB", - "connections": 1024, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "123.87MB/s", - "reconnects": 0, - "status_2xx": 266595, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 346287 - }, { "framework": "symfony-spawn-tas", "language": "PHP", @@ -615,25 +577,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "trillium-tuned", - "language": "Rust", - "rps": 0, - "avg_latency": "", - "p99_latency": "", - "cpu": "2590.0%", - "memory": "5.0GiB", - "connections": 1024, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "0", - "reconnects": 0, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 0 - }, { "framework": "true-async-server", "language": "PHP", diff --git a/site/data/static-h2-256.json b/site/data/static-h2-256.json index 264571722..7fe6959ff 100644 --- a/site/data/static-h2-256.json +++ b/site/data/static-h2-256.json @@ -558,25 +558,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "symfony-spawn-franken", - "language": "PHP", - "rps": 63518, - "avg_latency": "64.13ms", - "p99_latency": "64.13ms", - "cpu": "5491.9%", - "memory": "2.2GiB", - "connections": 256, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "637.75MB/s", - "reconnects": 0, - "status_2xx": 320134, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 286846 - }, { "framework": "symfony-spawn-tas", "language": "PHP", @@ -615,25 +596,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "trillium-tuned", - "language": "Rust", - "rps": 0, - "avg_latency": "", - "p99_latency": "", - "cpu": "1486.4%", - "memory": "5.1GiB", - "connections": 256, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "0", - "reconnects": 0, - "status_2xx": 0, - "status_3xx": 0, - "status_4xx": 0, - "status_5xx": 0 - }, { "framework": "true-async-server", "language": "PHP", diff --git a/site/data/upload-256.json b/site/data/upload-256.json index 3c66d6358..22f664dd6 100644 --- a/site/data/upload-256.json +++ b/site/data/upload-256.json @@ -798,26 +798,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "phoenix-bandit", - "language": "Elixir", - "rps": 685, - "avg_latency": "194.10ms", - "p99_latency": "894.30ms", - "cpu": "1835.1%", - "memory": "3.1GiB", - "connections": 256, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "183.74KB/s", - "input_bw": "5.43GB/s", - "reconnects": 3351, - "status_2xx": 3425, - "status_3xx": 0, - "status_4xx": 1714, - "status_5xx": 0 - }, { "framework": "php-fpm", "language": "PHP", diff --git a/site/data/upload-32.json b/site/data/upload-32.json index 843c16c90..cbfafdc20 100644 --- a/site/data/upload-32.json +++ b/site/data/upload-32.json @@ -798,26 +798,6 @@ "status_4xx": 0, "status_5xx": 0 }, - { - "framework": "phoenix-bandit", - "language": "Elixir", - "rps": 704, - "avg_latency": "24.18ms", - "p99_latency": "100.00ms", - "cpu": "1208.7%", - "memory": "878MiB", - "connections": 32, - "threads": 64, - "duration": "5s", - "pipeline": 1, - "bandwidth": "188.98KB/s", - "input_bw": "5.58GB/s", - "reconnects": 3518, - "status_2xx": 3523, - "status_3xx": 0, - "status_4xx": 1761, - "status_5xx": 0 - }, { "framework": "php-fpm", "language": "PHP", From feed522ba74e5869f30a58f96116544c3246c30d Mon Sep 17 00:00:00 2001 From: MDA2AV Date: Fri, 24 Jul 2026 21:17:25 +0100 Subject: [PATCH 2/2] Drop a stray .pyc and gitignore Python bytecode scripts/__pycache__/rebuild_site_data.cpython-312.pyc rode in with the previous commit: verifying the new failure_reason() meant importing rebuild_site_data as a module, which writes bytecode next to the source, and nothing in .gitignore covered it. Removes the file and ignores __pycache__/ and *.py[cod] so importing any script in scripts/ can't do this again. --- .gitignore | 4 ++++ .../rebuild_site_data.cpython-312.pyc | Bin 16484 -> 0 bytes 2 files changed, 4 insertions(+) delete mode 100644 scripts/__pycache__/rebuild_site_data.cpython-312.pyc diff --git a/.gitignore b/.gitignore index 8e3be1127..e05bdbfe7 100644 --- a/.gitignore +++ b/.gitignore @@ -28,3 +28,7 @@ node_modules/ # macOS Finder metadata .DS_Store + +# Python bytecode (generated when the site-data scripts are imported/run) +__pycache__/ +*.py[cod] diff --git a/scripts/__pycache__/rebuild_site_data.cpython-312.pyc b/scripts/__pycache__/rebuild_site_data.cpython-312.pyc deleted file mode 100644 index 8fc8a48c95503e5219c2f9bfa167adfed10627d7..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 16484 zcmd6OZBQInl3-PL_4oHD-GHD#fYgW&BMI9=9~J_vPe5O`R`STW=_;V1>26mQLZI$3 zGwZ!Uwr<_>%&pMcKA~O58J)Ff_RhB_Z%o|EyW<;q#I5a-RVP%^U423=-8CmP(p`H(3wJ7{8`huD zlZ1j`4Xk?5BR5sJ50(_la*0tf>h~4dr}xS1{e+n@F`8GY6BfqIXyI;UEQ}8BHh9*< z-OgAU1Kb^~lQ9mhCkRdptvcS%5haN+<(SfE8`%nk^@rKskzk0S`JliyGJe6|Slckj zN5XU;7a68GmLCZTeB-t#7wHRz*d2}A0+Dc-hkGCAA7)1*+|Z63K78gI^!j5QP~uzk z^b(SJTt16S)AawC_!&Jg*2@JMde|SOBYpH4Cdfxa{;{4g5N^Rou8w9{KEMT|LNF4p zqZ=9;>gZE`F6a*n{4??RLv7?)tk1ESYYH10X1RWrp`$EUpX*3Hma!|jF4BIE^N;Zl zQ4EZ592*w$EX!Q`IcC()vGfMI`v?FoG7@I!V7MRZ=gF)1Lc4K*0>B)}GGRKj`T=e7O5?&%w5%BH0|sDYB%GDu7CY zAn#?MaDj*sgG;|kD1Pypz98<^O zDW<^p;|AbMGY>(+_lDRy8V0K?5@zXOpEN2ssX+DUj#Df*Mu#GPhVEy>Eaw*h zF_>3|#!#@gQHVrBd?V10MmT{7b@K}&JRJZH75F+DhTM;hO5{cdB0NiPJbik@rqicU zJI7frLia}m8b)&fhrHJx7^3|G?T2y4S>&e!oSz@4gTjD6+%GZemuPWppQN@pq1XUw zLIZs;+a)9v1Xj2xzx@e`A(p2rM_HJYetra0RccqFA`HM(>K}-X2$gm8=s++qK*P)! zf++w~D!+u#AwMtBFbQDduuu{PfW%>x(64GhUl1R?(+vjAif@$*05T0PM}=@|NI#M6U3sG&|BW{p7kcS zP1{nswbSf-!OOu}<28S(W@l2jb7EHpzWdJl5(lO$Qu?atk@vZ5)E7@3N$P3*s6Th|mZmskcO?P|?^OMqW&NCXea2w9 z2h9_K{A7R!9sDbCf;T|_e`2{n1c|sp$WL+szGbh$rL}VJ1Y91ZD+Y^zH02Z$R3dyo zM`iiFojCcAI4yW;G3D~sXVmik5Dsj~=w-(n6)iRYyOCQNNUe(6*?Rh#eJ>x|F^Y{k*`i457 z=m)Jn31b;o$JC5V>Z2ihA9@O=9eK_YQ=eCyvr|Oe7*`AH=r+*D2hYBBto7N9KkEEXIqPA2-L$gBuEXa%qX))yeM)_n4L` zzFHzvlldYEp-9GIN@M1?h<6l)Z)2LZgn+AybX5^t9rUh%PsNmP5sWIPhFX-6w`vEA zQqnna2RgO`43srma7Y{am)Lx=qIOR~lLQr1YN*P|&1 zJ0KWoy=YK?y$QTTuB zi>!eFRye(VJH3>-I+_iE=>UcyY`^|c6wDS@2nPHidGjrZULX^h=wJ$zetD_8Z{y%j73>d%c4aO1Vl9(hCAz1iAvy< zD+6`oXau;daJXWK)Nq7hAF?@dqk$!uyk~?0+MEoD6Z#0@Q*?1vP`W|@QzXmd{z7xO zT0hWk3;Bn88UK!PpS&R#y4iyY*sv`W0U_`^ppZp90u_1mGAD@Hm#?}0^7I73x!@6q zD~H&zhj_wma1-*h84CY~+)t1aXD4AJ0b7=22;}grWQW3HV~ml1LS7(Yd9+c)82J~< zQPQW7SnuL+>G3H=MZ*&ut|+1=CsiJspy`Qhe{|c>h&&zTc8u3PzC$wZb3lSG1@O)P zCcc>0#F&_ zfv^BGnB#D>2V0Sar#?S;!J^t9MfXTl!9*K|mOy-hNHM{HAgYFkpdC>^iq3D3fFD%M z$Z(YRSwV3l5kb`D%{Ngk%|u?5CPt6Muexuv2MLhF(wK2*K8Q+izeEi#Ot5Z<=3JpC z*KbX3<>DZCfw;YcqFV5SpDS>U@P)x)K8hfiTWv!iT zUDOf|=Y{I?)l=pfb;{m2srl03&6Jd0YQES!y=GRCD%q5Ax-N8_@0hBYu1q zS~tBhrL9S8>yz608TPhz^Ws|Q!(t6lS_UR=Y2~E@7Z1GEb(heqN|IX7q$W|c;4Pol zBzw1jq!z$}*`9dn_2Eh79fKoNRx{a_(4Xzj6ctY@CwISQTy#O3iylH{ z{z>`3%-3T}flx}ig z;#6W_PG5eTqQANS3}N-&B@`fZqixPvHGO<~{|q-Se@cjj2gu4Vc8|)!%FP}jJ)|&K#QxE zv`Kx>HWep7$r_-KZBxBsn;M8+sK8uP#54ev?@*E{sRi>|w}cz@VBWkd@XJuLwB$RO z+)n~GdKjm?llGX=(aUw^d)yE+#B?#kAZ~8UuBD;RZSd{f(-hH9fDx@-Rd33F2Y$3N zAMzcjo50*R=FENIDsGG!AIDuxcZ!6O_QZ_f0F;0OV1`=eL9{@YT`}EiPK?Y2i4Kn? zW(EmA`~5hGTVvLk1-S4%f?ES)l+FH_B__FOEB0y$9>%etSG&Mrff~{X?4g!8wlT|U zePK-CcA4ciFe_nJXk&KCl_`~nZV+wCW!F`m+%s^=a($5;#P6#yic=`tg2vcD0!!(D z)BJs8QgCKt`M&lkKnp8C3mrhg0dls!;S4Ax?!eqd7ddW?J7Z30Uk6&_l<>_GKE8Kw zFEZA_y!%?{I6BfuPhQ*dvUM$yt9=!@y8an*MaSE?Qg1nQym8*4;mUhV1MhuMMhUby1+#?L*4w|CCe+W zIF9&F%5LtmI(5*$LVKE+UeZP5?E! zdK*V!_DD|nY9(dd8MiLQ^tt`B(Cud-EY12cc>Bo1BXB1S* zj>1 z1w#Q*5Ejdwf+L=+!Cp(Zv&={|1a39NJ;2S+laZRNqlZ{l@}QB}030b{C>&z!`RYf& zJ9@}D&qfO#CL>{U=y5(E>&I1+*EO!r9sr8EJT8YZJP!A04&yo;>S>p#gytmg?jb*x zf^T3Ek@t2%;rpEDthSqjN3QJRUc@$$1#odCr%V-!jDm*-C!4T}0?4}P98T@=a_OuO z!s61w<#&DZe~Xo{7Zb#_roU`IwBn% z!g)f7JxPx%>gb2Mi>8m{wr_;OQm0*7Xh}49VIo5u6d?HywZv&xNKU(QTtTmYK>B32 zoofI(;98>VsvI2{%Q@}29zKCQWoOI^J(QhvTyc1`KC(V~W!FdEYtC!@G&BAD%*L6( z%yY99^J_OJE4zNn<7oZZRYZ>4USPNgxVLat2nP%3Vsp*#jRO;p;GPEHbAxvbM;&Tu zYVh1n{LqR32U?^cp22f%_`C(5H83MN^k~&V9z9!*!A~s+a0t0}1RV&##37KXfUdRr zls`1W^3rjJNDW4UVXhOAx&QzNhTO196c1uW15^ky(vdMQD*6URL+*fAa;L#ZmX57O zEuKtCuT;|cwWz~`lpb7KL_?Sz?UB7~vQP58vyqc*XhI4zWlxYrO>g#iTI!Cd>I*_- zPgF_D0g;BR1UXjf5d(qAV2F3aLv~6phgoOb^pL)CuK+rVik}7NXo9$3Ost{b>X_J{ z(b?x*HA!6!oZM)1KN)rDEQ)He+;V>}%5Y%A~z= zQnLUx&ZNsXCd)Ub$~UJx&G(2+y1L0d8Ar*5XU;z}@2JXHJsD@|h0gPxlN}jP#q{WO zaOUM%*Q{oCWUgv^(zAVX@0T0uC-+0>U~12$&WoKh>rESI8NZaKFuYOoTx^Hnv#gvFD#lqW6a z^Og#bfUR<&eC?&Ni(^-vGr{-EQ{`I{dlrpEW$ov#V^gg&hdy}j%5$?@uPajZZ8x^w zs(=1X|D5aCJm4#dWFThIwP-qMBG=JGndKD!EH2O*IKJQ{=fY za(-m$g_+7!QGJ5SIEto5la891iVv!hp5`TGI{N zP1mQc8)xQz@M3Cj&s;xuYcDrVzNfmZnjV_nc$UOP5d-0@l4&3&KlyX}O=eV==dOuhV&1g@XCRe$(TZF8O@2^D}vJ<&-X zCO>x{zW(wQc}aCqH8nJ|G39Ca&Y{X7kpxyY2GWv@mnzulXrz zD)8o+Id|LS?xefzj>VnvuDRsB=!G@qdjEW7SIXO!P%l_ra~|J}?}LUb4RgLN*X-Ax zz4r8+yZy6GH(Ne!xov%3VzF)N=QkoHORlmM}Q3>U5?^h9? zin|1f%I5UWmDbH1nrX|$KZmWX z^0#+A1TDZ#O?Z#OO-~dZA%FXYyY07kwS>2w&rY@`c7u(CawbUo& z-tJ=UCtE0hH|*Z-a^sCUC%{ji_I6ike`Ti-mR5E9w7=HqyBqYswyObtR%Pwpto^K+ z0{Gvbbaa`Ne{UoaT6Ntj-QRmFx(VG)LIvfUT0?h{=BAqjxN00RomZADdK}H}3seCbCxT=y(=RpYjEWh&RFP>}W12qLK z`8znfu+8A<+DaiC$b!mo32uuEaBve@b}<@8D|5nV(OMbIhjj|5a(i)#B*G*G`B*Up zqw9yTlw221Gxc%p((u#&$8yP^F8+(jMVpli^gkci67bMMx(X+=OWx(fDC6K<%HtH* zfp%eVXm#y^v*v!q?@7PFUVNl}nPt8oEk{{%vbq(ATt3tv=%wLtz3e9#%b)$eX3<*g zC7j2rr4WObvE@ppm1j-9S9Y8+jAI;N+>q~^5&D)72Qp4sN-Kq6WJ;^fIwQ!-^bk$g z_oBJ_%CUQuk#)=TA3d^g%Kkg794qyY@yM;jO-t4lFH_Q=KTD6BajgMeDE%`sk8R1l zTeIYyl_`tczK2eg|K4;8j+j8FXj!LL*CY(Kt(ePNk?*x5W{)`-E$SL-UEG$H8DzES zVwxCaWW-D{Glm^wMyB#@eOwhYFeK(#K>maQa!l4@I1+-S&X^6}*kVqm3g0l*u#3Q2 zCHY>ixEo}vjOFz!UKA?=9k9iUV$PV9fpM+~@-Cb}!3i{LkOCY}tdoM`o|tE)EF1yd zQ0W?^bA=^0ptAm|Pu@r5-a;@4=4AF_YGa;<14(0L5SW4RR`!D65$sD+pr}(0%B=WZ zaW46Y?-1CDY2rF5u=BT%z{$3uIM?%-@A;gJ6?q`f{pJz%EtuyF`>K~I?pG|?Sxvwd zdG3!NQ7g+8d5)65`x-|63gpt}j|!0LMnl4)(Aa&X5wcr{Alq-njN5!FY&Q2c7x>u) z(UE$#55h5ml=LBmr=(mDaCbu?KAY+Y851v!gonbB(eO){e2JkINbP00&=|&>Ajx7F z6KBzn%Vur*Pl1PrIh&ZW2?2=+$J20Tn@fYxCzR^K0E#AEix zyw$Rr?X{ZSb%cPil``Q`13~{(m}L}zVC4}aKMsnnE2QM8Us1z}_((6f)d9$o6peua zHZar!eTJk>QPbY>eAjaa4v1>Vw}6}rbc`W)TZ#%`ret840Vg?on9hT#K+xH+21JrZ z!sF%Hv-4c)^sAG$k$?V}${ux*&~Vj1LH*aYcQY`$OQKdlkV3 z083+Le+Z^n4hMmcVosS&^&mBh*4uMA= z0%fb3_*kuLwJSfikpM+ihz~&E4cQ(ybb=~{q^M`h1G42Sv?oPX7*$Xto5meS2QVFK zJ@3Qxewd^j+XDGS9Lqw_2ga#RcDMx+aw&dWkcb@$X!CF z5v@Ht;SNIzcyO4F$Gd>CA{d2Cm|%DarH{jGo%uAOxa7I@}K{Hgp+_0v8F1`ocRvw<9_-(7;EA{iop@Wb@>U zfMClba4^O_fFo+<(nFs|^Plr2C9p7VT>9gW<)KiL`w?{6r{-QmA?qM>Uw8qQx|7`N z@XT$M#KEz!XN9E)7#K{(l*-(HhSCE_Pv_{El;!WUbAN;%{}qCtAeaOIt1QI+1sEcb zKPIz7Ha{Lgz>~R7^z$W5~fX^&c;u7A}R(^rNFDt%XMQJ zBdFXt1h@=v{|>=(03_XDIozVSD+u00fJ-3v9}xTn08ta+dn8RmyI<7jMjSMfyNWm; z;9C_Wc|nSoGRU0fKE$^u2bkTEFc=caQ``x7?>38+ZML7wbF#QORs2-Cczd#V`?Wx-cvnIr^8|Wewk6h0ZBCS@OcfK) z-my7vxjIv}u8DnjT%KFru9T~LqU%eOZQ|L7rF5%i*Jmy1{U?+APtKcPOqNqH zVi{p|WNdCQeln$X8H+t*c4h3IjJ-5dydL8CU%L!8HOv?fp|vIU-qNhelvF_?+J4A- z`_gKkXuG2^Wjtl@X3tp@*j`2IyM$I%a#jy-p8MKJm>h{sukW9PUxFDk-eO2)Q@yrt z(FPb_+X<~9G-#>Zn1tnp0LHJf>617IK8C0fk@4LVIN}qe> z$ykeCS0SF~+xyP%;(5ON%J|(^_S~zDci`tOqtjF(GCh{EH@&Vw!W+N6PcN_Xfd8+s z5dG^`Pq&HqxTXf+-+JL;4e@syYXJVjOLx~3zu4C5?S6*%Z*Dij##ZA&3-QZs6u=v# z7vY+92g`|1HCqoh62DS51N^nO)p$rxd{$Ej@TN)EU97z6B@wRCb=N9y`bdQJhVE_3 zn_EeQ&*%;sm2(CXp+$GFR5@2dA}rS(tW(ZyAQ3h?4{4N1HHlEKJ5;1hx=DZm=t2I% zI4)Q{Xo7} zxD(;E(r<27ddE-@yUuYBUJtQaS{)`~uaq*h8Xs`_t6Em=}eSd;BL{^n-s`!T|&>*CJCM;|Gr$B(P)>oFsF{E5Sm zP!CzjXzt<*dVS6}YoW8b+4!Se>F0{sU!wVZ%4}Bs>p)DT2G|f7k(kvCInZ#JDr$0v zvp$Qcg*;@S!Ly>dmFpiFhSSx9_{fQxo*u9idV0A3f<%mdhUo!Z2_tL3XnUkn*`6MW zULB|{@?SWv@$ zIAWfMm#v`b{9i+1f>ZhsV&viaowM6+B0w3n_Ik!l2lPfYyFw< z?$N93|NO;t_10wd)>QTObai{Ox;=G-W88 zXosK5y|Lx&mZfj3&cvznrfGH3S~IaXV|Pz^;V^b$|36YCQ|;5fd8#g>v8OfOq{ch1 zDOu3E5<}Bl=C$<;lsh3z9luSjlYUzuf&j5oJw0G{;`s-+9|4-390pUsYL#q1RQs#} zD3$QLNa4>;0Rbf?jm9zr?sFud$P$yvF77j|sE0ut!59q;EP6}auMo5#*ogo=gshp1 zNB)@mFATuXRkI_m>VuqoDbf8@kb|Ex;?WLgM&Lvs4RU^brLn?3qOSz4twR(S0z9pN zUunRPJs`_x2e$x^=qmCY09d$5@@vXKQvauzAlH6DSiT?((x2`N!uAEB{|CbJwUw~B z6TN8eT?n5K&sFcZu1vYxQ`U|N(>=9?wBD;~AS>?mk_Sk#?%s=v{R(*c0c9rFFIEzi WadP~Y(tC%}{?W6GN`fkuc=~_*3M23U