From 89d07ef724ff90aec8e06a3f66568f81bba562cb Mon Sep 17 00:00:00 2001 From: cosven Date: Tue, 30 Jun 2020 16:31:59 +0800 Subject: [PATCH 1/8] Add docs for rocksdb --- TOC.md | 2 ++ media/tikv-rocksdb.png | Bin 0 -> 27160 bytes rocksdb/rocksdb-overview.md | 58 ++++++++++++++++++++++++++++++++++++ 3 files changed, 60 insertions(+) create mode 100644 media/tikv-rocksdb.png create mode 100644 rocksdb/rocksdb-overview.md diff --git a/TOC.md b/TOC.md index d06064b0dbab1..222517d893cc8 100644 --- a/TOC.md +++ b/TOC.md @@ -407,6 +407,8 @@ + [Overview](/tiflash/tiflash-overview.md) + [Use TiFlash](/tiflash/use-tiflash.md) + [FAQ](/tiflash/tiflash-faq.md) + + TiKV + + [RocksDB Overview](/rocksdb/rocksdb-overview.md) + TiUP + [Overview](/tiup/tiup-overview.md) + [Manage TiUP Components](/tiup/manage-tiup-component.md) diff --git a/media/tikv-rocksdb.png b/media/tikv-rocksdb.png new file mode 100644 index 0000000000000000000000000000000000000000..639e38b1d391c95bf584bd8ec556849b6dd18678 GIT binary patch literal 27160 zcmeFYWmH{3(=K>$0>Le~2Zs3;LCg}p?MQ37k(@B_^-b&7{`gDu<6{bGhd8+4vw7O!dMeV6)yWE2U)-|8>frk37tMsRz#TCEgCR!YA>+oE3lU_j$x_=kAZ>l zeH^-5yb-!i7p{gUskVUz2J~sAm*s}=3Yau{_A0=Lp#if{(4`wz^H7m@# zW?D&;Q_LE3ZW-WJjC?iuSarEr_3>~eh1ufYqS#nX0srZSUpx#@B)fP#WW ziZj!22_pMBU;gVO+3>1?3x8AY8KZvnKSd9Af>r04AsY$2#~-%?3*0^tX;kPQ?C*on zF)=ZLmBz;}RxYNJva+|2jffy{f+8Qbuyt=aMNvXTp9P+bk$b3}HciJRB|rZeXJzxd zeI2*1TBtUi1qV{}kT0vG|9|u(@vbb$fw&)yFOX5Ee9k=E*X`lzfFSxk6&00$fZt+` z+2!tNX;Bd^3(M>3Z97-pVb^Y)HRDZfJuyo84JskofDu8j9AKzNqGI)AAbI@xbYjv> zE3HqWvgzb*6~F35U;GW&5Asy4s8x~3-^(WQa8N0izWe!KYfKt}ko}1$W@u<=d3hPI z+J}b+!RG@7QlCG<^71hvIpI-JL>I=T_6so?_A*h*l^;U-L+E&qFPe{tqt(OeyXi_2 zKCZhA;mj}TEM?*y^4Q5wgjit*f$*z%w;NFR^lC*p=yP&<{kB`xayrXv#e^sA@w@*S zIt``wVxE5G{DI4xH*Z{BU9){}QWtI74+_%ba!4Ljs@UOCRY$GoR}q!b~FW_nB%N*rG3LNj9bpX{e|=k$~tClZf?3z zp;Wa5&`3l?L}kaz!{gekpNh&fO}3A+p5Dsp>gvplX}xx~_h~boN&#Y!!}G)SU&SRe z>mi|`B~w{W=8C`nVY$`Ue+0pfm&^L`bJB`0nGsQu0jzgUwzWNbv+!XjgD&lT>7{b8 zt$G;`qu9P{ZoZb&oTd$%2|}}I$Se%tOcb8@oi|!Mh3OE7vNX_}yzof8kE?E zo7jjP3G`8M?eMO0oRI*Bo!%%--gj74Y4Oz(`9Cfx5p5?JL<=ii93Vnog!6f(c$fO? zex7RE?hiB1y|GMSuMk8_VD|!YCrVQf*K}{h0WQcaTEcl+j*CgRX=1~{^vjo*Q}>SH z!NKD4a(@4pCt~+yR^SpBhu)_DIw6Wi%;%w)l!+v~;WC6;czcN=uf>_R5E$1Eom-6DrvNiD>@+y28m!S1 zRO_$|%*t{576WaC#%ijPaYyF8M)^+46aU{>V9C_o|(PT zDTsKZM4w1QkKg3et*!;hA}S;i`7q@F*yv938_8*l_mu^o*pk^_zZNabIhrPox*iPbpCu{#e1qtG%@I zWiUq>@v<wGE>p1x6_)W-neEeyKf9HwzbFb@ zolryiF=fEV+*5Kp|$v)3jlf6i(~2SrN{q)=B#MeYlXZ6P{+BC22|UvK8< zcD?;VHpc1qrd z7%;Jalfm#QyelL5E8-@Bfihl63PFAvNnRF7c~*%^GrUrYjmIq2Ur$(}b*wScfT)Cl zRWK7uuZHyY1MYQC-+x{us5GVnL!Ek<+FKkwTACl~0O zgfiSn@e5xoB|Mtn%*}fkt}PCreIN0egPs@4`WYR*NDOf>)glyVgD3OTf)#aSJ{-u! zk!Q{1Bm1IemWKPz^vLjVixlYDTSFgrQ5T=O@z*+?+w@=4a22F% z{wk;1pZ6G+uDu@_#D~^a7wR*@=U27P?q3S2gL`RDM%@%z&tu=O32INOWT}Z>EQ| zwv)#pB}q#5OG;WhmmUWztTc~aDqJsU7vy=nzf8U>dguKk=E=`|(o&FK5^#U+(C)jx zJA$3IdI^f^D$ah0kGE!NOpMF7Je@@2r(diH{yJ$gd^}$htnGf@V!0(iSPh*`KE0c` z)>9vIYO4&?j@j-&0Udx8pZfaM+Q&XoEpb=8M&pu}AcTKAedXmP`BlQjX-37osHjXD zu>KG?mS}@rCqoLcx}nwv9^VK(+m+d5dKr_P9AVr@AW|~q?`F;o`epCQP1F3y8&B=% zN3kv-AU`MhSQ9~#-#w(q@yL0k*14j^y^w|mx)<+vv(Z^TqkFRz=Dngv6uZZGe%~WL z8+PAYQ4ez<KQ3m$p3YqQG4Y0fn~`#OO!|Bq6e}gFfO307Us5#CV+#15$zpV zLgI4C);KbLd>+4|d0Ij8(J5=y=1 zhxH?moRy!oU1n!CTjw8vH-E>r8K7!Cv_D>iA2)LD`=0tAWH6X}etEbkW1L6p7-6cu zN}bWKzXV%z*=d8XuQ9hK+wiS~33FykLe&Jlx)-<|+Z@jrbl6~pC#>o*w9D1er?__g zkFPKKOpAL}uP&p5thu$zf8x%)uy?fXg_SOKIM2FsRg5lay2uB$S?v`><2!RUt1gtCZq=tZBd{KH7tx3e4lzXmy-kO1M5<8;-%_v|Fxo z+ZvC|*H^lpy5&0c+=+hPk@V|OQv?eh&&@~jRzCe<0W^UG>p8PP&*uxhh@;MOrV|Z! z+PGG-e~qM$+q6sT%}~g2RfyciB#~s+{AG=EJPO<SI2j?w4@*K(&L`G0^x_&Hz|J3zVz%sEMM@zq8oc_`# z2Kn*gLC0e5Cb*4r>F1=j!s*S;oSk)Lk(ccxl;2jHt+Opw$M5$PxgdNQUo;rRu!Y)r z^5InGDr*i*gU#ySY(9PSmJ$(ceeJGW?-?x znHPINoq&v%MqDw%q0Gl*z70=0>;BbXGXs+4wiOk7B1p@<-WiwX7s19-hqXYxaaZ;C zI}BNeek|6FrE+SM+#O5xdDcT^dfskl%tJDGP@PUL=0gPI#p!h!LB9Eahj|9VMOr_j z-X)D8Fz$|Mow07;slmxo1M|xB=8!F%{R#ckD1EhtYP%y#K4flqwN|+Dg6*&JGbR~S zzfRx&xjTjNZg4^K9&RUKiQL~^cRux1%#I1HfO#O691)PTXL86!5!U}^&I}11OL6mA zbXLkjOV5HQdo-+9OER-|71zpFOWFy}l1DCEcFchk6w#e8+Z_tboUY!k>wnIsbLQdt z(o1p+f66md>0;lR(T3Lor?GXAdf?ltuQIjRKQYU&Ird2ciP)YJ&r>vDU zD#_>ltglnBN8_ZS=O=d5rua_#dcxiHB~(sKERdaKOqb~Bf%EZzDQm^`KB6c2A&X_7 zYHZvkn_RG=z6o;erIuwxQsY-ozPNWpJ-p_geKhc(7V9lHO?uOF`4ZD!_*s{{vJKp? zs5;er@iLyi?{sKobvRNZr!NS8v`jrd27Buf(o2k=`|KTsIg}PNy88aOZj&Vjc{lnC zU7X11o#7HQCx3l+@>z4y+tu%{R8SmQBw=xv>WioS(@bk}QTjDFpt$twGnt{RGy6Jt z-<#Ep7!Md#SXTA$EkKxZ1Job)^e)OUqC9ek+S9gOQ7BGeuMpzh1WKrpd zM#EV(#IXf16HKd44&Gf6ctcPL9aWq8_Nk3bw*rR2MrQIn7sG*@=E$X4@(SuudwejQ zWy_h|EAwhH{YVCfPa6v|gj}B4Wp~BUTWC{{Q9+5{7uGmY$5?AjpG>|hV9jZIkKu(I zJ|t!poN1N4<@jp{0glT2WsC#fbw^bKm(9OzS$;L$>^G+pCe08JTzMUZ*h!skX+ClW zBu6Oh*vVtXK27>+`$DEmea@^{79V@ov7kD~a=V+`WKvxRTV5I*9pH$02#^UTn#>u1S{+{Ji4lUWQZO zZr%1p>sQg;0zHq%Hz0*pjTUaE`afFpo?kOi+#k<1%eedwUK*;V()e|b`LA-M>0c(N z)H)au;vS_pl6JWHZ}fc;z*i&ss`{alC~fm@WfuJ;Cb{H`Ra6YlenL0F-@d+`r-gWy z)+TwJG#vOP^~72r;MdgtYW?!ux_w%y>bv;*A=9^o=LtDIJRI7RPnY8j%cG&=>O({$ zPZ*Jhare=x{ndQFS+jk~(``JZq>LLhXsU7VozE(ZhOg#+dyUb~DdT*7U`f_1*%v?i zTJJ+NE?hLXRlS*`l{VUrku&c=gM(MC$0-))?1j3V4%UXS0mz@(WV={uMuQ zVTI8-PZGZp*v8tykKgVltzPQLa`^c``~P*K0R%L&!CYVaFZn5z^X`hHxY`fF3Q($p zgnm2pAN93cdiZTF=faZjS}|oNhb!V?bV$x_O~XF$APgScFX9SVZoh_}(Ceh6m1uph`>oTAHn`bz3 z%v;0Sbw-AD^d;2yISt)zoIQE{)b{=mtsoTbZ)+@wiD7^0Dkd3% zFC(epxuvE-C?#nToB||)woeqE|M2~fQQXies@?7J9SIAO>6D%8NRlsy9$N1{%|%46 zozZQLlScNQ?%QUsrmFnC8h(2%JQm#zyb!czlxy(eOy~XyD?baWmt8OdCH!MR|G1ynW z2^u%fV1~o(42V|-*Ez@1!6J@0f0pr1gvlTq!qCn3_0q{+tIvehH8xk4hq$km`nbkd ztiKB$PB!dRWs)-U%S%PB4WAw(ms6<{55{>u0653Fh3C@VEHdcG77#>gseTLK^lH)P zT33S+5VsDal8PrbZdx#b=4HHo?;NRB6ZviWhwHOONc$fKz%LPK{fn0d1h?+8fZ&k= z3F%*$^*@m2|Nrt|nEU_7nqXIr?=}7o>SWeB%->)fRJQ+qOo@dZEiQ|b`zCA_2Ic$t zZR+GC941-jSZvJvZuW(9r6K*`^Q8Wp096uV(zYwEz5p>P6Oc-%#ikM*FNmPAN9>m+ zRQM$)5u>#1Or+na-u3xdGOFzW3|WCj0`^=F4LsFn0^}5WF)Y;5u)$F~Zl@wyyx92I z;XM0c?T+ODudP)|TLgPiE)eos};(%Z|NK_|smK2ttm; z-07uuNb!O(+EPA)U4H@ZrWTpFy_rn2+HCG_DOFR<1ZXde&j_pyF@eR zE*{W;mzNAD=x3)zA_KDC@bbkn?!1#*cAYi@^0Z#0EJWR?VgP!|U+w*{(_&=}l%d&J zlIsCzDw}tKa-i<`m>o2*sDrVM!gc>aExd?V9qmWjtR{%;QNyeTz}a_+K_o0yEvu!x zDX2D!8k7)vCs5h48s%^eaC> znX^UD8LFvZ$B#`)P*=n6U>rlx?N7s`0%1-p)-ug%Kpix{9t+H{mkkfAX&+LyaXR)0tBj$o?Kt&TctkB z%(}k(!h}jeZTp^b4-5KbwpTtrAfnXaTv{h&T$#Y$km3z{pp6Xo=0D>pqG>TS9(zpr z<}7}9zZ{*U0AbJy=q9p_gFpo(6lBv!{I?}yL<}Dq$`sHL0w^m@8J{o0I&3|mEqSfl zmvc3s+_sj?ox!7qC$-_mTjF%SSx3+fCUPIpj@?ZX z_IX8~E51yb57%n2D2KuHk8oV$Mh>RDRu*T5{V;BV>gU`&V@ak0p+h;k?Q$9hZ+wea zc|UzX+038;rBysF?y>lMwURtRYYZ%46-(<+Lnl2mb%!ZuqtsE|CVns*SHlV253}2A zYrQEaRCOIQzyN+kC&H;~x#|6Gc>DzfEY6;RdnMkcwk+%r)5}4-O1?%0d*Z5!PWoXd zy<%%QaC=MBsIzZ|u115cStE{`Hh_s+qq;D6^m|!iv;H+&;UqJEWYn~EDybTn-2HZz zH9Or>`QEIE-;yZGkyKyN1ZHAqD1@majK?*2@D?&xh8Nk2#e=#0^oLf9JV-6XQJv# zZ7BfBYPtEsO{5y?*UA*DwzQY1($CNT^n0KCEy`c-hrG8pc-AVh#qzA6yL&Er#jX)@ zk{Rt(IHs<%|He1#X;sVglUEJogdVM4yL{3@*MK)P-&UR+a z`II@9l$10#r#`Qd+|a-|uTeU?G&fhETwJGJUYc{f%m7YPl2fRdo11-sKm9PKmgcgS z>~d<@a^}k8E?%(`Kl)|bsy=5W#52U37MqWi>o-7OMM_$8*^&T))x(^!bcu+F@F4e$ z!KxIiLOqx@OaEvuo=w%O*eqbH*9DOOwbsKdZGa3!pTvx&EFFIg~dbVEXF!OD*M>a?_BaVsF2gmLGhs)8fxC#I_0N~CTB;QxW#KoIYjfw>`-^?v6 zETkHaj*bGv+Q!BPfMi$bS+2n%qoAO;9aH{nYinZzx8O--CjJn~*&kt;p9iSNd5tXQ z9I2q$?v(dZ@ryQ`>7z+@fDGN;-FoF}G#Wa%09A4v*B=M4W@G#YKfx*x5Cc(w|xx_b=^%ovX+&N zQZGeu8xW(6n(cl!wXmRRV6e9CxYZYZb90lD@@~FeZXN|KGfXsTr>{?9-fpUX-sMDK z*2-1ZUmA|{zj6U&Wo@`7Cnvj5-Kez6=KxyslpC1)J1neeuKxED2x^M#92~9%>gHy^ za`LD!F#l`x^z;kgF;S39!+ut4m(N!kR31s>P4@KmHZLUuiUUx2U{@H1aeu3;t80qj zoPVirZ`Y6i)zs8<_pbp|pls?#GoR}b1Aui%G^wbJr|RwRcOS2QqfQd7RWajKTv=IJ zT3QP^S-G?tN}y{^3hKjCN6V9pr&8gPlam8T%}iTKAD<_xV{X!|B?yGqy{-OV0~QOo z>D2}>rZn&0$JCP?0CNINV2lXBmiMzv_*RK|brH~*D`p~MW6wrPdwRt5FJ(-KO2ax^ z0Nts@wzRYqs}#QX>gnr)=jFM803@O8dlfmk(R%B7D=RBw;Gis@$z!`zXE|Na+uIxM z$0uKSIh4qd$qi@-xV4n}#j4)xN;EeIpy*t8W`S-51Ox@1ve^Q@KGzYdRrEas+&QKySl7@x`4HeZ$8i)1nNUD^SR4Y|zAEG)M zWkE~J13+owG3z!tZb>*fIVmeom1)&3bbatl8QqluxY$;Y>XH(AfGy+Z=BB2mrlYg` zm1JMLIF>DVHpE)5eKJ)2>lalPOZK0V(o#K7vyZ9D|C(LP!TYSR+;!vY54yl8JkGna zW@h;kd8usXV{%+8>?VCth6v)qZh+j*0K3EMUXh<~HkQE+{39nNg};=WA-{JpT{xb_ z?|pN;h!1E1=m~t{$+RIBax?6 ztV;0qtLNozb$Z}KjV5EELN+HsG_Y}t|24oI1jcmq^z@XJ&PTb{sNTOPcncrg)p_1r ztO7_oX;D^G)O5d_uA-u{H=2%j9B$!cbiLjfiN^w9S!FdfP5?=UjLWz_`*u4LFpeKI z3M7JlK7CR6mX?;a=Hn!W#(9&P`uf1;ot~P)WzJBjZ?(LVvF_#LIy&De4kjC*ZwZ412Yv0Z8gV6*K zQBd@1zsOZ>S@qKVToH6Vd!NM&K>jLQL3FGQAJuv4&-~{Yda>tW+ zJQcwBk&yVEwcWz=5~Yz14G(ub9G3b&6#8$58)1_Xy>&(ZQrq0xnjsm2O?JEH{|YF9 zMZa}mDo{ev%gYOR6=-Q`Rl(%Eu@1WBuWM+qTf52o)stgTv$`;~y0FB`X0-3FYEW){ z|0wxF&BSCoAoGs59avv?zuVob9z#@4hl8Q$Y=PIvYPNM@SJ&%`OwaDY>LDv+*pXC$ zD+QP80?W(ThEc#ufu3;r`T70=e_k;+gSbC+UuipCyy$d+w;o-^hzZo>D*DF7X zX6m*ASIjp0?UA-~kyN7F+uA-aB9D)xk&G}1R-_ShJUWc^SlqXz-QC{wSmuX`_zPa$ zhw*nlbyY=eFdH+nQt7|)b|(wJU2jVg7uE%9)KVV3_&fmRJ>B7LSUk>tSmg zoD-pZNa`YPa0i!N9OdLiH;y<4rq#o%Mbh?)Hw8~m=*Ji<~~_mMW` zZZ=q(S}aU6_+5ay8QSKOWJXaSGh<5j=RDk?CbvovRVN2$MllWnFZ6oynDzIu(%e~N zvn7ebc3_d&t)`QyBJ3_XPTPUIyHcW>^Q%UHjTbY4`bUFbp#5(zVc+C|y{KFj{PSDT zfmmQ45h0BxzpO*wnz@7|-XHt#60p)k1;BlY}Ob|DxyzI?kmj`z7*_9uw zHwu!v@UcGtSZP+Hf!wx@54crFdmljji1T*b7ed3{h}fp(J_e3{yZgTGAfl^l~AjI6CW<8XOJRFjO1E{Q6GTvd=V6-rB6 zVb-2;z`nOT{J2c?NatcewgvSIt>dd9j~tokFvBpuSO*eFGg8?Vv{d0`a^9cc$AGJp zoZ{#{7w_5P=R9c4D4=ShiSE1I9ZxUSXu4A}a{pk596xl6N6F9N+wwU5PLR3e+xUnq zAp=@dj+Fnd2nr}vF!&y4rZp(-H99Dfb}5XD8HN4F0rb?oh#!^?=J#6!O_(~I6Sd~U z{d{ZAg!IuqG_DHm_?8V@O0vX`8Y^)xX{%jSwh_Yl%Fi>qX~zL=U(9bxqPXc8=8`+L zTQ=HNPZT1-EM-<}C2Ea(>dlgN94DGp#wSWLhMNUhUsFveIQ^iNxQ{=i$YYJfT@2a4 z2c&Tswb6iV=bI_W1zcwzatWN}&I?dvP$usSf4w+KwJ5jHdy5ZcM4^O@&arS+XvU?0 z*Jf`+D>Vpfk55Z`xH5`Ap!(P^JIjy&Fg-{#=lp) z7w$1GVt+3{e{$T0MN)u@rfo)4p^kvc!;qAPsWANf%UMz%8g*s>?Ek$HtjVkymMfSf z4f+jbN?gV;?Clq(>kIO*ynK`>ibFKqsUT=}fUqE0-`I9{R4#e99330&-XaA&ZK~jc zrViqh1Ad1kNBK<5#6mfZctG8URb5gZ^Fu|}y2m8?z=3)Vx)(l{fdCD}liK1Lwg$A( z1mM$h2M_N_w4Zz=p#wF{4j{Na8TUi!h@&dH{3=<~RknmOZPTUiNKU@Z@>y@bN}k&$ z0F%h(jMEKuLGE`d1_XUz7HVGX-1wr%%SgP~)N&Cud2m1_@?;O^VH5m=$_=(E3qgi^q>7sS7vIA#=Z4jtYl>U9W4+hz?&pZlU?!l2kgNE)*ibQ4g7 zuQ2wy;-juM9x#*kLdq_~)s=!*?CG}*#8_5cD4KTN&thoWjCO8!C=&@lYR zBB%cJTUs8q_}Om9m5rUi@}466Kt9v!F*6SvDFkWe^NJIrr>8tm1TtzWNM_PR{D8&& zbJnr>Qa|nsC&;jy1w_V*#1Tzf3IBzR7Y-EjgmcHa?Fx!>M+@0gQV+|J{K=8G?nb2f z_S5+)&AY=BCEz5Vz_pnBP zK=?*~3;A0p3t1s;q}h|jMV5*DwttZAE3zj%Xn_xp4s29+JyPA#Z9Sg1DfjNH)*)Nw zdy3bkik+sP;zKlX&QLTj+8=-0CLc@?eoIvV32D2nK14BMH|w7XCf8u`ds3EowZZKt zZ41QIcd(4gNir+mIXr~JNmB;S!hlRyLQQdi`Si}}`8$E;1ivw|4UKnt5S#U7T`nxY z%78MUz6~|`n`2C&%_W!}LK&X9H~x0L!jIW>AI_IVp#XWyEru0o(Lv=RVv|-3rJ934 zDry*Ui*lpjy@Wc)eH(ixh31PJ0277D(|EjxOLe{^a}t(tgV+NNdgws|mmz^@{koK%lO>qV-gZ@4#?(tQX8T6cKucp?3cVlyI| zh*f1|PChj+^ujU3X(XfF!6uDrZ;z60bDG8YG!e)2Nl2PR zK9wAs@fY#nBr+8GCM@s;@S#x|0o5|P<-1sC0GTv z`jf>x#prvixF0wH-UZ`NJDbFn>^B~k=QG`qE95-*l)-!PoCD>retH+CCC1>r1rbmc zr?8detoW}N_QFYjEc?=&)fGKGynZIgVrdhNb52VJMBXYZQm@Af%> zs@eR!hD=53w4k2u=(2Wo2{TS6nFAXATHf#QoBN*sx%}Y^UcttGJy^p4%--1*owt=Z zFRQ)-tMEI}i4iuG;e5)@B3tCJbm1Bu^zO(@o*xdpOZkd6Ku`S5u+NmovJl3(fW*)J z<=3dilYU{>`R<@3;&t zQILlQ$%b8UeK9NZB?V^>5e(iXKnLu-`mQdz?`3r6_itVal-Mf2f3HT3f6Nd84*WKA~1z^i#SJgkj{#ohh5{|nmdA2cBAu<)|_0?$q! z!<-Jv@KpBvg7JDJBa-(8_x9l<$~xbr+~rDM^Edtm4|pjYILs+|v|p;hVR?+Y2yp8p2L3?(petI}5us)rzTt9GVtcmwPkF{Yym*qM#6|tqLGQF7W}0t3s~;bgtiTV0)U1}4t6Mtp2o~Df zviGDE_V!w1Pr@;Gjdm+7ri1alk$4K}oTL3ezl@?|70`hU3+^5a#FqUZzOQK$WKTk^ zIu%!Vno#4abNaCp->y*n-niSe*g7BPl*&uvWwBXg(vQskW?!R$7Odh?eEulUmi89S z4zjUhrgw7*Mo6>unuF51u@-j&Kc6(N#Y9E50&&gXJjj`unXWxYP*$ejOh(aOV75km zN4xCxWQEDmYWMHiH@8AVzO@9AKdcPqd>v2ohn(wN# zaqxoEK>eqBDKauL8mT}=T%5x=|Lw1Wf|A+bs3W_RRhkkMUv?T7g?81)x4$~{j?!h6 z`rd2R>A&lvT9%lE&#GNLw`;Mnpr`DR6vWI6H3VjcOv!b2#i)t(#PZyojK1Vuh z1et~{kH@v+@L6OZKPtxqd|?FuI6d7t3ZuXDzuu9)lnJNQ6WtYvIu-f9k~FlptlvXs zKqLwsbq?Tr8_AJFQmS&#%FR{xk0j~e*!-K}_s9KsIG}{&kADm^++-hmFnWG`kwX<0 zO~0TKyqEGngJ%ZAf6f8GF8~h(S}+2d-+3<{kP*zw3yBpEDy=-BKO2}|B~?AdNh zzk2B2WkFqG*EsO;PkytOIejMv8?Z+j74CJnSzS67`s3Ht+h1;B7qc1$#D+mi)=lts zWfYW@P`Vv{Kpy>(`FIw}n>PS3^6hdD-%x3~PRQCe>~bW#b$iNWxZp%z_i;61=JpIP zf0)Y+bmkxDXz(_mcvzni;?bn0&5sWz)#QiOEgE{Me0`|A+(WpsYt7LFMh2awjqVz} zTqIYV70RaO&L82i=$Fk}vFKg<%SoT#W*5xrCkZ^4TyIhzY;D2|?eWmYgFa;h7@t1V z)koTzcUZpU7Ho|i^znBRyw0;(9@NFyLK^N@ZF!X+WO4wQ>h)p11OBISF#x3ju$Lpm zwvjti9z0)+FsE?QWRL3rlJ0FuW;-qYA)mmN-AqCJT4J=H9p;6k#PhOa@L~qQF1py1!U4I6+8LO1VY_rrAERfZH6{$f*` z;|~ky4SHorL)7Gt`ugXOLpBTXI|cx3*3r?CmL8hQm%(Gw-dtVZ}^4uBM1bj#tYB%HJ=OMI)Co~%Z?W)YD;-7 zOv}h=d-#c^rH52O4Q_p&6_9Lt{yBNF9lPLz(2f8I z(f1?GLwoNhR*`JuWut4*CnL*XW}iQAXc7S&mh|(102L#hv zx7)cfvyF5OA#fm%!{Wj(qiHypoBb%G+H&*Xn2HbkT5cWI&?s}^N2pc8usTya^wz>A zo=Hb^aA81a9UeGy-|w-vP)iX$zZZLbYX*)J7;xqM5tE%CjA`Lfx!`1uy~LYgK2-VY z<4>4G!_T8Yr;iKLB(#ZFb3CZqiAiP*s2~ndX2n~05bn-=#YSOLJSeht39ovl>o8vH z;c+(Nis^~{Yid^YXqjfJ^F3Kg1-!yw8i(LS+sl{hl}8+N|74f7oJe;; zLuiLJO13Nm{pZWdrT1g?pREifs!ZM;5oC|JqW!6I@ceyBvpJt{9RL+iW7BVJ<9qVN zIs$nP@>{!!G*##8MU4z&CVoBUamPID8V|@z2L*7x*8;<;(rht(M18VHfHN; zCvm_$md@{JqWOPtKH8inbe4ziG=7dDTAA}1&bO^tqyb=h7*@5d0F3vYA0W@$JRpb7 z0mlUWU_1CEiM$dNE$3kT?RTuVFIv&JP`(F)o9 z<~iQ1zOqZy@Z>0kOM}jJBqIfc-o2TUvWna<7A$fAx7swq7AG+&%?fZ*Vj-gxH*#2r(p8n?S9%==rx*Oq_atg z)(aDWiEJ?D(RUc4Ol}2QKldk!uIeXKM|-9h42cp1T~<@WS|fjN$23C&b&eSrT65HN zM3-jQmm|p+ znTgS6Hz9^p7#*-Azju_`!6^;%V<+E28}LI69fl&?llgVpx0*k|Ze-1Yq~G&U+zGSr z0P7|nq;x_{_Ato0VxWYodK@Gi-3rd}#h`eV9dx$=Tc}nAT*zBN1uW;ElA<(H5Egyt zT80NLR=@~zIFUfX6Gdeg2|n+n^oCr887jgA6iIeP3Gs`BzNc!?W9QN2zY?81DGjS8 z7K$BS>lySdWtTS-s@!0dwCWRc_-(vgd8uYN=^5q<=Kfy(1RZCD&*X)@PR&9vy z8;F%agWX_+N>Jg6rvjs1BZJX8U7!iu;q6e2XW0EzP3)Ts1FA$bg39opw-=r}S*mVe zwpSuH*T_l#?A@cYA5@*>0?7Y*L;YULKHv}V`g zfs|CEZ4?MI5`{WiFuFHNrd{_KT6HZPf}Qv*9pCvIx2I2^0w+TNMz0K0{{cl-ZXiP9 zz@Q3?wvGDp_eqbwhDhiL4Y>zPBrCrNCR@$xY+BB|_5OwUk9V?S?K1L`i?>Ds!S!Gz z4!rzV$jXTo$DX;OyV zvL*<-Rp{%{zNPLbs+pK5$ikds9RblF&ECFs#>m?gbIr)-w;QPCswbBXae5cV=gL(k zn(a9=t9cx{IRUAFFJ0Q`N%#O+|8cYRgkrlJXO7xn?RtU`gVvUBbtVKR{@(r0YUhcA zeDMge7Ay2$S{Py(KqkF4zX529)77>|pmL9BOk@`zlQI_j6O8P}yTi~FvIY3LxjjE2 z<6A?V2Oz)l^74v`_H)8W2>jlf*rt1MjBMX8j{DtolfEbxPNSHUZyAd=zjb!jYxlVa zcrpK5cYm&o(5Sr8S!A6Tq4YanwC<;8y2K#_5)VUTnm~iE&2;j>BLFt*O#^?}lVh2> zwU}5BEQA2&-3tvsWPOW`CH-#L*KTCTTsXbByv*lWQ;OH{6xeq}5*`F(!G{L<2Vll}V*kdh`$Is6ggj)eXN4-UH&A@`@z+DAClTEH}##j)9s zb#8_MnlPC7x5w|}+yLWiA#8R&o?o3tZc0L|jp0FGCBv{kwVnn-N6f+~O;q_7b{g4A zL-ig8(poGu$_94-?PG@lhZ2iQ=a8a@tq7#V{in}=iSuVa4mwO@R#6D3=0zfCIK%!_ zCP()Z$cBMKc})6GpAn|qpBt0p{}Jl%WB&D76o=|vra&UlzZLq=WB!VEtW)q$w10!$ zFsZD60Mq_V(Ge5w&J#r<`mY)O_h5f#*jf<{9rkyy|BKB3=fVDap8ut={~pXalGegQ zkMMs=R%M_1zZmR)uYeZL|0&sj>;9iM|7){_15Z0Xl*S?|*2lo(?YIpH08Z11RLg>}*m@Oyg?JT%h*N z*f`5`H`Ne1ce1*I5}ZqVJ)rHqY;0{#g%h^0un3QiwzaeKyBMU|8%f1F{@GVinC-{I&OT$q2~-0&*3~_2(+En* z%D#0|+=AEfBdz{;qOtp-6z;!r0YpTCx;L79?wtTC6nMUbghUl3!5S{Jt_P4*0MO_F zA-!;N1Dvv9d29?(aT7CH%5-1#JebNa0AjfkdJs@>i}Lr| z&ECk;k`hMTNNZ<+Gl%2NJr7~(hy~t23<3fIbaaE)zNV(zKuAk6DN&G-oqb91@VJ0l*Y!TOz7?RdF%#S=9*OH}$8+(QFU?H19*n03Do~rw zqgZV2eR~d#$cB@ckkQZ(KG?6e4h#$!hOxC<)-M5lpwPw7v%F8=xP20SkNEp${ z?ri;5x^f)w|#f`Dc9SCcBfxHVzNfym2vW4m>7(f~z&Be&b2tJtn z@Bi*AuE{{*cyzR!ygaHK;})8ZpFx$^yU!LDMO}u9^ocfA5+T5c(piAXS5i{yilM&y zGM)uw?-VFxk4;ate|YznAbK~3_jFy4zCA_I-%nF>;h(1%XzEE?!z(dF_@E1{B9yJ% zm$3{j0|S81x8c+YP&caA?jI9(28vRFDx^T27l3pB5g!luv~~5#)zymW0|`?lprjAG z7egyS1E8VP1bkblqylxsfkL_>*ES$F?4*vB$om!@J$rNwwW2GX?=(|^iMWsOEhYvP z1tmH<8f9M_C_y z4-;mV%ig+u=vIX+yu%|*cexz3#K#8;`jMsI0x>R=pt{>bMs-dOIfa*xLHv8jPIy%j z+*k3&Rj;E7JJr&%GCp;Ezw?n)fSqr0^qZ^gK6o1&k;dl#0xWIG{1Hrb1Yn26f{w?( zdoMZWW#6c50beOm@?zi=r`u~fJf1Wn6q&s~991eLCL5Gjs7H@CH1dMUdU|^CahY4U zVtApwW+o4SfZM?DdM9;gdXz~n$k^vrWTl8rK`JFaBO(5K607^^vSHX(qoWo-5kSvR zqY3K%QXii?px{UY3=f)zC`M*f z(!7I`Sx=6uc5n3u@7;@z+p|gmPp5W~@7VZs@CQv+rW~#=8MT@*5Sz7@*Oz>m+Y4Ud_kO!{hJ7@jW)2XOHVJ^^1HONrjaIXI0gT#YjfDi4wcJi_g?V)q4A*#GkCtwd4Cr&S-9G+lS9g z4#yT&7ToUJhiQ+bM8($H)KU*85c{>6nFL(HcCdMC%kuKm-qO;w;GOU~^VHL3B#U&+ zliAo*DpsD|ey_*X(@1#wtKZ2Q0x!B?R4vRX;I`sn5ZZ~c*vMrNK*O6`IxKoXxH+DpiwMYxTraO6o zSPb9t)T`6UT0a7Y>6>~tJT+DKhQC5SM56X{`HiFX+yA4rw+f2miQm0vafbvA8UiE) zclTsNa0o7oySvKEm)^A&rkNjD@&Oha0}&K83B@H|hraJ>8UJV+hW~SA7)|6O)Vu zA`wNJH32}ej9QK$-8i3$r-!FV5L|L1^7I@hlg(Pw;aVse#g+y%M`Ka8to zQa?Gf>9?g&QdOkxPn64kDPz}bA_!y167_g%euwhBG@}r{U)2~ZH6s3{UzKm9Mo(9j zkZ@3Vzdwwi^$`)pzUq%4=2EGKVdM70PHDywjbR+Ip#P?6_e zT~#ut=)+^Y8c2ptfR;?({6E&}r_Yj~5&6m&^7>4Kx__~@o> zA7?@i8C68Towv4ED0~V1bo(6>V&iJ2IblkDvyyGY+jwOONl4`|c*?%px3sUf>hAU> z`%|9RJZp}w2BlJ#q#`doNTPO|YURgdh zt{#ZStd{aBJ+G<%d$?Q45WPEWTn)cJ9)qum{H4iB<{f_?{~a){LvZrALW@Eua`h|I z*UZ5Tu1WI{`;HraZquZ6uc_2g!+-T#(_@t8IQGv7IjQK~dPwxYdRaL9j%M`x!c(Va zmVZmjb=h;nZ9xEpZ137R*SR#S&7_Lww5DI3+2zS)j7kOTzuNL$bq2m(2TC2skB3y^ z_apI*6Bjn(rotCm<2I_959{+DxEj8Tc%n)FbS8{w;fojB&)$cX90l{9Aq4VWe@g?Z z7~t`%zY5`xUgM2LH-tGALOyc(-Qv$nt9?PDDXUB5S@+LNt@F^rN>_*VXDK9%8R#Si zG6*$XD~QK=id?xwtAc|O9K$$F9$Ydw8OGa*YJh$IVzLBn)>0sVUm|_Jn@@826&6x&`f32^D;2eyx&d~j& zw_H1WpQU=+YO#1=l{_xeITa7$Y2-;<9>i06K_5l%G5VM$sBp^e>&q*b z<1WuenpL~kwKlhQt+BgXTZj^Oafg`oqL^~C`*AOngKjX-6Ld2iQ zSGtXuVA-u{PQA%}GzqQlB&(Qy+3f0FMV;fTUxjA$q_@}MIbIi5A3(lc$!&0;eo+}e z_Jnl6UMc;$z1S4C&GxZTYz$BNONP!g!71*#!%suh2y6S&k>MA;Cy`~`4?^yvhOj@g zS>?F(-*lB;X)PcVK??|7I6hp@@Z({nWVnBFE-9By6CE_``w8X}+&_LnDIRL! zV5MdhN3W!@AaCK>+2h6;z}8vmqwId*pRvMN8To#@W;1=MMw)14)juEp`DjiWZDpu` zQ4ukNtTD%+fm77>0=Rq8M z6hnr!wWz@@7j)?3)%WLz4p|B=jks}>-4VuSDcV8b4FX?#;T8VA6+sIS&Ve(Tls^IK`B|bd{T!qVzC7 z>P#t}xUH|M`eelOcQI?kNG!|Zvx5v4F^_?!MaU|RGzy03-WTX(7yqg zCivY6J>3a1K0zq}hig@0Ml)Xv%R+pA-!3J4Nlu%>eJ{2#a+ix~YNu5oKd8%;lAY_x zTEDJI>b4_~7AU<~Yeoe-lW|Ea_*x`nR3cVFSBFND#7fn0op$9@1dfA;Efx?{1o-9QebB3q$0|qI{JPT+{UCaI{%~U{F z%mF+$YBP-(D_{U&T%SYo*C3HMS()PaU7e zWTuT;;Bi!v!(tJDchtOYt+M^ues;Rz;gfby=|mh6saPd9x$VIvU#O^-M>7s9i_X?S zsvSF7X;{RvoBhVgf;;r3A(wW`f%(rQY*D9=8hVxKi<(7Sb|@+CSLSc;I6y%B(8yk7 z2Px^pomQIY4AI&s@uNO6a?-u9$h<}T{+EJh5>nzJ^^o1lj?5U1i4}ophCc=c^%{gC zRAVZtzO}EPudNGyE00^oI`LJgW4H>xqhR&}3-R+sl8{$mO8i#m$n;p>sY z%kkhtlPn^WC{49;zEk)TuHw|h6YbGeNr6YT>#CK=v< zhe1Op4P^5 zY!Tf);!!~cc+`)5lwLrdP9B|BLnpl!v>ME3g(lsy?cCY(u}-K4jNjwoei;j;?OiLp)`yx+g!3 z51FaD9}ktNCVYMw#8>`uxzxcUnN3d#*x_v{h@G(pEJ2Lmaabr|5e3frOtqMkuxGJ{ zbW9rpkWarMIVHu!rsgLpz!1-B*f(@%1=Oo8lRQQXFEXP3+Eh?j$YXxNuCK z{p~pX8|W1S0zb1aVUmY736RUWt7CP|nq@1Di*+4RM}btf#kT+)GY;xm87wovrZB6= zaE4S#=H-2Gx#)Ti5(hiHXE(KD^Jsb^f$5f+CscDITJ}* zu;w2vq7Pg^NU+sP;?gy+gz6H?t(@f{c#9QyjTkN1qTCQ+@WUfJPEXjM)Xd#MpUOXH z38#ZlgwK%iAmjgxL1Z(&+;lDzF~-lHPdu5gcoIqRL6JSA0jAiVwZTKgxa>cjhh{Dy z>(@SGWH#RBqx_(d}Gi=!2mn=)Ov-gOOY&^ncF_40GB! zIk-i=v~Mcob6R^+U6cxgT<$13THl{ihc?SP)(x(qLUZ064WD_*Y|ducMv$9?CxzZm z0H7)K2TJHG{UM)xEaFN1t?!(4M1%4C#@nySayj0|{G2f_@ksE=!emo@O{X&`4s`iq z)V_spQUSD>nY$8WozpMbjIEq=#RpJQ=oV`Xr~lBbC9|*)U~oEZ#I?iIkR=IKKOnw= zTznAJ$z-*p$G~nFSMKkt%^tPHa6BXUH-#)G>`$Qzsx#EB$g z^6b^t@l(pj12@PBb8z_RJy{P<*sX;cVy!wS%j;^&92?*O)ijRBfH=Nc>9dk1lOT_~ zc*0Y1&ff;8vN?<;@%{@1rR@3$81M9BOUe)?H0w?nTm7L9(B$sDc^9oybNMWH)qxA_ zkk-;Aowz*++@&#Af{NQ{{84rtZbLgU*SSwl#YzqtRI$mm6wv(>%Q(n6B+s4<$DjD* z9*df5dIWcw%fBFc*Qz0u!?3&s^(}w*o&xz_c7a zYm&MuA_CUe9xB;#F29g-Pa($8T4AUO2H=`}Ch|E(7)&IM8l5)m^;|r#o=hb{fn|6& zwO?&6L0Ir(Z9G~e@xb*5UDnRB+}nup*JKDFVY9s0g-JRHrm?RxK++7O@Du^=qQ_}2r< zzJaFb<}b|i*AD~MEZ+f9=%RckR$U83OUT{OcUL?t`C!$?7_iDb^14YfUXSdsH61GsHGxDGyADp~P zho}IyU2Jgx54li18|Bh3AJAH}k_$we{u-0|D-r7O*GkF8ax%Z1xh;u0jqb0=?$NA? z5aG}EN>G<;WhZ`o3%kPK_h?nL!b3e+0Qt!ph@|6}&G*88Pm!@S-)iokZb@JK*tbGR zm9q#u&*vqOKOAE5u?Oio z{H2emvX@O)-E2nvxnM#a{Db;3)b-sO$EbpuQl8xel&&dOAHOk>vhypwz=t}jYRSe- z(s2a=Wj6eV;UBLLYpYutf>pRp*Ty(z0#fiOWftS{nnvf{oOw)Zq`r=wsd%k&2lh9y zK>94mk*LPr>SD8GOyJjWYDmqhX$HIzJsXT+6knmeRH2rf4k7>2Cw;v~j2sXtM%9Cw z&6JGXBPB*z2npp~32o%6Xseq$3n#XO&C0-X zwoIEcd?>I!aK&CC=`<_~2@n!#+xlM)@je% zv;#|o=9d*n>?!WWP{|9RBf`NhoC77@piM7YwjPV@ZjJ_S(Zb#=3hLTXpbRlK`oxI} z-0@}UFWvT~O3)YCLt4aUShss;7q4R1t3{9<&C)>1AQt}0JZ7JSBTG$#)gtKzPhhWd z1-_tqg4fBJnX~1E{n#x#7XnI2PQM@M{IR;rt3)FKsSrzmQftIl`uF zeyl1|YANDD$`o_)D@5&lwzA$No&HL*shigaa0KL#=F3lEC_6Sh(k)&bOw4U4>>+Ii zy*sfCFuMgGE9t!8qLNwqmW9sp^F@kAo}@5O^p3^GehG|N^0!x0D56!&OK!{iLvb&< zdG&k)WaduL?<+t_qJZZd_HTiG2b)J|aZ}c?o03gc}oN zfGy_Mj|~;I3-ZYM>`Udm=*CNVjeUS_8B2!J*7!HilIul$YYHo7 zCI-4*c|d8}gM$&x&S#a7^Sf)tCgauaT4P>g6*vXeIMW=8nT7G$jj74D5UhJH)h&tT z*q@x$!l>1cex?cBV?1w!*LDYKI|Be!-ye%cC51jw=#k|I3Z#69tzWu$+_`k{eroPJ z?}~+1rxNZC+y|9Hko8IWO-`@LQ#vkKrpToe6NEKP;g-eqDUH2Ph4kensIn~ng-@Qf)hR2+}X)SK=~w9m6m(w<>uTV7%H5G>BmiE@S~oj9~{ofo%@l9 z95lfpS?9>0StqtDC5W2MBU%CE^`J}@G06A?e`V~@ek@SmG*K zL;KxHFAb16eLVZW6F)D=zvdPx8~uJjiTcssjb%@=y)D=FpCWD7E2~cyu~*JtO=Ztf zt%9Vy<#_18yZnWhtUm%H;mZXQ+b!~5*u#ATd7Jo!BZo(+M{?8&A2i8cA#W1J3COW1 zPK*#!#{64M#%u&nFjdpGX1*aQitIbsT>jSMBw1v7W9S^(^Z7!-^&MNlS7JqmXW!a| z;UMfV{=QvJHGa*NgyWc&Z)f8;$cy`vC5(dcz7L9iAA&B_G^{cIj^n`QdEZaIwZ}oL zd~n>y%kKF3z1eRo(dT@(uyirv<+}FxcZlchCM+SCo$9O zQB<-;_ljw_3)Lyd#?856@Af>)arApw>M+AzUIf#ygShA>h>%Om6SlbVOiAV#4-ZT8 zYxp#`7H8>e-op+?vmTLwnzlBW9UHb5UoWpgx!*xO&$b1(OtxL@OA;DvfOyMJL zHY|uT<~Mc^l<)x=%xkECc=_oxGUL_xG>F3H*k$`ZPSE!AJu!zS+DFQ?dUsau`uEQt z$!#21D$T!LVw!yHgn;tx9-rPOocjp}3%~5+RU)@JS!1_rU8_yzs_D%wU&byP_oHYs zwBai3o0#`6>jntS!p>`)!P?EFKc;V1Dda!_@T-wKO$B6;tmP)p30+)-D?DjU;`%D% zt4%oPLf`q_-9uYe;CNTEgT%eE^|)JGa45nHQG%GY+&0dly<(xHIv;}vPb%v^#D{*x&6`l9xfHlZIS z?XH91yE3Z;$k>zb5>=*>jZE&hE$Iy!B;BHdSxeC5AT0cV$9xPx^j2+&z%$f;_%)>7;v}kI*_o#eDDte6Q?~*~Y@ZW=z zFFh;od0l2^R#wHqg}p^S$B@S{9m+85#n(sx3A@>ro`(s5_im0YA}U+e^Be#$%m3RW zK<;8#5Uq>fa0pMyW{1^gofg>FuGFVUBJgnh$E)9Fva8?Z$d3ww+b#(@62+1GOOJsR%vy z(?qAnD92Zde&!$q()<5yWzCwciQVnk!IT=rwx0X19ch*db2x>44BU?qd9bfb1aNawzwZXCRlEr5d&eu_DU+`^-WR!0K8aGm;d2y-%NsEAe{_Q z<$bpJnt6+oB-;+x#Cc+3PeG97^VtoXJ1ke)DO}Y!1{tui9)7nxNATk#YzkLkcazV} z<8(Va&aOYGcvC*>Pw?!^pVm4L;4&T{P(f9%duz zxn_T40|;v3*efnpXH`K*7%(!wVjna8*cs*(a4xXYxGVoPDnm-;Xv$L_A4m5Pz9_A8 zc4>sa?8k;p-F8t1PUEK^7odwnpAhLJo5*G4p|_62$A(72BakC{Phr)ZK-WS$UzCDo zEy&K_a3G_4iIv-VK&ErWXpeLvrA8?w6A55jH5%}&*hxEfysBzE#Dlf|{sKnKo=vu+oPD;I%lkVMw2Byy)Wj_}{LZA$YFm@2)tSfi)_ z&OOij5VPXep!hk5dDz5)%UMJ4bYZP7w>jZ!A7=5!OUEW!n_ZMKD&-7npzRC+SeW<2 z$+$$M3I_hKIDI`m_;S_0Oa{?D_FbzN{0O9O-}-IzUUmVgN1xj1?l@0+Yj@PBcA>o` z8DT_!3=M$QyB^H}mTK|jcRY94OOwU#i47%=9OSnzkT=+)-mj7!g+owNQ#2UEDhq&)2z ziC4U^)vO``E}z(l!Myqn z3uESw_N;lxv9TjpS0jUigA1M3#evkLl17%0KbBNDEC|Vt%0KemG0)l3)U+{HR^c{s zDsnEq*n`}4qsf2uXuhXu*zh0eH^SrRTbgvzqG~WvRl~)7G%8ZHf-ppIjm}j+{RSrbb*L$#yjr(Z z4&~+L%a37lHU?a}TM{a=Dz>D_Ck_Z+`=?*Q8k&M#*cbhwGoRh@aaiH&>&hCCifsKA z*Q#IOpI;DLaQvmvbvKVKWpW6xo&@Ec`{SB$^enf<;$*E>`zMKddU}2)jzGsnZWy)p zy1PjIA{aB^!`r$Qy=OIyUSEuEn`o#SUAj@(zp~QK&FrI?;+T;;!-{;Bs8gv-G&AdV zZtXhg)7=}9n@YW)wsxVVv7)DdAndyZs3MeOZ6&HMSt18dg?B76>cnHirOBVPuP#o@ z-#Zk?;cuhlo@?~tGG&vk5ETNM?VGzj?d04Q-ooDjMwqII@{W?x6hOA%)M3n zYs(ia+#1P&g|^W z(m7`j4-0KP=4KX~DV#3fqw4aztDSq$Q*FzdG=eC5y{d{(-j$Y8FX$Nj({*$6SDn7- ze5y}wa4azKxKV5ih2u-3!kN5-DWUYS z5(ia&o0(yfc5;)%hiL658u!bP(V?Mc->0^bG>4jtOWezgsFoG*`sJ76*=Qm+(xwV! z`7GX%f_JH`i^GG11Z!>mRs1zoeA7iomsj5`+7j2MWsQ;qoRGESg}J$Q;b0FHm8s4@O1R{*+>M;PoD})L z);i^wWJ@$uQ-|+))y;5Mx8B~~+_at09j5yK6a^4aDg^C^z1^+s{5P_KIQ&>c1tR)m ztii*XoZB3yR=D=m?Sq-w4uYAJZ`OwZ$wblmnI!FrAK~sV=+vfXd~#~mqod0cuMS2) z`_~XKniw_j`zyQ1-;ej!BD2#J)+Sp5&f5rR2|DiK^0FDk6}~DO{_%_Vp8J#oc&=e3 zHUFZXJDWm_`NsC*DWFcSK;md{e^C6&^fNhNfcz`5L-O~GGN7z Jl~RU*{|l^nIQ0Mk literal 0 HcmV?d00001 diff --git a/rocksdb/rocksdb-overview.md b/rocksdb/rocksdb-overview.md new file mode 100644 index 0000000000000..0c39c83cc47b8 --- /dev/null +++ b/rocksdb/rocksdb-overview.md @@ -0,0 +1,58 @@ +--- +title: RocksDB Overview +category: reference +--- + +# RocksDB Overview + +RocksDB is an LSM-based storage engine, providing key-value store and read-write functions. It is maintained by Facebook and is based on LevelDB. Key-value pairs put by the user are firstly inserted into Write Ahead Log (WAL) and then written to the SkipList in memory (a data structure called MemTable). LSM-tree engines convert the random modification (insertion) to sequential writes to the WAL file, so they provide better write throughput than B-tree engines. + +Once the data in memory reaches a certain size, RocksDB flushes the content into an SST file. SST files are organized in multiple levels (the default is up to 6 levels). When the total size of a level reaches the threshold, RocksDB chooses part of the SST files and merges them into the next level. Each subsequent level is 10 larger than the previous one, so 90% of the data is stored in the last layer. + +RocksDB allows users to create multiple Column Families. Column Families have their own SkipList and SST files, and they share the same WAL file. In this way, different Column Family can have different settings according to the application characteristics. It does not increase the number of writes to WAL at the same time. + +## TiKV Architecture + +The architecture of TiKV is illustrated as follows: + +![TiKV RocksDB](/media/tikv-rocksdb.png) + +As the storage engine of TiKV, RocksDB is used to store Raft logs and user data. All data in a TiKV node shares two RocksDB instances. One is for Raft log (often called raftdb), and the other is for data (often called kvdb). There are four Column Families (CF) in kvdb: raft, lock, default, and write: + +* raft CF: Store metadata of each Region. It occupies only a very small amount of space, and users do not need to care. +* lock CF: Store the pessimistic lock of pessimistic transactions and the Prewrite lock for distributed transactions. After the transaction is committed, the corresponding data in lock CF is deleted quickly. Therefore, the size of data in lock CF is usually very small (less than 1GB). If the data in lock CF increases a lot, it means that there are a large number of transactions waiting to be committed, and the system meets a bug or failure. +* write CF: Store the user's real written data and MVCC metadata (the start timestamp and commit timestamp of the transaction to which the data belongs). When the user writes a row of data, it is stored in the write CF if the data length is less than 255 bytes. Otherwise, it is stored in the default CF. In TiDB, the secondary index only occupies the space of write CF, since the value stored in the non-unique index is empty and the value stored in the unique index is the primary key index. +* default CF: Store data longer than 255 bytes. + +## RocksDB Memory Usage + +To improve the reading performance and reduce the reading operations to the disk, RocksDB divides the files stored on the disk into blocks based on a certain size (the default is 64KB). When reading a block, it first checks if the data already exists in BlockCache in memory. If true, it can be read directly from memory without accessing the disk. + +BlockCache discards the least recently used data according to the LRU algorithm. By default, TiKV devotes 45% of the system memory to BlockCache. Users can also modify the `storage.block-cache.capacit` configuration to an appropriate value by themselves. However, it is not recommended to exceed 60% of the total system memory. + +The data written to RocksDB will be written to MemTable firstly. When the size of a MemTable exceeds 128MB, it will switch to a new MemTable. There are 2 RocksDB instances in TiKV, a total of 4 ColumnFamily. The size limit of a single MemTable for each ColumnFamily is 128MB. A maximum of 5 MemTables can exist at the same time, otherwise, the foreground writes will be blocked. The memory occupied by this part is at most 2.5GB (4 x 5 x 128MB). It is not recommended to change since it costs less memory. + +**##** RocksDB Space Usaga + +* Multi-version: RocksDB is a key-value storage engine with LSM-tree structure, and the data in MemTable will be flushed to L0 first. There may be overlap between the ranges of SSTs at the L0 (because the file is arranged in the order of which they are generated). As a result, there may be multiple versions of the same key in L0. When a file is merged from L0 to L1, it will be cut into multiple files according to a certain size (the default is 8MB). The key range of each file on the same level does not overlap with each other, so there is only one version for each key on L1 and subsequent levels. +* Space amplification: The total size of files on each level is x (the default is 10) times that of the previous level, so 90% of the data is stored in the last level. It also means that the space amplification of RocksDB does not exceed 1.11 (L0 has fewer data and can be ignored). +* Space amplification of TiKV: TiKV has it's own MVCC strategy. When a user writes a key, the real data written to RocksDB is key + commit_ts, that is to say, the update and deletion also write a new key to RocksDB. TiKV deletes the old version of the data (through the Delete interface of RocksDB) at intervals, so it can be considered that the actual space of the data stored by the user on TiKV is enlarged to 1.11 plus the data written in the last 10 minutes (assuming that TiKV cleans up the old data promptly). See [《TiDB in Action》](https://github.com/pingcap-incubator/tidb-in-action/blob/master/session4/chapter7/compact.md#tikv-%E7%9A%84%E7%A9%BA%E9%97%B4%E6%94%BE%E5%A4%A7) for details. + +## RocksDB Background Threads and Compaction + +In RocksDB, operations such as converting the MemTable into SST files and merging SST files at various levels are performed in the background thread pool. The default size of the background thread pool is 8. When the number of CPUs in the machine is less than or equal to 8, the default size of the background thread pool is the number of CPUs minus one. Generally speaking, users do not need to change this configuration. If the user deploys multiple TiKV instances on a machine, or the machine has a relatively high read load and a low write load, you can adjust the `rocksdb/max-background-jobs` to 3 or 4 as appropriate. + +## WriteStall + +The L0 of RocksDB is different from other levels. The SSTs of L0 are arranged in the order of generation. The key ranges between the SSTs can overlap. Therefore, each SST in L0 must be queried in turn when performing a query. In order not to affect query performance, WriteStall will be triggered to block writing when there are too many files in L0. + +When encountering a sudden sharp increase in write delay, you can first check the WriteStall Reason metric on the Grafana RocksDB KV panel. If it is a WriteStall caused by too many L0 files, you can adjust the following configurations to 64, see [《TiDB in Action》](https://github.com/pingcap-incubator/tidb-in-action/blob/master/session4/chapter8/threadpool-optimize.md#5-rocksdb) for details. + +``` +rocksdb.defaultcf.level0-slowdown-writes-trigger +rocksdb.writecf.level0-slowdown-writes-trigger +rocksdb.lockcf.level0-slowdown-writes-trigger +rocksdb.defaultcf.level0-stop-writes-trigger +rocksdb.writecf.level0-stop-writes-trigger +rocksdb.lockcf.level0-stop-writes-trigger +``` \ No newline at end of file From 981cadf9af70cd2331f509d18d0bad1eb574bdac Mon Sep 17 00:00:00 2001 From: cosven Date: Tue, 30 Jun 2020 16:54:56 +0800 Subject: [PATCH 2/8] fix typo --- rocksdb/rocksdb-overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rocksdb/rocksdb-overview.md b/rocksdb/rocksdb-overview.md index 0c39c83cc47b8..200206c2737e4 100644 --- a/rocksdb/rocksdb-overview.md +++ b/rocksdb/rocksdb-overview.md @@ -32,7 +32,7 @@ BlockCache discards the least recently used data according to the LRU algorithm. The data written to RocksDB will be written to MemTable firstly. When the size of a MemTable exceeds 128MB, it will switch to a new MemTable. There are 2 RocksDB instances in TiKV, a total of 4 ColumnFamily. The size limit of a single MemTable for each ColumnFamily is 128MB. A maximum of 5 MemTables can exist at the same time, otherwise, the foreground writes will be blocked. The memory occupied by this part is at most 2.5GB (4 x 5 x 128MB). It is not recommended to change since it costs less memory. -**##** RocksDB Space Usaga +## RocksDB Space Usage * Multi-version: RocksDB is a key-value storage engine with LSM-tree structure, and the data in MemTable will be flushed to L0 first. There may be overlap between the ranges of SSTs at the L0 (because the file is arranged in the order of which they are generated). As a result, there may be multiple versions of the same key in L0. When a file is merged from L0 to L1, it will be cut into multiple files according to a certain size (the default is 8MB). The key range of each file on the same level does not overlap with each other, so there is only one version for each key on L1 and subsequent levels. * Space amplification: The total size of files on each level is x (the default is 10) times that of the previous level, so 90% of the data is stored in the last level. It also means that the space amplification of RocksDB does not exceed 1.11 (L0 has fewer data and can be ignored). @@ -55,4 +55,4 @@ rocksdb.lockcf.level0-slowdown-writes-trigger rocksdb.defaultcf.level0-stop-writes-trigger rocksdb.writecf.level0-stop-writes-trigger rocksdb.lockcf.level0-stop-writes-trigger -``` \ No newline at end of file +``` From 4ab955a47bb761b4b7bcbc85858f25a5dd3c7f9f Mon Sep 17 00:00:00 2001 From: cosven Date: Tue, 30 Jun 2020 17:11:00 +0800 Subject: [PATCH 3/8] remove redundant whitespace --- rocksdb/rocksdb-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rocksdb/rocksdb-overview.md b/rocksdb/rocksdb-overview.md index 200206c2737e4..04adbb4f00743 100644 --- a/rocksdb/rocksdb-overview.md +++ b/rocksdb/rocksdb-overview.md @@ -13,7 +13,7 @@ RocksDB allows users to create multiple Column Families. Column Families have th ## TiKV Architecture -The architecture of TiKV is illustrated as follows: +The architecture of TiKV is illustrated as follows: ![TiKV RocksDB](/media/tikv-rocksdb.png) From 73070346bc4be6a07868f966aeed60ef3ba49465 Mon Sep 17 00:00:00 2001 From: cosven Date: Wed, 8 Jul 2020 22:06:05 +0800 Subject: [PATCH 4/8] Apply suggestions from code review Co-authored-by: Ran --- rocksdb/rocksdb-overview.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/rocksdb/rocksdb-overview.md b/rocksdb/rocksdb-overview.md index 04adbb4f00743..9c7c5301f78c0 100644 --- a/rocksdb/rocksdb-overview.md +++ b/rocksdb/rocksdb-overview.md @@ -1,42 +1,43 @@ --- title: RocksDB Overview +summary: Learn the basic concepts and working principles of RocksDB. category: reference --- # RocksDB Overview -RocksDB is an LSM-based storage engine, providing key-value store and read-write functions. It is maintained by Facebook and is based on LevelDB. Key-value pairs put by the user are firstly inserted into Write Ahead Log (WAL) and then written to the SkipList in memory (a data structure called MemTable). LSM-tree engines convert the random modification (insertion) to sequential writes to the WAL file, so they provide better write throughput than B-tree engines. +[RocksDB](https://github.com/facebook/ rocksdb) is an LSM-tree storage engine that provides key-value store and read-write functions. It is developed by Facebook and based on LevelDB. Key-value pairs written by the user are firstly inserted into Write Ahead Log (WAL) and then written to the SkipList in memory (a data structure called MemTable). LSM-tree engines convert the random modification (insertion) to sequential writes to the WAL file, so they provide better write throughput than B-tree engines. -Once the data in memory reaches a certain size, RocksDB flushes the content into an SST file. SST files are organized in multiple levels (the default is up to 6 levels). When the total size of a level reaches the threshold, RocksDB chooses part of the SST files and merges them into the next level. Each subsequent level is 10 larger than the previous one, so 90% of the data is stored in the last layer. +Once the data in memory reaches a certain size, RocksDB flushes the content into an SST file in the disk. SST files are organized in multiple levels (the default is up to 6 levels). When the total size of a level reaches the threshold, RocksDB chooses part of the SST files and merges them into the next level. Each subsequent level is 10 larger than the previous one, so 90% of the data is stored in the last layer. -RocksDB allows users to create multiple Column Families. Column Families have their own SkipList and SST files, and they share the same WAL file. In this way, different Column Family can have different settings according to the application characteristics. It does not increase the number of writes to WAL at the same time. +RocksDB allows users to create multiple Column Families. Column Families have their own SkipList and SST files, and they share the same WAL file. In this way, different Column Families can have different settings according to the application characteristics. It does not increase the number of writes to WAL at the same time. -## TiKV Architecture +## TiKV architecture The architecture of TiKV is illustrated as follows: ![TiKV RocksDB](/media/tikv-rocksdb.png) -As the storage engine of TiKV, RocksDB is used to store Raft logs and user data. All data in a TiKV node shares two RocksDB instances. One is for Raft log (often called raftdb), and the other is for data (often called kvdb). There are four Column Families (CF) in kvdb: raft, lock, default, and write: +As the storage engine of TiKV, RocksDB is used to store Raft logs and user data. All data in a TiKV node shares two RocksDB instances. One is for Raft log (often called raftdb), and the other is for user data and MVCC metadata (often called kvdb). There are four Column Families (CF) in kvdb: raft, lock, default, and write: * raft CF: Store metadata of each Region. It occupies only a very small amount of space, and users do not need to care. -* lock CF: Store the pessimistic lock of pessimistic transactions and the Prewrite lock for distributed transactions. After the transaction is committed, the corresponding data in lock CF is deleted quickly. Therefore, the size of data in lock CF is usually very small (less than 1GB). If the data in lock CF increases a lot, it means that there are a large number of transactions waiting to be committed, and the system meets a bug or failure. +* lock CF: Store the pessimistic lock of pessimistic transactions and the Prewrite lock for distributed transactions. After the transaction is committed, the corresponding data in lock CF is deleted quickly. Therefore, the size of data in lock CF is usually very small (less than 1GB). If the data in lock CF increases a lot, it means that a large number of transactions are waiting to be committed, and that the system meets a bug or failure. * write CF: Store the user's real written data and MVCC metadata (the start timestamp and commit timestamp of the transaction to which the data belongs). When the user writes a row of data, it is stored in the write CF if the data length is less than 255 bytes. Otherwise, it is stored in the default CF. In TiDB, the secondary index only occupies the space of write CF, since the value stored in the non-unique index is empty and the value stored in the unique index is the primary key index. * default CF: Store data longer than 255 bytes. ## RocksDB Memory Usage -To improve the reading performance and reduce the reading operations to the disk, RocksDB divides the files stored on the disk into blocks based on a certain size (the default is 64KB). When reading a block, it first checks if the data already exists in BlockCache in memory. If true, it can be read directly from memory without accessing the disk. +To improve the reading performance and reduce the reading operations to the disk, RocksDB divides the files stored on the disk into blocks based on a certain size (the default is 64 KB). When reading a block, it first checks if the data already exists in BlockCache in memory. If true, it can read the data directly from memory without accessing the disk. BlockCache discards the least recently used data according to the LRU algorithm. By default, TiKV devotes 45% of the system memory to BlockCache. Users can also modify the `storage.block-cache.capacit` configuration to an appropriate value by themselves. However, it is not recommended to exceed 60% of the total system memory. -The data written to RocksDB will be written to MemTable firstly. When the size of a MemTable exceeds 128MB, it will switch to a new MemTable. There are 2 RocksDB instances in TiKV, a total of 4 ColumnFamily. The size limit of a single MemTable for each ColumnFamily is 128MB. A maximum of 5 MemTables can exist at the same time, otherwise, the foreground writes will be blocked. The memory occupied by this part is at most 2.5GB (4 x 5 x 128MB). It is not recommended to change since it costs less memory. +The data written to RocksDB is written to MemTable firstly. When the size of a MemTable exceeds 128 MB, it switches to a new MemTable. There are 2 RocksDB instances in TiKV, a total of 4 ColumnFamily. The size limit of a single MemTable for each ColumnFamily is 128 MB. A maximum of 5 MemTables can exist at the same time; otherwise, the foreground writes will be blocked. The memory occupied by this part is at most 2.5 GB (4 x 5 x 128 MB). It is not recommended to change this limit since it costs less memory. ## RocksDB Space Usage -* Multi-version: RocksDB is a key-value storage engine with LSM-tree structure, and the data in MemTable will be flushed to L0 first. There may be overlap between the ranges of SSTs at the L0 (because the file is arranged in the order of which they are generated). As a result, there may be multiple versions of the same key in L0. When a file is merged from L0 to L1, it will be cut into multiple files according to a certain size (the default is 8MB). The key range of each file on the same level does not overlap with each other, so there is only one version for each key on L1 and subsequent levels. +* Multi-version: As RocksDB is a key-value storage engine with LSM-tree structure, the data in MemTable is flushed to L0 first. Because the file is arranged in the order of which they are generated, there might be overlap between the ranges of SSTs at the L0. As a result, the same key might have multiple versions in L0. When a file is merged from L0 to L1, it is cut into multiple files according to a certain size (the default is 8 MB). The key range of each file on the same level does not overlap with each other, so there is only one version for each key on L1 and subsequent levels. * Space amplification: The total size of files on each level is x (the default is 10) times that of the previous level, so 90% of the data is stored in the last level. It also means that the space amplification of RocksDB does not exceed 1.11 (L0 has fewer data and can be ignored). -* Space amplification of TiKV: TiKV has it's own MVCC strategy. When a user writes a key, the real data written to RocksDB is key + commit_ts, that is to say, the update and deletion also write a new key to RocksDB. TiKV deletes the old version of the data (through the Delete interface of RocksDB) at intervals, so it can be considered that the actual space of the data stored by the user on TiKV is enlarged to 1.11 plus the data written in the last 10 minutes (assuming that TiKV cleans up the old data promptly). See [《TiDB in Action》](https://github.com/pingcap-incubator/tidb-in-action/blob/master/session4/chapter7/compact.md#tikv-%E7%9A%84%E7%A9%BA%E9%97%B4%E6%94%BE%E5%A4%A7) for details. +* Space amplification of TiKV: TiKV has it's own MVCC strategy. When a user writes a key, the real data written to RocksDB is key + commit_ts, that is to say, the update and deletion also write a new key to RocksDB. TiKV deletes the old version of the data (through the Delete interface of RocksDB) at intervals, so it can be considered that the actual space of the data stored by the user on TiKV is enlarged to 1.11 plus the data written in the last 10 minutes (assuming that TiKV cleans up the old data promptly). ## RocksDB Background Threads and Compaction @@ -44,9 +45,9 @@ In RocksDB, operations such as converting the MemTable into SST files and mergin ## WriteStall -The L0 of RocksDB is different from other levels. The SSTs of L0 are arranged in the order of generation. The key ranges between the SSTs can overlap. Therefore, each SST in L0 must be queried in turn when performing a query. In order not to affect query performance, WriteStall will be triggered to block writing when there are too many files in L0. +The L0 of RocksDB is different from other levels. The SSTs of L0 are arranged in the order of generation. The key ranges between the SSTs can overlap. Therefore, each SST in L0 must be queried in turn when a query is performed. In order not to affect query performance, WriteStall is triggered to block writing when there are too many files in L0. -When encountering a sudden sharp increase in write delay, you can first check the WriteStall Reason metric on the Grafana RocksDB KV panel. If it is a WriteStall caused by too many L0 files, you can adjust the following configurations to 64, see [《TiDB in Action》](https://github.com/pingcap-incubator/tidb-in-action/blob/master/session4/chapter8/threadpool-optimize.md#5-rocksdb) for details. +When encountering a sudden sharp increase in write delay, you can first check the **WriteStall Reason** metric on the Grafana RocksDB KV panel. If it is a WriteStall caused by too many L0 files, you can adjust the following configurations to 64. ``` rocksdb.defaultcf.level0-slowdown-writes-trigger From 9f71479c2b8c85294c10108a483fd1babf96c73c Mon Sep 17 00:00:00 2001 From: cosven Date: Wed, 8 Jul 2020 22:08:08 +0800 Subject: [PATCH 5/8] address comments --- TOC.md | 4 ++-- rocksdb/rocksdb-overview.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/TOC.md b/TOC.md index 411b4522f4c38..8d68286dfb490 100644 --- a/TOC.md +++ b/TOC.md @@ -404,11 +404,11 @@ + [MySQL System Variables](/system-variables.md) + [TiDB Specific System Variables](/tidb-specific-system-variables.md) + Storage Engines + + TiKV + + [RocksDB Overview](/rocksdb/rocksdb-overview.md) + TiFlash + [Overview](/tiflash/tiflash-overview.md) + [Use TiFlash](/tiflash/use-tiflash.md) - + TiKV - + [RocksDB Overview](/rocksdb/rocksdb-overview.md) + TiUP + [Overview](/tiup/tiup-overview.md) + [Manage TiUP Components](/tiup/manage-tiup-component.md) diff --git a/rocksdb/rocksdb-overview.md b/rocksdb/rocksdb-overview.md index 9c7c5301f78c0..779e0ba981312 100644 --- a/rocksdb/rocksdb-overview.md +++ b/rocksdb/rocksdb-overview.md @@ -6,7 +6,7 @@ category: reference # RocksDB Overview -[RocksDB](https://github.com/facebook/ rocksdb) is an LSM-tree storage engine that provides key-value store and read-write functions. It is developed by Facebook and based on LevelDB. Key-value pairs written by the user are firstly inserted into Write Ahead Log (WAL) and then written to the SkipList in memory (a data structure called MemTable). LSM-tree engines convert the random modification (insertion) to sequential writes to the WAL file, so they provide better write throughput than B-tree engines. +[RocksDB](https://github.com/facebook/rocksdb) is an LSM-tree storage engine that provides key-value store and read-write functions. It is developed by Facebook and based on LevelDB. Key-value pairs written by the user are firstly inserted into Write Ahead Log (WAL) and then written to the SkipList in memory (a data structure called MemTable). LSM-tree engines convert the random modification (insertion) to sequential writes to the WAL file, so they provide better write throughput than B-tree engines. Once the data in memory reaches a certain size, RocksDB flushes the content into an SST file in the disk. SST files are organized in multiple levels (the default is up to 6 levels). When the total size of a level reaches the threshold, RocksDB chooses part of the SST files and merges them into the next level. Each subsequent level is 10 larger than the previous one, so 90% of the data is stored in the last layer. From d95287e5e67da38e605c860baa2d6bc56172d5f8 Mon Sep 17 00:00:00 2001 From: cosven Date: Thu, 9 Jul 2020 20:59:00 +0800 Subject: [PATCH 6/8] Apply suggestions from code review Co-authored-by: Ran --- rocksdb/rocksdb-overview.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/rocksdb/rocksdb-overview.md b/rocksdb/rocksdb-overview.md index 779e0ba981312..b62e63ea2e50a 100644 --- a/rocksdb/rocksdb-overview.md +++ b/rocksdb/rocksdb-overview.md @@ -8,7 +8,7 @@ category: reference [RocksDB](https://github.com/facebook/rocksdb) is an LSM-tree storage engine that provides key-value store and read-write functions. It is developed by Facebook and based on LevelDB. Key-value pairs written by the user are firstly inserted into Write Ahead Log (WAL) and then written to the SkipList in memory (a data structure called MemTable). LSM-tree engines convert the random modification (insertion) to sequential writes to the WAL file, so they provide better write throughput than B-tree engines. -Once the data in memory reaches a certain size, RocksDB flushes the content into an SST file in the disk. SST files are organized in multiple levels (the default is up to 6 levels). When the total size of a level reaches the threshold, RocksDB chooses part of the SST files and merges them into the next level. Each subsequent level is 10 larger than the previous one, so 90% of the data is stored in the last layer. +Once the data in memory reaches a certain size, RocksDB flushes the content into a Sorted String Table (SST) file in the disk. SST files are organized in multiple levels (the default is up to 6 levels). When the total size of a level reaches the threshold, RocksDB chooses part of the SST files and merges them into the next level. Each subsequent level is 10 times larger than the previous one, so 90% of the data is stored in the last layer. RocksDB allows users to create multiple Column Families. Column Families have their own SkipList and SST files, and they share the same WAL file. In this way, different Column Families can have different settings according to the application characteristics. It does not increase the number of writes to WAL at the same time. @@ -21,27 +21,29 @@ The architecture of TiKV is illustrated as follows: As the storage engine of TiKV, RocksDB is used to store Raft logs and user data. All data in a TiKV node shares two RocksDB instances. One is for Raft log (often called raftdb), and the other is for user data and MVCC metadata (often called kvdb). There are four Column Families (CF) in kvdb: raft, lock, default, and write: * raft CF: Store metadata of each Region. It occupies only a very small amount of space, and users do not need to care. -* lock CF: Store the pessimistic lock of pessimistic transactions and the Prewrite lock for distributed transactions. After the transaction is committed, the corresponding data in lock CF is deleted quickly. Therefore, the size of data in lock CF is usually very small (less than 1GB). If the data in lock CF increases a lot, it means that a large number of transactions are waiting to be committed, and that the system meets a bug or failure. +* lock CF: Store the pessimistic lock of pessimistic transactions and the Prewrite lock for distributed transactions. After the transaction is committed, the corresponding data in lock CF is deleted quickly. Therefore, the size of data in lock CF is usually very small (less than 1 GB). If the data in lock CF increases a lot, it means that a large number of transactions are waiting to be committed, and that the system meets a bug or failure. * write CF: Store the user's real written data and MVCC metadata (the start timestamp and commit timestamp of the transaction to which the data belongs). When the user writes a row of data, it is stored in the write CF if the data length is less than 255 bytes. Otherwise, it is stored in the default CF. In TiDB, the secondary index only occupies the space of write CF, since the value stored in the non-unique index is empty and the value stored in the unique index is the primary key index. * default CF: Store data longer than 255 bytes. -## RocksDB Memory Usage +## RocksDB memory usage To improve the reading performance and reduce the reading operations to the disk, RocksDB divides the files stored on the disk into blocks based on a certain size (the default is 64 KB). When reading a block, it first checks if the data already exists in BlockCache in memory. If true, it can read the data directly from memory without accessing the disk. -BlockCache discards the least recently used data according to the LRU algorithm. By default, TiKV devotes 45% of the system memory to BlockCache. Users can also modify the `storage.block-cache.capacit` configuration to an appropriate value by themselves. However, it is not recommended to exceed 60% of the total system memory. +BlockCache discards the least recently used data according to the LRU algorithm. By default, TiKV devotes 45% of the system memory to BlockCache. Users can also modify the `storage.block-cache.capacity` configuration to an appropriate value by themselves. However, it is not recommended to exceed 60% of the total system memory. The data written to RocksDB is written to MemTable firstly. When the size of a MemTable exceeds 128 MB, it switches to a new MemTable. There are 2 RocksDB instances in TiKV, a total of 4 ColumnFamily. The size limit of a single MemTable for each ColumnFamily is 128 MB. A maximum of 5 MemTables can exist at the same time; otherwise, the foreground writes will be blocked. The memory occupied by this part is at most 2.5 GB (4 x 5 x 128 MB). It is not recommended to change this limit since it costs less memory. -## RocksDB Space Usage +## RocksDB space usage -* Multi-version: As RocksDB is a key-value storage engine with LSM-tree structure, the data in MemTable is flushed to L0 first. Because the file is arranged in the order of which they are generated, there might be overlap between the ranges of SSTs at the L0. As a result, the same key might have multiple versions in L0. When a file is merged from L0 to L1, it is cut into multiple files according to a certain size (the default is 8 MB). The key range of each file on the same level does not overlap with each other, so there is only one version for each key on L1 and subsequent levels. +* Multi-version: As RocksDB is a key-value storage engine with LSM-tree structure, the data in MemTable is flushed to L0 first. Because the file is arranged in the order of which they are generated, there might be overlap between the ranges of SSTs at the L0. As a result, the same key might have multiple versions in L0. When a file is merged from L0 to L1, it is cut into multiple files in a certain size (the default is 8 MB). The key range of each file on the same level does not overlap with each other, so there is only one version for each key on L1 and subsequent levels. * Space amplification: The total size of files on each level is x (the default is 10) times that of the previous level, so 90% of the data is stored in the last level. It also means that the space amplification of RocksDB does not exceed 1.11 (L0 has fewer data and can be ignored). * Space amplification of TiKV: TiKV has it's own MVCC strategy. When a user writes a key, the real data written to RocksDB is key + commit_ts, that is to say, the update and deletion also write a new key to RocksDB. TiKV deletes the old version of the data (through the Delete interface of RocksDB) at intervals, so it can be considered that the actual space of the data stored by the user on TiKV is enlarged to 1.11 plus the data written in the last 10 minutes (assuming that TiKV cleans up the old data promptly). -## RocksDB Background Threads and Compaction +## RocksDB background threads and compaction -In RocksDB, operations such as converting the MemTable into SST files and merging SST files at various levels are performed in the background thread pool. The default size of the background thread pool is 8. When the number of CPUs in the machine is less than or equal to 8, the default size of the background thread pool is the number of CPUs minus one. Generally speaking, users do not need to change this configuration. If the user deploys multiple TiKV instances on a machine, or the machine has a relatively high read load and a low write load, you can adjust the `rocksdb/max-background-jobs` to 3 or 4 as appropriate. +In RocksDB, operations such as converting the MemTable into SST files and merging SST files at various levels are performed in the background thread pool. The default size of the background thread pool is 8. When the number of CPUs in the machine is less than or equal to 8, the default size of the background thread pool is the number of CPUs minus one. + +Generally speaking, users do not need to change this configuration. If the user deploys multiple TiKV instances on a machine, or the machine has a relatively high read load and a low write load, you can adjust the `rocksdb/max-background-jobs` to 3 or 4 as appropriate. ## WriteStall From 5727e86618b60e8c3cb7ec10948c8f769511846d Mon Sep 17 00:00:00 2001 From: cosven Date: Thu, 9 Jul 2020 21:30:38 +0800 Subject: [PATCH 7/8] address comments --- rocksdb/rocksdb-overview.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rocksdb/rocksdb-overview.md b/rocksdb/rocksdb-overview.md index b62e63ea2e50a..91d8c8da17d10 100644 --- a/rocksdb/rocksdb-overview.md +++ b/rocksdb/rocksdb-overview.md @@ -10,7 +10,7 @@ category: reference Once the data in memory reaches a certain size, RocksDB flushes the content into a Sorted String Table (SST) file in the disk. SST files are organized in multiple levels (the default is up to 6 levels). When the total size of a level reaches the threshold, RocksDB chooses part of the SST files and merges them into the next level. Each subsequent level is 10 times larger than the previous one, so 90% of the data is stored in the last layer. -RocksDB allows users to create multiple Column Families. Column Families have their own SkipList and SST files, and they share the same WAL file. In this way, different Column Families can have different settings according to the application characteristics. It does not increase the number of writes to WAL at the same time. +RocksDB allows users to create multiple Column Families (CFs). CFs have their own SkipList and SST files, and they share the same WAL file. In this way, different CFs can have different settings according to the application characteristics. It does not increase the number of writes to WAL at the same time. ## TiKV architecture @@ -18,7 +18,7 @@ The architecture of TiKV is illustrated as follows: ![TiKV RocksDB](/media/tikv-rocksdb.png) -As the storage engine of TiKV, RocksDB is used to store Raft logs and user data. All data in a TiKV node shares two RocksDB instances. One is for Raft log (often called raftdb), and the other is for user data and MVCC metadata (often called kvdb). There are four Column Families (CF) in kvdb: raft, lock, default, and write: +As the storage engine of TiKV, RocksDB is used to store Raft logs and user data. All data in a TiKV node shares two RocksDB instances. One is for Raft log (often called raftdb), and the other is for user data and MVCC metadata (often called kvdb). There are four CFs in kvdb: raft, lock, default, and write: * raft CF: Store metadata of each Region. It occupies only a very small amount of space, and users do not need to care. * lock CF: Store the pessimistic lock of pessimistic transactions and the Prewrite lock for distributed transactions. After the transaction is committed, the corresponding data in lock CF is deleted quickly. Therefore, the size of data in lock CF is usually very small (less than 1 GB). If the data in lock CF increases a lot, it means that a large number of transactions are waiting to be committed, and that the system meets a bug or failure. @@ -31,7 +31,7 @@ To improve the reading performance and reduce the reading operations to the disk BlockCache discards the least recently used data according to the LRU algorithm. By default, TiKV devotes 45% of the system memory to BlockCache. Users can also modify the `storage.block-cache.capacity` configuration to an appropriate value by themselves. However, it is not recommended to exceed 60% of the total system memory. -The data written to RocksDB is written to MemTable firstly. When the size of a MemTable exceeds 128 MB, it switches to a new MemTable. There are 2 RocksDB instances in TiKV, a total of 4 ColumnFamily. The size limit of a single MemTable for each ColumnFamily is 128 MB. A maximum of 5 MemTables can exist at the same time; otherwise, the foreground writes will be blocked. The memory occupied by this part is at most 2.5 GB (4 x 5 x 128 MB). It is not recommended to change this limit since it costs less memory. +The data written to RocksDB is written to MemTable firstly. When the size of a MemTable exceeds 128 MB, it switches to a new MemTable. There are 2 RocksDB instances in TiKV, a total of 4 CFs. The size limit of a single MemTable for each CF is 128 MB. A maximum of 5 MemTables can exist at the same time; otherwise, the foreground writes is blocked. The memory occupied by this part is at most 2.5 GB (4 x 5 x 128 MB). It is not recommended to change this limit since it costs less memory. ## RocksDB space usage From e6dcd456793791a140eaecf55b264667a733b54f Mon Sep 17 00:00:00 2001 From: Ran Date: Tue, 14 Jul 2020 13:11:58 +0800 Subject: [PATCH 8/8] move the path of rocksdb-overview --- {rocksdb => storage-engine}/rocksdb-overview.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {rocksdb => storage-engine}/rocksdb-overview.md (100%) diff --git a/rocksdb/rocksdb-overview.md b/storage-engine/rocksdb-overview.md similarity index 100% rename from rocksdb/rocksdb-overview.md rename to storage-engine/rocksdb-overview.md