From abca46b2c5324a9b1415c2e9bba585f885143007 Mon Sep 17 00:00:00 2001 From: jiacai2050 Date: Tue, 27 Aug 2024 15:22:47 +0800 Subject: [PATCH 1/5] feat: add metric engine rfc --- docs/rfcs/20240827-metric-engine-cn.md | 231 +++++++++++++++++++++++++ docs/rfcs/resources/root-table.png | Bin 0 -> 39642 bytes 2 files changed, 231 insertions(+) create mode 100644 docs/rfcs/20240827-metric-engine-cn.md create mode 100644 docs/rfcs/resources/root-table.png diff --git a/docs/rfcs/20240827-metric-engine-cn.md b/docs/rfcs/20240827-metric-engine-cn.md new file mode 100644 index 0000000000..3a3691425d --- /dev/null +++ b/docs/rfcs/20240827-metric-engine-cn.md @@ -0,0 +1,231 @@ +- Feature Name: New engine for storing massive metrics data +- Tracking Issue: NA + +> Note: This document is only available in Chinese and can be translated into the desired language using services such as Google Translate. + +# Summary + +新引擎的目标： +- 管理大量 metrics 数据，单个指标下，时间线量级到亿级别 +- 高效的数据检索（Scan + Filter） + +# Motivation + +目前的 Analytic Engine 对于管理大量 metrics 数据有如下问题： +1. 以表为资源管理单元，不适合管理 10w+ 的表 +2. 基于 Arrow + Datafusion 的查询框架，丢失时间线信息，导致单个查询会占用较多资源 + +# Details + +## 表管理 + +主要有两个目标： +1. 复用现有的 meta 基于 Shard、表的调度能力 +2. 与 AnalyticMetric 平级，可以尽可能复用 Frontend 层（处理 GRPC/HTTP ）的逻辑（比如：转发）。 + +### 方案 + +![](resources/root-table.png) + +说明： +- Region 对应现在的 Shard 逻辑 +- meta 内只有一张表，即 root，它是一个 super 表，采用 range 分区 +- Range 分区管理的详见后面的扩容方案小节 +- 分区的计算方式是：`hash(metric + sorted_tags)` ，之后根据 ID 在 Range 的范围确定所在 Region + +本方案缺点： +1. 没法做到指标级别 TTL + +### 扩容方案 + +假设现在有如下分区布局： + +| Row Key | Region ID | +| --- | --- | +| [A, H) | 1 | +| [H, S) | 2 | +| [S, Z) | 3 | + +如果发现 Region 1 有热点，那么可以新增 Region 4，并添加如下规则： +```sql +alter table root + split partition region_1 into [A,E), [E, H); +``` +这样 `[A,E)`还会写到 Region1 中，而 `[E, H)`则会写入新的 Region（系统会自动创建）。 +这时 `[A, H)`的分区可以删掉，写入时根据最新的路由规则进行，查询时由于无法精确到 Region，所以只能查询所有 Region。 + +#### 讨论 +如果查询时可以定位到涉及的 Region，比如 ID 采用 `hash(metric+固定TagKey)` 的生成方式，那么在分裂时可以通过记录 Region TTL 的方式来减少查询所涉及的 Region 范围。 +对于上面的分裂 case 来说，对于 `[A, E)` 写入、查询路由不变 `[E, H)`则需要根据时间戳来区分，假设 split 的分区时间为 t，表的 TTL 为 30d，那么 + +| 查询范围 | 设计 Region | 分区规则 | 规则 TTL | +| --- | --- | --- | --- | +| [0, t) | 1 | [A, H) | t+30d | +| [t, MAX) | 4 | [E, H) | MAX | + +相当于每个分区规则也有一个 TTL，默认是 MAX，在分裂时，会自动更新老分区规则的 TTL。在分裂 region1 后各个规则的 TTL 如下： + +| 规则 | TTL | CreatedAt | +| --- | --- | --- | +| [A, H) | t+30 | + | +| [A, E) | MAX | t | +| [E, H) | MAX | t | +| [H, S) | MAX | + | +| [S, Z) | MAX | + | + +写入时是需要找到对应区间内 TTL 最大的即可，查询则需要根据时间戳来与分区键来路由。 + +## 索引设计 + +索引的设计主要参考 VM，VM 中的所有索引结构采用平铺方式（即 opaque byte slices ）保存在 [MergeSet](https://pkg.go.dev/github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset) 结构中，主要有如下几种： +1. `metricName -> metricID`， metricName 包含指标名与所有 label 对 +2. `metricID -> metricName` +3. `label=value -> metricID`， label 到 metricID 的倒排，用于加速待 label 的过滤查询 + + +不同的索引结构通过固定 prefix 来区分，这样可以保证 Scan 的高效。HoraeDB 在参考这种做法的基础上，设计了如下几种索引结构： + +``` +metrics: {Date}-{MetricName}-{MetricID}-{FieldName} + +series: {Date}-{TSID}-{SeriesKey} + +tags: {Date}-{TagKey}-{TagValue} <可选> + +index: {Date}-{TagKey}-{TagValue}-{TSID} +``` + +- Date 默认为天 +- TSID 采用 Hash 方式生成 +- SeriesKey 为所有排序后的 TagKV + +采用 Table 来管理上述结构，不同字段可以直接对应 parquet 的一个列，便于进行针对性 encoding。 +如果采用类似 VM 这种 opaque byte 的方式，就失去了对不同类型针对性优化的可能性。 +与 VM 不同的是，Date 属性通过 segment duration 来表示，并不会单独记录一列，每个 SST 会记录 min/max 时间戳，便于在 compact 阶段进行数据淘汰。 + +对于上述索引结构，需要四类表，结构定义如下： + +**metrics** + +| MetricName | MetricId | FieldName | FieldId | +| --- | --- | --- | --- | +| string | uint64 | string | uint32 | + +**series** + +| MetricId | TSID | SeriesKey | + | +| --- | --- | --- | --- | +| uint64 | uint64 | bytes | + | + +**tags(可选)** +这个表是为了加速 LabelValues 的查询，如果没有的话，也可以通过 index 表查出来，具体来说分两步： + +1. 先用 filter 的 label 过滤出 TSID +2. 用筛选出的 TSID 在 series 表中查找对应 SeriesKey，并从其中提取出对应 label 的 value + +VM 就是采用上述做法。 + +| MetricID | TagKey | TagValue | +| --- | --- | --- | +| uint64 | bytes | bytes | + +**index** + +| MetricID | TagKey | TagKey | TSID | +| --- | --- | --- | --- | +| uint64 | bytes | bytes | uint64 | + +每个表采用 4.0 中 segment_duration 的方式来管理，便于基于时间戳在 compact 阶段进行淘汰。 + +### 示例 + +``` +1 http_requests{url="/api/put", code="200", job="proxy"} 100 2024-08-15 +2 http_requests{url="/api/query", code="200", job="proxy"} 10 2024-08-15 +``` +需要的表数据 +#### metrics +metadata: min_timestamp: 2024-08-15, max_timestamp: 2024-08-15, + +| MetricName | MetricId | FieldName | FieldId | +| --- | --- | --- | --- | +| http_requests | 1 | value | 1 | + +#### series +| MetricId | TSID | SeriesKey | +| --- | --- | --- | +| 1 | 1 | {code=200, job=proxy, url=/api/put} | +| 1 | 2 | {code=200, job=proxy, url=/api/query} | + +#### Tags（可选） + +| MetricId | TagKey | TagValue | +| --- | --- | --- | +| 1 | code | 200 | +| 1 | job | proxy | +| 1 | url | /api/put | +| 2 | code | 200 | +| 2 | job | proxy | +| 2 | url | /api/query | + +#### index +| MetricId | TagKey | TagKey | TSID | +| --- | --- | --- | --- | +| 1 | code | 200 | 1 | +| 1 | code | 200 | 2 | +| 1 | job | proxy | 1 | +| 1 | job | proxy | 2 | +| 1 | url | /api/put | 1 | +| 1 | url | /api/query | 2 | + +### 问题讨论 + +在 VM 中，指标名通过 `__name__` 这个特殊的 TagKey 来表示，没有单独的 metric 这个表，这样的好处是简单，但可能会导致索引膨胀问题。分析如下： + +假设有如下数据 +``` +1 http_requests_latency_bucket{le="0.1"} +2 http_requests_latency_bucket{le="0.2"} +3 http_requests_latency_bucket{le="0.3"} +4 grpc_requests_latency_bucket{le="0.1"} +``` +在 VM 中，会有如下 index 结构 + +| 序号 | index rows | +| --- | --- | +| 1 | le=0.1 => 1 | +| 2 | le=0.1 => 4 | +| 3 | le=0.2 => 2 | +| 4 | le=0.3 => 3 | +| 5 | __name__=http_requests_latency_bucket => 1 | +| 6 | __name__=http_requests_latency_bucket => 2 | +| 7 | __name__=http_requests_latency_bucket => 3 | +| 8 | __name__=grpc_requests_latency_bucket => 4 | +| 9 | http_requests_latency_bucket, le=0.1 => 1 | +| 10 | http_requests_latency_bucket, le=0.1 => 4 | +| 11 | http_requests_latency_bucket, le=0.2 => 2 | +| 12 | grpc_requests_latency_bucket, le=0.3 => 3 | + +在 HoraeDB， 只会记录 9-12 这四条数据。 +VM 这样的好处是为了可以查询没有指定 metric name 的查询，为了满足这类查询，HoraeDB 也需要新建额外的索引 TagKV => metric ，查询分两步： + +1. 根据 filter 的 tag 查出对应 metric +2. 根据 metric 再去走正常的查询流程 + +很明显这这种方式会比较低效，但这类查询比较低频，RT 高些也是可以接受的，而这种方式的优势就是记录的索引大大减少了。 + + +## 数据设计 + +与索引设计类似，采用带 segment duation 的表来管理 + +| MetricID | TSID | Timestamp | Field1 | Field2 | +| --- | --- | --- | --- | --- | +| uint64 | uint64 | int64 | double... | double... | + +底层 SST 文件会对应一个 sequence，不同文件之间 compact 时，如果遇到主键相同时，根据 seq 来去重，seq 大的值，为最新的值。 diff --git a/docs/rfcs/resources/root-table.png b/docs/rfcs/resources/root-table.png new file mode 100644 index 0000000000000000000000000000000000000000..6e4464e9dd2e36c83c0132b3a17c3a163542ff04 GIT binary patch literal 39642 zcmeEuc{tTy*FLACl;WU>N+KFKG>{B&gpg2_d1yw4WXwzxB}AdjLrA7%o=<~R!XZOu zGS5Pp`K^5p)$@Fx_j=zye}6n(R|6fNVeh@}b+3Efd*45;BEM+^`vxj1s!fUtvT9UR zSV<}>OaSd#_>P$NHb?l2+FDKiI8|!p_5t|9?24A+Rb^$W{qS#EDl9cS6%G0b{FbEN z_WR!_sP|H>A%7o3MdfKigb zT26N=%)k6WprUD7_e#}fJvA0bp!%0T4p3uuO3?nxr_m4NwBY+5J9NbE{oAFeFj&c( zYybT!1kP9ReT;af&BlMZI=UIH^Z3SpKGAs^T4$`C@xa!9eNwm^Zl;6n-yV`cP7wZpP!e6Uxp+$kv!A*VDd3@`CarD19`a2i> zH<6)OBK((0{H84b1C#Jb!ZaVIFu4vk4^0CHx&3^6aryZ=^xKbe%KLEy&doWB+K+}A zH7Us8#`6jDzeacj1XRTANBKW}`czY3e_c^kHCXq0QQeSof(r$PbN7>gY?jusSXz^_ zlg4@keW{a!IX}K8sFpLUO6)2R!ar^>bSpaN(#?g%d_P=~GX!I>R2u!_g~HXU2>c$^2%&)Y?`7G9j2}&n zG;3Pq5AY7u#ChwvE&hygdlL`Kpd4#+?}=-Co8u6^vBa%pxFt`0|Ni~Yv$N~>ld5W` zOO{Rp{M@wbaWh^x{yHvr0)LkA?}GIC&wJXJ7u$Q{f(>4e zx((^J2YBvu)w)N&gWZIFP@eND%@K>1y*xZT6JH~jYB!GNcF}9UI`5GXOuYGHUh|72 z!Dc%Ewih+L%E1F4Z5XMsxAt{p8E)4|)!R4S?pD$>(qA3xxzMprinD|9i@cwK!&$-_ zPCv=H7qzvu-^^YbML3)|;m0xZsors9K6sD6@n-TW{{}qzj!(b?#02^`+#F!_yi3R0 z^o4%QOr?D>_ndIKR?NJFr3m&cA^6>?u%l-poF^35R6paK%W8NKTc-IsAC^P>4 zHPvCsiw;u*;uzW)9U`6E!MZSx{J8j~KyA0`kV97^ z24@%jd^?@S=_BkSjfk(Y0np}2$wF^Hv6~u1x{{;Li$jJvvhn$aa?77j!Dp_omzW~S zWDmMar z2{YGNTlK&(8WF2-GYM$880Ei z62Bnrcim-37kg|$Dr}0g10&-}zhyD`T9lep!DjR$)Yyy4@V?1DM;GPbi?^Jw%o-0R zl{QxJY&@q1#6b2maW+{jvX!>&!AnMR%`+ZGrKB$*vRepcedA~Dy6UC2dxD!S3Mzu~ zw;yxNyyDZtkHuN)M_Kn(X4m%g1RL79c(UdcyX8sJ3ZVfe5VrtR$bDO>j4rsm;jD&+ zPS?6k?A`4uBrEJixZxME$j+_1qsqy%dT>8bixcZ2EVyC)r)+tVnwqTy;osb}KbM{w ztB13c%&0W_c9+*uiSj;rjGW`08^lG>>n!kDb`Yse7nDU>|8G~foN%;(sLeWOxdTI+ z%h}<4HN5H-opMLe3Z!rmtkgBX#sig@d`m$VEYo<4!z-cr>)QmZwiuM)24E=V&Kya$gx<}B(gBzBJ` ztSWhu&G*8Fd|#A7ahEWfrCM$S0w}sxQtH87 zZx-b++1vO<(q%cxaPj5kt5=WM?I%}??X<6IOGRYlWMH&C=xAC(_K()P2o4`J+~rO% z#THJ?{yZtJbE!YU)OUwC@BRDtqj^nlHM`9Jx?5~9Tw3L}WPQVqs%E;(cL$$Ruz>k+ zEz3@)`H`F}V{Ju8)Gfz5-r$RUnbD;!*TdS=3#h&QJNvu4d(!+GYZqL{!13hy^XI&} z+8P@5-A*YkU9)|W!9TjnntS@HBCneZ>v_V@Wa_wY-1|iT8;( zOtiNjweEdbINjzHES@4aX6owSzBp=cHPQWvCA>-zhL@lO#eU}52b1L6QvMiEB)Uwu z8sXi#d_=ml8VpP`S*2z_sSe*vGbjqtr;k-sr=z3e5f(m|n%B!d@GU*Ihvu26b$q{* zis+5+cfu$8WBtTDhlkpVoIH!`SwoMQ2e>WuxdkPq=AX34?OZ$Cy+>+a$hzL8`LW<* zPQUal1q{(Nr@->9W*1JSfCD977tqCmO*C3i_OQR zPlUEFoDe#2$f$CAb=WSI>gO-M#w$O5_;BOw;+WfY&#H)HrhP{<6sDpKGMuI@`VKX| zzT&-+?U1a&&Ca`|gMFnFJ>@+?x)%PyhOUjG?@Q2tBmtUdKJfAWdqDi$dAHbJbva7y zdoLyJ<;ws_%HmUL%X4Wl9*?}s^Q`-1Z*vI?3;(M3b6dCs5S4W&QBF?I&a5F-|B&;H zrPE?wl}lw?sHBVIi~Y)}FW)fZ6I3IE3dV|?4cxl6wLe<3j$Vt{lT)&oA~DC+co*a%JgW{cg z_zeB!IUHnJ__fYA3=A`Dx|n&r-C(MyGF;qq@M~Iu{g{Q}(Cck`zNzWynU$MPg$U^9 zf4sAvdH3PNUn@aY zwlo0)`FH@?64I0Q?oo;$HDF!aVsNlU(C z@n++O)Ns6gb+|^5fd2mWgCkzXRls#VAk7Up zM)N^^3+o#snVkIAOp~uX4VEbSjeS6stABB0)9UcSKuT^Qb-af>aK*G}?ZHf;oKWxZ z@oKk;$;t5G2|yhkHMKtXwFgH_mR;x79tY903w?A@0+0$J4Ix}N7^=_K5lSBbw!B3w z^^>yjKw?JS$g{&n$H#tP{--;#Sl-&K!U_~~u+VJ6aMRg-CN{j-Hksgz%a<>I++p9c zOHolVo6oygQ$vHd!N4h{IMWKUgt%awQB}l5W0h-tvFrGobuwioGo^wjOI#Oat#9`q z$$`jYso;r;i;w@%_SK!1=0DtBkBgk~K9`q|xVg=JjTCT*1St}yPhC;o`Q%>O&!-tJ za)u1skDeCFGk#}Pcg{_T=7ny@5q<=K5R6`xgCsV#z`(%QrVRq(aR8XVW=nuMu^Vt1 z&ea4z?>v&1|Ncbz8U_hY^#qko;kxI}T~XA`!Y65_W#`)_u;J$-ES$(pO@|wm6LG?< zR^+L0FThkXq*pH;fo$4Z*E@=5Bqff{X&U0^yiZ3;y69N3^~s(g@`4a92I(`cm)aS-o75bv^p&oM9@|G&T45RIIJJl$Dh=G2gzt^nm?WpVYFp zjp?Thqbh@H+pveL5u1xAbi5m3>{y}%aHcdN$M2M~av&_Ab8Svz$M)^r`v?%f>qwy9 z?VT#sja&1}-~Mbc6bu))&o<{1e$(+>@o{%;qI#tX>%x4e=gyO7&iET2m(j0EQ*RWv z>E|qRnhr8=&epda1?V54v%9fX_}Y`Kq;$NXVD2LXys#_)eQD`l)$+V0p7n$0l08XG zN&IfnDx&7=nr07lFd!nrqV?uXr)S*r=euip!t0_HF%4;kZqgUukk-4oMfag<%cAhd z2a;|`=6HnDsDI?*uu=3S){?#a{7-f+jeFW@*BJTqRJ@KH&yX!j<2o7`?Jz3jP_ zIeaB{wa0|Ken8lLeyl2Kk|C}Vc63)|cHmqaz_RRPCXQo{ zXAxb~USs2V7xGd6t+;~-hIedm?B@^2X>(K(`Vqy8U;H3ae4*hOfZ4=bI>B$It&N>| zeGwTAb30r8J3HRowwkOE@$@CfCwV+9W!%|zTD-7Ug`wfxh_P5KtxcJ~=fVHzoKV=0 zXr(i*z<|62F5LLc?I*#-{+?vjPuf-ejBS(mTB+jcqOrzZoAfqvNNBBaKD%M@_E}$k zoy-@VJQHP7OVxG;BsPwm7Ax<5t8Mt*B1P_>9vj~5nH&nVyJ_`kwcbN@i=EnjS@Qf4 zcX%{r7#ZTGNE*r786VhVMMVVY9s9T0x4YyNr7gq%`4!b`?^1at;qZF$(9~Fht{+r3oM$($N%U%oXR zCTL^lCE!sVbNEPR{}&wIQ9@L-(ZFya@}82GO%CgZRj}+h2Su*I4z?z8EVvW5Y;gY0 z9GTnkAgA3`d@3u$hQ}uA4c(r^#EjArNJ=JScEJAMXtNgAhF=JqfIKQDILk_2e((oW zJ{}BJo7QilJQ$}0W{0@l#&gftD{tf5={)YSu+S%!lT!=`*HqG=XYI4-G&f~3pOd~k z++HGOlbaWHvVS+3%aG$p0LpsH^%2dK%!o7b*k+Nt%ZAQ~O{t&QZU~k39eNY1v-rx5 zI2ToWE0bhnrgI7K6T4?tpQmsygr#@v@lZ;H(B4L>Na?5fXz>hZ2!!C;rUUNPIoA!H zN-h}qLAp~)t=8t!%{EK#8WlxPZ&wUMi zBVKbUHqRBMv6TH#3>Jj6Txw^EblFFw?*2c|BwB795 zG+USE$^&UN@X3?SiYHG7n5GqFr+*#!_UZASxa@2Vn-89C33+p&$#+D!xH=xnWg0!u zZ(&Qz?D}r^Lm0&c;Q(rETI&sL!O*^Uch@aBnrqv>b*oyw7vVh(r5@y>uiQjl84|n3 z{bti6jOl$=PEqa3Ddvty5g3?)mw z+ZKDAk9-g`j zbix;)?_RU=U~zz}dT?&2ZD=STbDb8S?YPa5`obwc>_i%!yr}Kq*`qI`SC9h`TN#G2 z>3vx#Wf*1UI}l;t$S$kk!wrt4Mf*FzpIBX&fR4-cx5*TY{RW6 zs|S_RFJHdAn%1TKdrH+XyG4AS@c%?G%gAEMCmN+Y?Q_gg0pk+YLjZ5CFP{$gJdbS%VegF!8;G8bu9}4)bMZ{Ow-STFSEtaG%!J z-dA;OTCYF3E=jZe&FytPy`k5$FM-UmiRPH=Lf1}R&MuGwCnh78^+pRPRm2lAGtaut zH@7zp-CG81qhl3#5-abgQ}lR`YPLF3J2Dj5Big^2h9CTnJ+^k^R?f7-i3j~z?0X05 zl7jUQU3hzklq%#?_vW`lfs+YliYxkLIAx~pXh*@#Yk-fPPC@Lro6m6eflWW@jqgZX zQS;IciJPyG@;Px|)%kd8(J!4)nu*&x7dUNNJ=Ux{p0=%+^lWEiV=8g2f+xZX0)q+mGt?XH9b9@yc`4 zV`I*2a}CdaMz0Im@U-74#Hiaxq>U7O$YsuE_FI|c)wJ+s=Rx&ICjG0iSL$A#tLSCF zk=kkad$s6cH18&H4X#4aG#1JD1+$S$Q+uklw}cGISl=p~?G1hHw!Cz8n_aVcj_}+P z)ZBHXAKdc3cYuI=Cq>Id%ZQ0z2Ei;kEu0mhcUbU5rTExK*5dKIq=v%Zh{v@L;4>h< zmYfIO2~V&>2M!_uM`8zSaBo2HU|zVGZT<|?d_ELyF;J#y%3Bu{2yJxcHz>@l&S&SS z#aplF-hkJ$FGKS)!VpBg$zk9tp9+X~z782gR3|0+olWiTkG@9jUMr?>s2 z(4bIy9^T?-SbSTEZC5ZtmtgI1l;_y7RtL5@O?z#V)c&9xA09#dY%}ftL#=Ufi2;!99Ft)2 zOyzgdweC7Gl-n2K#Lm%6PH6~FaW^2dFz)GHyRw$$J3>rT*PnZyArV${;7or_oKk6H zUJocUj{^dj9J>AG!_-4t&RqHS>Ov50o#>A}QnUBJgb0@xFZeCb`0-A<_pcY*_P%jW zBh7T{!p!(P2K=;_wb}Rg_bc(g85qv%0inRJek{W8b9nB0gtP=Lnx-sMqn~o~&AGbM zoA@I9_~wBX3PRnP#bI2d)+!BJ)a(Q25mUnqeZ8qo$_g*b0I`N8DiUu@bSon6G>v4A z&maJFrxuI`^hLV0Ro+;NZMar_Cte3Mll@}c-PV!wVJ<~3^L96`2KJBgqbx9zTU_tV z<;Z|P<~a2T$ya=7g-kD%aw_WZ_G$SK0<-URzjdP|cfG9`i~DHAI&J|8(IfP&h})SO zQBhHxHi>K5Y&6P10^!?s#O&4i+r^2}bbI>!0|R-SehvGB$b5h8$*CeBK6Qg6ecjA2 z;o_Pc&g0TfDP!W&49Wp><%Ua#Hcye5!%PKSQ#|}d+HT6MW#BOTY21d>zZh#f;Ntz@ zL3e`G;y!7Hubonq$>PK*LsccK9_oY^-G$ATLxS;8+@PCIFIy|%tyhahcaee{ym zuVHPvYyH1o=Pz_4tc>!4ODz5H3o4J*&9R7n#$M?(H)gmzry1J%1CoKRB)!oQ`FGYj z6FQHdj<%N=4h{}>M=lLScK>R=9^7jeXrdS%#>_$8r+xp|sdR7Mu|PY$7J*{2PEQ*T zy32WAzM|t)9$RsiL#*p^Mz^hzImycCmFuo4Sz=%!|NafdE8)Het2Oxj{)H&UmACm# zLO5~jVNv>@j#F7hnBM^~eh27|>c%zyBvu{@K*hc8?6>_(GoxG)ZUl7a;#OlUC9D5+ zJK5tvcXppmWd56=k(q$RE}%Qi?+Dz#iOA}+{2TiJ#nL*^V@GUV%Z`-DzrGKT;SU-p zj0{$q6*^u0TS)%^?(1T7rd*V!V9%M|9&yroAB74d+vFh_}81jJqf!J zq8$$u`%93dKqrwIl9t55jY3GQ`i-XKe{sRPax=ZGB`@zkvjG2!fXD9qU#EzZh#6^K z9$6LxWtg?Yphg&bYY({xWI67r;77TcF-VN^T>{%te+g%NoKrxcvQ5-s;xe<5IpuPR8a%pis^yuw(5VCTV&li0=(X{gUzsAXfif(GOwb?NT8dCqq%LA4Cr2-=T z|MBUOt77jjQ=H+0k)w3MMC zec`45HAv$Dm`bC%+2Iot1}n+Wy@AY)Z>F5dT!$vLV*)eMl$G(gW#))0)X<8jPHk@G zuzq)M(;;9Z; zwP>bT6UN+@=Qa9b{UrAT-=H&HJPVY!6Pk61@60oi|VR<7hoWZJIiqC?^#=+|5LPZjikE$CDtJE{a)YZi8;F0a57_HLHN$ zUIwX!S3aNYizjzfr~OR(0{8Grb?==b_29+6%6O0VelLwPHke!5(#rVV0v?nKV?jHv zp15tW1wjbyQ}?BdSrGHX zCr&_u27cIGm8MYQT;X4qiq`+ZQo--}V<1C5qm_|#g@5iD>V#fiURr!@-KUZWeV?W# z9n(jy@iyf;Qp0WE!AvpwP{5uXuwKs*6b-70z?|ZwsneFfTfzc!_qi86}P09sY@ULu( zNHHL|HzF!Nq;_t_?C+4^#yeEvD1vWP3RFPmNt_KxJ_ZDl@nGJg^6@r<;B)#KnJYSu~}B+zA~&TKNN%KGbHL$_4%OCd~Mlze4n6T zF!ZEf{kjO0E_;DTP_Sw+T;zHL-mQB6%*wK)q85Q1lp9@njFN-A0*J*f^Ork_9s!}D zd}Bq^5f9mgKVNmYY-A*#OvA3VPe34B-}%tt!#y)Ux+Z`!D(UEijEpj;Z&N?GI(#HA zSbrx@A6)@&92R+pp#A#;= z|6RioAQg3`alhbuVx73iykaJl0QKoNJ*K<7_vDWF)EP^tWV{uSD zgR$#MQ>Ka4(!y-HCXsDrO}i85P&nfre5K6%cnEQ@F3DWfJLB9ih}qr9XVt1+SYIFC z^f)r|`mEFkzn#|;zJ!UY=R@mQQ7bh6Uf1fSh=H`3^ZjZ27qoj$zCVhz+Ho)U-czKv zIXC<^Dr^po7LVyyV`+_(D|e?}Bg=euu*N@|MIZNqWTuDM}vV_AsU#d*?gPX8L`3d3heUNs@!VavU5S>T{6InL)S*L%F2B9AJ|<7)_a>8nYi(OX`^t9YYl7eKw{)3&+aX@lhSWgN9TXB&!u5=O z=5w|mzWNaF_Dsz7rSR32NXh>bDZSWlb#bCQ+R(#E@Do#MIXJ?O2$>Jnr(|C$k2vO( z9HjV|_fwdtbvN`iEM!vfCAADyfxY6@t%fI+lu|B*d^>0F_UZHIYYy_FU(ZoXNf@PK zZcq%0uQq^#COHgI5?D=yA#5f1Ha)P$cXWaH)JazvP)^@ScLNk)}@$a^;;qN z`W(_sj+-u6D}{Ha+m9K9GJZ|JBp*s05EjOte@ekR2J-$2k>7GYk**54!#mWe;~&i1 zbg&D}gIJxNEN5?0T>}bub=Vfrjw~!SYz*gq=GDk192^>)V^CjN14#g>eYxL#I{rmRq^S^Tk+gRw3LGzebHG&`3!nN=F=_zpN zPI!bAa|fG&ZRoo031mtaa{G8rypNw>Ft{8~IymGRzg9VUlLm;PadUfbPmgwK$nIYs zwp|yR9B;edv@(XoAqclZ$>W&}WJH|C#7jiMWJ*bZgGWd^W08-mRjeip{6SlI-ZrcJJMf5{dl>Mf)|?U?foT4x(PORx99? z{*lhUx7Iw_bt9Mjj`2d8XKCGFKQ}cHQ}FKeTW+E?3y1tQN(y;~Qb_N6ZOU4?gJr?% zT+WaX>aUKLS@%rJt=L?1XgW&%#qEPjTquWVM*PPdlDM@Y8rg;bsb-+j;<^0cAq)Ba zVXY_%LNeM9LhvuTao}?J1bGXKR3-fPZ2!$9yqE2KGl+Nv6@?CyX0zTI$@>DP<5##O zjy4j)cU)VI8I2slou;=k+%!W1tRLnqM9}Zzl9NvX%CmPEe9fAfxe0VDXmIeF?gUsF zjP9EWdn#SI!b6}u6lcd8SbdUPcZYb9NpFPv(p#s#aQg?mCAH4`yFU12XmeaD^R-Po z(;_;E32Bdwi8&r5XlU_9pkmpdLb7lb)4x%44EYU5zmX1+rzV)jMoGOj#11$`O^U9F zHrS&0wO`AM*$#GFmFE`eVlI7nnBF6ZNGUIfwOU|1IH6=Vq!F5nHqf9k6A~n*3RKLG zBWq`eHnp9cR}0CkHNVcz4E4!dFV{R*MC&>aoJrjF>VCCftm`H`NMVBc^GYTriTck! zncGeWCnqNdX)?tcO@x*96Th0nqY-(KHvwB^PENyxdn`dNQ@Wg}`}G>B3yOyz5EBiA z60h)Mj}?k=wOO1|QwsokbD}=YP>_A>EXsn?&<;f7`WuIspmrO5#nc^`Wyr)KE1c3 zVxs@>@UZ*OC!;r&32V$sOH0-C_3==I$7DR@aGklJsHF55xMrGgFvV@pi8~6B;;~Il zj9ijfrW1!7GmvZR^_6m#ftI|CWSy*&XlFo(pT{j|GdTVn9paLLgV zRULT`Dv3i!a`NZP;T1;i5=Lh&MR)y67qn~1}sWKHFdyZ{P8nBjKkCg%{O@OZ6t zUs+wUI__}-{w(pXE8w}xS-@}&{&r8(sTL6DaK@qTOL0$@7vB+bXzSwmRR(|t7S4S4 z-m2ee{e9+#*9WjQJ?xAHl>~=9zHs z_6PEK>D_l9y%YUO$u^h;zh>>mM=>#6ipt7G0oRV^c(^|nKVc+#x@8L~#X^|+<>*q3 zT+Z8YNtgWo(E}@@5zhE;mCD_d4CvzL`4;G?2u*Z@}X+qaKXde$K-?#6G{YgPMS)>3yiOM5;X#m!B;dn2}rScFwfY!$alme)}@jPvq!@0~{8UnrY(iK>%s z$e;*bURmq5RVLe^1}i=|#FD&S1KL?3&4irz*7QrBHbeEwNFz`R;`fITc6bpeL<3t{ zkZ=)H(r=fe!y;&41~8N-xcXcIn>q6a?s7O&^czNx6y2OAUpzEQ(pi*4gg!AnHE{V^ z$6<2eK&Z5p`m#?_Z_!E}@FPFTe{?AzFtDejl5~;sl3bjiUSnRIC9Bt94*(yY4)GFH zbsCD`;5PSe0vR}`SS{5o9nvI=y2I#u*d3xpbr~S7J{wALEh->HpRCQ}q*vxZ9b%%o zw%36CWyOYMT`1dv#iSX+Gv*ynuAT+(A?>gvt)o!+hfvTCRv`T0ZwSv$u@n}er@XF* z))v?yx++jg?6&;=etl?9bU~&uk_sv&`dbTfJayQ%DPQ;?f8c{QbNJu{YsF%;oV{cX zfcUmMEAnVE+9*(eB5VouH#nHJvdx}VOjzFHuk%8ussaa7|? zh6V>OLl-n#&fL}2HEFO5ZSp$H3A_e+ZlM0iGBGh-26sV>bSbp{%M;ZT1d80bDT9YS z4GWlRU7!G>zy_>Zuz{*#VlN-xV_+WIxt7sSf`h$#xuYNn&HP3Bd*EsrBiDsFSEvJL zKT0T)Z>JD|k3VuBJ#x^pgYc=MA`tawUz~}BJOFIpQ%6x9M;q_Hmd%@78tZ_FWFCJ` zMa7rG_OBceS5Bia))@uZZ0<+iqf0nTR*&+9M_@C+bI|->U#r`%oo5wiEPw$fFRS{0 zp!(9x=QV?)eD*FPb|}FcHA}Ms)AfhKtiYg&qFyZ|7Bf7^q+8_yEWMxvgpaRc zsm&dOqk?nKRzM%O8K?*L%I<-6h|)+S{ir!a<2JY-ryfD0wlaX9X==RVeD(9En@NNQ zh^TI;dk$TEPpf+?K2kS4JPY{wQ=#au{!{8d`sVT)uVI;(yGybiErZfJl6 zNmLJ+C%CvVLy@a`YeRpZ9q2?y*2tDbKpwz*+=yT}c7jq=69tixqgxmmZd@36bjS$d$)9^V#l}i zOGnCO?2W4rohMT1evc8w>|J8ET#89k6x>Jq!rj6)Y5i&2X;ysxmQ-Fs9SB2|MLHu5 zxozi(`$x#3v_&h^*q`Q@N`%C|9bz`G+!61V*Hy~U;Q}?3=VZErZ)Gn;BB1};N5R2H z)Q@InX1-yg8_3rxW;~zhef{2pd+1VajHJt+U>6Dm>tA(L8So{rXrG{UylM2%csbEyV=F7EaR-H@m@66B{E~+?-!oCob8scS{fo(wVMD*5n0=c=f^m&Z z#qt4*mZPCP2knNn7_x(SVK>umn%`jk556E5T9FBy{Rl1D9a*y*jI`Nr>lO$r6A*)pWX9gs zQs|*#F?&?PUTJfn!qateu2E1Rr1>_JIO?*6r~95z6|PPcaI0?Fsbe#p0Nl#EckN0Y zfx`nvR|oV;qKNjSL7SHfvij?v=m;H*9&I~*{}}vtx=3$ze>yC6YFz80oDz6eXb3j; zTURs*f0px!O3&{m?nFTPis;WR-wiHq#?}A;8JT+<)A85VJKnrWGG@m}ZvLmL1676Y zimo$p!Oxi7&2W!Ig}O;5nU>*Bc+k+XXbFH(_R#%5qE5e#GrB)+y~lm#JxycuN{7IxGRzFDY4 z*VWZ!cgq=t_H=hEK$oCDVD&x!t`#K6N=|i(Nsd_~7r5zz&LwGl0y`_u!GqQBQsPbE z!DOi`#?FA60}S)k^MpTZK4f#?O1jO(Dp;@!D3U!*68ur7Lv$1#UVk#&B9#L0E!RSN z??Df-^>klEenkqeR5&OS7y6Inf{@_a9z&t^f2UbMJuqUibd#=LTk-|M$_CD`GF|1# z7Et^of(|KYB9jJkDT)RW_~>rv41;5&B&@!+ z7B@Bi>s7TF_M#5wzI}@I$=cjQ4QZ^K9F0|CRumlkUZ$@{Z8#7yjH-i`x4OHiZSxCo z&kS^;wJGT`_l=)HdAovihZTGzHw-r>uTC&}0S$->rh`K+38$WMfh39R6ZN^Oo6CwG z%-s!Y$-f=%>f)fEDk&lAg^v@TF^Vbu94cKUP-5jR*XJbtun*mSyZcSE7Xk zMsvSjo10|P;6aR$^XBhI_Mib$^lcgmsP-Q3Vu#>s7$0LA&5{bRe`HLQDJRP_ zN{Wh~0?)l~BqxZ2+`vw+-V5>NLaP`_GyjRtbJ{*GM(f3AL;tYv_wN@IVY@Dmpx8Y3 zo;_s~Yy-cS`6e2z!J5;zQSNasqnYw_ZVf_uN`1|Zt`ChhL5O`9_6Su8!d~hhkZLRM zfk>keTmpv8Z}g^zY5`o7E?&I&?(0gUKNdkywMRlTgrF4lVzfa{d|i}hN=CJ4|q&rkJNXD?hCtWDGab>G{3wj9Lzw@%#GDAjo!=wr^JWvEl9SSk#e%l>n?3=fI3tg-1@l4b3b*G4B~OBanvTIzH~%HCp`r19RFPtS zZG*a{bL2-f*xYdM{vf)tbWS{e|Na0V^kuI$==)_N1ff%R%fucr`*xoC5jg%D@gRm? zpne22)mYer;_JI93xuA7jW&e5L2xrg%D2760d9N7@E_8o*l4CS$Rr-|G&c9jK&Ujo zrw3zdzE;7?>UF=>Q}FwVxa^|TP*@e%KeAHwyRsYw5oR+Bi_R$hh&*i08uqlS7?GCd zWJjZV@6fbYHo@FHrDBN$$S$NBz$>-m@YPe9p>G$-lvEE{YcOIt$PvU&rt^4nQP+js zq7=nh*ps$RJn#~&>?vT=?~N0`t-<8irSvX;%d<|%;)U&%7*5#SF+mtI#-Jc1JO@aY zJsISP-hyQVF!}cw4yY9EO2Nu#7UZe&DV`H&j#3FrpQ{Bezg{jWy9AV(jAqlCerQQ5 zI6D_^tLTg%FIYwv&NBHs9*GDlj8)FRxkVFHE_0h>ZcWb80Rx~T*kd8;UB?iNm}8(N zo;h-Ta$+Kxt0JMc$jKg*WfR>}JxV4;E$@i!EbLI_Nwbpl*RG}Cz8C6htKKhcekG5p@sQ^jd=*LiCE}Q_jcF6K ze-)s?i#(~Xt$I#^<7G;b!5BGDaNGYEi0^QtwsM|v+c*G4!HjIw%a@<698lj@Cs{p3 z*uz9mFHF^uaQKomc)ZgOe4a|*N8Q@1@i?*)SiDNLxg{uxw{G2v7w9}nhCv3D6wg00 z=7NhE5^`SDaeA^;w4XV9b~ix!FSoLO=??e=)~2h`(*r!k92>TQ{rk=WkDff)jih?0WxZf4mEhM` z7v5d&{^%bg%}7rD?j>kXM%s^mNo}a_l5_i1`M}2sJ9vC zgDd_ZcT~ntkWY4ZmHFjJR)H%tyQJ>U3ko9|gEJQ1&0mtGV-i(AP>*4w?9g^^%~|ZFdA=*yNJcFFdBCe=qcT_)-whrbLQ0f>KHke zsiO`P&yn34!vU*Ha)_ovj&O&#PL&#iXC`U86LrOp=jDd< zJ0;B$Ob?v}IsWO>Zvo|^WM3HPafC?Zvru3c%hWoZR#an(Wn2PF!t2YQn0ECue)$ov z3C{*?_=g@C8R&H?KxI8L&&|ij_r*Sjyzkee9J*78 zW^lU`PSJ%ObE?O1gk-9CLmS;K?`z7neYDs!fPgspK*_v)uWodm&|P{B4O`O>UX0xQ zR8ZY^U(LuRdy>@A=^=@o@foz7mOLwJy9((NxH)2wPNR39s}EIZ>G&vdw~^L@hFiU` z2`!V26MTX(^Uie3^G_FD+Id zvVio?3v+$(2;sr}f)HNw2^rxZhE{wh?@#JL-A`Aq0Zv%X_lckjo0*h@M`+}betke> zq#MsUO94zVN_0b=u|Fgkkyg1}20q7>Qs6Z#RA}=7LLgM*xWO=uH{7h&f#-XKNIZYX z84ZoA_FJJd3qFQ{kDDRu!ayPKpsnut&bN1imSbVnha7@>>e<0d_du{LXcZbodqA-# z0nt3zRm31-Qy7sz(*sTj&!tz1QeHv9!C>X9H>T{>h0`5=_hlP-LeyV^Cj?5?nwtGb z$-;UR4b+4>?(d;>i91DI@k$Cb8erX3VWQlzCwu}x^!!v=Nq^N7?U(|-(g%mn7*IHT z%xcXFI?ZnnL#95wLGz|WYFPApQ7^VR#12jxLPxr+bYh!KciMUKk_*xqJ;J2Z*|TRq zKG?#?KakkF{PExP^h^VEE=Eq*NvM0G3E+O}nfR7v8k}X2z-Eg1nwxuc=dXLb+&EWA z0(aVbDVOL;I!E;1#V&VL~5NADlHTacw|lFX#NF$aUbyfTUVDhIU~ zQtfAs6V^Nm>7H5@X?zoxch+YZsW!g4a9ePWz>O{?2s++7rC4cQFph8`fPY;QV6sGy z&p(z@IRU~LF%E(?xAqPK(RsRZhYJVkD&e8uFAf8I`ik7(5$cm zqx@caCK@(xk*<0eMY?Ff6q}jqq?~F{9AaX<#bF*Fhklh`)Tp_wsgFw?Dr%pDfRvP*W=W&55TFCbb zZ1|yz(FG;O)hU*?v)-fhlq{bM`e7MG8J1-E55kO}lk8VUuVSDA{h`+K*e0&1a?u2e zHm^|u8>Y=>m7JZObzk;Cs1P_$11hYqwX9*oUR3XZ7B=``Ps5sI+2pu!r=*^c|2}EI zZ_XV3Xfp&(36qa65ODhhJtT`NGRF6(ULof$j}i#K4n-IOLptQ!cbw|Kp!=0~_wJ8> zx~qdjuGKfm!-Un@{~ysE+DD&*G~PMDpu@>#T7yjFGfc6%1h-s1*a;4Bq$OYGv&~3J z7doPwhkd__W4ZW=YQeN?)Q@4#;7Agh`3tB8XP2{Xr!y(2WH&GGiO$g${Fy?J61e8w9-gj^r{9r&D0l z837l@+2%z!ZE46VctR45>Q>-@9&9JYZV@bLkS~XtLHS#}9F!UEo)+iefB=ti=n>PV z?aA8JvBd+niZ)FI1B=&wd}4*g?@kl)w?mZ?BPfhYK!u?n^>)`Qi&inP@i6Y`KX~Qy z=4!T`*kf;_m3F}U0mO4f6+|M4U$P|qvE*#;1w|IPm1Ei5cpo>hj2vSfon2&VpSj>YUo>mHIAGysRrk);}F z3~q@sAloxmSPSjV6CGf=U>OJ5JmbhFwnOis96qZD5iKhZ2x$|RM~5+B(gHD^h5$P= zlC>Z6D!(4N-qqDrRC}rk$rpc}2}GNK+NRH8BTbAn!=kj38!expAVhWN*?0lUauwL} z^yrNkbGJ~afv;aPz_%`CKY!j`^H>yJX$Hd4lVP0?AmwiA3W5KO@MJMNIU-X9);ouM+HEr#If$XnG(F(JIq#>~9J=h>{2hra5=K4(7UF{EsW$mnM zfysJ#H3ss-@n=_$QBdAs?CSU6!3OR0i+7D{)G0m#AegMm4^E=GU|}!8mqE&x-TVZx z4+Bl*pz=%5Ev;t`Z$;!A+NF|XY!&jvWlZxtn4=JMizH8Zq8u^Z^<%#n@Ec!?fb zMp4TS_lH{#z72Xlxa>DQ+^7zkRd^7p=L^ez-vaAJ_oDa9_IXigB%YRfbBvQ$Ht&GiD1+Ww zfZ+keJ7q8Om>2GagH6dt4ZxTqDt`t4hsx_SlMj2NJL;BR_xv6LP?+-Ooa;G_J1GVm z&ES%iJ}i+B)!pB=2=Z75&bh{1 z>bE$Q@83X2_`!!eXpv(esH7VlX9>MvL@GB7+iv(~U>~Y3rNB`5&M)>vDC783O$MYk$K*gEpw&8Pp#<%{0eh&0_KZ)I3pNk?fn&7U1n?`XM-Yta75X2X z5ahmx*^_-3^nj7Jm;kL>8VS4MZx_Pr&U-@LHX{u;bVELwzTE0-un#6Guh?d`9ISdN z4h)hrQFu30(_XYT0do&l3v>)>4!!()pxIBYV?0R7J7-au?jnJUNJ=Q$(sk2fd_LqHRPR)n>!p6 z0JAn;f*@Y%-V`vF4aNemRv?eeJYfQ1oC9J+fKjp@4d>D!@bD|Iyw36ke{r+Z$%m zj*efWt)^}g==`u1&kUR_R)b7*umcL<9c{JZc4 zXJ=>q3wfY9;=T@M_vU#VC@}|l|3Y_%-{^jc$p-w)Ts)8{)Lz$mgB1ggLAprIfXvJ* z`iu-np@eZ%FStD%*o86J@(11T|3UYr%ji4~Nk*_isIGl*l?%5CiSG)qi2(T8;w~9C zDzN7RwpWw-ILM2D_I)-HoTlbv@7Qfp%f;3L`{00pfN%GrrxD#ly6dw?8(^;8uWASa z2?04p%LtC)GS~;T93b+ty)Oqa(k(z)yB_2oWrNq){nt|8G52f4wA`@f>sM|Y`Il&2 zp@S4Z#wxYrj6IMch60{ZmwG-2q9*Gt4*aVwiW~ri%tIm^=fjIkF&9fWRIjS=;1L&G zQQ`HFZ#z}i+nSN~5n}KaWCs6pzqoj_mBTCFBZS+hG;zjv;X^teA9!FXLKmU;S9%%8 zV<=ghTx_=KfqY<;giOdDL9pVXnHP4OK>QD=jx~!KSUC%U8b@*;{;Sa{3Mb(8R~anI?}HZDDuQ2c5X)uY*Uj&#nr=Mb*zEHPA^ ziSL2dOF9t&yJqOA0{GMwn1Nggsum6pW5nT=po?b?4st&b;!(X%>nOV>(g}q#TGv`M zaE{b|r|G5TU!&MB0Wp$PH=}W5}mNc@P^Mbr~ zVTIhI__=~{EVnr2X`Sg9H#KaayQlO-278N-miCU)o-sp3srE00&m63$PL&K8y0nkC z4W*{0F4d_hPr99}%b0XIb;=1!$P*lIU{AKIaB1e&`&85z7aFYO+yhWx0@mv}4#b(j zxhUq68w7gGK5Py~#|i}KCzkIbxWgy0gS&Cunr?8gPB)EeSH7ymkM|2R&04r~Tm7|?(;9p35T+oIZt0fC!mR{ zy4_DGk!>=|sF=XKFipwQ2uw<`z|uZ^43)!CY4J@SKQ^PIQy<^v5G#v`ny|E&J+zV+pI-7+gzn_0NPydp!X;0dmc=!@PG6^1*B@+M zN_7n&pIW-Fx9CHx33q>bt6g)P)>j$Qr2yf~)fa5qFkG^{Q7z^I6t?ES$P@LaGfWrkBwSdYN^gZ5Jj8 zQ$<8obH_oSo|Y~6~VqNrNceljypMNK<_eKZw@u0A%=G0K#`fo5P4z=hT&?3=D$Z1vyJRi2gW^d)W% zDsPYecw)%+&PxQ!bDSADsm3!a#!4u+Jy8Cit=u#M$Nkf$BNJh5gqQKT4vfMX9g5-4 z=KRPv6g#YgJTQVJ@$Z+*!x42F|| zt22woq$~Vhr_NvvbTi0d)z@er0rDvW>0#JQLCX&c`@D1NB+3UK4{IUka&XVEs2?N} zwb}^)SxKN1gtMI>^dXx4q5eFTl@9ep7x&HaR%uRt-kUn@o$Dizn5_%B-M)Q5Zm_u3 z4L1Ng7`!>-1j!JGYb`Gd;nE{e+^%oVF$cQ7|NcOTIh4nDz*96P9(j)==$Ny^;ejEC zS1n@=;C58g8Z|gsbiT zDN*lph?Kb1F7&|u&kZfW4Mb_}%Fvc(J$Cm9wa+)2aNA#Lra5AvhE)KUo`h~y-JFIO z0^jjg>0cMD@h+oeFZC)UzI0tvJn_FbdkL-yP<4pE z!9`$efwkn|Vs)26Onev!ln;kR+}k4s+<5ru2oAq%x%g`alsNlAI`tK{-jJQ`#ie*P zW`$#Rwy%rIh9gd9ziU;@A8D4D-Fztp$0orK?%9c2rfz%W?NS@AGP5reE|L&_y=)-I z2x${kntkucH(LDpbhF;Aqw(NzsWa5K_)X;!41Tf*osU|XP}ku@p-)8qk4tK2qIdsc ztyf009M=-7)n+dPCu8V=rmMpE&OK3UA}PK5eu}qm%uL!dBTq=<*XIU0=IeKWf9cP< z+6(QDr(`$b$a~Obs5Dp&tivpC;AMNP&j6LSk0`rR%IT{*8^GC-EC)BCdVULPI=&b$ zKj6MHn``tOz842q<<0vmC)z7;2Ry|eD6nfSZwEqYXSNUdvM_RGW`JU65>L;ktu#|& z>KSehQuV1I3q_S#_z@;U<*yUZyXW?h|G4t@aXw$}$tudWdn%2Uv8`OUm@~-b^l-+- z4*{qVDS>%u+E}Z_A^eM%o4^cuMT}3MJ2I|8k*_$l@g{rgt9@C`An;JA5BnaQrB}WJ z`Hz?82%l^gPQH2m&cutbPG+U)N46k>*jBv|HqI_!(~_N7AFh2f$6jg44#UaH;bow-j3tiSI=MaLU02Z6b2p!dYNgKwVf z!W%PI*H9(eJToLUpjpoqJZPIU2vwe%b3I%9+U4N&(Z#k$^dFI3V%-zr!MCY;U!bf? zUj$>@0a{25KOwaWgj!3(ky6ZeY&7hwmTK-@D}!;RF{g$d^*9fo7sv`9Iw*XX-``Q< z5uUvJHys%1A^WL5$Cp-h>-$(H*g1@@i#x3BL>P$sVqnE8C8tsBg&KO6LA!l7VGH5kH|cqUwHZ6+gROSy*xLeIyJ} z`JQT4qX?(N zpvqF8*l#nFRfg`{H{si#7e$dXwP|SUgS!f96fueZ`UT!NAt1TUyua(={Fkyx8$S-l zwFhqV4P8Y{vyM`DvHcT0uX@E&dM*(AE1o5zUkRpD3N&>=%SPWeV*Jk_{~j$JO@}A2 z8XdmyKgLJ?Wu^uoFxN~v`@f(0pNHf2fnGVDF827p)A{pXxevf>knKAs@#|b!(P&j2 z+{I)68Q|aF18spPIG5bB{QIlXp^h`a8il^n{&k*Q@Z2vq=u7{$5-_L^3(J}a-No2n z=P3)%ebvG2eD0SQdnzC>Z;W~+xbN=|#yi7v5$$#NmlqQ`2Md8V(*U*fw~@ZUb8|J7 za`*iDVs~L7_T@$#`}@byua3fV|JeZ0ApdN@pJ)4j@eQaOIQ}$G8+AzX=D+p{XM!gz zGgS;_A32UO3fb%a%0q#30Tf;0)~Hk$FiFJA4SIJSHC!iKd1OhH^$JSVEOj~ooZ^J= zyf1Jb0hfe;*i4;Z!BVE~0>SBO$ylOG_V*R}n*E~G4^SAG$ItNHJU9YfFOXTS0Jy6O|Vd?R5W0i(nCiaT6+V$YsOEsM~` z4==8j&Ax4xObUjYNO{9bFy%6VMdU3qW%V~BP+h8r7%v+eJ;b37(Su6UZ@l}Qal7S0fKE!$v0{tJPlmN+6D79 z-;h<`iWn{aAnyo|3yAyfol*BHzp!sQfU@4|dufGrI8|QrYfG{Ma4f8kJ}bzHz<}M0 zCs5gaYjyU4ogbpzro5EQ8Uh9)1M1x96IDk@H4jb`|mi{TE z*{7ihWw}usoCm$fC-%d`3vZlw#^ZtqMo>A@jYr_iDUn1X@3UKqNS&yJ4joxg%69#= zVLWsY*;tAt1RiU_?rg0GGD^O;@AGO?={LB3*{$?TC)i3UQ|-P6lL_W}c-R+3{P!h! zp_(;|D1`^6jdygUQO7k?KZ7?}VK-C7V4v&=R2Z!}irY(FUBC@l6`M--gqh|h*WKLw z@ZGi!@Pj9pfG71_h$<{gPOet$?E%lmEjf0j=vaPmL7DIt17>1|0^!o}<4Q)KqDgx1 zHjl?!cSdBSby&nMfq{{>LM#zlLXZGGau9`w!wLCKQLSmnXq zA`+1R%Ti!28a)}%vSn`VsZkW98dVfZp-`9k^4~d?7eD3+R@j_=XE;k-p@i%N(-ShW z)8-#pxFgE=e=Y;H;KW4}N2XlsxqHltxjnb}8|>0=R8g_-&Xn!vq;{;ui+s`g_RQS& z6JaglWQ+zKumu0X)@9=`D zf}pcVbHBs6I%+VNKC!76T;Cpss7&IKSyD>i!ly|PPc31jM0M=u*S3X9btvWciMVer zbXZ7vA*>-}Xt~f=R_iUP>-sS`yLx-^u}fheYG#3`R8nK7Z#8Eb^F3{&J7+;;jQrN| zidUvfVMVsjyH)hhlkHr4sKL{FxOU#&D)E`ty)Qv?}k1|qLu zP{WPyy+$bkA+jQaYOG;SlXJ853US=dwB6*2OdbpB!0H|n{1Y}f|_%6e4!LD!smRHwa>gl}>#SJ`M1EO^@_M(Q7PaU6W<{l^{- z8Ekp&tKfc!Q$OTvr;A(Ew6>LNvBqNXGCg+fT!hqeUlv)a zW9;+uF3CtYwDWN=81u(&7ayH3@CZ5?gfM#0Iz+6lrYCK#Wy{cnkCNB@XzSyf7e`<@ zJqkyFM!nl3jR|DI4dGLrv}~KpGi8V6H!um&!VXNrq^{QyZ082?KP`WNL$wPp#u$~2 z<@w}8GPSx5T5K%0@_pLijg08M;;$^cL{GC#VVM%XGNWW!CR8*__i{|LPD!v()cO3r z;nxr1(7aqHy>k%7M;hS}iWqW#(a{k>3V(b%PkN68y^4dKV`~6W+&E3Ufbw{y`3e+= zqb<>zeUuV5>4|>q<~wZj=E#W+K)d*5JSZtUi`bp60X6~)0&BImpfV;8YTX({RBc4# zGcCn$p}RYQDu4Z4WXCvjljC(JaOIqPCwyvufCD?;Ou(t>c_~c2e%C2kHT`&4R7@KC zcJAqu;x`eCCAuW}J;igtjoP+(^k9WL&XllUOh z*Ayx=n^4>t04%8HcH>fMF(99cE)OHTz+(X($VGTscXK1j;+WJ%so%^o zH(>r3MK#9Ou9OyKRn1oB`}YEtlxhGt(pCla>|D36Vl#qO*BSype2Zf>ApW@W{ZW}% zp4sgydUgXK@DkN?l5gqQa!965cdq8lEqwrGD$LK}5>&!{c|g3I8zqZ*^WF&os%dFB z_0N%w1PS8Sf+*iMCt>Z*Qiqn;gM)$Jc8lI+mbMNA?QSSta`XP-OCJeW#qVMYFiQqD zBh;AUpSPv&WF$1O#v)r*Z6^)js@4$ux>XvjQz`?wNfSb+l4)Z=7NRe$lGPO~k$Tn# zh^TXbu2YFV0=p1P$mE<=0Vs5|_RK3*`LD$jwk{EkJ?!U30O+aqiC;yA&_R@e^r*+# z)hESrHq6T6>2vb_^5RXcy>Kz zSN%xjWt`mCD^Frtg$}{BeIT${r-CiBnMAFL2G=NEd%K-bRkphRgH$cMvK_8avII8w7DCk#?sLAP#K%abKIb@T};71=(IH z&eA!L4SO$(voC1n>-BMQA(b}!(S9vN?lO3H@@uMS1z^8^Ds6~&i;V%`m<{g)!7-pl z*?Xb_7c~%`cZ`aqljiZv6lE7m0bQ*){pN!SbzmhnPPW%ms(SvsmuH>DStBIkV220) zD$~ln;MvhS(jDRV=iXC`mYzr#sWm>1V*_rg1PitX)U z<1Sx&&ZbtWmFPgb!?V*9vfYy<$>P&_K{!h)WW9M3ycNSn9X}#FMM7mOvY`UOLiq(l3^aR7A z^_H@3r~63W;HEX)1`oyYWUo7JQ8@+T&?%#)%xB>@iIg+$VimCh^Dvze5z0xNZHb=tfC!GplJ@Rl+p~j5VtE zu_XLJVQ{_Rj|SLW#h6W$dUEosu2!@$39Zh*vjuK>fAtGd35S+AYtw{~XaN3T@cg~) zjUjMKL(~tS+zejeAg*cLfle=`Xcx4KX16+XVm%U|7hKNpMKBUhLill_ z>6)q2pFrh*f$a1{f8u3Nq89!Rp2IFy>eq!6++C!XkN@}Mf8yne>?~^&2|Aj;4%H8L zyt07Q9sdj7K4AfPyQA2VU-5bOLoiMmyQ$r;133Z^6O(u4SG>JaAI9mo)C>A`pcrr; z6tf@4{`y=w80ViCKrE$ycHplw{QuDr!dED@iVEQ$qiJi6*z7I?B?;0L&IvFGz5Lg2 zbrKbQoRQ3-y-jkG{T{XJV=_5Hj*h|-8u=R=HBrTfgD)bnDGcnVz<2ULS}Wq z-LdJ}B}M!!rhP81=MsGH8bbWssWe25Sul8c1$CV9^Or8|<7sMC^xngeYa`zY*1gh= zRB{TEUA6JY!Qe-T9ZYT+S)l>0+Kf*`T(U{CuT^8o+lH!>2S2Bho+0{d>$R*7v|8rUuQkVW}PVEZ)$o7OPw_$N&yfunsc z#x1`VVhod+PYiUB9tB;7!3$5U*8+wXQ`uI(BVd9hgu)wEe%BE)dK+4#qqFzY);#*9 z+IA#yEi#y11xA~M-Z(A&bKqZ;G;q0O0#qrX|NL~8oh>)**gSa3Khm{akGQJ0q{Ym` zEQ{IOi=qu=YSR>c=4Nzaw?1H(PEH_rKGh34Q)P2T%kOM0t^(Xdhgv=a^DAsYpstdy z*OIaTmaZ{BM_h05=mbwPDa4^<1yaB|U5E$KViGJus>e`=MCXAUyD>Lo%c~xvb->~o z(m+=QA`WV2=CHI8EOSEnXWnQ7;`oa&zMoF$!%fO9xlO^4?{EhJn^PN>(zAPz0E6+f zNqZDpk)yrH$6eN}6_D_ZuLRjzr|F8Vg=R1Qz%G6H9HJ!X}_EeaEB3Wi{KacHpb zG?#C@&26WZg3UQ%NE!sC<_7AD77!->WiZV+Kj&T%@P>4_n&?W$_vC{H!fbuMTvX_S zMN35T>`FeJNXpbqR|tZ5kH>7}0dW5tzuo_;n|>akoWsrsE*`NIAfp~C494;WOcw{! zR|7$y|Il}!;Y;PPkJwcQCnX6872+K1s+NE(HfJ~USpoV34lk;a1R4rX>)SA-59+`< zaw~cfOr^Jnj!SxJJ6q$MhgabM=Kw~aTm#)l44TE?>Dlh`d3Zcd8)|fJLSAJ2#wbBn z{$(M8k=sfgBaehTasZUb);I%KY!V_3Q@#T@ZinxA4m#~Pe^+5#x+Q9(6b~sWbx82W z>%qi94kYmPJcB@i@I+$&Qi%S3z-c(uaYUt421^wrMV++u&(}Auy4v)Q#6xdW;oLBHb-pIEMrxnv?HR8(%Xr-5g&Y?N(Ob}VT{o!81#`ta3zOmdv_ zy$6FP)KsFexPmaIjatl`Nzq`gNMqS%*)La(lGcyCyY)HF1Inr_^<0uz~B23V6R)X-t8qdaIJNZrqUutNk9k@vr_2 z3$#3|R1?Vdlo`7zpV1zj^z$0Y@q6o<+?#|xDo8seLs=*9*1TeN?N$y(84}B+plq;c z*El$TOVFH3(IeuwPU$OBmkQlqAMYHn{lPyi%UAFo_H>~J`AL)rZ#mtMdS>Y(7f*?2SRco%-`+iqQ$i0 z26gS>3v!TY)U7HI$%#p@=cKvEN|Su%FA0EgLD3-6>J6m76W){$sQRWW4N| zMpP~<-W|yta(Bv0>^^fTBBoTq()9D!7~zU~!&nFBCb@oB(?C7A%9p)%e0pqp4sfYI zupkIMP}`ZI(A=1v)pJw;MA>2sbtwG_RKO}h-x=+0#Bp@l2;!uHS97fFk0JHCmz?Zt zX*=s_{(X8s?BH}R$|Q=MY$EK^>Yg)A+b|;z4`ZUG?`LaAW4Ky=8-(;n10)%)xcMX` z9UV)zcP#kUUrICjdCZo6-Xslt8yY+tZMQqAYS{vg_0b9qGDjMK#k)bfLqSwQcVl z$Q9S}ewbY)4o4(c#cFmHkzRHoVXjID#wFcv7|y2th=*1cDJ^aQ3PR5Ew)@-)zdo_5 z&pAJ&VYr8kEsA6(+0Yui|&My2CAu0fX+^zm* z!p^#fvbnOM%fnsgUk5-?Dj<_A{JcJb7LDO`BS^{Bg0fk6!>n=M;Pg50Jmy6pi7jq; zt+vQeb^VH$sh-;r5t8+Cu5Xpr06SeWKBhEf7k>6IXjlDyk|ke(SMx}S>#(R5q&3re zlX0RDU*m#v(<^)R^&(~}SUo~k#UZGAjc2t(`ndHs$LZJ4%nD(O6}3A2YIy&#=fb{) z#i?4E!l{0OL?mL=Rzywn>z;wmxI7yPIx+Gsqjjm2FkyTM# zd;iGRBn?tKB&DTLtK}5=| zK%@J*5uVfCJX25v(`ClBSmiao33_GR_~=Y_Vf}nHr|jG5x6NaM4w+iYd#N-)+j#qrqb?r0v2)S)i*ueTnr0{Pa!JFP*QOXu zq8}~AWX-Bj64LNy6kbjz zxMQtKBTf-Wem^lPT2^Awd$AWIG>?GW!TzoSMjGI-iIqhtGKsOIxMnsyF-crU5~9a8 zE!JiMxpfVWG3ErI-HOK3=ngjTdeBIoWQ#JQjcs#9<+gsB6LpM-4x3wd3}CyHIA=LT z$(&!2e1VUThZ8KGE=SsJIndriV)=1A;d}##vio2lKbqL)Qp?S-i)Qm$nS;I$&}n%4 zMqQePE0Dg0;)GEfgDW~<-356yLSxjg)^%7pq}$=-BQ@p=NSBG$LyXBf2-=4TY|&-c z{q@TKN*H5xxu#Cf^I^uJPbcuCALL$VbN+$py6nKF8MVIxL7E%o_;mZN*RkZjvyjHn z^;;NkT+XNZK7Oj!n}qYG%J;MU=LI~t2wnOi%jO}^c36@jR^m@bwpbHoKV1W*ObfBb zE7F1zx+AF8?@_8pA4@ z`iPa#E!HXB$n!)JP75yXvu~3Tgv)Qiy;<#~09~RSL(mMao4oMBeqD8K&?HilqjexU84>w9zP6GqDGtn`$AjRU~&D<6^S{0$e@>%$e zI){Lq)hCno5i0v&=vT(ZQoOFFNT5?TzU$erIZLHE+JxwBPN+Be?OHS)p&2$aL5r@Y}7L3 z4*B`d>zpup43g}^n}~wiIwS(pkP@7hS+&`&NE7MBMt;) zf+WJ=&Jr(GK~UZXL3zdFw;uV+=+{)~%a)x!2do58N2rZrozUi}%d?R;-hshhy#!n% z^0js$@*K+&r(Dr)K)69Q_o363j?2LL)lsmE*~Iu*$Q+D>zH9 z@bZVnXBqYjDJlWCz3^QYNeK0|L8SQGkG-i#nh^^mP!ccwm2i0~9Beigo+<`2?Qb*D z9uh7-M z1C}y3I0J*d&Ofa=K!8-1luZ2s9m%r!SuquTJr4Wyw~S9&3}k#>P^;4WI|5~-1;Z$o sRl1z`=i~pZ%s+?VpX>2|OM*4KaYdgB{~o>6UGQ^3!$7_8y#3w(1BN5&X8-^I literal 0 HcmV?d00001 From 7bca22e26018e3d33382e2f76217711eb7b13ffd Mon Sep 17 00:00:00 2001 From: jiacai2050 Date: Tue, 27 Aug 2024 15:30:43 +0800 Subject: [PATCH 2/5] style --- ...c-engine-cn.md => 20240827-metric-engine.md} | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) rename docs/rfcs/{20240827-metric-engine-cn.md => 20240827-metric-engine.md} (97%) diff --git a/docs/rfcs/20240827-metric-engine-cn.md b/docs/rfcs/20240827-metric-engine.md similarity index 97% rename from docs/rfcs/20240827-metric-engine-cn.md rename to docs/rfcs/20240827-metric-engine.md index 3a3691425d..1af511b7d7 100644 --- a/docs/rfcs/20240827-metric-engine-cn.md +++ b/docs/rfcs/20240827-metric-engine.md @@ -54,7 +54,7 @@ alter table root 这样 `[A,E)`还会写到 Region1 中，而 `[E, H)`则会写入新的 Region（系统会自动创建）。 这时 `[A, H)`的分区可以删掉，写入时根据最新的路由规则进行，查询时由于无法精确到 Region，所以只能查询所有 Region。 -#### 讨论 +### 问题讨论 如果查询时可以定位到涉及的 Region，比如 ID 采用 `hash(metric+固定TagKey)` 的生成方式，那么在分裂时可以通过记录 Region TTL 的方式来减少查询所涉及的 Region 范围。 对于上面的分裂 case 来说，对于 `[A, E)` 写入、查询路由不变 `[E, H)`则需要根据时间戳来区分，假设 split 的分区时间为 t，表的 TTL 为 30d，那么 @@ -67,14 +67,11 @@ alter table root | 规则 | TTL | CreatedAt | | --- | --- | --- | -| [A, H) | t+30 | - | +| [A, H) | t+30 | | | [A, E) | MAX | t | | [E, H) | MAX | t | -| [H, S) | MAX | - | -| [S, Z) | MAX | - | +| [H, S) | MAX | | +| [S, Z) | MAX | | 写入时是需要找到对应区间内 TTL 最大的即可，查询则需要根据时间戳来与分区键来路由。 @@ -99,7 +96,7 @@ index: {Date}-{TagKey}-{TagValue}-{TSID} ``` - Date 默认为天 -- TSID 采用 Hash 方式生成 +- TSID，与 metricID 作用类似，但采用 Hash 方式生成 - SeriesKey 为所有排序后的 TagKV 采用 Table 来管理上述结构，不同字段可以直接对应 parquet 的一个列，便于进行针对性 encoding。 @@ -117,10 +114,8 @@ index: {Date}-{TagKey}-{TagValue}-{TSID} **series** | MetricId | TSID | SeriesKey | - | -| --- | --- | --- | --- | +| --- | --- | --- | | uint64 | uint64 | bytes | - | **tags(可选)** 这个表是为了加速 LabelValues 的查询，如果没有的话，也可以通过 index 表查出来，具体来说分两步： From 4b6245e836c3238eeb80c47a79392a13d2e35a8b Mon Sep 17 00:00:00 2001 From: jiacai2050 Date: Wed, 28 Aug 2024 11:10:35 +0800 Subject: [PATCH 3/5] fix data design --- docs/rfcs/20240827-metric-engine.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/rfcs/20240827-metric-engine.md b/docs/rfcs/20240827-metric-engine.md index 1af511b7d7..f47a5b4f7c 100644 --- a/docs/rfcs/20240827-metric-engine.md +++ b/docs/rfcs/20240827-metric-engine.md @@ -197,10 +197,10 @@ metadata: min_timestamp: 2024-08-15, max_timestamp: 2024-08-15, | 2 | le=0.1 => 4 | | 3 | le=0.2 => 2 | | 4 | le=0.3 => 3 | -| 5 | __name__=http_requests_latency_bucket => 1 | -| 6 | __name__=http_requests_latency_bucket => 2 | -| 7 | __name__=http_requests_latency_bucket => 3 | -| 8 | __name__=grpc_requests_latency_bucket => 4 | +| 5 | `__name__=http_requests_latency_bucket => 1` | +| 6 | `__name__=http_requests_latency_bucket => 2` | +| 7 | `__name__=http_requests_latency_bucket => 3` | +| 8 | `__name__=grpc_requests_latency_bucket => 4` | | 9 | http_requests_latency_bucket, le=0.1 => 1 | | 10 | http_requests_latency_bucket, le=0.1 => 4 | | 11 | http_requests_latency_bucket, le=0.2 => 2 | @@ -219,8 +219,10 @@ VM 这样的好处是为了可以查询没有指定 metric name 的查询，为 与索引设计类似，采用带 segment duation 的表来管理 -| MetricID | TSID | Timestamp | Field1 | Field2 | +| MetricID | TSID | FieldID | Timestamp | Value | | --- | --- | --- | --- | --- | -| uint64 | uint64 | int64 | double... | double... | +| uint64 | uint64 | int32 | opaque bytes | opaque bytes | + +Timestamp 与 value 上层自己编码，会进行数据攒批，比如会把 30 分钟的数据压缩到一行里面。 底层 SST 文件会对应一个 sequence，不同文件之间 compact 时，如果遇到主键相同时，根据 seq 来去重，seq 大的值，为最新的值。 From eda24e51e033d00b747385e38803b1d8789f4bde Mon Sep 17 00:00:00 2001 From: jiacai2050 Date: Wed, 28 Aug 2024 16:02:38 +0800 Subject: [PATCH 4/5] add field type --- docs/rfcs/20240827-metric-engine.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/rfcs/20240827-metric-engine.md b/docs/rfcs/20240827-metric-engine.md index f47a5b4f7c..a9de9142ae 100644 --- a/docs/rfcs/20240827-metric-engine.md +++ b/docs/rfcs/20240827-metric-engine.md @@ -107,9 +107,9 @@ index: {Date}-{TagKey}-{TagValue}-{TSID} **metrics** -| MetricName | MetricId | FieldName | FieldId | -| --- | --- | --- | --- | -| string | uint64 | string | uint32 | +| MetricName | MetricId | FieldName | FieldId | FieldType | +| --- | --- | --- | --- | --- | +| string | uint64 | string | uint32 | uint8 | **series** @@ -147,9 +147,9 @@ VM 就是采用上述做法。 #### metrics metadata: min_timestamp: 2024-08-15, max_timestamp: 2024-08-15, -| MetricName | MetricId | FieldName | FieldId | -| --- | --- | --- | --- | -| http_requests | 1 | value | 1 | +| MetricName | MetricId | FieldName | FieldId | FieldType | +| --- | --- | --- | --- | --- | +| http_requests | 1 | value | 1 | 0 | #### series | MetricId | TSID | SeriesKey | From 4f788192f776572d2fe62c1a246a0c27c51e9509 Mon Sep 17 00:00:00 2001 From: jiacai2050 Date: Wed, 28 Aug 2024 16:12:36 +0800 Subject: [PATCH 5/5] add data pk --- docs/rfcs/20240827-metric-engine.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/rfcs/20240827-metric-engine.md b/docs/rfcs/20240827-metric-engine.md index a9de9142ae..8f6748473b 100644 --- a/docs/rfcs/20240827-metric-engine.md +++ b/docs/rfcs/20240827-metric-engine.md @@ -223,6 +223,10 @@ VM 这样的好处是为了可以查询没有指定 metric name 的查询，为 | --- | --- | --- | --- | --- | | uint64 | uint64 | int32 | opaque bytes | opaque bytes | -Timestamp 与 value 上层自己编码，会进行数据攒批，比如会把 30 分钟的数据压缩到一行里面。 +Timestamp 与 Value 上层自己编码，会进行数据攒批，比如会把 30 分钟的数据压缩到一行里面。 -底层 SST 文件会对应一个 sequence，不同文件之间 compact 时，如果遇到主键相同时，根据 seq 来去重，seq 大的值，为最新的值。 +为了能够快速定位数据，做如下几点设计： +1. 前三列为主键，在写入时会排好序，这样就可以利用 parquet 自身的 min/max 信息来过滤不需要的数据。 +2. 由于 Timestamp 会进行压缩，因此这一列的 min/max 数据需要我们自己来更新，不能依赖 parquet-rs。 + +此外，每个 SST 文件会对应一个 sequence，不同文件之间 compact 时，如果遇到主键相同，时间戳相同的数据，需根据 seq 来进行数据去重，seq 大的为最新的值。