From af8cca460e420eb8f0651e434fe6792b8a63f45d Mon Sep 17 00:00:00 2001 From: angelozerr Date: Wed, 5 May 2021 05:34:39 +0200 Subject: [PATCH] String Encoding serialization support Fixes #181 Signed-off-by: azerr --- CHANGELOG.md | 1 + docs/Consuming.md | 11 +- docs/Producing.md | 6 +- ...le-consumer-string-encoding-completion.png | Bin 0 -> 12529 bytes ...le-producer-string-encoding-completion.png | Bin 0 -> 22126 bytes src/client/consumer.ts | 32 +++- src/client/producer.ts | 2 +- src/client/serialization.ts | 30 ++-- src/commands/producers.ts | 16 +- src/kafka-file/kafkaFileClient.ts | 2 +- src/kafka-file/languageservice/model.ts | 13 +- .../languageservice/parser/kafkaFileParser.ts | 117 +++++++++++-- .../services/codeLensProvider.ts | 52 ++++-- .../languageservice/services/completion.ts | 117 +++++++++++-- .../languageservice/services/diagnostics.ts | 18 +- .../languageservice/services/hover.ts | 46 ++++-- .../consumerVirtualTextDocumentProvider.ts | 12 +- .../languageservice/codeLens.test.ts | 128 ++++++++++++++- .../completionProperties.test.ts | 154 ++++++++++++++++++ .../kafka-file/languageservice/hover.test.ts | 8 +- 20 files changed, 652 insertions(+), 113 deletions(-) create mode 100644 docs/assets/kafka-file-consumer-string-encoding-completion.png create mode 100644 docs/assets/kafka-file-producer-string-encoding-completion.png diff --git a/CHANGELOG.md b/CHANGELOG.md index d9e3cfcb..51b1c988 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ All notable changes to `Tools for Apache Kafka®` are documented in this file. - Validation for available topics in `.kafka` files. See [#153](https://github.com/jlandersen/vscode-kafka/issues/153). - Simplify snippets. See [#180](https://github.com/jlandersen/vscode-kafka/pull/180). - Hover support in `.kafka` files. See [#149](https://github.com/jlandersen/vscode-kafka/issues/149). +- String encoding serialization support. See [#181](https://github.com/jlandersen/vscode-kafka/issues/181). ## [0.12.0] - 2021-04-26 ### Added diff --git a/docs/Consuming.md b/docs/Consuming.md index 892da260..69871686 100644 --- a/docs/Consuming.md +++ b/docs/Consuming.md @@ -20,11 +20,6 @@ Once this command is launched, it creates a consumer group (with an auto-generat In this case, the starting offset can be only be configured via the [kafka.consumers.offset](#kafkaconsumersoffset) preference. -Known limitations: - -* UTF-8 encoded keys and values only. If data is encoded differently, it will not be pretty. -* One consumer group is created per topic (may change in the future to just have one for the extension). - ### Kafka file Define simple consumers in a `.kafka` file, using the following format: @@ -58,7 +53,7 @@ The `CONSUMER` block defines: The deserializers can have the following value: * `none`: no deserializer (ignores content). - * `string`: similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java) which currently only supports `UTF-8` encoding. + * `string`: similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java). By default it supports `UTF-8` encoding, but you can specify the encoding as parameter like this `string(base64)`. The valid encoding values are defined in [Node.js' buffers and character encodings](https://nodejs.org/api/buffer.html#buffer_buffers_and_character_encodings). * `double`: similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.DoubleDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/DoubleDeserializer.java). * `float`: similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.FloatDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/FloatDeserializer.java). * `integer`: similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.IntegerDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java). @@ -85,6 +80,10 @@ Completion is available for ![Property value completion](assets/kafka-file-consumer-property-value-completion.png) + * string encoding: + +![String encoding completion](assets/kafka-file-consumer-string-encoding-completion.png) + * topic: ![Topic completion](assets/kafka-file-consumer-topic-completion.png) diff --git a/docs/Producing.md b/docs/Producing.md index a865224d..b52d0f0e 100644 --- a/docs/Producing.md +++ b/docs/Producing.md @@ -36,7 +36,7 @@ The `PRODUCER` block defines: The serializers can have the following value: - * `string`: similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java) which currently only supports `UTF-8` encoding. + * `string`: similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java). By default it supports `UTF-8` encoding, but you can specify the encoding as parameter like this `string(base64)`. The valid encoding values are defined in [Node.js' buffers and character encodings](https://nodejs.org/api/buffer.html#buffer_buffers_and_character_encodings). * `double`: similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.DoubleSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/DoubleSerializer.java). * `float`: similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.FloatSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/FloatSerializer.java). * `integer`: similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.IntegerSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/IntegerSerializer.java). @@ -59,6 +59,10 @@ Completion is available for ![Property value completion](assets/kafka-file-producer-property-value-completion.png) + * string encoding: + +![String encoding completion](assets/kafka-file-producer-string-encoding-completion.png) + * randomized content (see following section): ![FakerJS completion](assets/kafka-file-producer-fakerjs-completion.png) diff --git a/docs/assets/kafka-file-consumer-string-encoding-completion.png b/docs/assets/kafka-file-consumer-string-encoding-completion.png new file mode 100644 index 0000000000000000000000000000000000000000..2be08513c1768f9797880b368c85b451015571b3 GIT binary patch literal 12529 zcmc(`cT|&K*Crf6KtoeW=v8_EK@?D=cMzn9Dj4Zq=^ZHn>Am;fArwKRX(-Z*6hoKZ zd+!tQ_sqQS%slhG^Q~{K?~kmk+;?)$K6_u+wf8>vNubgjX?!3#5C8z+%gRWo004L5 z(669-80ae*VLNo_w>x$!(qe$JKFU?}hr1@C3Zei&ML6yS3=94F{s$Q?I{@I3{mtvn zv@wk{0KnxhD>@L88Opr5-#Mu)412%5MP#!sHgQJB94X)wl8hT*zV0TK4E(U&x%qg zXj7MkMMJz0tRxGop02KGF5gW0QtZxpQeG}!7YoO~6ciNn&|Avisyptvh<3@K*ZuIr z{jR852BI%x`sH0w79~v+rEN+dS`bUgvjQgq(FZ`#DB19<0NQ(4z;(Q}RPhIX7;p#d z!#=uuV%|u8U)SXacQZv`siKW`_k5K8Uk$S=4s)`aJjuNzj9rY=vH5uT-SpOk7$MNn z$zb;K)dXi(rOAw*M!D$H(vs(X^2~tZ@mK1ZtD{=slRv)4({9^!%W3C}oL;AU-JJE{ z?1dn1>*MC@OC6iW!(2{}&eGz5Ta)eG9>4sATwqGSI>=8}MpYVh zd+rA7NANK+RGOrFjE)p|-+t{N6H#3DZEZ=>^Own@0$*=W&sT?7JhPynUK}M-{2!*# z_1jI!aTKbkaL)sJa`%@G_dU-iW~c?62h`W#wPDVSY0r|LBzA-`=+zVeiysDOG zRD09y?pO7~@MY`=SgJDVZ#tZhFUgiS#;O|FCzwrtu*ufQ3+EQSNfVHz*yH9|cjE!z z5Kx{>ZF??||5=q$3{Y!UGvPeWYMuDcbIV+08F1rCdc)1S&zWbEQ|+ zCA@@SDZ;{sWjzDhkQ{Y2Z|W`u?Cgzj$eWZGA7|kt-{k&rDvUWw_+pTiJhhITktGm8 zo5L-(z+`TBKODKjj6c_4q~>wRrAAtsC1xA69c3Hm`hxUi2zr-H@=Qtd09R_F#-@2C zD_A$wcH?E)1w;IEg4zDmbU`=XBD%Vl>KRrpu^OZt-MmH~MY66%P||p};7(%~C(meN ze}Hv)gh)I9g!T*8sZ`0AS15(lXw4zMS0_>9jLw7)O*d& z#5pme%TTjfHu#oxa^gVDmhAt(82KNF4fsTW3BLp2#{*#lR=AW%0U0eKu>X9q1!B27 z*LLlBZ0%qto1Wy9co*>L2PqWB=d~AYqW7xUdUsQGS#Ku$k8-e^Qg}CK^F_r^YPY3C z6VnzBof?6|Vb$(jT&W&%1JQcZs1&D=CGrC3GYG1ZhxH* zGx!7oZ4N-j zjX6d|Ui5{O5_#{oxfbqezZSmq(TN;Xg{gVB%NfN*qB&AyJ;i^{Djn{Z z4uL2UJf>nX^6R)+lnSXR{R^|F>iR->V5ws3rc-0c!CMuF3@$*sHw+9t!phdMZfqN2 zkxpGuX{W2(sk4L3)L2oF$ZZ$&nu#|Dfu(RO6|~Ze1}n>H)JaR{Y<)PEk{JuQnW6yI zZ<2{*eL_SY??7PlO7}%^ODMEfEVym=3y}Br9*)59E1aB(er1*G=W%K0rC0Z)&Kg<6 zW|}-~i2x#O)g@d204X*L9$)~5Hbx}C8wmhVVFNJ%_N4z87fYx$EXTcCtGtH5Wu$5d z=lLsoLL2Q3>d;Rk6VWxz&Odo&uOBR*kme!*LBpeKn-d!Ka8ZKHR-EY*3^*gw zzQnfb1>y7g5(=#5Dw<|!l&(uNf7Szly_LaL)X)kTS__H36C{h4@8RF+1cT&3TuxyUn)Y``TI-5qmN--kiKF@c5z?@icTNe5#X~X5 zP}PF306jQ^_gPR-E5?fS-;)Fsn^H@rox=5bMzwYOYC^@mMS+d|I$s8%WzYoScaxE< z=YzpLrd%&uW z7FCbVIUK*+$LbfG8kNzu&Eg0hG1~S|Jt5)aEL%I3gKnhWn^DkPe1!>8Ek-!R_a!@B zol(@JEX|@~tE`P!m!hQ?0{kvhIs>BIV_*K7{F%f$6Xd8|VD{i=Y|F$?&y^})QO^PT zO*J<5ZN-mH#%w}DhsrUVVuzU`6-GZ?Uh@h*uRVYEda~v5YoRK)txSd2j}0o`&)mC~ zTe>u?G+|EvQmzWss^;V{=uB_2k85jaJ}pWsH(Z>~-ZuV7W}2rB^T)mJ{dw_W!|_Sc zch9~H<%eCL^n#IXbh6S|QixgyVf=HlOc73Bv8SQZv_ZDU$ejukJt(7e zQMuBG|B+oR*DTejl+xhXgH^C98LokiGiOoh$+M{Cy;s38(bq(qgml5I$DynFSQGE@gL_JYX6TfhAgJbdNdtFfQ5+fd9$vm>ZmaacV2vFtdumNPBPKON)pK#;XGrFO>P zv6i#SDXz|XY)OSvu)C%u5R~W$PrW|X*?MW97(k`1;t2H`;XO?xl7n0XLxk<}-W-_` zA140t3UG;?(Kuaoce0Q5SRt_)C#EhlE}a~x)l!6gqX_1+KJE0@6>?;a*r9MVExz@u}>E>}kPPlg$KsfheH`MXd&^ zwbQt!sK#UK2*P4LL#{SpH;%s36a7`FCIpr%&V7c$Q|C+e>`|hX2WmAH+9=&3EdQ~3Q$J%;Nn71*% zq%?OJhq|eAz)2<>T<09PB-lvOUtihE&VOZIytaL;%XqH3CpU_!cU=(ztZ4ySsv#R2 z9><5|?%s5(r3jaO;*zVm$=?LMR4K0xzSMD+JvlcBdYaW2jiJCD3Ow&H_=2tfPG*gI zbGcU^@!`kCTC*t?MF!%d0Xdl0X8JVV`rBRKa^+`{(yu_g*7EsixhBH+_vOO>T0@B7 zXzujj7WXwEHk1~@o1@#hj!W>|<9caT^e`egpO9Uq(l9n&;diKwGWXx?$JA~+!-|zD zy3FOg92%)F7C3{PS0`oFxwntRCloIYyBoJ23is52t&*^V>l11Xx(m}S({*0iOpiM- z2gPm{HRKE^##@H#sxA~#HDJg;Q_>{Tt%Xm1S^NfG^6F*V_?Dk@@AH@SG~4%;#SKa* z&YxvfgeAx~*20;oi`3e`HQAn4mrfx3k*)rHT3GPFGC6m8;^_y($*viQNW0Z*eu!>SP#FuJSlM{g)?WOAGiTqv56ovU*HWt}~jH?r$sIuzJT$%*{*?_eK$22)w zf!)t2-qINm`dV)OxB%+qT~NfAI6WherrEea==`+QO|mgHH%_p9iFvk@X=BL2*}Q;b ztSqz8Q|7a#jpqdFxy@=9R!qQ#oKwl-r$!4zmnUGj9%w-PN2yG|Ey6*5jKfhD9~{v*ecI#4cl0p&vO`qjlabl z4#?$?-q9^EbJN=>wHn>*II2P5&ZN(EtM5K<6UN(OtaMYpc2+ac8R#!NIcr^)=U2_) zvO~r9)RbWOsE_KZz>I$c(kIBY=i4@djVZj$rAPN@fmKt{QpH(os%3;oWMn%|=8Lru zrHjZJ#VR>i1G|01b~OUm*Agq)lZ`8PyU}E50?FSOg}a3rd3v7x>+I za?yehtO1Zt*m5)R^XMlvwj2IrJyxxPWz@!A#H1oK89T;n&iZGum@KWCxTtSkR zQX_~PUcuzkl0EESlV@5qhbmLkbpA}dl8FFrH2HA@k$Y~0pYMwa#0TY*O%P`zB;5>59!rcXF(MEyb;GL>TCbGkb}X z3kupq34TjzaGJ67e9#VB^3%zp7_-!0conLg$#qF6vYAiT;HrKd5%WM_V=# z-?(0}1z5ENL_JEK47~@Hk3+(2BqJp8%&U>kDT|j#-wXlsvM$@6x#ZG(uGk(^ZI}ms zYo%A&+n|7AK}e;#%7e!Chj9GM%yonw;;9ff&IdvGL&xT<#RBd#iCWg=QZ-W8W_0e- zP>D0KVCMv=^EkOVHm^UA8-)E`qJ@`LCJ0BNOIr5+)g3_LC*=2-hsmwzQZE|-q##ow zP2ff@0BDadR7K6%G@YL$+mtwlKl=3*DRjP&Xt%64xlWyn?!TroUEtuR)u|Pd z1`c{v&H0cwFo1o4KiK$U@ry&}xK*zX9uV_gl&RC>u@&`}Ew>^010R`GL5{VC^SXLn z){3*-V;Zm3RTrnm3mK@WJn0mOdlIso$yW)iNqtO_MoV%?mo50SSAbKZ=DQVt*!IU< zU2EgdZX3FTAC~Sv)n8y`ps#tfK=F$pn+KzZX_#-2@yo0~vtm71Vwz?sk>prYbwZ)i zywafY6&_Oxs|u+qKd>s!Wg+~*rJffNYetvFHO;FrEp7wqk8ca>;g1PJ zeibdh&z8E;8h8A8VgN#G+FJAAD8M5!e%qx2n*MCPuB+zb$1JP<-DirPqa6EBQrdd< z5})O-1||^qr3i^H)?9BDju|zVPQ7j}r&|#!h4f649eZ0@aDF7D$drINGN_Z%j76n4 zHK2qL0cg2D>E7atE4@jV2L?HP#VL~L)>+x4=i*%Alkd} z>+Ck7rroXz=z2|&QvMDHWaAM%0;i@lTEmK3@FQGnkcF``NeY(O$DX-22V{F{43q;I z=O@ZNI31*3YZ)f-tZ235IVt4ZOp=2KUMBQufJ;&Y zrX6(&{$f@KcE`pjYruTj!15pkwh|++hoeHS07=kWdXq>AW!R|o>;Xb7tWsdahGEJd zP8Zu2w~Sfr;~Iku;q21BcV&Cjq(&NySAkv5_fISsU2{D>Y^Y=keTo$}fyLW-bSRm+| zS@#rd=KSxU|4UtI9=asf*G@59XsheGG7WCN7bv`n@%9}#*!U8xr}F+=N3#MIHwJuG z=gvEc=OtWD6;Q=}4J(IQ)}*3HjSvVt6u^QP@|jl}Ryrxo?YSJbiV4Jw7Of|ewIix& z=k(IdLTvcJ=bmfr`V7L?eNNSvg8YZcZH$r}NT^ha)Xyc6SFx~g_kf0lxXgoJZ^y83 z+srr{trC%j$rasX(8K&4BUvxm@=6;eF%w#RbozBW9Og>2xHu%3qOAI026cP*`y#1q z&N4|Ir8vXrlE`2EZi86!S7B!J>k#tp6Y;4tmRfAV9&iiEivyLN->-TcO^(Z+6l%T% zb~9+^6ba82`Naux5>x!VT?N>fQ>-y}kYbh`A_arneq0XwK&1NK=;!ZP-Vwj03vI>g zY_9cttEmuN5JPT=R6=91sOk)`cuNrhy#x1tIOu~s**lHPvw3V*sG@lxL{*_~Ol#b3 z3WbeA=WNe2k!MV{Jrs(uA@e}&3XHYfZcp%Nv9L8#)A;Gl^ibv~ch<{Z{O}fR)dc?9 zHal5z_fR>Q<0x?y8+!?t3jA;*e4;B;9rtFctvar&T*DT-PJ5t*Gl&Tx_AtrR2n6-t z45=gqVN2Zw#VKbZI+&Vr@{ni3B+#c;3!^#jrOFXqS9ksL+t0=+KaE!xA=YaW_F)DG z(qJQCv@5q^l}uI|vgb|-4pZYF7cKh*72QR0nhIpmnZX&Y*V}hLb=#P`#h*D09pY5r=7d+_7&}O)b{DDDkye<_>K}||8>AJ%eABBhp8aZ zt8oMjbL699$vk?uZ`g<6ijec{!}Dr$Ja8WWDkj_k?1|jE1Mm=2IT^sB32JAvTtz7j z>Oe{V`%f@{6p9%LSCoy?H&D|B3UFkRFg8s;vvt?#Yxh;osgN-({7_mk-EFb+uzKIG?pEGAuU+-09F4sDT{B1TrVA1DC1FLEk$kQV= zeuCmag@{ANCflt^Hfg@P0DP7beRmsm4OS&h;MR*)3}2+{J23R){S1F^q^08!O6tu( zyB16!Rlw74Ac3{1PPuK5BU=`UzE zE2!3t0dO#TVcC^7feurM9Z=tEaJw3%W7AX8s^2>Z*@#S6sVfx6%`@1O%mAKN%{v0~ zXE9!pkS)wrmlR#tkEG2;U8{AYPD)PAHq@(VESr@JGvl#}N|0(cO^D zr5-WMBXv!Ae%JiLw+p3#{9s_Qt9j=KgDmb~b+(fDC$GUwxQkB?l%D*~iuOs20r!Qe zW&x{+#9wYyDn2u=P!f6QubPl@AqEqF*FWcFk?%I#fHbbXSiVSj9$*cNQ~S!_)H?vI z@+s>MFTYAg~$WT;f-M{p%|p)D|n2+mza+Atzwwk5HMJtVamBQ3~BmYJ!dC? zfYGELf@4a}QsT!1t~nYw#_ci}g;kw5`t}U680Pdn3AF}|(s;1SZ`sqrwY&o3iqpnf z{?x$+-z%{7^ikUFof=cYWy1@U6F&93FA}$~Sdx;fX`4*Pv0^ z%st%%HTgQ^XJkkBXlotfj@gyH=NpP_gqD!fw%aPcLj(yn?a@FkW_=tIDBbjy7-*+} zFYq9kEkr$vFy3;xQ41TIz;UpN+Ot&m=fKm83la6CuUl^W&Lj!j9bg~sGL?cYv9_M4 z%Vw_eMNP}KvtpMce3^6JmxQGI~mfPO!43k!WhT7LE%BS=|^T5lUe;DVF)SK!cSitgVw1G^*Z!1hBN6 z6~6#hiNF4p9w7!(65j<@y(hz!usK9L1g=ArNLM5+4dZ*Q7QeO=wC+87KVQN%q3$ki z`}h-5MA1i?G(GScqY4bZjPXW( z1A?&q>6A3nKB^~dG2@_zM{lTO`{rb9WzWuo+iRyuaUibUz-<1lKnF^6VWS-Zf{=>C zWAC7o#d8`Q4)l8jfIsePRoYr2i4eU3gOP!#!MM4|Qc{Y}1`HW_$g+m3aVoEG3_WW+ zzYCINdT%d>A*~w^le|vGeXd=ce_X%F*$dV1~H+(-%FTabGbF)4n3rKMfy^q8-tbxkyjvwsT z7kt|7a^3;2opE%~0VaRCowW(s*KFJ76Yh~B(O%~~wkP~`!y4o6{X^W#`<&@c zX$hCf=bV{|+2uWj)bP#-XhST@)Yg(wwUn9Q#sf0&LA719Z@e6StLV?!om*)Pbp-d% zy#As*%TYUeHpIn9MQtL^(-SF^d77<(3Mkj6V@?=suHJzu z%I4V1sFzhcKMASHdP*KT>@IurSmoVkNC-Ccs#?(KI$eJjyu@~lmnh?!BhPcnbxPV2%hyOO%9GZFsY}B{RP9>)qWiF z@MT{qk;$b|GBu~%2$OjIQ5hu~=l}*0$VR%pYs>BwjmLV$g+x>T_1U=3Gkv|2@^bB2 zHhOXN10vcxDzF%I{LgwRt~B-D1XLZ)a9nD{HoHrl;osmm1{u-*AAt-2TCkxj{QIAf z<7m$O51gx@ut<8O{~h+NPf-__PJMo3&hsHffy3wrqaqk%m1rY+&hdx<t!W^%JL@=V^g##osf5Z_FeMxJwtMI;@*K zhwW3*92UXIL3Zc|2np|d(ex`ipEm5DY&bdg4D<|YEHP*#C?+NsOj3l#8zocS1|kP( zT69t+ZG3F?`69BF1e&}{lHMF|o{j9E$zijT%Haa?N$zE(SC@=BGT!*~GsJRv-2ES{ z^TdPm646lq6JigN5;jZJ-l4DT!@BXMGJNoRS9(o8<+SBO8*o3?%@_pC?rm&-VOjm8 zk1-Ryx(Dj0TwGjn4ssX#a>M|Svh1nEZ~ss~m36tBwLefNl}y{=|6&rMOlsDaUi|*> zz=#Gy2yPdDX(M$e+f5}%kB!#JPoC&=4|{7nbr2tks*m$hDp6{rZN;HV(Om^d7R#r# z_*eWipID_Kma{&I%@aK5;o*zSdmhvL0_Vqs{dyi+o_8PrcQi}lF3lX9c}P-qrSD2vLHmD5xOQE5&c&0#9euxrazpO<4<6M2$nfVadM%F% zsv`~C-#dIQXsy z{Qm2rP+)R%I76;H%shHP0S${L=i$6hNC2r5I^`^%%h+ZCarp`h5eRKebek_Dq6W+U z8qN@IpMxx7c;AGAwg|y+9LeC}oQY_TL7FDYnZ|NPl8L}dn70*( zqs@hjxPYe;?youyiyTMhM3A6#4MQ3DUwbi~`_aZu6-K%E%n=(a35qeYsouIQUeZ{e@K7+`~j2x`^JtBD8XB-vp7yM(67r zwfjKR$#mmLKp_`!i+)V=uXg%BC%&=lzZjfipJlTA3kb9~H<&Efz5xlc=r;2;DFv-X)dDJ(ostf-fz__srO=tC0Gj@i=k;_YmE2v96O{<4%-EU?pWEHZtJ2E!$o()QggOjmRd~v zDY(c`uP~xS&i*snNQ-QFx(gYZ#|Ay^gX}d zh2H2w+rtc(+L}C+! z(U^|pEbjvGdK7)e6d+Ha0y9HT&Z?S#_j{L~sf!C3eV%1BR>FzKWWs|Sl}20Bqj)BOc5a4E7Vxz#jqDpW*ao>)fi60l0P~0H zno+mvPtO-=(ea{5qK@fX?9ItHE(x$yBy+W-AC}fb%LRMWYc8 z@W0LaK@qIz(}Eb&oC};9p!#qIbl!)}?A{#$8#&Xcv$2oUKn`Ll{=b)qf2qHcDhw07 zQ+&}E5p)fVJS&=yxh*0cK)fgqhD^lTE&jrgN9c8QrU8@p{!s?#kqeD1OUd8*v>f4y z_Azjx&Te^q3@h)_Z%~mga3|}XbhwbL1)ufgN_laM#`tb#TwO|rk$zwNTxM4_eJ%e4 ztNw&^pJp8)NdMP|8^|h2!+fWtqZY)(=uk=wkps|$m;$5>UsH<5!e)#AoKca;NJHDj zNb3r-zG%;OjE3yYZjIWH_~*CU%>rj^2xmAITVsv6mn#OFR`s-8Wa~h%Xld$3V?#ZU zy3*$I3>APpEHu2g_>+O>Hik2>ijTJ{PP*SmCw7%5?0MYq&MsY5tzQ90i@hDGti@yU zf2jmrMh-Es7xF^b>%((YG^-5(@Lln!$oxEHPK{}}n(Ail?2TjoCU!Ms5P|!dIw@zj z@p?FtrHPw^CGqc^65zN6p(r`n0&68Wxg%d6-q}ej4c#i_D%)Af4p;f9rueso_zSc` zE@~e*W;&Wo)CW;n@G~OWu<@L>zw>Yf3Cuc}XPnFIeeR_1MKam4aX0*dfTSXORWeI- z(S4JPL#pvDEpI{A2M0?DtGn#~&4j4uXtK2H&$`#d-i#&!3E>_?*B4qQ&TFN&{kjON zrAmv*?$4k4pc|^n2O`^V4s!l4ZO^~ai+{TRPf53lJpYC~H%-icDis3GWx{B0ZNy0oEL?TiO8D;j~5=jmHtl|09`d_a_Rp_ zyK6TfuCFj2su-k@)^>xpf$ddy^)S3!Sj)f_T09tjGT*ogUB$^Cp+BH^e*pKDfZ~6} w{eRkP`G2V6KY_5d@BH7LjEr;HzY{PIKpO#vV)`9`8l?48el1OI8~FaQ7m literal 0 HcmV?d00001 diff --git a/docs/assets/kafka-file-producer-string-encoding-completion.png b/docs/assets/kafka-file-producer-string-encoding-completion.png new file mode 100644 index 0000000000000000000000000000000000000000..3987c5eed1470ce5e74b7fb9b8ee5b78309010b1 GIT binary patch literal 22126 zcmbsRV{~L+^gf8k>^SMz9jnu^opj8KZQDl2R>yY5wrzB5+jgctzwgYfnKv_kytu3C z-m10kxqF}eoM-Rz>>I8iCxL{3hX4i!h9vo0R0#|WJO*?m!a;zp)G!*_f;_n2 zZcyikNpu7TqIE>**pNG!74Dam>Kb~%cmc^(n5T4T#cryhyI|ExcjM1zKgMyFO~3=qFQNYRTS z;(HxUr1E;YyjUn#v$Q;2t~Jug7&MMk-RSc1Ql)%5{qJe?IRreP^9XVV+BIxOM*X;ZrUw#?97|-@S5^~qDiwg9IAl=>F)z{Y> znV`9xE&3cJtEqzvq3Wd41El#sSXqt3YNGIPeJ`K3G%|d9c2*lK5{2`QH`<>nwVI*! z;8b&ezub;9Sy0#?&ycE0pFUhvO^(EqSEK4fN9^t)diC~7#1dU>Z-e;~t6LKCxE4NO zzRoqU+?zYQ?W86{p3}ICsm|mZ^oMM;yO}{l2{qKr=?5p%smZ1OB&DJ{nlDq{7M?3b zH4lxAola#o$~9d80!W}#NOCtvN5dEGwz_n-E?4~%Iqfz(c-$_Xk7o>)s&woHf2!+y zfyJO`3dW*Pej{jVXuw>rBI5JZ?($B%MDCI1uvx`C75IEMq7J0NL_-sb1q%%h>X)8j zkUcp0S8{N0P~lG!77j82qQD0$<)eR>$?3g?b$bLDp~Y6$$F#uL`$g*%94_jG<0-E&-u z%N7%84CX6!6xuINhZA|1STk)dR;vwuU%qyBb}dc4R|RZCVElyd&_Q9A1Y+0QO*U($ z>Lxv?kXf&vZ&%PZ`ToAMm$Q3n={|3dIxY4>B?b@2v*g#@8vnb)?Ps&!KHPkyZEYFX zrr>&1_ZfWl^fp^}ZP#jwP0^AX31ssn0Jc*csWD3;Y_va|W02 z@4lZR2_DPb&R0@vprzNS+h1+8s>t018I0F)e*_fO>bU<+tzqzYhdaua)Rpf;@qhoS zHrsAw^F72+LU*IELhu$;>9p6;e)!ATY<9d%W%1S$w`rnL$7{j!kGH3l@1FjML=)5~C@7jxI$$R7h=>Ja5Tc@@3Xrt4v{I#*ap!Wf zL^T=a@etx^xvB{lW|QdvgH(7?BpE89%FE48M896e;BI@tW7=-Tg^{T#->c52_xJan zejCt4z1r-Q#RJBWuzgP~qP^Viy;}07(jaTLWuT=s)H)7@$?FL&L>Ebh_GU##Y=d^i zjmBtgX_*R6e7aC>whUhAuNgoJW9b44 zy5n!Dnyl$Cx7ftQ)9k{i1E;(wD2wyd*S&a!riOJO0EIL3`TrL+|8F1*f=Un&&j9~l zLAi%F2n!QI?6}6j3`Ac-HF)FVP242vwBoFdoRL(>|qljRFe`tGf7ugTrlB zK%mv->^q%ScxWxKYq~ASb)IZ-rEm?;A)6}>eD1E{^N@uS zq7U>S=mEL!av2K?!%Oh!cJEMjD74D+;S@RovQ9X%`|Fbj4!l$Z6qR}E>gsTQy;T~` zwuEerv{24KkmpL|vbY^~p(?|pqA(Ygm6W!hVeVJ^`ymT4+J2qNvhFM6jq; zW;G6V4{<3OYPz~p>z~#$sBaWk2Yp&c<#^h6&4;I z9@Jm+XBfNkBRaH?jv9OTbBrJf>YA(2h231RN=m56k zDJLbxosmkdacR6^pmhwJ$>Znum7AN}<@080V$$nCB;ZRxK=4g&al1DtkA|P0sL5hB zUkt?l%!`EcPy>|{Haa^x*4NidMX1)n|Jd~G1=;wEgdzus!SulngEQTNFfSU9ZSDw6 zX$#dAHuw~@s#NQC=Gx65NFXC21(W+<2Ly#9lLV@#AHDSAK?!B!Pwat_!3C;g{{KP| z|8HOyONTF;7?+mVUOTGi-&c>eHFQ`bOaYJ6p6lj2kRAl!m2n}nC-9Tfyoxfso}jT& z0)EE8_h@lcP-wx<8U3}9H9Siem7g~G_7&|S&}eOSK=m=>D=(@Rkp0A~zaD6yy&!J15M5CXL;R@Q`WAR zkDgS9!G?B{8Anz4W0r-Oj4hRqjYMel|NpK-EqF_fX! zQwjMJ8abaTpCzAA=+;2+zV@{Iy&AlTsl79PXdSBZOUc(RoZBsT-HOSL&Z2W(TiYY) zw-}Q)KT>2}L!1r*uh18_I{d69r7aFo&aQ0bpU-G?(?f-El|QaSA#n4v_G3{C z=iTMq9Sys6z9W9IApwHh%YW(lSdOaw3w=1;a;mIn6TKMP=+nK}s8#F?Maj7x^n4&X z)ZuUBcsL!ZoS>ilbSW8zTqv#`5CDhp( zJ^okGWT!i&%8Y}tKp1kXgQlMIRcF6D*w&2vZ~PU-%PZ~Oi_JG7eyWQ8{%1wOuhH8Sa5eQE z8jIVXeC)S=-EM?@uMe&--Qt9fCmgoLUqz{<%K|bZBPeQUYWCP|!%w=Nivwn)N>h4i zh(iwdV-+gXYy!P;sL$nB(aF1f6@1=?TBF>6_b?Mtrsrn2a~;}X&zR~XOfVg)asDw4 zWo<5``mneTw&K0OnG{ImJ~RkjOv( zbHUZ-%klZS_0#MN-+?}IAzU#_1t|sq9_uvq`98VIx!IBat0k)(-Y$gfM@NR$Dvc+_ zUFi}QX=i70xT0|4D!>|juD$5i8nb=7wXpJS?l_$k0>*g{om@rlBXc?Ex^q{Lj8FUl ziM+VQ&Wwy%X->Y(cW2b$k1hqVs)x*LO!XTkwWs`dM#u}Re(GDltuE6^ev{#)5O-6e zu&#Y>^PzQ)HX2>yN-R`PTbzzsiC{Xpy$T)Z9B8>K{rMH)tC)^jg|h6s9Q{}RUI?j( z;EjjQTFo#|oM%UlOY0Uy0zXTOC)eqM1u1-`C*XC7-$w0r#w1 z3wq17Z*V*(OpMn*c@KRX(F)BH|B|utS~M#Z>W@tV?BkJBQ8+3FmtYLT&y4UCDcPVX0e-!%t7` z)k?*Y%W{cgsq3SoD6o*9CW$t1qyrCDkFB%x3hXpYqIC8vnEA5+Q1E7caxWg`kG92C zWIrC&=iBK*v^8?YtN;F2JdeGacj^n%W7Fk&qE$pi5l>dM#AqaVwpt?jl*@sI;@HCo zNeNCj8z*`Gl(%a(!Sej?iks|p6>d7Z1+7hcQ@h_^4|iEl`82MBgBX5(LcNgW&3Vmf zZr4ZO_9~mvHp=@#1)$2jBcS1WzcTwhw4ylS8YZlm2S#9fA2>3qX1UI~@5hXjP>oCs zS71(z;Jg zyq?8pxn1eewNNb^gdGxw<82bb!~FcQml0Agu|&#Q;nlOJoKWE6U*YaXIGk~b#WMU6 zU=-Nq_nS1Bh9^~Bf$BZ!JjCFX{QX7Z`#A)hnXl>RMl14WyIAdZ_Rs0hbN1bS0o&vr zQ)3-LA0=u4ZzH}NKA+Rjc-2vJRXAsoDxfTaFZmJ=lkpo@Ks9tEoNFV7A5(~U}xM1OhK z|2u*-Foyok?cFrvwXEKRLHuZ&e-+IZw$G{E!`Ws#a3fYiLh)kR*0yFZb5cr;^^SXk z#<&Xu6gWT_ZB5O;goiJOH>g8U)ct(8hJNJ=oTsg~Qxu4!YfY|Hc^Mx=4E5`z|ELH~ z{3TB$xRQ6gzQN4LjI~*%S#=R2=k4V$@ft9>7PCeFl4z3)Nwswu@Q6#rXz9Zl)3wpw zvLQ_tO1`hnt2lQ!$r18I(V&RX+$pVuVY~c<|8Ypt)wPgS8A|MCp)2l2FF9HBgum?f z(oaEWhN%H#NKiC0Ppv?M&xHo1`UFDo(8 znCHS0IfW&P#oT|OLBglOc^dudBhfv|-Sq&+MX@6W_{~E;Gf-_;!24R<)`W!1VH;-l z*z2>j@%HV(%hhxZn)$YiO@%2_d`yhAM0@QKS^FPd|A2XkGLt)BucTkwV zK5Jp}$+r!h)&47B6t1#wu1_O!ZF$~93y+sP%hjyRmzv;Pw>2cy>Sy8|r-hcsL2Teg zX6*cJhu8I+1;dXHliUP=)&SnHS?UoulAB{v@$DAq3nr$uw0{>&J4_tP*->(?-WI>9 zF6X*Rm#w_z6`WnPv|G%hXf=L0Qjd$O`dH9{K#--SMZ^{AS_*6JtlT?BKQz*55 z(U)yEV0ra$)IO|je5B@2ZmHijo*DK|2%T~uD0&u+s_YX0-jAOCX ztdzUrA^-GK1xy`afW)Hq#qz~jUc;{scwy)DGP$M9oP%J7!Pfv>^qWqk?szGig&IZAw<}0;t zUhGz4FuFPip#RuK9AiwRfBWZq!ahTeO0sZ?->UFn0IrD9UHT_k;v@gDqdxAHpiUOOg$ssR z(Qz2RnG8TFm&9U1IYXXUU?E^U3UAmZS0c zU!aOM;dZ*LJQl7LAirXFATZtE+xhrV+w*a4U-8Q@QSxTVK;5Uy6q%mpgu!RU#QW!y zHQLeqkcM+FkE`XVgT+{mIlQyM-=j|Rw!P%~%p>au=x;=Np|(#-zcIMz%Q*r=WGUGu zh_JGz8?Eacgo6aSIw3Z$FMlTorwDAO6@^AmJt1S1LEckM@MS$vdCcE++mT zmvB$+FFd7(84uo+!>`9WS96WdS|D$jmMusb4coCq=Fu{kVBQrYJ;Ijp;;0^Rx~X|3 z6IlqK6J9Qw7>YOksf@78oM&~F+bHjOu63*IamTqT@C~P(Jq{Ypa=x#DpzM-&IUR~P ztAMGRQ)}nK%;_uw(!7M3L$xXp59=`2*GUNQNN`jlUvN|YO)!>V^|A2Rlc7C6EF(jP z%bw?FE$io6@3~aahY(OOIt3_BEQCxHb6ywKTLNr&-pb6n9( z-YKwXkbcicgK)ad5j*SuXqDcIRUAHvCVBqwulb9fWq5iwY@yh|^FgXzdMWA@o72VA zX5Jc3xUrz-g@4pErOiElC*JaTP4pWf$301RlXh=hMKsV+-pSM=8X0x&G9Y!aARIsD zh}T~)|7WcqZJ;$x)7!m>oakWW<&?Fo?5s$-&!W9w9Ib*#(O7%V=~#(@qeE_b@Itpd zO#fzm4q7iIAHP~bF_CrUn0d&;pc^{C;NfFiZp96OjiUJSd?mWupJ+eKB|?L2`>M2n zVf!Cft+=UZskh+fo3ubx9ZAPouPNvkH&$*E=&(JW8XS)OQ!z1-Pf^oI*UXK{05%9% zmhd1LOEFD00Q>~?W*MgIQWU^KykLMzsqV?TY;0m*MDs(_ad0WJ^hr&~FK$d(q)5sK z_F4c#^f4**84LDBUs*7gxgoQdelJs-qC2J&KQQ6IL3S@xLy*`@DPHAe+9$~$AZRSvUtU;>1liHkB*T|e! z`!$)(C>R4{PHZp}f*442Rr}c?G)aVvPcB{UdTmnL?L{V9!Dn<+eVs6uhxhlAUn}bFnSc;4{&p|f^ zen|Fhjq?2-g;3M@u(JB?vz*wt;v}~eUQcN_+bVy(L(Ws19m%)2owZUmD?OhyEdT7$ z1Rz)b9Ib)qInI@f+x|qwp5~G9^wNbQ)^tTP>bHnsPpJokngbAecP?ig%}8wQpkKtTh9E^{OgMEhc5w|>dqy&fThqr7@pET z0tgntGon=Ou-OarSemLR&sqTDn(;e_t;5c5KE@56+3ph@xM)>Go>k15o$7q++T8Zq zPMMgwT*$gs>aB-3o*3fgR^L2I&@Bd1AHvfM7RtK&y%-blK+6e=c4s|r){kV&U;O@+ zcl#3M>%W17YPn&pxJvjz%UGFu8U`_(IH7Hc z2}HXpb0)zK^31@xM0e}XdjjwW3%qj_C5xY+%mY45CESpe4L)Bv{Z!Tu3rXa#72vOn}i&n$n;sV6V~)Es#KA2u)pF8cqpg_`V<#_sI)+DCq3*Ev4+ z%XIY=XS+M{t^QZwasnPLAi@OuI}sP=h?Jx0_AVhab9qTEdLVE0vZZ$N5XQKwr-BC5 z?}hvA^-f4VNkgnU=)%+f+{}%<`lWgVC=bTkaKPXhrkghd#i^1*eHTj~kh_^6kYt)= z!JvV=wx%Mv<-U^qq>Yka8cjo;Zj<|$W}~d)u4DIyLj`IJ;|IX@ErC$i>F~*sg3W@b zlfzG9@tJxdkOaG}sx~Fv4TZsbBc-y$TJM(+krvgxUc=rV5ptRMnuKz9H10}7Wh5GL z08t{A?Il81V?fpI!$r0Ci>bK##BX9HWzWNL#YjJj@*h%cfvQ&L6o#jk3o+M$?ZReO zc6rdqyCVk?svIcEZeAD1t?<{qxeM!M0W&49+ z?{#uB?DEazRm(#0sCBMmqkKZ%c4b0O?+Rr1${bnCZkc&5JuqTh;uFY-9?q+;%39DC z8rq#hfqES_!Ma&tsep2K)@UU;1K7!QjF@l%nQ@yIeI~>a#_d)bVU_R8L7Gw(mRE_O ztC2W0dUvr`!avqFfv~7j7^Q_Uc`zkw%UABmpEk!NdNppU4EiZvQ-KTN!P?_O@d%oS zidr&Uh>s->QXUlOX9q4rv?bQP=b-aYSW&5=_)H)X!}He8b&%?(sT*PHa8 zZ_M1=A<;q)S5)j#g`?t+2Rw-yHM02!V>Fp-8tCDXTqsFyR`ru3Be_CMF`$X*RR2#6Uf=L!038F^Bnyx6e6au2&rJ4@jKIz_ z)O*sk$WRmu(rMDBB%o`*dlqdMrpZA(LUCe70U{zU z!`;Ge8^#PFFAs9anTfOtMMK&{G)g`-X)2BQ>;*Yg)ynZD*4wWK#s{Z0$-6O|)ccw+ z75ZY?76IGWh3{2f&sYvP{}R}2_fP(GWZF>$i7dZd+f%?e5smss0`?b~+5MZ3)(!kQ zjSDX5%F<4^#yGE{Ix=r+{m4(glwf}@qPwMCs^zz<8P!Cqtf-c_+wRkH9px$-0w5Nz zVx@PyB@mwuF~?0|TQMmvXK2L#g>>9vDF70hzpDycv2Nr~>-$=UHzz@_{%Ua@e!uGZhEv{~d$ ztlX4D!^UGRw80H1Tzy@;jJ;M_PqDZsa-M_Tf5~pn6eDBe{{7$}9vvbm+i&LZ<{r5N zms7Wf=vZW(Qy@+wi-(6Q`&{;}fe0k|J@|jXI4ig_*h2vtQ(|B*dEwaqarOTZ-2cvK~9vh zBpCPvWEjcs;N?y52>}oi;NwZ^;#T-Du>gC`Tod%jh6k?MA7C<*Z3P-MolMHc{cT{DHsz zU1V;{OH@})eO|=%kIOp7X^FI{C}bFD?yAI3noFDaeV!g}Y)!8IA)eP?rt&u}!{kYn zMMwsngqS>&n&mI~r?|9eLlH^UQo?^c+ArzyJrXg3#q0B#ZmEU&XjB;nGcVUlZlNJS zqHl)>@@9d&7N@lSIW?*6Wm2M$n&JuLEV)K-hE~>0fQyg^t09xD`re5_eGJ7crRlpe zb@rrc#mQ;OTQGz>yy^C&K8G$HNJ;QqntHj?1Q>MM2H1%JUa9~16v*n0CX3U?|r69E;hu7<<%Q5?Jpy^ zEFq0XNQ=GAYH^4q$r;m2gtR^qcCG^sHu!(2=Jgn(WO1)^{E5`Xx38bzhqD5tlGpC zGN=DeaKsBHdn!&BNhB~}5XU`zvi@Sw`o>T9tIh;2WW;3}A0x)aBJR^4i_9`~AYkmrpKIm%4=7=r(@=CJbs>(mbyLz*KN zC7dz^hd^HbmdlyFz4*!>t->6cuv-er>h`D2#IO!U?^zH~wKVMUKvx6$Lb>B{Lap{i z)mCYz@6Q*qJK90mH6;+e^LQ;J^>LsXZB--{KwV!Og~pXfft-1zP+u> zKTrXeRT4riMah|&C4qz*P;Hn}`&+7CPC5T{xpVDd7@)sT=20i&L zSkag|#L|sJ`AM(#fBZ}prvR)5_D1}5eI1K?+&ZBzn}cA7oRBSoWC14%@Sj@m{}hS* zmMz2&z}gp#b^^P&xLCmpOr0u}Ok7T9G&C|=(R?8J@#9aWMSNV`_VIBlNY$2A`9fmt z0u2)N$$%AFiWFXI5hxGKRfZ@PRX6)~`BDQGK&;fn#H1mNg!1OH{i-TPeG(ykBKw!u zbT(gw0v`^)cT3miV1`akNxU%_rDWjROm3e;%YHflOd=2{b<57y~*nwK*Ui4-}y=EzQYrpieAZ~Zm6TyoP_lA1cme1$$-$E*` zf)UKy;QY;maNA?xYYkpCg>VP?^!x%dxle>lc3sFJY(WyRxEM_>LXofMck%G(7FOH1 zRjB2yyMZ3}IMN&Q{3(i`v7C{_WGLBae3nK4gam*3QoyD%Wa8TZ1Hv4&9le6lbnxzss0 z{Ldy$&io<5p5~ylgCcPNJ&l}xfoJJyoB z4K;*Au|S#AM+E}>g{;H(TQF6m&J$>)zPU2S9}u&*E~FK{VU*9i^`+?W$`6ZA0L2&p z(bg=Y2Y?L!T*d)OmqZ3*`E?`jWjuxAJ9o41Q(yVItNDR8H}ss3Py&Q+$?%T=E>3z6 zG2cz0S7FiYWtiR8Bl!1$vCJPn2EnM3QX68F{;-DllZAIFwy9!c?*t`3#`6%OVJo$w zqh4zIvAE;kFI7T`i0ezgJADYd^_o>-_)0XZ*G;CFNr@#E11aH@*@Un<+RP2B!6DbU z9W9O;bp&S8oBXb`ai)hNO>~16eez?RF$P-+hto@P9PxSDsjLEZ%)YkT%0AZzqbt}F z@aY?rmSP1+%29i@yH6Z&D}*@&GLJAAU9zVl9Q#$3*ghi_uI9ElsyKVc7L7#aY$hiP zj6Z2e^ZGj`^ATfnx>KEOH@bl!s011Arya{e;K2+Mqb!A_bm~cip+XG{j#OY-GOLdl z9u*fojwtQjMf=u{TAmP#C*xF=$D2iGxpWoKXzH?Z)GzKwV{ePEUnXZ!D%!ONowE$1 zW!$IkWqxK^hj1EeBC&atnn)*-FiV8yMEiIq)N+-K-m*?uP4h3zL29su| zsw_2^yxx#_o}=@y4bdtOpXUA?DZcmR$`#fC{uj19^&}5xF)BUD5Mwk4+@}>yKq*P5 zKE+P7kR({-4*(XiD!CU9lXqWbe}i!{jwDNzg#$VEpE*qS2^i%8)~ldMNQmo2lf& zY~~Vn$<}ziKCbrJE%Fru#4@tk`+FQZ=#qv}J`XXPRQPx3>iC&}Ctzw0^tgFk5UsMJ zr;vS%sBEC08I;%y$8tIzb}XfJvBE|zlrUL%ym(RCYmdK-N__v0^Qb;2|Je0 zml)}4n&AV>Y0mpIWeBQ!zCUsEjyG<#D%30-jiLR<^sI{=%1&*U%0o5>=?p2~>s5yO z3Jd|4uSl{-($lBtBl7Y5y}ns|IW!|+p^UmSmlmc_a{xX^wnUh&jO-a7qDw5CwGSH6 zL7);$R)%n_gl(U|)7x6P@z%u&CX^GD4QCKPZAWEm5;S#q z`qZW$P^vPlcR?`E%ATsv$feA)Odg*q<_?*>zs%kAZV$_R#j>E>cYBYi17*a^&r&VS z)ABHlBTJwT!KuY-^}3cyl3+5I#frZ0h=x9)DDk|5EU0O1e!sIYg(_ z?JpJ=mvB>cO{9#76^sBXRT&TEpoQ)fM{Q5mZcDM0xb7(*p31XTC6Uyy;xM~ExL05a zrGIHP<;4Z!*dowQpy_%$TAs<4wpaQXx_ex%^SSFW(xGa>gVi_NLmLQtGAp!~cPV^O zo>UHzx91lOEyxvZqL)!qlzK8W>3Q;sNL$OWbYWh<&d~TLn;thB)`Im#+0=q(C49&n zz$8BIzD&z6inIB=30ms{wHlIY4(DyAy6%i05}7y5K9!yQYEEUILYWRaqfUA2Ho=(-#}(_fSC5iNti7-$2Wi98msh{4S+-e@n5a@So}kx3*U#;D{A*M1Y#l}!-#Ak{B9 zu!J`c-We#w42^*Ko5q>0z=InR)fv=}49CT5(ArH=cRPNZ>u?aVDp5n=k$x3Od{Xad zzDhTq-k&}pZ+sV4>_lV-4T^{VEe&6K&3`oO4kbJ^`|3IB2_d6CYpD`{oKR&0TxGC( zsA6uiF0p?wZ;@x%|th$FBOtE6!*v z+g^jWsuTqzc#E$yP%SmBh~PIaFzV=Q>-LFP*k8_5b0Grf*SGN7N*G`|F7UgBHUhvI zLU0g1)7~^r?c<$uM|*M+Pm$y+Q;bdWp3rn-x4mTabEqz}jDhaJ;e~Axxu} z$N&Fqz7U*r1zSVTTF=NKA$gl0`tX~f4G8JA>sZ{3-;=8Pymfl#Q?)?K940>A{+rrx znkZ2r+{CK0>R_p0aUXZ@RM7`!BTV^Z0rOQaZ@cc7r3&8y8;iHO8NL@${a&}%z$So@ z$UFyrzPWF(b%>%aG;r{a{ZwKpe;Rt-vAx^(3uB&h8v7_q3g>=c*wcZ z?IDH1%db7NY>&r8^7J*@PUTB(Dy_h;4q;`Nb*4bNo@%L?I=W@hnTZXa`r&x3Bqv|x zHXzAFH=rrXlkn~SME&Uz%gT?0a{KbqxBMPjb*BG%vFHmoeNF79RC-V1U2pbQGh>ds zv1P07v|hGSY`?A*vG`B3)e1d3e;T0d`o_A$1mju_3#5CeARzZ4sN z=+V_IdNDt*xdUt=)ycZ_u04*}k3Z7M(PAW`t|)js zv7-+A)#k>sc}lBJd!oGj->NX!+92>(Y$X<&2W`%RR%8g{w$gELC#kz{ShT$(8;JxX zl1Vk-kNBg!yzIzGno}a<$yIT*xpLoX#ndwNmNV|t%^RJcu?bpPnp^0k4szw!D&8c} z`j4z3Kpzn7M$g;iy2YkyfcU%}7Mab72_KVW4qbDKyn?&@ahXiPb2go}lu)#Y0h_7rw1I7E90hWA%&hI%+3L%zSHU~TeZ7*Yl0^sh*J;4fW|fSCu1%IGQ@0&Q)_lcx znB9}|fl9oRvicm)O^=AY*a}a5nF>o+j)M%sh^^&~Lf?zq*T$ULF@f<5Tdc*+M{M8Q zh8>>tIVsU^o8YWiImn1dt$KP6o}+R9r1KrMgCK_MB{wsNr}tfjDKzqxv)ZQ5I7RO< z>8x9S?)SpDl)GUsLaYj5#$9-cky{XIzlds>pM{84y>(;*)%IJ4bgSD<1UWc~tGQV6 z)Fk04w9COJc{7xV4pQu^Y+W=YgXXk#O9T}>6YN`n0`_A(_>qjn$Gp`Eg;=_ZXvLP$ z0p!BsZhtnvCt4!R(lADMLDm@ghFv4iQ^#yob@PuNOq;nnwb5XAs|cpZdv2Z0A2Y8B zqg=%z@-#n4w-M-d|7dZ-Ula_}W>+)+Sc07+ec1IekX9{k#kTIZgy+wWe)3_-VbA)! zxvLdO3AMR6Wf>QxN|HM#%nSF(b!)iXfuVY%hecj+OM&fK{ksak~#HCjK zlWD#HbCj^N^shGvPUByZZe_0IPc7C`o*$y%;?erx8!tbCu?tX>?KvHn+~x?gTFPla zyEbR|?*QK%2Z)7EkOiAEo4mDc6jleu_;gI zfLO)_D%YgD9ug{(d?3EkX&aB@Sh>`K*8OqM8TaM4!!4F==nXnd>6~w+Cc`i6?sKiP z_CdVg7Nj_uaL(j7)&&m&+C z|0~#;U}B#^(k3Jv7=x&^3x3=naLR0=)Rn|i32*b&v)jCT9#z&PCu#AmAuq4{FNuFo zjHeeJbmAQ^Q+|#zxKJ-h-iAB^m5|SzDW1c4QnG+NyK>cL;vsJ5@Q~!#6ZoKq%^niD{DQ<1tji@G`T|zRPgod8d0RGEWc97TTW4l3F#Eoln z>;B_py`@7rUF9#waPD-q%-XYCO*s?1Eey5;x<5$Kz~-osAqwK0qX@4v@BuHB7@ed2i}M ziPkk&iUZ?cQXrP=6F7?p>waq9uo)}S&w`#iikP1sk@L0{sa`h%cI)n&4{Xmkn=|MY z0*`j<&yjOEU+*ERUcI+9Urq|+byy|%)T)gM^jAdJ$HnfPEha13if(^Kxan>1cy)8E z=eP6PpOOLzp~e8}N{dtNY0LE(aHq$JTG?It;|?&Ud3e9E)HS4jo@Z^~*X9SeB5yrK z<1716$`dUpdorx^bV9Di*hB7U&V)g!z(tE}MHw34S&WQMm}~MSgAY_yN$h3PZA$I@TcZ2ekbYAONh%fsL zUMem@9ADfy-3rjCB?@X{!}$BD)%mzNW2F_oCq>-F_CCnnr~!Gi7n9WvVJW6HSPLAo zW!z>s++;E~Sy3Vc)nauJsqBZU4r=QFE!pXVTx@m)YqP&c=?~MNQ*Ya4-&+zI$SJ9s z$l}Qrox}fBsSp5kQ$j|OxTCo3iKa7t;(_}z9)RZr4lOkz!7eJ#CaZi`E;z&|me?DN z+=^2SFe?cZs7P$*_8=y1Gb~r?aX4I%I#*FzUvuYnTsdiz^=IFdZAq9oTO?Xk0$l!G z=t|I2Q@5pEXoj=HUg%0yQ&V@mMH+Dsr9vOV^O=AUeD=I6;9 zD0`FH`I~e1Y!981o=2t57>GcJ__9P;S0AJmPW=t}Y7{U2Fd+t)STB)<>end~_YNh! zc$>RnAW!qny}3KfSrux)94u#$Q`ah-CG-$xWouR$aV=Yd5QJ;0YD*yx5Qq1n9Q-D$C988#ESsMiN7PS&3 zi%2?|J*oMFe8cEMt>P%B6zdp&HYoPcjfEzCv|G_d1kobz2#~-5obR|;zDM0^nW{O- z#wCa14Kx~?0)C&>m+jT~Z|YFTvk_c-z5IbLaoKt?6mmrT_Fn6D_qs$QGnpJB&^PE? zR|Wom8}AN@VPUI^%%XL!kt)JsX%v(m?Kgo8H`vwEF_Hl(Fy%3%iq`qH{aWt}SBqIKU@m(t>odHCNRTY5r5{)C>><;N%E=N163bNJ~skA;3;cte^VkmjjjJ--)p$>H5 zkOQh4jt)`E9%NF;I-iAK{<5sUCVp2eucpp+?94kdnk*dI* zFj|Xun+~QFkR9+nAqIvqivuZP);dqko~9Pfj6D&N7P|*wjf-teEv#h>55}qq3A6q( zXt};U+Ne?lo~nq855W?9j{x^WYz#pg9eHx3Z0RA$t~_S*+;?tG1Yfy}+sNG7!cMcT z^~CRdo+`ulLb(p@Zddu^iEyBH8R_6%5XxFH`M3F(U#$C34xH0E zPFCO%CCYkE#wR8Wp&{H=td^>vwX84KpTwhZ;Qgefqy%1%`%lYMo8oBQ)TpY|N5kuT zLp8YckOG$@{xsa$gimMl)4|HyUyfU+@#U&&1(TK|SQE!#k!nLCZ%hl=<*-_i>3|27 z0?Jj1h@!h*4!7p{(d{Zw1iGh?@LGn3zr=mSpApYoPUmm#jajq+bk-z*OWP@eqj}~y zo{^G8haqN&R{dQ0qG zK=`I`fJ3JHuZQc$l-|SK}{0=%JagvlA%{5eMJ)F)#IdgDu0CkrcU!4Xc z71DP19RGlUBhy+ZXyNa2wTex7dfrVl)@W(&aAIjs(Oj$alP1&%9c(9k+8P*J|14)6 zPhpA;3%kC(g;1vZ`4enWXhU5KPpJRGT>_nGL=zL-_)_+LKYDImmJY4O}oBJ*dC~(zGbR? z$hAUM32L?7Pf6MuIrww%=+mgOL9_qaqQywRN?u7nRw#+ly9c?2YE4HSs0Mddso5Cl z*(TK3$6gDxdRrtKw_c3h{K%M{eo>lV^;!I26GF9$Po45{YSC72%IF_FTVIKd;lUo6 z?mjv`-ab4W8WR_UJY89#{}2#d8Vq6N0&-#~GU3>mxc$v;FqM2Z6zo%eKi)P|g$UC8KO`X5bIUJKOy2~t zUB;~Y4K1vC<-M1KgAaf||#N2M5Co zwhiHfYz9zP1eHtV2tn`5@tAZ--4|= zcu=f+2?&gv(!|9_2!URCP*3Wi-Rett2@v2pdaa1@*ZavPY$9 zfR_=lBbJlH1>&ImfAAgz&ME|mcj!9hw8K}rx9kaAC$_h!xelhTlh86LCdF;|%65P&&yY%d@#=>RKbkH%tj!o=aT# z8F_ZZAA40rrRv?0f`<;*lRAq;zpg^1y|u3g>X=_)<8&u^1inoWs0%m3AnmIbCzt9Q z35ntJwvev8o9_ao>eDaE=k*X!?S#oKn6QLQM-Hb|cn{ZN$+{;fL`tI`w~ocO$BN}J zMCYc}Vn>)t*E(w=!4-dNB~}DZ<}I|JGbKCQ#>%RbaAnY6k*MKzGp6IwN50ROt+e*l zSIJen%YD?}RSwZ~56>`rn^${@uIKOemUk=lB4=1qOr%QkrDAj2y? zb&cs`l@|8Z@@XY%&(ht*nlV1)7m))iJsd}gD1U&R*xlWowYdgM$cE9fa0m$<_A8V- z9^jKM_jPx(|MKfp45V$83X$w3nQKYEa}%mtMdF_SX_+af+SVjt;7YKtD4ny|cU>L3 zdhIe!K38i)%uJV2oxW2S6GAgqC>FSa+PNL394(VZIB(K?YPQ#Hdof-_FWr%(E02ra z(nz?X3^cvIm|>flnORQo!Pt2WE+IUe%HJWMt{MhzJx_-O-s*%q7!%1}9pC@tI`!lR z%SlbHXR`_t{iD8XqZ?x0AK2ApYczz=ZM?=+F1$IML;#P_zQU22iVHtBxfQV?*3L`N6lx*NtbQ5%^5x6Nk7)#f zt~@**I&KOlV9;1K#iCi8q*jCIx2vnONQ{WHL0Ro%y-~lf;RTFs@I2?$TYOw_Pnd4eQE z_-UDH`g4C94Db*(kGQn5VLqePvWic)cI)G@yOgoAniLj^>hi|Vi$p`K-#3d znAaqgWHL2}&bSY8Kj9y8m5Fb{2VQN}=gPXFG(?m>D^?g~AtIt)9y9`va5u}}gzzKO zj)d%FQdU-!XOy%7R^YCoH6dtc(AC@pFPhF^7OKzjgnC3S#|atHbyxP=`7`t#9Kn$O z)N8$xIK_32f`Wc)8~E?TLK}!yB$(oCOclRj4hMJA;$@3dTbf03*aJ_5ispyUsj&-7 z83>^b&W=g4yLd!$|9E)pP7M*8ukJIO&@AMG} z?ePChXn`b_1MPE~ZxYX)r+TU^p9V&oTz_ufTo%j>@bdO=$S$Q*7k?~e?oz0N{h%tJ z0_MA%+m!Es#K?rq`FZd;CjovYBq2pSr-qbY`Jvve#P*2$RPsZh@+e4-0ziM)s=~W3j^=>f*}osC^!#~j@y!6pc$d`lJk-vxE(RfAxK6dxYP{vuk*1)k>?En*0HVQcl$6ui>{4qhbYSD{ zX^lnc7`SLHZNJF4LDt4aC8c0MGl<}82s=8gt*sr%m&ig*Ta9ZF$i_miEmuawLOY4~ zOwWiJ6zwfRz=%Qn{58=n3f0bR#^!8R)`P4fr7`pw?n zr7)+Zqtd0bvJVb-$y89b(5B`Ir?An&N%|LiDfYD&daLa#y};vY9q zD8$L>jQH`&yC&Q+1%bM1iF@d_$rv|NtvHwn{y@e6)Y?f}!gmMUDUrI--W3~5FA9(` z>d&q2E)G!T>+9-PGz!BZ0=k&PtAP~+q`a(91wO3zWTN#lOEGmdeJW8m)lj8!bVF5o* zfALGPL8kg`T)`(83>Lb#bmrydTfDJ@Bq~|Pq(4PxbMJ;|&CtI;M8pQc}y1LjnzPii!&R~HZC7n_`obTqea*?9^w0V|7a|RyY zw*IvZ(0X8>j5IWExVx7kP+?I_NQg z3XH_{3=ZLSZkY)vl5y&hAR75hHMJ6uC%~dw!@vmuwVEe5-jK!j9UlB@O{N+uG{Pxr zyvP>TNhFa-!66|Qs?y1HZ1pcJN)mB${toVO@$s8v@^gmqB0Y0k&UI8cO0qhFxMn65 zFstTA76{yV1TZ881qCC5&|Jp8mo$vrgI8fnP_y`?qkfOb^OS|CwLa6FaV5!p-%c{@ z8`laKF&!)-?A3`P4*;clmq@&rn39>E4q!RaRTjDNFP=jho;x8ujaw7JE@gCZbT^=p zEk)D~)cw>;*^*Aw^mKAzj?KZFBA*i>pZf;W=_HkZvG87#H9cov3gPRbSGzr>NF>7G zbu1L4vFS^_WXl|Y;!d8YlFeJ@ief?{O+L6*RK(e<-v2rN`LjT1n^rt` zoVS!mLyge|%88YA275Ta)K*7Z`TZfYqvgW~=a_F<7!^j)7y)yDD`V4sE>gi3sO=0x z&zn(V*4J}0(NQy&j*czBvs#&p$Du(y_ep$pr*$-W@NA+=WYK0$x&$L7EMeh zu|jxERHja}>vsL5OC4&y>$U-noVC)6L!+k{=1HUz$`Ohn>1=7kvwII`o=U!t16db0 z^YubX=wdT~i;PKMI`)gRG7{j->wltE*DfJhAL|tGUy{ZC&Zxn@o}+Ne>YNfRF&9~J zfAXQbp;(L?sKo6WJkpbsn?iT*SYZy(bZqJs`U_;p;h%FR!rBRW$~(UU-LrC$m1T|! zdPt*m>|8iou*Cw*{zMKDIB-1_y+;=T2}U?0#vwLf10W3}sK8ZZw%^Dd#|lbHFI7X> zW~kD%SRD^ooj$zB;O45Q8r*Zh=5V=A%^|PRW=*!xx8dS(25!w!{(S3QySc`Qo-|K7 zmPqx)d`Q3B-_psctuukI4zPzn_!#t#Dy#U`+wuO0_H!FM7en+-7Nh|dbUdsXHi1fE z->6`5I;M3&8>zLm(#hvvQXwO%b*`(So}8~_*G#)IzxW59&d=XJOF58b4hC~K(n$z?tbwkY(zYizAvx^F^<1Zlb)+Q=6DY?1 zD@t99-*E{icop+l1v?vKYn?9g4lk1$9P>Q1?PN^gu-!Dc@7)^co}QX%{?=TG;ZC z3EtbZ3C>8h+j6C`8ch+7X=`g+9w~#$X&CCqnz!zyoi_o2@N7{O1oRx%e*8Ge*Br3- zbIJb=kKVh$4gZUvVEM)+1;9ZA18;=qaV3}Z7xUQZI^0h=x&Yg$?_8nq`mD~Kt_-!C zttBX5gQS$dKB zvVkPyKVy3gb7WKG!@pg79h72xSCF0kD<~m)v~op>8$ipsFWH}YI0WEbK=uMwr?S1< z{T@Ed7m)QeGBI&;VWdC+Gv2)phO}l;Hy3hq#?tVbLTQ39_rnJLi+C>CV?o8SDm#=B)H{%WfKwZku&qB1q7(yO zKYXwd&4?U@=U+T|O~aX55od%mt#C}<-w*LJ($dqbVe6v1K?jbj=^_(VYFnQP#rtX#3m#{Z|i$`8>kj;PrppZVB!F|475V% z706aq`;$R*Y`E4ffHjKTaT~&-DASn8cpEar{&mn$8-7UPk9|N}yS~nI?Jb{-tEr}l zVLHuW!%GQ@VX`^vF%rP6s8tf^o(7eBBvF(s?KcUGc= zMlaq?rkT5t2VSPXf;Hoh%MJV^-E^ate~c*nADYiUs*f2*8jzl8vse+Y0AVTM0cfpp z@dyN_ZF>58S;dS~_SMO|&wjTmceZ$CY|AJ+8r-^?cku$l3BLW=_yChUMKf#B%@{v% zMihujn4IRx=}U8+UXB*<_{`l!fy`e}%3_=6|1vJMVM<-t*_O zY-nF!6O7D>6BSG|1xCr(dZXOl&r+ro5Ye5H{d%UQ>yo0asy2@>WP=Zo36&jEcL^SF=|Jl zy9$4VTG-l3-_(t*_E`pwuCYA#K575pV0vnbMM#C-?jevpW`0>B$+?q64Q>V0aFP0< zq6OVEvmK0}tyagoCWABxZ{hcj7W5|bQ2{V$9QTstGsoywY6%TvbTFm7<8FN( z4Cxs3FPt?Do0tehcd4l64$XoFCv1Dn6J}lWE+IeTShXXxHX(6-fj;g&K5yNJG)`O5 zq_$65dVP6aoB)xi(7#Pdd0lF*2%$l*WCu-gdyTuhLkmw7v>EN~H%a|Rdt+}ei?qZU z>O8>z{U}GENv_{qr=0r?KO7G5Yf_+eHNDAPOZ~qludjG#{@cN?9x3O}LB*F*0Md4E zP3Y; { - message.key = deserialize(message.key, this.options.messageKeyFormat); - message.value = deserialize(message.value, this.options.messageValueFormat); + message.key = deserialize(message.key, this.options.messageKeyFormat, this.options.messageKeyFormatSettings); + message.value = deserialize(message.value, this.options.messageValueFormat, this.options.messageValueFormatSettings); this.onDidReceiveMessageEmitter.fire({ uri: this.uri, record: { topic: topic, partition: partition, ...message }, @@ -360,14 +364,18 @@ export interface ConsumerInfoUri { fromOffset?: string; partitions?: string; messageKeyFormat?: MessageFormat; + messageKeyFormatSettings?: SerializationSetting[]; messageValueFormat?: MessageFormat; + messageValueFormatSettings?: SerializationSetting[]; } const TOPIC_QUERY_PARAMETER = 'topic'; const FROM_QUERY_PARAMETER = 'from'; const PARTITIONS_QUERY_PARAMETER = 'partitions'; const KEY_FORMAT_QUERY_PARAMETER = 'key'; +const KEY_FORMAT_SETTINGS_QUERY_PARAMETER = 'key-settings'; const VALUE_FORMAT_QUERY_PARAMETER = 'value'; +const VALUE_FORMAT_SETTINGS_QUERY_PARAMETER = 'value-settings'; export function createConsumerUri(info: ConsumerInfoUri): vscode.Uri { const path = `kafka:${info.clusterId}/${info.consumerGroupId}`; @@ -376,7 +384,9 @@ export function createConsumerUri(info: ConsumerInfoUri): vscode.Uri { query = addQueryParameter(query, FROM_QUERY_PARAMETER, info.fromOffset); query = addQueryParameter(query, PARTITIONS_QUERY_PARAMETER, info.partitions); query = addQueryParameter(query, KEY_FORMAT_QUERY_PARAMETER, info.messageKeyFormat); + query = addQueryParameter(query, KEY_FORMAT_SETTINGS_QUERY_PARAMETER, info.messageKeyFormatSettings?.map(p => p.value).join(',')); query = addQueryParameter(query, VALUE_FORMAT_QUERY_PARAMETER, info.messageValueFormat); + query = addQueryParameter(query, VALUE_FORMAT_SETTINGS_QUERY_PARAMETER, info.messageValueFormatSettings?.map(p => p.value).join(',')); return vscode.Uri.parse(path + query); } @@ -387,7 +397,9 @@ export function extractConsumerInfoUri(uri: vscode.Uri): ConsumerInfoUri { const from = urlParams.get(FROM_QUERY_PARAMETER); const partitions = urlParams.get(PARTITIONS_QUERY_PARAMETER); const messageKeyFormat = urlParams.get(KEY_FORMAT_QUERY_PARAMETER); + const messageKeyFormatSettings = urlParams.get(KEY_FORMAT_SETTINGS_QUERY_PARAMETER); const messageValueFormat = urlParams.get(VALUE_FORMAT_QUERY_PARAMETER); + const messageValueFormatSettings = urlParams.get(VALUE_FORMAT_SETTINGS_QUERY_PARAMETER); const result: ConsumerInfoUri = { clusterId, consumerGroupId, @@ -402,9 +414,19 @@ export function extractConsumerInfoUri(uri: vscode.Uri): ConsumerInfoUri { if (messageKeyFormat && messageKeyFormat.trim().length > 0) { result.messageKeyFormat = messageKeyFormat as MessageFormat; } + if (messageKeyFormatSettings) { + const settings = messageKeyFormatSettings.split(','). + map(value => { value }); + result.messageKeyFormatSettings = settings; + } if (messageValueFormat && messageValueFormat.trim().length > 0) { result.messageValueFormat = messageValueFormat as MessageFormat; } + if (messageValueFormatSettings) { + const settings = messageValueFormatSettings.split(','). + map(value => { value }); + result.messageValueFormatSettings = settings; + } return result; } diff --git a/src/client/producer.ts b/src/client/producer.ts index c144cc76..099d51ce 100644 --- a/src/client/producer.ts +++ b/src/client/producer.ts @@ -184,7 +184,7 @@ export interface ProducerInfoUri { clusterId: string; topicId?: string; key?: string; - value: string; + value?: string; } const TOPIC_QUERY_PARAMETER = 'topic'; diff --git a/src/client/serialization.ts b/src/client/serialization.ts index d04b071b..45a5a544 100644 --- a/src/client/serialization.ts +++ b/src/client/serialization.ts @@ -1,18 +1,23 @@ -export type MessageFormat = "none" | "string" | "double" | "float" | "integer" | "long" | "short" ; +export type MessageFormat = "none" | "string" | "double" | "float" | "integer" | "long" | "short"; export type SerializationdResult = any | Error; export class SerializationException extends Error { } +export interface SerializationSetting { + name?: string; + value?: string; +} + // ---------------- Serializers ---------------- interface Serializer { - serialize(data: string): Buffer | string | null; + serialize(data: string, settings?: SerializationSetting[]): Buffer | string | null; } const serializerRegistry: Map = new Map(); -export function serialize(data?: string, format?: MessageFormat): Buffer | string | null { +export function serialize(data?: string, format?: MessageFormat, settings?: SerializationSetting[]): Buffer | string | null { if (!data || !format) { return data || null; } @@ -20,7 +25,7 @@ export function serialize(data?: string, format?: MessageFormat): Buffer | strin if (!serializer) { throw new SerializationException(`Cannot find a serializer for ${format} format.`); } - return serializer.serialize(data); + return serializer.serialize(data, settings); } function getSerializer(format: MessageFormat): Serializer | undefined { @@ -79,7 +84,11 @@ class ShortSerializer implements Serializer { class StringSerializer implements Serializer { - serialize(value: string): Buffer | string | null { + serialize(value: string, settings?: SerializationSetting[]): Buffer | string | null { + const encoding = settings?.[0].value; + if (encoding) { + return Buffer.from(value, encoding); + } return value; }; } @@ -94,12 +103,12 @@ serializerRegistry.set("string", new StringSerializer()); // ---------------- Deserializers ---------------- interface Deserializer { - deserialize(data: Buffer): any; + deserialize(data: Buffer, settings?: SerializationSetting[]): any; } const deserializerRegistry: Map = new Map(); -export function deserialize(data: Buffer | null, format?: MessageFormat): SerializationdResult | null { +export function deserialize(data: Buffer | null, format?: MessageFormat, settings?: SerializationSetting[]): SerializationdResult | null { if (data === null || !format) { return data; } @@ -111,7 +120,7 @@ export function deserialize(data: Buffer | null, format?: MessageFormat): Serial if (!deserializer) { throw new SerializationException(`Cannot find a deserializer for ${format} format.`); } - return deserializer.deserialize(data); + return deserializer.deserialize(data, settings); } catch (e) { return e; @@ -189,11 +198,12 @@ class ShortDeserializer implements Deserializer { class StringDeserializer implements Deserializer { - deserialize(data: Buffer | null): any { + deserialize(data: Buffer | null, settings?: SerializationSetting[]): any { if (data === null) { return null; } - return data.toString(); + const encoding = settings?.[0].value; + return data.toString(encoding); } } diff --git a/src/commands/producers.ts b/src/commands/producers.ts index 0afe0a90..c56834e3 100644 --- a/src/commands/producers.ts +++ b/src/commands/producers.ts @@ -7,7 +7,7 @@ import { OutputChannelProvider } from "../providers/outputChannelProvider"; import { KafkaExplorer } from "../explorer"; import { WorkspaceSettings } from "../settings"; import { pickClient } from "./common"; -import { MessageFormat, serialize } from "../client/serialization"; +import { MessageFormat, SerializationSetting, serialize } from "../client/serialization"; import { createProducerUri, ProducerCollection, ProducerInfoUri, ProducerLaunchState } from "../client/producer"; import { ProducerRecord } from "kafkajs"; import { ProducerValidator } from "../validators/producer"; @@ -15,7 +15,9 @@ import { getErrorMessage } from "../errors"; export interface ProduceRecordCommand extends ProducerInfoUri { messageKeyFormat?: MessageFormat; + messageKeyFormatSettings?: SerializationSetting[]; messageValueFormat?: MessageFormat; + messageValueFormatSettings?: SerializationSetting[]; } export class ProduceRecordCommandHandler { @@ -46,6 +48,10 @@ export class ProduceRecordCommandHandler { channel.appendLine("No topic"); return; } + if (value === undefined) { + channel.appendLine("No value"); + return; + } if (this.settings.producerFakerJSEnabled) { faker.setLocale(this.settings.producerFakerJSLocale); } @@ -61,15 +67,15 @@ export class ProduceRecordCommandHandler { faker.seed(seed); const randomizedValue = faker.fake(value); return { - key: serialize(randomizedKey, command.messageKeyFormat), - value: serialize(randomizedValue, command.messageValueFormat) + key: serialize(randomizedKey, command.messageKeyFormat, command.messageKeyFormatSettings), + value: serialize(randomizedValue, command.messageValueFormat, command.messageValueFormatSettings) }; } // Return key/value message as-is return { - key: serialize(key, command.messageKeyFormat), - value: serialize(value, command.messageValueFormat) + key: serialize(key, command.messageKeyFormat, command.messageKeyFormatSettings), + value: serialize(value, command.messageValueFormat, command.messageValueFormatSettings) }; }); diff --git a/src/kafka-file/kafkaFileClient.ts b/src/kafka-file/kafkaFileClient.ts index 39852fc0..5b132f80 100644 --- a/src/kafka-file/kafkaFileClient.ts +++ b/src/kafka-file/kafkaFileClient.ts @@ -157,7 +157,7 @@ export function startLanguageClient( // Completion const completion = new KafkaFileCompletionItemProvider(kafkaFileDocuments, languageService, workspaceSettings); context.subscriptions.push( - vscode.languages.registerCompletionItemProvider(documentSelector, completion, ':', '{', '.') + vscode.languages.registerCompletionItemProvider(documentSelector, completion, ':', '{', '.', '(') ); // Validation diff --git a/src/kafka-file/languageservice/model.ts b/src/kafka-file/languageservice/model.ts index b602cadf..25f08278 100644 --- a/src/kafka-file/languageservice/model.ts +++ b/src/kafka-file/languageservice/model.ts @@ -21,7 +21,10 @@ export class Model { return this.getDefinitionEnum(name, value) !== undefined; } - public getDefinitionEnum(name: string, value: string): ModelDefinition | undefined { + public getDefinitionEnum(name: string, value?: string): ModelDefinition | undefined { + if (!value) { + return; + } const definition = this.getDefinition(name); if (!definition) { return undefined; @@ -73,7 +76,7 @@ const consumerProperties = [ }, { name: "string", - description: "Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java) which currently only supports `UTF-8` encoding." + description: "Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java)." }, { name: "double", @@ -107,7 +110,7 @@ const consumerProperties = [ }, { name: "string", - description: "Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java) which currently only supports `UTF-8` encoding." + description: "Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java)." }, { name: "double", @@ -158,7 +161,7 @@ const producerProperties = [ enum: [ { name: "string", - description: "Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java) which currently only supports `UTF-8` encoding." + description: "Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java)." }, { name: "double", @@ -188,7 +191,7 @@ const producerProperties = [ enum: [ { name: "string", - description: "Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java) which currently only supports `UTF-8` encoding." + description: "Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java)." }, { name: "double", diff --git a/src/kafka-file/languageservice/parser/kafkaFileParser.ts b/src/kafka-file/languageservice/parser/kafkaFileParser.ts index db3fc849..64895c94 100644 --- a/src/kafka-file/languageservice/parser/kafkaFileParser.ts +++ b/src/kafka-file/languageservice/parser/kafkaFileParser.ts @@ -11,8 +11,10 @@ export enum NodeKind { property, propertyKey, propertyAssigner, - propertyValue, - mustacheExpression + propertyValue, + mustacheExpression, + calleeFunction, + parameter } export interface Node { start: Position; @@ -178,7 +180,7 @@ export class Property extends BaseNode { return true; } - findNodeAt(position : Position) : Node { + findNodeAt(position: Position): Node { if (this.isBeforeAssigner(position)) { return this.key?.findNodeAt(position) || this; } @@ -242,6 +244,34 @@ export class DynamicChunk extends ChildrenNode { } +export class Parameter extends Chunk { + name?: string; + + public get value() : string { + return this.content?.trim(); + } + +} + +export class CalleeFunction extends ChildrenNode { + startParametersCharacter?: number; + endParametersCharacter?: number; + + constructor(public readonly content: string, start: Position, end: Position) { + super(start, end, NodeKind.calleeFunction); + parseParameters(this); + } + + public get functionName() : string { + return this.startParametersCharacter ? this.content.substring(0, this.startParametersCharacter - this.start.character).trim() : this.content.trim(); + } + + public get parameters(): Array { + return this.children; + } + +} + /** * Mustache expression AST (ex : {{random.words}}) */ @@ -459,11 +489,18 @@ function createProperty(lineText: string, lineNumber: number, parent: Block): Pr const content = lineText.substr(start, end); if (withinValue) { const propertyName = propertyKey?.content.trim(); - if (propertyName === 'key') { - propertyValue = new DynamicChunk(content, new Position(lineNumber, start), new Position(lineNumber, end), NodeKind.propertyValue); - } else { - propertyValue = new Chunk(content, new Position(lineNumber, start), new Position(lineNumber, end), NodeKind.propertyValue); - } + switch (propertyName) { + case "key": + propertyValue = new DynamicChunk(content, new Position(lineNumber, start), new Position(lineNumber, end), NodeKind.propertyValue); + break; + case "key-format": + case "value-format": + propertyValue = new CalleeFunction(content, new Position(lineNumber, start), new Position(lineNumber, end)); + break; + default: + propertyValue = new Chunk(content, new Position(lineNumber, start), new Position(lineNumber, end), NodeKind.propertyValue); + break; + } } else { propertyKey = new Chunk(content, new Position(lineNumber, start), new Position(lineNumber, end), NodeKind.propertyKey); } @@ -483,6 +520,66 @@ export class ExpressionEdge { } } +function parseParameters(parent: CalleeFunction) { + const content = parent.content; + let startLine = parent.start.line; + let startColumn = parent.start.character; + let currentLine = startLine; + let currentColumn = startColumn; + let previousChar: string | undefined; + let startParameter: number | undefined; + + function addParameterIfNeeded() { + if (startParameter) { + const value = content.substring(startParameter - startColumn, currentColumn - startColumn); + const start = new Position(currentLine, startParameter); + const end = new Position(currentLine, currentColumn); + parent.addChild(new Parameter(value, start, end, NodeKind.parameter)); + } + } + + for (let currentOffset = 0; currentOffset < content.length; currentOffset++) { + const currentChar = content[currentOffset]; + switch (currentChar) { + case '\r': + // compute line, column position + currentLine++; + currentColumn = 0; + break; + case '\n': { + if (previousChar !== '\r') { + // compute line, column position + currentLine++; + currentColumn = 0; + } + break; + } + case '(': { + if (!parent.startParametersCharacter) { + parent.startParametersCharacter = currentColumn; + startParameter = currentColumn + 1; + } + break; + } + case ')': { + parent.endParametersCharacter = currentColumn; + addParameterIfNeeded(); + startParameter = undefined; + break; + } + case ',': { + addParameterIfNeeded(); + startParameter = currentColumn + 1; + break; + } + } + if (currentChar !== '\r' && currentChar !== '\n') { + currentColumn++; + } + } + addParameterIfNeeded(); +} + function parseMustacheExpressions(parent: DynamicChunk) { const content = parent.content; let startLine = parent.start.line; @@ -532,7 +629,7 @@ function parseMustacheExpressions(parent: DynamicChunk) { } // 2. create mustache expression AST by visiting collected edges - + let previousEdge = new ExpressionEdge(new Position(startLine, startColumn), 0, true); const endOfValueEdge = new ExpressionEdge(new Position(currentLine, currentColumn), currentOffset, false); for (let i = 0; i < edges.length; i++) { @@ -548,7 +645,7 @@ function parseMustacheExpressions(parent: DynamicChunk) { const openedEdge = currentEdge; let closedEdge = endOfValueEdge; - let closed = false; + let closed = false; if (matchingClosedEdge) { // '}}' has been found closed = true; diff --git a/src/kafka-file/languageservice/services/codeLensProvider.ts b/src/kafka-file/languageservice/services/codeLensProvider.ts index 54633634..70db451c 100644 --- a/src/kafka-file/languageservice/services/codeLensProvider.ts +++ b/src/kafka-file/languageservice/services/codeLensProvider.ts @@ -1,9 +1,10 @@ import { TextDocument, CodeLens, Range } from "vscode"; import { ClientState, ConsumerLaunchState } from "../../../client"; import { createProducerUri, ProducerLaunchState } from "../../../client/producer"; +import { SerializationSetting } from "../../../client/serialization"; import { LaunchConsumerCommand, ProduceRecordCommand, ProduceRecordCommandHandler, SelectClusterCommandHandler, StartConsumerCommandHandler, StopConsumerCommandHandler } from "../../../commands"; import { ProducerLaunchStateProvider, ConsumerLaunchStateProvider, SelectedClusterProvider } from "../kafkaFileLanguageService"; -import { Block, BlockType, ConsumerBlock, KafkaFileDocument, ProducerBlock } from "../parser/kafkaFileParser"; +import { Block, BlockType, ConsumerBlock, KafkaFileDocument, CalleeFunction, ProducerBlock } from "../parser/kafkaFileParser"; /** * Kafka file codeLens support. @@ -43,7 +44,7 @@ export class KafkaFileCodeLenses { getClusterStatus(state: ClientState | undefined) { switch (state) { case ClientState.disconnected: - return `$(eye-closed) `; + return `$(eye-closed) `; case ClientState.connecting: return `$(sync~spin) `; case ClientState.connected: @@ -110,7 +111,9 @@ export class KafkaFileCodeLenses { let key; let value = block.value?.content; let keyFormat; + let keyFormatSettings: Array | undefined; let valueFormat; + let valueFormatSettings: Array | undefined; block.properties.forEach(property => { switch (property.propertyName) { case 'topic': @@ -119,12 +122,18 @@ export class KafkaFileCodeLenses { case 'key': key = property.propertyValue; break; - case 'key-format': - keyFormat = property.propertyValue; + case 'key-format': { + const callee = property.value; + keyFormat = callee.functionName; + keyFormatSettings = this.getSerializationSettings(callee); break; - case 'value-format': - valueFormat = property.propertyValue; + } + case 'value-format': { + const callee = property.value; + valueFormat = callee.functionName; + valueFormatSettings = this.getSerializationSettings(callee); break; + } } }); return { @@ -133,10 +142,19 @@ export class KafkaFileCodeLenses { key, value, messageKeyFormat: keyFormat, - messageValueFormat: valueFormat + messageKeyFormatSettings: keyFormatSettings, + messageValueFormat: valueFormat, + messageValueFormatSettings: valueFormatSettings } as ProduceRecordCommand; } + private getSerializationSettings(callee: CalleeFunction): SerializationSetting[] | undefined { + const parameters = callee.parameters; + if (parameters.length > 0) { + return parameters.map(p => { return { value: p.value }; }); + } + } + private createConsumerLens(block: ConsumerBlock, lineRange: Range, range: Range, clusterName: string | undefined, clusterId: string | undefined, clusterState: ClientState | undefined): CodeLens[] { const launchCommand = this.createLaunchConsumerCommand(block, range, clusterId); const lenses: CodeLens[] = []; @@ -193,7 +211,9 @@ export class KafkaFileCodeLenses { let partitions; let offset; let keyFormat; + let keyFormatSettings; let valueFormat; + let valueFormatSettings; block.properties.forEach(property => { switch (property.propertyName) { case 'topic': @@ -205,12 +225,18 @@ export class KafkaFileCodeLenses { case 'partitions': partitions = property.propertyValue; break; - case 'key-format': - keyFormat = property.propertyValue; + case 'key-format': { + const callee = property.value; + keyFormat = callee.functionName; + keyFormatSettings = this.getSerializationSettings(callee); break; - case 'value-format': - valueFormat = property.propertyValue; + } + case 'value-format': { + const callee = property.value; + valueFormat = callee.functionName; + valueFormatSettings = this.getSerializationSettings(callee); break; + } } }); @@ -221,7 +247,9 @@ export class KafkaFileCodeLenses { fromOffset: offset, partitions, messageKeyFormat: keyFormat, - messageValueFormat: valueFormat + messageValueFormat: valueFormat, + messageKeyFormatSettings: keyFormatSettings, + messageValueFormatSettings: valueFormatSettings } as LaunchConsumerCommand; } } diff --git a/src/kafka-file/languageservice/services/completion.ts b/src/kafka-file/languageservice/services/completion.ts index 36aec62a..177a3923 100644 --- a/src/kafka-file/languageservice/services/completion.ts +++ b/src/kafka-file/languageservice/services/completion.ts @@ -1,7 +1,14 @@ import { TextDocument, Position, CompletionList, CompletionItem, SnippetString, MarkdownString, CompletionItemKind, Range } from "vscode"; import { createTopicDocumentation, SelectedClusterProvider, TopicProvider } from "../kafkaFileLanguageService"; import { consumerModel, fakerjsAPIModel, Model, ModelDefinition, producerModel } from "../model"; -import { Block, BlockType, Chunk, ConsumerBlock, KafkaFileDocument, MustacheExpression, NodeKind, ProducerBlock, Property } from "../parser/kafkaFileParser"; +import { Block, BlockType, Chunk, ConsumerBlock, KafkaFileDocument, CalleeFunction, MustacheExpression, NodeKind, Parameter, ProducerBlock, Property } from "../parser/kafkaFileParser"; + +/** + * Supported encoding by nodejs Buffer. + * + * @see https://nodejs.org/api/buffer.html#buffer_buffers_and_character_encodings + */ +const bufferEncoding = ["utf8", "utf16le", "base64", "latin1", "hex"]; /** * Kafka file completion support. @@ -18,7 +25,7 @@ export class KafkaFileCompletion { return; } - // Following comments with use the '|' character to show the position where the complation is trigerred + // Following comments with use the '|' character to show the position where the compilation is triggered const items: Array = []; switch (node.kind) { case NodeKind.consumerBlock: { @@ -80,23 +87,53 @@ export class KafkaFileCompletion { } } else { const propertyValue = property.value; - const expression = propertyValue?.findNodeBefore(position); - if (expression && expression.kind === NodeKind.mustacheExpression) { - // Completion was triggered inside a mustache expression which is inside the property value + // Property value can be: + // - a simple value -> abcd + // - a mustache expression -> {{...}} + // - a method parameter -> string(utf-8) + const previousNode = propertyValue?.findNodeBefore(position); + switch (previousNode?.kind) { + case NodeKind.mustacheExpression: { + // Completion was triggered inside a mustache expression which is inside the property value + + // PRODUCER + // key: abcd-{{|}} + const expression = previousNode; + this.collectFakerJSExpressions(expression, producerFakerJSEnabled, position, items); + break; + } + case NodeKind.calleeFunction: { + // Check if completion was triggered inside an empty method parameter + + const callee = previousNode; + if (callee.startParametersCharacter) { + // PRODUCER + // key-format: string(|) + this.collectMethodParameters(callee, position, items); + } else { + // PRODUCER + // key-format: | + await this.collectDefaultPropertyValues(property, propertyValue, items); + } + break; + } + case NodeKind.parameter: { + // Completion was triggered inside a method parameter which is inside the property value + + // PRODUCER + // key-format: string(ut|f) + + // OR - // PRODUCER - // key: abcd-{{|}} - this.collectFakerJSExpressions(expression, producerFakerJSEnabled, position, items); - } else { - const block = property.parent; - if (block.type === BlockType.consumer) { - // CONSUMER - // key-format: | - await this.collectConsumerPropertyValues(propertyValue, property, block, items); - } else { // PRODUCER - // key-format: | - await this.collectProducerPropertyValues(propertyValue, property, block, items); + // key-format: string(utf8, u|t) + + const parameter = previousNode; + this.collectMethodParameters(parameter.parent, position, items); + break; + } + default: { + await this.collectDefaultPropertyValues(property, propertyValue, items); } } } @@ -205,6 +242,19 @@ export class KafkaFileCompletion { item.insertText = insertText; item.range = valueRange; items.push(item); + + if (value === 'string' && (propertyName === 'key-format' || propertyName === 'value-format')) { + const item = new CompletionItem('string with encoding...'); + item.kind = CompletionItemKind.Value; + const insertText = new SnippetString(' '); + insertText.appendText(value); + insertText.appendText('('); + insertText.appendChoice(bufferEncoding); + insertText.appendText(')'); + item.insertText = insertText; + item.range = valueRange; + items.push(item); + } }); } @@ -275,9 +325,40 @@ export class KafkaFileCompletion { }*/ } } + + collectMethodParameters(callee: CalleeFunction, position: Position, items: CompletionItem[]) { + const parameter = callee.parameters.find(p => position.isAfterOrEqual(p.start) && position.isBeforeOrEqual(p.end)); + if (!parameter) { + return; + } + switch (callee.functionName) { + case 'string': { + const range = parameter.range(); + bufferEncoding.forEach(encoding => { + const item = new CompletionItem(encoding); + item.kind = CompletionItemKind.EnumMember; + item.range = range; + items.push(item); + }); + } + } + } + + async collectDefaultPropertyValues(property: Property, propertyValue: Chunk | undefined, items: CompletionItem[]) { + const block = property.parent; + if (block.type === BlockType.consumer) { + // CONSUMER + // key-format: | + await this.collectConsumerPropertyValues(propertyValue, property, block, items); + } else { + // PRODUCER + // key-format: | + await this.collectProducerPropertyValues(propertyValue, property, block, items); + } + } } -function createMarkdownString(contents : string) { +function createMarkdownString(contents: string) { const doc = new MarkdownString(contents); doc.isTrusted = true; return doc; diff --git a/src/kafka-file/languageservice/services/diagnostics.ts b/src/kafka-file/languageservice/services/diagnostics.ts index 94e07152..4c62ee1b 100644 --- a/src/kafka-file/languageservice/services/diagnostics.ts +++ b/src/kafka-file/languageservice/services/diagnostics.ts @@ -1,5 +1,5 @@ import { Diagnostic, DiagnosticSeverity, Position, Range, TextDocument } from "vscode"; -import { Block, BlockType, ConsumerBlock, DynamicChunk, KafkaFileDocument, MustacheExpression, ProducerBlock, Property } from "../parser/kafkaFileParser"; +import { Block, BlockType, Chunk, ConsumerBlock, DynamicChunk, KafkaFileDocument, CalleeFunction, MustacheExpression, ProducerBlock, Property } from "../parser/kafkaFileParser"; import { ConsumerValidator } from "../../../validators/consumer"; import { ProducerValidator } from "../../../validators/producer"; import { CommonsValidator } from "../../../validators/commons"; @@ -236,7 +236,7 @@ export class KafkaFileDiagnostics { if (!range) { return; } - const errorMessage = await this.validateValue(propertyName, type, propertyValue); + const errorMessage = await this.validateValue(propertyName, type, property.value); if (errorMessage) { diagnostics.push(new Diagnostic(range, errorMessage, DiagnosticSeverity.Error)); } @@ -249,18 +249,20 @@ export class KafkaFileDiagnostics { } } - private async validateValue(propertyName: string, type: BlockType, propertyValue?: string): Promise { + private async validateValue(propertyName: string, type: BlockType, propertyValue?: Chunk): Promise { switch (propertyName) { case 'topic': - return CommonsValidator.validateTopic(propertyValue); + return CommonsValidator.validateTopic(propertyValue?.content.trim()); case 'key-format': - return type === BlockType.consumer ? ConsumerValidator.validateKeyFormat(propertyValue) : ProducerValidator.validateKeyFormat(propertyValue); + const keyFormat = (propertyValue).functionName; + return type === BlockType.consumer ? ConsumerValidator.validateKeyFormat(keyFormat) : ProducerValidator.validateKeyFormat(keyFormat); case 'value-format': - return type === BlockType.consumer ? ConsumerValidator.validateValueFormat(propertyValue) : ProducerValidator.validateValueFormat(propertyValue); + const valueFormat = (propertyValue).functionName; + return type === BlockType.consumer ? ConsumerValidator.validateValueFormat(valueFormat) : ProducerValidator.validateValueFormat(valueFormat); case 'from': - return ConsumerValidator.validateOffset(propertyValue); + return ConsumerValidator.validateOffset(propertyValue?.content.trim()); case 'partitions': { - return ConsumerValidator.validatePartitions(propertyValue); + return ConsumerValidator.validatePartitions(propertyValue?.content.trim()); } } } diff --git a/src/kafka-file/languageservice/services/hover.ts b/src/kafka-file/languageservice/services/hover.ts index 36218c71..25c2b5f4 100644 --- a/src/kafka-file/languageservice/services/hover.ts +++ b/src/kafka-file/languageservice/services/hover.ts @@ -2,7 +2,7 @@ import { Hover, MarkdownString, Position, Range, TextDocument } from "vscode"; import { getDocumentationPageUri } from "../../../docs/markdownPreviewProvider"; import { createTopicDocumentation, SelectedClusterProvider, TopicProvider } from "../kafkaFileLanguageService"; import { consumerModel, Model, producerModel } from "../model"; -import { Block, BlockType, Chunk, ConsumerBlock, KafkaFileDocument, MustacheExpression, NodeKind, ProducerBlock, Property } from "../parser/kafkaFileParser"; +import { Block, BlockType, CalleeFunction, Chunk, ConsumerBlock, KafkaFileDocument, MustacheExpression, NodeKind, ProducerBlock, Property } from "../parser/kafkaFileParser"; export class KafkaFileHover { @@ -52,19 +52,16 @@ export class KafkaFileHover { } } + case NodeKind.calleeFunction: { + const property = node.parent; + const propertyValue = (node).functionName; + return this.getHoverForPropertyValue(property, propertyValue); + } + case NodeKind.propertyValue: { - const propertyValue = node; - const property = propertyValue.parent; - const block = property.parent; - if (block.type === BlockType.consumer) { - // CONSUMER - // key-format: | - return await this.getHoverForConsumerPropertyValues(propertyValue, property, block); - } else { - // PRODUCER - // key-format: | - return await this.getHoverForProducerPropertyValues(propertyValue, property, block); - } + const property = node.parent; + const propertyValue = property.propertyValue; + return this.getHoverForPropertyValue(property, propertyValue); } case NodeKind.mustacheExpression: { @@ -93,7 +90,22 @@ export class KafkaFileHover { } } - async getHoverForConsumerPropertyValues(propertyValue: Chunk, property: Property, block: ConsumerBlock): Promise { + async getHoverForPropertyValue(property: Property, propertyValue?: string) { + if (!propertyValue) { + return; + } + const block = property.parent; + if (block.type === BlockType.consumer) { + // CONSUMER + // key-format: | + return await this.getHoverForConsumerPropertyValues(propertyValue, property, block); + } else { + // PRODUCER + // key-format: | + return await this.getHoverForProducerPropertyValues(propertyValue, property, block); + } + } + async getHoverForConsumerPropertyValues(propertyValue: string, property: Property, block: ConsumerBlock): Promise { const propertyName = property.propertyName; switch (propertyName) { case 'topic': @@ -108,7 +120,7 @@ export class KafkaFileHover { } - async getHoverForProducerPropertyValues(propertyValue: Chunk, property: Property, block: ProducerBlock): Promise { + async getHoverForProducerPropertyValues(propertyValue: string, property: Property, block: ProducerBlock): Promise { const propertyName = property.propertyName; switch (propertyName) { case 'topic': @@ -150,12 +162,12 @@ export class KafkaFileHover { return undefined; } - async getHoverForPropertyValues(propertyValue: Chunk, property: Property, block: Block, metadata: Model): Promise { + async getHoverForPropertyValues(propertyValue: string, property: Property, block: Block, metadata: Model): Promise { const propertyName = property.propertyName; if (!propertyName) { return; } - const definition = metadata.getDefinitionEnum(propertyName, propertyValue.content.trim()); + const definition = metadata.getDefinitionEnum(propertyName, propertyValue); if (definition && definition.description) { return createHover(definition.description, property.propertyTrimmedValueRange); } diff --git a/src/providers/consumerVirtualTextDocumentProvider.ts b/src/providers/consumerVirtualTextDocumentProvider.ts index e3efb224..498a9301 100644 --- a/src/providers/consumerVirtualTextDocumentProvider.ts +++ b/src/providers/consumerVirtualTextDocumentProvider.ts @@ -1,6 +1,7 @@ import * as vscode from "vscode"; import { ConsumedRecord, Consumer, ConsumerChangedStatusEvent, ConsumerCollection, ConsumerCollectionChangedEvent, ConsumerLaunchState, RecordReceivedEvent } from "../client"; +import { SerializationSetting } from "../client/serialization"; import { CommonMessages } from "../constants"; import { ClusterSettings } from "../settings/clusters"; @@ -86,15 +87,22 @@ export class ConsumerVirtualTextDocumentProvider implements vscode.TextDocumentC line += ` - partitions: ${consumer.options.partitions}\n`; } if (consumer.options.messageKeyFormat) { - line += ` - key format: ${consumer.options.messageKeyFormat}\n`; + line += ` - key format: ${consumer.options.messageKeyFormat}${this.getFormatSettings(consumer.options.messageKeyFormatSettings)}\n`; } if (consumer.options.messageValueFormat) { - line += ` - value format: ${consumer.options.messageValueFormat}\n`; + line += ` - value format: ${consumer.options.messageValueFormat}${this.getFormatSettings(consumer.options.messageValueFormatSettings)}\n`; } line += `\n`; this.updateBuffer(consumer.uri, line); } + getFormatSettings(messageKeyFormatSettings: SerializationSetting[] | undefined) { + if (!messageKeyFormatSettings) { + return ''; + } + return `(${messageKeyFormatSettings.map(s => s.value).join(',')})`; + } + onDidConsumerError(uri: vscode.Uri, error: any): void { if (!this.isActive(uri)) { return; diff --git a/src/test/suite/kafka-file/languageservice/codeLens.test.ts b/src/test/suite/kafka-file/languageservice/codeLens.test.ts index 88538c96..b96da7bf 100644 --- a/src/test/suite/kafka-file/languageservice/codeLens.test.ts +++ b/src/test/suite/kafka-file/languageservice/codeLens.test.ts @@ -1,4 +1,5 @@ import { ConsumerLaunchState } from "../../../../client"; +import { LaunchConsumerCommand, ProduceRecordCommand } from "../../../../commands"; import { getLanguageService } from "../../../../kafka-file/languageservice/kafkaFileLanguageService"; import { assertCodeLens, codeLens, LanguageServiceConfig, position } from "./kafkaAssert"; @@ -76,10 +77,12 @@ suite("Kafka File PRODUCER CodeLens Test Suite", () => { clusterId: 'cluster1', key: undefined, messageKeyFormat: undefined, + messageKeyFormatSettings: undefined, messageValueFormat: undefined, + messageValueFormatSettings: undefined, topicId: undefined, value: undefined - }, + } as ProduceRecordCommand, 1 ] }), @@ -91,10 +94,12 @@ suite("Kafka File PRODUCER CodeLens Test Suite", () => { clusterId: 'cluster1', key: undefined, messageKeyFormat: undefined, + messageKeyFormatSettings: undefined, messageValueFormat: undefined, + messageValueFormatSettings: undefined, topicId: undefined, value: undefined - }, + } as ProduceRecordCommand, 10 ] }), @@ -120,10 +125,12 @@ suite("Kafka File PRODUCER CodeLens Test Suite", () => { clusterId: 'cluster1', key: 'a-key', messageKeyFormat: 'long', + messageKeyFormatSettings: undefined, messageValueFormat: 'string', + messageValueFormatSettings: undefined, topicId: 'abcd', value: 'ABCD\nEFGH' - }, + } as ProduceRecordCommand, 1 ] }), @@ -135,10 +142,68 @@ suite("Kafka File PRODUCER CodeLens Test Suite", () => { clusterId: 'cluster1', key: 'a-key', messageKeyFormat: 'long', + messageKeyFormatSettings: undefined, messageValueFormat: 'string', + messageValueFormatSettings: undefined, topicId: 'abcd', value: 'ABCD\nEFGH' - }, + } as ProduceRecordCommand, + 10 + ] + }), + codeLens(position(0, 0), position(0, 0), { + command: 'vscode-kafka.explorer.selectcluster', + title: 'CLUSTER_1' + }) + ], languageService); + + }); + + test("PRODUCER with string encoding", async () => { + + const languageServiceConfig = new LanguageServiceConfig(); + languageServiceConfig.setSelectedCluster({ clusterId: 'cluster1', clusterName: 'CLUSTER_1' }); + const languageService = getLanguageService(languageServiceConfig, languageServiceConfig, languageServiceConfig, languageServiceConfig); + + await assertCodeLens( + 'PRODUCER\n' + + 'key: a-key\n' + + 'topic: abcd\n' + + 'key-format: string(base64)\n' + + 'value-format: string(ascii)\n' + + 'ABCD\n' + + 'EFGH', [ + codeLens(position(0, 0), position(0, 0), { + command: 'vscode-kafka.producer.produce', + title: '$(run) Produce record', + arguments: [ + { + clusterId: 'cluster1', + key: 'a-key', + messageKeyFormat: 'string', + messageKeyFormatSettings: [{ value: 'base64' }], + messageValueFormat: 'string', + messageValueFormatSettings: [{ value: 'ascii' }], + topicId: 'abcd', + value: 'ABCD\nEFGH' + } as ProduceRecordCommand, + 1 + ] + }), + codeLens(position(0, 0), position(0, 0), { + command: 'vscode-kafka.producer.produce', + title: '$(run-all) Produce record x 10', + arguments: [ + { + clusterId: 'cluster1', + key: 'a-key', + messageKeyFormat: 'string', + messageKeyFormatSettings: [{ value: 'base64' }], + messageValueFormat: 'string', + messageValueFormatSettings: [{ value: 'ascii' }], + topicId: 'abcd', + value: 'ABCD\nEFGH' + } as ProduceRecordCommand, 10 ] }), @@ -199,10 +264,12 @@ suite("Kafka File CONSUMER CodeLens Test Suite", () => { consumerGroupId: 'group-1', fromOffset: undefined, messageKeyFormat: undefined, + messageKeyFormatSettings: undefined, messageValueFormat: undefined, + messageValueFormatSettings: undefined, partitions: undefined, topicId: '' - } + } as LaunchConsumerCommand ] }), codeLens(position(0, 0), position(0, 0), { @@ -227,10 +294,12 @@ suite("Kafka File CONSUMER CodeLens Test Suite", () => { consumerGroupId: 'group-1', fromOffset: '10', messageKeyFormat: 'long', + messageKeyFormatSettings: undefined, messageValueFormat: 'string', + messageValueFormatSettings: undefined, partitions: '1,2,3', topicId: 'abcd' - } + } as LaunchConsumerCommand ] }), codeLens(position(0, 0), position(0, 0), { @@ -253,10 +322,12 @@ suite("Kafka File CONSUMER CodeLens Test Suite", () => { consumerGroupId: 'group-1', fromOffset: undefined, messageKeyFormat: undefined, + messageKeyFormatSettings: undefined, messageValueFormat: undefined, + messageValueFormatSettings: undefined, partitions: undefined, topicId: '' - } + } as LaunchConsumerCommand ] }), codeLens(position(0, 0), position(0, 0), { @@ -274,10 +345,12 @@ suite("Kafka File CONSUMER CodeLens Test Suite", () => { consumerGroupId: 'group-2', fromOffset: undefined, messageKeyFormat: undefined, + messageKeyFormatSettings: undefined, messageValueFormat: undefined, + messageValueFormatSettings: undefined, partitions: undefined, topicId: '' - } + } as LaunchConsumerCommand ] }), codeLens(position(1, 0), position(1, 0), { @@ -287,4 +360,43 @@ suite("Kafka File CONSUMER CodeLens Test Suite", () => { ], languageService); }); + test("CONSUMER with string encoding", async () => { + + const languageServiceConfig = new LanguageServiceConfig(); + languageServiceConfig.setSelectedCluster({ clusterId: 'cluster1', clusterName: 'CLUSTER_1' }); + languageServiceConfig.setConsumerLaunchState('cluster1', 'group-1', ConsumerLaunchState.started); + const languageService = getLanguageService(languageServiceConfig, languageServiceConfig, languageServiceConfig, languageServiceConfig); + + await assertCodeLens( + 'CONSUMER group-1\n' + + 'topic: abcd\n' + + 'from: 10\n' + + 'partitions: 1,2,3\n' + + 'key-format: string(base64)\n' + + 'value-format: string(ascii)\n', [ + codeLens(position(0, 0), position(0, 0), { + command: 'vscode-kafka.consumer.stop', + title: '$(debug-stop) Stop consumer', + arguments: [ + { + clusterId: 'cluster1', + consumerGroupId: 'group-1', + fromOffset: '10', + messageKeyFormat: 'string', + messageKeyFormatSettings: [{ value: 'base64' }], + messageValueFormat: 'string', + messageValueFormatSettings: [{ value: 'ascii' }], + partitions: '1,2,3', + topicId: 'abcd' + } as LaunchConsumerCommand + ] + }), + codeLens(position(0, 0), position(0, 0), { + command: 'vscode-kafka.explorer.selectcluster', + title: 'CLUSTER_1' + }) + ], languageService); + + }); + }); diff --git a/src/test/suite/kafka-file/languageservice/completionProperties.test.ts b/src/test/suite/kafka-file/languageservice/completionProperties.test.ts index 783c0cce..924f4914 100644 --- a/src/test/suite/kafka-file/languageservice/completionProperties.test.ts +++ b/src/test/suite/kafka-file/languageservice/completionProperties.test.ts @@ -392,6 +392,11 @@ suite("Kafka File CONSUMER Completion Test Suite", () => { insertText: ' string', range: range(position(1, 11), position(1, 11)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(1, 11), position(1, 11)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -437,6 +442,11 @@ suite("Kafka File CONSUMER Completion Test Suite", () => { insertText: ' string', range: range(position(1, 13), position(1, 13)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(1, 13), position(1, 13)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -466,6 +476,65 @@ suite("Kafka File CONSUMER Completion Test Suite", () => { }); }); + test("CONSUMER property value for string encoding of key-format", async () => { + await testCompletion( + 'CONSUMER a\n' + + 'key-format: string(|' + , { + items: [ + { + label: 'utf8', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'utf16le', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'base64', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'latin1', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'hex', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + } + ] + }); + }); + + test("CONSUMER property value for string encoding of value-format", async () => { + await testCompletion( + 'CONSUMER a\n' + + 'value-format: string(|' + , { + items: [ + { + label: 'utf8', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'utf16le', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'base64', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'latin1', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'hex', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + } + ] + }); + }); }); suite("Kafka File PRODUCER Completion Test Suite", () => { @@ -694,6 +763,11 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { insertText: ' string', range: range(position(1, 11), position(1, 11)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(1, 11), position(1, 11)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -734,6 +808,11 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { insertText: ' string', range: range(position(1, 11), position(1, 13)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(1, 11), position(1, 13)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -775,6 +854,11 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { insertText: ' string', range: range(position(2, 11), position(2, 13)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(2, 11), position(2, 13)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -817,6 +901,11 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { insertText: ' string', range: range(position(2, 11), position(2, 13)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(2, 11), position(2, 13)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -857,6 +946,11 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { insertText: ' string', range: range(position(1, 13), position(1, 13)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(1, 13), position(1, 13)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -886,4 +980,64 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { }); }); + test("PRODUCER property value for string encoding of key-format", async () => { + await testCompletion( + 'PRODUCER a\n' + + 'key-format: string(|' + , { + items: [ + { + label: 'utf8', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'utf16le', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'base64', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'latin1', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'hex', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + } + ] + }); + }); + + test("PRODUCER property value for string encoding of value-format", async () => { + await testCompletion( + 'PRODUCER a\n' + + 'value-format: string(|' + , { + items: [ + { + label: 'utf8', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'utf16le', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'base64', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'latin1', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'hex', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + } + ] + }); + }); + }); diff --git a/src/test/suite/kafka-file/languageservice/hover.test.ts b/src/test/suite/kafka-file/languageservice/hover.test.ts index c78a4619..19b8326d 100644 --- a/src/test/suite/kafka-file/languageservice/hover.test.ts +++ b/src/test/suite/kafka-file/languageservice/hover.test.ts @@ -116,7 +116,7 @@ suite("Kafka File CONSUMER Hover Test Suite", () => { 'CONSUMER\n' + 'key-format: stri|ng', hover( - 'Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java) which currently only supports `UTF-8` encoding.', + 'Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java).', position(1, 12), position(1, 18) ) @@ -144,7 +144,7 @@ suite("Kafka File CONSUMER Hover Test Suite", () => { 'CONSUMER\n' + 'value-format: stri|ng', hover( - 'Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java) which currently only supports `UTF-8` encoding.', + 'Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java).', position(1, 14), position(1, 20) ) @@ -270,7 +270,7 @@ suite("Kafka File PRODUCER Hover Test Suite", () => { 'PRODUCER\n' + 'key-format: stri|ng', hover( - 'Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java) which currently only supports `UTF-8` encoding.', + 'Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java).', position(1, 12), position(1, 18) ) @@ -298,7 +298,7 @@ suite("Kafka File PRODUCER Hover Test Suite", () => { 'PRODUCER\n' + 'value-format: stri|ng', hover( - 'Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java) which currently only supports `UTF-8` encoding.', + 'Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java).', position(1, 14), position(1, 20) )