diff --git a/CHANGELOG.md b/CHANGELOG.md index d9e3cfcb..51b1c988 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ All notable changes to `Tools for Apache Kafka®` are documented in this file. - Validation for available topics in `.kafka` files. See [#153](https://github.com/jlandersen/vscode-kafka/issues/153). - Simplify snippets. See [#180](https://github.com/jlandersen/vscode-kafka/pull/180). - Hover support in `.kafka` files. See [#149](https://github.com/jlandersen/vscode-kafka/issues/149). +- String encoding serialization support. See [#181](https://github.com/jlandersen/vscode-kafka/issues/181). ## [0.12.0] - 2021-04-26 ### Added diff --git a/docs/Consuming.md b/docs/Consuming.md index 892da260..69871686 100644 --- a/docs/Consuming.md +++ b/docs/Consuming.md @@ -20,11 +20,6 @@ Once this command is launched, it creates a consumer group (with an auto-generat In this case, the starting offset can be only be configured via the [kafka.consumers.offset](#kafkaconsumersoffset) preference. -Known limitations: - -* UTF-8 encoded keys and values only. If data is encoded differently, it will not be pretty. -* One consumer group is created per topic (may change in the future to just have one for the extension). - ### Kafka file Define simple consumers in a `.kafka` file, using the following format: @@ -58,7 +53,7 @@ The `CONSUMER` block defines: The deserializers can have the following value: * `none`: no deserializer (ignores content). - * `string`: similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java) which currently only supports `UTF-8` encoding. + * `string`: similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java). By default it supports `UTF-8` encoding, but you can specify the encoding as parameter like this `string(base64)`. The valid encoding values are defined in [Node.js' buffers and character encodings](https://nodejs.org/api/buffer.html#buffer_buffers_and_character_encodings). * `double`: similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.DoubleDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/DoubleDeserializer.java). * `float`: similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.FloatDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/FloatDeserializer.java). * `integer`: similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.IntegerDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java). @@ -85,6 +80,10 @@ Completion is available for ![Property value completion](assets/kafka-file-consumer-property-value-completion.png) + * string encoding: + +![String encoding completion](assets/kafka-file-consumer-string-encoding-completion.png) + * topic: ![Topic completion](assets/kafka-file-consumer-topic-completion.png) diff --git a/docs/Producing.md b/docs/Producing.md index a865224d..b52d0f0e 100644 --- a/docs/Producing.md +++ b/docs/Producing.md @@ -36,7 +36,7 @@ The `PRODUCER` block defines: The serializers can have the following value: - * `string`: similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java) which currently only supports `UTF-8` encoding. + * `string`: similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java). By default it supports `UTF-8` encoding, but you can specify the encoding as parameter like this `string(base64)`. The valid encoding values are defined in [Node.js' buffers and character encodings](https://nodejs.org/api/buffer.html#buffer_buffers_and_character_encodings). * `double`: similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.DoubleSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/DoubleSerializer.java). * `float`: similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.FloatSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/FloatSerializer.java). * `integer`: similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.IntegerSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/IntegerSerializer.java). @@ -59,6 +59,10 @@ Completion is available for ![Property value completion](assets/kafka-file-producer-property-value-completion.png) + * string encoding: + +![String encoding completion](assets/kafka-file-producer-string-encoding-completion.png) + * randomized content (see following section): ![FakerJS completion](assets/kafka-file-producer-fakerjs-completion.png) diff --git a/docs/assets/kafka-file-consumer-string-encoding-completion.png b/docs/assets/kafka-file-consumer-string-encoding-completion.png new file mode 100644 index 00000000..2be08513 Binary files /dev/null and b/docs/assets/kafka-file-consumer-string-encoding-completion.png differ diff --git a/docs/assets/kafka-file-producer-string-encoding-completion.png b/docs/assets/kafka-file-producer-string-encoding-completion.png new file mode 100644 index 00000000..3987c5ee Binary files /dev/null and b/docs/assets/kafka-file-producer-string-encoding-completion.png differ diff --git a/src/client/consumer.ts b/src/client/consumer.ts index 59fb9294..f5a41bda 100644 --- a/src/client/consumer.ts +++ b/src/client/consumer.ts @@ -4,7 +4,7 @@ import * as vscode from "vscode"; import { ClientAccessor } from "."; import { getWorkspaceSettings, InitialConsumerOffset, ClusterSettings } from "../settings"; import { addQueryParameter, Client, ConnectionOptions } from "./client"; -import { deserialize, MessageFormat, SerializationdResult } from "./serialization"; +import { deserialize, MessageFormat, SerializationdResult, SerializationSetting } from "./serialization"; interface ConsumerOptions extends ConnectionOptions { consumerGroupId: string; @@ -12,7 +12,9 @@ interface ConsumerOptions extends ConnectionOptions { fromOffset: InitialConsumerOffset | string; partitions?: number[]; messageKeyFormat?: MessageFormat; + messageKeyFormatSettings?: SerializationSetting[]; messageValueFormat?: MessageFormat; + messageValueFormatSettings?: SerializationSetting[]; } export interface RecordReceivedEvent { @@ -62,7 +64,7 @@ export class Consumer implements vscode.Disposable { public error: any; constructor(public uri: vscode.Uri, clusterSettings: ClusterSettings, private clientAccessor: ClientAccessor) { - const { clusterId, consumerGroupId, topicId, fromOffset, partitions, messageKeyFormat, messageValueFormat } = extractConsumerInfoUri(uri); + const { clusterId, consumerGroupId, topicId, fromOffset, partitions, messageKeyFormat, messageKeyFormatSettings, messageValueFormat,messageValueFormatSettings } = extractConsumerInfoUri(uri); this.clusterId = clusterId; const cluster = clusterSettings.get(clusterId); @@ -81,7 +83,9 @@ export class Consumer implements vscode.Disposable { fromOffset: fromOffset || settings.consumerOffset, partitions: parsePartitions(partitions), messageKeyFormat, - messageValueFormat + messageKeyFormatSettings, + messageValueFormat, + messageValueFormatSettings }; } catch (e) { @@ -114,8 +118,8 @@ export class Consumer implements vscode.Disposable { this.consumer.run({ eachMessage: async ({ topic, partition, message }) => { - message.key = deserialize(message.key, this.options.messageKeyFormat); - message.value = deserialize(message.value, this.options.messageValueFormat); + message.key = deserialize(message.key, this.options.messageKeyFormat, this.options.messageKeyFormatSettings); + message.value = deserialize(message.value, this.options.messageValueFormat, this.options.messageValueFormatSettings); this.onDidReceiveMessageEmitter.fire({ uri: this.uri, record: { topic: topic, partition: partition, ...message }, @@ -360,14 +364,18 @@ export interface ConsumerInfoUri { fromOffset?: string; partitions?: string; messageKeyFormat?: MessageFormat; + messageKeyFormatSettings?: SerializationSetting[]; messageValueFormat?: MessageFormat; + messageValueFormatSettings?: SerializationSetting[]; } const TOPIC_QUERY_PARAMETER = 'topic'; const FROM_QUERY_PARAMETER = 'from'; const PARTITIONS_QUERY_PARAMETER = 'partitions'; const KEY_FORMAT_QUERY_PARAMETER = 'key'; +const KEY_FORMAT_SETTINGS_QUERY_PARAMETER = 'key-settings'; const VALUE_FORMAT_QUERY_PARAMETER = 'value'; +const VALUE_FORMAT_SETTINGS_QUERY_PARAMETER = 'value-settings'; export function createConsumerUri(info: ConsumerInfoUri): vscode.Uri { const path = `kafka:${info.clusterId}/${info.consumerGroupId}`; @@ -376,7 +384,9 @@ export function createConsumerUri(info: ConsumerInfoUri): vscode.Uri { query = addQueryParameter(query, FROM_QUERY_PARAMETER, info.fromOffset); query = addQueryParameter(query, PARTITIONS_QUERY_PARAMETER, info.partitions); query = addQueryParameter(query, KEY_FORMAT_QUERY_PARAMETER, info.messageKeyFormat); + query = addQueryParameter(query, KEY_FORMAT_SETTINGS_QUERY_PARAMETER, info.messageKeyFormatSettings?.map(p => p.value).join(',')); query = addQueryParameter(query, VALUE_FORMAT_QUERY_PARAMETER, info.messageValueFormat); + query = addQueryParameter(query, VALUE_FORMAT_SETTINGS_QUERY_PARAMETER, info.messageValueFormatSettings?.map(p => p.value).join(',')); return vscode.Uri.parse(path + query); } @@ -387,7 +397,9 @@ export function extractConsumerInfoUri(uri: vscode.Uri): ConsumerInfoUri { const from = urlParams.get(FROM_QUERY_PARAMETER); const partitions = urlParams.get(PARTITIONS_QUERY_PARAMETER); const messageKeyFormat = urlParams.get(KEY_FORMAT_QUERY_PARAMETER); + const messageKeyFormatSettings = urlParams.get(KEY_FORMAT_SETTINGS_QUERY_PARAMETER); const messageValueFormat = urlParams.get(VALUE_FORMAT_QUERY_PARAMETER); + const messageValueFormatSettings = urlParams.get(VALUE_FORMAT_SETTINGS_QUERY_PARAMETER); const result: ConsumerInfoUri = { clusterId, consumerGroupId, @@ -402,9 +414,19 @@ export function extractConsumerInfoUri(uri: vscode.Uri): ConsumerInfoUri { if (messageKeyFormat && messageKeyFormat.trim().length > 0) { result.messageKeyFormat = messageKeyFormat as MessageFormat; } + if (messageKeyFormatSettings) { + const settings = messageKeyFormatSettings.split(','). + map(value => { value }); + result.messageKeyFormatSettings = settings; + } if (messageValueFormat && messageValueFormat.trim().length > 0) { result.messageValueFormat = messageValueFormat as MessageFormat; } + if (messageValueFormatSettings) { + const settings = messageValueFormatSettings.split(','). + map(value => { value }); + result.messageValueFormatSettings = settings; + } return result; } diff --git a/src/client/producer.ts b/src/client/producer.ts index c144cc76..099d51ce 100644 --- a/src/client/producer.ts +++ b/src/client/producer.ts @@ -184,7 +184,7 @@ export interface ProducerInfoUri { clusterId: string; topicId?: string; key?: string; - value: string; + value?: string; } const TOPIC_QUERY_PARAMETER = 'topic'; diff --git a/src/client/serialization.ts b/src/client/serialization.ts index d04b071b..45a5a544 100644 --- a/src/client/serialization.ts +++ b/src/client/serialization.ts @@ -1,18 +1,23 @@ -export type MessageFormat = "none" | "string" | "double" | "float" | "integer" | "long" | "short" ; +export type MessageFormat = "none" | "string" | "double" | "float" | "integer" | "long" | "short"; export type SerializationdResult = any | Error; export class SerializationException extends Error { } +export interface SerializationSetting { + name?: string; + value?: string; +} + // ---------------- Serializers ---------------- interface Serializer { - serialize(data: string): Buffer | string | null; + serialize(data: string, settings?: SerializationSetting[]): Buffer | string | null; } const serializerRegistry: Map = new Map(); -export function serialize(data?: string, format?: MessageFormat): Buffer | string | null { +export function serialize(data?: string, format?: MessageFormat, settings?: SerializationSetting[]): Buffer | string | null { if (!data || !format) { return data || null; } @@ -20,7 +25,7 @@ export function serialize(data?: string, format?: MessageFormat): Buffer | strin if (!serializer) { throw new SerializationException(`Cannot find a serializer for ${format} format.`); } - return serializer.serialize(data); + return serializer.serialize(data, settings); } function getSerializer(format: MessageFormat): Serializer | undefined { @@ -79,7 +84,11 @@ class ShortSerializer implements Serializer { class StringSerializer implements Serializer { - serialize(value: string): Buffer | string | null { + serialize(value: string, settings?: SerializationSetting[]): Buffer | string | null { + const encoding = settings?.[0].value; + if (encoding) { + return Buffer.from(value, encoding); + } return value; }; } @@ -94,12 +103,12 @@ serializerRegistry.set("string", new StringSerializer()); // ---------------- Deserializers ---------------- interface Deserializer { - deserialize(data: Buffer): any; + deserialize(data: Buffer, settings?: SerializationSetting[]): any; } const deserializerRegistry: Map = new Map(); -export function deserialize(data: Buffer | null, format?: MessageFormat): SerializationdResult | null { +export function deserialize(data: Buffer | null, format?: MessageFormat, settings?: SerializationSetting[]): SerializationdResult | null { if (data === null || !format) { return data; } @@ -111,7 +120,7 @@ export function deserialize(data: Buffer | null, format?: MessageFormat): Serial if (!deserializer) { throw new SerializationException(`Cannot find a deserializer for ${format} format.`); } - return deserializer.deserialize(data); + return deserializer.deserialize(data, settings); } catch (e) { return e; @@ -189,11 +198,12 @@ class ShortDeserializer implements Deserializer { class StringDeserializer implements Deserializer { - deserialize(data: Buffer | null): any { + deserialize(data: Buffer | null, settings?: SerializationSetting[]): any { if (data === null) { return null; } - return data.toString(); + const encoding = settings?.[0].value; + return data.toString(encoding); } } diff --git a/src/commands/producers.ts b/src/commands/producers.ts index 0afe0a90..c56834e3 100644 --- a/src/commands/producers.ts +++ b/src/commands/producers.ts @@ -7,7 +7,7 @@ import { OutputChannelProvider } from "../providers/outputChannelProvider"; import { KafkaExplorer } from "../explorer"; import { WorkspaceSettings } from "../settings"; import { pickClient } from "./common"; -import { MessageFormat, serialize } from "../client/serialization"; +import { MessageFormat, SerializationSetting, serialize } from "../client/serialization"; import { createProducerUri, ProducerCollection, ProducerInfoUri, ProducerLaunchState } from "../client/producer"; import { ProducerRecord } from "kafkajs"; import { ProducerValidator } from "../validators/producer"; @@ -15,7 +15,9 @@ import { getErrorMessage } from "../errors"; export interface ProduceRecordCommand extends ProducerInfoUri { messageKeyFormat?: MessageFormat; + messageKeyFormatSettings?: SerializationSetting[]; messageValueFormat?: MessageFormat; + messageValueFormatSettings?: SerializationSetting[]; } export class ProduceRecordCommandHandler { @@ -46,6 +48,10 @@ export class ProduceRecordCommandHandler { channel.appendLine("No topic"); return; } + if (value === undefined) { + channel.appendLine("No value"); + return; + } if (this.settings.producerFakerJSEnabled) { faker.setLocale(this.settings.producerFakerJSLocale); } @@ -61,15 +67,15 @@ export class ProduceRecordCommandHandler { faker.seed(seed); const randomizedValue = faker.fake(value); return { - key: serialize(randomizedKey, command.messageKeyFormat), - value: serialize(randomizedValue, command.messageValueFormat) + key: serialize(randomizedKey, command.messageKeyFormat, command.messageKeyFormatSettings), + value: serialize(randomizedValue, command.messageValueFormat, command.messageValueFormatSettings) }; } // Return key/value message as-is return { - key: serialize(key, command.messageKeyFormat), - value: serialize(value, command.messageValueFormat) + key: serialize(key, command.messageKeyFormat, command.messageKeyFormatSettings), + value: serialize(value, command.messageValueFormat, command.messageValueFormatSettings) }; }); diff --git a/src/kafka-file/kafkaFileClient.ts b/src/kafka-file/kafkaFileClient.ts index 39852fc0..5b132f80 100644 --- a/src/kafka-file/kafkaFileClient.ts +++ b/src/kafka-file/kafkaFileClient.ts @@ -157,7 +157,7 @@ export function startLanguageClient( // Completion const completion = new KafkaFileCompletionItemProvider(kafkaFileDocuments, languageService, workspaceSettings); context.subscriptions.push( - vscode.languages.registerCompletionItemProvider(documentSelector, completion, ':', '{', '.') + vscode.languages.registerCompletionItemProvider(documentSelector, completion, ':', '{', '.', '(') ); // Validation diff --git a/src/kafka-file/languageservice/model.ts b/src/kafka-file/languageservice/model.ts index b602cadf..25f08278 100644 --- a/src/kafka-file/languageservice/model.ts +++ b/src/kafka-file/languageservice/model.ts @@ -21,7 +21,10 @@ export class Model { return this.getDefinitionEnum(name, value) !== undefined; } - public getDefinitionEnum(name: string, value: string): ModelDefinition | undefined { + public getDefinitionEnum(name: string, value?: string): ModelDefinition | undefined { + if (!value) { + return; + } const definition = this.getDefinition(name); if (!definition) { return undefined; @@ -73,7 +76,7 @@ const consumerProperties = [ }, { name: "string", - description: "Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java) which currently only supports `UTF-8` encoding." + description: "Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java)." }, { name: "double", @@ -107,7 +110,7 @@ const consumerProperties = [ }, { name: "string", - description: "Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java) which currently only supports `UTF-8` encoding." + description: "Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java)." }, { name: "double", @@ -158,7 +161,7 @@ const producerProperties = [ enum: [ { name: "string", - description: "Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java) which currently only supports `UTF-8` encoding." + description: "Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java)." }, { name: "double", @@ -188,7 +191,7 @@ const producerProperties = [ enum: [ { name: "string", - description: "Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java) which currently only supports `UTF-8` encoding." + description: "Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java)." }, { name: "double", diff --git a/src/kafka-file/languageservice/parser/kafkaFileParser.ts b/src/kafka-file/languageservice/parser/kafkaFileParser.ts index db3fc849..64895c94 100644 --- a/src/kafka-file/languageservice/parser/kafkaFileParser.ts +++ b/src/kafka-file/languageservice/parser/kafkaFileParser.ts @@ -11,8 +11,10 @@ export enum NodeKind { property, propertyKey, propertyAssigner, - propertyValue, - mustacheExpression + propertyValue, + mustacheExpression, + calleeFunction, + parameter } export interface Node { start: Position; @@ -178,7 +180,7 @@ export class Property extends BaseNode { return true; } - findNodeAt(position : Position) : Node { + findNodeAt(position: Position): Node { if (this.isBeforeAssigner(position)) { return this.key?.findNodeAt(position) || this; } @@ -242,6 +244,34 @@ export class DynamicChunk extends ChildrenNode { } +export class Parameter extends Chunk { + name?: string; + + public get value() : string { + return this.content?.trim(); + } + +} + +export class CalleeFunction extends ChildrenNode { + startParametersCharacter?: number; + endParametersCharacter?: number; + + constructor(public readonly content: string, start: Position, end: Position) { + super(start, end, NodeKind.calleeFunction); + parseParameters(this); + } + + public get functionName() : string { + return this.startParametersCharacter ? this.content.substring(0, this.startParametersCharacter - this.start.character).trim() : this.content.trim(); + } + + public get parameters(): Array { + return this.children; + } + +} + /** * Mustache expression AST (ex : {{random.words}}) */ @@ -459,11 +489,18 @@ function createProperty(lineText: string, lineNumber: number, parent: Block): Pr const content = lineText.substr(start, end); if (withinValue) { const propertyName = propertyKey?.content.trim(); - if (propertyName === 'key') { - propertyValue = new DynamicChunk(content, new Position(lineNumber, start), new Position(lineNumber, end), NodeKind.propertyValue); - } else { - propertyValue = new Chunk(content, new Position(lineNumber, start), new Position(lineNumber, end), NodeKind.propertyValue); - } + switch (propertyName) { + case "key": + propertyValue = new DynamicChunk(content, new Position(lineNumber, start), new Position(lineNumber, end), NodeKind.propertyValue); + break; + case "key-format": + case "value-format": + propertyValue = new CalleeFunction(content, new Position(lineNumber, start), new Position(lineNumber, end)); + break; + default: + propertyValue = new Chunk(content, new Position(lineNumber, start), new Position(lineNumber, end), NodeKind.propertyValue); + break; + } } else { propertyKey = new Chunk(content, new Position(lineNumber, start), new Position(lineNumber, end), NodeKind.propertyKey); } @@ -483,6 +520,66 @@ export class ExpressionEdge { } } +function parseParameters(parent: CalleeFunction) { + const content = parent.content; + let startLine = parent.start.line; + let startColumn = parent.start.character; + let currentLine = startLine; + let currentColumn = startColumn; + let previousChar: string | undefined; + let startParameter: number | undefined; + + function addParameterIfNeeded() { + if (startParameter) { + const value = content.substring(startParameter - startColumn, currentColumn - startColumn); + const start = new Position(currentLine, startParameter); + const end = new Position(currentLine, currentColumn); + parent.addChild(new Parameter(value, start, end, NodeKind.parameter)); + } + } + + for (let currentOffset = 0; currentOffset < content.length; currentOffset++) { + const currentChar = content[currentOffset]; + switch (currentChar) { + case '\r': + // compute line, column position + currentLine++; + currentColumn = 0; + break; + case '\n': { + if (previousChar !== '\r') { + // compute line, column position + currentLine++; + currentColumn = 0; + } + break; + } + case '(': { + if (!parent.startParametersCharacter) { + parent.startParametersCharacter = currentColumn; + startParameter = currentColumn + 1; + } + break; + } + case ')': { + parent.endParametersCharacter = currentColumn; + addParameterIfNeeded(); + startParameter = undefined; + break; + } + case ',': { + addParameterIfNeeded(); + startParameter = currentColumn + 1; + break; + } + } + if (currentChar !== '\r' && currentChar !== '\n') { + currentColumn++; + } + } + addParameterIfNeeded(); +} + function parseMustacheExpressions(parent: DynamicChunk) { const content = parent.content; let startLine = parent.start.line; @@ -532,7 +629,7 @@ function parseMustacheExpressions(parent: DynamicChunk) { } // 2. create mustache expression AST by visiting collected edges - + let previousEdge = new ExpressionEdge(new Position(startLine, startColumn), 0, true); const endOfValueEdge = new ExpressionEdge(new Position(currentLine, currentColumn), currentOffset, false); for (let i = 0; i < edges.length; i++) { @@ -548,7 +645,7 @@ function parseMustacheExpressions(parent: DynamicChunk) { const openedEdge = currentEdge; let closedEdge = endOfValueEdge; - let closed = false; + let closed = false; if (matchingClosedEdge) { // '}}' has been found closed = true; diff --git a/src/kafka-file/languageservice/services/codeLensProvider.ts b/src/kafka-file/languageservice/services/codeLensProvider.ts index 54633634..70db451c 100644 --- a/src/kafka-file/languageservice/services/codeLensProvider.ts +++ b/src/kafka-file/languageservice/services/codeLensProvider.ts @@ -1,9 +1,10 @@ import { TextDocument, CodeLens, Range } from "vscode"; import { ClientState, ConsumerLaunchState } from "../../../client"; import { createProducerUri, ProducerLaunchState } from "../../../client/producer"; +import { SerializationSetting } from "../../../client/serialization"; import { LaunchConsumerCommand, ProduceRecordCommand, ProduceRecordCommandHandler, SelectClusterCommandHandler, StartConsumerCommandHandler, StopConsumerCommandHandler } from "../../../commands"; import { ProducerLaunchStateProvider, ConsumerLaunchStateProvider, SelectedClusterProvider } from "../kafkaFileLanguageService"; -import { Block, BlockType, ConsumerBlock, KafkaFileDocument, ProducerBlock } from "../parser/kafkaFileParser"; +import { Block, BlockType, ConsumerBlock, KafkaFileDocument, CalleeFunction, ProducerBlock } from "../parser/kafkaFileParser"; /** * Kafka file codeLens support. @@ -43,7 +44,7 @@ export class KafkaFileCodeLenses { getClusterStatus(state: ClientState | undefined) { switch (state) { case ClientState.disconnected: - return `$(eye-closed) `; + return `$(eye-closed) `; case ClientState.connecting: return `$(sync~spin) `; case ClientState.connected: @@ -110,7 +111,9 @@ export class KafkaFileCodeLenses { let key; let value = block.value?.content; let keyFormat; + let keyFormatSettings: Array | undefined; let valueFormat; + let valueFormatSettings: Array | undefined; block.properties.forEach(property => { switch (property.propertyName) { case 'topic': @@ -119,12 +122,18 @@ export class KafkaFileCodeLenses { case 'key': key = property.propertyValue; break; - case 'key-format': - keyFormat = property.propertyValue; + case 'key-format': { + const callee = property.value; + keyFormat = callee.functionName; + keyFormatSettings = this.getSerializationSettings(callee); break; - case 'value-format': - valueFormat = property.propertyValue; + } + case 'value-format': { + const callee = property.value; + valueFormat = callee.functionName; + valueFormatSettings = this.getSerializationSettings(callee); break; + } } }); return { @@ -133,10 +142,19 @@ export class KafkaFileCodeLenses { key, value, messageKeyFormat: keyFormat, - messageValueFormat: valueFormat + messageKeyFormatSettings: keyFormatSettings, + messageValueFormat: valueFormat, + messageValueFormatSettings: valueFormatSettings } as ProduceRecordCommand; } + private getSerializationSettings(callee: CalleeFunction): SerializationSetting[] | undefined { + const parameters = callee.parameters; + if (parameters.length > 0) { + return parameters.map(p => { return { value: p.value }; }); + } + } + private createConsumerLens(block: ConsumerBlock, lineRange: Range, range: Range, clusterName: string | undefined, clusterId: string | undefined, clusterState: ClientState | undefined): CodeLens[] { const launchCommand = this.createLaunchConsumerCommand(block, range, clusterId); const lenses: CodeLens[] = []; @@ -193,7 +211,9 @@ export class KafkaFileCodeLenses { let partitions; let offset; let keyFormat; + let keyFormatSettings; let valueFormat; + let valueFormatSettings; block.properties.forEach(property => { switch (property.propertyName) { case 'topic': @@ -205,12 +225,18 @@ export class KafkaFileCodeLenses { case 'partitions': partitions = property.propertyValue; break; - case 'key-format': - keyFormat = property.propertyValue; + case 'key-format': { + const callee = property.value; + keyFormat = callee.functionName; + keyFormatSettings = this.getSerializationSettings(callee); break; - case 'value-format': - valueFormat = property.propertyValue; + } + case 'value-format': { + const callee = property.value; + valueFormat = callee.functionName; + valueFormatSettings = this.getSerializationSettings(callee); break; + } } }); @@ -221,7 +247,9 @@ export class KafkaFileCodeLenses { fromOffset: offset, partitions, messageKeyFormat: keyFormat, - messageValueFormat: valueFormat + messageValueFormat: valueFormat, + messageKeyFormatSettings: keyFormatSettings, + messageValueFormatSettings: valueFormatSettings } as LaunchConsumerCommand; } } diff --git a/src/kafka-file/languageservice/services/completion.ts b/src/kafka-file/languageservice/services/completion.ts index 36aec62a..177a3923 100644 --- a/src/kafka-file/languageservice/services/completion.ts +++ b/src/kafka-file/languageservice/services/completion.ts @@ -1,7 +1,14 @@ import { TextDocument, Position, CompletionList, CompletionItem, SnippetString, MarkdownString, CompletionItemKind, Range } from "vscode"; import { createTopicDocumentation, SelectedClusterProvider, TopicProvider } from "../kafkaFileLanguageService"; import { consumerModel, fakerjsAPIModel, Model, ModelDefinition, producerModel } from "../model"; -import { Block, BlockType, Chunk, ConsumerBlock, KafkaFileDocument, MustacheExpression, NodeKind, ProducerBlock, Property } from "../parser/kafkaFileParser"; +import { Block, BlockType, Chunk, ConsumerBlock, KafkaFileDocument, CalleeFunction, MustacheExpression, NodeKind, Parameter, ProducerBlock, Property } from "../parser/kafkaFileParser"; + +/** + * Supported encoding by nodejs Buffer. + * + * @see https://nodejs.org/api/buffer.html#buffer_buffers_and_character_encodings + */ +const bufferEncoding = ["utf8", "utf16le", "base64", "latin1", "hex"]; /** * Kafka file completion support. @@ -18,7 +25,7 @@ export class KafkaFileCompletion { return; } - // Following comments with use the '|' character to show the position where the complation is trigerred + // Following comments with use the '|' character to show the position where the compilation is triggered const items: Array = []; switch (node.kind) { case NodeKind.consumerBlock: { @@ -80,23 +87,53 @@ export class KafkaFileCompletion { } } else { const propertyValue = property.value; - const expression = propertyValue?.findNodeBefore(position); - if (expression && expression.kind === NodeKind.mustacheExpression) { - // Completion was triggered inside a mustache expression which is inside the property value + // Property value can be: + // - a simple value -> abcd + // - a mustache expression -> {{...}} + // - a method parameter -> string(utf-8) + const previousNode = propertyValue?.findNodeBefore(position); + switch (previousNode?.kind) { + case NodeKind.mustacheExpression: { + // Completion was triggered inside a mustache expression which is inside the property value + + // PRODUCER + // key: abcd-{{|}} + const expression = previousNode; + this.collectFakerJSExpressions(expression, producerFakerJSEnabled, position, items); + break; + } + case NodeKind.calleeFunction: { + // Check if completion was triggered inside an empty method parameter + + const callee = previousNode; + if (callee.startParametersCharacter) { + // PRODUCER + // key-format: string(|) + this.collectMethodParameters(callee, position, items); + } else { + // PRODUCER + // key-format: | + await this.collectDefaultPropertyValues(property, propertyValue, items); + } + break; + } + case NodeKind.parameter: { + // Completion was triggered inside a method parameter which is inside the property value + + // PRODUCER + // key-format: string(ut|f) + + // OR - // PRODUCER - // key: abcd-{{|}} - this.collectFakerJSExpressions(expression, producerFakerJSEnabled, position, items); - } else { - const block = property.parent; - if (block.type === BlockType.consumer) { - // CONSUMER - // key-format: | - await this.collectConsumerPropertyValues(propertyValue, property, block, items); - } else { // PRODUCER - // key-format: | - await this.collectProducerPropertyValues(propertyValue, property, block, items); + // key-format: string(utf8, u|t) + + const parameter = previousNode; + this.collectMethodParameters(parameter.parent, position, items); + break; + } + default: { + await this.collectDefaultPropertyValues(property, propertyValue, items); } } } @@ -205,6 +242,19 @@ export class KafkaFileCompletion { item.insertText = insertText; item.range = valueRange; items.push(item); + + if (value === 'string' && (propertyName === 'key-format' || propertyName === 'value-format')) { + const item = new CompletionItem('string with encoding...'); + item.kind = CompletionItemKind.Value; + const insertText = new SnippetString(' '); + insertText.appendText(value); + insertText.appendText('('); + insertText.appendChoice(bufferEncoding); + insertText.appendText(')'); + item.insertText = insertText; + item.range = valueRange; + items.push(item); + } }); } @@ -275,9 +325,40 @@ export class KafkaFileCompletion { }*/ } } + + collectMethodParameters(callee: CalleeFunction, position: Position, items: CompletionItem[]) { + const parameter = callee.parameters.find(p => position.isAfterOrEqual(p.start) && position.isBeforeOrEqual(p.end)); + if (!parameter) { + return; + } + switch (callee.functionName) { + case 'string': { + const range = parameter.range(); + bufferEncoding.forEach(encoding => { + const item = new CompletionItem(encoding); + item.kind = CompletionItemKind.EnumMember; + item.range = range; + items.push(item); + }); + } + } + } + + async collectDefaultPropertyValues(property: Property, propertyValue: Chunk | undefined, items: CompletionItem[]) { + const block = property.parent; + if (block.type === BlockType.consumer) { + // CONSUMER + // key-format: | + await this.collectConsumerPropertyValues(propertyValue, property, block, items); + } else { + // PRODUCER + // key-format: | + await this.collectProducerPropertyValues(propertyValue, property, block, items); + } + } } -function createMarkdownString(contents : string) { +function createMarkdownString(contents: string) { const doc = new MarkdownString(contents); doc.isTrusted = true; return doc; diff --git a/src/kafka-file/languageservice/services/diagnostics.ts b/src/kafka-file/languageservice/services/diagnostics.ts index 94e07152..4c62ee1b 100644 --- a/src/kafka-file/languageservice/services/diagnostics.ts +++ b/src/kafka-file/languageservice/services/diagnostics.ts @@ -1,5 +1,5 @@ import { Diagnostic, DiagnosticSeverity, Position, Range, TextDocument } from "vscode"; -import { Block, BlockType, ConsumerBlock, DynamicChunk, KafkaFileDocument, MustacheExpression, ProducerBlock, Property } from "../parser/kafkaFileParser"; +import { Block, BlockType, Chunk, ConsumerBlock, DynamicChunk, KafkaFileDocument, CalleeFunction, MustacheExpression, ProducerBlock, Property } from "../parser/kafkaFileParser"; import { ConsumerValidator } from "../../../validators/consumer"; import { ProducerValidator } from "../../../validators/producer"; import { CommonsValidator } from "../../../validators/commons"; @@ -236,7 +236,7 @@ export class KafkaFileDiagnostics { if (!range) { return; } - const errorMessage = await this.validateValue(propertyName, type, propertyValue); + const errorMessage = await this.validateValue(propertyName, type, property.value); if (errorMessage) { diagnostics.push(new Diagnostic(range, errorMessage, DiagnosticSeverity.Error)); } @@ -249,18 +249,20 @@ export class KafkaFileDiagnostics { } } - private async validateValue(propertyName: string, type: BlockType, propertyValue?: string): Promise { + private async validateValue(propertyName: string, type: BlockType, propertyValue?: Chunk): Promise { switch (propertyName) { case 'topic': - return CommonsValidator.validateTopic(propertyValue); + return CommonsValidator.validateTopic(propertyValue?.content.trim()); case 'key-format': - return type === BlockType.consumer ? ConsumerValidator.validateKeyFormat(propertyValue) : ProducerValidator.validateKeyFormat(propertyValue); + const keyFormat = (propertyValue).functionName; + return type === BlockType.consumer ? ConsumerValidator.validateKeyFormat(keyFormat) : ProducerValidator.validateKeyFormat(keyFormat); case 'value-format': - return type === BlockType.consumer ? ConsumerValidator.validateValueFormat(propertyValue) : ProducerValidator.validateValueFormat(propertyValue); + const valueFormat = (propertyValue).functionName; + return type === BlockType.consumer ? ConsumerValidator.validateValueFormat(valueFormat) : ProducerValidator.validateValueFormat(valueFormat); case 'from': - return ConsumerValidator.validateOffset(propertyValue); + return ConsumerValidator.validateOffset(propertyValue?.content.trim()); case 'partitions': { - return ConsumerValidator.validatePartitions(propertyValue); + return ConsumerValidator.validatePartitions(propertyValue?.content.trim()); } } } diff --git a/src/kafka-file/languageservice/services/hover.ts b/src/kafka-file/languageservice/services/hover.ts index 36218c71..25c2b5f4 100644 --- a/src/kafka-file/languageservice/services/hover.ts +++ b/src/kafka-file/languageservice/services/hover.ts @@ -2,7 +2,7 @@ import { Hover, MarkdownString, Position, Range, TextDocument } from "vscode"; import { getDocumentationPageUri } from "../../../docs/markdownPreviewProvider"; import { createTopicDocumentation, SelectedClusterProvider, TopicProvider } from "../kafkaFileLanguageService"; import { consumerModel, Model, producerModel } from "../model"; -import { Block, BlockType, Chunk, ConsumerBlock, KafkaFileDocument, MustacheExpression, NodeKind, ProducerBlock, Property } from "../parser/kafkaFileParser"; +import { Block, BlockType, CalleeFunction, Chunk, ConsumerBlock, KafkaFileDocument, MustacheExpression, NodeKind, ProducerBlock, Property } from "../parser/kafkaFileParser"; export class KafkaFileHover { @@ -52,19 +52,16 @@ export class KafkaFileHover { } } + case NodeKind.calleeFunction: { + const property = node.parent; + const propertyValue = (node).functionName; + return this.getHoverForPropertyValue(property, propertyValue); + } + case NodeKind.propertyValue: { - const propertyValue = node; - const property = propertyValue.parent; - const block = property.parent; - if (block.type === BlockType.consumer) { - // CONSUMER - // key-format: | - return await this.getHoverForConsumerPropertyValues(propertyValue, property, block); - } else { - // PRODUCER - // key-format: | - return await this.getHoverForProducerPropertyValues(propertyValue, property, block); - } + const property = node.parent; + const propertyValue = property.propertyValue; + return this.getHoverForPropertyValue(property, propertyValue); } case NodeKind.mustacheExpression: { @@ -93,7 +90,22 @@ export class KafkaFileHover { } } - async getHoverForConsumerPropertyValues(propertyValue: Chunk, property: Property, block: ConsumerBlock): Promise { + async getHoverForPropertyValue(property: Property, propertyValue?: string) { + if (!propertyValue) { + return; + } + const block = property.parent; + if (block.type === BlockType.consumer) { + // CONSUMER + // key-format: | + return await this.getHoverForConsumerPropertyValues(propertyValue, property, block); + } else { + // PRODUCER + // key-format: | + return await this.getHoverForProducerPropertyValues(propertyValue, property, block); + } + } + async getHoverForConsumerPropertyValues(propertyValue: string, property: Property, block: ConsumerBlock): Promise { const propertyName = property.propertyName; switch (propertyName) { case 'topic': @@ -108,7 +120,7 @@ export class KafkaFileHover { } - async getHoverForProducerPropertyValues(propertyValue: Chunk, property: Property, block: ProducerBlock): Promise { + async getHoverForProducerPropertyValues(propertyValue: string, property: Property, block: ProducerBlock): Promise { const propertyName = property.propertyName; switch (propertyName) { case 'topic': @@ -150,12 +162,12 @@ export class KafkaFileHover { return undefined; } - async getHoverForPropertyValues(propertyValue: Chunk, property: Property, block: Block, metadata: Model): Promise { + async getHoverForPropertyValues(propertyValue: string, property: Property, block: Block, metadata: Model): Promise { const propertyName = property.propertyName; if (!propertyName) { return; } - const definition = metadata.getDefinitionEnum(propertyName, propertyValue.content.trim()); + const definition = metadata.getDefinitionEnum(propertyName, propertyValue); if (definition && definition.description) { return createHover(definition.description, property.propertyTrimmedValueRange); } diff --git a/src/providers/consumerVirtualTextDocumentProvider.ts b/src/providers/consumerVirtualTextDocumentProvider.ts index e3efb224..498a9301 100644 --- a/src/providers/consumerVirtualTextDocumentProvider.ts +++ b/src/providers/consumerVirtualTextDocumentProvider.ts @@ -1,6 +1,7 @@ import * as vscode from "vscode"; import { ConsumedRecord, Consumer, ConsumerChangedStatusEvent, ConsumerCollection, ConsumerCollectionChangedEvent, ConsumerLaunchState, RecordReceivedEvent } from "../client"; +import { SerializationSetting } from "../client/serialization"; import { CommonMessages } from "../constants"; import { ClusterSettings } from "../settings/clusters"; @@ -86,15 +87,22 @@ export class ConsumerVirtualTextDocumentProvider implements vscode.TextDocumentC line += ` - partitions: ${consumer.options.partitions}\n`; } if (consumer.options.messageKeyFormat) { - line += ` - key format: ${consumer.options.messageKeyFormat}\n`; + line += ` - key format: ${consumer.options.messageKeyFormat}${this.getFormatSettings(consumer.options.messageKeyFormatSettings)}\n`; } if (consumer.options.messageValueFormat) { - line += ` - value format: ${consumer.options.messageValueFormat}\n`; + line += ` - value format: ${consumer.options.messageValueFormat}${this.getFormatSettings(consumer.options.messageValueFormatSettings)}\n`; } line += `\n`; this.updateBuffer(consumer.uri, line); } + getFormatSettings(messageKeyFormatSettings: SerializationSetting[] | undefined) { + if (!messageKeyFormatSettings) { + return ''; + } + return `(${messageKeyFormatSettings.map(s => s.value).join(',')})`; + } + onDidConsumerError(uri: vscode.Uri, error: any): void { if (!this.isActive(uri)) { return; diff --git a/src/test/suite/kafka-file/languageservice/codeLens.test.ts b/src/test/suite/kafka-file/languageservice/codeLens.test.ts index 88538c96..b96da7bf 100644 --- a/src/test/suite/kafka-file/languageservice/codeLens.test.ts +++ b/src/test/suite/kafka-file/languageservice/codeLens.test.ts @@ -1,4 +1,5 @@ import { ConsumerLaunchState } from "../../../../client"; +import { LaunchConsumerCommand, ProduceRecordCommand } from "../../../../commands"; import { getLanguageService } from "../../../../kafka-file/languageservice/kafkaFileLanguageService"; import { assertCodeLens, codeLens, LanguageServiceConfig, position } from "./kafkaAssert"; @@ -76,10 +77,12 @@ suite("Kafka File PRODUCER CodeLens Test Suite", () => { clusterId: 'cluster1', key: undefined, messageKeyFormat: undefined, + messageKeyFormatSettings: undefined, messageValueFormat: undefined, + messageValueFormatSettings: undefined, topicId: undefined, value: undefined - }, + } as ProduceRecordCommand, 1 ] }), @@ -91,10 +94,12 @@ suite("Kafka File PRODUCER CodeLens Test Suite", () => { clusterId: 'cluster1', key: undefined, messageKeyFormat: undefined, + messageKeyFormatSettings: undefined, messageValueFormat: undefined, + messageValueFormatSettings: undefined, topicId: undefined, value: undefined - }, + } as ProduceRecordCommand, 10 ] }), @@ -120,10 +125,12 @@ suite("Kafka File PRODUCER CodeLens Test Suite", () => { clusterId: 'cluster1', key: 'a-key', messageKeyFormat: 'long', + messageKeyFormatSettings: undefined, messageValueFormat: 'string', + messageValueFormatSettings: undefined, topicId: 'abcd', value: 'ABCD\nEFGH' - }, + } as ProduceRecordCommand, 1 ] }), @@ -135,10 +142,68 @@ suite("Kafka File PRODUCER CodeLens Test Suite", () => { clusterId: 'cluster1', key: 'a-key', messageKeyFormat: 'long', + messageKeyFormatSettings: undefined, messageValueFormat: 'string', + messageValueFormatSettings: undefined, topicId: 'abcd', value: 'ABCD\nEFGH' - }, + } as ProduceRecordCommand, + 10 + ] + }), + codeLens(position(0, 0), position(0, 0), { + command: 'vscode-kafka.explorer.selectcluster', + title: 'CLUSTER_1' + }) + ], languageService); + + }); + + test("PRODUCER with string encoding", async () => { + + const languageServiceConfig = new LanguageServiceConfig(); + languageServiceConfig.setSelectedCluster({ clusterId: 'cluster1', clusterName: 'CLUSTER_1' }); + const languageService = getLanguageService(languageServiceConfig, languageServiceConfig, languageServiceConfig, languageServiceConfig); + + await assertCodeLens( + 'PRODUCER\n' + + 'key: a-key\n' + + 'topic: abcd\n' + + 'key-format: string(base64)\n' + + 'value-format: string(ascii)\n' + + 'ABCD\n' + + 'EFGH', [ + codeLens(position(0, 0), position(0, 0), { + command: 'vscode-kafka.producer.produce', + title: '$(run) Produce record', + arguments: [ + { + clusterId: 'cluster1', + key: 'a-key', + messageKeyFormat: 'string', + messageKeyFormatSettings: [{ value: 'base64' }], + messageValueFormat: 'string', + messageValueFormatSettings: [{ value: 'ascii' }], + topicId: 'abcd', + value: 'ABCD\nEFGH' + } as ProduceRecordCommand, + 1 + ] + }), + codeLens(position(0, 0), position(0, 0), { + command: 'vscode-kafka.producer.produce', + title: '$(run-all) Produce record x 10', + arguments: [ + { + clusterId: 'cluster1', + key: 'a-key', + messageKeyFormat: 'string', + messageKeyFormatSettings: [{ value: 'base64' }], + messageValueFormat: 'string', + messageValueFormatSettings: [{ value: 'ascii' }], + topicId: 'abcd', + value: 'ABCD\nEFGH' + } as ProduceRecordCommand, 10 ] }), @@ -199,10 +264,12 @@ suite("Kafka File CONSUMER CodeLens Test Suite", () => { consumerGroupId: 'group-1', fromOffset: undefined, messageKeyFormat: undefined, + messageKeyFormatSettings: undefined, messageValueFormat: undefined, + messageValueFormatSettings: undefined, partitions: undefined, topicId: '' - } + } as LaunchConsumerCommand ] }), codeLens(position(0, 0), position(0, 0), { @@ -227,10 +294,12 @@ suite("Kafka File CONSUMER CodeLens Test Suite", () => { consumerGroupId: 'group-1', fromOffset: '10', messageKeyFormat: 'long', + messageKeyFormatSettings: undefined, messageValueFormat: 'string', + messageValueFormatSettings: undefined, partitions: '1,2,3', topicId: 'abcd' - } + } as LaunchConsumerCommand ] }), codeLens(position(0, 0), position(0, 0), { @@ -253,10 +322,12 @@ suite("Kafka File CONSUMER CodeLens Test Suite", () => { consumerGroupId: 'group-1', fromOffset: undefined, messageKeyFormat: undefined, + messageKeyFormatSettings: undefined, messageValueFormat: undefined, + messageValueFormatSettings: undefined, partitions: undefined, topicId: '' - } + } as LaunchConsumerCommand ] }), codeLens(position(0, 0), position(0, 0), { @@ -274,10 +345,12 @@ suite("Kafka File CONSUMER CodeLens Test Suite", () => { consumerGroupId: 'group-2', fromOffset: undefined, messageKeyFormat: undefined, + messageKeyFormatSettings: undefined, messageValueFormat: undefined, + messageValueFormatSettings: undefined, partitions: undefined, topicId: '' - } + } as LaunchConsumerCommand ] }), codeLens(position(1, 0), position(1, 0), { @@ -287,4 +360,43 @@ suite("Kafka File CONSUMER CodeLens Test Suite", () => { ], languageService); }); + test("CONSUMER with string encoding", async () => { + + const languageServiceConfig = new LanguageServiceConfig(); + languageServiceConfig.setSelectedCluster({ clusterId: 'cluster1', clusterName: 'CLUSTER_1' }); + languageServiceConfig.setConsumerLaunchState('cluster1', 'group-1', ConsumerLaunchState.started); + const languageService = getLanguageService(languageServiceConfig, languageServiceConfig, languageServiceConfig, languageServiceConfig); + + await assertCodeLens( + 'CONSUMER group-1\n' + + 'topic: abcd\n' + + 'from: 10\n' + + 'partitions: 1,2,3\n' + + 'key-format: string(base64)\n' + + 'value-format: string(ascii)\n', [ + codeLens(position(0, 0), position(0, 0), { + command: 'vscode-kafka.consumer.stop', + title: '$(debug-stop) Stop consumer', + arguments: [ + { + clusterId: 'cluster1', + consumerGroupId: 'group-1', + fromOffset: '10', + messageKeyFormat: 'string', + messageKeyFormatSettings: [{ value: 'base64' }], + messageValueFormat: 'string', + messageValueFormatSettings: [{ value: 'ascii' }], + partitions: '1,2,3', + topicId: 'abcd' + } as LaunchConsumerCommand + ] + }), + codeLens(position(0, 0), position(0, 0), { + command: 'vscode-kafka.explorer.selectcluster', + title: 'CLUSTER_1' + }) + ], languageService); + + }); + }); diff --git a/src/test/suite/kafka-file/languageservice/completionProperties.test.ts b/src/test/suite/kafka-file/languageservice/completionProperties.test.ts index 783c0cce..924f4914 100644 --- a/src/test/suite/kafka-file/languageservice/completionProperties.test.ts +++ b/src/test/suite/kafka-file/languageservice/completionProperties.test.ts @@ -392,6 +392,11 @@ suite("Kafka File CONSUMER Completion Test Suite", () => { insertText: ' string', range: range(position(1, 11), position(1, 11)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(1, 11), position(1, 11)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -437,6 +442,11 @@ suite("Kafka File CONSUMER Completion Test Suite", () => { insertText: ' string', range: range(position(1, 13), position(1, 13)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(1, 13), position(1, 13)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -466,6 +476,65 @@ suite("Kafka File CONSUMER Completion Test Suite", () => { }); }); + test("CONSUMER property value for string encoding of key-format", async () => { + await testCompletion( + 'CONSUMER a\n' + + 'key-format: string(|' + , { + items: [ + { + label: 'utf8', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'utf16le', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'base64', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'latin1', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'hex', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + } + ] + }); + }); + + test("CONSUMER property value for string encoding of value-format", async () => { + await testCompletion( + 'CONSUMER a\n' + + 'value-format: string(|' + , { + items: [ + { + label: 'utf8', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'utf16le', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'base64', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'latin1', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'hex', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + } + ] + }); + }); }); suite("Kafka File PRODUCER Completion Test Suite", () => { @@ -694,6 +763,11 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { insertText: ' string', range: range(position(1, 11), position(1, 11)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(1, 11), position(1, 11)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -734,6 +808,11 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { insertText: ' string', range: range(position(1, 11), position(1, 13)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(1, 11), position(1, 13)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -775,6 +854,11 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { insertText: ' string', range: range(position(2, 11), position(2, 13)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(2, 11), position(2, 13)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -817,6 +901,11 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { insertText: ' string', range: range(position(2, 11), position(2, 13)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(2, 11), position(2, 13)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -857,6 +946,11 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { insertText: ' string', range: range(position(1, 13), position(1, 13)) }, + { + label: 'string with encoding...', kind: CompletionItemKind.Value, + insertText: ' string(${1|utf8,utf16le,base64,latin1,hex|})', + range: range(position(1, 13), position(1, 13)) + }, { label: 'double', kind: CompletionItemKind.Value, insertText: ' double', @@ -886,4 +980,64 @@ suite("Kafka File PRODUCER Completion Test Suite", () => { }); }); + test("PRODUCER property value for string encoding of key-format", async () => { + await testCompletion( + 'PRODUCER a\n' + + 'key-format: string(|' + , { + items: [ + { + label: 'utf8', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'utf16le', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'base64', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'latin1', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + }, + { + label: 'hex', kind: CompletionItemKind.EnumMember, + range: range(position(1, 19), position(1, 19)) + } + ] + }); + }); + + test("PRODUCER property value for string encoding of value-format", async () => { + await testCompletion( + 'PRODUCER a\n' + + 'value-format: string(|' + , { + items: [ + { + label: 'utf8', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'utf16le', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'base64', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'latin1', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + }, + { + label: 'hex', kind: CompletionItemKind.EnumMember, + range: range(position(1, 21), position(1, 21)) + } + ] + }); + }); + }); diff --git a/src/test/suite/kafka-file/languageservice/hover.test.ts b/src/test/suite/kafka-file/languageservice/hover.test.ts index c78a4619..19b8326d 100644 --- a/src/test/suite/kafka-file/languageservice/hover.test.ts +++ b/src/test/suite/kafka-file/languageservice/hover.test.ts @@ -116,7 +116,7 @@ suite("Kafka File CONSUMER Hover Test Suite", () => { 'CONSUMER\n' + 'key-format: stri|ng', hover( - 'Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java) which currently only supports `UTF-8` encoding.', + 'Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java).', position(1, 12), position(1, 18) ) @@ -144,7 +144,7 @@ suite("Kafka File CONSUMER Hover Test Suite", () => { 'CONSUMER\n' + 'value-format: stri|ng', hover( - 'Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java) which currently only supports `UTF-8` encoding.', + 'Similar deserializer to the Kafka Java client [org.apache.kafka.common.serialization.StringDeserializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java).', position(1, 14), position(1, 20) ) @@ -270,7 +270,7 @@ suite("Kafka File PRODUCER Hover Test Suite", () => { 'PRODUCER\n' + 'key-format: stri|ng', hover( - 'Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java) which currently only supports `UTF-8` encoding.', + 'Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java).', position(1, 12), position(1, 18) ) @@ -298,7 +298,7 @@ suite("Kafka File PRODUCER Hover Test Suite", () => { 'PRODUCER\n' + 'value-format: stri|ng', hover( - 'Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java) which currently only supports `UTF-8` encoding.', + 'Similar serializer to the Kafka Java client [org.apache.kafka.common.serialization.StringSerializer](https://github.com/apache/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java).', position(1, 14), position(1, 20) )