Adding index.name.override config #193
Conversation
| @@ -88,6 +88,13 @@ Data Conversion | |||
| * Valid Values: ``insert``, ``upsert`` | |||
| * Importance: low | |||
|
|
|||
| ``index.name.override`` | |||
There was a problem hiding this comment.
@porto88 I feel like this is not the right way to write into dedicated index. First, there are many topics (and consequently) many different schemas which would not fit the same index. Secondly, you could always configure index aliases on OpenSearch side for whatever topics you would like to funnel into the same index.
There was a problem hiding this comment.
@reta I feel this is still a nice feature too have even in the case of just one kafka topic. If you are pulling from kafka topic "A" but want to insert into an index that is named differently, this will allow that.
There was a problem hiding this comment.
@porto88 the connector is not dealing with just one topic, the risk of misusing this feature (and unintended consequences of moving from one topic to many with this property being set) greatly outperform the benefits (in my opinion).
There was a problem hiding this comment.
this could be a mapping between topic and index name, similar to the Snowflake snowflaketopic2table.map that can be set to "snowflake.topic2table.map": "topic1:table1,topic2:table2"
There was a problem hiding this comment.
The same thing can be achieved with the RegexRouter smt while still supporting the multiple topics case @reta mentioned.
New Feature:
Adding new connector configuration index.name.override which allows for defining a static index name that documents will be written to.
Current Behavior:
Current logic for the index name is based off of the incoming kafka topic or data stream topic.
Problem this Solves:
There currently isn't a way to change the topic with connector configuration. The only solution is to use the kafka connect transformer "org.apache.kafka.connect.transforms.RegexRouter" to remap the incoming topic(s).
The problem with this solution is when using the key.ignore.id.strategy with value of "topic.partition.offset", the topic in this key is always mapped to the static value defined by the RegexRouter, and will cause document collisions.
Proposed Solution:
Add new configuration index.name.override which allows for defining a static index name that documents will be written to regardless of source topic(s).
Existing Issues:
#89