ElasticSearch connector with date in index and custom document type

[robvadai]

The ElasticSearch connector is updated with:

• support for auto-generated time/date in index names; this is to create indexes per day instead of one ever-increasing index, as per the official guidelines on Retiring Data
• support for custom document type for the messages to be saved; document types allow custom indexing policies

[stheppi]

@robvadai
Shouldn't we add support for this in KCQL? The reason i am asking is for scenarios where you route messages from topic1 to index1 and topic2 to index2 and maybe you want one of the index to have suffix. Furthermore you might use different document type for messages coming from two different topics.
Thoughts?

[stheppi]

KCQL example: INSERT INTO $THE_INDEX SELECT * FROM $THE_TOPIC [WITHDOCTYPE($docType)] [WITHINDEXSUFFIX($suffix)] [AUTOCREATE]

Autocreate - already exists in the KCQL grammar

[robvadai]

1. Yes this could be part of KCQL, good idea, obviously updating KCQL was out of scope for me, we needed a way to support date string in ES index names
2. We had a requirement to make sure indexes are never auto created because we create indexes with custom policies and document types. The Connector was automatically creating missing indexes which is what we wanted to avoid. Hence the conditional index creation was added here: https://github.com/ConnectedHomes/stream-reactor/blob/9bc01bd61034037cacb14f5154bbfa01ae8e4b47/kafka-connect-elastic/src/main/scala/com/datamountaineer/streamreactor/connect/elastic/ElasticJsonWriter.scala#L42

[stheppi]

should probably use null and avoid toOption over an empty string

[stheppi]

@robvadai thank you for the improvement

[joscas]

👍