Add option to use auto-generated IDs on indexing

[srishti-saraswat]

Problem

Original PR - #679
PR to expose this on cloud - https://github.com/confluentinc/kafka-connect-elasticsearch-private/pull/44

Solution

Does this solution apply anywhere else?

• yes
• no

If yes, where?

Test Strategy

Tested on docker playround.
Case1: when use.autogenerated.ids is false
connector config -

{
     "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
     "tasks.max": "1",
     "topics": "test-elasticsearch-sink-testing1",
     "key.ignore": "true",
     "connection.url": "**************",
     "connection.username": "*******",
     "connection.password": "******",
     "type.name": "kafka-connect",
     "data.stream.namespace": "ssaraswat${topic}",
     "use.autogenerated.ids": "false"
}

result -

17:34:38 ℹ️ Check that the data is available in Elasticsearch Cloud
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2625  100  2625    0     0   2843      0 --:--:-- --:--:-- --:--:--  2840
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test-elasticsearch-sink-testing1",
        "_type" : "_doc",
        "_id" : "test-elasticsearch-sink-testing1+0+0",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value1"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing1",
        "_type" : "_doc",
        "_id" : "test-elasticsearch-sink-testing1+0+6",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value7"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing1",
        "_type" : "_doc",
        "_id" : "test-elasticsearch-sink-testing1+0+7",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value8"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing1",
        "_type" : "_doc",
        "_id" : "test-elasticsearch-sink-testing1+0+8",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value9"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing1",
        "_type" : "_doc",
        "_id" : "test-elasticsearch-sink-testing1+0+9",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value10"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing1",
        "_type" : "_doc",
        "_id" : "test-elasticsearch-sink-testing1+0+1",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value2"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing1",
        "_type" : "_doc",
        "_id" : "test-elasticsearch-sink-testing1+0+2",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value3"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing1",
        "_type" : "_doc",
        "_id" : "test-elasticsearch-sink-testing1+0+3",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value4"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing1",
        "_type" : "_doc",
        "_id" : "test-elasticsearch-sink-testing1+0+4",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value5"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing1",
        "_type" : "_doc",
        "_id" : "test-elasticsearch-sink-testing1+0+5",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value6"
        }
      }
    ]
  }
}
          "f1" : "value1"
          "f1" : "value10"
          "f1" : "value10"
17:34:39 ℹ️ ####################################################
17:34:39 ℹ️ ✅ RESULT: SUCCESS for elasticsearch-cloud-sink.sh (took: 2min 14sec - )
17:34:39 ℹ️ ####################################################

17:34:43 ℹ️ 🧩 Displaying status for 🌎onprem connector elasticsearch-cloud-sink
Name                           Status       Tasks                                                        Stack Trace                                       
-------------------------------------------------------------------------------------------------------------
elasticsearch-cloud-sink       ✅ RUNNING  0:🟢 RUNNING[connect]        -                                                 
-------------------------------------------------------------------------------------------------------------

Case 2: when use.autogenerated.ids is true
connector config -

{
     "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
     "tasks.max": "1",
     "topics": "test-elasticsearch-sink-testing2",
     "key.ignore": "true",
     "connection.url": "**************",
     "connection.username": "*******",
     "connection.password": "******",
     "type.name": "kafka-connect",
     "data.stream.namespace": "ssaraswat${topic}",
     "use.autogenerated.ids": "true"
}

result -

17:38:21 ℹ️ Check that the data is available in Elasticsearch Cloud
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2465  100  2465    0     0   2285      0  0:00:01  0:00:01 --:--:--  2286
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test-elasticsearch-sink-testing2",
        "_type" : "_doc",
        "_id" : "joEgtJIBFHOU-SZarWdj",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value1"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing2",
        "_type" : "_doc",
        "_id" : "j4EgtJIBFHOU-SZar2fH",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value2"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing2",
        "_type" : "_doc",
        "_id" : "kIEgtJIBFHOU-SZar2fH",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value3"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing2",
        "_type" : "_doc",
        "_id" : "kYEgtJIBFHOU-SZar2fH",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value4"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing2",
        "_type" : "_doc",
        "_id" : "koEgtJIBFHOU-SZar2fH",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value5"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing2",
        "_type" : "_doc",
        "_id" : "k4EgtJIBFHOU-SZar2fH",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value6"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing2",
        "_type" : "_doc",
        "_id" : "lIEgtJIBFHOU-SZar2fH",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value7"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing2",
        "_type" : "_doc",
        "_id" : "lYEgtJIBFHOU-SZar2fH",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value8"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing2",
        "_type" : "_doc",
        "_id" : "loEgtJIBFHOU-SZar2fH",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value9"
        }
      },
      {
        "_index" : "test-elasticsearch-sink-testing2",
        "_type" : "_doc",
        "_id" : "l4EgtJIBFHOU-SZar2fH",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value10"
        }
      }
    ]
  }
}
          "f1" : "value1"
          "f1" : "value10"
          "f1" : "value10"
17:38:22 ℹ️ ####################################################
17:38:22 ℹ️ ✅ RESULT: SUCCESS for elasticsearch-cloud-sink.sh (took: 2min 19sec - )
17:38:22 ℹ️ ####################################################

17:38:28 ℹ️ 🧩 Displaying status for 🌎onprem connector elasticsearch-cloud-sink
Name                           Status       Tasks                                                        Stack Trace                                       
-------------------------------------------------------------------------------------------------------------
elasticsearch-cloud-sink       ✅ RUNNING  0:🟢 RUNNING[connect]        -                                                 
-------------------------------------------------------------------------------------------------------------

Case 3: when use.autogenerated.ids is true but key.ignore is false
connector config -

{
     "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
     "tasks.max": "1",
     "topics": "test-elasticsearch-sink-testing6",
     "key.ignore": "false",
     "connection.url": "**************",
     "connection.username": "*******",
     "connection.password": "******",
     "type.name": "kafka-connect",
     "data.stream.namespace": "ssaraswat${topic}",
     "use.autogenerated.ids": "true"
}

result -

      {
        "_index" : "test-elasticsearch-sink-testing6",
        "_type" : "_doc",
        "_id" : "U4Htt5IBFHOU-SZax6Qn",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value10"
        }
      }
    ]
  }
}
          "f1" : "value1"
          "f1" : "value10"
          "f1" : "value10"
11:21:14 ℹ️ ####################################################
11:21:14 ℹ️ ✅ RESULT: SUCCESS for elasticsearch-cloud-sink.sh (took: 1min 45sec - )
11:21:14 ℹ️ ####################################################

11:21:18 ℹ️ 🧩 Displaying status for 🌎onprem connector elasticsearch-cloud-sink
Name                           Status       Tasks                                                        Stack Trace                                       
-------------------------------------------------------------------------------------------------------------
elasticsearch-cloud-sink       ✅ RUNNING  0:🟢 RUNNING[connect]        -                                                 
-------------------------------------------------------------------------------------------------------------

Testing done:

• Unit tests
• Integration tests
• System tests
• Manual tests

Release Plan

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[sonarqube-confluent]

Analysis Details

1 Issue

• 0 Bugs
• 0 Vulnerabilities
• 1 Code Smell

Coverage and Duplications

• 81.00% Coverage (58.50% Estimated after merge)
• No duplication information (1.20% Estimated after merge)

Project ID: kafka-connect-elasticsearch

View in SonarQube