All Tasks fail

[yoyoraso]

Hi, I have set up the connector with the following configuration :

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: my-source-connector
  namespace: kafka
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: io.aiven.kafka.connect.opensearch.OpensearchSinkConnector
  tasksMax: 10
  config:
    name: my-source-connector
    topics: os-topic
    connection.url: http://MY_OC_INGEST_URL:9200  
    connection.username: *****
    connection.password: *******
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter: org.apache.kafka.connect.storage.StringConverter
    tasks.max: 1
    schema.ignore: true
    key.ignore: true
    value.converter.schemas.enable: false 

but I get this error in all tesks :

org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception. at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:632) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:350) at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:250) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:219) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259) at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to check index os-topic exists after total of 6 attempt(s) at io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:144) at io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:105) at io.aiven.kafka.connect.opensearch.OpensearchClient.withRetry(OpensearchClient.java:339) at io.aiven.kafka.connect.opensearch.OpensearchClient.indexOrDataStreamExists(OpensearchClient.java:133) at io.aiven.kafka.connect.opensearch.OpensearchSinkTask.ensureIndexOrDataStreamExists(OpensearchSinkTask.java:156) at io.aiven.kafka.connect.opensearch.OpensearchSinkTask.tryWriteRecord(OpensearchSinkTask.java:132) at io.aiven.kafka.connect.opensearch.OpensearchSinkTask.put(OpensearchSinkTask.java:122) at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:601) ... 11 more Caused by: org.apache.http.ConnectionClosedException: Connection is closed at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:941) at org.opensearch.client.RestClient.performRequest(RestClient.java:332) at org.opensearch.client.RestClient.performRequest(RestClient.java:320) at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918) at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1901) at org.opensearch.client.IndicesClient.exists(IndicesClient.java:920) at io.aiven.kafka.connect.opensearch.OpensearchClient.lambda$indexOrDataStreamExists$1(OpensearchClient.java:135) at io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:119) ... 18 more Caused by: org.apache.http.ConnectionClosedException: Connection is closed at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:356) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ... 1 more
 
I am not sure what is wrong or what should I do

[yoyoraso]

this issue was because opensearch is configured to use https for 9200 by default so connector failed to connect using http but I still have issue as now opensearch have privet certificate and the connector can't verify the certificate so it fails

[muralibasani]

Can you try with setting in the config ssl_certificate_verification: false

[yoyoraso]

hello @muralibasani it still the same, I don't think ssl_certificate_verification is a vaild attribute in configration as I don't see it handled in the code