Create-only actions in OpenSearch bulk requests

[dlvenable]

Is your feature request related to a problem? Please describe.

Data Prepper only uses the index action of the Bulk API. In some cases, teams want to create only if the document does not already exist. The OpenSearch _bulk API does support this via a create action.

Describe the solution you'd like

Create a new action property of the OpenSearch sink. The default value will be index (this is the current behavior). Pipeline authors can define this value to be either create or index.

Data Prepper will send bulk requests with all events using the configured action.

If OpenSearch responds that the document could not be created because it exists, this document will be dropped without any retry or DLQ. This is a reasonable expectation for pipeline authors who use create since their goals are to avoid updating existing documents.

Describe alternatives you've considered (Optional)

Data Prepper could key off of values directly in the document to determine if create or index should be used. This would be somewhat more complicated since pipeline authors would need to define conditions and fallbacks. This approach could be added later if needed. But, the proposed solution should be simpler for pipeline authors.

Additional context

In addition to index and create, the Bulk API also supports update and delete. The proposal above would allow us to add these in future iterations.

[jzonthemtn]

@dlvenable Happy to take on this one if it's ready to be worked.

[dlvenable]

@jzonthemtn , Yes, this one is ready to work when you are available. I'll assign it to you. Thanks!

[jzonthemtn]

Hi @dlvenable, I'm working on integration tests for this. It looks like a running OpenSearch is required for the OpenSearchSinkIT tests. The OpenSearchIntegrationHelper gets the host names via the tests.opensearch.host system property. I just wanted to verify my understanding here is correct before getting too deep into it.

[dlvenable]

Thanks @jzonthemtn ! Yes, you would need to run OpenSearch. You can do this via Docker.

I normally run them using the GitHub Actions as a template.

Start Docker:

data-prepper/.github/workflows/opensearch-sink-os-integration-tests.yml

Lines 26 to 27 in d737daf

	docker pull opensearchproject/opensearch:${{ matrix.opensearch }}
	docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -d opensearchproject/opensearch:${{ matrix.opensearch }}

Run the tests:

data-prepper/.github/workflows/opensearch-sink-os-integration-tests.yml

Lines 31 to 32 in d737daf

	./gradlew :data-prepper-plugins:opensearch:integrationTest --tests "com.amazon.dataprepper.plugins.sink.opensearch.OpenSearchIT" -Dtests.opensearch.host=localhost:9200 -Dtests.opensearch.user=admin -Dtests.opensearch.password=admin
	./gradlew :data-prepper-plugins:opensearch:integrationTest -Dtests.opensearch.host=localhost:9200 -Dtests.opensearch.user=admin -Dtests.opensearch.password=admin -Dtests.opensearch.bundle=true

[dlvenable]

This was resolved in #1561 .