[BUG] Tests against OpenSearch main are failing

[VachaShah]

What is the bug?

Tests against OpenSearch main branch are failing in the CI. https://github.com/opensearch-project/opensearch-go/actions/runs/3114891404/jobs/5051224763

How can one reproduce the bug?

Run the integration-unreleased workflow against OpenSearch main.

What is the expected behavior?

Tests should pass successfully.

What is your host/environment?

Operating system, version.

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[VachaShah]

TestBulkIndexerIntegration test with body compression fails when testing against OpenSearch main:

FAIL: TestBulkIndexerIntegration/With_body_compression (9.46s)
        --- FAIL: TestBulkIndexerIntegration/With_body_compression/Default (9.25s)
            bulk_indexer_integration_test.go:114: Unexpected NumIndexed: want=100000, got=0
            bulk_indexer_integration_test.go:118: Unexpected NumFailed: want=0, got=100000
            bulk_indexer_integration_test.go:122: Unexpected countSuccessful: want=100000, got=0
        --- FAIL: TestBulkIndexerIntegration/With_body_compression/Multiple_indices (0.21s)
            bulk_indexer_integration_test.go:188: Unexpected NumIndexed: want=30, got=0

Stacktrace from OpenSearch end:

{"type": "server", "timestamp": "2022-10-20T22:44:46,490Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "docker-cluster", "node.name": "8e1ce1e71492", "message": "path: /test-bulk-integration, params: {wait_for_active_shards=1, index=test-bulk-integration}", "cluster.uuid": "3bPso61HQZWlstl9afmOZg", "node.id": "jZN3cDgWQWGNTA6E4eQpcg" , 
"stacktrace": ["org.opensearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes",
"at org.opensearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:71) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:144) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.rest.action.admin.indices.RestCreateIndexAction.prepareRequest(RestCreateIndexAction.java:79) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:103) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.rest.RestController.dispatchRequest(RestController.java:312) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.rest.RestController.dispatchRequest(RestController.java:241) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",
"at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]",

[VachaShah]

Another test failure due to compression

--- FAIL: TestTransportCompression (0.01s)
    opensearchtransport_integration_test.go:162: Unexpected StatusCode, expected 201, got: 400

[VachaShah]

For TestTransportCompression, PUT and POST requests are failing with 400 bad request (against OpenSearch main):

&{PUT /shiny_new_index HTTP/1.1 1 1 map[] <nil> <nil> 0 [] false  map[] map[] <nil> map[]   <nil> <nil> <nil> 0xc0000160c0}
&{400 Bad Request 400 HTTP/1.1 1 1 map[Content-Type:[application/json; charset=UTF-8]] 0xc00000f680 -1 [] false true map[] 0xc0001cbb00 <nil>}

&{GET /shiny_new_index HTTP/1.1 1 1 map[] <nil> <nil> 0 [] false  map[] map[] <nil> map[]   <nil> <nil> <nil> 0xc0000160c0}
&{200 OK 200 HTTP/1.1 1 1 map[Content-Type:[application/json; charset=UTF-8]] 0xc00000f8c0 -1 [] false true map[] 0xc00014df00 <nil>}

&{POST /shiny_new_index/_doc HTTP/1.1 1 1 map[Content-Type:[application/json]] {0xc00000f9a0} 0x856a00 19 [] false  map[] map[] <nil> map[]   <nil> <nil> <nil> 0xc0000160c0}
&{400 Bad Request 400 HTTP/1.1 1 1 map[Content-Type:[application/json; charset=UTF-8]] 0xc0002e68c0 -1 [] false true map[] 0xc000442100 <nil>}

Against OpenSearch 1.x and 2.x, the requests succeed.

[VachaShah]

Upon picking the commits from OpenSearch main, looks like opensearch-project/OpenSearch#3847 is the change from which the tests have started failing. Looking into why this is the case.

[VachaShah]

@reta Any suggestions on how this can be fixed?

This is affecting only for requests when compression is enabled. Doesn't look like this is a breaking change from OpenSearch's end but I am not sure how the go client is breaking because of this.

[reta]

@VachaShah thanks a lot for bringing this up, looking into it

[VachaShah]

@VachaShah thanks a lot for bringing this up, looking into it

Thank you!

[reta]

@VachaShah ah, found the issue, the fix on the way ... sorry about that ...

[VachaShah]

@VachaShah ah, found the issue, the fix on the way ... sorry about that ...

Thats awesome! Thank you @reta :)

[reta]

@VachaShah the fix is in, could you please rerun the failing action(s), I don't have permissions sadly to do that, thank you

[VachaShah]

@VachaShah the fix is in, could you please rerun the failing action(s), I don't have permissions sadly to do that, thank you

Thank you @reta! I have rerun it for PR #142, will rerun for others as well.

[VachaShah]

The tests are passing now, thank you @reta for the fix!

Closing this now.