[BUG] Create Snapshot Repository (Attached to S3 Bucket) Fails, when HTTP Compression Enabled

[omamoo]

What is the bug?

While trying to create Elasticsearch repository with client that HTTP compression enabled, the HTTP requests fails with 401 response, saying that the authentication failed.

How can one reproduce the bug?

1. Create newest Opensearch on AWS.
2. Initialize client with HTTP compressions enabled.
3. With the above client, create snapshot repository attached to S3

What is the expected behavior?

Snapshot repository created, HTTP status 200 (OK)

What is your host/environment?

Windows 11 - Client
Opensearch Version: 2.3 (latest)
Opensearch Service Software Version: R20221114-P1

Do you have any screenshots?

2022-12-11 16:00:11.0202 ERR CreateElasticsearchDomainSnapshotRepositoryCmdlet RunAsync failed {DebugInformation="Invalid OpenSearch.Client response built from a unsuccessful (401) low level call on PUT: /_snapshot/devuse2dep2es1snapshotrepository?verify=false\r\n# Audit trail of this API call:\r\n - [1] BadResponse: Node: https://vpc-devuse2dep2es1-yrdn4ehgxv26lwnfq7kft6kz5e.us-east-2.es.amazonaws.com/ Exception: PipelineException Took: 00:00:00.2514008\r\n# OriginalException: OpenSearch.Net.OpenSearchClientException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.. Call: Status code 401 from: PUT /_snapshot/devuse2dep2es1snapshotrepository?verify=false\r\n ---> OpenSearch.Net.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.\r\n   at OpenSearch.Net.RequestPipeline.ThrowBadAuthPipelineExceptionWhenNeeded(IApiCallDetails details, IOpenSearchResponse response)\r\n   at OpenSearch.Net.RequestPipeline.CallOpenSearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)\r\n   at OpenSearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)\r\n   --- End of inner exception stack trace ---\r\n# Audit exception in step 1 BadResponse:\r\nOpenSearch.Net.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.\r\n   at OpenSearch.Net.RequestPipeline.ThrowBadAuthPipelineExceptionWhenNeeded(IApiCallDetails details, IOpenSearchResponse response)\r\n   at OpenSearch.Net.RequestPipeline.CallOpenSearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)\r\n   at OpenSearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)\r\n# Request:\r\n<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>\r\n# Response:\r\n<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>\r\n", ServerError="(null)"}
Ermetic.Sil.Infrastructure.AppException: ExecuteOperationAsync failed
   from ExecuteOperation in C:\Dev\sil3\src\Common\DataStores\Elasticsearch\ElasticsearchClient.cs:line 366
 ---> OpenSearch.Net.OpenSearchClientException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.. Call: Status code 401 from: PUT /_snapshot/devuse2dep2es1snapshotrepository?verify=false
 ---> OpenSearch.Net.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.
   at OpenSearch.Net.RequestPipeline.ThrowBadAuthPipelineExceptionWhenNeeded(IApiCallDetails details, IOpenSearchResponse response)
   at OpenSearch.Net.RequestPipeline.CallOpenSearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at OpenSearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at Ermetic.Sil.Common.ElasticsearchClient.ExecuteOperationAsync[TResponse](Func`1 operationAsync, Func`2 validateOperationErrorResponse) in C:\Dev\sil3\src\Common\DataStores\Elasticsearch\ElasticsearchClient.cs:line 366
   at Ermetic.Sil.Common.ElasticsearchClient.CreateSnapshotRepositoryAsync(String bucketName, String bucketObjectKeyPrefix, String name, Boolean readOnly, AwsResourceArn roleArn) in C:\Dev\sil3\src\Common\DataStores\Elasticsearch\ElasticsearchClient.cs:line 92
   at Ermetic.Sil.Common.System.AwsElasticsearchDomain.CreateSnapshotRepositoryAsync(String bucketName, String bucketObjectKeyPrefix, Boolean readOnly, String repositoryName, AwsResourceArn roleArn) in C:\Dev\sil3\src\Common.System\Resources\Aws\Es\AwsElasticsearchDomain.cs:line 120
   at Ermetic.Sil.Shell.CreateElasticsearchDomainSnapshotRepositoryCmdlet.RunAsync() in C:\Dev\sil3\src\Shell\Cmdlets\Deployment\ElasticsearchDomain\SnapshotRepositories\CreateElasticsearchDomainSnapshotRepositoryCmdlet.cs:line 37
   at Ermetic.Sil.Infrastructure.TaskExtensions.Block(Task task) in C:\Dev\sil3\src\Infrastructure\Extensions\TaskExtensions.cs:line 8
   at Ermetic.Sil.Shell.CmdletBase.ProcessRecord() in C:\Dev\sil3\src\Shell\Cmdlets\CmdletBase.cs:line 111
2022-12-11 16:00:11.0783 ERR CreateElasticsearchDomainSnapshotCmdlet RunAsync failed {DebugInformation="Invalid OpenSearch.Client response built from a unsuccessful (401) low level call on PUT: /_snapshot/devuse2dep2es1snapshotrepository?verify=false\r\n# Audit trail of this API call:\r\n - [1] BadResponse: Node: https://vpc-devuse2dep2es1-yrdn4ehgxv26lwnfq7kft6kz5e.us-east-2.es.amazonaws.com/ Exception: PipelineException Took: 00:00:00.2514008\r\n# OriginalException: OpenSearch.Net.OpenSearchClientException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.. Call: Status code 401 from: PUT /_snapshot/devuse2dep2es1snapshotrepository?verify=false\r\n ---> OpenSearch.Net.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.\r\n   at OpenSearch.Net.RequestPipeline.ThrowBadAuthPipelineExceptionWhenNeeded(IApiCallDetails details, IOpenSearchResponse response)\r\n   at OpenSearch.Net.RequestPipeline.CallOpenSearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)\r\n   at OpenSearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)\r\n   --- End of inner exception stack trace ---\r\n# Audit exception in step 1 BadResponse:\r\nOpenSearch.Net.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.\r\n   at OpenSearch.Net.RequestPipeline.ThrowBadAuthPipelineExceptionWhenNeeded(IApiCallDetails details, IOpenSearchResponse response)\r\n   at OpenSearch.Net.RequestPipeline.CallOpenSearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)\r\n   at OpenSearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)\r\n# Request:\r\n<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>\r\n# Response:\r\n<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>\r\n", ServerError="(null)"}
System.Management.Automation.CmdletInvocationException: ExecuteOperationAsync failed
   from ExecuteOperation in C:\Dev\sil3\src\Common\DataStores\Elasticsearch\ElasticsearchClient.cs:line 366
 ---> Ermetic.Sil.Infrastructure.AppException: ExecuteOperationAsync failed
   from ExecuteOperation in C:\Dev\sil3\src\Common\DataStores\Elasticsearch\ElasticsearchClient.cs:line 366
 ---> OpenSearch.Net.OpenSearchClientException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.. Call: Status code 401 from: PUT /_snapshot/devuse2dep2es1snapshotrepository?verify=false
 ---> OpenSearch.Net.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.
   at OpenSearch.Net.RequestPipeline.ThrowBadAuthPipelineExceptionWhenNeeded(IApiCallDetails details, IOpenSearchResponse response)
   at OpenSearch.Net.RequestPipeline.CallOpenSearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at OpenSearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at Ermetic.Sil.Common.ElasticsearchClient.ExecuteOperationAsync[TResponse](Func`1 operationAsync, Func`2 validateOperationErrorResponse) in C:\Dev\sil3\src\Common\DataStores\Elasticsearch\ElasticsearchClient.cs:line 366
   at Ermetic.Sil.Common.ElasticsearchClient.CreateSnapshotRepositoryAsync(String bucketName, String bucketObjectKeyPrefix, String name, Boolean readOnly, AwsResourceArn roleArn) in C:\Dev\sil3\src\Common\DataStores\Elasticsearch\ElasticsearchClient.cs:line 92
   at Ermetic.Sil.Common.System.AwsElasticsearchDomain.CreateSnapshotRepositoryAsync(String bucketName, String bucketObjectKeyPrefix, Boolean readOnly, String repositoryName, AwsResourceArn roleArn) in C:\Dev\sil3\src\Common.System\Resources\Aws\Es\AwsElasticsearchDomain.cs:line 120
   at Ermetic.Sil.Shell.CreateElasticsearchDomainSnapshotRepositoryCmdlet.RunAsync() in C:\Dev\sil3\src\Shell\Cmdlets\Deployment\ElasticsearchDomain\SnapshotRepositories\CreateElasticsearchDomainSnapshotRepositoryCmdlet.cs:line 37
   at Ermetic.Sil.Infrastructure.TaskExtensions.Block(Task task) in C:\Dev\sil3\src\Infrastructure\Extensions\TaskExtensions.cs:line 8
   at Ermetic.Sil.Shell.CmdletBase.ProcessRecord() in C:\Dev\sil3\src\Shell\Cmdlets\CmdletBase.cs:line 111
   at System.Management.Automation.Cmdlet.DoProcessRecord()
   at System.Management.Automation.CommandProcessor.ProcessRecord()
   --- End of inner exception stack trace ---
   at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)
   at System.Management.Automation.Runspaces.Pipeline.Invoke()
   at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke)
   at System.Management.Automation.PowerShell.Worker.CreateRunspaceIfNeededAndDoWork(Runspace rsToUse, Boolean isSync)
   at System.Management.Automation.PowerShell.CoreInvokeHelper[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.CoreInvoke[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.Invoke(IEnumerable input, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.Invoke()
   at Ermetic.Sil.Shell.CmdletBase.RunCmdlet(CmdletBase cmdlet) in C:\Dev\sil3\src\Shell\Cmdlets\CmdletBase.cs:line 176
   at Ermetic.Sil.Shell.EnvironmentCmdlet.RunCmdlet(EnvironmentCmdlet environmentCmdlet) in C:\Dev\sil3\src\Shell\Cmdlets\EnvironmentCmdlet.cs:line 82
   at Ermetic.Sil.Shell.ApplicationTenantCmdlet.RunCmdlet(ApplicationTenantCmdlet applicationTenantCmdlet) in C:\Dev\sil3\src\Shell\Cmdlets\ApplicationTenantCmdlet.cs:line 123
   at Ermetic.Sil.Shell.DeploymentCmdletBase.RunCmdlet(DeploymentCmdletBase deploymentCmdletBase) in C:\Dev\sil3\src\Shell\Cmdlets\Deployment\DeploymentCmdletBase.cs:line 137
   at Ermetic.Sil.Shell.ElasticsearchDomainCmdlet.RunCmdlet(ElasticsearchDomainCmdlet elasticsearchDomainCmdlet) in C:\Dev\sil3\src\Shell\Cmdlets\Deployment\ElasticsearchDomain\ElasticsearchDomainCmdlet.cs:line 147
   at Ermetic.Sil.Shell.CreateElasticsearchDomainSnapshotCmdlet.RunAsync() in C:\Dev\sil3\src\Shell\Cmdlets\Deployment\ElasticsearchDomain\Snapshots\CreateElasticsearchDomainSnapshotCmdlet.cs:line 15
   at Ermetic.Sil.Infrastructure.TaskExtensions.Block(Task task) in C:\Dev\sil3\src\Infrastructure\Extensions\TaskExtensions.cs:line 8
   at Ermetic.Sil.Shell.CmdletBase.ProcessRecord() in C:\Dev\sil3\src\Shell\Cmdlets\CmdletBase.cs:line 111
Create-SilElasticsearchDomainSnapshotRepository: ExecuteOperationAsync failed
   from ExecuteOperation in C:\Dev\sil3\src\Common\DataStores\Elasticsearch\ElasticsearchClient.cs:line 366

Do you have any additional context?

Just to be clear, the following procedure works without HTTP compression enabled, there is no error with the credentials at all.
Here is the client creation as code:

var connectionSettings = CreateConnectionSettings(Serializer);
OpenSearchClient = new OpenSearchClient(connectionSettings);

ConnectionSettings CreateConnectionSettings(OpenSearchSerializer serializer) =>
                new ConnectionSettings(
                        new SingleNodeConnectionPool(domainUrl),
                        new AwsSigV4HttpConnection(),
                        (_, _) => serializer).
                    ConnectionLimit((int)(ConcurrencyLimiter.ConcurrencyLevel * 1.1)).
                    DefaultFieldNameInferrer(fieldName => fieldName).
                    EnableHttpCompression().
                    RequestTimeout(5.Minutes());
```

[Xtansia]

@omamoo Would you be able to provide an example of your CreateSnapshotRepository request (redacting anything sensitive)?

This may be a similar issue to #104

[michfa]

Replying on behalf of @omamoo (all names replaced):

Request:

PUT https://domain1-yrdn4ehgxv26lwnfq7kft6kz5e.us-east-2.es.amazonaws.com/_snapshot/snapshotrepository?verify=false HTTP/1.1
Host: domain1-yrdn4ehgxv26lwnfq7kft6kz5e.us-east-2.es.amazonaws.com
Accept: application/json
User-Agent: opensearch-net/1.2.0+abd6402041bbb1b078ce45f5f9918129d6137172 (Microsoft Windows 10.0.22621; .NET 6.0.11; OpenSearch.Client)
opensearch-client-meta: opensearch=1.2.0,a=1,net=6.0.11,so=6.0.11
x-amz-date: 20221211T140010Z
Authorization: AWS4-HMAC-SHA256 Credential=ASIAQJOVEAFN3YFQETGG/20221211/us-east-2/es/aws4_request, SignedHeaders=accept;content-encoding;content-type;host;opensearch-client-meta;x-amz-date;x-amz-security-token, Signature=b77b0109eeea8f0b0d64dbd9b59d93abe533bc5b716028af4e3a781dda540cca
x-amz-security-token: FwoGZXIvYXdzEHMaDL9wHWrA5kwqIFNB9yK6Aft6F+C04+LFRCfphrblMwj2XlZKrcoENaoNnkVyLLjOsozKkDxl9vpQaCzWoLdnnoEITgP6NEV4JOSvNgMby+iA/LS1/hyoQ0U3eLGfdTAmSLtshPF9EGpr6vNwjU6L36ygSi8jOMh3qIyRnF7Wmm0U9QnIm2oXjtDyDUwiowApcahVw3/lQHHzPmaZPfrpjff+d+B7H1X5kivmM78x79khgwX5rpcp3dIK9QYISH/H35dxg0CAv3gwGyj1vdacBjItp4J1WSFnChuPmud5FaMSsImOJR+Ukea1Dvr5juxr3FEz9oozjgKYHttLatyv
Accept-Encoding: gzip, deflate
Content-Type: application/json
Content-Length: 212

{
    "settings": {
        "base_path": "Es/SnapshotRepository",
        "bucket": "bucket",
        "readonly": "false",
        "role_arn": "arn:aws:iam::012345678901:role/EsSnapshotRepositoryRole"
    },
    "type": "s3"
}

Response:

HTTP/1.1 401 Unauthorized
Date: Sun, 11 Dec 2022 14:00:11 GMT
Content-Type: application/json
Content-Length: 134
Connection: keep-alive
x-amzn-requestid: d959bb4e-628c-431d-afbf-10c4225c4edc
access-control-allow-origin: *

{
    "Message": "Your request: '/_snapshot/snapshotrepository?verify=false' is not allowed due to invalid input parameters."
}

[michfa]

@Xtansia - we encountered the same issue when trying to update the cluster settings.
We ran the following code (with the same client defined in the issue description):

                clusterPutSettingsDescriptor =>
                    clusterPutSettingsDescriptor.Persistent(
                        settingNameToValueMap =>
                            settingNameToValueMap.Add("action.auto_create_index", value: false)));

The generated request was:

PUT https://domain1-2gghghuw5pw6kdqnezzrckakbi.us-east-2.es.amazonaws.com/_cluster/settings HTTP/1.1
Host: domain1-2gghghuw5pw6kdqnezzrckakbi.us-east-2.es.amazonaws.com
Accept: application/json
User-Agent: opensearch-net/1.2.0+abd6402041bbb1b078ce45f5f9918129d6137172 (Microsoft Windows 10.0.22000; .NET 6.0.12; OpenSearch.Client)
opensearch-client-meta: opensearch=1.2.0,a=1,net=6.0.12,so=6.0.12
x-amz-date: 20221221T160437Z
Authorization: AWS4-HMAC-SHA256 Credential=ASIAQJOVEAFN3WL7TIVW/20221221/us-east-2/es/aws4_request, SignedHeaders=accept;content-encoding;content-type;host;opensearch-client-meta;x-amz-date;x-amz-security-token, Signature=d2b244190f15f0e26f15ccfb560fc628b7fe2adb342d77d9994134d001c5e3ab
x-amz-security-token: FwoGZXIvYXdzEGcaDFB7zX6a5CQSJfdvVSK3AesJGoeDjmDQzlJM3sId5X7hNf4fMaMY7WQI4Z6GuuFtw16aGA3/HCnrE8gBFnpjYkELqf8DejfCwh0kHfgXukMbA9suSYdN+IsKHwrfKGvN3hdnEvVhch8uqzz4TMhRKYeU+i808ITpkVwjoXPI95fYZYpJ0bBjvReEAj0rqzlRVYR7ifLngk9AjFp8lRCHrinzu5N5toLe0NW4UMQSrKR4Qq9bMZebT/jxEXIVISFEXv7ZqCSwRCj3m4ydBjIt/j5sNQ1Orx71WqaLsL59hY/OSDdMuRKyhgeV2kv6M2oeOAAiLyo5yBjpHwrO
Accept-Encoding: gzip, deflate
Content-Type: application/json
Content-Length: 49

{"persistent":{"action.auto_create_index":false}}

And the response:

HTTP/1.1 401 Unauthorized
Date: Wed, 21 Dec 2022 16:04:36 GMT
Content-Type: application/json
Content-Length: 72
Connection: keep-alive
x-amzn-requestid: 967df0c5-2321-448e-8ac6-fc22bd74ea9b
access-control-allow-origin: *

{"Message":"Your request: '/_cluster/settings' payload is not allowed."}

with an opensearch client created without http compression, the request was successfull

[wbeckler]

Do any requests at all ever work with compression enabled? It sounds like it might just be an issue of sigv4 + compression.

[michfa]

Hi @wbeckler,
Sorry for the late response - we manage to query the data and bulk insert with compression enabled.
For the requests in the previous comments - create snapshot repository/take snapshot/update cluster config - we had to create another client with encryption disabled.

[omamoo]

Any updates?

[dblock]

@omamoo I don't think anyone is working on this, please contribute!

[wbeckler]

There were some changes to how the sigv4 signing works, committed in the main branch but not yet released. Maybe give it a try to see if it magically fixed this issue?