CI Failure (BadLogLines ptree_bad_path (No such node (DeleteResult)) in ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy

[andijcr]

https://buildkite.com/redpanda/vtools/builds/9804#018afee9-476c-416b-b96d-5a2a2d6f674c

Module: rptest.tests.e2e_shadow_indexing_test
Class:  ShadowIndexingWhileBusyTest
Method: test_create_or_delete_topics_while_busy
Arguments:
{
  "cloud_storage_type": 1,
  "short_retention": true
}

  | <BadLogLines nodes=ip-172-31-2-8(2),ip-172-31-6-244(2),ip-172-31-5-105(2),ip-172-31-8-73(2),ip-172-31-12-62(2),ip-172-31-0-120(2) example="ERROR 2023-10-05 09:50:29,100 [shard 2:au  ] s3 - s3_client.cc:889 - DeleteObjects response parse failed: boost::wrapexcept<boost::property_tree::ptree_bad_path> (No such node (DeleteResult))"> Traceback (most recent call last):   File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run     data = self.run_test()   File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test     return self.test_context.function(self.test)   File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper     return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)   File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 142, in wrapped     redpanda.raise_on_bad_logs(   File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1244, in raise_on_bad_logs     raise BadLogLines(bad_lines) rptest.services.utils.BadLogLines: <BadLogLines nodes=ip-172-31-2-8(2),ip-172-31-6-244(2),ip-172-31-5-105(2),ip-172-31-8-73(2),ip-172-31-12-62(2),ip-172-31-0-120(2) example="ERROR 2023-10-05 09:50:29,100 [shard 2:au  ] s3 - s3_client.cc:889 - DeleteObjects response parse failed: boost::wrapexcept<boost::property_tree::ptree_bad_path> (No such node (DeleteResult))">
-- | --

        {
            "title": "<BadLogLines nodes=ip-172-31-2-8(2),ip-172-31-6-244(2),ip-172-31-5-105(2),ip-172-31-8-73(2),ip-172-31-12-62(2),ip-172-31-0-120(2) example=\"ERROR 2023-10-05 09:50:29,100 [shard 2:au  ] s3 - s3_client.cc:889 - DeleteObjects response parse failed: boost::wrapexcept<boost::property_tree::ptree_bad_path> (No such node (DeleteResult))\">",
            "id": 15190,
            "ts": 1696520189.320468,
            "type": "cdt",
            "build": "release",
            "arch": "amd64",
            "link": "https://buildkite.com/redpanda/vtools/builds/9804"
        }

[andijcr]

sev/medium because the exception may be a symptom that we are not correctly handling an error response

[abhijat]

It looks like the 200 OK response contains the error (quite possibly slowdown as there are many of them in the broker logs).

Some AWS operations can return an error with 200 OK status eg https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html

If the error occurs during the copy operation, the error response is embedded in the 200 OK response. This means that a 200 OK response can contain either a success or an error. If you call the S3 API directly, make sure to design your application to parse the contents of the response and handle it appropriately. If you use AWS SDKs, SDKs handle this condition. 

The DeleteObjects API does not mention this but quite possibly we are facing the same issue here. The AWS rust SDK has a very similar issue recently:

awslabs/aws-sdk-rust#873

We might need to handle the 200 response as potentially containing an error. AFAIK there have been some changes to this area recently.

[rockwotj]

This happened again yesterday on nightly: https://buildkite.com/redpanda/vtools/builds/10097

[ztlpn]

https://buildkite.com/redpanda/vtools/builds/10163

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/10219
*https://buildkite.com/redpanda/vtools/builds/10266
*https://buildkite.com/redpanda/vtools/builds/10266
*https://buildkite.com/redpanda/vtools/builds/10278
*https://buildkite.com/redpanda/vtools/builds/10287
*https://buildkite.com/redpanda/vtools/builds/10294
*https://buildkite.com/redpanda/vtools/builds/10300
*https://buildkite.com/redpanda/vtools/builds/10304
*https://buildkite.com/redpanda/vtools/builds/10308
*https://buildkite.com/redpanda/vtools/builds/10328
*https://buildkite.com/redpanda/vtools/builds/10344
*https://buildkite.com/redpanda/vtools/builds/10365
*https://buildkite.com/redpanda/vtools/builds/10378
*https://buildkite.com/redpanda/vtools/builds/10378
*https://buildkite.com/redpanda/vtools/builds/10393
*https://buildkite.com/redpanda/vtools/builds/10398

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/10423

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/10446
*https://buildkite.com/redpanda/vtools/builds/10450

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/10456
*https://buildkite.com/redpanda/vtools/builds/10461