CI Failure (failed to hydrate chunk starting at 0) in `CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves` #10868

travisdowns · 2023-05-18T18:52:49Z

https://buildkite.com/redpanda/redpanda/builds/29339#01882bb2-7bdb-49d9-b79b-7bcc668a9a40

Module: rptest.tests.cloud_storage_timing_stress_test
Class:  CloudStorageTimingStressTest
Method: test_cloud_storage_with_partition_moves

Failure:

    <BadLogLines nodes=docker-rp-18(1) example="ERROR 2023-05-17 22:06:46,417 [shard 1] cloud_storage - [fiber278 bee5085b/kafka/test-topic/0_20/7905-8036-4213422-2-v1.log.2] - segment_chunk_data_source.cc:84 - failed to hydrate chunk starting at 0, error: seastar::abort_requested_exception (abort requested)">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 87, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 2274, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-18(1) example="ERROR 2023-05-17 22:06:46,417 [shard 1] cloud_storage - [fiber278 bee5085b/kafka/test-topic/0_20/7905-8036-4213422-2-v1.log.2] - segment_chunk_data_source.cc:84 - failed to hydrate chunk starting at 0, error: seastar::abort_requested_exception (abort requested)">

The text was updated successfully, but these errors were encountered:

dlex · 2023-05-22T03:43:21Z

Same in CloudStorageTimingStressTest.test_cloud_storage

on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29363#01882d81-f9a1-4cd9-b15f-043e47d4e971
on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29359#01882d6d-edc7-4782-9019-4f5a02e6ab3a
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/29359#01882d7c-84b5-48d8-abd4-c0c05e97ffef
on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7603#01882de4-7d22-4e20-95b5-653d0959ae48

dlex · 2023-05-22T04:13:27Z

on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29356#01882d67-471f-48bf-88e3-117033bbebd0
on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29363#01882d81-f99d-4e78-bbcf-4bb4999f297c
on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7603#01882de4-7d22-4e20-95b5-653d0959ae48

abhijat · 2023-05-22T07:12:24Z

This should have been fixed with the extended allow list in #10852. Will keep open for observation and close in a couple of days if no new instances are seen.

Removed the bug label as the behavior is expected.

michael-redpanda · 2023-05-25T12:47:07Z

https://buildkite.com/redpanda/vtools/builds/7764#0188513f-293d-459b-8036-df2377b38bc5
https://buildkite.com/redpanda/vtools/builds/7764#0188513f-293b-49dd-acaf-a19422321c9f

michael-redpanda · 2023-05-25T15:19:21Z

https://buildkite.com/redpanda/redpanda/builds/29872#018851a8-2e67-41df-9a77-a4e1c42746ab

NyaliaLui · 2023-05-25T16:03:41Z

https://buildkite.com/redpanda/redpanda/builds/29886#01885298-cb14-4abc-97fb-53526689cf7f
https://buildkite.com/redpanda/redpanda/builds/29799#01884ebb-4a91-49c6-a0f0-83e94bdc2bac

NyaliaLui · 2023-05-26T13:15:31Z

FAIL test: CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=compact.delete (7/31 runs)
  failure at 2023-05-26T07:26:58.598Z: <BadLogLines nodes=docker-rp-30(1) example="ERROR 2023-05-26 06:53:35,183 [shard 1] cloud_storage - [fiber658 889ded78/kafka/test-topic/0_20/6804-6933-4210586-1-v1.log.1] - segment_chunk_data_source.cc:108 - failed to hydrate chunk starting at 0, error: seastar::abort_requested_exception (abort requested)">
      on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29955#018856bd-c63f-4485-97e7-488126129857
FAIL test: CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=delete (4/30 runs)
  failure at 2023-05-25T19:47:10.649Z: <BadLogLines nodes=docker-rp-21(1) example="ERROR 2023-05-25 19:09:24,798 [shard 0] cloud_storage - [fiber175 e7fb37e0/kafka/test-topic/0_20/6726-7118-12640220-1-v1.log.1] - segment_chunk_data_source.cc:108 - failed to hydrate chunk starting at 0, error: seastar::abort_requested_exception (abort requested)">
      on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/29913#0188543e-447d-4d57-bc2d-7ec634915b39

abhijat · 2023-05-30T07:53:52Z

It looks like the chain of exception handlers passes the exception down:

DEBUG 2023-05-29 22:22:42,203 [shard 3] cloud_storage - [fiber254 a95c2145/kafka/test-topic/0_21/7565-7693-4213193-1-v1.log.1] - segment_chunk_data_source.cc:85 - abort requested during hydration of chunk starting at 0: seastar::abort_requested_exception (abort requested)
ERROR 2023-05-29 22:22:42,203 [shard 3] cloud_storage - [fiber254 a95c2145/kafka/test-topic/0_21/7565-7693-4213193-1-v1.log.1] - segment_chunk_data_source.cc:108 - failed to hydrate chunk starting at 0, error: seastar::abort_requested_exception (abort requested)

The first debug log is intended, but the exception returned from that handler is passed onto the last handler which results in the error log.

abhijat · 2023-05-30T09:25:23Z

https://buildkite.com/redpanda/redpanda/builds/30090#01886640-337a-4ad4-b2d8-1bc3bb6ea98a

[INFO  - 2023-05-29 07:21:11,123 - runner_client - log - lineno:278]: RunnerClient: rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=delete: Summary: <BadLogLines nodes=docker-rp-17(1) example="ERROR 2023-05-29 07:20:22,496 [shard 0] cloud_storage - [fiber455 84bcf2f1/kafka/test-topic/0_21/7630-8024-12640498-2-v1.log.2] - segment_chunk_data_source.cc:108 - failed to hydrate chunk starting at 0, error: seastar::abort_requested_exception (abort requested)">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 87, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 2663, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-17(1) example="ERROR 2023-05-29 07:20:22,496 [shard 0] cloud_storage - [fiber455 84bcf2f1/kafka/test-topic/0_21/7630-8024-12640498-2-v1.log.2] - segment_chunk_data_source.cc:108 - failed to hydrate chunk starting at 0, error: seastar::abort_requested_exception (abort requested)">

https://buildkite.com/redpanda/vtools/builds/7827#0188691e-0212-4567-a5bb-e59ae97dd98d

21/7565-7693-4213193-1-v1.log.1] - segment_chunk_data_source.cc:108 - failed to hydrate chunk starting at 0, error: seastar::abort_requested_exception (abort requested)">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 87, in wrapped
    redpanda.raise_on_bad_logs(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2663, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=ip-172-31-0-250(1) example="ERROR 2023-05-29 22:22:42,203 [shard 3] cloud_storage - [fiber254 a95c2145/kafka/test-topic/0_21/7565-7693-4213193-1-v1.log.1] - segment_chunk_data_source.cc:108 - failed to hydrate chunk starting at 0, error: seastar::abort_requested_exception (abort requested)">

vshtokman · 2023-05-31T16:16:09Z

https://buildkite.com/redpanda/vtools/builds/7860#01887231-1573-45a7-83e2-f9d42654da8d

twmb · 2023-06-02T00:42:30Z

https://buildkite.com/redpanda/redpanda/builds/30361#0188779d-3bc9-4bcc-8e59-1476ea9dc8cf

travisdowns added kind/bug Something isn't working ci-failure labels May 18, 2023

travisdowns mentioned this issue May 18, 2023

Optimizations to fetch planning #10385

Merged

7 tasks

mmaslankaprv added the area/cloud-storage Shadow indexing subsystem label May 20, 2023

abhijat self-assigned this May 22, 2023

abhijat removed the kind/bug Something isn't working label May 22, 2023

graphcareful mentioned this issue May 24, 2023

Kafka: Add support for the delete-records API #10061

Merged

7 tasks

dotnwat mentioned this issue May 25, 2023

Address clang16 use after move #11018

Merged

7 tasks

BenPope mentioned this issue May 25, 2023

net: Add GNUTLS_E_INVALID_SESSION to is_reconnect_error #11029

Merged

7 tasks

NyaliaLui mentioned this issue May 25, 2023

CI Failure (failed to hydrate chunk) in ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy #10931

Closed

ballard26 mentioned this issue May 26, 2023

Per-handler kafka request metrics #10623

Merged

7 tasks

dlex mentioned this issue May 26, 2023

Limit memory used while fetching many partitions #10905

Merged

7 tasks

rockwotj mentioned this issue May 30, 2023

Require conn_quota::units stay on a single shard #10681

Merged

7 tasks

abhijat mentioned this issue May 30, 2023

cloud_storage: Handle expected hydrate errors #11093

Merged

7 tasks

ztlpn mentioned this issue May 30, 2023

partition_allocator: preserve shards for replicas on original nodes #11077

Merged

7 tasks

r-vasquez mentioned this issue May 30, 2023

rpk debug bundle: pass SASL info to admin client #11104

Merged

1 task

mmaslankaprv mentioned this issue May 31, 2023

Skip controller log limiting mm2 test in debug mode #11100

Merged

7 tasks

BenPope mentioned this issue Jun 2, 2023

ducktape/test_simple_live_change: wait for all nodes #11156

Merged

7 tasks

abhijat mentioned this issue Jun 2, 2023

cloud_storage: Handle nested benign exception #10909

Merged

7 tasks

ztlpn mentioned this issue Jun 2, 2023

c/partition_allocator: maintain and use final partition counts #11154

Merged

7 tasks

piyushredpanda closed this as completed in #11093 Jun 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI Failure (failed to hydrate chunk starting at 0) in `CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves` #10868

CI Failure (failed to hydrate chunk starting at 0) in `CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves` #10868

travisdowns commented May 18, 2023

dlex commented May 22, 2023

dlex commented May 22, 2023

abhijat commented May 22, 2023 •

edited

Loading

michael-redpanda commented May 25, 2023 •

edited

Loading

michael-redpanda commented May 25, 2023

NyaliaLui commented May 25, 2023 •

edited

Loading

NyaliaLui commented May 26, 2023

abhijat commented May 30, 2023 •

edited

Loading

abhijat commented May 30, 2023

vshtokman commented May 31, 2023

twmb commented Jun 2, 2023

CI Failure (failed to hydrate chunk starting at 0) in CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves #10868

CI Failure (failed to hydrate chunk starting at 0) in CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves #10868

Comments

travisdowns commented May 18, 2023

dlex commented May 22, 2023

dlex commented May 22, 2023

abhijat commented May 22, 2023 • edited Loading

michael-redpanda commented May 25, 2023 • edited Loading

michael-redpanda commented May 25, 2023

NyaliaLui commented May 25, 2023 • edited Loading

NyaliaLui commented May 26, 2023

abhijat commented May 30, 2023 • edited Loading

abhijat commented May 30, 2023

vshtokman commented May 31, 2023

twmb commented Jun 2, 2023

CI Failure (failed to hydrate chunk starting at 0) in `CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves` #10868

CI Failure (failed to hydrate chunk starting at 0) in `CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves` #10868

abhijat commented May 22, 2023 •

edited

Loading

michael-redpanda commented May 25, 2023 •

edited

Loading

NyaliaLui commented May 25, 2023 •

edited

Loading

abhijat commented May 30, 2023 •

edited

Loading