Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Failure (producer timeout) in NodesDecommissioningTest.test_decommissioning_cancel_ongoing_movements #11751

Closed
VladLazar opened this issue Jun 28, 2023 · 6 comments · Fixed by #11903
Closed

CI Failure (producer timeout) in NodesDecommissioningTest.test_decommissioning_cancel_ongoing_movements #11751

VladLazar opened this issue Jun 28, 2023 · 6 comments · Fixed by #11903
Assignees
@mmaslankaprv
Labels
area/redpanda ci-failure kind/bug Something isn't working

Comments

@VladLazar
Copy link
Contributor

https://buildkite.com/redpanda/redpanda/builds/32125#018901a0-d89b-41a7-924b-0c833bc6e8b9

Module: rptest.tests.nodes_decommissioning_test
Class:  NodesDecommissioningTest
Method: test_decommissioning_cancel_ongoing_movements

test_id:    rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_cancel_ongoing_movements
status:     FAIL
run time:   11 minutes 7.171 seconds


    TimeoutError(None)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 79, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/nodes_decommissioning_test.py", line 408, in test_decommissioning_cancel_ongoing_movements
    self.verify()
  File "/root/tests/rptest/tests/nodes_decommissioning_test.py", line 278, in verify
    self.producer.wait()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/service.py", line 261, in wait
    if not self.wait_node(node, end - now):
  File "/root/tests/rptest/services/kgo_verifier_services.py", line 639, in wait_node
    self._redpanda.wait_until(lambda: self._status_thread.errored or
  File "/root/tests/rptest/services/redpanda.py", line 1997, in wait_until
    wait_until(wrapped,
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: None
@VladLazar VladLazar added kind/bug Something isn't working ci-failure labels Jun 28, 2023
@VladLazar VladLazar mentioned this issue Jun 28, 2023
Merged
7 tasks
@dotnwat
Copy link
Member

dotnwat commented Jun 28, 2023

https://buildkite.com/redpanda/redpanda/builds/32117#018900fb-5d4d-4fa7-ac90-e897a44f542e

@dotnwat dotnwat mentioned this issue Jun 29, 2023
Merged
7 tasks
@jcsp jcsp mentioned this issue Jun 29, 2023
Merged
9 tasks
@dlex dlex mentioned this issue Jun 29, 2023
Closed
7 tasks
@andrwng andrwng mentioned this issue Jun 29, 2023
Merged
7 tasks
@jcsp jcsp mentioned this issue Jun 30, 2023
Merged
7 tasks
@VladLazar VladLazar mentioned this issue Jun 30, 2023
Merged
7 tasks
@bharathv bharathv mentioned this issue Jun 30, 2023
Merged
7 tasks
@BenPope BenPope mentioned this issue Jul 2, 2023
Merged
7 tasks
@jcsp jcsp mentioned this issue Jul 3, 2023
Merged
7 tasks
@vshtokman
Copy link
Contributor

https://buildkite.com/redpanda/vtools/builds/8306#01890e37-4e3c-4410-8295-fb5937d43c23

@jcsp
Copy link
Contributor

jcsp commented Jul 3, 2023

https://buildkite.com/redpanda/redpanda/builds/32402#01891bff-1c1d-40f4-8e1e-e869f8c40f3b

@jcsp jcsp mentioned this issue Jul 3, 2023
Merged
7 tasks
@BenPope BenPope mentioned this issue Jul 3, 2023
Merged
7 tasks
@vshtokman
Copy link
Contributor

https://buildkite.com/redpanda/redpanda/builds/32422#01891c9f-7d12-4fec-8a78-d2ea6767bafb

@jcsp
Copy link
Contributor

jcsp commented Jul 4, 2023

https://buildkite.com/redpanda/vtools/builds/8324#01891aca-be80-40c4-871f-ed05fddd9c27

@VladLazar VladLazar mentioned this issue Jul 4, 2023
Merged
7 tasks
@jcsp jcsp mentioned this issue Jul 4, 2023
Merged
7 tasks
@jcsp
Copy link
Contributor

jcsp commented Jul 5, 2023

https://buildkite.com/redpanda/redpanda/builds/32519#_

@mmaslankaprv mmaslankaprv self-assigned this Jul 6, 2023
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jul 6, 2023
@mmaslankaprv
tests: fixed querying for the node in decommission test
4ec279e 
In current redpanda version node ids are dynamically assigned we can not
use a node id as an index in the `RedpandaService` nodes list as it may
result in incorrect results. In this particular example a node that was
stopped was not the one that was decommissioned, leading to some of the
rf=1 partitions being not available which prevented test from finishing.

Fixes: redpanda-data#11751

Signed-off-by: Michal Maslanka <[email protected]>
@mmaslankaprv mmaslankaprv mentioned this issue Jul 6, 2023
Merged
7 tasks
@andijcr andijcr mentioned this issue Jul 6, 2023
Merged
7 tasks
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue Jul 6, 2023
@mmaslankaprv @vbotbuildovich
tests: fixed querying for the node in decommission test
5c706ce 
In current redpanda version node ids are dynamically assigned we can not
use a node id as an index in the `RedpandaService` nodes list as it may
result in incorrect results. In this particular example a node that was
stopped was not the one that was decommissioned, leading to some of the
rf=1 partitions being not available which prevented test from finishing.

Fixes: redpanda-data#11751

Signed-off-by: Michal Maslanka <[email protected]>
(cherry picked from commit 4ec279e)
@vbotbuildovich vbotbuildovich mentioned this issue Jul 6, 2023
Closed
@vbotbuildovich vbotbuildovich mentioned this issue Jul 6, 2023
Closed
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue Jul 6, 2023
@mmaslankaprv @vbotbuildovich
tests: fixed querying for the node in decommission test
f95588c 
In current redpanda version node ids are dynamically assigned we can not
use a node id as an index in the `RedpandaService` nodes list as it may
result in incorrect results. In this particular example a node that was
stopped was not the one that was decommissioned, leading to some of the
rf=1 partitions being not available which prevented test from finishing.

Fixes: redpanda-data#11751

Signed-off-by: Michal Maslanka <[email protected]>
(cherry picked from commit 4ec279e)
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue Jul 6, 2023
@mmaslankaprv @vbotbuildovich
tests: fixed querying for the node in decommission test
145ef3b 
In current redpanda version node ids are dynamically assigned we can not
use a node id as an index in the `RedpandaService` nodes list as it may
result in incorrect results. In this particular example a node that was
stopped was not the one that was decommissioned, leading to some of the
rf=1 partitions being not available which prevented test from finishing.

Fixes: redpanda-data#11751

Signed-off-by: Michal Maslanka <[email protected]>
(cherry picked from commit 4ec279e)
@vbotbuildovich vbotbuildovich mentioned this issue Jul 6, 2023
Closed
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jul 20, 2023
@mmaslankaprv
tests: fixed querying for the node in decommission test
1d5a6a6 
In current redpanda version node ids are dynamically assigned we can not
use a node id as an index in the `RedpandaService` nodes list as it may
result in incorrect results. In this particular example a node that was
stopped was not the one that was decommissioned, leading to some of the
rf=1 partitions being not available which prevented test from finishing.

Fixes: redpanda-data#11751

Signed-off-by: Michal Maslanka <[email protected]>
(cherry picked from commit 4ec279e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmaslankaprv mmaslankaprv

Labels
area/redpanda ci-failure kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixed querying for the node in decommission test mmaslankaprv/redpanda
5 participants
@dotnwat @jcsp @vshtokman @VladLazar @mmaslankaprv