Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wr_arp] add timeout to thread join #2986

Merged
merged 2 commits into from
Feb 16, 2021
Merged

[wr_arp] add timeout to thread join #2986

merged 2 commits into from
Feb 16, 2021

Conversation

yxieca
Copy link
Collaborator

@yxieca yxieca commented Feb 12, 2021

Description of PR

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Approach

What is the motivation for this PR?

test_wr_arp.py got stuck in nightly test until the test was aborted. This is because paramiko didn't return shell command return code from exec_command() call. So if there were error and warm-reboot didn't proceed, the command will return as success. Without return code, we cannot solely rely on stderr to judge if the command failed or not (because we could ignored the error and proceeded).

How did you do it?

When warm reboot fails to reboot the dut and test failed to detect the failure. The thread.join() would fail and the test will stuck
indefinitely.

Add a timeout to join and check if the thread quit after join returns.

Signed-off-by: Ying Xie [email protected]

How did you verify/test it?

While warm reboot fails, with the fix, the thread join will timeout and give properly error information:

E "======================================================================",
E "FAIL: wr_arp.ArpTest",
E "----------------------------------------------------------------------",
E "Traceback (most recent call last):",
E " File "ptftests/wr_arp.py", line 278, in runTest",
E " self.assertTrue(False, "Timed out waiting for warm reboot")",
E "AssertionError: Timed out waiting for warm reboot",
E "",
E "----------------------------------------------------------------------",
E "Ran 1 test in 384.692s",
E "",
E "FAILED (failures=1)"

When warm reboot fails to reboot the dut and test failed to detect
the failure. The thread.join() would fail and the test will stuck
indefinitely.

Add a timeout to join and check if the thread quit after join
returns.

Signed-off-by: Ying Xie <[email protected]>
@yxieca yxieca requested a review from a team as a code owner February 12, 2021 21:44
ansible/roles/test/files/ptftests/wr_arp.py Outdated Show resolved Hide resolved
@vaibhavhd
Copy link
Contributor

/AzurePipelines run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@yxieca yxieca merged commit a39bd2d into sonic-net:master Feb 16, 2021
@yxieca yxieca deleted the wr branch February 16, 2021 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants