Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[3006.x] Fix test_system flaky setup_teardown #66861

Merged
merged 1 commit into from
Sep 18, 2024

Conversation

m-czernek
Copy link
Contributor

What does this PR do?

I noticed that when running the testsuite, we intermittently get the following error:

04:28:52  _________________ ERROR at teardown of test_set_computer_desc __________________
04:28:52  Traceback (most recent call last):
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/_pytest/runner.py", line 341, in from_call
04:28:52  result: Optional[TResult] = func()
04:28:52  ^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/_pytest/runner.py", line 262, in <lambda>
04:28:52  lambda: ihook(item=item, **kwds), when=when, reraise=reraise
04:28:52  ^^^^^^^^^^^^^^^^^^^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
04:28:52  return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
04:28:52  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
04:28:52  return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
04:28:52  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
04:28:52  return outcome.get_result()
04:28:52  ^^^^^^^^^^^^^^^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
04:28:52  raise ex[1].with_traceback(ex[2])
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
04:28:52  res = hook_impl.function(*args)
04:28:52  ^^^^^^^^^^^^^^^^^^^^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/_pytest/runner.py", line 182, in pytest_runtest_teardown
04:28:52  item.session._setupstate.teardown_exact(nextitem)
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/_pytest/runner.py", line 537, in teardown_exact
04:28:52  raise exceptions[0]
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/_pytest/runner.py", line 526, in teardown_exact
04:28:52  fin()
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/_pytest/fixtures.py", line 701, in <lambda>
04:28:52  subrequest.node.addfinalizer(lambda: fixturedef.finish(request=subrequest))
04:28:52  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/_pytest/fixtures.py", line 1031, in finish
04:28:52  raise exc
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/_pytest/fixtures.py", line 1024, in finish
04:28:52  func()
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/_pytest/fixtures.py", line 911, in _teardown_yield_fixture
04:28:52  next(it)
04:28:52  File "/usr/lib/venv-salt-minion/lib/python3.11/site-packages/salt-testsuite/tests/pytests/functional/modules/test_system.py", line 83, in setup_teardown_vars
04:28:52  res = service.start("systemd-timesyncd")
04:28:52  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/salt/loader/lazy.py", line 149, in __call__
04:28:52  return self.loader.run(run_func, *args, **kwargs)
04:28:52  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/salt/loader/lazy.py", line 1234, in run
04:28:52  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
04:28:52  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib64/python3.11/site-packages/salt/loader/lazy.py", line 1249, in _run_as
04:28:52  return _func_or_method(*args, **kwargs)
04:28:52  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
04:28:52  File "/usr/lib/venv-salt-minion/lib/python3.11/site-packages/salt/modules/systemd_service.py", line 872, in start
04:28:52  raise CommandExecutionError(_strip_scope(ret["stderr"]))
04:28:52  salt.exceptions.CommandExecutionError: Job for systemd-timesyncd.service failed.
04:28:52  See "systemctl  status systemd-timesyncd.service" and "journalctl  -xe" for details.
04:28:52  ---------------------------- Captured log teardown -----------------------------
04:28:52  ERROR    tests.pytests.functional.conftest.loaded.int.module.cmdmod:impl.py:295 Command '/usr/bin/systemd-run' failed with return code: 1
04:28:52  ERROR    tests.pytests.functional.conftest.loaded.int.module.cmdmod:impl.py:295 stderr: Running scope as unit: run-r0c01f3d9cc894f94aa5db15d337abb62.scope
04:28:52  Job for systemd-timesyncd.service failed.
04:28:52  See "systemctl  status systemd-timesyncd.service" and "journalctl  -xe" for details.
04:28:52  ERROR    tests.pytests.functional.conftest.loaded.int.module.cmdmod:impl.py:295 retcode: 1

The reason for that is the service start request repeated too quickly, refusing to start error, which seems to happen when restarting a service more than 5 times in a 10s window as per [0].

This diff should fix the test flakiness; when service.start throws an exception, we wait 10s and try again. If the service start still fails, we fail. Otherwise, we continue.

[0] https://serverfault.com/questions/845471/service-start-request-repeated-too-quickly-refusing-to-start-limit

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

Commits signed with GPG?

Yes

Please review Salt's Contributing Guide for best practices, including the
PR Guidelines.

See GitHub's page on GPG signing for more information about signing commits with GPG.

@dwoz dwoz merged commit 6c38c1a into saltstack:3006.x Sep 18, 2024
160 checks passed
Copy link

welcome bot commented Sep 18, 2024

Congratulations on your first PR being merged! 🎉

@m-czernek m-czernek deleted the fix-test-system branch September 23, 2024 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants