Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dev: override bazel timeout when run under stress #72046

Merged
merged 1 commit into from
Oct 28, 2021

Conversation

irfansharif
Copy link
Contributor

When running under stress and no timeout is specified, we want
to respect the timeout passed down to stress1. Not doing so has bazel
default to a timeout based on the test target's size2, which is not
what we want.

Release note: None

Footnotes

  1. Through --stress-arg=-maxtime or if nothing is specified, a
    -maxtime of 0 that's taken as "run forever")

  2. https://docs.bazel.build/versions/main/test-encyclopedia.html#role-of-the-test-runner

When running under stress and no timeout is specified, we want
to respect the timeout passed down to stress[^1]. Not doing so has bazel
default to a timeout based on the test target's size[^2], which is not
what we want.

[^1]: Through --stress-arg=-maxtime or if nothing is specified, a
     -maxtime of 0 that's taken as "run forever")
[^2]: https://docs.bazel.build/versions/main/test-encyclopedia.html#role-of-the-test-runner

Release note: None
@irfansharif irfansharif requested a review from a team as a code owner October 27, 2021 17:27
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@irfansharif
Copy link
Contributor Author

irfansharif commented Oct 27, 2021

Aside: I see we ripped out the automatic recording stuff in #68514, but the datadriven tests here still consults these files. I had to edit them by hand, which I stumbled with expecting --rewrite to pick it up like we do elsewhere (and the fact that two files needed updating -- definitely worth adding in the README here).

In the future we may consider migrating to a proper mocking framework for dev.c

I'm curious what we're imagining a proper mocking framework for dev to look like? And what would it give us over using recordings in the way we were before?

@rickystewart
Copy link
Collaborator

Aside: I see we ripped out the automatic recording stuff in #68514, but the datadriven tests here still consults these files.

Yeah, the recording wasn't useful. It introduced so many diffs that heavy manual editing of the testdata was still necessary, and at that point it's way less time-consuming to just manually make the changes you were expecting. This also scaled poorly as more tests were added.

I'm curious what we're imagining a proper mocking framework for dev to look like? And what would it give us over using recordings in the way we were before?

For example, a limitation of the current design is that the recording framework doesn't allow us to capture that certain events should fail, which means we can't test edge/failure cases with the datadriven tests.

@irfansharif
Copy link
Contributor Author

It introduced so many diffs that heavy manual editing of the testdata was still necessary

That sounds surprising and was probably a bug if anything. The intent was that -record -from-checkout=<path> -rewrite with obviate all need for any manual editing. Do you remember what you were doing that made these edits necessary? I'd love to learn.

we can't test edge/failure cases with the datadriven tests.

We use datadriven tests to test failure cases all throughout our logic tests, etc, and I think it's particularly well suited for it too. Here are some examples of us doing just that:

# Logging to stderr without stderr capture causes an error in the default config.
run
start
--log=capture-stray-errors: {enable: false}, sinks: {stderr: {filter: INFO}}
----
error: yaml: did not find expected key

kvaccessor-update
upsert [d,e):Y
----
err: expected to find single row containing upserted spans

It's ends up being a matter of writing the error output in way that's distinguishable from the happy path, which the recorder didn't do, but I think it's a minor change to make.

@irfansharif
Copy link
Contributor Author

bors r+

@craig
Copy link
Contributor

craig bot commented Oct 28, 2021

Build failed (retrying...):

@craig
Copy link
Contributor

craig bot commented Oct 28, 2021

Build succeeded:

@craig craig bot merged commit 90881be into cockroachdb:master Oct 28, 2021
@irfansharif irfansharif deleted the 211027.dev-stress-timeout branch October 28, 2021 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants