forked from cockroachdb/cockroach
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This commit groups a few changes to speed up dev and make it easier to test. 1. Update `dev bench` to run benchmarks using `bazel test` directly instead of first grepping for benchmarks, querying test targets, and only then invoking test binaries using `bazel run`. The latter approach was both slower and more difficult to test. `dev bench` now looks identical to the `dev test` implementation. 2. Simplify `dev test`; now that protobuf targets are directly buildable, we don't need any manual querying of build targets and filtering for only go_test ones -- we can more easy `bazel test` whatever was top-level package was specified. This reduces how much I/O needs to happen with the host environment which both speeds things up and makes it easier to test. a. It also allows us to partially revert cockroachdb#74808 while still retaining informative error messages about missing targets (bazel's ones out-of-the-box are pretty good). 3. Finally, introduce TestDataDriven and TestRecorderDriven. a. TestDataDriven makes use of datadriven to capture all operations executed by individual dev invocations. The testcases are defined under testdata/datadriven/*. DataDriven divvies up these files as subtests, so individual "files" are runnable through: dev test pkg/cmd/dev -f TestDataDrivenDriven/<fname> [--rewrite] OR go test ./pkg/cmd/dev -run TestDataDrivenDriven/<fname> [-rewrite] b. TestRecorderDriven makes use of datadriven/recorder to record (if --rewrite is specified) or play back (if --rewrite is omitted) all operations executed by individual dev invocations. The testcases are defined under testdata/recorderdriven/*; each test file corresponds to a captured recording found in testdata/recorderdriven/*.rec. dev test pkg/cmd/dev -f TestRecorderDriven/<fname> OR go test ./pkg/cmd/dev -run TestRecorderDriven/<fname> [-rewrite] Recordings are used to mock out "system" behavior. When --rewrite is specified, attempts to shell out to bazel or perform other OS operations (like creating, removing, symlinking filepaths) are intercepted and system responses are recorded for future playback. --- It's worth comparing TestDataDriven and TestRecorderDriven. 1. TestDataDriven is well suited for exercising flows that don't depend on reading external state in order to function (simply translating a `dev test <target>` to its corresponding bazel invocation for e.g.) All operations are run in "dry-run" mode when --rewrite is specified; all exec/os commands return successfully with no error + an empty response, and just the commands are recorded as test data. 2. TestRecorderDriven in contrast works better for flows that do depend on external state during execution (like reading the set of targets from a file for e.g., or hoisting files from a sandbox by searching through the file system directly). With these, when --rewrite is specified, we actually do shell out and record the data for future playback. We previously (cockroachdb#68514) ripped out the equivalent of TestRecorderDriven because it was difficult to use (large recordings, execution errors failing the test, etc.) -- issues this PR tries to address. There are some real limitations here though that may still make this approach untenable: When --rewrite is used for TestRecorderDriven, because it shells out, will: - takes time proportional to the actual dev invocations; - makes it difficult to run under bazel (through without --rewrite it works just fine); The few remaining tests for this recorder stuff are probably examples of dev doing too much, when it should instead push the logic down into bazel rules. As we continue doing so, we should re-evaluate whether this harness provides much value. Release note: None
- Loading branch information
1 parent
3651e3c
commit 30a0874
Showing
57 changed files
with
1,245 additions
and
1,933 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2103,6 +2103,16 @@ def go_deps(): | |
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/dustin/go-humanize/com_github_dustin_go_humanize-v1.0.0.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_dvyukov_go_fuzz", | ||
| build_file_proto_mode = "disable_global", | ||
| importpath = "github.com/dvyukov/go-fuzz", | ||
| sha256 = "0a4c4bc0a550c729115d74f6a636e5802894b33bc50aa8af99c4a70196d5990b", | ||
| strip_prefix = "github.com/dvyukov/[email protected]", | ||
| urls = [ | ||
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/dvyukov/go-fuzz/com_github_dvyukov_go_fuzz-v0.0.0-20210103155950-6a8e9d1f2415.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_eapache_go_resiliency", | ||
| build_file_proto_mode = "disable_global", | ||
|
|
@@ -2173,6 +2183,16 @@ def go_deps(): | |
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/elastic/gosigar/com_github_elastic_gosigar-v0.14.1.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_elazarl_go_bindata_assetfs", | ||
| build_file_proto_mode = "disable_global", | ||
| importpath = "github.com/elazarl/go-bindata-assetfs", | ||
| sha256 = "ee91e4dedf0efd24ddf201e8f8b62f0b79a64efd0d205b30bcd9fa95f905cd15", | ||
| strip_prefix = "github.com/elazarl/[email protected]", | ||
| urls = [ | ||
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/elazarl/go-bindata-assetfs/com_github_elazarl_go_bindata_assetfs-v1.0.1.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_elazarl_goproxy", | ||
| build_file_proto_mode = "disable_global", | ||
|
|
@@ -4107,6 +4127,16 @@ def go_deps(): | |
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/influxdata/usage-client/com_github_influxdata_usage_client-v0.0.0-20160829180054-6d3895376368.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_irfansharif_recorder", | ||
| build_file_proto_mode = "disable_global", | ||
| importpath = "github.com/irfansharif/recorder", | ||
| sha256 = "4a2f085d5339eba18558059c51110de1ff6d9ab8389ece8818fd2f62b7b2e7ab", | ||
| strip_prefix = "github.com/irfansharif/[email protected]", | ||
| urls = [ | ||
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/irfansharif/recorder/com_github_irfansharif_recorder-v0.0.0-20211218081646-a21b46510fd6.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_iris_contrib_blackfriday", | ||
| build_file_proto_mode = "disable_global", | ||
|
|
@@ -4567,6 +4597,16 @@ def go_deps(): | |
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/julienschmidt/httprouter/com_github_julienschmidt_httprouter-v1.3.0.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_julusian_godocdown", | ||
| build_file_proto_mode = "disable_global", | ||
| importpath = "github.com/Julusian/godocdown", | ||
| sha256 = "1bd26f1d29b20d40b3eb0a5678691a2e6e153c473efe079b8b1bbd97a7cc1f57", | ||
| strip_prefix = "github.com/Julusian/[email protected]", | ||
| urls = [ | ||
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Julusian/godocdown/com_github_julusian_godocdown-v0.0.0-20170816220326-6d19f8ff2df8.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_jung_kurt_gofpdf", | ||
| build_file_proto_mode = "disable_global", | ||
|
|
@@ -6603,6 +6643,16 @@ def go_deps(): | |
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/retailnext/hllpp/com_github_retailnext_hllpp-v1.0.1-0.20180308014038-101a6d2f8b52.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_robertkrimen_godocdown", | ||
| build_file_proto_mode = "disable_global", | ||
| importpath = "github.com/robertkrimen/godocdown", | ||
| sha256 = "789ed4a63a797e0dbac7c358eafa8fec4c9885f67ee61da941af4bad2d8c3b55", | ||
| strip_prefix = "github.com/robertkrimen/[email protected]", | ||
| urls = [ | ||
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/robertkrimen/godocdown/com_github_robertkrimen_godocdown-v0.0.0-20130622164427-0bfa04905481.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_robfig_cron_v3", | ||
| build_file_proto_mode = "disable_global", | ||
|
|
@@ -7093,6 +7143,16 @@ def go_deps(): | |
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/stefanberger/go-pkcs11uri/com_github_stefanberger_go_pkcs11uri-v0.0.0-20201008174630-78d3cae3a980.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_stephens2424_writerset", | ||
| build_file_proto_mode = "disable_global", | ||
| importpath = "github.com/stephens2424/writerset", | ||
| sha256 = "a5444ddf04cda5666c4511e5ca793a80372d560376c4193a1fa2e2294d0760dc", | ||
| strip_prefix = "github.com/stephens2424/[email protected]", | ||
| urls = [ | ||
| "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/stephens2424/writerset/com_github_stephens2424_writerset-v1.0.2.zip", | ||
| ], | ||
| ) | ||
| go_repository( | ||
| name = "com_github_stoewer_go_strcase", | ||
| build_file_proto_mode = "disable_global", | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.