-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
67325: build: add TeamCity build script to stress roachtests r=tbg a=erikgrinaker ### roachtest: handle multiple runs of same test on TeamCity The TeamCity markers output via `--teamcity` currently do not handle multiple runs of the same test via `--count`, since they use the same flow ID for all runs, making TeamCity unable to distinguish between them. A similar problem exists with artifacts, where each run would publish the root `artifacts/` directory rather than the `run_` subdirectories. This patch fixes the above issues by including run identifiers where appropriate. Release note: None ### roachtest: add roachtest/noop test This adds a `roachtest/noop` test that instantly succeeds without spinning up any servers, and a `roachtest/noop-maybefail` test that randomly fails with 20% probability. These can be useful when testing e.g. CI builds or other test infrastructure. They are disabled by default and require the `roachtest` tag. Release note: None ### roachtest: add --disable-issue flag to disable GitHub issues Release note: None ### build: add TeamCity build script to stress roachtests This adds a build script that can be used to stress roachtests on TeamCity, in order to reproduce failures. The corresponding TeamCity build is `Cockroach_Nightlies_RoachtestStress`. Release note: None ### build: explicitly build binary for TeamCity nightly tests The TeamCity nightly tests depend on the "publish bleeding edge" build, which pulls in a ton of additional builds (e.g. for multiple OSes) that we don't really need to run the nightlies. This can cause unrelated build failures to fail the nightly tests. This patch explicitly builds a CRDB binary in `teamcity-nightly-roachtest.sh` such that the build can be decoupled from "publish bleeding edge". This is a temporary mitigation until the TeamCity builds and dependencies are cleaned up. Touches https://cockroachlabs.atlassian.net/browse/DEVINF-16. Touches #64263. Release note: None --- This is still a work-in-progress, but I suggest we merge the script itself which makes it easier to generalize the TeamCity build and work on e.g. branch specification. There's a bit of work remaining on the actual build, which will be addressed separately: * Notifications: TeamCity does not expose the email address of the person who started the build, so for now we'll send notifications to the Slack channel `#roachtest-stress-ops`. * Running clusters: We need to let the user know about any test clusters that are still running, so that these can be inspected and shut down as appropriate. We'll probably also need to change how roachtest extends cluster lifetimes, since we want to reuse clusters for tests but make sure every test invocation will extend the lifetime to 36 hours from the start of the test (not the cluster creation, which is currently the case). * Trigger link: We should include a link on test failures to a TeamCity form for starting a stress build (if possible), or at least instructions for how to start one. * Branch/commit selection: We need to make sure it's easy for users to pick which branch or commit to build. 68134: tenantcostserver: add TenantID to TokenBucketRequest r=RaduBerinde a=RaduBerinde This should address the "error issuing TokenBucket RPC" loop observed in #65830. Informs #65830. #### tenantcostserver: add Error field This change adds an Error field to TokenBucketResponse, allowing the Connector to differentiate between RPC errors and logical errors (similar to RangeLookup). This fixes an infinite retry loop caused by a logical issue. Release note: None #### tenantcostserver: add TenantID to TokenBucketRequest We are using TenantFromContext to identify the tenant from which a TenantBucketRequest is coming from (similarly to other connector APIs). Unfortunately, this does not work when the connections are insecure (as is the case in some tests and possibly when testing manually during development). This commit adds a TenantID field. It is cross-checked against the tenant when running in secure mode. Release note: None Co-authored-by: Erik Grinaker <[email protected]> Co-authored-by: Radu Berinde <[email protected]>
- Loading branch information
Showing
23 changed files
with
821 additions
and
599 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
#!/usr/bin/env bash | ||
set -euo pipefail | ||
|
||
source "$(dirname "${0}")/teamcity-support.sh" | ||
|
||
google_credentials="$GOOGLE_EPHEMERAL_CREDENTIALS" | ||
generate_ssh_key | ||
log_into_gcloud | ||
|
||
set -x | ||
|
||
export ROACHPROD_USER=teamcity | ||
|
||
# TODO(erikgrinaker): We should use a dedicated or more appropriate project | ||
# here, but use andrei-jepsen for now. We don't want to use the regular test | ||
# project here to avoid disturbing test runs due to e.g. quota limits. See: | ||
# https://cockroachlabs.atlassian.net/browse/DEVINF-140 | ||
export GCE_PROJECT=${GCE_PROJECT-andrei-jepsen} | ||
|
||
mkdir -p artifacts | ||
|
||
build/builder/mkrelease.sh amd64-linux-gnu build bin/workload bin/roachtest bin/roachprod \ | ||
> artifacts/build.txt 2>&1 || (cat artifacts/build.txt; false) | ||
|
||
build/teamcity-roachtest-invoke.sh \ | ||
--cloud=gce \ | ||
--zones=us-central1-b,us-west1-b,europe-west2-b \ | ||
--debug="${DEBUG-false}" \ | ||
--count="${COUNT-16}" \ | ||
--parallelism="${PARALLELISM-16}" \ | ||
--cpu-quota="${CPUQUOTA-1024}" \ | ||
--cluster-id="${TC_BUILD_ID}" \ | ||
--build-tag="${BUILD_TAG}" \ | ||
--create-args="--lifetime=36h" \ | ||
--cockroach="${PWD}/cockroach-linux-2.6.32-gnu-amd64" \ | ||
--artifacts="${PWD}/artifacts" \ | ||
--disable-issue \ | ||
"${TESTS}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -68,6 +68,7 @@ var ( | |
clusterWipe bool | ||
zonesF string | ||
teamCity bool | ||
disableIssue bool | ||
) | ||
|
||
type encryptValue string | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.