Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
87938: bazci: move the logic of posting github issues to bazci r=srosenberg a=healthy-pod

This code change moves the logic of filtering tests JSON output files and posting github issues to bazci.

Release note: None
Epic CRDB-15060

89257: upgrades: remove unused struct field r=stevendanna a=stevendanna

Epic: None

Release note: None

89873: cliccl: add `encryption-registry-list` command r=jbowenns a=nicktrav

The existing `enc_util` package contains a tool that could be used to dump the files in an encryption registry. This command has been broken since the file registry format was updated.

Add the `(*PebbleFileRegistry).List` function, that returns a map of files in the registry. Adapt existing test cases.

Add a `debug encryption-registry-list` command that will print all files contained in the registry of an encrypted store. This is useful for debugging which store / data key was used to encrypt each file, replacing the equivalent functionality in `enc_util`.

Touches: #89095.
Epic: None.

Release note (ops change): Adds a new command that can be used by an operator to list the files present in the Encryption-At-Rest file registry.

89988: rangedesciter: carve out library for range desc iteration r=irfansharif a=irfansharif

Informs #87503; pure code-movement. Going to use it in future commits as part of multi-tenant replication reports (#89987) where we'll need to iterate over the set of range descriptors.

Release note: None

Co-authored-by: healthy-pod <[email protected]>
Co-authored-by: Steven Danna <[email protected]>
Co-authored-by: Nick Travers <[email protected]>
Co-authored-by: irfan sharif <[email protected]>
  • Loading branch information
5 people committed Oct 14, 2022
5 parents 4aaf5e9 + 7122bac + e158acf + 0d467be + 75688c2 commit b0ed2bf
Show file tree
Hide file tree
Showing 61 changed files with 1,802 additions and 1,363 deletions.
2 changes: 1 addition & 1 deletion build/bazelutil/check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ pkg/util/timeutil/zoneinfo.go://go:generate go run gen/main.go
EXISTING_BROKEN_TESTS_IN_BAZEL="
pkg/acceptance/BUILD.bazel
pkg/cmd/cockroach-oss/BUILD.bazel
pkg/cmd/github-post/BUILD.bazel
pkg/cmd/bazci/githubpost/BUILD.bazel
pkg/cmd/prereqs/BUILD.bazel
pkg/cmd/roachtest/BUILD.bazel
pkg/cmd/teamcity-trigger/BUILD.bazel
Expand Down
68 changes: 0 additions & 68 deletions build/teamcity-bazel-support.sh
Original file line number Diff line number Diff line change
Expand Up @@ -47,71 +47,3 @@ _tc_release_branch() {
branch=$(_tc_build_branch)
[[ "$branch" == master || "$branch" == release-* || "$branch" == provisional_* ]]
}

# process_test_json processes logs and submits failures to GitHub
# Requires GITHUB_API_TOKEN set for the release branches.
# Accepts 5 arguments:
# testfilter: path to the `testfilter` executable, usually
# `$BAZEL_BIN/pkg/cmd/testfilter/testfilter_/testfilter`
# github_post: path to the `github-post` executable, usually
# `$BAZEL_BIN/pkg/cmd/github-post/github-post_/github-post`
# artifacts_dir: usually `/artifacts`
# test_json: path to test's JSON output, usually generated by `rules_go`'s and
# `GO_TEST_JSON_OUTPUT_FILE`.
# create_tarball: whether to create a tarball with full logs. If the test's
# exit code is passed, the tarball is generated on failures.
#
# The variable BAZEL_SUPPORT_EXTRA_GITHUB_POST_ARGS can be set to add extra
# arguments to $github_post.
process_test_json() {
local testfilter=$1
local github_post=$2
local artifacts_dir=$3
local test_json=$4
local create_tarball=$5

$testfilter -mode=strip < "$test_json" | $testfilter -mode=omit | $testfilter -mode=convert > "$artifacts_dir"/failures.txt
failures_size=$(stat --format=%s "$artifacts_dir"/failures.txt)
if [ $failures_size = 0 ]; then
rm -f "$artifacts_dir"/failures.txt
fi

if _tc_release_branch; then
if [ -z "${GITHUB_API_TOKEN-}" ]; then
# GITHUB_API_TOKEN must be in the env or github-post will barf if it's
# ever asked to post, so enforce that on all runs.
# The way this env var is made available here is quite tricky. The build
# calling this method is usually a build that is invoked from PRs, so it
# can't have secrets available to it (for the PR could modify
# build/teamcity-* to leak the secret). Instead, we provide the secrets
# to a higher-level job (Publish Bleeding Edge) and use TeamCity magic to
# pass that env var through when it's there. This means we won't have the
# env var on PR builds, but we'll have it for builds that are triggered
# from the release branches.
echo "GITHUB_API_TOKEN must be set"
exit 1
else
$github_post ${BAZEL_SUPPORT_EXTRA_GITHUB_POST_ARGS:+$BAZEL_SUPPORT_EXTRA_GITHUB_POST_ARGS} < "$test_json"
fi
fi

if [ "$create_tarball" -ne 0 ]; then
# Keep the debug file around for failed builds. Compress it to avoid
# clogging the agents with stuff we'll hopefully rarely ever need to
# look at.
# If the process failed, also save the full human-readable output. This is
# helpful in cases in which tests timed out, where it's difficult to blame
# the failure on any particular test. It's also a good alternative to poking
# around in test.json.txt itself when anything else we don't handle well happens,
# whatever that may be.
$testfilter -mode=convert < "$test_json" > "$artifacts_dir"/full_output.txt
(cd "$artifacts_dir" && tar --strip-components 1 -czf full_output.tgz full_output.txt $(basename $test_json))
rm -rf "$artifacts_dir"/full_output.txt
fi

# Some unit tests test automatic ballast creation. These ballasts can be
# larger than the maximum artifact size. Remove any artifacts with the
# EMERGENCY_BALLAST filename.
find "$artifacts_dir" -name "EMERGENCY_BALLAST" -delete
}

4 changes: 4 additions & 0 deletions build/teamcity-testrace.sh
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ tc_end_block "Compile C dependencies"
TESTTIMEOUT=${TESTTIMEOUT:-45m}

for pkg in $pkgspec; do
# Skip known-bad tests (generally Bazel-specific ones).
if [[ "$pkg" == "./pkg/cmd/bazci" ]]; then
continue
fi
tc_start_block "Run ${pkg} under race detector"
run_json_test build/builder.sh env \
COCKROACH_LOGIC_TESTS_SKIP=true \
Expand Down
14 changes: 2 additions & 12 deletions build/teamcity/cockroach/nightlies/cloud_unit_tests_impl.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,12 @@
set -xeuo pipefail

dir="$(dirname $(dirname $(dirname $(dirname "${0}"))))"
source "$dir/teamcity-bazel-support.sh" # For process_test_json
source "$dir/teamcity-support.sh" # For log_into_gcloud

bazel build //pkg/cmd/bazci //pkg/cmd/github-post //pkg/cmd/testfilter --config=ci
bazel build //pkg/cmd/bazci --config=ci
BAZEL_BIN=$(bazel info bazel-bin --config=ci)

ARTIFACTS_DIR=/artifacts
GO_TEST_JSON_OUTPUT_FILE=$ARTIFACTS_DIR/test.json.txt

google_credentials="$GOOGLE_EPHEMERAL_CREDENTIALS"
log_into_gcloud
Expand All @@ -25,11 +23,10 @@ export AWS_CONFIG_FILE="$PWD/.aws/config"
log_into_aws

exit_status=0
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci test -- --config=ci \
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci --process_test_failures -- test --config=ci \
//pkg/cloud/gcp:gcp_test //pkg/cloud/amazon:amazon_test //pkg/ccl/cloudccl/gcp:gcp_test //pkg/ccl/cloudccl/amazon:amazon_test \
--test_env=GO_TEST_WRAP_TESTV=1 \
--test_env=GO_TEST_WRAP=1 \
--test_env=GO_TEST_JSON_OUTPUT_FILE=$GO_TEST_JSON_OUTPUT_FILE \
--test_env=GOOGLE_CREDENTIALS_JSON="$GOOGLE_EPHEMERAL_CREDENTIALS" \
--test_env=GOOGLE_APPLICATION_CREDENTIALS="$GOOGLE_APPLICATION_CREDENTIALS" \
--test_env=GOOGLE_BUCKET="nightly-cloud-unit-tests" \
Expand All @@ -51,11 +48,4 @@ $BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci test -- --config=ci \
--test_timeout=900 \
|| exit_status=$?

process_test_json \
$BAZEL_BIN/pkg/cmd/testfilter/testfilter_/testfilter \
$BAZEL_BIN/pkg/cmd/github-post/github-post_/github-post \
$ARTIFACTS_DIR \
$GO_TEST_JSON_OUTPUT_FILE \
$exit_status

exit $exit_status
12 changes: 2 additions & 10 deletions build/teamcity/cockroach/nightlies/compose.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ source "$dir/teamcity-bazel-support.sh"

tc_start_block "Run compose tests"

bazel build //pkg/cmd/bazci //pkg/cmd/github-post //pkg/cmd/testfilter --config=ci
bazel build //pkg/cmd/bazci --config=ci
BAZEL_BIN=$(bazel info bazel-bin --config=ci)
BAZCI=$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci

Expand All @@ -18,24 +18,16 @@ COCKROACH=$CROSSBIN/pkg/cmd/cockroach/cockroach_/cockroach
COMPAREBIN=$CROSSBIN/pkg/compose/compare/compare/compare_test_/compare_test
ARTIFACTS_DIR=$PWD/artifacts
mkdir -p $ARTIFACTS_DIR
GO_TEST_JSON_OUTPUT_FILE=$ARTIFACTS_DIR/test.json.txt

exit_status=0
$BAZCI --artifacts_dir=$ARTIFACTS_DIR -- \
$BAZCI --process_test_failures --artifacts_dir=$ARTIFACTS_DIR -- \
test --config=ci //pkg/compose:compose_test \
"--sandbox_writable_path=$ARTIFACTS_DIR" \
"--test_tmpdir=$ARTIFACTS_DIR" \
--test_env=GO_TEST_WRAP_TESTV=1 \
--test_env=GO_TEST_JSON_OUTPUT_FILE=$GO_TEST_JSON_OUTPUT_FILE \
--test_arg -cockroach --test_arg $COCKROACH \
--test_arg -compare --test_arg $COMPAREBIN \
--test_timeout=1800 || exit_status=$?
process_test_json \
$BAZEL_BIN/pkg/cmd/testfilter/testfilter_/testfilter \
$BAZEL_BIN/pkg/cmd/github-post/github-post_/github-post \
$ARTIFACTS_DIR \
$GO_TEST_JSON_OUTPUT_FILE \
$exit_status

tc_end_block "Run compose tests"
exit $exit_status
10 changes: 1 addition & 9 deletions build/teamcity/cockroach/nightlies/lint_urls_impl.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,8 @@
set -xeuo pipefail

dir="$(dirname $(dirname $(dirname $(dirname "${0}"))))"
source "$dir/teamcity-bazel-support.sh" # For process_test_json

bazel build //pkg/cmd/bazci //pkg/cmd/github-post //pkg/cmd/testfilter --config=ci
bazel build //pkg/cmd/bazci --config=ci
BAZEL_BIN=$(bazel info bazel-bin --config=ci)
GO_TEST_JSON_OUTPUT_FILE=/artifacts/test.json.txt
exit_status=0
Expand All @@ -15,10 +14,3 @@ XML_OUTPUT_FILE=/artifacts/test.xml GO_TEST_WRAP_TESTV=1 GO_TEST_WRAP=1 GO_TEST_
# The schema of the output test.xml will be slightly wrong -- ask `bazci` to fix
# it up.
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci munge-test-xml /artifacts/test.xml
process_test_json \
$BAZEL_BIN/pkg/cmd/testfilter/testfilter_/testfilter \
$BAZEL_BIN/pkg/cmd/github-post/github-post_/github-post \
/artifacts \
$GO_TEST_JSON_OUTPUT_FILE \
$exit_status
exit $exit_status
25 changes: 5 additions & 20 deletions build/teamcity/cockroach/nightlies/optimizer_tests_impl.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,27 +3,19 @@
set -xeuo pipefail

dir="$(dirname $(dirname $(dirname $(dirname "${0}"))))"
source "$dir/teamcity-bazel-support.sh"
source "$dir/teamcity/util.sh"

bazel build //pkg/cmd/bazci //pkg/cmd/github-post //pkg/cmd/testfilter --config=ci
bazel build //pkg/cmd/bazci --config=ci
BAZEL_BIN=$(bazel info bazel-bin --config=ci)

tc_start_block "Run opt tests with fast_int_set_large"
ARTIFACTS_DIR=/artifacts/fast_int_set_large
mkdir $ARTIFACTS_DIR
GO_TEST_JSON_OUTPUT_FILE=$ARTIFACTS_DIR/test.json.txt
exit_status_large=0
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci --artifacts_dir $ARTIFACTS_DIR -- \
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci --artifacts_dir $ARTIFACTS_DIR --process_test_failures -- \
test //pkg/sql/opt:opt_test --config=ci \
--define gotags=bazel,crdb_test,fast_int_set_large \
--test_env=GO_TEST_JSON_OUTPUT_FILE=$GO_TEST_JSON_OUTPUT_FILE || exit_status_large=$?
process_test_json \
$BAZEL_BIN/pkg/cmd/testfilter/testfilter_/testfilter \
$BAZEL_BIN/pkg/cmd/github-post/github-post_/github-post \
$ARTIFACTS_DIR \
$GO_TEST_JSON_OUTPUT_FILE \
$exit_status_large
|| exit_status_large=$?
tc_end_block "Run opt tests with fast_int_set_large"

# NOTE(ricky): Running both tests in the same configuration with different
Expand All @@ -32,19 +24,12 @@ tc_end_block "Run opt tests with fast_int_set_large"
tc_start_block "Run opt tests with fast_int_set_small"
ARTIFACTS_DIR=/artifacts/fast_int_set_small
mkdir $ARTIFACTS_DIR
GO_TEST_JSON_OUTPUT_FILE=$ARTIFACTS_DIR/test.json.txt
exit_status_small=0
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci --artifacts_dir $ARTIFACTS_DIR -- \
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci --artifacts_dir $ARTIFACTS_DIR --process_test_failures -- \
test --config=ci \
//pkg/sql/opt:opt_test \
--define gotags=bazel,crdb_test,fast_int_set_small \
--test_env=GO_TEST_JSON_OUTPUT_FILE=$GO_TEST_JSON_OUTPUT_FILE || exit_status_small=$?
process_test_json \
$BAZEL_BIN/pkg/cmd/testfilter/testfilter_/testfilter \
$BAZEL_BIN/pkg/cmd/github-post/github-post_/github-post \
$ARTIFACTS_DIR \
$GO_TEST_JSON_OUTPUT_FILE \
$exit_status_large
|| exit_status_small=$?
tc_end_block "Run opt tests with fast_int_set_small"

if [ $exit_status_large -ne 0 ]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,37 +1,29 @@
#!/usr/bin/env bash

dir="$(dirname $(dirname $(dirname $(dirname "${0}"))))"
source "$dir/teamcity-bazel-support.sh" # For process_test_json

set -euxo pipefail
ARTIFACTS_DIR=/artifacts/meta
mkdir -p $ARTIFACTS_DIR
GO_TEST_JSON_OUTPUT_FILE=/artifacts/test.json.txt

echo "TC_SERVER_URL is $TC_SERVER_URL"

bazel build //pkg/cmd/bazci //pkg/cmd/github-post //pkg/cmd/testfilter --config=ci
bazel build //pkg/cmd/bazci --config=ci

BAZEL_BIN=$(bazel info bazel-bin --config ci)

exit_status=0
# NB: If adjusting the metamorphic test flags below, be sure to also update
# pkg/cmd/github-post/main.go to ensure the GitHub issue poster includes the
# correct flags in the reproduction command.
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci -- test --config=ci \
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci --process_test_failures --formatter=pebble-metamorphic -- test --config=ci \
@com_github_cockroachdb_pebble//internal/metamorphic:metamorphic_test \
--test_timeout=25200 '--test_filter=TestMeta$' \
--define gotags=bazel,invariants \
"--test_env=GO_TEST_JSON_OUTPUT_FILE=$GO_TEST_JSON_OUTPUT_FILE" \
--run_under "@com_github_cockroachdb_stress//:stress -bazel -shardable-artifacts 'GO_TEST_JSON_OUTPUT_FILE=cat,XML_OUTPUT_FILE=$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci merge-test-xmls' -maxtime 6h -maxfails 1 -stderr -p 1" \
--run_under "@com_github_cockroachdb_stress//:stress -bazel -shardable-artifacts 'XML_OUTPUT_FILE=$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci merge-test-xmls' -maxtime 6h -maxfails 1 -stderr -p 1" \
--test_arg -dir --test_arg $ARTIFACTS_DIR \
--test_arg -ops --test_arg "uniform:5000-10000" \
--test_output streamed \
|| exit_status=$?

BAZEL_SUPPORT_EXTRA_GITHUB_POST_ARGS=--formatter=pebble-metamorphic process_test_json \
$BAZEL_BIN/pkg/cmd/testfilter/testfilter_/testfilter \
$BAZEL_BIN/pkg/cmd/github-post/github-post_/github-post \
/artifacts $GO_TEST_JSON_OUTPUT_FILE $exit_status

exit $exit_status
Original file line number Diff line number Diff line change
@@ -1,37 +1,29 @@
#!/usr/bin/env bash

dir="$(dirname $(dirname $(dirname $(dirname "${0}"))))"
source "$dir/teamcity-bazel-support.sh" # For process_test_json

set -euxo pipefail
ARTIFACTS_DIR=/artifacts/meta
mkdir -p $ARTIFACTS_DIR
GO_TEST_JSON_OUTPUT_FILE=/artifacts/test.json.txt

echo "TC_SERVER_URL is $TC_SERVER_URL"

bazel build //pkg/cmd/bazci //pkg/cmd/github-post //pkg/cmd/testfilter --config=ci
bazel build //pkg/cmd/bazci --config=ci

BAZEL_BIN=$(bazel info bazel-bin --config ci)

exit_status=0
# NB: If adjusting the metamorphic test flags below, be sure to also update
# pkg/cmd/github-post/main.go to ensure the GitHub issue poster includes the
# correct flags in the reproduction command.
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci -- test --config=race --config=ci \
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci --process_test_failures --formatter=pebble-metamorphic -- test --config=race --config=ci \
@com_github_cockroachdb_pebble//internal/metamorphic:metamorphic_test \
--test_timeout=14400 '--test_filter=TestMeta$' \
--define gotags=bazel,invariants \
"--test_env=GO_TEST_JSON_OUTPUT_FILE=$GO_TEST_JSON_OUTPUT_FILE" \
--run_under "@com_github_cockroachdb_stress//:stress -bazel -shardable-artifacts 'GO_TEST_JSON_OUTPUT_FILE=cat,XML_OUTPUT_FILE=$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci merge-test-xmls' -maxtime 3h -maxfails 1 -stderr -p 1" \
--run_under "@com_github_cockroachdb_stress//:stress -bazel -shardable-artifacts 'XML_OUTPUT_FILE=$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci merge-test-xmls' -maxtime 3h -maxfails 1 -stderr -p 1" \
--test_arg -dir --test_arg $ARTIFACTS_DIR \
--test_arg -ops --test_arg "uniform:5000-10000" \
--test_output streamed \
|| exit_status=$?

BAZEL_SUPPORT_EXTRA_GITHUB_POST_ARGS=--formatter=pebble-metamorphic process_test_json \
$BAZEL_BIN/pkg/cmd/testfilter/testfilter_/testfilter \
$BAZEL_BIN/pkg/cmd/github-post/github-post_/github-post \
/artifacts $GO_TEST_JSON_OUTPUT_FILE $exit_status

exit $exit_status
15 changes: 4 additions & 11 deletions build/teamcity/cockroach/nightlies/random_syntax_tests_impl.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,15 @@
set -xeuo pipefail

dir="$(dirname $(dirname $(dirname $(dirname "${0}"))))"
source "$dir/teamcity-bazel-support.sh"

bazel build //pkg/cmd/bazci //pkg/cmd/github-post //pkg/cmd/testfilter --config=ci
bazel build //pkg/cmd/bazci --config=ci
BAZEL_BIN=$(bazel info bazel-bin --config=ci)
GO_TEST_JSON_OUTPUT_FILE=/artifacts/test.json.txt
exit_status=0
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci -- test --config=ci \
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci --process_test_failures -- test --config=ci \
//pkg/sql/tests:tests_test \
--test_arg -rsg=5m --test_arg -rsg-routines=8 --test_arg -rsg-exec-timeout=1m \
--test_timeout 3600 --test_filter 'TestRandomSyntax' \
--test_sharding_strategy=disabled \
--test_env=GO_TEST_JSON_OUTPUT_FILE=$GO_TEST_JSON_OUTPUT_FILE || exit_status=$?
process_test_json \
$BAZEL_BIN/pkg/cmd/testfilter/testfilter_/testfilter \
$BAZEL_BIN/pkg/cmd/github-post/github-post_/github-post \
/artifacts \
$GO_TEST_JSON_OUTPUT_FILE \
$exit_status
|| exit_status=$?

exit $exit_status
15 changes: 4 additions & 11 deletions build/teamcity/cockroach/nightlies/sqlite_logic_test_impl.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,13 @@
set -xeuo pipefail

dir="$(dirname $(dirname $(dirname $(dirname "${0}"))))"
source "$dir/teamcity-bazel-support.sh"

bazel build //pkg/cmd/bazci //pkg/cmd/github-post //pkg/cmd/testfilter --config=ci
bazel build //pkg/cmd/bazci --config=ci
BAZEL_BIN=$(bazel info bazel-bin --config=ci)
GO_TEST_JSON_OUTPUT_FILE=/artifacts/test.json.txt
exit_status=0
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci -- test --config=ci --config=crdb_test_off \
$BAZEL_BIN/pkg/cmd/bazci/bazci_/bazci --process_test_failures -- test --config=ci --config=crdb_test_off \
//pkg/sql/sqlitelogictest/tests/... \
--test_arg -bigtest --test_arg -flex-types --test_timeout 86400 \
--test_env=GO_TEST_JSON_OUTPUT_FILE=$GO_TEST_JSON_OUTPUT_FILE || exit_status=$?
process_test_json \
$BAZEL_BIN/pkg/cmd/testfilter/testfilter_/testfilter \
$BAZEL_BIN/pkg/cmd/github-post/github-post_/github-post \
/artifacts \
$GO_TEST_JSON_OUTPUT_FILE \
$exit_status
|| exit_status=$?

exit $exit_status
Loading

0 comments on commit b0ed2bf

Please sign in to comment.