testing: start using btlr

[tmatsuo]

fixes #3958
fixes #3994
fixes #3912
fixes #3992
fixes #3996
fixes #3997
fixes #3998
fixes #3999
fixes #4000
fixes #4001
fixes #4002

We embed btlr binary from gs://cloud-devrel-kokoro-resources/btlr/v0.0.1/btlr

[tmatsuo]

For the record, here are latencies for the last periodic builds:

lint: 14:26
py2.7: 24:26
py3.6: 3 hr, 27 min, 8 sec
py3.7: 3 hr, 35 min, 34 sec
py3.8: 3 hr, 21 min, 28 sec

Here are latencies for the presubmit builds for this PR:
(will update here once the builds finish)
lint: 10:01
py2.7: 20:16
py3.6: 1 hr, 15 min, 57 sec
py3.7: 1 hr, 0 min, 48 sec
py3.8: 55 min, 53 sec

and interestingly, the test in appengine/flexible/websockets fails with btlr.

[tmatsuo]

Spanner is failing with:

google.api_core.exceptions.GoogleAPICallError: None Project 1012616486416 cannot add 1 nodes in region us-central1.

It seems like there are too many instances.

[tmatsuo]

There were around 10 stale Spanner instances in the main test project. I deleted them all.

I'll restart the py-3.6 build.

[tmatsuo]

The number of tests for the current presubmit builds are exactly the same as the last periodic builds.
py-2.7: 20
py-3.6: 212
py-3.7: 212
py-3.8: 209

Latencies:
Lint: 8:55
py-2.7: 21:25
py-3.6: 47:51
py-3.7: 52:08
py-3.8: 56:07

Pretty good :)

[tmatsuo]

Only failing test is dlp/risk_test.py in py-3.8 among 4 builds.

[tmatsuo]

I'll update the branch after merging some more PRs. Let's be extra cautious before merging this.

[tmatsuo]

3 failures:

iot:

Traceback (most recent call last):
  File "/workspace/iot/api-client/manager/manager_test.py", line 57, in clean_up_registries
    test_date = datetime.datetime.utcfromtimestamp(int(time_str))
ValueError: invalid literal for int() with base 10: 'b32153211df0'

It looks like a bug in the cleanup code.

dlp:

Traceback (most recent call last):
  File "/workspace/dlp/inspect_content_test.py", line 437, in test_inspect_datastore_no_results
    assert "No findings" in out
AssertionError: assert 'No findings' in 'Inspection operation started: projects/python-docs-samples-tests/locations/global/dlpJobs/i-7027593330947527120\nNo event received before the timeout. Please verify that the subscription provided is subscribed to the topic provided.\n'

This is a known flake.

run/markdown-preview:

Traceback (most recent call last):
  File "/workspace/run/markdown-preview/e2e_test.py", line 35, in services
    subprocess.run(
  File "/usr/local/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['gcloud', 'builds', 'submit', '--project', 'python-docs-samples-tests', '--substitutions', '_SUFFIX=7837734e97b844a9a0b587057d8797e2', '--config', 'e2e_test_setup.yaml', '--quiet']' returned non-zero exit status 1.

This is the most concerning one. gcloud sdk crashed, probably because of the low memory situation.

[tmatsuo]

This time inspect_content_test.py::test_inspect_datastore_no_results failed.
Maybe I can bump the wait time for those tests.

For run/markdown-preview,

ERROR: gcloud crashed (FileNotFoundError): [Errno 2] No such file or directory: 'renderer/noxfile.py'

I think I understand the problem. This is happening because

1. When first gcloud scan the filesystem, there is a file renderer/noxfile.py
2. Then the test for run/markdown-preview/renderer finishes and the cleanup code delete that file.
3. After that gcloud tries to archive the file for uploading, but that file in question is gone.

I'll put noxfile.py in the child directories and it should be fine.

[kurtisvg]

You can also do rm -f $file, which won't fail even if the file doesn't exist

[tmatsuo]

@kurtisvg It's not failing to delete the file. gcloud crashes because there is not the file existing any more which used to be there.

[kurtisvg]

Ah sorry, I misunderstood. I think there is a .gitignore equivalent for Cloud Run/Cloud Build as well that might help.

[tmatsuo]

@kurtisvg

Thanks, yeah maybe we can use .gcloudignore. However, considering we'll sprinkle noxfile.py anyways soon, I'd do it now.

[tmatsuo]

Seeing some new failures, filed #3994 and #3995
At a glance, they seems to be server side errors. Let me re-run the builds.

[tmatsuo]

For iam failure, I think it is sensible to display description only when the key exists anyways.

Actually, I found that there is a role roles/accesscontextmanager.vpcScTroubleshooterViewer which doesn't have the description.

[tmatsuo]

For iot failure, it's an ongoing flake tracked at #3912, so I don't worry about it too much.

[tmatsuo]

Failure analysis:

video/cloud-client/analyze
beta_snippets_test.test_annotation_to_storage_streaming

This test is using an intact uuid1 value as the bucket name. Since we now running the tests in parallel, other tests could get the same uuid1. Although it's low probability, it's still way higher than the probability when we use uuid4.

bucket_name = str(uuid.uuid1())

There are several cases of this:

$ grep -r 'bucket_name = str(uuid.uuid1())' *
translate/automl/translate_v3_batch_translate_text_with_model_test.py:    bucket_name = str(uuid.uuid1())
translate/cloud-client/beta_snippets_test.py:    bucket_name = str(uuid.uuid1())
translate/cloud-client/translate_v3_batch_translate_text_with_glossary_test.py:    bucket_name = str(uuid.uuid1())
video/cloud-client/analyze/beta_snippets_test.py:    bucket_name = str(uuid.uuid1())

Maybe we can just use uuid4, which will make the probability much less.

py-3.8 failed during pip install:

zipfile.BadZipFile: Bad CRC-32 for file 'uWSGI-2.0.18.data/scripts/uwsgi'

Maybe it's because we run tests in parallel, maybe not.