More tweaks based on writing run_tests.dart

README.md

@@ -1,18 +1,20 @@
 # Flutter Tests
 
-This repository contains references to tests that are run with every commit to
-Flutter to verify that no breaking changes have been introduced.
+This repository contains references to tests that are run with every
+commit to Flutter to verify that no breaking changes have been
+introduced.
 
 ## Adding more tests
 
 You are welcome to add a new batch of tests. To do so, copy the file
-`registry/template.test` to create a new file in the `registry/` directory. Fill
-in the fields and delete all the comments. Then, submit a PR with this new file.
+`registry/template.test` to create a new file in the `registry/`
+directory. Fill in the fields and delete all the comments. Then,
+submit a PR with this new file.
 
 Tests must fulfill the following criteria to be added:
 
-* All the code must be available publicly on GitHub under a license compatible
-  with this effort.
+* All the code must be available publicly on GitHub under a license
+  compatible with this effort.
 
 * Tests must be hermetic. For example, a test should not involve
   network activity, spawn processes, or access the local file system
@@ -21,21 +23,25 @@ Tests must fulfill the following criteria to be added:
 * Tests must be resilient to being run concurrently with other tests,
   including concurrent runs of themselves.
 
-* Tests must be reliable. A test must not claim to pass if it is failing.
-  Running a test multiple times in a row must always have the same result.
+* Tests must be reliable. A test must not claim to pass if it is
+  failing. Running a test multiple times in a row must always have the
+  same result.
 
-* Tests must be as fast as possible given the hardware. For example, tests must
-  not use real timers or depend on the wall clock.
+* Tests must have no output when they are passing.
 
-* The time taken by tests must be proportional to their value. A few thousands
-  tests are expected to run within a few minutes. An upper limit of about five
-  minutes will be applied to each contributed test suite (not including the
-  time to download the tests), but it is expected that most suites will
-  complete in seconds.
+* Tests must be as fast as possible given the hardware. For example,
+  tests must not use real timers or depend on the wall clock.
 
-* The tests must be compatible with any tools for automatically updating
-  Flutter code (e.g. they cannot rely on custom code generation unless such
-  code generation can hook into the automatic update mechanism).
+* The time taken by tests must be proportional to their value. A few
+  thousands tests are expected to run within a few minutes. An upper
+  limit of about five minutes will be applied to each contributed test
+  suite (not including the time to download the tests), but it is
+  expected that most suites will complete in seconds.
+
+* The tests must be compatible with any tools for automatically
+  updating Flutter code (e.g. they cannot rely on custom code
+  generation unless such code generation can hook into the automatic
+  update mechanism).
 
 * The tests must represent good practices as described in Flutter's
   documentation. For example, using an object after calling its
@@ -49,35 +55,41 @@ Tests must fulfill the following criteria to be added:
 
 ## If a test is broken
 
-The point of these tests is to make sure we don't break existing code, while still
-being able to make improvements to Flutter APIs.
+The point of these tests is to make sure we don't break existing code,
+while still being able to make improvements to Flutter APIs.
 
-If you find that a PR you have created causes on these tests to fail, you have the
-following options:
+If you find that a PR you have created causes one these tests to fail,
+you have the following options:
 
-1. Change your PR so that the test no longer fails. This is the preferred option,
-   so long as the result is one we can be proud of. Is the resulting API something
-   that you would plausibly come up with without the backwards-compatibility
-   constraint? That's good. Is the resulting API something that, as soon as you
-   see it, you think "why?" or "that's weird"? That's bad.
+1. Change your PR so that the test no longer fails. This is the
+   preferred option, so long as the result is one we can be proud of.
+   Is the resulting API something that you would plausibly come up
+   with without the backwards-compatibility constraint? That's good.
+   Is the resulting API something that, as soon as you see it, you
+   think "why?" or "that's weird"? That's bad. Consider the advice in
+   the Style guide:
+   https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo
 
 2. Go through the breaking change process, as documented here:
    https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes
-   If you're going to do this, you will need to contact the relevant people
-   responsible for the breaking test(s) (see the relevant .test files), help them
-   fix their code, and update this repository to use the new version of their
-   tests, in addition to the steps described on the wiki.
+   If you're going to do this, you will need to contact the relevant
+   people responsible for the breaking test(s) (see the relevant .test
+   files), help them fix their code, and update this repository to use
+   the new version of their tests, in addition to the steps described
+   on the wiki.
 
-3. Remove the test in question. This is by far the least ideal solution. To go
-   down this path, we must first establish that one of the following is true:
+3. Remove the test in question. This is by far the least ideal
+   solution. To go down this path, we must first establish that one of
+   the following is true:
 
     - the people listed as contacts for the test are not responsive.
 
-    - the test is poorly written (e.g. it contains a race condition or relies on
-      assumptions that violate clearly documented API contracts), and the people
-      listed as contacts are not willing to fix the test or accept fixes for the
-      test.
+    - the test is poorly written (e.g. it contains a race condition or
+      relies on assumptions that violate clearly documented API
+      contracts), and the people listed as contacts are not willing to
+      fix the test or accept fixes for the test.
 
-    - we have gone through the breaking change process cited above, but are unable
-      to update the test accordingly (e.g. the people listed as contacts are not
-      willing to work with us to update their code).
+    - we have gone through the breaking change process cited above,
+      but are unable to update the test accordingly (e.g. the people
+      listed as contacts are not willing to work with us to update
+      their code).

registry/flutter_svg.test

@@ -1,5 +1,5 @@
 [email protected]
 fetch=git clone [email protected]:dnfield/flutter_svg.git tests
 fetch=git -C tests checkout 4687e1122cde7d10bfab2a594f153ce453fd87d7
-directory=.
+update=.
 test=flutter test

registry/template.test

@@ -1,71 +1,99 @@
 # This is a template for adding tests to the flutter/tests registry.
 #
-# To submit a new batch of tests, copy this file to create a new file in
-# this directory.
+# To submit a new batch of tests, copy this file to create a new file
+# in this directory, update it as described below, and submit it as a
+# new PR to this repository.
 #
 # New files should be named `foo.test` where `foo` is a description of
-# the project whose tests are being added.
+# the project whose tests are being added. (The file named
+# `template.test` is skipped automatically.)
 #
-# Lines beginning with a "#" are ignored. Delete all such lines from this
-# template before submitting a new `.test` file (you may add new comments
-# if you have something to say that doesn't fit in one of the meaningful
-# lines). Also delete blank lines like the following:
+# Lines beginning with a "#" are ignored. Delete all such lines from
+# this template before submitting a new `.test` file (you may add new
+# comments if you have something to say that doesn't fit in one of the
+# meaningful lines). Also delete blank lines like the following:
 
-# The contact lines must contain the e-mail addresses, one to a line, of
-# one or more people who will be able to help if a problem is found with
-# a test. People on this list should be able to respond to e-mails within
-# about 72 hours. If there is a problem with a test and we cannot reach
-# any of the people listed, then we will probably remove the tests.
+# The contact lines must contain the e-mail addresses, one to a line,
+# of one or more people who will be able to help if a problem is found
+# with a test. People on this list should be able to respond to
+# e-mails within about 72 hours. If there is a problem with a test and
+# we cannot reach any of the people listed, then we will probably
+# remove the tests.
 [email protected]
 [email protected]
 
 # The fetch line(s) must contain one or more shell commands that fetch
-# the tests into a new directory called "tests" in the current directory.
-# The example below clones the dnfield/flutter_svg GitHub repository then
-# checks out a specific revision.
+# the tests into a new directory called "tests" in the current
+# directory. The example below clones the dnfield/flutter_svg GitHub
+# repository then checks out a specific revision.
 #
 # Our current policy is that all tests should be in public GitHub
 # repositories. (This way, we don't take on additional dependencies on
-# other services that may or not go down. We already depend on GitHub, so
-# if GitHub goes down it doesn't matter that we can't reach contributed
-# tests.) If you have some tests to contribute that are not and will not
-# be in GitHub, please contact [email protected] before submitting a PR
-# proposing to add such tests. We may make exceptions if your tests are
-# especially valuable.
+# other services that may or not go down. We already depend on GitHub,
+# so if GitHub goes down it doesn't matter that we can't reach
+# contributed tests.) If you have some tests to contribute that are
+# not and will not be in GitHub, please contact [email protected] before
+# submitting a PR proposing to add such tests. We may make exceptions
+# if your tests are especially valuable.
 #
-# The checkout must always check out the exact same set of tests. Do not
-# check out the current master branch or some such; always fetch a
+# The checkout must always check out the exact same set of tests. Do
+# not check out the current master branch or some such; always fetch a
 # specific revision. You may regularly submit PRs that update the
 # revision being used.
 #
-# The commands specified on the "fetch" lines must return the exit code 0
-# or else the tests will be skipped entirely. The "git" program will be
-# on the path.
+# The commands specified on the "fetch" lines must return the exit
+# code 0 or else the tests will be skipped entirely. The "git" program
+# will be on the path.
+#
+# There must be at least one "fetch" line. (In fact currently there
+# must be two, and they must follow the pattern below more or less
+# exactly. If this is a problem, let us know in your PR.)
 fetch=git clone [email protected]:dnfield/flutter_svg.git tests
 fetch=git -C tests checkout 4687e1122cde7d10bfab2a594f153ce453fd87d7
 
-# The directory lines specify paths (relative to the newly created
-# "tests" directory) in which to run a flutter command to automatically
-# update the code to fix breaking changes. The command will be run in
-# turn in each specified directory. The root can be specified by using
-# the path "." as that represents the "current" directory. Multiple paths
-# may be specified by specifying multiple "directory" lines.
-directory=.
+# The "update" lines specify paths (relative to the newly created
+# "tests" directory) in which to run a flutter command to
+# automatically update the code to fix breaking changes. The command
+# will be run in turn in each specified directory. The root can be
+# specified by using the path "." as that represents the "current"
+# directory. Multiple paths may be specified by specifying multiple
+# "update" lines.
+#
+# The command that updates the code may fail, in which case the tests
+# are considered to have failed. This helps catch cases where
+# automatically updating the code doesn't work.
+#
+# There must be at least one "update" line.
+update=.
 
 # The test lines specify commands to run to execute the actual tests.
-# Each of these should return either 0 to indicate success, or a non-zero
-# exit code to report failure.
+# Each of these should return either 0 to indicate success, or a
+# non-zero exit code to report failure.
 #
 # The "flutter" tool and the "dart" program will be in the path.
 #
 # If there are multiple directories, create a shell script in the
-# repository to enter each directory and run the tests in that directory,
-# returning non-zero as soon as any set of tests fails. You can have
-# multiple such scripts. In the example below, we run "flutter test" in
-# the root of the repository, a "more_tests.sh" shell script from that
-# same directory (the "./" indicates it isn't a command to search for on
-# the path), and the "extra_tests.dart" program in the "dev" directory,
-# run via "dart".
+# repository to enter each directory and run the tests in that
+# directory, returning non-zero as soon as any set of tests fails. You
+# can have multiple such scripts. In the example below, we run
+# "flutter test" in the root of the repository, a "more_tests.sh"
+# shell script from that same directory (the "./" indicates it isn't a
+# command to search for on the path), and the "extra_tests.dart"
+# program in the "dev" directory, run via "dart".
 test=flutter test
 test=./more_tests.sh
 test=dart dev/extra_tests.dart
+
+# To test your tests, check out the flutter/flutter repository and
+# then, from the root of that repository, run:
+#   bin/flutter; time bin/cache/dart-sdk/bin/dart dev/customer_testing/run_tests.dart --repeat=100 <path>
+# ...where <path> is the path to this file. This should run with no
+# output, without failing, and should in total take less than 15
+# minutes (a hundred runs of your tests taking just a few seconds
+# each). If your tests take longer, mention this in your PR and we
+# will evaluate them to see how valuable they are (e.g. how unique and
+# different they are compared to other tests we already have). The
+# more valuable the tests, the more likely we are to accept them
+# despite them taking a long time to run.
+
+# See also the `README.md` file in the root of this repository.

scripts/verify_tests_on_master.sh

@@ -6,4 +6,4 @@ flutter/bin/flutter doctor -v
 
 # Now run the tests a bunch of times to try to find flakes (tests that sometimes pass
 # even though they should be failing).
-flutter/bin/cache/dart-sdk/bin/dart flutter/dev/customer_testing/run_tests.dart --repeat 15 registry/
+flutter/bin/cache/dart-sdk/bin/dart flutter/dev/customer_testing/run_tests.dart --repeat=15 --skip-template registry/*.test