diff --git a/README.md b/README.md index c6c14cb6..5c2062a8 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,20 @@ # Flutter Tests -This repository contains references to tests that are run with every commit to -Flutter to verify that no breaking changes have been introduced. +This repository contains references to tests that are run with every +commit to Flutter to verify that no breaking changes have been +introduced. ## Adding more tests You are welcome to add a new batch of tests. To do so, copy the file -`registry/template.test` to create a new file in the `registry/` directory. Fill -in the fields and delete all the comments. Then, submit a PR with this new file. +`registry/template.test` to create a new file in the `registry/` +directory. Fill in the fields and delete all the comments. Then, +submit a PR with this new file. Tests must fulfill the following criteria to be added: -* All the code must be available publicly on GitHub under a license compatible - with this effort. +* All the code must be available publicly on GitHub under a license + compatible with this effort. * Tests must be hermetic. For example, a test should not involve network activity, spawn processes, or access the local file system @@ -21,21 +23,25 @@ Tests must fulfill the following criteria to be added: * Tests must be resilient to being run concurrently with other tests, including concurrent runs of themselves. -* Tests must be reliable. A test must not claim to pass if it is failing. - Running a test multiple times in a row must always have the same result. +* Tests must be reliable. A test must not claim to pass if it is + failing. Running a test multiple times in a row must always have the + same result. -* Tests must be as fast as possible given the hardware. For example, tests must - not use real timers or depend on the wall clock. +* Tests must have no output when they are passing. -* The time taken by tests must be proportional to their value. A few thousands - tests are expected to run within a few minutes. An upper limit of about five - minutes will be applied to each contributed test suite (not including the - time to download the tests), but it is expected that most suites will - complete in seconds. +* Tests must be as fast as possible given the hardware. For example, + tests must not use real timers or depend on the wall clock. -* The tests must be compatible with any tools for automatically updating - Flutter code (e.g. they cannot rely on custom code generation unless such - code generation can hook into the automatic update mechanism). +* The time taken by tests must be proportional to their value. A few + thousands tests are expected to run within a few minutes. An upper + limit of about five minutes will be applied to each contributed test + suite (not including the time to download the tests), but it is + expected that most suites will complete in seconds. + +* The tests must be compatible with any tools for automatically + updating Flutter code (e.g. they cannot rely on custom code + generation unless such code generation can hook into the automatic + update mechanism). * The tests must represent good practices as described in Flutter's documentation. For example, using an object after calling its @@ -49,35 +55,41 @@ Tests must fulfill the following criteria to be added: ## If a test is broken -The point of these tests is to make sure we don't break existing code, while still -being able to make improvements to Flutter APIs. +The point of these tests is to make sure we don't break existing code, +while still being able to make improvements to Flutter APIs. -If you find that a PR you have created causes on these tests to fail, you have the -following options: +If you find that a PR you have created causes one these tests to fail, +you have the following options: -1. Change your PR so that the test no longer fails. This is the preferred option, - so long as the result is one we can be proud of. Is the resulting API something - that you would plausibly come up with without the backwards-compatibility - constraint? That's good. Is the resulting API something that, as soon as you - see it, you think "why?" or "that's weird"? That's bad. +1. Change your PR so that the test no longer fails. This is the + preferred option, so long as the result is one we can be proud of. + Is the resulting API something that you would plausibly come up + with without the backwards-compatibility constraint? That's good. + Is the resulting API something that, as soon as you see it, you + think "why?" or "that's weird"? That's bad. Consider the advice in + the Style guide: + https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo 2. Go through the breaking change process, as documented here: https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes - If you're going to do this, you will need to contact the relevant people - responsible for the breaking test(s) (see the relevant .test files), help them - fix their code, and update this repository to use the new version of their - tests, in addition to the steps described on the wiki. + If you're going to do this, you will need to contact the relevant + people responsible for the breaking test(s) (see the relevant .test + files), help them fix their code, and update this repository to use + the new version of their tests, in addition to the steps described + on the wiki. -3. Remove the test in question. This is by far the least ideal solution. To go - down this path, we must first establish that one of the following is true: +3. Remove the test in question. This is by far the least ideal + solution. To go down this path, we must first establish that one of + the following is true: - the people listed as contacts for the test are not responsive. - - the test is poorly written (e.g. it contains a race condition or relies on - assumptions that violate clearly documented API contracts), and the people - listed as contacts are not willing to fix the test or accept fixes for the - test. + - the test is poorly written (e.g. it contains a race condition or + relies on assumptions that violate clearly documented API + contracts), and the people listed as contacts are not willing to + fix the test or accept fixes for the test. - - we have gone through the breaking change process cited above, but are unable - to update the test accordingly (e.g. the people listed as contacts are not - willing to work with us to update their code). + - we have gone through the breaking change process cited above, + but are unable to update the test accordingly (e.g. the people + listed as contacts are not willing to work with us to update + their code). diff --git a/registry/flutter_svg.test b/registry/flutter_svg.test index dd67bf36..1f5acdf0 100644 --- a/registry/flutter_svg.test +++ b/registry/flutter_svg.test @@ -1,5 +1,5 @@ contact=dfield@gmail.com fetch=git clone git@github.com:dnfield/flutter_svg.git tests fetch=git -C tests checkout 4687e1122cde7d10bfab2a594f153ce453fd87d7 -directory=. +update=. test=flutter test diff --git a/registry/template.test b/registry/template.test index 82a09f95..4e12066e 100644 --- a/registry/template.test +++ b/registry/template.test @@ -1,71 +1,99 @@ # This is a template for adding tests to the flutter/tests registry. # -# To submit a new batch of tests, copy this file to create a new file in -# this directory. +# To submit a new batch of tests, copy this file to create a new file +# in this directory, update it as described below, and submit it as a +# new PR to this repository. # # New files should be named `foo.test` where `foo` is a description of -# the project whose tests are being added. +# the project whose tests are being added. (The file named +# `template.test` is skipped automatically.) # -# Lines beginning with a "#" are ignored. Delete all such lines from this -# template before submitting a new `.test` file (you may add new comments -# if you have something to say that doesn't fit in one of the meaningful -# lines). Also delete blank lines like the following: +# Lines beginning with a "#" are ignored. Delete all such lines from +# this template before submitting a new `.test` file (you may add new +# comments if you have something to say that doesn't fit in one of the +# meaningful lines). Also delete blank lines like the following: -# The contact lines must contain the e-mail addresses, one to a line, of -# one or more people who will be able to help if a problem is found with -# a test. People on this list should be able to respond to e-mails within -# about 72 hours. If there is a problem with a test and we cannot reach -# any of the people listed, then we will probably remove the tests. +# The contact lines must contain the e-mail addresses, one to a line, +# of one or more people who will be able to help if a problem is found +# with a test. People on this list should be able to respond to +# e-mails within about 72 hours. If there is a problem with a test and +# we cannot reach any of the people listed, then we will probably +# remove the tests. contact=foo@example.com contact=bar@example.com # The fetch line(s) must contain one or more shell commands that fetch -# the tests into a new directory called "tests" in the current directory. -# The example below clones the dnfield/flutter_svg GitHub repository then -# checks out a specific revision. +# the tests into a new directory called "tests" in the current +# directory. The example below clones the dnfield/flutter_svg GitHub +# repository then checks out a specific revision. # # Our current policy is that all tests should be in public GitHub # repositories. (This way, we don't take on additional dependencies on -# other services that may or not go down. We already depend on GitHub, so -# if GitHub goes down it doesn't matter that we can't reach contributed -# tests.) If you have some tests to contribute that are not and will not -# be in GitHub, please contact ian@hixie.ch before submitting a PR -# proposing to add such tests. We may make exceptions if your tests are -# especially valuable. +# other services that may or not go down. We already depend on GitHub, +# so if GitHub goes down it doesn't matter that we can't reach +# contributed tests.) If you have some tests to contribute that are +# not and will not be in GitHub, please contact ian@hixie.ch before +# submitting a PR proposing to add such tests. We may make exceptions +# if your tests are especially valuable. # -# The checkout must always check out the exact same set of tests. Do not -# check out the current master branch or some such; always fetch a +# The checkout must always check out the exact same set of tests. Do +# not check out the current master branch or some such; always fetch a # specific revision. You may regularly submit PRs that update the # revision being used. # -# The commands specified on the "fetch" lines must return the exit code 0 -# or else the tests will be skipped entirely. The "git" program will be -# on the path. +# The commands specified on the "fetch" lines must return the exit +# code 0 or else the tests will be skipped entirely. The "git" program +# will be on the path. +# +# There must be at least one "fetch" line. (In fact currently there +# must be two, and they must follow the pattern below more or less +# exactly. If this is a problem, let us know in your PR.) fetch=git clone git@github.com:dnfield/flutter_svg.git tests fetch=git -C tests checkout 4687e1122cde7d10bfab2a594f153ce453fd87d7 -# The directory lines specify paths (relative to the newly created -# "tests" directory) in which to run a flutter command to automatically -# update the code to fix breaking changes. The command will be run in -# turn in each specified directory. The root can be specified by using -# the path "." as that represents the "current" directory. Multiple paths -# may be specified by specifying multiple "directory" lines. -directory=. +# The "update" lines specify paths (relative to the newly created +# "tests" directory) in which to run a flutter command to +# automatically update the code to fix breaking changes. The command +# will be run in turn in each specified directory. The root can be +# specified by using the path "." as that represents the "current" +# directory. Multiple paths may be specified by specifying multiple +# "update" lines. +# +# The command that updates the code may fail, in which case the tests +# are considered to have failed. This helps catch cases where +# automatically updating the code doesn't work. +# +# There must be at least one "update" line. +update=. # The test lines specify commands to run to execute the actual tests. -# Each of these should return either 0 to indicate success, or a non-zero -# exit code to report failure. +# Each of these should return either 0 to indicate success, or a +# non-zero exit code to report failure. # # The "flutter" tool and the "dart" program will be in the path. # # If there are multiple directories, create a shell script in the -# repository to enter each directory and run the tests in that directory, -# returning non-zero as soon as any set of tests fails. You can have -# multiple such scripts. In the example below, we run "flutter test" in -# the root of the repository, a "more_tests.sh" shell script from that -# same directory (the "./" indicates it isn't a command to search for on -# the path), and the "extra_tests.dart" program in the "dev" directory, -# run via "dart". +# repository to enter each directory and run the tests in that +# directory, returning non-zero as soon as any set of tests fails. You +# can have multiple such scripts. In the example below, we run +# "flutter test" in the root of the repository, a "more_tests.sh" +# shell script from that same directory (the "./" indicates it isn't a +# command to search for on the path), and the "extra_tests.dart" +# program in the "dev" directory, run via "dart". test=flutter test test=./more_tests.sh test=dart dev/extra_tests.dart + +# To test your tests, check out the flutter/flutter repository and +# then, from the root of that repository, run: +# bin/flutter; time bin/cache/dart-sdk/bin/dart dev/customer_testing/run_tests.dart --repeat=100 +# ...where is the path to this file. This should run with no +# output, without failing, and should in total take less than 15 +# minutes (a hundred runs of your tests taking just a few seconds +# each). If your tests take longer, mention this in your PR and we +# will evaluate them to see how valuable they are (e.g. how unique and +# different they are compared to other tests we already have). The +# more valuable the tests, the more likely we are to accept them +# despite them taking a long time to run. + +# See also the `README.md` file in the root of this repository. diff --git a/scripts/verify_tests_on_master.sh b/scripts/verify_tests_on_master.sh index 193ed337..686d485e 100755 --- a/scripts/verify_tests_on_master.sh +++ b/scripts/verify_tests_on_master.sh @@ -6,4 +6,4 @@ flutter/bin/flutter doctor -v # Now run the tests a bunch of times to try to find flakes (tests that sometimes pass # even though they should be failing). -flutter/bin/cache/dart-sdk/bin/dart flutter/dev/customer_testing/run_tests.dart --repeat 15 registry/ +flutter/bin/cache/dart-sdk/bin/dart flutter/dev/customer_testing/run_tests.dart --repeat=15 registry/*.test