Add testing guideline for naming test files and adding comments #1664
Conversation
|
@fippo can you review this? |
|
philip(p) confusion. Paging @foolip |
|
@foolip could you take a look? |
|
It is really up to the people doing most of the work here to decide what amount of metadata and structure is useful. If I were to recommend anything, it would be to err on the side of less metadata to update, to optimize for ease of adding new tests. Given the permission given to not bother with metadata in new tests or when updating tests, unless someone is continuously going to update the metadata, then it will get out of sync. As soon as that begins to happen, I think it might be better to delete the metadata. |
|
@foolip yes agree we can remove a number of metadata from the individual test files, in particular the todo list. I think some coverage accounting is still essential until we reach 100% coverage. Some considerations are how essential are such metadata to others, and where should we put the metadata. Do we put them in web-platform-tests (web-platform-tests/wpt#8051)? or as unofficial forks (https://rwaldron.github.io/webrtc-pc/)? or in webrtc-pc? |
|
@soareschen, all the attempts to determine coverage from test-to-spec linkage so far don't seem very successful. Linking to just sections isn't granular enough. If you search for "data-tested-assertations" you can see an attempt at something more granular dating back to web-platform-tests/wpt@6bb5305. It's a brittle approach, and I can only assume it's bitrotted. The approach in speced/bikeshed#1116 is more promising to me, and should allow for views similar to https://rwaldron.github.io/webrtc-pc/. But that will be a Bikeshed feature, and WebRTC uses ReSpec. AFAICT, the least amount of work at this point would be to keep https://rwaldron.github.io/webrtc-pc/ up to date manually, by noting which wpt commit it's in sync with, and going through the changes since then for each update. |
|
Updating the coverage can be pretty tedious. If not done incrementally, it could take a whole week to track changes to new editor's draft. I guess that means the best approach is to pull out the metadata and put them in unofficial forks. If the fork gets bitrotten, it is simply forgotten and affect neither web-platform-tests nor webrtc-pc. |
| is being tested. New tests may be written with minimal or no comments up to the | ||
| author's preference. If a test case is testing specific steps in a spec, it is | ||
| recommended to leave at least one line of comment referencing the spec to help | ||
| readers quickly understand the objective of the test. |
There was a problem hiding this comment.
How does this relate to the "help link" that should point to the tested section in the spec?
There was a problem hiding this comment.
Mention about "help link": web-platform-tests/wpt#8450 (comment)
I can add more about other aspects of testing later on. These are the two draft test guidelines that I come out with while still at TPAC.
We might also want to consider whether to put the main guidelines in webrtc-pc or web-platform-tests. There may be other contributors who work on web-platform-tests but not webrtc-pc. So it might be better off to put the guidelines in WPT and link to it from webrtc-pc.