Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add script to find list of specs potentially in scope #70

Merged
merged 18 commits into from
Jun 12, 2020

Conversation

dontcallmedom
Copy link
Member

per #50
Looks into the report from validate-repo for repos associated with browser-spec producing WGs and well-known browser-spec producing CGs (both lists curated manually) and match that list of repos with URLs known in browser-spec

per #50
Looks into the report from validate-repo for repos associated with browser-spec producing WGs and well-known browser-spec producing CGs (both lists curated manually) and match that list of repos with URLs known in browser-spec
@dontcallmedom dontcallmedom requested a review from tidoust June 10, 2020 09:08
@dontcallmedom
Copy link
Member Author

dontcallmedom commented Jun 10, 2020

current output (which has false positives but also true ones that need review/discusison):
URLs from a repo of a browser-spec producing WG with no matching URL in spec list
[
'https://drafts.css-houdini.org/',
'https://drafts.csswg.org/',
'https://drafts.fxtf.org/',
'https://w3c.github.io/adpt/',
'https://w3c.github.io/imsc/',
'https://w3c.github.io/tt-module-karaoke/',
'https://w3c.github.io/tt-module-live/tt-live-1/spec/tt-live.html',
'https://w3c.github.io/ttml1/',
'https://w3c.github.io/ttml2/',
'https://w3c.github.io/ttml3/',
'https://w3c.github.com/mediacapture-depth/',
'https://w3c.github.io/mediacapture-record/MediaRecorder.html',
'https://w3c.github.io/webrtc-extensions/',
'https://w3c.github.io/webrtc-ice/',
'https://w3c.github.io/webappsec-cowl/',
'https://w3c.github.io/webappsec-suborigins/',
'https://w3c.github.io/webappsec-uisecurity/',
'https://w3c.github.io/webdriver-bidi/',
'https://w3c.github.io/openscreenprotocol/',
'https://w3c.github.io/dpub-aam/',
'https://w3c.github.io/graphics-aam/',
'https://w3c.github.io/graphics-aria/',
'https://w3c.github.io/mathml-aam/',
'https://w3c.github.io/editing/',
'https://w3c.github.io/html-aam/',
'https://w3c.github.io/html-aria/'
]
URLs from a repo of a browser-spec producing CG with no matching URL in spec list
[
'https://w3c.github.io/badging/',
'https://wicg.github.io/aom/',
'https://wicg.github.io/change-password-url/',
'https://wicg.github.io/crash-reporting/',
'https://wicg.github.io/cross-origin-embedder-policy/',
'https://wicg.github.io/custom-state-pseudo-class/',
'https://wicg.github.io/datacue/',
'https://wicg.github.io/deprecation-reporting/',
'https://wicg.github.io/devtools-protocol/specification/docs/domains/',
'https://wicg.github.io/encrypted-media-encryption-scheme/',
'https://wicg.github.io/focus-visible/demo/',
'https://wicg.github.io/frame-timing/',
'https://wicg.github.io/get-installed-related-apps/spec/index.html',
'https://wicg.github.io/hdcp-detection/',
'https://wicg.github.io/intervention-reporting/',
'https://wicg.github.io/is-input-pending/',
'https://wicg.github.io/kv-storage/',
'https://wicg.github.io/local-font-access/',
'https://wicg.github.io/media-feeds/',
'https://wicg.github.io/media-source/',
'https://open-ui.org',
'https://wicg.github.io/purification/',
'https://wicg.github.io/savedata/',
'https://wicg.github.io/ua-client-hints/',
'https://github.com/WICG/web-otp ',
'https://webmonetization.org',
'https://immersive-web.github.io/anchors/',
'https://immersive-web.github.io/cardboard-vr-display/',
'https://www.w3.org/immersive-web/',
'https://immersiveweb.dev',
'https://immersive-web.github.io/layers/',
'https://immersive-web.github.io/webvr-polyfill/examples/',
'https://dpdb.webvr.rocks/dpdb.json',
'https://webvr.rocks/',
'https://immersive-web.github.io/webxr-input-profiles/',
'https://immersive-web.github.io/webxr-reference/',
'https://immersive-web.github.io/webxr-samples/',
'https://immersive-web.github.io/webxr-test-api/',
'https://gpuweb.github.io/cts/',
'https://webgpu.io',
'https://gpuweb.github.io/WSL/',
'https://www.w3.org/community/audio-comgp/',
'https://privacycg.github.io/private-click-measurement/',
'https://privacycg.github.io/storage-access/',
'https://privacycg.github.io/storage-partitioning/'
]
Unadvertized URLs from a repo of a browser-spec producing WG with no matching URL in spec list
[
'https://w3c.github.io/svg-aam/',
'https://w3c.github.io/svgwg/',
'https://w3c.github.io/system-wake-lock/',
'https://w3c.github.io/webrtc-provisional-stats/',
'https://immersive-web.github.io/dom-overlays/'
]
Unadvertized URLs from a repo of a browser-spec producing CG with no matching URL in spec list
[
'https://wicg.github.io/audio-focus/',
'https://wicg.github.io/contact-api/',
'https://wicg.github.io/container-queries/',
'https://wicg.github.io/content-index/',
'https://wicg.github.io/controls-list/',
'https://wicg.github.io/conversion-measurement-api/',
'https://wicg.github.io/display-locking/',
'https://wicg.github.io/file-handling/',
'https://wicg.github.io/first-party-sets/',
'https://wicg.github.io/indexed-db-observers/',
'https://wicg.github.io/inert/',
'https://wicg.github.io/input-for-workers/',
'https://wicg.github.io/media-latency-hint/',
'https://wicg.github.io/overscroll-scrollend-events/',
'https://wicg.github.io/sw-launch/',
'https://wicg.github.io/virtual-scroller/',
'https://wicg.github.io/webhid/',
'https://wicg.github.io/webpackage/',
'https://immersive-web.github.io/immersive-web.github.io/',
'https://immersive-web.github.io/webvr/',
'https://immersive-web.github.io/webxr-polyfill/',
'https://gpuweb.github.io/admin/',
'https://privacycg.github.io/template/'
]

@dontcallmedom
Copy link
Member Author

right now, the script doesn't have a way to track repos/URLs that would have been reviewed as not worth watching - input on where/how to track this would be useful (could be done the same was as WGs/CGs with a static list in the code?)

I'm also not clear how to handle URLs which right now have uninteresting content, but might become interesting (e.g. https://w3c.github.io/system-wake-lock/) - maybe the script should have a bit more heuristic than just 404 check.

Some of the false positives could be fixed by fixing the underlying repos (e.g. adding a proper w3c.json); others could be by relaxing some of the tests (e.g. the CSS specific URLs could be checked as a prefix of a known spec URL).

@dontcallmedom
Copy link
Member Author

conversely, the script right now is not useful in identifying new specs in csswg-drafts and similar multi-spec repos; this would probably require additional repo-specific heuristics

@dontcallmedom
Copy link
Member Author

dontcallmedom commented Jun 10, 2020

I've added a test that compares with the data from the spec-dashboard, which I think will help with #62 and should also help with some of the CSS spec finding. The data from that new test gives:
TR specs from browser-producing WGs
[
'https://www.w3.org/TR/service-workers-1/',
'https://www.w3.org/TR/html-aam-1.0/',
'https://www.w3.org/TR/html-aria/',
'https://www.w3.org/TR/input-events-1/',
'https://www.w3.org/TR/dom/',
'https://www.w3.org/TR/svg-aam-1.0/',
'https://www.w3.org/TR/css-box-4/',
'https://www.w3.org/TR/css-sizing-4/',
'https://www.w3.org/TR/ttml-imsc1.2/',
'https://www.w3.org/TR/ttml2/',
'https://www.w3.org/TR/geolocation-sensor/',
'https://www.w3.org/TR/reporting-1/',
'https://www.w3.org/TR/resource-timing-1/',
'https://www.w3.org/TR/user-timing-3/',
'https://www.w3.org/TR/COWL/',
'https://www.w3.org/TR/feature-policy-1/',
'https://www.w3.org/TR/UISecurity/',
'https://www.w3.org/TR/webdriver2/',
'https://www.w3.org/TR/html-imports/',
'https://www.w3.org/TR/html53/',
'https://www.w3.org/TR/microdata/',
'https://www.w3.org/TR/accname-1.2/',
'https://www.w3.org/TR/core-aam-1.2/',
'https://www.w3.org/TR/webpayments-http-messages/'
]

The false positives here probably also imply some TR cleanup (in addition to clean up in specs.json)

@dontcallmedom dontcallmedom requested a review from foolip June 10, 2020 10:03
src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
Copy link
Member

@tidoust tidoust left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While you're at improving readability, a couple of additional suggestions :)

src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
@dontcallmedom
Copy link
Member Author

dontcallmedom commented Jun 10, 2020

Having reviewed all the URLs highlighted by find specs, here is my raw and rough first pass analysis of how they need to be handled:
multi-spec repos: use substring matching

not for browsers (blocklist)

review whether to add
'https://w3c.github.com/mediacapture-depth/', // impl?
'https://w3c.github.io/webrtc-extensions/', // diff?
'https://w3c.github.io/webrtc-ice/', // impl?
'https://w3c.github.io/webdriver-bidi/', // impl?
'https://w3c.github.io/openscreenprotocol/', // protocol?
'https://w3c.github.io/dpub-aam/',
'https://w3c.github.io/graphics-aam/',
'https://w3c.github.io/graphics-aria/',
'https://w3c.github.io/mathml-aam/',
'https://w3c.github.io/html-aam/',
'https://w3c.github.io/html-aria/'
'https://w3c.github.io/badging/',
'https://wicg.github.io/change-password-url/',
'https://wicg.github.io/crash-reporting/',
'https://wicg.github.io/cross-origin-embedder-policy/',
'https://wicg.github.io/custom-state-pseudo-class/',
'https://wicg.github.io/deprecation-reporting/',
'https://wicg.github.io/intervention-reporting/'
'https://wicg.github.io/is-input-pending/',
'https://wicg.github.io/local-font-access/',
'https://wicg.github.io/media-feeds/',
'https://wicg.github.io/purification/',
'https://wicg.github.io/savedata/',
'https://wicg.github.io/ua-client-hints/',
'https://github.com/WICG/web-otp ', // should be https://wicg.github.io//WebOTP/index.html (fix repo)
'https://webmonetization.org', // spec is https://webmonetization.org/specification.html
'https://immersive-web.github.io/anchors/'
'https://immersive-web.github.io/layers/',
'https://immersive-web.github.io/webxr-test-api/',
'https://gpuweb.github.io/WSL/',
'https://privacycg.github.io/private-click-measurement/'
'https://privacycg.github.io/storage-access/',
'https://w3c.github.io/svg-aam/',
'https://w3c.github.io/webrtc-provisional-stats/',
'https://immersive-web.github.io/dom-overlays/'
https://wicg.github.io/contact-api/spec/
'https://wicg.github.io/content-index/',
'https://wicg.github.io/conversion-measurement-api/'
'https://wicg.github.io/input-for-workers/',
'https://wicg.github.io/overscroll-scrollend-events/',
'https://wicg.github.io/webhid/'
https://wicg.github.io/webpackage/loading.html

not yet (monitor)

fix repo

fix specs.json

deal with spec series (not sure how)
https://www.w3.org/TR/service-workers-1/ vs https://www.w3.org/TR/service-workers ?
'https://www.w3.org/TR/input-events-1/', vs -2

pending deprecation

Status check?
'https://w3c.github.io/webappsec-cowl/',
'https://w3c.github.io/webappsec-suborigins/',
'https://w3c.github.io/webappsec-uisecurity/',
'https://wicg.github.io/encrypted-media-encryption-scheme/' // belongs to Media WG now?
'https://wicg.github.io/hdcp-detection/' // Media WG?
'https://wicg.github.io/media-source/', // Media WG
https://github.com/immersive-web/webvrrocks (should it point to webxr now)
'https://wicg.github.io/container-queries/', // relationship to CSS work? depreacate old spec text?
'https://wicg.github.io/controls-list/', // WICG/controls-list#12
'https://wicg.github.io/file-handling/',
'https://www.w3.org/TR/html-imports/',
'https://www.w3.org/TR/webpayments-http-messages/'
// @@@ discuss with WICG regularly paced checkin on liveness of a repo?

Sorry, something went wrong.

@foolip
Copy link
Member

foolip commented Jun 10, 2020

https://w3c.github.io/webdriver-bidi/ doesn't have any implementation yet, but is intended for browser implementation (and nothing else). It's not important to include it now, but also not bad. It doesn't define any CSS or IDL that will trickle down anywhere else.

@foolip
Copy link
Member

foolip commented Jun 11, 2020

This is still a draft, should I do a final review pass, or wait until it's marked ready?

@dontcallmedom
Copy link
Member Author

dontcallmedom commented Jun 11, 2020

@tidoust and I discussed this morning: to finalize this PR, I'll

  •  create separate files to track specs we want to ignore and specs that can't be added now but may be worth adding later, recording the dates and the reasons of the assessment

  • handle minimally the case of multi-spec repos (right now, ignoring them)

Later improvements (e.g. setting up a regular github action to run it, post the results as github issue or pull request) will be raised as issues for improvement. Likewise will I raise issues for the specs identified by the script as needing reviews or fixes in our list.

Sorry, something went wrong.

They'll have to be identified through other mechanisms
…arily

Defines a JSON list of specs and repos that can be ignored (e.g. because the work is not targeted at browsers)
Defines a JSON list of specs with no relevant content at the moment but that may become relevant later (to be regularly checked)
@dontcallmedom dontcallmedom changed the title [WIP] Add script to find list of specs potentially in scope Add script to find list of specs potentially in scope Jun 11, 2020
@dontcallmedom dontcallmedom marked this pull request as ready for review June 11, 2020 16:17
Copy link
Member

@tidoust tidoust left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of editorial comments which are up to you.

Otherwise, I note that we will need to add some code as part of the GitHub action that runs this script periodically to check on lastreviewed dates and report on entries that need to be checked again (e.g. if they haven't been reviewed in the last 3 months or so). One option is to have this script filter out old entries from monitor-repos.json when it parses it. Doing it separately would probably help clarify that the entry is a actually a known one, just one that needs to be evaluated again. Anyway, that can be done separately.

src/find-specs.js Outdated Show resolved Hide resolved
src/find-specs.js Outdated Show resolved Hide resolved
@dontcallmedom dontcallmedom merged commit 117b4b5 into master Jun 12, 2020
tidoust added a commit to tidoust/browser-specs that referenced this pull request Jun 12, 2020
- Use TR URL for geolocation-sensor
- Use TR URL for reporting-1
- Use TR URL for feature-policy-1
- Use TR URL for accname-1.2
- Use TR URL for core-aam-1.2
- Add css-box-4 (on top of -3, which is still "active")
- Add user-timing-3 (on top of -2, which is still the "default" for user-timing)
- Replace webdriver1 with webdriver2
- Adjust shortname for webxr-ar-module-1 (fixed in W3C API)
- Adjust shortname for webxr-gamepads-module-1 (fixed in W3C API)
tidoust added a commit to tidoust/browser-specs that referenced this pull request Jun 12, 2020
- Use TR URL for geolocation-sensor
- Use TR URL for reporting-1
- Use TR URL for feature-policy-1
- Use TR URL for accname-1.2
- Use TR URL for core-aam-1.2
- Move to W3C URL for badging
- Add css-box-4 (on top of -3, which is still "active")
- Add user-timing-3 (on top of -2, which is still the "default" for user-timing)
- Replace webdriver1 with webdriver2
- Adjust shortname for webxr-ar-module-1 (fixed in W3C API)
- Adjust shortname for webxr-gamepads-module-1 (fixed in W3C API)
tidoust added a commit that referenced this pull request Jun 12, 2020
- Use TR URL for geolocation-sensor
- Use TR URL for reporting-1
- Use TR URL for feature-policy-1
- Use TR URL for accname-1.2
- Use TR URL for core-aam-1.2
- Move to W3C URL for badging
- Add css-box-4 (on top of -3, which is still "active")
- Add user-timing-3 (on top of -2, which is still the "default" for user-timing)
- Replace webdriver1 with webdriver2
- Adjust shortname for webxr-ar-module-1 (fixed in W3C API)
- Adjust shortname for webxr-gamepads-module-1 (fixed in W3C API)
@tidoust tidoust deleted the find-specs branch July 1, 2021 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants