Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapping between Test262 and web-features #4043

Open
foolip opened this issue Apr 8, 2024 · 10 comments
Open

Mapping between Test262 and web-features #4043

foolip opened this issue Apr 8, 2024 · 10 comments

Comments

@foolip
Copy link

foolip commented Apr 8, 2024

Let's find a way to find the tests for a given web-features identifier in Test262!

Background:

web-features is an effort to describe the whole web platform, including JavaScript, as a set of features. It's part of Baseline, which powers the "Baseline 2022" badge at the top of MDN pages, such as on Array.prototype.at().

Separately from Baseline, we have a mechanism for finding the tests in web-platform-tests that belong to a particular feature:
https://wpt.fyi/results/css/css-transforms?label=master&label=experimental&aligned&q=feature%3Atransforms3d
https://github.com/web-platform-tests/wpt/blob/master/css/css-transforms/WEB_FEATURES.yml

Alternatives:

Test262 already has a features.txt, so rather than mapping web-features to individual tests we should map to one of those features. There are two obvious options:

  1. Maintain a mapping from Tests262 features to web-features in Test262, similar to how WPT controls the mapping from WPT to web-features
  2. Add a test262 field to web-features accepting one or more Test262 feature identifiers

With one of these in place, it should be possible to list all tests for a given JavaScript features in web-features. This would help web-features maintainers understand the level of interoperability when determining the Baseline status of JavaScript features.

@nicolo-ribaudo
Copy link
Member

For ecmascript features, could web-features just reuse the same name as in test262? Test262 tests are written before that implementations ship, so the identifier should always be available when you'll want to include it in web-features.

@foolip
Copy link
Author

foolip commented Apr 8, 2024

I think we could a lot of the time, but not always. Our identifiers cannot have periods, and we might have larger groupings. For example, we have "array-flat" where Test262 has "Array.prototype.flat" + "Array.prototype.flatMap".

@mathiasbynens
Copy link
Member

IIUC, there’s a few separate requests here, and it’s worth considering them separately:

  1. For each Test262 feature, you want the complete list of Test262 tests tagged with that feature. That mapping is something the Test262 project could provide as a JSON build artifact, potentially published for each commit. (Of course, this could be built as a standalone project that pulls in Test262, but ideally it’s done upstream.)

  2. Then, web-features could consume that JSON file to figure out the overall test pass rate for each feature.

  3. The mapping from web-features identifiers to Test262 feature identifiers is also something web-features could maintain. It could be done at the Test262 level too, but assuming point 1 above is solved in some way, it doesn’t seem necessary.

The biggest missing piece that affects the tc39/test262 project seems to be the first item. Everything else can be built on top of that.

@nicolo-ribaudo
Copy link
Member

With https://github.com/bocoup/test262-stream, the first step is trivial:

await new Test262Stream("./test262")
  .filter(test => test.attrs.features?.includes("Array.prototype.flat"))
  .map(test => test.file)
  .toArray();

@mathiasbynens
Copy link
Member

mathiasbynens commented Apr 8, 2024

With https://github.com/bocoup/test262-stream, the first step is trivial:

await new Test262Stream("./test262")
  .filter(test => test.attrs.features?.includes("Array.prototype.flat"))
  .map(test => test.file)
  .toArray();

TIL about test262-stream — that makes that part easier indeed:

import {default as Test262Stream} from 'test262-stream';

const stream = new Test262Stream('../test262');

const featureIdentifierToTests = new Map();

for await (const test of stream) {
  const filePath = test.file;
  if (!test.attrs.features) continue;
  for (const feature of test.attrs.features) {
    if (featureIdentifierToTests.has(feature)) {
      featureIdentifierToTests.get(feature).add(filePath);
    } else {
      featureIdentifierToTests.set(feature, new Set([filePath]));
    }
  }
}

console.log(featureIdentifierToTests);
Example output:
Map(169) {
  // …
  'Array.prototype.flat' => Set(15) {
    'test/built-ins/Array/prototype/methods-called-as-functions.js',
    'test/built-ins/Array/prototype/flat/array-like-objects.js',
    'test/built-ins/Array/prototype/flat/bound-function-call.js',
    'test/built-ins/Array/prototype/flat/empty-array-elements.js',
    'test/built-ins/Array/prototype/flat/empty-object-elements.js',
    'test/built-ins/Array/prototype/flat/length.js',
    'test/built-ins/Array/prototype/flat/name.js',
    'test/built-ins/Array/prototype/flat/non-numeric-depth-should-not-throw.js',
    'test/built-ins/Array/prototype/flat/non-object-ctor-throws.js',
    'test/built-ins/Array/prototype/flat/null-undefined-elements.js',
    'test/built-ins/Array/prototype/flat/null-undefined-input-throws.js',
    'test/built-ins/Array/prototype/flat/positive-infinity.js',
    'test/built-ins/Array/prototype/flat/prop-desc.js',
    'test/built-ins/Array/prototype/flat/proxy-access-count.js',
    'test/built-ins/Array/prototype/flat/symbol-object-create-null-depth-throws.js'
  },
  'Array.prototype.flatMap' => Set(20) {
    'test/built-ins/Array/prototype/methods-called-as-functions.js',
    'test/built-ins/Array/prototype/flatMap/array-like-objects-nested.js',
    'test/built-ins/Array/prototype/flatMap/array-like-objects-poisoned-length.js',
    'test/built-ins/Array/prototype/flatMap/array-like-objects-typedarrays.js',
    'test/built-ins/Array/prototype/flatMap/array-like-objects.js',
    'test/built-ins/Array/prototype/flatMap/bound-function-argument.js',
    'test/built-ins/Array/prototype/flatMap/length.js',
    'test/built-ins/Array/prototype/flatMap/depth-always-one.js',
    'test/built-ins/Array/prototype/flatMap/name.js',
    'test/built-ins/Array/prototype/flatMap/non-callable-argument-throws.js',
    'test/built-ins/Array/prototype/flatMap/not-a-constructor.js',
    'test/built-ins/Array/prototype/flatMap/prop-desc.js',
    'test/built-ins/Array/prototype/flatMap/proxy-access-count.js',
    'test/built-ins/Array/prototype/flatMap/this-value-ctor-non-object.js',
    'test/built-ins/Array/prototype/flatMap/this-value-ctor-object-species-bad-throws.js',
    'test/built-ins/Array/prototype/flatMap/this-value-ctor-object-species-custom-ctor-poisoned-throws.js',
    'test/built-ins/Array/prototype/flatMap/this-value-ctor-object-species-custom-ctor.js',
    'test/built-ins/Array/prototype/flatMap/this-value-ctor-object-species.js',
    'test/built-ins/Array/prototype/flatMap/this-value-null-undefined-throws.js',
    'test/built-ins/Array/prototype/flatMap/thisArg-argument.js'
  },
  // …
}

@foolip
Copy link
Author

foolip commented Apr 8, 2024

That's very nice! It look like I still need a Test262 checkout, so a build step to create these mappings as build artifacts would be great. In WPT we do this as artifacts published for every merged PR. Example:
https://github.com/web-platform-tests/wpt/releases/tag/merge_pr_45598

@ljharb
Copy link
Member

ljharb commented Apr 8, 2024

Why can't your identifiers have periods?

@foolip
Copy link
Author

foolip commented Apr 8, 2024

That's a self-imposed limitation that we never discussed much, but it's the style of caniuse identifiers and we do want web-features to mix well with caniuse.

@ljharb
Copy link
Member

ljharb commented Apr 8, 2024

@foolip https://caniuse.com/?search=array.prototype.flat seems to work just fine.

@foolip
Copy link
Author

foolip commented Apr 8, 2024

https://caniuse.com/?search=flatten also works, but the canonical URL is https://caniuse.com/array-flat, with the identifier "array-flat" coming from this file:
https://github.com/Fyrd/caniuse/blob/main/features-json/array-flat.json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants