Definitive decision for Test262 console runner

[jugglinmike]

This project includes a console runner to facilitate running tests from console
hosts. I understand that some folks at Mozilla rely on it in some capacity
(@tschneidereit may be able to give more detail), and the V8 team uses some of
its internals in their own Python-based runner.

TC-39 agreed to deprecate the runner in 2014. This decision wasn't
well-communicated, so there was some contention a few months later when
@domenic filed gh-128 to remove the runner. The discussion died off without a
change to TC-39's decision or to the runner itself.

While implementing new features for the test runner (first for the proposed
async flag, and later for executing module code), I came across a few of bugs
in the implementation (see gh-512). I submitted a bug fix (gh-514), but as an
alternative, I also suggested formal deprecation (gh-513) in light of the
long-standing TC-39 decision and because I was afraid I was devoting energy to
a rarely-used/updated tool.

As it stands, the runner is still defined on the project's master branch, but
its "deprecated" status is formally documented. As @littledan pointed out,
though:

I don't really see the point in leaving something in the repository frozen
and deprecated. Why don't we either let Till maintain it here for Mozilla's
use, or remove it?

[...]

Let's bring this up at the next TC39 meeting to see if we should reverse the
deprecation decision. To address the concern about wasted maintanence work:
Whether or not the runner is deprecated, I don't think @jugglinmike is
personally responsible for maintaining it. But it would definitely be
unfortunate if Mozilla and V8 (for the inner parts Andreas mentioned) needed
to maintain separate copies of the same code.

I am opening this issue to solicit feedback on this from any stakeholders. Some
ideas for moving forward:

• Simply remove the tool (expecting dependents to maintain their own versions)
• Formally re-instate the tool (expecting Test262 maintainers to continue
  maintenance responsibilities)
• Move the tool to a new repository, possibly within the TC39 GitHub
  organzation (allowing for shared maintenance of a single tool by some
  combination of Test262 maintainers and interested consumers)

@leobalter has plans to raise this subject at next week's TC-39 meeting, as
well.

[tschneidereit]

Just as v8, we're currently using parts of the runner imported into our own repository. We were planning on using the upstream runner unmodified (with a small driver script), but when I learned about the deprecation I put that project on ice. While it'd obviously be nice for us to be able to use the runner, if we'd be the only ones to do so, I agree that removing it makes sense.

That being said, the fact that nobody uses the runner still confuses me. Why does it make sense to have a test suite that's used by multiple parties without having an official test runner that's usable in automation be part of said suite? I get that actually driving the shell running the tests is something that's specific to each implementation, but the test configuration parsing and result interpretation seem completely transferable, and are by far the bigger part of the runner.

[bterlson]

I think the reason is that we have different constraints in our test environment. Python is easy for some, while Node is a non-starter (this used to be the case for v8 teams, I think?) while for others the opposite is true.

[littledan]

The decision at TC39, driven by @bterlson , was that the test runner is out of scope and deleted from the test262 repository. Projects like V8 and Spidermoney can collaborate on the test runner elsewhere if they wish.

[avdg]

Any news on a official alternative test runner yet?

[jugglinmike]

@avdg Actually, yes. We've created test262-utils/test262-harness-py to facilitate ongoing development of the Python-based test harness. I am currently coordinating with the V8 and SpiderMonkey teams to migrate; I expect to remove the runner from this project very soon.

You may also be interested in the Node.js-based harness: https://github.com/bterlson/test262-harness

[avdg]

Thanks

[leobalter]

this seems fixed.