util: add tokens to parseArgs

[shadowspawn]

Offer additional meta-data for building custom and additional behaviour on top of parseArgs.

parseArgs supports the primary goal of a useful CLI parser for direct use, but is not offering much for the secondary goal of a basis for building richer functionality, or for customising the behaviour.

@bakkot proposed returning additional meta-data, such as reporting indexes. We have developed that idea into optionally returning a tokens structure with parsing meta-data in pkgjs/parseargs#129 with input from @aaronccasanova, @bcoe, and @ljharb.

The returned fields are a pragmatic mix of information useful to an author for custom behaviours or fancy error messages. The tokens are used internally by parseArgs for generating the returned values and positionals and performing error detection, using the same meta-data now available to callers. The downstream PR includes some example files of add-on processing: pkgjs/parseargs#129

Edit: fixed link to downstream PR.

[bakkot]

The point of this design is that it lets script authors do stuff like support --no-foo properties or stopping at the first positional argument (so you can build an API like npm or node). And it accomplishes this without needing parseArgs to explicitly support all that stuff - script authors can build more advanced functionality on top of tokens in a pretty straightforward way.

---

One thing none of us were particularly happy with: we wanted a way to indicate whether a given option was --long or -l style, and the way that we've done that is with a field called rawName which holds "--long" or "-l". (Assuming l is an alias for long, the name field of the token will hold "long" in either case.) rawName is maybe a little misleading because if you have something like -xy you'll get two tokens which have rawNames "-x" and "-y" respectively, even though -y never appeared literally in the input.

If anyone has a better idea how to represent this, please suggest it.

[bricss]

Sold! 🔨 Lets merge it 🤠 coz that's exactly 🎯 what we need 👀

[shadowspawn]

Example uses of tokens in parseArgs itself are the strict mode error detection for:

• unknown options
• "string" options used without an option-argument
• "boolean" options used with an option-argument
• option-argument ambiguous as looks like an option (this was not feasible for client to reproduce before tokens)
• unexpected positional

And populating the returned values and positionals.

Example custom client uses of tokens in downstream PR:

• implement negatable options, like --no-foo
• error if option used more than once
• enforce option order, like --experimental before --unstable-x
• restricting long options to using equals (--foo=bar allows, --foo bar produces error)

None of these custom client cases were previously possible without client doing some parsing of inputs args themselves.

[bcoe]

@shadowspawn thanks for opening this PR 👏 will make an effort to give a through review soon.

[bcoe]

Left some initial review feedback.

[bcoe]

Just a thought, I would be tempted to make tokens a class rather than a POJO with a kind property:

class Token {}
class OptionTerminator extends Token {}
class PositionalToken extends Token {}

With the classes exposed on the parseArgs export:

const {Token, PositionalToken} = parseArgs

Then I think you could just document the tokens as classes, similar to Buffer.blob, nested under the top level parseArgs docs, similar to how the docs are laid out for the JavaScript embedder API.

[bakkot]

Even if these are made classes they should definitely expose a kind property - it's really awkward to work with tree nodes which lack such a property, which is why pretty much every parser in the world has them. (Similarly, the DOM has nodeName in addition to nodes being instances of distinct classes.)

[bcoe]

Even if these are made classes they should definitely expose a kind property

Sounds good to me. I think it might be worth the class based approach, so that people could opt to check instanceof, if they're so inclined?

Should we consider making kind a Symbol, or is a string preferable?

[bakkot]

Strings are traditional - they're a lot easier to work with, since you don't have to import them.

[shadowspawn]

Just a thought, I would be tempted to make tokens a class rather than a POJO

It did cross my mind as soon as I added the kind property. 😄 .

The examples of Blob and AsyncResource and AsyncLocalStorage are all objects that authors will create themselves, with methods and state, where I think a Class adds more value.

I'm not against classes as such, but not seeing benefits in this case.

[aduh95]

My 2 cents is we should go with the first one and make it less verbose, the "existing" version can be a bit confusing I think. But feel free to disagree, none option is perfect anyway, and we can always edit the docs later if we find a better layout.

[ljharb]

I definitely find the existing one massively clearer; muddling things up with unnecessary TS/Java-ish terms and structure makes things more confusing.

[shadowspawn]

Mixed opinions straight away! Thanks both for feedback.

I will leave it as is, barring further feedback.

[bcoe]

I like that we could map the interface based approach directly to the TypeScript types we release on DefinitelyTyped.

But I don't feel super strongly, and would rather not block progress on the API (as @aduh95 says, we can edit the docs later).

Reapproving this PR 👍, @aduh95 are you happy as is?

[shadowspawn]

The compact POJO description is a bit subtle to read. How about an expanded version with all the properties listed?

Proposed expanded

A returned token has two properties which are always defined,
and some other properties which vary depending on the kind:

• kind {string} One of 'option', 'positional', or 'option-terminator'.
• index {number} Index of element in args containing token. So the
  source argument for a token is args[token.index].

An option token has additional parse details
for an option detected in the input args:

• kind = 'option'
• index {number} Index of element in args containing token.
• name {string} Long name of option.
• rawName {string} How option used in args, like -f of --foo.
• value {string | undefined} Option value specified in args.
  Undefined for boolean options.
• inlineValue {boolean | undefined} Whether option value specified inline,
  like --foo=bar.

A positional token has just one additional property with the positional value:

• kind = 'positional'
• index {number} Index of element in args containing token.
• value {string} The value of the positional argument in args (i.e. args[index]).

An option-terminator token has only the base properties:

• kind = 'option-terminator'
• index {number} Index of element in args containing token.

Old compact

The returned tokens have properties describing:

• all tokens
  • kind {string} One of 'option', 'positional', or 'option-terminator'.
  • index {number} Index of element in args containing token. So the
    source argument for a token is args[token.index].
• option tokens
  • name {string} Long name of option.
  • rawName {string} How option used in args, like -f of --foo.
  • value {string | undefined} Option value specified in args.
    Undefined for boolean options.
  • inlineValue {boolean | undefined} Whether option value specified inline,
    like --foo=bar.
• positional tokens
  • value {string} The value of the positional argument in args (i.e. args[index]).
• option-terminator token

(Duplicate post of: pkgjs/parseargs#129 (comment))

[bcoe]

CC: @aduh95, @cjihrig, @ruyadorno, who provided review for the first parseArgs PR.

[aduh95]

We would need to add an entry to the YAML comment to document the history of the API.

[shadowspawn]

Added a changelog entry to the YAML.

[bcoe]

I like the negate example 👌good demonstration of how this can be leveraged.

I'm fine with the POJO for tokens, but feels like we could figure out slightly better markdown syntax for describing the hierarchical relationship between tokens -- not a blocker for me.

Perhaps @Trott has some thoughts, who I believe has put a lot of work into making the reference docs top notch.

[Trott]

Perhaps @Trott has some thoughts, who I believe has put a lot of work into making the reference docs top notch.

I'm afraid that's not an aspect of things I've put a lot of thought to before and I don't have much to add to what you, @aduh95, and others have already said.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/45478/ ✅

[bcoe]

Tests are green, @aduh95 are you happy to merge? with the understanding we're likely to iterate more on docs over time.

[nodejs-github-bot]

Landed in d0dbb75