Option to allow Commas in Parse Parameters Tokenizer

[zachleat]

While upgrading to use Liquid’s parameter parser in Eleventy (and away from our home-grown one that is old): 11ty/eleventy#2679 one of the biggest backwards compatibility issues that I think we’ll have is that our custom parser allowed the use of commas to separate parameters and the Liquid one does not (nor does it fail with an error—looks like it hangs).

https://liquidjs.com/tutorials/parse-parameters.html#Parse-Parameters-as-Values

I’m curious if this can be improved—and best case scenario would be to add support for optional commas! Or maybe an opt-in option? Thanks!

[harttle]

.peek(). .advance() can be used for such cases. Feel free to open another issue if this doesn't resolve the problem. I guess it hangs because .readValue() keep returns null and the while loop won't exit. The following snippet resolves the issue.

const { Liquid, Tokenizer, evalToken } = require('liquidjs')
const liquid = new Liquid()

liquid.registerTag('random', {
  parse(tagToken) {
    const tokenizer = new Tokenizer(tagToken.args)
    this.items = []
    let value
    while (value = tokenizer.readValue()) {
      this.items.push(value)
      tokenizer.skipBlank()
      while (tokenizer.peek() === ',') tokenizer.advance()
    }
    assert(tokenizer.end(), 'invalid chars')
  },
  * render(context, emitter) {
    const index = Math.floor(this.items.length * Math.random())
    const token = this.items[index]
    // in LiquidJS, we use yield to wait for async call
    const value = yield evalToken(token, context)
    emitter.write(value)
  }
})

const html = liquid.parseAndRenderSync(`
  {% for i in (1..10) %}{% random "a", 1, 2 3,4,, 5 %}{% endfor %}
`)

console.log(html)

// throws 'invalid chars'
// liquid.parseAndRenderSync(`{% random 1,2< %}`)

[zachleat]

Amazing, thank you!