gsub breaks \b for checking word boundaries

[thedward]

When I run the following program with jq -n:

"hello world"|"\\b(?<i>[[:lower:]])" as $rx|gsub($rx;.i|ascii_upcase)

Expected Output:

"Hello World"

Actual Output:

"HELLO WORLD"

I've got a workaround for my particular use case:

"hello world"|"\\b(?<fst>[[:lower:]])(?<rst>[[:lower:]]*)\\b" as $rx|gsub($rx;"\(.fst|ascii_upcase)\(.rst)")

[pkoppstein]

@thedward - As you've discovered, jq's interpretation of the "g" in gsub is currently different from the one you have in mind. Here's one way to achieve the effect you want:

def capitalize:
  def c: sub("\\b(?<i>[[:lower:]])";.i|ascii_upcase);
  c as $c | if . == $c then $c else ($c|c) end;

Example:

"hello world" | capitalize

[thedward]

It's also currently different than documented. :-)

I'm trying to write a correct implementation — if I succeed, I'll make a pull request.

[itchyny]

I'm uncertain but this looks the same instance of gsub bugs, like #1206, #2148.

[pkoppstein]

@itchyny wrote:

I'm uncertain but this looks the same instance of gsub bugs, like #1206, #2148.

With respect to the example given at the top of this thread, please note that gsub is working as intended, that is, in accordance with the idea that gsub(re; _) should repeatedly apply the transformation specified by sub(re;_) to non-overlapping substrings of the original input from left to right, until quiescence. That's not to say that this "specification" is desirable, but it is defensible.

[thedward]

@pkoppstein where is this behavior specified?

[pkoppstein]

@thedward - I wrote this "specification" to make it clear that I was referring to the previous sentence.
If you think that the existing documentation of gsub precludes the interpretation as iterative application, please just say so.

In any case, my hope is that whatever issues there may regarding the "iterative application" interpretation of gsub should not get in the way of fixing the infinite looping behavior described in #1206 and #2148.

[thedward]

@pkoppstein, I was just hoping there was a source other than the manual to shed more light on this.

Regardless, I don't agree that the existing behavior is consistent with the manual or with your informal specification because the regular expression \b(?<i>[[:lower:]] does not match the 'e' in "hello world". That it matches the 'e' in "ello world" should be irrelevant based on either the manual's description —"gsub is like sub but all the non-overlapping occurrences of the regex are replaced by the string, after interpolation."— or your informal specification.

I also have a hard time accepting the idea that someone intended the existing behavior both because it violates the principal of least surprise and because it makes the function less useful without gaining anything in return.

I really don't understand why you'd want to reframe this as anything other than a bug.

Since changing the behavior could conceivably break existing programs, it's certainly possible to argue it's a documentation bug at this point, but it's definitely a bug.

p.s. I'm really excited to see jq development picking up some speed!

[pkoppstein]

@thedward @itchyny - The proposed new version of gsub (#2624) addresses this and many other issues.
This would be a good time to try to ensure that known gsub bugs are resolved if possible.
With that end in mind, here is an extract from my current tests/onig.test showing the new test cases.
Please feel free to suggest others.

The new version uses match(_; "g"), which unfortunately seems to have a bug in the case of match("";"g").
For example, "a" | match("";"g") only returns one match rather than two. That's why
the last test case (involving gsub(""; "a")) has the comment about its being a regression test case.
I propose retaining it, as at least it verifies termination (whereas with e.g. jq 1.6 it does not).

(The tests are presented as triples, each of which can be read as: asserteq(LINE1 | LINE2; LINE3).)

gsub( ""; "a";  "g")
""
"a"

gsub( "^"; "";  "g")
"a"
"a"

gsub( "$"; "a";  "g")
"a"
"aa"

gsub( "^"; "a")
""
"a"

gsub("(?=u)"; "u")
"qux"
"quux"

gsub("^.*a"; "b")
"aaa"
"b"

gsub("^.*?a"; "b")
"aaa"
"baa"

gsub("(?<x>.)(?<y>[0-9])"; "\(.x|ascii_downcase)\(.y)")
"A1 B2 CD"
"a1 b2 CD"

gsub("\\b(?<x>.)"; "\(.x|ascii_downcase)")
"ABC DEF"
"aBC dEF"

# Note: the following is just a regression test case.  The result should be "aaa".
gsub( ""; "a";  "g")
"a"
"aa"