Fix loop in parser

[odersky]

I noted a case in Metals where the compiler would keep running at 100+ % until the process was killed.

Using jstack I tracked it down to an infinite skip caused by a syntaxError in pattern3. In fact,
the syntaxError should not skip at this point since the offending expression was already fully parsed.
I fixed this in this commit.

The parser was invoked from the signatureHelp method. It seems it parsed something that was not
syntactically correct (specifically, a postfix * appeared in a pattern where none was allowed).

`

[odersky]

The fix is actually not sufficient. I just found another infinite loop in the Parser, this time here:

"pool-4-thread-1" #82 prio=5 os_prio=31 tid=0x00007f7eb4308800 nid=0x12d53 runnable [0x0000700012a20000]
   java.lang.Thread.State: RUNNABLE
	at dotty.tools.dotc.parsing.Scanners$TokenData.copyFrom(Scanners.scala:61)
	at dotty.tools.dotc.parsing.Scanners$TokenData.copyFrom$(Scanners.scala:35)
	at dotty.tools.dotc.parsing.Scanners$ScannerCommon.copyFrom(Scanners.scala:97)
	at dotty.tools.dotc.parsing.Scanners$Scanner.getNextToken(Scanners.scala:382)
	at dotty.tools.dotc.parsing.Scanners$Scanner.nextToken(Scanners.scala:390)
	at dotty.tools.dotc.parsing.Scanners$Scanner.skip(Scanners.scala:309)
	at dotty.tools.dotc.parsing.Parsers$Parser.skip(Parsers.scala:256)
	at dotty.tools.dotc.parsing.Parsers$Parser.syntaxErrorOrIncomplete(Parsers.scala:281)
	at dotty.tools.dotc.parsing.Parsers$Parser.simpleExpr(Parsers.scala:2306)
	at dotty.tools.dotc.parsing.Parsers$Parser.$init$$$anonfun$9(Parsers.scala:2237)

It's again called from signatureHelp. It seems it always stays in the skip method of Scanner, in the while loop at the bottom of this snippet:

      def atStop =
        token == EOF
        || (currentRegion eq lastRegion)
            && (isStatSep
                || closingParens.contains(token) && lastRegion.toList.exists(_.closedBy == token)
                || token == COMMA && lastRegion.toList.exists(_.commasExpected)
                || token == OUTDENT && indentWidth(offset) < lastKnownIndentWidth)
          // stop at OUTDENT if the new indentwidth is smaller than the indent width of
          // currentRegion. This corrects for the problem that sometimes we don't see an INDENT
          // when skipping and therefore might erroneously end up syncing on a nested OUTDENT.
      if debugTokenStream then
        println(s"\nSTART SKIP AT ${sourcePos().line + 1}, $this in $currentRegion")
      while !atStop do
        nextToken()

It looks like something is wrong with the way the virtual file passed to the parser is set up that makes atStop be false forever. Does it end in EOF?

[odersky]

Another thing to try would be to make indent syntax be false for signature helps. (I assume it's not needed). That would prevent spurious insertions of indent and outdent. You can achieve this by passing a context with -noindent setting to the parser.

[tgodzik]

Thanks for looking into this! This might have caused the issue with the stuck compiler

[griggt]

@tgodzik I've hit this issue several times in the past week using Metals 0.11.6 with projects that are on Scala 3.1. Is there a corresponding workaround on the Metals side for those codebases not on 3.2? (since this looping bug in parser seems to be longstanding)

[alishangtian]

@tgodzik I've hit this issue several times in the past week using Metals 0.11.6 with projects that are on Scala 3.1. Is there a corresponding workaround on the Metals side for those codebases not on 3.2? (since this looping bug in parser seems to be longstanding)

The fix is actually not sufficient. I just found another infinite loop in the Parser, this time here:

"pool-4-thread-1" #82 prio=5 os_prio=31 tid=0x00007f7eb4308800 nid=0x12d53 runnable [0x0000700012a20000]
   java.lang.Thread.State: RUNNABLE
	at dotty.tools.dotc.parsing.Scanners$TokenData.copyFrom(Scanners.scala:61)
	at dotty.tools.dotc.parsing.Scanners$TokenData.copyFrom$(Scanners.scala:35)
	at dotty.tools.dotc.parsing.Scanners$ScannerCommon.copyFrom(Scanners.scala:97)
	at dotty.tools.dotc.parsing.Scanners$Scanner.getNextToken(Scanners.scala:382)
	at dotty.tools.dotc.parsing.Scanners$Scanner.nextToken(Scanners.scala:390)
	at dotty.tools.dotc.parsing.Scanners$Scanner.skip(Scanners.scala:309)
	at dotty.tools.dotc.parsing.Parsers$Parser.skip(Parsers.scala:256)
	at dotty.tools.dotc.parsing.Parsers$Parser.syntaxErrorOrIncomplete(Parsers.scala:281)
	at dotty.tools.dotc.parsing.Parsers$Parser.simpleExpr(Parsers.scala:2306)
	at dotty.tools.dotc.parsing.Parsers$Parser.$init$$$anonfun$9(Parsers.scala:2237)

It's again called from signatureHelp. It seems it always stays in the skip method of Scanner, in the while loop at the bottom of this snippet:

      def atStop =
        token == EOF
        || (currentRegion eq lastRegion)
            && (isStatSep
                || closingParens.contains(token) && lastRegion.toList.exists(_.closedBy == token)
                || token == COMMA && lastRegion.toList.exists(_.commasExpected)
                || token == OUTDENT && indentWidth(offset) < lastKnownIndentWidth)
          // stop at OUTDENT if the new indentwidth is smaller than the indent width of
          // currentRegion. This corrects for the problem that sometimes we don't see an INDENT
          // when skipping and therefore might erroneously end up syncing on a nested OUTDENT.
      if debugTokenStream then
        println(s"\nSTART SKIP AT ${sourcePos().line + 1}, $this in $currentRegion")
      while !atStop do
        nextToken()

It looks like something is wrong with the way the virtual file passed to the parser is set up that makes atStop be false forever. Does it end in EOF?

oh this looks so bad

[tgodzik]

I am trying to fix it for older versions in https://github.com/scalameta/metals/pull/4074/files and it seems to work, but I am not sure if it should. Does anyone knows how to get an infinitly compiling code? Would be cool to test out cancelation in that case.