Fix slow token rescanning during preprocessing.

[abbeyj]

The loop to rescan tokens during preprocessing can become very slow when expanding a macro that consists of many tokens. For instance, preprocessing the following can take several minutes:

void foo() {}

#define A foo();
#define FOO() \
    A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A \
    A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A \
    A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A \
    A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A \
    A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A \
    A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A \
    A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A \
    A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A \
    A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A \
    A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A

int main()
{
    FOO()
    return 0;
}

You can add more lines of As to make it take an arbitrarily long amount of time.

The problem is the calls like replTokens = replTokens.subList(...) on each iteration of the loop. subList doesn't return a new list but a SubList that's a "view" of the existing list. And calling subList on a SubList doesn't create a view of the underlying list but instead creates a view of the SubList or a 'view of a view". After running through the loop a few times you now have a "view of a view of a view of a view of ..." which gets very slow.

Modifying the existing list in-place solves this problem.

I'd like to have a test for this but I'm not sure how to write one when the only observable difference in behavior is how long it takes to run.

[guwirth]

There are many unit test for the preprocessor. If everything is still green it should work.