[Bugfix] Fix bug in detokenizer.py

[cafeii]

Fix the bug where the length of prev_tokens and next_iter_tokens in detokenizer.py would grow exponentially in some cases.

The original code assign the reference of next_iter_tokens to prev_tokens directly, making them point to a same object. When prev_tokens is extended by next_iter_tokens, both of their length will be doubled, which makes the length grows exponentially.

ps: I didn't find any issue that directly point out this bug.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

• Comment /ready on the PR
• Add ready label to the PR
• Enable auto-merge.

🚀

[cafeii]

/ready

[zifeitong]

Do you have a reproducer?

#5919 is almost the same.

#6223 is also related.

[cafeii]

reproduce.zip

• vllm: 0.5.1
• transformers 4.45.0.dev0
• tokenizers 0.19.1

If successfully reproduced, it will slow down at the 8th sample, and completely stuck at 9th sample.
The bug's behavior is similar to #5872, but I didn't encounter the same bug as #5872 while reproducing it.
#6223 can also fix the bug.
#5919 is almost the same as this PR.

[njhill]

Addressed by #5919