Fix UnicodeDecodeError for BPE-based Models (especially GLM-4)

[GralchemOz]

Checklist:

• I have read the Contributing guidelines.

Description
This PR addresses a UnicodeDecodeError encountered in text-generation-webui when using models that rely on byte pair encoding (BPE) tokenizers, such as the GLM-4-9b model. The error occurs with certain characters that cannot be decoded due to incomplete byte sequences being passed to the utf-8 decoder.

Issue Example
When the model attempts to output characters like "簌", which is UTF-8 encoded as b'\xe7\xb0\xa8', the web UI tries to decode the output after each token. If only a partial sequence is available (e.g., b'\xe7\xb0'), a UnicodeDecodeError is raised, such as UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: unexpected end of data

[oobabooga]

Thanks for the fix