Support transformers original behavior to load processors

[hans00]

As title.

I'm trying implement custom processor which has multiple tokenizer.
The original behavior will help to load it.

[xenova]

Thanks for this! This is exactly what I was planning to do for v3. Can you give the model id of the processor/tokenizer you are loading so I can add a unit test?

[hans00]

Thanks for this! This is exactly what I was planning to do for v3. Can you give the model id of the processor/tokenizer you are loading so I can add a unit test?

The processor still in WIP.
https://github.com/mybigday/Bert-VITS2/tree/convert-hf/hf

this model uses multiple tokenizer.