Is there a good method to set the generation_kwargs for tasks so that different kv caches can be used for optimization?

[CoderChen01]

In this transformers documentation, there are many kv cache methods available. However, I find it inconvenient to set past_key_values in the generation_kwargs of a task, for example, when using SinkCache. My proposed solution is to build a subclass of Task to implement setting past_key_values in the generation_kwargs. Is there a better way?

[baberabb]

Hi! you can pass it on to the command for ex. --gen_kwargs cache_implentation=....,<other kwargs to model.generate>

[CoderChen01]

Hi! you can pass it on to the command for ex. --gen_kwargs cache_implentation=....,<other kwargs to model.generate>

Thank you for your reply, but something like SinkCache cannot use parameters directly because it needs to be initialized in advance and passed to model.generate through past_key_values.