Goal

PEFT library provides adapter based parameter efficient fine-tuning method called LoRA. We already integrate with peft library in peft text generation module. This milestone is to extend this support and add LoRA as one of the supported methods. However, we want to provide support for LoRA in 2 fashion:

1. Ability to train LoRA and save merged weights in the form of new base model as output.
2. Ability to train LoRA and generate adapters as output (not base model)

To enable this, we would essentially leverage both of our text generation modules in more specific defined form:

1. peft_prompt_tuning will be used for all module that generates "adapters" or "tuned prompts" as output.
2. text_generation_local would be used for generating modules in "fine-tuning" fashion, thus always returning modified base model.

To accomplish above task cleaning and add support for LoRA in text_generation_local module, we would need to integrate peft library in both of the modules above. To avoid duplicate codes, the best way to achieve above would be to abstract out prompt tuning and fine-tuning code to external toolkit functions and use those in both prompt tuning and fine-tuning modules.

This does mean that fine-tuning parameters will now look a bit similar to prompt tuning ones and also we would need to add a new set of parameters for prompt tuning one. This would require generation of new tuning configs in data models.