Research Study: Effectiveness of Adaptive Merging for Recycling LoRA Modules from Public Repositories
By
PaulHoule
Toasted to a respectable shade. No regrets, no crumbs left.
Summary
This research paper examines the effectiveness of adaptive merging methods for recycling LoRA (Low-Rank Adaptation) modules from public repositories like Hugging Face Hub. The study analyzes nearly 1,000 user-contributed LoRAs trained from the Llama 3.1 8B-Instruct model, comparing adaptive and non-adaptive merging methods. Key findings reveal that while adaptive merging can improve performance over base models, it offers limited benefits compared to training new LoRAs on the same data. Surprisingly, the specific choice of LoRAs to merge has little importance, and even randomly initialized LoRAs yield similar performance, suggesting the method works primarily through regularization rather than positive cross-task transfer. The research confirms positive transfer is only possible when highly relevant LoRAs are available in the pool.
Key quotes
· 4 pulledWhile adaptive merging methods have demonstrated improvements in some settings, no past work has attempted to recycle LoRAs found 'in the wild' on model repositories like the Hugging Face Hub.
We demonstrate that adaptive merging methods can improve performance over the base model but provide limited benefit over training a new LoRA on the same data used to set merging coefficients.
We additionally find not only that the specific choice of LoRAs to merge has little importance, but that using LoRAs with randomly initialized parameter values yields similar performance.
This raises the possibility that adaptive merging from recycled LoRAs primarily works via some kind of regularization effect, rather than by enabling positive cross-task transfer.
