4 Commits

Author	SHA1	Message	Date
Ivan Fioravanti	6ae6c72c2e	reduction moved to CPU in case of distributed training (#1200)	2025-01-14 17:20:42 -08:00
Angelos Katharopoulos	331148d8ec	Enable distributed LoRA training (#821)	2024-11-02 18:02:31 -07:00
Zai Thottakath	418d9a5511	Feature: QDoRA (#891) ... * feat: QDoRA with tests and a small bug fix for recalculation of self.m * some simplifications and fixes --------- Co-authored-by: Awni Hannun <awni@apple.com>	2024-09-30 08:01:11 -07:00
Zai Thottakath	4e01700816	Allow the entire model to be targed for LoRA and DoRA fine tuning: LoRA and DoRA embeddings with small DoRALinear bug fix (#914) ... * feature: LoRA adapter for Embeddings * feature: wire in LoRAEmbedding into the tuner. Allow the embedding and non model.layers Linear layers to be targeted for fine tuning * feature: DoRA adapter for Embeddings * feature: wire in DoRAEmbedding * bugfix: ensure self.m is recalculated when the linear layer is changed in DoRALinear.from_linear * refactor: prefer from_base over from_linear or from_embedding. prefer fuse over to_linear or to_embedding * cleanup: remove unused imports in test_dora.py * refactor: remove unnecessary non_layer_modules * cleanup: remove wrong comments for lora embedding dropout. remove uncessary parens in dora embedding dropout * nits --------- Co-authored-by: Awni Hannun <awni@apple.com>	2024-08-16 07:38:36 -07:00