7 Commits

Author	SHA1	Message	Date
Angelos Katharopoulos	9f671228cd	Block sparse MM MoEs (#782) ... - Adds SwitchLinear - Adds QuantizedSwitchLinear	2024-05-21 15:58:08 -07:00
alexC-nonsense4k	42458914c8	support dora finetune in mlx-examples/llms/mlx_lm (#779) ... * support dora finetune * solve problems in lora.py and tuner.utils.py * add use_dora (bool) in functions of load adapters * delete all unsupported quantization code and fix all the calculate problems in mlx_lm/tuner/dora.py * Using stop_gradient to prevent gradients from flowing through ‘norm’ during backpropagation * set DEFAULT_USE_DORA in mlx_lm/generate.py * add annotation for all the use_dora * mlx_lm/fuse.py support fuse dora layers and fix a bug of to_linear() in mlx_lm/tuner/dora.py * simplify code of juding type of a fused layer in mlx_lm/fuse.py * add use_dora in mlx_lm/fuse.py when apply_lora_layers() * style + nits * style + nits * more updates --------- Co-authored-by: chenyifei08 <chenyifei08@baidu.com> Co-authored-by: Awni Hannun <awni@apple.com>	2024-05-16 08:21:26 -07:00
Awni Hannun	2bd64b78cf	Save lora config (#636) ... * lora config * comments * version bump	2024-04-02 13:52:53 -07:00
Anchen	fe96ef342f	feat(mlx-lm): export the GGUF (fp16) format model weights from fuse.py (#555) ... * wip * wip * feat: convert mlx model to gguf f16 * chore: conver norm layer to float32 to avoid overflow issue * chore: add support for mixtral * chore: clean up * chore: remove unused import statement * chore: clean up weight name mapping * version and readme * actual version bump --------- Co-authored-by: Awni Hannun <awni@apple.com>	2024-03-21 10:34:11 -07:00
Sugato Ray	2cd793dd69	feat: add update_config functionality (#531) ... * feat: add `update_config` finctionality - sorts the config for better readability - updates "_name_or_path" key in config with upload_repo - sets indentation of 4 spaces - allows adding other key-value pairs via kwargs - reduces code duplication - standardizes config-update across mlx-lm * feat: standardize updating config Impactes: - fuse.py - merge.py * update formatting * remove commented out code * update func: update_config to save_config - drop kwards - rename func as save_config - incorporate review suggestions * update func: save_config - ensure only config-saving functionality - function oes not return config as a dict anymore - added review suggestions * fixed formatting * update formatting instruction in contribution guide * nits --------- Co-authored-by: Awni Hannun <awni@apple.com>	2024-03-14 06:36:05 -07:00
Anchen	854ad8747a	feat(mlx-lm): add de-quant for fuse.py (#365) ... * feat(mlx-lm): add de-quant for fuse * chore: disable quant in to linear when de-quant enabled * chore: add better error handling for adapter file not found	2024-01-25 18:59:32 -08:00
Anchen	362e88a744	feat: move lora into mlx-lm (#337) ... * feat: Add lora and qlora training to mlx-lm --------- Co-authored-by: Awni Hannun <awni@apple.com>	2024-01-23 08:44:37 -08:00