59 Commits

Author	SHA1	Message	Date
Anchen	0ab01b4626	fix(mlx-lm): sorted probs in top_p implementation. (#610) ... * fix(mlx-lm): the top p imp * chore: address comment	2024-03-25 15:07:55 -07:00
Awni Hannun	b8a348c1b8	Switch to fast RMS/LN Norm (#603) ... * use nn.RMSNorm, use sdpa, cleanup * bump mlx versions * minor update * use fast layer norm * version bump * update requirement for whisper * update requirement for gguf	2024-03-23 07:13:51 -07:00
Anchen	fbed720d6f	chore(mlx-lm): fix the top_p implementation. (#602) ... * chore(mlx-lm): clean up the top p imp * chore: clean up * chore: add test * chore: address comments * chore: clean up docs string * chore: clean up test	2024-03-21 12:18:23 -07:00
Anchen	949f63f309	chore(mlx-lm): fix print_trainable_parameters for quant models (#581) ... * chore(mlx-lm): fix print_trainable_parameters for quant models * chore: clean up * refactor: use layer type to check quant bits * chore: address comment	2024-03-20 08:41:03 -07:00
madroid	b0bcd86a40	Support for OpenAI’s fine-tuning dataset format (#548) ... * LoRA: move load_dataset to tuner/datasets.py file * LoRA: support OpenAI chat format datasets see https://platform.openai.com/docs/guides/fine-tuning/example-format * LoRA: support OpenAI completion format datasets * LoRA: formatting dataset timing to reduce memory footprint * Refactor dataset item access in PromptCompletionDataset * Update mlx_lm/LORA.md * Update mlx_lm/LORA.md * check Unsupported data format * add tests, fine-tune doc * add tests, fine-tune doc * add jinja2 for chat template * nits in readme * nits in readme --------- Co-authored-by: Awni Hannun <awni@apple.com>	2024-03-19 16:45:46 -07:00
Prince Canuma	76c3244cc5	Add support for Cohere's Command-R (#565) ... * initial commit for command-R * update mlp, layernorm, lm_head and model args * add custom layernorm * add default to tie_word_embeddings * add layernorm weight type and refactor * update layernorm (bias conditional) in model/layers * fix layer norm use traditional rope * add test --------- Co-authored-by: Awni Hannun <awni@apple.com>	2024-03-13 07:03:36 -07:00
Anchen	3535408c99	chore(mlx-lm): fix tie_word_embeddings for qwen2 (#566) ... * chore: fix tie_word_embeddings for qwen2 * chore: default tie_word_embeddings to True	2024-03-12 21:34:32 -07:00
Chime Ogbuji	e56d9015ef	LoRA on all linear transformer block layers (#546) ... * Add --lora-all-linear option to apply LoRa to all linear transfer block layers * Moved to YAML config and added specification of rank & alpha * nits in conifg, more tests * nit * run tests for prs --------- Co-authored-by: Awni Hannun <awni@apple.com>	2024-03-12 07:37:40 -07:00
Awni Hannun	7cdd1b69ac	Enable unit testing in Circle and start some MLX LM tests (#545) ... * add a few tests for mlx lm * add a few tests for mlx lm * add a few tests for mlx lm * more tests / cleanup	2024-03-07 09:31:57 -08:00