feat(mlx_lm): Nemotron (#949)

* feat: Nemotron

https://huggingface.co/nvidia/Minitron-4B-Base

This is basically Llama with partial RoPE and LayerNorm instead of
BatchNorm. Also they add 1 to the LayerNorm weight for some reason.

* fixup! feat: Nemotron

* nits

---------

Co-authored-by: Awni Hannun <awni@apple.com>
This commit is contained in:
L
2024-08-29 21:08:57 -07:00
committed by GitHub
parent b1186e2a81
commit fc93c55723
2 changed files with 228 additions and 0 deletions

View File

@@ -93,6 +93,7 @@ def linear_to_lora_layers(
"llama",
"phi",
"mixtral",
"nemotron",
"stablelm",
"qwen2",
"qwen2_moe",