zhangyiss/mlx-examples
1
0
Fork 0
mirror of https://github.com/ml-explore/mlx-examples.git synced 2025-09-01 04:14:38 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

use official HF for mixtral

Browse Source
This commit is contained in:
Awni Hannun
2023-12-14 15:30:32 -08:00
parent 09fff84a85
commit 078fed3d8d
4 changed files with 54 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
mixtral/params.json Normal file
View File

@@ -0,0 +1 @@
{"dim": 4096, "n_layers": 32, "head_dim": 128, "hidden_dim": 14336, "n_heads": 32, "n_kv_heads": 8, "norm_eps": 1e-05, "vocab_size": 32000, "moe": {"num_experts_per_tok": 2, "num_experts": 8}}