zhangyiss/mlx-examples
1
0
Fork 0
mirror of https://github.com/ml-explore/mlx-examples.git synced 2025-06-25 01:41:19 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

incude instruct option

Browse Source
This commit is contained in:
Awni Hannun 2023-12-14 15:40:38 -08:00
parent 078fed3d8d
commit e434e7e5c2
1 changed files with 29 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

35
mixtral/README.md
View File

@ -2,6 +2,8 @@
Run the Mixtral[^mixtral] 8x7B mixture-of-experts (MoE) model in MLX on Apple silicon. Run the Mixtral[^mixtral] 8x7B mixture-of-experts (MoE) model in MLX on Apple silicon.
This example also supports the instruction fine-tuned Mixtral model.[^instruct]
Note, for 16-bit precision this model needs a machine with substantial RAM (~100GB) to run. Note, for 16-bit precision this model needs a machine with substantial RAM (~100GB) to run.
### Setup ### Setup
@ -16,9 +18,23 @@ brew install git-lfs
Download the models from Hugging Face: Download the models from Hugging Face:
For the base model use:
``` ```
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/ export MIXTRAL_MODEL=Mixtral-8x7B-v0.1
cd Mixtral-8x7B-v0.1/ && \ ```
For the instruction fine-tuned model use:
```
export MIXTRAL_MODEL=Mixtral-8x7B-Instruct-v0.1
```
Then run:
```
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mistralai/${MIXTRAL_MODEL}/
cd $MIXTRAL_MODEL/ && \
git lfs pull --include "consolidated.*.pt" && \ git lfs pull --include "consolidated.*.pt" && \
git lfs pull --include "tokenizer.model" git lfs pull --include "tokenizer.model"
``` ```
@ -27,7 +43,7 @@ Now from `mlx-exmaples/mixtral` convert and save the weights as NumPy arrays so
MLX can read them: MLX can read them:
``` ```
python convert.py --model_path Mixtral-8x7B-v0.1/ python convert.py --model_path $MIXTRAL_MODEL/
``` ```
The conversion script will save the converted weights in the same location. The conversion script will save the converted weights in the same location.
@ -37,8 +53,15 @@ The conversion script will save the converted weights in the same location.
As easy as: As easy as:
``` ```
python mixtral.py --model_path Mixtral-8x7B-v0.1/ python mixtral.py --model_path $MIXTRAL_MODEL/
``` ```
[^mixtral]: Refer to Mistral's [blog For more options including how to prompt the model, run:
post](https://mistral.ai/news/mixtral-of-experts/) for more details.
```
python mixtral.py --help
```
[^mixtral]: Refer to Mistral's [blog post](https://mistral.ai/news/mixtral-of-experts/) for more details.
[^instruc]: Refer to the [Hugging Face repo](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) for more
details