zhangyiss/mlx-examples
1
0
Fork 0
mirror of https://github.com/ml-explore/mlx-examples.git synced 2025-11-01 03:28:08 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

incude instruct option

Browse Source
This commit is contained in:
Awni Hannun
2023-12-14 15:40:38 -08:00
parent 078fed3d8d
commit e434e7e5c2
1 changed files with 29 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
mixtral/README.md
View File

@@ -2,6 +2,8 @@
Run the Mixtral[^mixtral] 8x7B mixture-of-experts (MoE) model in MLX on Apple silicon.
This example also supports the instruction fine-tuned Mixtral model.[^instruct]
Note, for 16-bit precision this model needs a machine with substantial RAM (~100GB) to run.
### Setup
@@ -16,9 +18,23 @@ brew install git-lfs
Download the models from Hugging Face:
For the base model use:
```
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/
cd Mixtral-8x7B-v0.1/ && \
export MIXTRAL_MODEL=Mixtral-8x7B-v0.1
```
For the instruction fine-tuned model use:
```
export MIXTRAL_MODEL=Mixtral-8x7B-Instruct-v0.1
```
Then run:
```
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mistralai/${MIXTRAL_MODEL}/
cd $MIXTRAL_MODEL/ && \
git lfs pull --include "consolidated.*.pt" && \
git lfs pull --include "tokenizer.model"
```
@@ -27,7 +43,7 @@ Now from `mlx-exmaples/mixtral` convert and save the weights as NumPy arrays so
MLX can read them:
```
python convert.py --model_path Mixtral-8x7B-v0.1/
python convert.py --model_path $MIXTRAL_MODEL/
```
The conversion script will save the converted weights in the same location.
@@ -37,8 +53,15 @@ The conversion script will save the converted weights in the same location.
As easy as:
```
python mixtral.py --model_path Mixtral-8x7B-v0.1/
python mixtral.py --model_path $MIXTRAL_MODEL/
```
[^mixtral]: Refer to Mistral's [blog
post](https://mistral.ai/news/mixtral-of-experts/) for more details.
For more options including how to prompt the model, run:
```
python mixtral.py --help
```
[^mixtral]: Refer to Mistral's [blog post](https://mistral.ai/news/mixtral-of-experts/) for more details.
[^instruc]: Refer to the [Hugging Face repo](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) for more
details