From e434e7e5c2877535aea0aa6384fbe1f0f91f5646 Mon Sep 17 00:00:00 2001 From: Awni Hannun Date: Thu, 14 Dec 2023 15:40:38 -0800 Subject: [PATCH] incude instruct option --- mixtral/README.md | 35 +++++++++++++++++++++++++++++------ 1 file changed, 29 insertions(+), 6 deletions(-) diff --git a/mixtral/README.md b/mixtral/README.md index a90f7abf..3b0c50d0 100644 --- a/mixtral/README.md +++ b/mixtral/README.md @@ -2,6 +2,8 @@ Run the Mixtral[^mixtral] 8x7B mixture-of-experts (MoE) model in MLX on Apple silicon. +This example also supports the instruction fine-tuned Mixtral model.[^instruct] + Note, for 16-bit precision this model needs a machine with substantial RAM (~100GB) to run. ### Setup @@ -16,9 +18,23 @@ brew install git-lfs Download the models from Hugging Face: +For the base model use: + ``` -GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/ -cd Mixtral-8x7B-v0.1/ && \ +export MIXTRAL_MODEL=Mixtral-8x7B-v0.1 +``` + +For the instruction fine-tuned model use: + +``` +export MIXTRAL_MODEL=Mixtral-8x7B-Instruct-v0.1 +``` + +Then run: + +``` +GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mistralai/${MIXTRAL_MODEL}/ +cd $MIXTRAL_MODEL/ && \ git lfs pull --include "consolidated.*.pt" && \ git lfs pull --include "tokenizer.model" ``` @@ -27,7 +43,7 @@ Now from `mlx-exmaples/mixtral` convert and save the weights as NumPy arrays so MLX can read them: ``` -python convert.py --model_path Mixtral-8x7B-v0.1/ +python convert.py --model_path $MIXTRAL_MODEL/ ``` The conversion script will save the converted weights in the same location. @@ -37,8 +53,15 @@ The conversion script will save the converted weights in the same location. As easy as: ``` -python mixtral.py --model_path Mixtral-8x7B-v0.1/ +python mixtral.py --model_path $MIXTRAL_MODEL/ ``` -[^mixtral]: Refer to Mistral's [blog - post](https://mistral.ai/news/mixtral-of-experts/) for more details. +For more options including how to prompt the model, run: + +``` +python mixtral.py --help +``` + +[^mixtral]: Refer to Mistral's [blog post](https://mistral.ai/news/mixtral-of-experts/) for more details. +[^instruc]: Refer to the [Hugging Face repo](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) for more +details