mirror of
https://github.com/ml-explore/mlx-examples.git
synced 2025-07-25 22:01:17 +08:00
758f05c09a
* refactor: merge deepseek coder example into hf_llm example * remove deepseek example * chore: fix format in readme * chore: remove default rope_scaling dict and use get to access type and factor to avoid key error * Update llms/hf_llm/models.py Co-authored-by: Awni Hannun <awni.hannun@gmail.com> * chore: fix lint --------- Co-authored-by: Awni Hannun <awni.hannun@gmail.com>
78 lines
2.4 KiB
Markdown
78 lines
2.4 KiB
Markdown
## Generate Text with MLX and :hugs: Hugging Face
|
|
|
|
This an example large language model text generation that can pull models from
|
|
the Hugging Face Hub.
|
|
|
|
### Setup
|
|
|
|
Install the dependencies:
|
|
|
|
```
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
### Run
|
|
|
|
```
|
|
python generate.py --model <model_path> --prompt "hello"
|
|
```
|
|
|
|
For example:
|
|
|
|
```
|
|
python generate.py --model mistralai/Mistral-7B-v0.1 --prompt "hello"
|
|
```
|
|
|
|
will download the Mistral 7B model and generate text using the given prompt.
|
|
|
|
The `<model_path>` should be either a path to a local directory or a Hugging
|
|
Face repo with weights stored in `safetensors` format. If you use a repo from
|
|
the Hugging Face Hub, then the model will be downloaded and cached the first
|
|
time you run it. See the [Models](#models) section for a full list of supported models.
|
|
|
|
Run `python generate.py --help` to see all the options.
|
|
|
|
|
|
### Models
|
|
|
|
The example supports Hugging Face format Mistral and Llama-style models. If the
|
|
model you want to run is not supported, file an
|
|
[issue](https://github.com/ml-explore/mlx-examples/issues/new) or better yet,
|
|
submit a pull request.
|
|
|
|
Here are a few examples of Hugging Face models which work with this example:
|
|
|
|
- [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
|
|
- [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)
|
|
- [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T)
|
|
- [deepseek-ai/deepseek-coder-6.7b-instruct](https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct)
|
|
- [01-ai/Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat)
|
|
|
|
Most
|
|
[Mistral](https://huggingface.co/models?library=transformers,safetensors&other=mistral&sort=trending)
|
|
and
|
|
[Llama](https://huggingface.co/models?library=transformers,safetensors&other=llama&sort=trending)
|
|
style models should work out of the box.
|
|
|
|
### Convert new models
|
|
|
|
You can convert (change the data type or quantize) models using the
|
|
`convert.py` script. This script takes a Hugging Face repo as input and outputs
|
|
a model directory (which you can optionally also upload to Hugging Face).
|
|
|
|
For example, to make 4-bit quantized a model, run:
|
|
|
|
```
|
|
python convert.py --hf-path <hf_repo> -q
|
|
```
|
|
|
|
For more options run:
|
|
|
|
```
|
|
python convert.py --help
|
|
```
|
|
|
|
You can upload new models to the [Hugging Face MLX
|
|
Community](https://huggingface.co/mlx-community) by specifying `--upload-name``
|
|
to `convert.py`.
|