diff --git a/lora/README.md b/lora/README.md index 69ca3899..09911bd3 100644 --- a/lora/README.md +++ b/lora/README.md @@ -27,10 +27,10 @@ If you do not have access to the Llama weights you will need to [request access](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform) from Meta. -Convert the weights with: +Convert the model with: ``` -python convert.py +python convert.py ``` ## Run @@ -52,7 +52,7 @@ python lora.py --model \ ``` Note, the model path should have the MLX weights, the tokenizer, and the -`params.json` configuration which will all be output by the `conver.py` script. +`params.json` configuration which will all be output by the `convert.py` script. By default, the adapter weights are saved in `adapters.npz`. You can specify the output location with `--adapter_file`. @@ -96,8 +96,6 @@ training and validation loss at a few points over the course of training. | 800 | 1.017 | 1.255 | | 1000 | 1.070 | 1.230 | -After training for 1000 iterations, the validation perplexity reduces to XX. - The model trains at around 475 tokens per second on an M2 Ultra. [^lora]: Refer to the [arXiv paper](https://arxiv.org/abs/2106.09685) for more details on LoRA. diff --git a/lora/requirements.txt b/lora/requirements.txt index c036fa59..7111f1d4 100644 --- a/lora/requirements.txt +++ b/lora/requirements.txt @@ -1,2 +1,3 @@ +mlx sentencepiece torch