Muhtasham Oblokulov 81e2a80026 Add Starcoder 2 (#502) ...

* Add Starcoder2 model and update utils.py

* Refactor model arguments and modules in starcoder2.py

* Refactor FeedForward class to MLP in starcoder2.py

* Fix typo

* pre-commit

* Refactor starcoder2.py: Update model arguments and modules

* Fix LM head and MLP layers

* Rename  input layer norm

* Update bias in linear layers

* Refactor token embeddings in Starcoder2Model

* Rename to standard HF attention layer name

* Add LayerNorm

* Add transposed token embeddings (like in Gemma)

* Refactor MLP and TransformerBlock classes

* Add tie_word_embeddings option to ModelArgs and update Model implementation

* Add conditional check for tying word embeddings in Starcoder2Model

* Fix bias in lm_head linear layer

* Remove unused LayerNorm in stablelm

* Update transformers dependency to use GitHub repository

* fix lm head bug, revert transformer req

* Update RoPE initialization in Attention class

---------

Co-authored-by: Awni Hannun <awni@apple.com>

2024-03-02 19:39:23 -08:00

..

__init__.py

feat: move lora into mlx-lm (#337)

2024-01-23 08:44:37 -08:00

lora.py

feat(mlx-lm): add de-quant for fuse.py (#365)

2024-01-25 18:59:32 -08:00

trainer.py

LoRA: adapter file Support path information (#505)

2024-02-29 22:20:49 -08:00

utils.py

Add Starcoder 2 (#502)

2024-03-02 19:39:23 -08:00