Commit Graph

3 Commits

Author SHA1 Message Date
Awni Hannun
9900d2230a fix 2024-10-31 11:57:13 -07:00
Awni Hannun
8fe9539af7 Fix detokenizer space match for quote (#1072)
* fix + test

* remove transformer flax/torch warning

* format
2024-10-27 15:06:07 -07:00
Awni Hannun
8dca1a2f60 Tokenizer updates + tests (#1024)
* tokenizer updates + tests

* nit

* add can_trim_prompt_cache

* nits
2024-10-14 10:48:46 -07:00