6 Commits

Author	SHA1	Message	Date
Billel Mokeddem	845efddc8c	Fix decoding manually added tokens (#1164) ... * Fix decoding manually added tokens * fix + test * nit * nit * no lag bpe --------- Co-authored-by: Awni Hannun <awni@apple.com>	2024-12-17 09:54:29 -08:00
Awni Hannun	0f135396ae	Generation refactor: part 2 (#1099) ... * unify with stream_generate * fixes * nit * some cleanup, warnings, tests * fix test + faster min p + test * version	2024-11-23 11:47:06 -08:00
Awni Hannun	6fd1f70f73	fix spm decoder multi-byte (#1092)	2024-11-05 06:06:26 -08:00
Awni Hannun	0f799947d0	fix (#1079)	2024-11-01 16:30:32 -07:00
Awni Hannun	8fe9539af7	Fix detokenizer space match for quote (#1072) ... * fix + test * remove transformer flax/torch warning * format	2024-10-27 15:06:07 -07:00
Awni Hannun	8dca1a2f60	Tokenizer updates + tests (#1024) ... * tokenizer updates + tests * nit * add can_trim_prompt_cache * nits	2024-10-14 10:48:46 -07:00