6 Commits

Author	SHA1	Message	Date
Alex Barron	2e0690374e	single sdpa function	2024-10-31 12:05:27 -07:00
Alex Barron	29f21e7fe4	add tests	2024-10-31 12:05:27 -07:00
Alex Barron	37a3723823	simplify	2024-10-31 12:05:27 -07:00
Alex Barron	48655a7f83	add QuantizedKVCache	2024-10-31 12:05:27 -07:00
Awni Hannun	8dca1a2f60	Tokenizer updates + tests (#1024) ... * tokenizer updates + tests * nit * add can_trim_prompt_cache * nits	2024-10-14 10:48:46 -07:00
Awni Hannun	fca087be49	More cache improvements (#1015) ... * fix rotating kv cache for chat use case * reorg + fixes to caching, unify prompt caching across types and use cases for e.g. caching during a chat * nit in chat * fix tests * fix tests * fix tests * docs * chat command * comments + docs * Define meta_state on all Cache implementations * fixes + trim_prompt_cache api * fix default model --------- Co-authored-by: Angelos Katharopoulos <a_katharopoulos@apple.com>	2024-10-07 20:45:51 -07:00