4 Commits

Author	SHA1	Message	Date
Awni Hannun	fca087be49	More cache improvements (#1015) ... * fix rotating kv cache for chat use case * reorg + fixes to caching, unify prompt caching across types and use cases for e.g. caching during a chat * nit in chat * fix tests * fix tests * fix tests * docs * chat command * comments + docs * Define meta_state on all Cache implementations * fixes + trim_prompt_cache api * fix default model --------- Co-authored-by: Angelos Katharopoulos <a_katharopoulos@apple.com>	2024-10-07 20:45:51 -07:00
Awni Hannun	7be292c0c9	Handle longer prompt/generation (#931) ... * rebase * nits * nit * fix rotating cache with step prefill * update version	2024-08-16 15:28:39 -07:00
Awni Hannun	68e88d42fb	Fix server for openai package (#877) ... * fix * fixes for 9b	2024-07-08 12:34:31 -07:00
Awni Hannun	20e221f7f7	Add recurrent gemma (#856) ... * add recurrent gemma * fix window cache	2024-07-07 12:10:04 -07:00