6 Commits

Author	SHA1	Message	Date
N8	ac58a95fbd	Add rotating kvcache to save space	2024-12-14 17:08:06 -05:00
Prince Canuma	406c7f300f	add coauthor :) ... Co-authored-by: n8programs <43304488+N8python@users.noreply.github.com>	2024-12-14 17:12:49 +01:00
Prince Canuma	d7d70487eb	fix sliding window	2024-12-14 17:06:57 +01:00
Prince Canuma	52595dafae	fix tests and sliding window attention	2024-12-14 16:39:22 +01:00
Prince Canuma	5d8b36ce7c	revert to act_fn to silu	2024-12-14 16:22:00 +01:00
Prince Canuma	d7c64c621a	add support for cohere2	2024-12-14 16:15:24 +01:00