3 Commits

Author	SHA1	Message	Date
Awni Hannun	2146bcd7ee	Quantize embedding / Update quantize API (#680) ... * more async eval * quantize embedding / update quantize api * more updates for quantize * update for quantize embeddings * update sd quant API * update sdxl quants * error for datasets < batch_size * async * fix config loading * fix quant * fix tests * fix req * remove lm head if tie weights is true * fix test	2024-04-18 18:16:10 -07:00
Angelos Katharopoulos	e55a9e8cb4	Add an SPM detokenizer that doesn't trim initial space (#681)	2024-04-15 14:15:25 -07:00
Angelos Katharopoulos	1278994b56	Add streaming detokenizers (#651)	2024-04-08 22:36:01 -07:00