Juarez Bochi 4fe2fa2a64 GGUF: Avoid dequantization when format is compatible (#426) ...

* GGUF: Don't dequantize q4_1

* Fix weight order. First in low bits

* Add unpacking for q4_0

* Don't dequantize q8_0

* rebase quants and split file

* don't quantize every weight

* reapply patch

* error handling

---------

Co-authored-by: Awni Hannun <awni@apple.com>

2024-01-23 15:43:57 -08:00

14 KiB

Raw Blame History

View Raw