Google unveils TurboQuant, a breakthrough compression algorithm for LLM key-value caches

Google Research has introduced TurboQuant, a novel vector quantization algorithm designed to dramatically reduce memory usage and speed up large language model (LLM) key-value (KV) caches without sacrificing accuracy.