Enabling Efficient Low-bit Quantization based on Matrix Product Operators for KV Cache Compression

摘要

In this work, we introduce MPOQ, a novel data-free quantization technique based on matrix product operators (MPO) to effectively compress the KV cache.