Definition
KV cache is an inference optimization that stores intermediate key/value states from attention during generation to speed up next-token computation. It is important for reducing latency in long outputs.
GlossaryAI term
KV Cache
KVキャッシュ
KVキャッシュ
Definition
KV cache is an inference optimization that stores intermediate key/value states from attention during generation to speed up next-token computation. It is important for reducing latency in long outputs.