Search Results for author:
Found 1 papers, 0 papers with code
Hydragen: High-Throughput LLM Inference with Shared Prefixes
Decoding in this large-batch setting can be bottlenecked by the attention operation, which reads large key-value (KV) caches from memory and computes inefficient matrix-vector products for every sequence in the batch.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
