Search Results for author:
Found 2 papers, 0 papers with code
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
However, batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency.
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
SARATHI employs chunked-prefills, which splits a prefill request into equal sized chunks, and decode-maximal batching, which constructs a batch using a single prefill chunk and populates the remaining slots with decodes.
Cannot find the paper you are looking for? You can
Submit a new open access paper.
