Search Results for author:
Found 11 papers, 3 papers with code
CacheGen: KV Cache Compression and Streaming for Fast Language Model Serving
Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3. 7-4. 3x and the total delay in fetching and processing contexts by 2. 7-3. 2x while having negligible impact on the LLM response quality in accuracy or perplexity.
OneAdapt: Fast Adaptation for Deep Learning Applications via Backpropagation
While the high demand for network bandwidth and GPU resources could be substantially reduced by optimally adapting the configuration knobs, such as video resolution and frame rate, current adaptation techniques fail to meet three requirements simultaneously: adapt configurations (i) with minimum extra GPU or bandwidth overhead; (ii) to reach near-optimal decisions based on how the data affects the final DNN's accuracy, and (iii) do so for a range of configuration knobs.
GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edge
Video analytics pipelines have steadily shifted to edge deployments to reduce bandwidth overheads and privacy violations, but in doing so, face an ever-growing resource tension.
Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers
Compressed models that are deployed on the edge servers for inference suffer from data drift, where the live video data diverges from the training data.
Visor: Privacy-Preserving Video Analytics as a Cloud Service
Video-analytics-as-a-service is becoming an important offering for cloud providers.
Machine Learning at the Network Edge: A Survey
To address this issue, efforts have been made to place additional computing devices at the edge of the network, i. e close to the IoT devices where the data is generated.
Collage Inference: Achieving low tail latency during distributed image classification using coded redundancy models
In this work, we argue that MLaaS platforms also provide unique opportunities to cut the cost of redundancy.
Collage Inference: Using Coded Redundancy for Low Variance Distributed Image Classification
Deploying the collage-cnn models in the cloud, we demonstrate that the 99th percentile tail latency of inference can be reduced by 1. 2x to 2x compared to replication based approaches while providing high accuracy.
ReXCam: Resource-Efficient, Cross-Camera Video Analytics at Scale
Enterprises are increasingly deploying large camera networks for video analytics.
Scaling Video Analytics Systems to Large Camera Deployments
Driven by advances in computer vision and the falling costs of camera hardware, organizations are deploying video cameras en masse for the spatial monitoring of their physical premises.
Focus: Querying Large Video Datasets with Low Latency and Low Cost
Focus handles the lower accuracy of the cheap CNNs by judiciously leveraging expensive CNNs at query-time.
