Search Results for author:
Found 12 papers, 7 papers with code
Retrieval Head Mechanistically Explains Long-Context Factuality
Despite the recent progress in long-context language models, it remains elusive how transformer-based models exhibit the capability to retrieve relevant information from arbitrary locations within the long context.
BitDelta: Your Fine-Tune May Only Be Worth One Bit
Large Language Models (LLMs) are typically trained in two phases: pre-training on large internet-scale datasets, and fine-tuning for downstream tasks.
InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory
To alleviate these issues, existing efforts employ sliding attention windows and discard distant tokens to achieve the processing of extremely long sequences.
Efficient Streaming Language Models with Attention Sinks
In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a "sink" even if they are not semantically important.
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
We then propose to search for the optimal per-channel scaling that protects the salient weights by observing the activation, not weights.
FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention
FastComposer proposes delayed subject conditioning in the denoising step to maintain both identity and editability in subject-driven image generation.
Ranked #7 on
Diffusion Personalization Tuning Free
on AgeDB
Sparse and Local Networks for Hypergraph Reasoning
Reasoning about the relationships between entities from input facts (e. g., whether Ari is a grandparent of Charlie) generally requires explicit consideration of other entities that are not mentioned in the query (e. g., the parents of Charlie).
Offsite-Tuning: Transfer Learning without Full Model
In this paper, we propose Offsite-Tuning, a privacy-preserving and efficient transfer learning framework that can adapt billion-parameter foundation models to downstream data without access to the full model.
FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Training
A key performance bottleneck when training graph neural network (GNN) models on large, real-world graphs is loading node features onto a GPU.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs.
Efficient Training and Inference of Hypergraph Reasoning Networks
To leverage the sparsity in hypergraph neural networks, SpaLoc represents the grounding of relationships such as parent and grandparent as sparse tensors and uses neural networks and finite-domain quantification operations to infer new facts based on the input.
Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attacks
In this work, we demonstrate the universal vulnerability of PTMs, where fine-tuned PTMs can be easily controlled by backdoor attacks in arbitrary downstream tasks.
