Search Results for author:
Found 21 papers, 6 papers with code
Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention
The quadratic complexity of self-attention in Transformers has hindered the processing of long text.
Accelerating Generic Graph Neural Networks via Architecture, Compiler, Partition Method Co-Design
However, designing GNN accelerators faces two fundamental challenges: the high bandwidth requirement of GNN models and the diversity of GNN models.
AdaptGear: Accelerating GNN Training via Adaptive Subgraph-Level Kernels on GPUs
Graph neural networks (GNNs) are powerful tools for exploring and learning from graph structures and features.
Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator
Many researchers have focused on designing new forms of self-attention or introducing new parameters to overcome this limitation, however a large portion of them prohibits the model to inherit weights from large pretrained models.
Ranked #1 on
Open-Domain Question Answering
on ELI5
Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Training
An activation function is an element-wise mathematical function and plays a crucial role in deep neural networks (DNN).
ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization
In this work, we propose a fixed-length adaptive numerical data type called ANT to achieve low-bit quantization with tiny hardware overheads.
Efficient Adaptive Activation Rounding for Post-Training Quantization
To deal with the runtime overhead, we use a coarse-grained version of the border function.
SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences
However, the quadratic complexity of self-attention w. r. t the sequence length incurs heavy computational and memory burdens, especially for tasks with long sequences.
Transkimmer: Transformer Learns to Layer-wise Skim
To address the above limitations, we propose the Transkimmer architecture, which learns to identify hidden state tokens that are not required by each layer.
SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation
This paper proposes an on-the-fly DFQ framework with sub-second quantization time, called SQuant, which can quantize networks on inference-only devices with low computation and memory requirements.
Block-Skim: Efficient Question Answering for Transformer
We further prune the hidden states corresponding to the unnecessary positions early in lower layers, achieving significant inference-time speedup.
Dubhe: Towards Data Unbiasedness with Homomorphic Encryption in Federated Learning Client Selection
Federated learning (FL) is a distributed machine learning paradigm that allows clients to collaboratively train a model over their own local data.
Dual-side Sparse Tensor Core
We demonstrate the feasibility of our design with minimal changes to the existing production-scale inner-product-based Tensor Core.
Block Skim Transformer for Efficient Question Answering
Following this idea, we proposed Block Skim Transformer (BST) to improve and accelerate the processing of transformer QA models.
How Far Does BERT Look At: Distance-based Clustering and Analysis of BERT's Attention
Recent research on the multi-head attention mechanism, especially that in pre-trained models such as BERT, has shown us heuristics and clues in analyzing various aspects of the mechanism.
How Far Does BERT Look At:Distance-based Clustering and Analysis of BERT$'$s Attention
Recent research on the multi-head attention mechanism, especially that in pre-trained models such as BERT, has shown us heuristics and clues in analyzing various aspects of the mechanism.
Architectural Implications of Graph Neural Networks
Graph neural networks (GNN) represent an emerging line of deep learning models that operate on graph structures.
Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity
Network pruning can reduce the high computation cost of deep neural network (DNN) models.
Balancing Efficiency and Flexibility for DNN Acceleration via Temporal GPU-Systolic Array Integration
We propose Simultaneous Multi-mode Architecture (SMA), a novel architecture design and execution model that offers general-purpose programmability on DNN accelerators in order to accelerate end-to-end applications.
Adversarial Defense Through Network Profiling Based Path Extraction
Recently, researchers have started decomposing deep neural network models according to their semantics or functions.
Effective Path: Know the Unknowns of Neural Network
Despite their enormous success, there is still no solid understanding of deep neural network’s working mechanism.
