LightConv is a type of depthwise convolution for sequential modelling which shares certain output channels and whose weights are normalized across the temporal dimension using a softmax. Compared to self-attention, LightConv has a fixed context window and it determines the importance of context elements with a set of weights that do not change over time steps. LightConv computes the following for the $i$-th element in the sequence and output channel $c$:
$$ \text{LightConv}\left(X, W_{\text{ceil}\left(\frac{cH}{d}\right),:}, i, c\right) = \text{DepthwiseConv}\left(X,\text{softmax}\left(W_{\text{ceil}\left(\frac{cH}{d}\right),:}\right), i, c\right) $$
Source: Pay Less Attention with Lightweight and Dynamic ConvolutionsPaper | Code | Results | Date | Stars |
---|
Task | Papers | Share |
---|---|---|
Abstractive Text Summarization | 1 | 25.00% |
Language Modelling | 1 | 25.00% |
Machine Translation | 1 | 25.00% |
Translation | 1 | 25.00% |