## Non-Local Operation

Introduced by Wang et al. in Non-local Neural Networks

A Non-Local Operation is a component for capturing long-range dependencies with deep neural networks. It is a generalization of the classical non-local mean operation in computer vision. Intuitively a non-local operation computes the response at a position as a weighted sum of the features at all positions in the input feature maps. The set of positions can be in space, time, or spacetime, implying that these operations are applicable for image, sequence, and video problems.

Following the non-local mean operation, a generic non-local operation for deep neural networks is defined as:

$$\mathbb{y}_{i} = \frac{1}{\mathcal{C}\left(\mathbb{x}\right)}\sum_{\forall{j}}f\left(\mathbb{x}_{i}, \mathbb{x}_{j}\right)g\left(\mathbb{x}_{j}\right)$$

Here $i$ is the index of an output position (in space, time, or spacetime) whose response is to be computed and $j$ is the index that enumerates all possible positions. x is the input signal (image, sequence, video; often their features) and $y$ is the output signal of the same size as $x$. A pairwise function $f$ computes a scalar (representing relationship such as affinity) between $i$ and all $j$. The unary function $g$ computes a representation of the input signal at the position $j$. The response is normalized by a factor $C\left(x\right)$.

The non-local behavior is due to the fact that all positions ($\forall{j}$) are considered in the operation. As a comparison, a convolutional operation sums up the weighted input in a local neighborhood (e.g., $i − 1 \leq j \leq i + 1$ in a 1D case with kernel size 3), and a recurrent operation at time $i$ is often based only on the current and the latest time steps (e.g., $j = i$ or $i − 1$).

The non-local operation is also different from a fully-connected (fc) layer. The equation above computes responses based on relationships between different locations, whereas fc uses learned weights. In other words, the relationship between $x_{j}$ and $x_{i}$ is not a function of the input data in fc, unlike in nonlocal layers. Furthermore, the formulation in the equation above supports inputs of variable sizes, and maintains the corresponding size in the output. On the contrary, an fc layer requires a fixed-size input/output and loses positional correspondence (e.g., that from $x_{i}$ to $y_{i}$ at the position $i$).

A non-local operation is a flexible building block and can be easily used together with convolutional/recurrent layers. It can be added into the earlier part of deep neural networks, unlike fc layers that are often used in the end. This allows us to build a richer hierarchy that combines both non-local and local information.

In terms of parameterisation, we usually parameterise $g$ as a linear embedding of the form $g\left(x_{j}\right) = W_{g}\mathbb{x}_{j}$ , where $W_{g}$ is a weight matrix to be learned. This is implemented as, e.g., 1×1 convolution in space or 1×1×1 convolution in spacetime. For $f$ we use an affinity function, a list of which can be found here.

Source: Non-local Neural Networks

#### Latest Papers

PAPER DATE
not-so-BigGAN: Generating High-Fidelity Images on a Small Compute Budget
Seungwook HanAkash SrivastavaCole HurwitzPrasanna SattigeriDavid D. Cox
2020-09-09
Neural Crossbreed: Neural Based Image Metamorphosis
Sanghun ParkKwanggyoon SeoJunyong Noh
2020-09-02
Brain Stroke Lesion Segmentation Using Consistent Perception Generative Adversarial Network
Shuqiang WangZhuo ChenWen YuBaiying Lei
2020-08-30
PNEN: Pyramid Non-Local Enhanced Networks
Feida ZhuChaowei FangKai-Kuang Ma
2020-08-22
DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting
| Chu-Tak LiWan-Chi SiuZhi-Song LiuLi-Wen WangDaniel Pak-Kong Lun
2020-08-17
Cascaded Non-local Neural Network for Point Cloud Semantic Segmentation
Mingmei ChengLe HuiJin XieJian YangHui Kong
2020-07-30
Instance Selection for GANs
Terrance DeVriesMichal DrozdzalGraham W. Taylor
2020-07-30
Interpolating GANs to Scaffold Autotelic Creativity
Ziv EpsteinOcéane BoulaisSkylar GordonMatt Groh
2020-07-21
Region-based Non-local Operation for Video Classification
2020-07-17
Differentiable Augmentation for Data-Efficient GAN Training
| Shengyu ZhaoZhijian LiuJi LinJun-Yan ZhuSong Han
2020-06-18
Disentangled Non-Local Neural Networks
Minghao YinZhuliang YaoYue CaoXiu LiZheng ZhangStephen LinHan Hu
2020-06-11
Training Generative Adversarial Networks with Limited Data
| Tero KarrasMiika AittalaJanne HellstenSamuli LaineJaakko LehtinenTimo Aila
2020-06-11
PNL: Efficient Long-Range Dependencies Extraction with Pyramid Non-Local Module for Action Recognition
Yuecong XuHaozhi CaoJianfei YangKezhi MaoJianxiong YinSimon See
2020-06-09
Learning disconnected manifolds: a no GANs land
Ugo TanielianThibaut IssenhuthElvis DohmatobJeremie Mary
2020-06-08
Big GANs Are Watching You: Towards Unsupervised Object Segmentation with Off-the-Shelf Generative Models
| Andrey VoynovStanislav MorozovArtem Babenko
2020-06-08
TESA: Tensor Element Self-Attention via Matricization
Francesca Babiloni Ioannis Marras Gregory Slabaugh Stefanos Zafeiriou
2020-06-01
A U-Net Based Discriminator for Generative Adversarial Networks
Edgar Schonfeld Bernt Schiele Anna Khoreva
2020-06-01
Network Fusion for Content Creation with Conditional INNs
Robin RombachPatrick EsserBjörn Ommer
2020-05-27
Learning Context-Based Non-local Entropy Modeling for Image Compression
Mu LiKai ZhangWangmeng ZuoRadu TimofteDavid Zhang
2020-05-10
Correlating Edge, Pose with Parsing
| Ziwei ZhangChi SuLiang ZhengXiaodong Xie
2020-05-04
LRTD: Long-Range Temporal Dependency based Active Learning for Surgical Workflow Recognition
| Xueying ShiYueming JinQi DouPheng-Ann Heng
2020-04-21
GANSpace: Discovering Interpretable GAN Controls
| Erik HärkönenAaron HertzmannJaakko LehtinenSylvain Paris
2020-04-06
Evolving Normalization-Activation Layers
| Hanxiao LiuAndrew BrockKaren SimonyanQuoc V. Le
2020-04-06
Feature Quantization Improves GAN Training
| Yang ZhaoChunyuan LiPing YuJianfeng GaoChangyou Chen
2020-04-05
BigGAN-based Bayesian reconstruction of natural images from human brain activity
Kai QiaoJian ChenLinyuan WangChi ZhangLi TongBin Yan
2020-03-13
PBRnet: Pyramidal Bounding Box Refinement to Improve Object Localization Accuracy
Li XiaoYufan LuoChunlong LuoLianhe ZhaoQuanshui FuGuoqing YangAnpeng HuangYi Zhao
2020-03-10
A U-Net Based Discriminator for Generative Adversarial Networks
| Edgar SchönfeldBernt SchieleAnna Khoreva
2020-02-28
Improved Consistency Regularization for GANs
Zhengli ZhaoSameer SinghHonglak LeeZizhao ZhangAugustus OdenaHan Zhang
2020-02-11
Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN
2020-01-31
Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures
Mohamed El Amine SeddikCosme LouartMohamed TamaazoustiRomain Couillet
2020-01-21
CNN-generated images are surprisingly easy to spot... for now
| Sheng-Yu WangOliver WangRichard ZhangAndrew OwensAlexei A. Efros
2019-12-23
LOGAN: Latent Optimisation for Generative Adversarial Networks
| Yan WuJeff DonahueDavid BalduzziKaren SimonyanTimothy Lillicrap
2019-12-02
Detecting GAN generated errors
Xiru ZhuFengdi CheTianzi YangTzuyang YuDavid MegerGregory Dudek
2019-12-02
One-Shot Object Detection with Co-Attention and Co-Excitation
| Ting-I HsiehYi-Chen LoHwann-Tzong ChenTyng-Luh Liu
2019-11-28
Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
| Ceyuan YangYujun ShenBolei Zhou
2019-11-21
Qingchao ZhangYunmei Chen
2019-11-17
Improving sample diversity of a pre-trained, class-conditional GAN by changing its class embeddings
| Qi LiLong MaiMichael A. AlcornAnh Nguyen
2019-10-10
Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations
| Peng Yi Zhongyuan Wang Kui Jiang Junjun Jiang Jiayi Ma
2019-10-01
Learnable Tree Filter for Structure-preserving Feature Transform
| Lin SongYanwei LiZeming LiGang YuHongbin SunJian SunNanning Zheng
2019-09-27
A Global-Local Emebdding Module for Fashion Landmark Detection
| Sumin LeeSungchan OhChanho JungChangick Kim
2019-08-28
Asymmetric Non-local Neural Networks for Semantic Segmentation
| Zhen ZhuMengde XuSong BaiTengteng HuangXiang Bai
2019-08-21
Adversarial Video Generation on Complex Datasets
Aidan ClarkJeff DonahueKaren Simonyan
2019-07-15
| Jeff DonahueKaren Simonyan
2019-07-04
6D-VNet: End-to-end 6DoF Vehicle Pose Estimation from Monocular RGB Images
| Di WuZhaoyong ZhuangCanqun XiangWenbin Zou and Xia Li
2019-06-15
GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
| Yue CaoJiarui XuStephen LinFangyun WeiHan Hu
2019-04-25
Lung Nodule Classification using Deep Local-Global Networks
| Mundher Al-ShabiBoon Leong LanWai Yee ChanKwan-Hoong NgMaxine Tan
2019-04-23
Improved Precision and Recall Metric for Assessing Generative Models
| Tuomas KynkäänniemiTero KarrasSamuli LaineJaakko LehtinenTimo Aila
2019-04-15
Libra R-CNN: Towards Balanced Learning for Object Detection
| Jiangmiao PangKai ChenJianping ShiHuajun FengWanli OuyangDahua Lin
2019-04-04
Spatial-Aware Non-Local Attention for Fashion Landmark Detection
Yixin LiShengqin TangYun YeJinwen Ma
2019-03-11
High-Fidelity Image Generation With Fewer Labels
| Mario LucicMichael TschannenMarvin RitterXiaohua ZhaiOlivier BachemSylvain Gelly
2019-03-06
CCNet: Criss-Cross Attention for Semantic Segmentation
| Zilong HuangXinggang WangYunchao WeiLichao HuangHumphrey ShiWenyu LiuThomas S. Huang
2018-11-28
Relational Long Short-Term Memory for Video Action Recognition
Zexi ChenBharathkumar RamachandraTianfu WuRanga Raju Vatsavai
2018-11-16
Metropolis-Hastings view on variational inference and adversarial training
Kirill NeklyudovEvgenii EgorovPavel ShvechikovDmitry Vetrov
2018-10-16
Large Scale GAN Training for High Fidelity Natural Image Synthesis
| Andrew BrockJeff DonahueKaren Simonyan
2018-09-28
Video-based Person Re-identification via 3D Convolutional Networks and Non-local Attention
Xingyu LiaoLingxiao HeZhouwang YangChi Zhang
2018-07-12
Non-local Neural Networks
| Xiaolong WangRoss GirshickAbhinav GuptaKaiming He
2017-11-21