DropConnect

Introduced by Wan et al. in Regularization of Neural Networks using DropConnect

DropConnect generalizes Dropout by randomly dropping the weights rather than the activations with probability $1-p$. DropConnect is similar to Dropout as it introduces dynamic sparsity within the model, but differs in that the sparsity is on the weights $W$, rather than the output vectors of a layer. In other words, the fully connected layer with DropConnect becomes a sparsely connected layer in which the connections are chosen at random during the training stage. Note that this is not equivalent to setting $W$ to be a fixed sparse matrix during training.

For a DropConnect layer, the output is given as:

$$ r = a \left(\left(M * W\right){v}\right)$$

Here $r$ is the output of a layer, $v$ is the input to a layer, $W$ are weight parameters, and $M$ is a binary matrix encoding the connection information where $M_{ij} \sim \text{Bernoulli}\left(p\right)$. Each element of the mask $M$ is drawn independently for each example during training, essentially instantiating a different connectivity for each example seen. Additionally, the biases are also masked out during training.

Source: Regularization of Neural Networks using DropConnect

Latest Papers

PAPER DATE
Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language
| Dan John Velasco
2020-10-13
[email protected]: Pre-training ULMFiT on Synthetically Generated Code-Mixed Data for Hate Speech Detection
Gaurav Arora
2020-10-05
Fine-tuning Pre-trained Contextual Embeddings for Citation Content Analysis in Scholarly Publication
Haihua ChenHuyen Nguyen
2020-09-12
HinglishNLP: Fine-tuned Language Models for Hinglish Sentiment Detection
Meghana BhangeNirant Kasliwal
2020-08-22
Composer Style Classification of Piano Sheet Music Images Using Language Model Pretraining
TJ TsaiKevin Ji
2020-07-29
RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr
Xingjian LiHaoyi XiongHaozhe AnChengzhong XuDejing Dou
2020-07-07
Probing for Referential Information in Language Models
Ionut-Teodor SorodocKristina GulordavaGemma Boleda
2020-07-01
WCP: Worst-Case Perturbations for Semi-Supervised Deep Learning
| Liheng Zhang Guo-Jun Qi
2020-06-01
Locally Constant Networks
| Guang-He LeeTommi S. Jaakkola
2020-05-01
Text Categorization for Conflict Event Annotation
Fredrik OlssonMagnus SahlgrenFehmi ben AbdesslemAriel EkgrenKristine Eck
2020-05-01
Offensive language detection in Arabic using ULMFiT
Mohamed AbdellatifAhmed Elgammal
2020-05-01
Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings
Abdul MoeedYang AnGerhard HagererGeorg Groh
2020-05-01
Dynamic Bayesian Neural Networks
Lorenzo RimellaNick Whiteley
2020-04-15
Inferring the source of official texts: can SVM beat ULMFiT?
| Pedro Henrique Luz de AraujoTeófilo Emidio de CamposMarcelo Magalhães Silva de Sousa
2020-03-02
MaxUp: A Simple Way to Improve Generalization of Neural Network Training
Chengyue GongTongzheng RenMao YeQiang Liu
2020-02-20
Localized Flood DetectionWith Minimal Labeled Social Media Data Using Transfer Learning
Neha SinghNirmalya RoyAryya Gangopadhyay
2020-02-10
Locally Constant Networks
Anonymous
2020-01-01
DeFINE: Deep Factorized Input Word Embeddings for Neural Sequence Modeling
Anonymous
2020-01-01
iSparse: Output Informed Sparsification of Neural Networks
Anonymous
2020-01-01
Natural language processing of MIMIC-III clinical notes for identifying diagnosis and procedures with neural networks
Siddhartha NuthakkiSunil NeelaJudy W. GichoyaSaptarshi Purkayastha
2019-12-28
A Comparative Study of Pretrained Language Models on Thai Social Text Categorization
Thanapapas HorsuwanKasidis KanwatcharaPeerapon VateekulBoonserm Kijsirikul
2019-12-03
Continuous Dropout
Xu ShenXinmei TianTongliang LiuFang XuDacheng Tao
2019-11-28
DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
Sachin MehtaRik Koncel-KedziorskiMohammad RastegariHannaneh Hajishirzi
2019-11-27
Parameters Estimation for the Cosmic Microwave Background with Bayesian Neural Networks
Hector J. HortuaRiccardo VolpiDimitri MarinelliLuigi Malagò
2019-11-19
A Subword Level Language Model for Bangla Language
Aisha KhatunAnisur RahmanHemayet Ahmed ChowdhuryMd. Saiful IslamAyesha Tasnim
2019-11-15
On the Regularization Properties of Structured Dropout
Ambar PalConnor LaneRené VidalBenjamin D. Haeffele
2019-10-30
Evolution of transfer learning in natural language processing
Aditya MaltePratik Ratadiya
2019-10-16
The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews
Benjamin van der BurghSuzan Verberne
2019-10-02
Oblique Decision Trees from Derivatives of ReLU Networks
| Guang-He LeeTommi S. Jaakkola
2019-09-30
Analyzing Customer Feedback for Product Fit Prediction
Stephan Baier
2019-08-28
Low-Shot Classification: A Comparison of Classical and Deep Transfer Machine Learning Approaches
Peter UsherwoodSteven Smit
2019-07-17
Spatially-Coupled Neural Network Architectures
Arman HasanzadehNagaraj T. JanakiramanVamsi K. AmalladinneKrishna R. Narayanan
2019-07-03
Evaluating Language Model Finetuning Techniques for Low-resource Languages
| Jan Christian Blaise CruzCharibeth Cheng
2019-06-30
Exploiting Unsupervised Pre-training and Automated Feature Engineering for Low-resource Hate Speech Detection in Polish
Renard KorzeniowskiRafał RolczyńskiPrzemysław SadownikTomasz KorbakMarcin Możejko
2019-06-17
Speak up, Fight Back! Detection of Social Media Disclosures of Sexual Harassment
| Arijit Ghosh ChowdhuryRamit SawhneyPuneet MathurDebanjan MahataRajiv Ratn Shah
2019-06-01
Figure Eight at SemEval-2019 Task 3: Ensemble of Transfer Learning Methods for Contextual Emotion Detection
Joan Xiao
2019-06-01
An Empirical Evaluation of Text Representation Schemes on Multilingual Social Web to Filter the Textual Aggression
Sandip ModhaPrasenjit Majumder
2019-04-16
Low Resource Text Classification with ULMFit and Backtranslation
Sam Shleifer
2019-03-21
Pay Less Attention with Lightweight and Dynamic Convolutions
| Felix WuAngela FanAlexei BaevskiYann N. DauphinMichael Auli
2019-01-29
DropFilter: A Novel Regularization Method for Learning Convolutional Neural Networks
Hengyue PanHui JiangXin NiuYong Dou
2018-11-16
Radius-margin bounds for deep neural networks
Mayank SharmaJayadevaSumit Soman
2018-11-03
Language Informed Modeling of Code-Switched Text
ChKhyathi uThomas ManziniSumeet SinghAlan W. Black
2018-07-01
Deep Network Regularization via Bayesian Inference of Synaptic Connectivity
Harris PartaouridesSotirios P. Chatzis
2018-03-04
Universal Language Model Fine-tuning for Text Classification
| Jeremy HowardSebastian Ruder
2018-01-18
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
| Zhilin YangZihang DaiRuslan SalakhutdinovWilliam W. Cohen
2017-11-10
Regularizing and Optimizing LSTM Language Models
| Stephen MerityNitish Shirish KeskarRichard Socher
2017-08-07
Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognition
Weixin YangTerry LyonsHao NiCordelia SchmidLianwen Jin
2017-07-13
Robustly representing uncertainty in deep neural networks through sampling
Patrick McClureNikolaus Kriegeskorte
2016-11-05
Regularized Dynamic Boltzmann Machine with Delay Pruning for Unsupervised Learning of Temporal Sequences
Sakyasingha DasguptaTakayuki YoshizumiTakayuki Osogami
2016-09-22
SoftTarget Regularization: An Effective Technique to Reduce Over-Fitting in Neural Networks
Armen Aghajanyan
2016-09-21
GraphConnect: A Regularization Framework for Neural Networks
Jiaji HuangQiang QiuRobert CalderbankGuillermo Sapiro
2015-12-21
Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines
Emre O. NeftciBruno U. PedroniSiddharth JoshiMaruan Al-ShedivatGert Cauwenberghs
2015-11-14
Prediction of breast cancer recurrence using Classification Restricted Boltzmann Machine with Dropping
Jakub M. Tomczak
2013-08-28
Regularization of Neural Networks using DropConnect
Li WanMatthew ZeilerSixin ZhangYann LeCunRob Fergus
2013-06-13

Components

COMPONENT TYPE
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign

Categories