Search Results for author:
Found 26 papers, 10 papers with code
Phonetic and Lexical Discovery of a Canine Language using HuBERT
This paper delves into the pioneering exploration of potential communication patterns within dog vocalizations and transcends traditional linguistic analysis barriers, which heavily relies on human priori knowledge on limited datasets to find sound units in dog vocalization.
SPAR: Personalized Content-Based Recommendation via Long Engagement Attention
Leveraging users' long engagement histories is essential for personalized content recommendations.
Effective Long-Context Scaling of Foundation Models
We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models
As a result, their performance suffers drastically on inputs longer than those encountered during training, substantially limiting their applications in real-world tasks involving long contexts such as encoding scientific articles, code repositories, or long dialogues.
Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefixes
Fine-tuning a large pre-trained language model for each downstream task causes computational burdens in the inference time due to several times of forward passes.
Defending Against Patch-based Backdoor Attacks on Self-Supervised Learning
It was shown that an adversary can poison a small part of the unlabeled data so that when a victim trains an SSL model on it, the final model will have a backdoor that the adversary can exploit.
Representation Deficiency in Masked Language Modeling
In this work, we offer a new perspective on the consequence of such a discrepancy: We demonstrate empirically and theoretically that MLM pretraining allocates some model dimensions exclusively for representing $\texttt{[MASK]}$ tokens, resulting in a representation deficiency for real tokens and limiting the pretrained model's expressiveness when it is adapted to downstream data without $\texttt{[MASK]}$ tokens.
Improved Adaptive Algorithm for Scalable Active Learning with Weak Labeler
We design a novel algorithmic template, Weak Labeler Active Cover (WL-AC), that is able to robustly leverage the lower quality weak labelers to reduce the query complexity while retaining the desired level of accuracy.
BayesFormer: Transformer with Uncertainty Estimation
Transformer has become ubiquitous due to its dominant performance in various NLP and image processing tasks.
Detection, Disambiguation, Re-ranking: Autoregressive Entity Linking as a Multi-Task Problem
Without the use of a knowledge base or candidate sets, our model sets a new state of the art in two benchmark datasets of entity linking: COMETA in the biomedical domain, and AIDA-CoNLL in the news domain.
IDPG: An Instance-Dependent Prompt Generation Method
Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage.
Reducing Target Group Bias in Hate Speech Detectors
The ubiquity of offensive and hateful content on online fora necessitates the need for automatic solutions that detect such content competently across target groups.
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models
Distilling state-of-the-art transformer models into lightweight student models is an effective way to reduce computation cost at inference time.
Luna: Linear Unified Nested Attention
Specifically, with the first attention function, Luna packs the input sequence into a sequence of fixed length.
Entailment as Few-Shot Learner
Large pre-trained language models (LMs) have demonstrated remarkable ability as few-shot learners.
Ranked #1 on
Topic Classification
on OS
On the Influence of Masking Policies in Intermediate Pre-training
Current NLP models are predominantly trained through a two-stage "pre-train then fine-tune" pipeline.
On Unifying Misinformation Detection
In this paper, we introduce UnifiedM2, a general-purpose misinformation model that jointly models multiple domains of misinformation with a single, unified setup.
Studying Strategically: Learning to Mask for Closed-book QA
Thus, our policy packs task-relevant knowledge into the parameters of a language model.
CLEAR: Contrastive Learning for Sentence Representation
Pre-trained language models have proven their unique powers in capturing implicit language features.
Ranked #5 on
Question Answering
on Quora Question Pairs
To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks
Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks.
To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks
Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks.
Linformer: Self-Attention with Linear Complexity
Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications.
Language Models as Fact Checkers?
Recent work has suggested that language models (LMs) store both common-sense and factual knowledge learned from pre-training data.
Blockwise Self-Attention for Long Document Understanding
We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies.
UCBoost: A Boosting Approach to Tame Complexity and Optimality for Stochastic Bandits
We show that UCBoost($D$) enjoys $O(1)$ complexity for each arm per round as well as regret guarantee that is $1/e$-close to that of the kl-UCB algorithm.
A New Alternating Direction Method for Linear Programming
It is well known that, for a linear program (LP) with constraint matrix $\mathbf{A}\in\mathbb{R}^{m\times n}$, the Alternating Direction Method of Multiplier converges globally and linearly at a rate $O((\|\mathbf{A}\|_F^2+mn)\log(1/\epsilon))$.
