Search Results for author:
Found 19 papers, 3 papers with code
Limitations of Knowledge Distillation for Zero-shot Transfer Learning
We demonstrate that in contradiction to the previous observation in the case of monolingual distillation, in multilingual settings, distillation during pretraining is more effective than distillation during fine-tuning for zero-shot transfer learning.
Controlled Data Generation via Insertion Operations for NLU
Use of synthetic data is rapidly emerging as a realistic alternative to manually annotating live traffic for industry-scale model building.
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection.
Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning
We present a novel approach which uses prompt-tuning to control the extraction rates of memorized content in LLMs.
Low-Resource Compositional Semantic Parsing with Concept Pretraining
In this work, we present an architecture to perform such domain adaptation automatically, with only a small amount of metadata about the new domain and without any new training data (zero-shot) or with very few examples (few-shot).
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model
In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks.
Ranked #14 on
Natural Language Inference
on CommitmentBank
Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9. 3B, their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
Cross-Lingual Natural Language Inference
intent-classification
+5
Unfreeze with Care: Space-Efficient Fine-Tuning of Semantic Parsing Models
We examine two such promising techniques, prefix tuning and bias-term tuning, specifically on semantic parsing.
RescoreBERT: Discriminative Speech Recognition Rescoring with BERT
Specifically, training a bidirectional model like BERT on a discriminative objective such as minimum WER (MWER) has not been explored.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Output Randomization: A Novel Defense for both White-box and Black-box Adversarial Models
Adversarial examples pose a threat to deep neural network models in a variety of scenarios, from settings where the adversary has complete knowledge of the model in a "white box" setting and to the opposite in a "black box" setting.
Using multiple ASR hypotheses to boost i18n NLU performance
For IC and NER multi-task experiments, when evaluating on the mismatched test set, we see improvements across all domains in German and in 17 out of 19 domains in Portuguese (improvements based on change in SeMER scores).
Abstractive Text Summarization
Automatic Speech Recognition
+10
Compressing Transformer-Based Semantic Parsing Models using Compositional Code Embeddings
The current state-of-the-art task-oriented semantic parsing models use BERT or RoBERTa as pretrained encoders; these models have huge memory footprints.
Don't Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding
For complex parsing tasks, the state-of-the-art method is based on autoregressive sequence to sequence models to generate the parse directly.
Optimal Mini-Batch Size Selection for Fast Gradient Descent
This paper presents a methodology for selecting the mini-batch size that minimizes Stochastic Gradient Descent (SGD) learning time for single and multiple learner problems.
Deep density ratio estimation for change point detection
In this work, we propose new objective functions to train deep neural network based density ratio estimators and apply it to a change point detection problem.
Thwarting finite difference adversarial attacks with output randomization
Adversarial examples pose a threat to deep neural network models in a variety of scenarios, from settings where the adversary has complete knowledge of the model and to the opposite "black box" setting.
Generation & Evaluation of Adversarial Examples for Malware Obfuscation
There has been an increased interest in the application of convolutional neural networks for image based malware classification, but the susceptibility of neural networks to adversarial examples allows malicious actors to evade classifiers.
Learning filter widths of spectral decompositions with wavelets
Our results show that the WD layer can improve neural network based time series classifiers both in accuracy and interpretability by learning directly from the input signal.
Focal onset seizure prediction using convolutional networks
Significance: We demonstrate that a robust set of features can be learned from scalp EEG that characterize the preictal state of focal seizures.
