Search Results for author:
Found 118 papers, 66 papers with code
Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
Large language models (LLMs) have exhibited impressive performance in language comprehension and various reasoning tasks.
Multi-Level Label Correction by Distilling Proximate Patterns for Semi-supervised Semantic Segmentation
Recent semi-supervised semantic segmentation approaches mainly resort to pseudo-labeling methods to exploit unlabeled data.
EDDA: A Encoder-Decoder Data Augmentation Framework for Zero-Shot Stance Detection
To address these issues, we propose an encoder-decoder data augmentation (EDDA) framework.
Large Generative Model Assisted 3D Semantic Communication
Semantic Communication (SC) is a novel paradigm for data transmission in 6G.
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs).
Towards Optimal Learning of Language Models
This work studies the general principles of improving the learning of language models (LMs), which aims at reducing the necessary training steps for achieving superior performance.
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs).
Cross-target Stance Detection by Exploiting Target Analytical Perspectives
In this paper, we propose a Multi-Perspective Prompt-Tuning (MPPT) model for CTSD that uses the analysis perspective as a bridge to transfer knowledge.
Large Language Model Enhanced Multi-Agent Systems for 6G Communications
The rapid development of the Large Language Model (LLM) presents huge opportunities for 6G communications, e. g., network optimization and management by allowing users to input task requirements to LLMs by nature language.
BioCLIP: A Vision Foundation Model for the Tree of Life
We then develop BioCLIP, a foundation model for the tree of life, leveraging the unique properties of biology captured by TreeOfLife-10M, namely the abundance and variety of images of plants, animals, and fungi, together with the availability of rich structured biological knowledge.
BitNet: Scaling 1-bit Transformers for Large Language Models
The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption.
Kosmos-G: Generating Images in Context with Multimodal Large Language Models
These limitations keep them far from the ultimate goal of "image as a foreign language in image generation."
Kosmos-2.5: A Multimodal Literate Model
We present Kosmos-2. 5, a multimodal literate model for machine reading of text-intensive images.
Large Language Model for Science: A Study on P vs. NP
In this work, we use large language models (LLMs) to augment and accelerate research on the P versus NP problem, one of the most important open problems in theoretical computer science and mathematics.
Large AI Model Empowered Multimodal Semantic Communications
To this end, we propose a Large AI Model-based Multimodal SC (LAM-MSC) framework, in which we first present the MLM-based Multimodal Alignment (MMA) that utilizes the MLM to enable the transformation between multimodal and unimodal data while preserving semantic consistency.
LAMBO: Large Language Model Empowered Edge Intelligence
Next-generation edge intelligence is anticipated to bring huge benefits to various applications, e. g., offloading systems.
Universal Defensive Underpainting Patch: Making Your Text Invisible to Optical Character Recognition
In this work, we propose a novel and effective defense mechanism termed the Universal Defensive Underpainting Patch (UDUP) that modifies the underpainting of text images instead of the characters.
Optical Character Recognition
Optical Character Recognition (OCR)
Retentive Network: A Successor to Transformer for Large Language Models
In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance.
Large AI Model-Based Semantic Communications
Semantic communication (SC) is an emerging intelligent paradigm, offering solutions for various future applications like metaverse, mixed-reality, and the Internet of everything.
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Scaling sequence length has become a critical demand in the era of large language models.
Fake the Real: Backdoor Attack on Deep Speech Classification via Voice Conversion
This work explores a backdoor attack that utilizes sample-specific triggers based on voice conversion.
Kosmos-2: Grounding Multimodal Large Language Models to the World
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e. g., bounding boxes) and grounding text to the visual world.
Ranked #11 on
Visual Question Answering
on ViP-Bench
Semi-Offline Reinforcement Learning for Optimized Text Generation
In reinforcement learning (RL), there are two major settings for interacting with the environment: online and offline.
MiniLLM: Knowledge Distillation of Large Language Models
In this work, we propose a KD approach that distills LLMs into smaller language models.
Augmenting Language Models with Long-Term Memory
Such a decoupled memory design can easily cache and update long-term past contexts for memory retrieval without suffering from memory staleness.
Pre-Training to Learn in Context
In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP community.
Generic-to-Specific Distillation of Masked Autoencoders
Lightweight ViT models limited by the model capacity, however, benefit little from those pre-training mechanisms.
Language Is Not All You Need: Aligning Perception with Language Models
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence.
Semi-Supervised Learning with Pseudo-Negative Labels for Image Classification
We also propose a mechanism to select a few pseudo-negative labels to feed into submodels.
Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is emerging.
Language Models as Inductive Reasoners
To this end, we propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts, and create a dataset termed DEER containing 1. 2k rule-fact pairs for the task, where rules and facts are written in natural language.
Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
We comprehensively compare the behaviors of in-context learning and explicit finetuning on real tasks to provide empirical evidence that supports our understanding.
GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator
Inspired by the idea of Generative Adversarial Networks (GANs), we propose a GAN-style model for encoder-decoder pre-training by introducing an auxiliary discriminator, unifying the ability of language understanding and generation in a single model.
Optimizing Prompts for Text-to-Image Generation
Instead of laborious human engineering, we propose prompt adaptation, a general framework that automatically adapts original user input to model-preferred prompts.
Structured Prompting: Scaling In-Context Learning to 1,000 Examples
Large language models have exhibited intriguing in-context learning capability, achieving promising zero- and few-shot performance without updating the parameters.
Extensible Prompts for Language Models on Zero-shot Language Style Customization
We propose eXtensible Prompt (X-Prompt) for prompting a large language model (LLM) beyond natural language (NL).
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across many tasks.
Joint Optimization of Deployment and Trajectory in UAV and IRS-Assisted IoT Data Collection System
To address this issue, we propose a joint optimization framework of deployment and trajectory (JOLT), where an adaptive whale optimization algorithm (AWOA) is applied to optimize the deployment of the UAV, and an elastic ring self-organizing map (ERSOM) is introduced to optimize the trajectory of the UAV.
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
In this paper, we elaborate upon recipes for building multilingual representation models that are not only competitive with existing state-of-the-art models but are also more parameter efficient, thereby promoting better adoption in resource-constrained scenarios and practical applications.
A Unified View of Masked Image Modeling
Masked image modeling has demonstrated great potential to eliminate the label-hungry problem of training large-scale vision Transformers, achieving impressive performance on various downstream tasks.
Non-Contrastive Learning Meets Language-Image Pre-Training
Contrastive language-image pre-training (CLIP) serves as a de-facto standard to align images and texts.
CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation
Specifically, the target sequence is first translated into the source language and then tagged by a source NER model.
Foundation Transformers
A big convergence of model architectures across language, vision, speech, and multimodal is emerging.
Semi-Supervised Semantic Segmentation with Cross Teacher Training
In experiments, the cross-teacher module significantly improves the performance of traditional student-teacher approaches, and our framework outperforms stateof-the-art methods on benchmark datasets.
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is emerging.
Ranked #1 on
Visual Reasoning
on NLVR2 Test
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
The large-size BEiT v2 obtains 87. 3% top-1 accuracy for ImageNet-1K (224 size) fine-tuning, and 56. 7% mIoU on ADE20K for semantic segmentation.
Ranked #27 on
Self-Supervised Image Classification
on ImageNet
Detecting and Recovering Adversarial Examples from Extracting Non-robust and Highly Predictive Adversarial Perturbations
Experimental results show that the proposed method can not only detect the adversarial examples with high accuracy, but also detect the specific category of the AEs.
Language Models are General-Purpose Interfaces
Experimental results across various language-only and vision-language benchmarks show that our model outperforms or is competitive with specialized models on finetuning, zero-shot generalization, and few-shot learning.
Ranked #2 on
Image Captioning
on nocaps val
VL-BEiT: Generative Vision-Language Pretraining
Our minimalist solution conducts masked prediction on both monomodal and multimodal data with a shared Transformer.
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
As more and more pre-trained language models adopt on-cloud deployment, the privacy issues grow quickly, mainly for the exposure of plain-text user data (e. g., search history, medical record, bank account).
Prototypical Calibration for Few-shot Learning of Language Models
In-context learning of GPT-like models has been recognized as fragile across different hand-crafted templates, and demonstration permutations.
Visually-Augmented Language Modeling
With the visually-augmented context, VaLM uses a visual knowledge fusion layer to enable multimodal grounded language modeling by attending to both text context and visual knowledge in images.
On the Representation Collapse of Sparse Mixture of Experts
We also present a comprehensive analysis on the representation and routing behaviors of our models.
StableMoE: Stable Routing Strategy for Mixture of Experts
We point out that existing learning-to-route MoE methods suffer from the routing fluctuation issue, i. e., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference.
CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment
We first evaluate CLIP's zero-shot performance on a typical visual question answering task and demonstrate a zero-shot cross-modality transfer capability of CLIP on the visual entailment task.
DeepNet: Scaling Transformers to 1,000 Layers
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers.
Controllable Natural Language Generation with Contrastive Prefixes
We propose a novel supervised method and also an unsupervised method to train the prefixes for single-aspect control while the combination of these two methods can achieve multi-aspect control.
A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models
With the increasing of model capacity brought by pre-trained language models, there emerges boosting needs for more knowledgeable natural language processing (NLP) models with advanced functionalities including providing and making flexible use of encyclopedic and commonsense knowledge.
AdaPrompt: Adaptive Model Training for Prompt-based NLP
However, for prompt learning, there are still two salient gaps between NLP tasks and pretraining.
Corrupted Image Modeling for Self-Supervised Visual Pre-Training
Given this corrupted image, an enhancer network learns to either recover all the original image pixels, or predict whether each visual token is replaced by a generator sample or not.
Kformer: Knowledge Injection in Transformer Feed-Forward Layers
In this work, we propose a simple model, Kformer, which takes advantage of the knowledge stored in PTMs and external knowledge via knowledge injection in Transformer FFN layers.
Swin Transformer V2: Scaling Up Capacity and Resolution
Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images.
Ranked #4 on
Image Classification
on ImageNet V2
(using extra training data)
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network.
Ranked #2 on
Image Retrieval
on PhotoChat
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation.
s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Pretrained bidirectional Transformers, such as BERT, have achieved significant improvements in a wide variety of language understanding tasks, while it is not straightforward to directly apply them for natural language generation.
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on XTREME
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG).
Learning to Sample Replacements for ELECTRA Pre-Training
Moreover, we propose to use a focal loss for the generator in order to relieve oversampling of correct tokens as replacements.
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
In this paper, we present a general approach to developing small, fast and effective pre-trained models for specific domains.
Consistency Regularization for Cross-Lingual Fine-Tuning
Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others.
BEiT: BERT Pre-Training of Image Transformers
We first "tokenize" the original image into visual tokens.
Ranked #10 on
Document Layout Analysis
on PubLayNet val
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
The cross-lingual language models are typically pretrained with masked language modeling on multilingual text or parallel sentences.
A Semi-supervised Multi-task Learning Approach to Classify Customer Contact Intents
We improve the performance significantly by evolving the model from multiclass classification to semi-supervised multi-task learning by leveraging the negative cases, domain- and task-adaptively pretrained ALBERT on customer contact texts, and a number of un-curated data with no labels.
Memory-Efficient Differentiable Transformer Architecture Search
To this end, we propose a multi-split reversible network and combine it with DARTS.
Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
In this paper, we focus on a zero-shot cross-lingual transfer task in NMT.
MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Multilingual T5 (mT5) pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks.
Knowledge Neurons in Pretrained Transformers
In this paper, we present preliminary studies on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons.
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers.
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Multilingual machine translation enables a single model to translate between different languages.
Investigating Learning Dynamics of BERT Fine-Tuning
The recently introduced pre-trained language model BERT advances the state-of-the-art on many NLP tasks through the fine-tuning approach, but few studies investigate how the fine-tuning process improves the model performance on downstream tasks.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.
Ranked #16 on
Zero-Shot Cross-Lingual Transfer
on XTREME
Distributed Resource Scheduling for Large-Scale MEC Systems: A Multi-Agent Ensemble Deep Reinforcement Learning with Imitation Acceleration
We consider the optimization of distributed resource scheduling to minimize the sum of task latency and energy consumption for all the Internet of things devices (IoTDs) in a large-scale mobile edge computing (MEC) system.
Harvesting and Refining Question-Answer Pairs for Unsupervised QA
Our approach outperforms previous unsupervised approaches by a large margin and is competitive with early supervised models.
Ranked #189 on
Question Answering
on SQuAD1.1
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
The great success of Transformer-based models benefits from the powerful multi-head self-attention mechanism, which learns token dependencies and encodes contextual information from the input.
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks.
Ranked #1 on
Image Retrieval
on MS COCO
(Recall@10 metric)
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Ranked #4 on
Question Generation
on SQuAD1.1
(using extra training data)
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher).
Ranked #8 on
Zero-shot Text Search
on BEIR
AI Driven Heterogeneous MEC System with UAV Assistance for Dynamic Environment -- Challenges and Solutions
By taking full advantage of Computing, Communication and Caching (3C) resources at the network edge, Mobile Edge Computing (MEC) is envisioned as one of the key enablers for the next generation networks.
Stacked Auto Encoder Based Deep Reinforcement Learning for Online Resource Scheduling in Large-Scale MEC Networks
An online resource scheduling framework is proposed for minimizing the sum of weighted task latency for all the Internet of things (IoT) users, by optimizing offloading decision, transmission power and resource allocation in the large-scale mobile edge computing (MEC) system.
Can Monolingual Pretrained Models Help Cross-Lingual Classification?
Multilingual pretrained language models (such as multilingual BERT) have achieved impressive results for cross-lingual transfer.
Transforming Wikipedia into Augmented Data for Query-Focused Summarization
The limited size of existing query-focused summarization datasets renders training data-driven summarization models challenging.
Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension
Most machine reading comprehension (MRC) models separately handle encoding and matching with different network architectures.
Learning A Unified Named Entity Tagger From Multiple Partially Annotated Corpora For Efficient Adaptation
Named entity recognition (NER) identifies typed entity mentions in raw text.
Ranked #14 on
Named Entity Recognition (NER)
on NCBI-disease
Cross-Lingual Natural Language Generation via Pre-Training
In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages.
Visualizing and Understanding the Effectiveness of BERT
Language model pre-training, such as BERT, has achieved remarkable results in many NLP tasks.
Learning to Ask Unanswerable Questions for Machine Reading Comprehension
We also present a way to construct training data for our question generation models by leveraging the existing reading comprehension dataset.
Data-to-text Generation with Entity Modeling
Recent approaches to data-to-text generation have shown great promise thanks to the use of large-scale datasets and the application of neural network architectures which are trained end-to-end.
Ranked #3 on
Data-to-Text Generation
on MLB Dataset
Unified Language Model Pre-training for Natural Language Understanding and Generation
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.
Ranked #2 on
Generative Question Answering
on CoQA
(using extra training data)
Multi-Task Learning for Semantic Parsing with Cross-Domain Sketch
Semantic parsing which maps a natural language sentence into a formal machine-readable representation of its meaning, is highly constrained by the limited annotated training data.
An Evaluation of Transfer Learning for Classifying Sales Engagement Emails at Large Scale
Our results show that fine-tuning of the BERT model outperforms with as few as 300 labeled samples, but underperforms with fewer than 300 labeled samples, relative to all the feature-based approaches using different embeddings.
Data-to-Text Generation with Content Selection and Planning
Recent advances in data-to-text generation have led to the use of large-scale datasets and neural network models which are trained end-to-end, without explicitly modeling what to say and in what order.
Coarse-to-Fine Decoding for Neural Semantic Parsing
Semantic parsing aims at mapping natural language utterances into structured meaning representations.
Ranked #1 on
Semantic Parsing
on Geo
Confidence Modeling for Neural Semantic Parsing
In this work we focus on confidence modeling for neural semantic parsers which are built upon sequence-to-sequence models.
Learning to Paraphrase for Question Answering
Question answering (QA) systems are sensitive to the many different ways natural language expresses the same information need.
Learning to Generate Product Reviews from Attributes
This paper presents an attention-enhanced attribute-to-sequence model to generate product reviews for given attribute information, such as user, product, and rating.
Proactive Resource Management for LTE in Unlicensed Spectrum: A Deep Learning Perspective
LTE in unlicensed spectrum using licensed assisted access LTE (LTE-LAA) is a promising approach to overcome the wireless spectrum scarcity.
Unsupervised Word and Dependency Path Embeddings for Aspect Term Extraction
In this paper, we develop a novel approach to aspect term extraction based on unsupervised learning of distributed representations of words and dependency paths.
Long Short-Term Memory-Networks for Machine Reading
In this paper we address the question of how to render sequence-level networks better at handling structured input.
Ranked #55 on
Natural Language Inference
on SNLI
Language to Logical Form with Neural Attention
Semantic parsing aims at mapping natural language to machine interpretable meaning representations.
A Statistical Parsing Framework for Sentiment Classification
Unlike previous works that employ syntactic parsing results for sentiment analysis, we develop a statistical parser to directly analyze the sentiment structure of a sentence.
