Search Results for author:
Found 80 papers, 51 papers with code
Towards Making the Most of Cross-Lingual Transfer for Zero-Shot Neural Machine Translation
When applied to zero-shot cross-lingual abstractive summarization, it produces an average performance gain of 12. 3 ROUGE-L over mBART-ft. We conduct detailed analyses to understand the key ingredients of SixT+, including multilinguality of the auxiliary parallel data, positional disentangled encoder, and the cross-lingual transferability of its encoder.
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+5
Improving Multilingual Neural Machine Translation with Auxiliary Source Languages
Multilingual neural machine translation models typically handle one source language at a time.
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs).
When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
This technical report presents LongViT, a vision Transformer that can process gigapixel images in an end-to-end manner.
Auto-ICL: In-Context Learning without Human Supervision
In the era of Large Language Models (LLMs), human-computer interaction has evolved towards natural language, offering unprecedented flexibility.
BitNet: Scaling 1-bit Transformers for Large Language Models
The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption.
Kosmos-2.5: A Multimodal Literate Model
We present Kosmos-2. 5, a multimodal literate model for machine reading of text-intensive images.
Retentive Network: A Successor to Transformer for Large Language Models
In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance.
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Scaling sequence length has become a critical demand in the era of large language models.
Kosmos-2: Grounding Multimodal Large Language Models to the World
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e. g., bounding boxes) and grounding text to the visual world.
Ranked #11 on
Visual Question Answering
on ViP-Bench
Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus
Several recent papers claim human parity at sentence-level Machine Translation (MT), especially in high-resource languages.
On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation
We conduct experiments on a multilingual machine translation benchmark in 11 languages.
Are More Layers Beneficial to Graph Transformers?
Despite that going deep has proven successful in many neural architectures, the existing graph transformers are relatively shallow.
Language Is Not All You Need: Aligning Perception with Language Models
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence.
HanoiT: Enhancing Context-aware Translation via Selective Context
Context-aware neural machine translation aims to use the document-level context to improve translation quality.
Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
We comprehensively compare the behaviors of in-context learning and explicit finetuning on real tasks to provide empirical evidence that supports our understanding.
GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator
Inspired by the idea of Generative Adversarial Networks (GANs), we propose a GAN-style model for encoder-decoder pre-training by introducing an auxiliary discriminator, unifying the ability of language understanding and generation in a single model.
Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Models
Despite the success of multilingual sequence-to-sequence pre-training, most existing approaches rely on document-level monolingual corpora in many different languages, sentence-level bilingual corpora,\footnote{In this paper, we use `bilingual corpora' to denote parallel corpora with `bilingual translation pairs' in many different language pairs, each consisting of two sentences/documents with the same meaning written in different languages.
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+4
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across many tasks.
A Bilingual Parallel Corpus with Discourse Annotations
The BWB corpus consists of Chinese novels translated by experts into English, and the annotated test set is designed to probe the ability of machine translation systems to model various discourse phenomena.
CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation
Specifically, the target sequence is first translated into the source language and then tagged by a source NER model.
Foundation Transformers
A big convergence of model architectures across language, vision, speech, and multimodal is emerging.
Revamping Multilingual Agreement Bidirectionally via Switched Back-translation for Multilingual Neural Machine Translation
Despite the fact that multilingual agreement (MA) has shown its importance for multilingual neural machine translation (MNMT), current methodologies in the field have two shortages: (i) require parallel data between multiple language pairs, which is not always realistic and (ii) optimize the agreement in an ambiguous direction, which hampers the translation performance.
GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation
Transformer structure, stacked by a sequence of encoder and decoder network layers, achieves significant development in neural machine translation.
UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translation
Most translation tasks among languages belong to the zero-resource translation problem where parallel corpora are unavailable.
HLT-MT: High-resource Language-specific Training for Multilingual Neural Machine Translation
Nonetheless, multilingual training is plagued by language interference degeneration in shared parameters because of the negative interference among different translation directions, especially on high-resource languages.
Language Models are General-Purpose Interfaces
Experimental results across various language-only and vision-language benchmarks show that our model outperforms or is competitive with specialized models on finetuning, zero-shot generalization, and few-shot learning.
Ranked #2 on
Image Captioning
on nocaps val
On the Representation Collapse of Sparse Mixture of Experts
We also present a comprehensive analysis on the representation and routing behaviors of our models.
StableMoE: Stable Routing Strategy for Mixture of Experts
We point out that existing learning-to-route MoE methods suffer from the routing fluctuation issue, i. e., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference.
DeepNet: Scaling Transformers to 1,000 Layers
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers.
Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt
To collocate with the unified prompt, we propose a new initialization method for the target label word to further improve the model's transferability across languages.
A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model
Synthetic data construction of Grammatical Error Correction (GEC) for non-English languages relies heavily on human-designed and language-specific rules, which produce limited error-corrected patterns.
PAEG: Phrase-level Adversarial Example Generation for Neural Machine Translation
While end-to-end neural machine translation (NMT) has achieved impressive progress, noisy input usually leads models to become fragile and unstable.
SMDT: Selective Memory-Augmented Neural Document Translation
Existing document-level neural machine translation (NMT) models have sufficiently explored different context settings to provide guidance for target generation.
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation.
Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
When applied to zero-shot cross-lingual abstractive summarization, it produces an average performance gain of 12. 3 ROUGE-L over mBART-ft. We conduct detailed analyses to understand the key ingredients of SixT+, including multilinguality of the auxiliary parallel data, positional disentangled encoder, and the cross-lingual transferability of its encoder.
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+5
Multilingual Agreement for Multilingual Neural Machine Translation
Although multilingual neural machine translation (MNMT) enables multiple language translations, the training process is based on independent multilingual objectives.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on XTREME
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG).
Smart-Start Decoding for Neural Machine Translation
Most current neural machine translation models adopt a monotonic decoding order of either left-to-right or right-to-left.
How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?
While non-autoregressive (NAR) models are showing great promise for machine translation, their use is limited by their dependence on knowledge distillation from autoregressive models.
MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Multilingual T5 (mT5) pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks.
Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
In this paper, we focus on a zero-shot cross-lingual transfer task in NMT.
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation
Standard automatic metrics, e. g. BLEU, are not reliable for document-level MT evaluation.
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Multilingual machine translation enables a single model to translate between different languages.
A Simple and Effective Unified Encoder for Document-Level Machine Translation
Most of the existing models for document-level machine translation adopt dual-encoder structures.
Improving Neural Machine Translation with Soft Template Prediction
Although neural machine translation (NMT) has achieved significant progress in recent years, most previous NMT models only depend on the source text to generate translation.
Multimodal Matching Transformer for Live Commenting
In this work, we aim to improve the relevance between live comments and videos by modeling the cross-modal interactions among different modalities.
Group, Extract and Aggregate: Summarizing a Large Amount of Finance News for Forex Movement Prediction
In this work, we propose a BERT-based Hierarchical Aggregation Model to summarize a large amount of finance news to predict forex movement.
Recursive Graphical Neural Networks for Text Classification
Graph is a natural structure to describe the complicated relation between tokens.
Key Fact as Pivot: A Two-Stage Model for Low Resource Table-to-Text Generation
We propose a novel model to separate the generation into two stages: key fact prediction and surface realization.
A Deep Reinforced Sequence-to-Set Model for Multi-Label Classification
In this way, we can reduce the dependence of the model on the label order, as well as capture high-order correlations between labels.
Phrase-level Self-Attention Networks for Universal Sentence Encoding
As a result, the memory consumption can be reduced because the self-attention is performed at the phrase level instead of the sentence level.
Unsupervised Machine Commenting with Neural Variational Topic Model
To fully exploit the unpaired data, we completely remove the need for parallel data and propose a novel unsupervised approach to train an automatic article commenting model, relying on nothing but unpaired articles and comments.
LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
We introduce the task of automatic live commenting.
A Deep Reinforced Sequence-to-Set Model for Multi-Label Text Classification
However, the Seq2Seq model is not suitable for the MLTC task in essence.
Semantic-Unit-Based Dilated Convolution for Multi-Label Text Classification
We propose a novel model for multi-label text classification, which is based on sequence-to-sequence learning.
Identifying High-Quality Chinese News Comments Based on Multi-Target Text Matching Model
In this work, we introduce a novel task: high-quality comment identification (HQCI), which aims to automatically assess the quality of online comments.
A Neural Question Answering Model Based on Semi-Structured Tables
Then the system measures the relevance between each question and candidate table cells, and choose the most related cell as the source of answer.
SGM: Sequence Generation Model for Multi-label Classification
Further analysis of experimental results demonstrates that the proposed methods not only capture the correlations between labels, but also select the most informative words automatically when predicting different labels.
Deconvolution-Based Global Decoding for Neural Machine Translation
A great proportion of sequence-to-sequence (Seq2Seq) models for Neural Machine Translation (NMT) adopt Recurrent Neural Network (RNN) to generate translation word by word following a sequential order.
Ranked #9 on
Machine Translation
on IWSLT2015 English-Vietnamese
Autoencoder as Assistant Supervisor: Improving Text Representation for Chinese Social Media Text Summarization
In this work, we supervise the learning of the representation of the source content with that of the summary.
Bag-of-Words as Target for Neural Machine Translation
However, most of the existing neural machine translation models only use one of the correct translations as the targets, and the other correct sentences are punished as the incorrect sentences in the training stage.
Automatic Academic Paper Rating Based on Modularized Hierarchical Convolutional Neural Network
As more and more academic papers are being submitted to conferences and journals, evaluating all these papers by professionals is time-consuming and can cause inequality due to the personal factors of the reviewers.
Global Encoding for Abstractive Summarization
To tackle the problem, we propose a global encoding framework, which controls the information flow from the encoder to the decoder based on the global information of the source context.
Ranked #29 on
Text Summarization
on GigaWord
A Hierarchical End-to-End Model for Jointly Improving Text Summarization and Sentiment Classification
Text summarization and sentiment classification both aim to capture the main ideas of the text but at different levels.
Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generation
The existing sequence-to-sequence model tends to memorize the words and the patterns in the training dataset instead of learning the meaning of the words.
Decoding-History-Based Adaptive Control of Attention for Neural Machine Translation
ACA learns to control the attention by keeping track of the decoding history and the current information with a memory vector, so that the model can take the translated contents and the current information into consideration.
Complex Structure Leads to Overfitting: A Structure Regularization Decoding Method for Natural Language Processing
The decoding of the complex structure model is regularized by the additionally trained simple structure model.
Does Higher Order LSTM Have Better Accuracy for Segmenting and Labeling Sequence Data?
In our work, we first design a new model called "high order LSTM" to predict multiple tags for the current token which contains not only the current tag but also the previous several tags.
Training Simplification and Model Simplification for Deep Learning: A Minimal Effort Back Propagation Method
Based on the sparsified gradients, we further simplify the model by eliminating the rows or columns that are seldom updated, which will reduce the computational cost both in the training and decoding, and potentially accelerate decoding in real-world applications.
Label Embedding Network: Learning Label Representation for Soft Training of Deep Networks
We propose a method, called Label Embedding Network, which can learn label representation (label embedding) during the training process of deep networks.
A Semantic Relevance Based Neural Network for Text Summarization and Text Simplification
In this work, our goal is to improve semantic relevance between source texts and simplified texts for text summarization and text simplification.
meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
In back propagation, only a small subset of the full gradient is computed to update the model parameters.
Improving Semantic Relevance for Sequence-to-Sequence Learning of Chinese Social Media Text Summarization
In this work, our goal is to improve semantic relevance between source texts and summaries for Chinese social media summarization.
A Generic Online Parallel Learning Framework for Large Margin Models
To speed up the training process, many existing systems use parallel technology for online learning algorithms.
Lock-Free Parallel Perceptron for Graph-based Dependency Parsing
To deal with this problem, we propose a parallel algorithm called parallel perceptron.
A New Recurrent Neural CRF for Learning Non-linear Edge Features
Conditional Random Field (CRF) and recurrent neural models have achieved success in structured prediction.
Towards Easier and Faster Sequence Labeling for Natural Language Processing: A Search-based Probabilistic Online Learning Framework (SAPO)
We show that this method with fast training and theoretical guarantee of convergence, which is easy to implement, can support search-based optimization and obtain top accuracy.
