Search Results for author:
Found 34 papers, 22 papers with code
Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs).
When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
This technical report presents LongViT, a vision Transformer that can process gigapixel images in an end-to-end manner.
Kosmos-2.5: A Multimodal Literate Model
We present Kosmos-2. 5, a multimodal literate model for machine reading of text-intensive images.
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Scaling sequence length has become a critical demand in the era of large language models.
Kosmos-2: Grounding Multimodal Large Language Models to the World
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e. g., bounding boxes) and grounding text to the visual world.
Ranked #11 on
Visual Question Answering
on ViP-Bench
Language Is Not All You Need: Aligning Perception with Language Models
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence.
Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is emerging.
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across many tasks.
Foundation Transformers
A big convergence of model architectures across language, vision, speech, and multimodal is emerging.
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is emerging.
Ranked #1 on
Visual Reasoning
on NLVR2 Test
Language Models are General-Purpose Interfaces
Experimental results across various language-only and vision-language benchmarks show that our model outperforms or is competitive with specialized models on finetuning, zero-shot generalization, and few-shot learning.
Ranked #2 on
Image Captioning
on nocaps val
VL-BEiT: Generative Vision-Language Pretraining
Our minimalist solution conducts masked prediction on both monomodal and multimodal data with a shared Transformer.
TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation
Since event camera is a novel sensor, its potential has not been fulfilled due to the lack of processing algorithms.
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Our framework AutoDistil addresses above challenges with the following steps: (a) Incorporates inductive bias and heuristics to partition Transformer search space into K compact sub-spaces (K=3 for typical student sizes of base, small and tiny); (b) Trains one SuperLM for each sub-space using task-agnostic objective (e. g., self-attention distillation) with weight-sharing of students; (c) Lightweight search for the optimal student without re-training.
Distilled Dual-Encoder Model for Vision-Language Understanding
We propose a cross-modal attention distillation framework to train a dual-encoder model for vision-language understanding tasks, such as visual reasoning and visual question answering.
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network.
Ranked #2 on
Image Retrieval
on PhotoChat
s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Pretrained bidirectional Transformers, such as BERT, have achieved significant improvements in a wide variety of language understanding tasks, while it is not straightforward to directly apply them for natural language generation.
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
In this paper, we present a general approach to developing small, fast and effective pre-trained models for specific domains.
Consistency Regularization for Cross-Lingual Fine-Tuning
Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others.
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.
Ranked #16 on
Zero-Shot Cross-Lingual Transfer
on XTREME
Harvesting and Refining Question-Answer Pairs for Unsupervised QA
Our approach outperforms previous unsupervised approaches by a large margin and is competitive with early supervised models.
Ranked #189 on
Question Answering
on SQuAD1.1
Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences
Neuromorphic data, recording frameless spike events, have attracted considerable attention for the spatiotemporal information components and the event-driven processing fashion.
Ranked #11 on
Gesture Recognition
on DVS128 Gesture
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).
Ranked #4 on
Question Generation
on SQuAD1.1
(using extra training data)
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher).
Ranked #8 on
Zero-shot Text Search
on BEIR
Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension
Most machine reading comprehension (MRC) models separately handle encoding and matching with different network architectures.
Cross-Lingual Natural Language Generation via Pre-Training
In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages.
Learning to Ask Unanswerable Questions for Machine Reading Comprehension
We also present a way to construct training data for our question generation models by leveraging the existing reading comprehension dataset.
Unified Language Model Pre-training for Natural Language Understanding and Generation
This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.
Ranked #2 on
Generative Question Answering
on CoQA
(using extra training data)
Improved Dependency Parsing using Implicit Word Connections Learned from Unlabeled Data
Pre-trained word embeddings and language model have been shown useful in a lot of tasks.
Multiway Attention Networks for Modeling Sentence Pairs
Modeling sentence pairs plays the vital role for judging the relationship between two sentences, such as paraphrase identification, natural language inference, and answer sentence selection.
Ranked #11 on
Paraphrase Identification
on Quora Question Pairs
(Accuracy metric)
Gated Self-Matching Networks for Reading Comprehension and Question Answering
We first match the question and passage with gated attention-based recurrent networks to obtain the question-aware passage representation.
Ranked #35 on
Question Answering
on SQuAD1.1 dev
