Search Results for author:
Found 25 papers, 17 papers with code
ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders
This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation.
Ranked #45 on
Semantic Segmentation
on ADE20K
UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding
Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships.
Scaling Language-Image Pre-training via Masking
We present Fast Language-Image Pre-training (FLIP), a simple and more efficient method for training CLIP.
Exploring Long-Sequence Masked Autoencoders
Masked Autoencoding (MAE) has emerged as an effective approach for pre-training representations across multiple domains.
FLAVA: A Foundational Language And Vision Alignment Model
State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks.
Ranked #4 on
Image Retrieval
on MS COCO
UniT: Multimodal Multitask Learning with a Unified Transformer
We propose UniT, a Unified Transformer model to simultaneously learn the most prominent tasks across different domains, ranging from object detection to natural language understanding and multimodal reasoning.
Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image
We present Worldsheet, a method for novel view synthesis using just a single RGB image as input.
TextCaps: a Dataset for Image Captioning with Reading Comprehension
Image descriptions can help visually impaired people to quickly understand the image content.
Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question.
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
The actual grounding can connect language to the environment through multiple modalities, e. g. "stop at the door" might ground into visual objects, while "turn right" might rely only on the geometric structure of a route.
Language-Conditioned Graph Networks for Relational Reasoning
E. g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction.
Ranked #3 on
Referring Expression Comprehension
on CLEVR-Ref+
Grounding Visual Explanations
Our model improves the textual explanation quality of fine-grained classification decisions on the CUB dataset by mentioning phrases that are grounded in the image.
Explainable Neural Computation via Stack Neural Module Networks
In complex inferential tasks like question answering, machine learning models must confront two challenges: the need to implement a compositional reasoning process, and, in many applications, the need for this reasoning process to be interpretable to assist users in both development and prediction.
Ranked #14 on
Referring Expression Comprehension
on Talk2Car
Generating Counterfactual Explanations with Natural Language
We call such textual explanations counterfactual explanations, and propose an intuitive method to generate counterfactual explanations by inspecting which evidence in an input is missing, but might contribute to a different classification decision if present in the image.
Speaker-Follower Models for Vision-and-Language Navigation
We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction.
Learning to Segment Every Thing
Most methods for object instance segmentation require all training examples to be labeled with segmentation masks.
Grounding Visual Explanations (Extended Abstract)
Existing models which generate textual explanations enforce task relevance through a discriminative term loss function, but such mechanisms only weakly constrain mentioned object parts to actually be present in the image.
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems.
Ranked #43 on
Visual Question Answering (VQA)
on VQA v2 test-dev
Modeling Relationships in Referential Expressions with Compositional Modular Networks
In this paper we instead present a modular deep architecture capable of analyzing referential expressions into their component parts, identifying entities and relationships mentioned in the input expression and grounding them all in the scene.
Ranked #1 on
Visual Question Answering (VQA)
on Visual7W
Utilizing Large Scale Vision and Text Datasets for Image Segmentation from Referring Expressions
Image segmentation from referring expressions is a joint vision and language modeling task, where the input is an image and a textual expression describing a particular region in the image; and the goal is to localize and segment the specific image region based on the given expression.
Segmentation from Natural Language Expressions
To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information.
Ranked #16 on
Referring Expression Segmentation
on J-HMDB
Natural Language Object Retrieval
In this paper, we address the task of natural language object retrieval, to localize a target object within a given image based on a natural language query of the object.
Ranked #12 on
Referring Expression Comprehension
on Talk2Car
Grounding of Textual Phrases in Images by Reconstruction
We propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly.
Ranked #12 on
Phrase Grounding
on Flickr30k Entities Test
Spatial Semantic Regularisation for Large Scale Object Detection
Our approach proves to be especially useful in large scale settings with thousands of classes, where spatial and semantic interactions are very frequent and only weakly supervised detectors can be built due to a lack of bounding box annotations.
LSDA: Large Scale Detection Through Adaptation
A major challenge in scaling object detection is the difficulty of obtaining labeled images for large numbers of categories.
