Search Results for author:
Found 54 papers, 38 papers with code
Massive Activations in Large Language Models
We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e. g., 100, 000 times larger).
Revisiting Feature Prediction for Learning Visual Representations from Video
This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision.
Deconstructing Denoising Diffusion Models for Self-Supervised Learning
In this study, we examine the representation learning abilities of Denoising Diffusion Models (DDM) that were originally purposed for image generation.
Learning to (Learn at Test Time)
Our inner loop turns out to be equivalent to linear attention when the inner-loop learner is only a linear model, and to self-attention when it is a kernel estimator.
Test-Time Training on Video Streams
Before making a prediction on each test instance, the model is trained on the same instance using a self-supervised task, such as image reconstruction with masked autoencoders.
Improving Selective Visual Question Answering by Learning from Your Peers
In this work, we explore Selective VQA in both in-distribution (ID) and OOD scenarios, where models are presented with mixtures of ID and OOD data.
R-MAE: Regions Meet Masked Autoencoders
In this work, we explore regions as a potential visual analogue of words for self-supervised image representation learning.
ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders
This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation.
Ranked #45 on
Semantic Segmentation
on ADE20K
UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding
Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships.
EurNet: Efficient Multi-Range Relational Modeling of Spatial Multi-Relational Data
We study EurNets in two important domains for image and protein structure modeling.
Exploring Long-Sequence Masked Autoencoders
Masked Autoencoding (MAE) has emerged as an effective approach for pre-training representations across multiple domains.
Test-Time Training with Masked Autoencoders
Test-time training adapts to a new test distribution on the fly by optimizing a model for each test input using self-supervision.
On the Importance of Asymmetry for Siamese Representation Learning
Many recent self-supervised frameworks for visual representation learning are based on certain forms of Siamese networks.
LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval
In this work, we propose LoopITR, which combines them in the same network for joint learning.
Point-Level Region Contrast for Object Detection Pre-Training
In this work we present point-level region contrast, a self-supervised pre-training approach for the task of object detection.
Benchmarking Detection Transfer Learning with Vision Transformers
The complexity of object detection methods can make this benchmarking non-trivial when new architectures, such as Vision Transformer (ViT) models, arrive.
Masked Autoencoders Are Scalable Vision Learners
Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
Towards Demystifying Representation Learning with Non-contrastive Self-supervision
Non-contrastive methods of self-supervised learning (such as BYOL and SimSiam) learn representations by minimizing the distance between two views of the same image.
NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict aware Supernet Training
In this work, we observe that the poor performance is due to a gradient conflict issue: the gradients of different sub-networks conflict with that of the supernet more severely in ViTs than CNNs, which leads to early saturation in training and inferior convergence.
Ranked #7 on
Neural Architecture Search
on ImageNet
A Data-Efficient Approach to Behind-the-Meter Solar Generation Disaggregation
With the emergence of cost effective battery storage and the decline in the solar photovoltaic (PV) levelized cost of energy (LCOE), the number of behind-the-meter solar PV systems is expected to increase steadily.
An Empirical Study of Training Self-Supervised Vision Transformers
In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
Out-of-Distribution Generalization
Self-Supervised Image Classification
+1
Understanding self-supervised Learning Dynamics without Contrastive Pairs
While contrastive approaches of self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing views from different data points (negative pairs), recent \emph{non-contrastive} SSL (e. g., BYOL and SimSiam) show remarkable performance {\it without} negative pairs, with an extra learnable predictor and a stop-gradient operation.
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image.
Ranked #6 on
Visual Question Answering (VQA)
on A-OKVQA
Exploring Simple Siamese Representation Learning
Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing.
Ranked #94 on
Self-Supervised Image Classification
on ImageNet
Representation Learning
Self-Supervised Image Classification
Understanding Self-supervised Learning with Dual Deep Networks
We propose a novel theoretical framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks (e. g., SimCLR).
Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation
We also demonstrate that reducing the task of room navigation to point navigation improves the performance further.
Overcoming Statistical Shortcuts for Open-ended Visual Counting
First, we propose the Modifying Count Distribution (MCD) protocol, which penalizes models that over-rely on statistical shortcuts.
MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
This paper focuses on visual counting, which aims to predict the number of occurrences given a natural image and a query (e. g. a question or a category).
Ranked #1 on
Object Counting
on HowMany-QA
Improved Baselines with Momentum Contrastive Learning
Contrastive unsupervised learning has recently shown encouraging progress, e. g., in Momentum Contrast (MoCo) and SimCLR.
Ranked #3 on
Contrastive Learning
on imagenet-1k
ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes
Compared to prior work on multi-modal detection, we explicitly extract both geometric and semantic features from the 2D images.
Ranked #2 on
3D Object Detection
on SUN-RGBD
(using extra training data)
In Defense of Grid Features for Visual Question Answering
Popularized as 'bottom-up' attention, bounding box (or region) based visual features have recently surpassed vanilla grid-based convolutional features as the de facto standard for vision and language tasks like visual question answering (VQA).
Ranked #18 on
Visual Question Answering (VQA)
on VQA v2 test-std
Towards VQA Models That Can Read
We show that LoRRA outperforms existing state-of-the-art VQA models on our TextVQA dataset.
Ranked #3 on
Visual Question Answering (VQA)
on VizWiz 2018
Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentation
Accurate multi-organ abdominal CT segmentation is essential to many clinical applications such as computer-aided intervention.
Embodied Visual Recognition
Passive visual systems typically fail to recognize objects in the amodal setting where they are heavily occluded.
Multi-Target Embodied Question Answering
To address this, we propose a modular architecture composed of a program generator, a controller, a navigator, and a VQA module.
TensorMask: A Foundation for Dense Object Segmentation
To formalize this, we treat dense instance segmentation as a prediction task over 4D tensors and present a general framework called TensorMask that explicitly captures this geometry and enables novel operators on 4D tensors.
Ranked #90 on
Instance Segmentation
on COCO test-dev
Cycle-Consistency for Robust Visual Question Answering
Despite significant progress in Visual Question Answering over the years, robustness of today's VQA models leave much to be desired.
nocaps: novel object captioning at scale
To encourage the development of image captioning models that can learn visual concepts from alternative data sources, such as object detection datasets, we present the first large-scale benchmark for this task.
Grounded Video Description
Our dataset, ActivityNet-Entities, augments the challenging ActivityNet Captions dataset with 158k bounding box annotations, each grounding a noun phrase.
Pythia v0.1: the Winning Entry to the VQA Challenge 2018
We demonstrate that by making subtle but important changes to the model architecture and the learning rate schedule, fine-tuning image features, and adding data augmentation, we can significantly improve the performance of the up-down model on VQA v2. 0 dataset -- from 65. 67% to 70. 22%.
Ranked #10 on
Visual Question Answering (VQA)
on A-OKVQA
Iterative Visual Reasoning Beyond Convolutions
The framework consists of two core modules: a local module that uses spatial memory to store previous beliefs with parallel updates; and a global graph-reasoning module.
CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
The game involves two players: a Teller and a Drawer.
Spatial Memory for Context Reasoning in Object Detection
On the other hand, modeling object-object relationships requires {\bf spatial} reasoning -- not only do we need a memory to store the spatial layout, but also a effective reasoning module to extract spatial patterns.
PixelNet: Representation of the pixels, by the pixels, and for the pixels
We explore design principles for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation.
An Implementation of Faster RCNN with Study for Region Sampling
We adapted the join-training scheme of Faster RCNN framework from Caffe to TensorFlow as a baseline implementation for object detection.
PixelNet: Towards a General Pixel-level Architecture
We explore architectures for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation.
Learning Visual Storylines with Skipping Recurrent Neural Networks
What does a typical visit to Paris look like?
Visualizing and Understanding Neural Models in NLP
While neural networks have been successfully applied to many NLP tasks the resulting vector-based models are very difficult to interpret.
Mind's Eye: A Recurrent Visual Representation for Image Caption Generation
Results are better than or comparable to state-of-the-art results on the image and sentence retrieval tasks for methods using similar visual features.
Sense Discovery via Co-Clustering on Images and Text
We present a co-clustering framework that can be used to discover multiple semantic and visual senses of a given Noun Phrase (NP).
Webly Supervised Learning of Convolutional Networks
Specifically inspired by curriculum learning, we present a two-step approach for CNN training.
Microsoft COCO Captions: Data Collection and Evaluation Server
In this paper we describe the Microsoft COCO Caption dataset and evaluation server.
Learning a Recurrent Visual Representation for Image Caption Generation
Results are better than or comparable to state-of-the-art results on the image and sentence retrieval tasks for methods using similar visual features.
Enriching Visual Knowledge Bases via Object Discovery and Segmentation
In this paper, we propose to enrich these knowledge bases by automatically discovering objects and their segmentations from noisy Internet images.
