Search Results for author:
Found 23 papers, 7 papers with code
HAMMR: HierArchical MultiModal React agents for generic VQA
We start from a multimodal ReAct-based system and make it hierarchical by enabling our HAMMR agents to call upon other specialized agents.
How (not) to ensemble LVLMs for VQA
So it is a trivial exercise to create an ensemble with substantial real gains.
Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
Empirically, we show that our dataset poses a hard challenge for large vision+language models as they perform poorly on our dataset: PaLI [14] is state-of-the-art on OK-VQA [37], yet it only achieves 13. 0% accuracy on our dataset.
The Missing Link: Finding label relations across datasets
To find relations between labels across datasets, we propose methods based on language, on vision, and on their combination.
How stable are Transferability Metrics evaluations?
Transferability metrics is a maturing field with increasing interest, which aims at providing heuristics for selecting the most suitable source models to transfer to a given target dataset, without fine-tuning them all.
Transferability Metrics for Selecting Source Model Ensembles
We address the problem of ensemble selection in transfer learning: Given a large pool of source models we want to select an ensemble of models which, after fine-tuning on the target training set, yields the best performance on the target test set.
Transferability Estimation using Bhattacharyya Class Separability
Then, we estimate their pairwise class separability using the Bhattacharyya coefficient, yielding a simple and effective measure of how well the source model transfers to the target task.
Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types
Our study leads to several insights and concrete recommendations: (1) for most tasks there exists a source which significantly outperforms ILSVRC'12 pre-training; (2) the image domain is the most important factor for achieving positive transfer; (3) the source dataset should \emph{include} the image domain of the target dataset to achieve best results; (4) at the same time, we observe only small negative effects when the image domain of the source task is much broader than that of the target; (5) transfer across task types can be beneficial, but its success is heavily dependent on both the source and target task types.
Towards Reusable Network Components by Learning Compatible Representations
This paper proposes to make a first step towards compatible and hence reusable network components.
Connecting Vision and Language with Localized Narratives
We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing.
Ranked #2 on
Image Captioning
on Localized Narratives
Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections
Our approach enables the adaptation to a particular object and its background, to distributions shifts in a test set, to specific object classes, and even to large domain changes, where the imaging modality changes between training and testing.
Ranked #1 on
Interactive Segmentation
on DRIONS-DB
The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
We present Open Images V4, a dataset of 9. 2M images with unified annotations for image classification, object detection and visual relationship detection.
Learning Intelligent Dialogs for Bounding Box Annotation
We demonstrate that (1) our agents are able to learn efficient annotation strategies in several scenarios, automatically adapting to the image difficulty, the desired quality of the boxes, and the detector strength; (2) in all scenarios the resulting annotation dialogs speed up annotation compared to manual box drawing alone and box verification alone, while also outperforming any fixed combination of verification and drawing in most scenarios; (3) in a realistic scenario where the detector is iteratively re-trained, our agents evolve a series of strategies that reflect the shifting trade-off between verification and drawing as the detector grows stronger.
Revisiting knowledge transfer for training object class detectors
We propose to revisit knowledge transfer for training object detectors on target classes from weakly supervised training images, helped by a set of source classes with bounding-box annotations.
The Devil is in the Decoder: Classification, Regression and GANs
Many machine vision applications, such as semantic segmentation and depth prediction, require predictions for every pixel of the input image.
COCO-Stuff: Thing and Stuff Classes in Context
To understand stuff and things in context we introduce COCO-Stuff, which augments all 164K images of the COCO 2017 dataset with pixel-wise annotations for 91 stuff classes.
Ranked #1 on
Semantic Segmentation
on COCO-Stuff
Region-based semantic segmentation with end-to-end training
We propose a novel method for semantic segmentation, the task of labeling each pixel in an image with a semantic class.
Ranked #1 on
Semantic Segmentation
on SIFT-flow
Joint Calibration for Semantic Segmentation
Semantic segmentation is the task of assigning a class-label to each pixel in an image.
Ranked #2 on
Semantic Segmentation
on SIFT-flow
Situational Object Boundary Detection
Intuitively, the appearance of true object boundaries varies from image to image.
Learning to Group Objects
It has been shown that such object regions can be used to focus computer vision techniques on the parts of an image that matter most leading to significant improvements in both object localisation and semantic segmentation in recent years.
