Search Results for author:
Found 15 papers, 6 papers with code
A Small but Informed and Diverse Model: The Case of the Multimodal GuessWhat!? Guessing Game
Pre-trained Vision and Language Transformers achieve high performance on downstream tasks due to their ability to transfer representational knowledge accumulated during pretraining on substantial amounts of data.
They Are Not All Alike: Answering Different Spatial Questions Requires Different Grounding Strategies
By analyzing LXMERT errors and its attention mechanisms, we find that our classification helps to gain a better understanding of the skills required to answer different spatial questions.
Visually Grounded Follow-up Questions: a Dataset of Spatial Questions Which Require Dialogue History
We call the question that restricts the context: trigger, and we call the spatial question that requires the trigger question to be answered: zoomer.
ACT-Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments
This paves the way for a systematic way of evaluating embodied AI agents that understand grounded actions.
Naming, Describing, and Quantifying Visual Objects in Humans and LLMs
While human speakers use a variety of different expressions when describing the same object in an image, giving rise to a distribution of plausible labels driven by pragmatic constraints, the extent to which current Vision \& Language Large Language Models (VLLMs) can mimic this crucial feature of language use is an open question.
Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions
Clarification questions are an essential dialogue tool to signal misunderstanding, ambiguities, and under-specification in language use.
Are Current Decoding Strategies Capable of Facing the Challenges of Visual Dialogue?
Decoding strategies play a crucial role in natural language generation systems.
Looking for Confirmations: An Effective and Human-Like Visual Dialogue Strategy
Inspired by the cognitive literature on information search and cross-situational word learning, we design Confirm-it, a model based on a beam search re-ranking algorithm that guides an effective goal-oriented strategy by asking questions that confirm the model's conjecture about the referent.
``I've Seen Things You People Wouldn't Believe'': Hallucinating Entities in GuessWhat?!
We also analyse where hallucinations tend to occur more often through the dialogue: hallucinations are less frequent in earlier turns, cause a cascade hallucination effect, and are often preceded by negative answers, which have been shown to be harder to ground.
Overprotective Training Environments Fall Short at Testing Time: Let Models Contribute to Their Own Training
Despite important progress, conversational systems often generate dialogues that sound unnatural to humans.
The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues
When training a model on referential dialogue guessing games, the best model is usually chosen based on its task success.
On the role of effective and referring questions in GuessWhat?!
Regarding our first metric, we find that successful dialogues do not have a higher percentage of effective questions for most models.
Effective questions in referential visual dialogue
Task success is the standard metric used to evaluate these systems.
Quantifiers in a Multimodal World: Hallucinating Vision with Language and Sound
Inspired by the literature on multisensory integration, we develop a computational model to ground quantifiers in perception.
Grounded Textual Entailment
Capturing semantic relations between sentences, such as entailment, is a long-standing challenge for computational semantics.
Ranked #2 on
Natural Language Inference
on V-SNLI
