Search Results for author:
Found 25 papers, 16 papers with code
Less Descriptive yet Discriminative: Quantifying the Properties of Multimodal Referring Utterances via CLIP
In this work, we use a transformer-based pre-trained multimodal model, CLIP, to shed light on the mechanisms employed by human speakers when referring to visual entities.
Probing Cross-Modal Representations in Multi-Step Relational Reasoning
We investigate the representations learned by vision and language models in tasks that require relational reasoning.
Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms
Most studies determine which edges belong in a LM's circuit by performing causal interventions on each edge independently, but this scales poorly with model size.
Naming, Describing, and Quantifying Visual Objects in Humans and LLMs
While human speakers use a variety of different expressions when describing the same object in an image, giving rise to a distribution of plausible labels driven by pragmatic constraints, the extent to which current Vision \& Language Large Language Models (VLLMs) can mimic this crucial feature of language use is an open question.
Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST!
In this work, we propose a novel Dataset of semantically Underspecified Sentences grouped by Type (DUST) and use it to study whether pre-trained language models (LMs) correctly identify and interpret underspecified sentences.
Describing Images $\textit{Fast and Slow}$: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes
There is an intricate relation between the properties of an image and how humans behave while describing the image.
GROOViST: A Metric for Grounding Objects in Visual Storytelling
A proper evaluation of stories generated for a sequence of images -- the task commonly referred to as visual storytelling -- must consider multiple aspects, such as coherence, grammatical correctness, and visual grounding.
When Language Models Fall in Love: Animacy Processing in Transformer Language Models
However, we also show that even when presented with stories about atypically animate entities, such as a peanut in love, LMs adapt: they treat these entities as animate, though they do not adapt as well as humans.
The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models
Despite the impressive performance achieved by pre-trained language-and-vision models in downstream tasks, it remains an open question whether this reflects a proper understanding of image-text interaction.
Fixing confirmation bias in feature attribution methods via semantic match
Feature attribution methods have become a staple method to disentangle the complex behavior of black box models.
Dealing with Semantic Underspecification in Multimodal NLP
Intelligent systems that aim at mastering language as humans do must deal with its semantic underspecification, namely, the possibility for a linguistic signal to convey only part of the information needed for communication to succeed.
Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind
Inspired by psycholinguistic theories, we endow our speaker with the ability to adapt its referring expressions via a simulation module that monitors the effectiveness of planned utterances from the listener's perspective.
A Psycholinguistic Analysis of BERT's Representations of Compounds
We build on recent studies that explore semantic information in Transformers at the word level and test whether BERT aligns with human semantic intuitions when dealing with expressions (e. g., sunlight) whose overall meaning depends -- to a various extent -- on the semantics of the constituent words (sun, light).
EaSe: A Diagnostic Tool for VQA based on Answer Diversity
We propose EASE, a simple diagnostic tool for Visual Question Answering (VQA) which quantifies the difficulty of an image, question sample.
Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts
We propose a generation model that produces referring utterances grounded in both the visual and the conversational context.
Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze
When speakers describe an image, they tend to look at objects before mentioning them.
Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision
This paper introduces BD2BB, a novel language and vision benchmark that requires multimodal models combine complementary information from the two modalities.
Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts
This work aims at modeling how the meaning of gradable adjectives of size (`big', `small') can be learned from visually-grounded contexts.
The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
We show that our metric is effective in providing a more fine-grained evaluation both on the quantitative and qualitative level.
Some of Them Can be Guessed! Exploring the Effect of Linguistic Context in Predicting Quantifiers
We study the role of linguistic context in predicting quantifiers (`few', `all').
Comparatives, Quantifiers, Proportions: A Multi-Task Model for the Learning of Quantities from Vision
The present work investigates whether different quantification mechanisms (set comparison, vague quantification, and proportional estimation) can be jointly learned from visual scenes by a multi-task computational model.
FOIL it! Find One mismatch between Image and Language caption
In this paper, we aim to understand whether current language and vision (LaVi) models truly grasp the interaction between the two modalities.
Pay Attention to Those Sets! Learning Quantification from Images
We however argue that precisely identifying the composition of the sets is not only beyond current state-of-the-art models but perhaps even detrimental to a task that is most efficiently performed by refining the approximate numerosity estimator of the system.
Be Precise or Fuzzy: Learning the Meaning of Cardinals and Quantifiers from Vision
People can refer to quantities in a visual scene by using either exact cardinals (e. g. one, two, three) or natural language quantifiers (e. g. few, most, all).
The LAMBADA dataset: Word prediction requiring a broad discourse context
We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task.
