Search Results for author:
Found 16 papers, 6 papers with code
Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain
Using brain recordings of participants watching a popular TV show, we analyze the effects of multi-modal connections and interactions in a pre-trained multi-modal video transformer on the alignment with uni- and multi-modal brain regions.
Speech language models lack important brain-relevant semantics
We investigate this question via a direct approach, in which we eliminate information related to specific low-level stimulus features (textual, speech, and visual) in the language model representations, and observe how this intervention affects the alignment with fMRI brain recordings acquired while participants read versus listened to the same naturalistic stories.
Perturbed examples reveal invariances shared by language models
Possessing a wide variety of invariances may be a key reason for the recent successes of large language models, and our framework can shed light on the types of invariances that are retained by or emerge in new models.
Getting aligned on representational alignment
Finally, we lay out open problems in representational alignment where progress can benefit all three of these fields.
What Happens During Finetuning of Vision Transformers: An Invariance Based Investigation
The pretrain-finetune paradigm usually improves downstream performance over training a model from scratch on the same task, becoming commonplace across many areas of machine learning.
Pointwise Representational Similarity
Concretely, we show how PNKA can be leveraged to develop a deeper understanding of (a) the input examples that are likely to be misclassified, (b) the concepts encoded by (individual) neurons in a layer, and (c) the effects of fairness interventions on learned representations.
Large language models can segment narrative events similarly to humans
In the future, GPT-3 may thereby help to elucidate the principles underlying human event perception.
Training language models to summarize narratives improves brain alignment
We show that training language models for deeper narrative understanding results in richer representations that have improved alignment to human brain activity.
Language models and brain alignment: beyond word-level semantics and prediction
The first perturbation is to improve the model's ability to predict the next word in the specific naturalistic stimulus text that the brain recordings correspond to.
Same Cause; Different Effects in the Brain
It is then natural to ask: "Is the activity in these different brain zones caused by the stimulus properties in the same way?"
A roadmap to reverse engineering real-world generalization by combining naturalistic paradigms, deep sampling, and predictive computational models
Real-world generalization, e. g., deciding to approach a never-seen-before animal, relies on contextual information as well as previous experiences.
Does injecting linguistic structure into language models lead to better alignment with brain recordings?
We evaluate across two fMRI datasets whether language models align better with brain recordings, if their attention is biased by annotations from syntactic or semantic formalisms.
Modeling Task Effects on Meaning Representation in the Brain via Zero-Shot MEG Prediction
These results suggest that only the end of semantic processing of a word is task-dependent, and pose a challenge for future research to formulate new hypotheses for earlier task effects as a function of the task and stimuli.
Inducing brain-relevant bias in natural language processing models
Progress in natural language processing (NLP) models that estimate representations of word sequences has recently been leveraged to improve the understanding of language processing in the brain.
Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)
Our results reveal differences in the context-related representations across these models.
An Empirical Study of Example Forgetting during Deep Neural Network Learning
Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks.
