Search Results for author:
Found 31 papers, 15 papers with code
“No, They Did Not”: Dialogue Response Dynamics in Pre-trained Language Models
A critical component of competence in language is being able to identify relevant components of an utterance and reply appropriately.
When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models
In this paper, we set out to clarify these capabilities under a more stringent evaluation setting in which we disallow any kind of external feedback.
Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently
Recent zero-shot evaluations have highlighted important limitations in the abilities of language models (LMs) to perform meaning extraction.
The Generative AI Paradox: "What It Can Create, It May Not Understand"
Specifically, we propose and test the Generative AI Paradox hypothesis: generative models, having been trained directly to reproduce expert-like outputs, acquire generative capabilities that are not contingent upon -- and can therefore exceed -- their ability to understand those same types of outputs.
"You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation
We compare models' analysis of this semantic structure across two settings: 1) direct production of AMR parses based on zero- and few-shot prompts, and 2) indirect partial reconstruction of AMR via metalinguistic natural language queries (e. g., "Identify the primary event of this sentence, and the predicate corresponding to that event.").
Can You Follow Me? Testing Situational Understanding in ChatGPT
Understanding sentence meanings and updating information states appropriately across time -- what we call "situational understanding" (SU) -- is a critical ability for human-like AI agents.
Faith and Fate: Limits of Transformers on Compositionality
We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures.
Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios
Current pre-trained language models have enabled remarkable improvements in downstream tasks, but it remains difficult to distinguish effects of statistical correlation from more systematic logical reasoning grounded on the understanding of real world.
Counterfactual reasoning: Do language models need world knowledge for causal understanding?
Current pre-trained language models have enabled remarkable improvements in downstream tasks, but it remains difficult to distinguish effects of statistical correlation from more systematic logical reasoning grounded on understanding of the real world.
"No, they did not": Dialogue response dynamics in pre-trained language models
A critical component of competence in language is being able to identify relevant components of an utterance and reply appropriately.
COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models
A characteristic feature of human semantic cognition is its ability to not only store and retrieve the properties of concepts observed through experience, but to also facilitate the inheritance of properties (can breathe) from superordinate concepts (animal) to their subordinates (dog) -- i. e. demonstrate property inheritance.
A Property Induction Framework for Neural Language Models
To what extent can experience from language contribute to our conceptual knowledge?
Variation and generality in encoding of syntactic anomaly information in sentence embeddings
While sentence anomalies have been applied periodically for testing in NLP, we have yet to establish a picture of the precise status of anomaly information in representations from NLP models.
Pragmatic competence of pre-trained language models through the lens of discourse connectives
We focus on testing models' ability to use pragmatic cues to predict discourse connectives, models' ability to understand implicatures relating to connectives, and the extent to which models show humanlike preferences regarding temporal dynamics of connectives.
Sorting through the noise: Testing robustness of information processing in pre-trained language models
Pre-trained LMs have shown impressive performance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to processing, retaining, and applying information presented in their input.
On the Interplay Between Fine-tuning and Composition in Transformers
Here we investigate the impact of fine-tuning on the capacity of contextualized embeddings to capture phrase meaning information beyond lexical content.
Do language models learn typicality judgments from text?
Building on research arguing for the possibility of conceptual and categorical knowledge acquisition through statistics contained in language, we evaluate predictive language models (LMs) -- informed solely by textual input -- on a prevalent phenomenon in cognitive science: typicality.
Adding Recurrence to Pretrained Transformers
Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years.
Assessing Phrasal Representation and Composition in Transformers
Deep transformer models have pushed performance on NLP tasks to new limits, suggesting sophisticated treatment of complex linguistic inputs, such as phrases.
Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks
Long document coreference resolution remains a challenging task due to the large memory and runtime requirements of current models.
Ranked #9 on
Coreference Resolution
on CoNLL 2012
Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming
Models trained to estimate word probabilities in context have become ubiquitous in natural language processing.
Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size
Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years.
PeTra: A Sparsely Supervised Memory Model for People Tracking
We propose PeTra, a memory-augmented neural network designed to track entities in its memory slots.
Ranked #1 on
Coreference Resolution
on GAP
(F1 metric)
Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words
Although models using contextual word embeddings have achieved state-of-the-art results on a host of NLP tasks, little is known about exactly what information these embeddings encode about the context words that they are understood to reflect.
What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models
Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models.
Assessing Composition in Sentence Vector Representations
We describe the details of the method and generation system, and then present results of experiments applying our method to probe for compositional information in embeddings from a number of existing sentence composition models.
Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task
This paper presents a summary of the first Workshop on Building Linguistically Generalizable Natural Language Processing Systems, and the associated Build It Break It, The Language Edition shared task.
