no code implementations • COLING 2022 • Sanghee J. Kim, Lang Yu, Allyson Ettinger
A critical component of competence in language is being able to identify relevant components of an utterance and reply appropriately.
no code implementations • 14 Apr 2024 • Yanhong Li, Chenghao Yang, Allyson Ettinger
In this paper, we set out to clarify these capabilities under a more stringent evaluation setting in which we disallow any kind of external feedback.
no code implementations • 12 Jan 2024 • Kanishka Misra, Allyson Ettinger, Kyle Mahowald
Recent zero-shot evaluations have highlighted important limitations in the abilities of language models (LMs) to perform meaning extraction.
no code implementations • 31 Oct 2023 • Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, Yejin Choi
Specifically, we propose and test the Generative AI Paradox hypothesis: generative models, having been trained directly to reproduce expert-like outputs, acquire generative capabilities that are not contingent upon -- and can therefore exceed -- their ability to understand those same types of outputs.
no code implementations • 26 Oct 2023 • Allyson Ettinger, Jena D. Hwang, Valentina Pyatkin, Chandra Bhagavatula, Yejin Choi
We compare models' analysis of this semantic structure across two settings: 1) direct production of AMR parses based on zero- and few-shot prompts, and 2) indirect partial reconstruction of AMR via metalinguistic natural language queries (e. g., "Identify the primary event of this sentence, and the predicate corresponding to that event.").
1 code implementation • 24 Oct 2023 • Chenghao Yang, Allyson Ettinger
Understanding sentence meanings and updating information states appropriately across time -- what we call "situational understanding" (SU) -- is a critical ability for human-like AI agents.
1 code implementation • NeurIPS 2023 • Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, Yejin Choi
We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures.
1 code implementation • 26 May 2023 • Jiaxuan Li, Lang Yu, Allyson Ettinger
Current pre-trained language models have enabled remarkable improvements in downstream tasks, but it remains difficult to distinguish effects of statistical correlation from more systematic logical reasoning grounded on the understanding of real world.
1 code implementation • 6 Dec 2022 • Jiaxuan Li, Lang Yu, Allyson Ettinger
Current pre-trained language models have enabled remarkable improvements in downstream tasks, but it remains difficult to distinguish effects of statistical correlation from more systematic logical reasoning grounded on understanding of the real world.
1 code implementation • 5 Oct 2022 • Kanishka Misra, Julia Taylor Rayz, Allyson Ettinger
A characteristic feature of human semantic cognition is its ability to not only store and retrieve the properties of concepts observed through experience, but to also facilitate the inheritance of properties (can breathe) from superordinate concepts (animal) to their subordinates (dog) -- i. e. demonstrate property inheritance.
1 code implementation • 5 Oct 2022 • Sanghee J. Kim, Lang Yu, Allyson Ettinger
A critical component of competence in language is being able to identify relevant components of an utterance and reply appropriately.
1 code implementation • 13 May 2022 • Kanishka Misra, Julia Taylor Rayz, Allyson Ettinger
To what extent can experience from language contribute to our conceptual knowledge?
no code implementations • EMNLP (BlackboxNLP) 2021 • Qinxuan Wu, Allyson Ettinger
While sentence anomalies have been applied periodically for testing in NLP, we have yet to establish a picture of the precise status of anomaly information in representations from NLP models.
no code implementations • CoNLL (EMNLP) 2021 • Lalchand Pandia, Yan Cong, Allyson Ettinger
We focus on testing models' ability to use pragmatic cues to predict discourse connectives, models' ability to understand implicatures relating to connectives, and the extent to which models show humanlike preferences regarding temporal dynamics of connectives.
no code implementations • EMNLP 2021 • Lalchand Pandia, Allyson Ettinger
Pre-trained LMs have shown impressive performance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to processing, retaining, and applying information presented in their input.
2 code implementations • Findings (ACL) 2021 • Lang Yu, Allyson Ettinger
Here we investigate the impact of fine-tuning on the capacity of contextualized embeddings to capture phrase meaning information beyond lexical content.
1 code implementation • 6 May 2021 • Kanishka Misra, Allyson Ettinger, Julia Taylor Rayz
Building on research arguing for the possibility of conceptual and categorical knowledge acquisition through statistics contained in language, we evaluate predictive language models (LMs) -- informed solely by textual input -- on a prevalent phenomenon in cognitive science: typicality.
no code implementations • 1 Jan 2021 • Davis Yoshida, Allyson Ettinger, Kevin Gimpel
Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years.
1 code implementation • EMNLP 2020 • Lang Yu, Allyson Ettinger
Deep transformer models have pushed performance on NLP tasks to new limits, suggesting sophisticated treatment of complex linguistic inputs, such as phrases.
3 code implementations • EMNLP 2020 • Shubham Toshniwal, Sam Wiseman, Allyson Ettinger, Karen Livescu, Kevin Gimpel
Long document coreference resolution remains a challenging task due to the large memory and runtime requirements of current models.
Ranked #9 on Coreference Resolution on CoNLL 2012
1 code implementation • Findings of the Association for Computational Linguistics 2020 • Kanishka Misra, Allyson Ettinger, Julia Taylor Rayz
Models trained to estimate word probabilities in context have become ubiquitous in natural language processing.
no code implementations • 16 Aug 2020 • Davis Yoshida, Allyson Ettinger, Kevin Gimpel
Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years.
1 code implementation • ACL 2020 • Shubham Toshniwal, Allyson Ettinger, Kevin Gimpel, Karen Livescu
We propose PeTra, a memory-augmented neural network designed to track entities in its memory slots.
Ranked #1 on Coreference Resolution on GAP (F1 metric)
no code implementations • ACL 2020 • Josef Klafka, Allyson Ettinger
Although models using contextual word embeddings have achieved state-of-the-art results on a host of NLP tasks, little is known about exactly what information these embeddings encode about the context words that they are understood to reflect.
2 code implementations • TACL 2020 • Allyson Ettinger
Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models.
1 code implementation • COLING 2018 • Allyson Ettinger, Ahmed Elgohary, Colin Phillips, Philip Resnik
We describe the details of the method and generation system, and then present results of experiments applying our method to probe for compositional information in embeddings from a number of existing sentence composition models.
no code implementations • WS 2017 • Allyson Ettinger, Sudha Rao, Hal Daumé III, Emily M. Bender
This paper presents a summary of the first Workshop on Building Linguistically Generalizable Natural Language Processing Systems, and the associated Build It Break It, The Language Edition shared task.