Search Results for author:
Found 89 papers, 29 papers with code
Controlled tasks for model analysis: Retrieving discrete information from sequences
In recent years, the NLP community has shown increasing interest in analysing how deep learning models work.
MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models
Transformer-based language models (LMs) track contextual information through large, hard-coded input windows.
Unnatural language processing: How do language models handle machine-generated prompts?
Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are routinely outperformed by automatically generated token sequences with no apparent meaning or syntactic structure, including sequences of vectors from a model's embedding space.
Bridging Information-Theoretic and Geometric Compression in Language Models
For a language model (LM) to faithfully model human language, it must compress vast, potentially infinite information into relatively few dimensions.
Cross-Domain Image Captioning with Discriminative Finetuning
However, when the model is used without further tuning to generate captions for out-of-domain datasets, our discriminatively-finetuned captioner generates descriptions that resemble human references more than those produced by the same captioner without finetuning.
Can discrete information extraction prompts generalize across language models?
We study whether automatically-induced prompts that effectively extract information from a language model can also be used, out-of-the-box, to probe other language models for the same information.
Referential communication in heterogeneous communities of pre-trained visual deep networks
As a first step in this direction, we systematically explore the task of \textit{referential communication} in a community of heterogeneous state-of-the-art pre-trained visual networks, showing that they can develop, in a self-supervised way, a shared protocol to refer to a target object among a set of candidates.
Communication breakdown: On the low mutual intelligibility between human and neural captioning
We compare the 0-shot performance of a neural caption-based image retriever when given as input either human-produced captions or captions generated by a neural captioner.
How BPE Affects Memorization in Transformers
In this work, we demonstrate that the size of the subword vocabulary learned by Byte-Pair Encoding (BPE) greatly affects both ability and tendency of standard Transformer models to memorize training data, even when we control for the number of learned parameters.
On the proper role of linguistically-oriented deep net analysis in linguistic theorizing
A lively research field has recently emerged that uses experimental methods to probe the linguistic behavior of modern deep networks.
Interpretable agent communication from scratch (with a generic visual processor emerging on the side)
As deep networks begin to be deployed as autonomous agents, the issue of how they can communicate with each other becomes important.
Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans
We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing, namely the storing of grammatical number and gender information in working memory and its use in long-distance agreement (e. g., capturing the correct number agreement between subject and verb when they are separated by other phrases).
Emergent Multi-Agent Communication in the Deep Learning Era
The ability to cooperate through language is a defining feature of humans.
Permutation Equivariant Models for Compositional Generalization in Language
Humans understand novel sentences by composing meanings and roles of core language components.
Syntactic Structure from Deep Learning
Modern deep neural networks achieve impressive performance in engineering applications that require extensive linguistic skills, such as machine translation.
Compositionality and Generalization in Emergent Languages
Third, while compositionality is not necessary for generalization, it provides an advantage in terms of language transmission: The more compositional a language is, the more easily it will be picked up by new learners, even when the latter differ in architecture from the original agents.
Emergent Language Generalization and Acquisition Speed are not tied to Compositionality
Studies of discrete languages emerging when neural agents communicate to solve a joint task often look for evidence of compositional structure.
Rat big, cat eaten! Ideas for a useful deep-agent protolanguage
Deep-agent communities developing their own language-like communication protocol are a hot (or at least warm) topic in AI.
A Benchmark for Systematic Generalization in Grounded Language Understanding
In this paper, we introduce a new benchmark, gSCAN, for evaluating compositional generalization in situated language understanding.
Focus on What's Informative and Ignore What's not: Communication Strategies in a Referential Game
Research in multi-agent cooperation has shown that artificial agents are able to learn to play a simple referential game while developing a shared lexicon.
On the Distribution of Deep Clausal Embeddings: A Large Cross-linguistic Study
Embedding a clause inside another ({``}the girl [who likes cars [that run fast]] has arrived{''}) is a fundamental resource that has been argued to be a key driver of linguistic expressiveness.
EGG: a toolkit for research on Emergence of lanGuage in Games
There is renewed interest in simulating language emergence among deep neural agents that communicate to jointly solve a task, spurred by the practical aim to develop language-enabled interactive AIs, as well as by theoretical questions about the evolution of human language.
Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text
Recurrent neural networks (RNNs) have reached striking performance in many natural language processing tasks.
Entropy Minimization In Emergent Languages
There is growing interest in studying the languages that emerge when neural agents are jointly trained to solve tasks requiring communication through a discrete channel.
Word-order biases in deep-agent emergent communication
We train models to communicate about paths in a simple gridworld, using miniature languages that reflect or violate various natural language trends, such as the tendency to avoid redundancy or to minimize long-distance dependencies.
Anti-efficient encoding in emergent communication
Despite renewed interest in emergent language simulations with neural networks, little is known about the basic properties of the induced code, and how they compare to human language.
Miss Tools and Mr Fruit: Emergent communication in agents learning about object affordances
Recent research studies communication emergence in communities of deep network agents assigned a joint task, hoping to gain insights on human language evolution.
CNNs found to jump around more skillfully than RNNs: Compositional generalization in seq2seq convolutional networks
Lake and Baroni (2018) introduced the SCAN dataset probing the ability of seq2seq models to capture compositional generalizations, such as inferring the meaning of "jump around" 0-shot from the component words.
Linguistic generalization and compositionality in modern artificial neural networks
In the last decade, deep artificial neural networks have achieved astounding performance in many natural language processing tasks.
The emergence of number and syntax units in LSTM language models
Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure.
Human few-shot learning of compositional instructions
There have been striking recent improvements in machine learning for natural language processing, yet the best algorithms require vast amounts of experience and struggle to generalize new concepts in compositional ways.
Jump to better conclusions: SCAN both left and right
Lake and Baroni (2018) recently introduced the SCAN data set, which consists of simple commands paired with action sequences and is intended to test the strong generalization abilities of recurrent sequence-to-sequence models.
How agents see things: On visual representations in an emergent language game
There is growing interest in the language developed by agents interacting in emergent-communication settings.
Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks
Systematic compositionality is the ability to recombine meaningful units with regular and predictable outcomes, and it's seen as key to humans' capacity for generalization in language.
What you can cram into a single \$\&!\#* vector: Probing sentence embeddings for linguistic properties
Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing.
What you can cram into a single vector: Probing sentence embeddings for linguistic properties
Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing.
Colorless green recurrent networks dream hierarchically
Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language.
Memorize or generalize? Searching for a compositional RNN in a haystack
Neural networks are very powerful learning systems, but they do not readily generalize from one task to the other.
Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks
Humans can understand and produce new utterances effortlessly, thanks to their systematic compositional skills.
Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
Humans can understand and produce new utterances effortlessly, thanks to their compositional skills.
High-risk learning: acquiring new word vectors from tiny data
Distributional semantics models are known to struggle with small data.
Causal Discovery Using Proxy Variables
In this paper, we develop a framework to estimate the cause-effect relation between two static entities $x$ and $y$: for instance, an art masterpiece $x$ and its fraudulent copy $y$.
Living a discrete life in a continuous world: Reference with distributed representations
Reference is a crucial property of language that allows us to connect linguistic expressions to the world.
CommAI: Evaluating the first steps towards a useful general AI
With machine learning successfully applied to new daunting problems almost every day, general AI starts looking like an attainable goal.
Multi-Agent Cooperation and the Emergence of (Natural) Language
The sender is told one of them is the target and is allowed to send a message from a fixed, arbitrary vocabulary to the receiver.
"Show me the cup": Reference with Continuous Representations
One of the most basic functions of language is to refer to objects in a shared scene.
The LAMBADA dataset: Word prediction requiring a broad discourse context
We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task.
Towards Multi-Agent Communication-Based Language Learning
We propose an interactive multimodal framework for language learning.
The red one!: On learning to refer to things based on their discriminative properties
As a first step towards agents learning to communicate about their visual environment, we propose a system that, given visual representations of a referent (cat) and a context (sofa), identifies their discriminative attributes, i. e., properties that distinguish them (has_tail).
A Roadmap towards Machine Intelligence
The development of intelligent machines is one of the biggest unsolved challenges in computer science.
Unveiling the Dreams of Word Embeddings: Towards Language-Driven Image Generation
We introduce language-driven image generation, the task of generating an image visualizing the semantic contents of a word embedding, e. g., given the word embedding of grasshopper, we generate a natural image of a grasshopper.
From Visual Attributes to Adjectives through Decompositional Distributional Semantics
By building on the recent "zero-shot learning" approach, and paying attention to the linguistic nature of attributes as noun modifiers, and specifically adjectives, we show that it is possible to tag images with attribute-denoting adjectives even when no training data containing the relevant annotation are available.
Combining Language and Vision with a Multimodal Skip-gram Model
We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual information into account.
Deriving Boolean structures from distributional vectors
Corpus-based distributional semantic models capture degrees of semantic relatedness among the words of very large vocabularies, but have problems with logical phenomena such as entailment, that are instead elegantly handled by model-theoretic approaches, which, in turn, do not scale up.
Improving zero-shot learning by mitigating the hubness problem
The zero-shot paradigm exploits vector-based word representations extracted from text corpora with unsupervised methods to learn general mapping functions from other feature spaces onto word space, where the words associated to the nearest neighbours of the mapped vectors are used as their linguistic labels.
A SICK cure for the evaluation of compositional distributional semantic models
Shared and internationally recognized benchmarks are fundamental for the development of any computational system.
