Search Results for author:
Found 21 papers, 8 papers with code
Rationality Report Cards: Assessing the Economic Rationality of Large Language Models
We begin by surveying the economic literature on rational decision making, taxonomizing a large set of fine-grained "elements" that an agent should exhibit, along with dependencies between them.
Tradeoffs Between Alignment and Helpfulness in Language Models
Representation engineering yields gains in alignment oriented tasks such as resistance to adversarial attacks and reduction of social biases, but was also shown to cause a decrease in the ability of the model to perform basic tasks.
Generating Benchmarks for Factuality Evaluation of Language Models
FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements.
Human or Not? A Gamified Approach to the Turing Test
Over the course of a month, the game was played by over 1. 5 million users who engaged in anonymous two-minute chat sessions with either another human or an AI language model which was prompted to behave like humans.
Fundamental Limitations of Alignment in Large Language Models
An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users.
In-Context Retrieval-Augmented Language Models
Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance.
Parallel Context Windows for Large Language Models
We present Parallel Context Windows (PCW), a method that alleviates the context window restriction for any off-the-shelf LLM without further training.
MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning
Huge language models (LMs) have ushered in a new era for AI, serving as a gateway to natural-language-based knowledge tasks.
Standing on the Shoulders of Giant Frozen Language Models
To demonstrate this, we introduce three novel methods for leveraging frozen models: input-dependent prompt tuning, frozen readers, and recursive LMs, each of which vastly improves on current frozen-model approaches.
Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks
Recently, several works have demonstrated high gains by taking a straightforward approach for incorporating intermediate supervision in compounded natural language problems: the sequence-to-sequence LM is fed with an augmented input, in which the decomposed tasks' labels are simply concatenated to the original input.
The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design
We highlight a bias introduced by this common practice: we prove that the pretrained NLM can model much stronger dependencies between text segments that appeared in the same training example, than it can between text segments that appeared in different training examples.
Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
After their successful debut in natural language processing, Transformer architectures are now becoming the de-facto standard in many domains.
PMI-Masking: Principled masking of correlated spans
Specifically, we show experimentally that PMI-Masking reaches the performance of prior masking approaches in half the training time, and consistently improves performance at the end of training.
The Depth-to-Width Interplay in Self-Attention
Our guidelines elucidate the depth-to-width trade-off in self-attention networks of sizes up to the scale of GPT3 (which we project to be too deep for its size), and beyond, marking an unprecedented width of 30K as optimal for a 1-Trillion parameter network.
SenseBERT: Driving Some Sense into BERT
The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding.
Ranked #11 on
Word Sense Disambiguation
on Words in Context
Deep autoregressive models for the efficient variational simulation of many-body quantum systems
Artificial Neural Networks were recently shown to be an efficient representation of highly-entangled many-body quantum states.
Quantum Entanglement in Deep Learning Architectures
Modern deep learning has enabled unprecedented achievements in various domains.
Benefits of Depth for Long-Term Memory of Recurrent Networks
We prove that deep recurrent networks support Start-End separation ranks which are exponentially higher than those supported by their shallow counterparts.
On the Long-Term Memory of Deep Recurrent Networks
A key attribute that drives the unprecedented success of modern Recurrent Neural Networks (RNNs) on learning tasks which involve sequential data, is their ability to model intricate long-term temporal dependencies.
Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositions
Expressive efficiency refers to the ability of a network architecture to realize functions that require an alternative architecture to be much larger.
Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design
This description enables us to carry a graph-theoretic analysis of a convolutional network, with which we demonstrate a direct control over the inductive bias of the deep network via its channel numbers, that are related to the min-cut in the underlying graph.
