Search Results for author: Kevin Leyton-Brown

Found 41 papers, 16 papers with code

Rationality Report Cards: Assessing the Economic Rationality of Large Language Models

no code implementations14 Feb 2024 Narun Raman, Taylor Lundy, Samuel Amouyal, Yoav Levine, Kevin Leyton-Brown, Moshe Tennenholtz

We begin by surveying the economic literature on rational decision making, taxonomizing a large set of fine-grained "elements" that an agent should exhibit, along with dependencies between them.

Decision Making

Utilitarian Algorithm Configuration

1 code implementation NeurIPS 2023 Devon R. Graham, Kevin Leyton-Brown, Tim Roughgarden

We prove upper bounds on the runtime of these procedures that are similar to theoretical lower bounds, while also demonstrating their performance empirically.

Generating Benchmarks for Factuality Evaluation of Language Models

2 code implementations13 Jul 2023 Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, Yoav Shoham

FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements.

Language Modelling Retrieval

How to Evaluate Behavioral Models

no code implementations7 Jun 2023 Greg d'Eon, Sophie Greenwood, Kevin Leyton-Brown, James R. Wright

Researchers building behavioral models, such as behavioral game theorists, use experimental data to evaluate predictive models of human behavior.

In-Context Retrieval-Augmented Language Models

1 code implementation31 Jan 2023 Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham

Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance.

Language Modelling Retrieval +1

Parallel Context Windows for Large Language Models

1 code implementation21 Dec 2022 Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham

We present Parallel Context Windows (PCW), a method that alleviates the context window restriction for any off-the-shelf LLM without further training.

In-Context Learning Playing the Game of 2048 +2

UNSAT Solver Synthesis via Monte Carlo Forest Search

1 code implementation22 Nov 2022 Chris Cameron, Jason Hartford, Taylor Lundy, Tuan Truong, Alan Milligan, Rex Chen, Kevin Leyton-Brown

We introduce Monte Carlo Forest Search (MCFS), a class of reinforcement learning (RL) algorithms for learning policies in {tree MDPs}, for which policy execution involves traversing an exponential-sized tree.

reinforcement-learning Reinforcement Learning (RL)

Artificial Intelligence and Life in 2030: The One Hundred Year Study on Artificial Intelligence

no code implementations31 Oct 2022 Peter Stone, Rodney Brooks, Erik Brynjolfsson, Ryan Calo, Oren Etzioni, Greg Hager, Julia Hirschberg, Shivaram Kalyanakrishnan, Ece Kamar, Sarit Kraus, Kevin Leyton-Brown, David Parkes, William Press, AnnaLee Saxenian, Julie Shah, Milind Tambe, Astro Teller

In September 2016, Stanford's "One Hundred Year Study on Artificial Intelligence" project (AI100) issued the first report of its planned long-term periodic assessment of artificial intelligence (AI) and its impact on society.

Better Peer Grading through Bayesian Inference

1 code implementation2 Sep 2022 Hedayat Zarkoob, Greg d'Eon, Lena Podina, Kevin Leyton-Brown

Peer grading systems aggregate noisy reports from multiple students to approximate a true grade as closely as possible.

Bayesian Inference

Formalizing Preferences Over Runtime Distributions

1 code implementation25 May 2022 Devon R. Graham, Kevin Leyton-Brown, Tim Roughgarden

We propose a principled alternative, taking a utility-theoretic approach to characterize the scoring functions that describe preferences over algorithms.

Standing on the Shoulders of Giant Frozen Language Models

no code implementations21 Apr 2022 Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes, Daniel Jannai, Dor Muhlgay, Yoni Osin, Opher Lieber, Barak Lenz, Shai Shalev-Shwartz, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham

To demonstrate this, we introduce three novel methods for leveraging frozen models: input-dependent prompt tuning, frozen readers, and recursive LMs, each of which vastly improves on current frozen-model approaches.

Matching Papers and Reviewers at Large Conferences

1 code implementation24 Feb 2022 Kevin Leyton-Brown, Mausam, Yatin Nandwani, Hedayat Zarkoob, Chris Cameron, Neil Newman, Dinesh Raghu

Peer-reviewed conferences, the main publication venues in CS, rely critically on matching highly qualified reviewers for each paper.

The Perils of Learning Before Optimizing

no code implementations18 Jun 2021 Chris Cameron, Jason Hartford, Taylor Lundy, Kevin Leyton-Brown

Typically, learning the prediction model used to generate the optimization problem and solving that problem are performed in two separate stages.

Stochastic Optimization

Dynamic Weighted Matching with Heterogeneous Arrival and Departure Rates

no code implementations1 Dec 2020 Natalie Collina, Nicole Immorlica, Kevin Leyton-Brown, Brendan Lucier, Neil Newman

The value of a match is determined by the types of the matched agents.

Computer Science and Game Theory Data Structures and Algorithms

Exemplar Guided Active Learning

no code implementations NeurIPS 2020 Jason Hartford, Kevin Leyton-Brown, Hadas Raviv, Dan Padnos, Shahar Lev, Barak Lenz

The challenge is that we are not informed which labels are common and which are rare, and the true label distribution may exhibit extreme skew.

Active Learning Word Sense Disambiguation

PMI-Masking: Principled masking of correlated spans

1 code implementation ICLR 2021 Yoav Levine, Barak Lenz, Opher Lieber, Omri Abend, Kevin Leyton-Brown, Moshe Tennenholtz, Yoav Shoham

Specifically, we show experimentally that PMI-Masking reaches the performance of prior masking approaches in half the training time, and consistently improves performance at the end of training.

Valid Causal Inference with (Some) Invalid Instruments

no code implementations19 Jun 2020 Jason Hartford, Victor Veitch, Dhanya Sridhar, Kevin Leyton-Brown

The technique is simple to apply and is "black-box" in the sense that it may be used with any instrumental variable estimator as long as the treatment effect is identified for each valid instrument independently.

Causal Inference valid

Learning under Invariable Bayesian Safety

no code implementations8 Jun 2020 Gal Bahar, Omer Ben-Porat, Kevin Leyton-Brown, Moshe Tennenholtz

A recent body of work addresses safety constraints in explore-and-exploit systems.

Smarter Parking: Using AI to Identify Parking Inefficiencies in Vancouver

no code implementations21 Mar 2020 Devon Graham, Satish Kumar Sarraf, Taylor Lundy, Ali MohammadMehr, Sara Uppal, Tae Yoon Lee, Hedayat Zarkoob, Scott Duke Kominers, Kevin Leyton-Brown

To see where this might be true in downtown Vancouver, we used artificial intelligence techniques to estimate the amount of time it would take drivers to both park on and off street for destinations throughout the city.

Fiduciary Bandits

no code implementations ICML 2020 Gal Bahar, Omer Ben-Porat, Kevin Leyton-Brown, Moshe Tennenholtz

Recommendation systems often face exploration-exploitation tradeoffs: the system can only learn about the desirability of new options by recommending them to some user.

Recommendation Systems

Procrastinating with Confidence: Near-Optimal, Anytime, Adaptive Algorithm Configuration

1 code implementation NeurIPS 2019 Robert Kleinberg, Kevin Leyton-Brown, Brendan Lucier, Devon Graham

Unfortunately, Structured Procrastination is not $\textit{adaptive}$ to characteristics of the parameterized algorithm: it treats every input like the worst case.

Deep Models of Interactions Across Sets

1 code implementation ICML 2018 Jason Hartford, Devon R Graham, Kevin Leyton-Brown, Siamak Ravanbakhsh

In experiments, our models achieved surprisingly good generalization performance on this matrix extrapolation task, both within domains (e. g., new users and new movies drawn from the same distribution used for training) and even across domains (e. g., predicting music ratings after training on movies).

Collaborative Filtering Matrix Completion +2

Deep IV: A Flexible Approach for Counterfactual Prediction

1 code implementation ICML 2017 Jason Hartford, Greg Lewis, Kevin Leyton-Brown, Matt Taddy

Counterfactual prediction requires understanding causal relationships between so-called treatment and outcome variables.

counterfactual

Deep Optimization for Spectrum Repacking

no code implementations11 Jun 2017 Neil Newman, Alexandre Fréchette, Kevin Leyton-Brown

A crucial element of the auction design was the construction of a solver, dubbed SATFC, that determined whether sets of stations could be "repacked" in this way; it needed to run every time a station was given a price quote.

Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates

no code implementations30 Mar 2017 Katharina Eggensperger, Marius Lindauer, Holger H. Hoos, Frank Hutter, Kevin Leyton-Brown

In our experiments, we construct and evaluate surrogate benchmarks for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems, drawing training data from the runs of existing AC procedures.

Benchmarking Hyperparameter Optimization

Counterfactual Prediction with Deep Instrumental Variables Networks

no code implementations30 Dec 2016 Jason Hartford, Greg Lewis, Kevin Leyton-Brown, Matt Taddy

We are in the middle of a remarkable rise in the use and capability of artificial intelligence.

counterfactual

Deep Learning for Predicting Human Strategic Behavior

no code implementations NeurIPS 2016 Jason S. Hartford, James R. Wright, Kevin Leyton-Brown

Predicting the behavior of human participants in strategic settings is an important problem in many domains.

ASlib: A Benchmark Library for Algorithm Selection

2 code implementations8 Jun 2015 Bernd Bischl, Pascal Kerschke, Lars Kotthoff, Marius Lindauer, Yuri Malitsky, Alexandre Frechette, Holger Hoos, Frank Hutter, Kevin Leyton-Brown, Kevin Tierney, Joaquin Vanschoren

To address this problem, we introduce a standardized format for representing algorithm selection scenarios and a repository that contains a growing number of data sets from the literature.

The Configurable SAT Solver Challenge (CSSC)

no code implementations5 May 2015 Frank Hutter, Marius Lindauer, Adrian Balint, Sam Bayless, Holger Hoos, Kevin Leyton-Brown

It is well known that different solution strategies work well for different types of instances of hard combinatorial problems.

Computational Analysis of Perfect-Information Position Auctions

no code implementations4 Aug 2014 David R. M Thompson, Kevin Leyton-Brown

After experimentation with other designs, the major search engines converged on the weighted, generalized second-price auction (wGSP) for selling keyword advertisements.

Position

Empirically Evaluating Multiagent Learning Algorithms

no code implementations31 Jan 2014 Erik Zawadzki, Asher Lipson, Kevin Leyton-Brown

Most such claims in the literature are based on small experiments, which has hampered understanding as well as the development of new multiagent learning (MAL) algorithms.

Q-Learning

ParamILS: An Automatic Algorithm Configuration Framework

no code implementations15 Jan 2014 Frank Hutter, Thomas Stuetzle, Kevin Leyton-Brown, Holger H. Hoos

The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms.

Hyperparameter Optimization

Bayesian Optimization With Censored Response Data

no code implementations7 Oct 2013 Frank Hutter, Holger Hoos, Kevin Leyton-Brown

Bayesian optimization (BO) aims to minimize a given blackbox function using a model that is updated whenever new evidence about the function becomes available.

Bayesian Optimization

Algorithm Runtime Prediction: Methods & Evaluation

no code implementations5 Nov 2012 Frank Hutter, Lin Xu, Holger H. Hoos, Kevin Leyton-Brown

We also comprehensively describe new and existing features for predicting algorithm runtime for propositional satisfiability (SAT), travelling salesperson (TSP) and mixed integer programming (MIP) problems.

Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms

1 code implementation18 Aug 2012 Chris Thornton, Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown

Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall.

Bayesian Optimization BIG-bench Machine Learning +3

Sequential Model-Based Optimization for General Algorithm Configuration

1 code implementation LION 2011 2011 Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown

State-of-the-art algorithms for hard computational problems often expose many parameters that can be modified to improve empirical performance.

Hyperparameter Optimization

Bayesian Action-Graph Games

no code implementations NeurIPS 2010 Albert X. Jiang, Kevin Leyton-Brown

Games of incomplete information, or Bayesian games, are an important game-theoretic model and have many applications in economics.

Cannot find the paper you are looking for? You can Submit a new open access paper.