Fine-tuning language models to find agreement among humans with diverse preferences

no code implementations • 28 Nov 2022 • Michiel A. Bakker, Martin J. Chadwick, Hannah R. Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew M. Botvinick, Christopher Summerfield

Recent work in large language modeling (LLMs) has used fine-tuning to align outputs with the preferences of a prototypical user.

Language Modelling

Improving alignment of dialogue agents via targeted human judgements

no code implementations • 28 Sep 2022 • Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soňa Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, Geoffrey Irving

We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines.

Language Modelling

Teaching language models to support answers with verified quotes

no code implementations • 21 Mar 2022 • Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, Nat McAleese

We measure the performance of GopherCite by conducting human evaluation of answers to questions in a subset of the NaturalQuestions and ELI5 datasets.

Fact Checking Natural Questions

HCMD-zero: Learning Value Aligned Mechanisms from Data

no code implementations • 21 Feb 2022 • Jan Balaguer, Raphael Koster, Ari Weinstein, Lucy Campbell-Gillingham, Christopher Summerfield, Matthew Botvinick, Andrea Tacchetti

Our analysis shows HCMD-zero consistently makes the mechanism policy more and more likely to be preferred by human participants over the course of training, and that it results in a mechanism with an interpretable and intuitive policy.

Human-centered mechanism design with Democratic AI

no code implementations • 27 Jan 2022 • Raphael Koster, Jan Balaguer, Andrea Tacchetti, Ari Weinstein, Tina Zhu, Oliver Hauser, Duncan Williams, Lucy Campbell-Gillingham, Phoebe Thacker, Matthew Botvinick, Christopher Summerfield

Building artificial intelligence (AI) that aligns with human values is an unsolved problem.

reinforcement-learning Reinforcement Learning (RL)