Search Results for author: Dawn Drain

Found 22 papers, 9 papers with code

In-context Learning and Induction Heads

no code implementations24 Sep 2022 Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah

In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all "in-context learning" in large transformer models (i. e. decreasing loss at increasing token indices).

In-Context Learning

Toy Models of Superposition

1 code implementation21 Sep 2022 Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, Christopher Olah

Neural networks often pack many unrelated concepts into a single neuron - a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging.

Exploring and Evaluating Personalized Models for Code Generation

no code implementations29 Aug 2022 Andrei Zlotchevski, Dawn Drain, Alexey Svyatkovskiy, Colin Clement, Neel Sundaresan, Michele Tufano

Large Transformer models achieved the state-of-the-art status for Natural Language Understanding tasks and are increasingly becoming the baseline model architecture for modeling source code.

Code Generation Natural Language Understanding +1

Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy

no code implementations EMNLP 2021 Colin B. Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, Alexey Svyatkovskiy

While there are many efforts to extend the context window, we introduce an architecture-independent approach for leveraging the syntactic hierarchies of source code for incorporating entire file-level context into a fixed-length window.

Code Completion Code Generation +3

Distilling Transformers for Neural Cross-Domain Search

no code implementations6 Aug 2021 Colin B. Clement, Chen Wu, Dawn Drain, Neel Sundaresan

Pre-trained transformers have recently clinched top spots in the gamut of natural language tasks and pioneered solutions to software engineering tasks.

Code Search Data Augmentation +3

DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons

no code implementations19 May 2021 Dawn Drain, Colin B. Clement, Guillermo Serrato, Neel Sundaresan

The joint task of bug localization and program repair is an integral part of the software development process.

Program Repair

Generating Bug-Fixes Using Pretrained Transformers

no code implementations16 Apr 2021 Dawn Drain, Chen Wu, Alexey Svyatkovskiy, Neel Sundaresan

In this work we introduce DeepDebug: a data-driven program repair approach which learns to detect and fix bugs in Java methods mined from real-world GitHub repositories.

Denoising Program Repair

Generating Code with the Help of Retrieved Template Functions and Stack Overflow Answers

no code implementations12 Apr 2021 Dawn Drain, Changran Hu, Chen Wu, Mikhail Breslav, Neel Sundaresan

To demonstrate the effectiveness of our model designs, we perform extensive experiments with CodeSearchNet which contains template functions and CoNaLa which contains Stack Overflow intent-snippet pairs.

Code Search Retrieval

PyMT5: multi-mode translation of natural language and Python code with transformers

no code implementations EMNLP 2020 Colin B. Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, Neel Sundaresan

Simultaneously modeling source code and natural language has many exciting applications in automated software development and understanding.

Translation

GraphCodeBERT: Pre-training Code Representations with Data Flow

1 code implementation ICLR 2021 Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, Ming Zhou

Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables.

Clone Detection Code Completion +7

Unit Test Case Generation with Transformers and Focal Context

1 code implementation11 Sep 2020 Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, Neel Sundaresan

We execute the test cases, collect test coverage information, and compare them with test cases generated by EvoSuite and GPT-3, finding that our approach outperforms GPT-3 and has comparable coverage w. r. t.

Denoising

Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers

no code implementations11 Sep 2020 Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Neel Sundaresan

In this paper we present an approach to support developers in writing unit test cases by generating accurate and useful assert statements.

Cannot find the paper you are looking for? You can Submit a new open access paper.