Search Results for author:
Found 67 papers, 30 papers with code
Learning to Progressively Plan
For problem solving, making reactive decisions based on problem description is fast but inaccurate, while search-based planning using heuristics gives better solutions but could be exponentially slow.
Vulnerability Detection with Code Language Models: How Far Are We?
Evaluating code LMs on PrimeVul reveals that existing benchmarks significantly overestimate the performance of these models.
A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts
Current Large Language Models (LLMs) are not only limited to some maximum context length, but also are not able to robustly consume long inputs.
Premise Order Matters in Reasoning with Large Language Models
We first examine the effect of premise ordering on deductive reasoning on a variety of LLMs, and our evaluation shows that permuting the premise order can cause a performance drop of over 30%.
Transformers Can Achieve Length Generalization But Not Robustly
We show that the success of length generalization is intricately linked to the data format and the type of position encoding.
Self-Discover: Large Language Models Self-Compose Reasoning Structures
We introduce SELF-DISCOVER, a general framework for LLMs to self-discover the task-intrinsic reasoning structures to tackle complex reasoning problems that are challenging for typical prompting methods.
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
Chain of Code: Reasoning with a Language Model-Augmented Code Emulator
For example, consider prompting an LM to write code that counts the number of times it detects sarcasm in an essay: the LM may struggle to write an implementation for "detect_sarcasm(string)" that can be executed by the interpreter (handling the edge cases would be insurmountable).
Universal Self-Consistency for Large Language Model Generation
Self-consistency with chain-of-thought prompting (CoT) has demonstrated remarkable performance gains on various challenging tasks, by utilizing multiple reasoning paths sampled from large language models (LLMs).
Steady-State Analysis and Online Learning for Queues with Hawkes Arrivals
We investigate the long-run behavior of single-server queues with Hawkes arrivals and general service distributions and related optimization problems.
Large Language Models can Learn Rules
In the deduction stage, the LLM is then prompted to employ the learned rule library to perform reasoning to answer test questions.
Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
We present Step-Back Prompting, a simple prompting technique that enables LLMs to do abstractions to derive high-level concepts and first principles from instances containing specific details.
Large Language Models Cannot Self-Correct Reasoning Yet
Large Language Models (LLMs) have emerged as a groundbreaking technology with their unparalleled text generation capabilities across various applications.
Large Language Models as Analogical Reasoners
Chain-of-thought (CoT) prompting for language models demonstrates impressive performance across reasoning tasks, but typically needs labeled exemplars of the reasoning process.
Large Language Models as Optimizers
In this work, we propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as optimizers, where the optimization task is described in natural language.
BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual Pragmatic Knowledge
This is evident from the performance gain of GPT-3. 5/4 compared to the smaller models on the benchmark (50% vs up to ~25%).
Large Language Models as Tool Makers
Our approach consists of two phases: 1) tool making: an LLM acts as the tool maker that crafts tools for a set of tasks.
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.
Symbol tuning improves in-context learning in language models
We present symbol tuning - finetuning language models on in-context input-label pairs where natural language labels (e. g., "positive/negative sentiment") are replaced with arbitrary symbols (e. g., "foo/bar").
Teaching Large Language Models to Self-Debug
In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i. e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language.
Ranked #9 on
Code Generation
on MBPP
DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection
Combining our new dataset with previous datasets, we present an analysis of the challenges and promising research directions of using deep learning for detecting software vulnerabilities.
Larger language models do in-context learning differently
We next study semantically-unrelated label ICL (SUL-ICL), in which labels are semantically unrelated to their inputs (e. g., foo/bar instead of negative/positive), thereby forcing language models to learn the input-label mappings shown in in-context exemplars in order to perform the task.
Online Learning and Optimization for Queues with Unknown Demand Curve and Service Distribution
A major drawback of PTO is that its solution accuracy can often be highly sensitive to the parameter estimation errors because PTO is unable to properly link these errors (step 1) to the quality of the optimized solutions (step 2).
Large Language Models Can Be Easily Distracted by Irrelevant Context
We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included.
Data-pooling Reinforcement Learning for Personalized Healthcare Intervention
Motivated by the emerging needs of personalized preventative intervention in many healthcare applications, we consider a multi-stage, dynamic decision-making problem in the online setting with unknown model parameters.
Benchmarking Language Models for Code Syntax Understanding
Our key observation is that existing language models pretrained on code still lack the understanding of code syntax.
Scaling Instruction-Finetuned Language Models
We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation).
Ranked #1 on
Multi-task Language Understanding
on BBH-nlp
Compositional Semantic Parsing with Large Language Models
Humans can reason compositionally when presented with new tasks.
Ranked #1 on
Semantic Parsing
on CFQ
Measuring and Improving Compositional Generalization in Text-to-SQL via Component Alignment
To deal with this problem, we modify a number of state-of-the-art models to train on the segmented data of Spider-SS, and we show that this method improves the generalization performance.
Competition-Level Code Generation with AlphaCode
Programming is a powerful and ubiquitous problem-solving tool.
Ranked #1 on
Code Generation
on CodeContests
Learning Bounded Context-Free-Grammar via LSTM and the Transformer:Difference and Explanations
With the forced decomposition, we show that the performance upper bounds of LSTM and Transformer in learning CFL are close: both of them can simulate a stack and perform stack operation along with state transitions.
SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs
There are two important reasoning tasks on KGs: (1) single-hop knowledge graph completion, which involves predicting individual links in the KG; and (2), multi-hop reasoning, where the goal is to predict which KG entities satisfy a given logical query.
RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
Thus, we propose RobustART, the first comprehensive Robustness investigation benchmark on ImageNet regarding ARchitecture design (49 human-designed off-the-shelf architectures and 1200+ networks from neural architecture search) and Training techniques (10+ techniques, e. g., data augmentation) towards diverse noises (adversarial, natural, and system noises).
Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization
Recently, there has been significant progress in studying neural networks for translating text descriptions into SQL queries under the zero-shot cross-domain setting.
Natural SQL: Making SQL Easier to Infer from Natural Language Specifications
Addressing the mismatch between natural language descriptions and the corresponding SQL queries is a key challenge for text-to-SQL translation.
PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context
Creating effective visualization is an important part of data analytics.
Latent Execution for Neural Program Synthesis
While recent works demonstrated limited success on domain-specific languages (DSL), it remains highly challenging to apply them to real-world programming languages, such as C. Due to complicated syntax and token variation, there are three major challenges: (1) unlike many DSLs, programs in languages like C need to compile first and are not executed via interpreters; (2) the program search space grows exponentially when the syntax and semantics of the programming language become more complex; and (3) collecting a large-scale dataset of real-world programs is non-trivial.
SpreadsheetCoder: Formula Prediction from Semi-structured Context
In this work, we present the first approach for synthesizing spreadsheet formulas from tabular context, which includes both headers and semi-structured tabular data.
Understanding the Spread of COVID-19 Epidemic: A Spatio-Temporal Point Process View
In this paper, we model the propagation of the COVID-19 as spatio-temporal point processes and propose a generative and intensity-free model to track the spread of the disease.
Towards Robustness of Text-to-SQL Models against Synonym Substitution
We observe that the accuracy dramatically drops by eliminating such explicit correspondence between NL questions and table schemas, even if the synonyms are not adversarially selected to conduct worst-case adversarial attacks.
Latent Execution for Neural Program Synthesis Beyond Domain-Specific Languages
Program synthesis from input-output (IO) examples has been a long-standing challenge.
A High-fidelity, Machine-learning Enhanced Queueing Network Simulation Model for Hospital Ultrasound Operations
To address the issue, we propose a novel two-level routing component to the queueing network model.
Understanding Robustness in Teacher-Student Setting: A New Perspective
Extensive experiments show that student specialization correlates strongly with model robustness in different scenarios, including student trained via standard training, adversarial training, confidence-calibrated adversarial training, and training with robust feature dataset.
Perturbation Type Categorization for Multiple $\ell_p$ Bounded Adversarial Robustness
In addition, we demonstrate the realization of this trade-off in deep networks by adding random noise to the model input at test time, enabling enhanced robustness against strong adaptive attacks.
N-Bref : A High-fidelity Decompiler Exploiting Programming Structures
In software development, decompilation aims to reverse engineer binary executables.
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
As machine learning systems grow in scale, so do their training data requirements, forcing practitioners to automate and outsource the curation of training data in order to achieve state-of-the-art performance.
Towards Defending Multiple $\ell_p$-norm Bounded Adversarial Perturbations via Gated Batch Normalization
In this paper, we observe that different $\ell_p$ bounded adversarial perturbations induce different statistical properties that can be separated and characterized by the statistics of Batch Normalization (BN).
An online learning approach to dynamic pricing and capacity sizing in service systems
In this work we propose an online learning framework designed for solving this problem which does not require the system's scale to increase.
Compositional Generalization via Neural-Symbolic Stack Machines
Despite achieving tremendous success, existing deep learning models have exposed limitations in compositional generalization, the capability to learn compositional rules and apply them to unseen cases in a systematic manner.
Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis
The use of deep learning techniques has achieved significant progress for program synthesis from input-output examples.
Spatiotemporal Attacks for Embodied Agents
Adversarial attacks are valuable for providing insights into the blind-spots of deep learning models and help improve their robustness.
Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension
Integrating distributed representations with symbolic operations is essential for reading comprehension requiring complex reasoning, such as counting, sorting and arithmetics, but most existing approaches are hard to scale to more domains or more complex reasoning.
Ranked #5 on
Question Answering
on DROP Test
Deep Symbolic Superoptimization Without Human Knowledge
Deep symbolic superoptimization refers to the task of applying deep learning methods to simplify symbolic expressions.
Coda: An End-to-End Neural Program Decompiler
Furthermore, Coda outperforms the sequence-to-sequence model with attention by a margin of 70% program accuracy.
REFIT: A Unified Watermark Removal Framework For Deep Learning Systems With Limited Data
The experimental results demonstrate that our fine-tuning based watermark removal attacks could pose real threats to the copyright of pre-trained models, and thus highlight the importance of further investigating the watermarking problem and proposing more robust watermark embedding schemes against the attacks.
Infinite-horizon Off-Policy Policy Evaluation with Multiple Behavior Policies
We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies.
A Neural-based Program Decompiler
Reverse engineering of binary executables is a critical problem in the computer security domain.
Execution-Guided Neural Program Synthesis
Most existing neural program synthesis approaches employ an encoder-decoder architecture, which uses an encoder to compute the embedding of the given input-output examples, as well as a decoder to generate the program from the embedding following a given syntax.
Learning to Perform Local Rewriting for Combinatorial Optimization
Search-based methods for hard combinatorial optimization are often guided by heuristics.
Tree-to-tree Neural Networks for Program Translation
We observe that program translation is a modular procedure, in which a sub-tree of the source tree is translated into the corresponding target sub-tree at each step.
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
In this work, we consider a new type of attacks, called backdoor attacks, where the attacker's goal is to create a backdoor into a learning-based authentication system, so that he can easily circumvent the system by leveraging the backdoor.
Fooling Vision and Language Models Despite Localization and Attention Mechanism
Our work sheds new light on understanding adversarial attacks on vision systems which have a language component and shows that attention, bounding box localization, and compositional internal structures are vulnerable to adversarial attacks.
Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong
We ask whether a strong defense can be created by combining multiple (possibly weak) defenses.
Towards Synthesizing Complex Programs from Input-Output Examples
In our evaluation, we show that using our novel approach, neural parsing programs can be learned to achieve 100% test accuracy on test inputs that are 500x longer than the training samples.
Delving into Transferable Adversarial Examples and Black-box Attacks
In this work, we are the first to conduct an extensive study of the transferability over large models and a large scale dataset, and we are also the first to study the transferability of targeted adversarial examples with their target labels.
Latent Attention For If-Then Program Synthesis
Automatic translation from natural language descriptions into programs is a longstanding challenging problem.
A General Retraining Framework for Scalable Adversarial Classification
We propose the first systematic and general-purpose retraining framework which can: a) boost robustness of an \emph{arbitrary} learning algorithm, in the face of b) a broader class of adversarial models than any prior methods.
