EffiBench: Benchmarking the Efficiency of Automatically Generated Code

1 code implementation • 3 Feb 2024 • Dong Huang, Jie M. Zhang, Yuhao QING, Heming Cui

This paper presents EffiBench, a benchmark with 1, 000 efficiency-critical coding problems for assessing the efficiency of code generated by code generation models.

Benchmarking Code Completion +1

AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation

1 code implementation • 20 Dec 2023 • Dong Huang, Qingwen Bu, Jie M. Zhang, Michael Luck, Heming Cui

The advancement of natural language processing (NLP) has been significantly boosted by the development of transformer-based large language models (LLMs).

Code Generation Prompt Engineering

Bias Testing and Mitigation in LLM-based Code Generation

no code implementations • 3 Sep 2023 • Dong Huang, Qingwen Bu, Jie Zhang, Xiaofei Xie, Junjie Chen, Heming Cui

To mitigate bias for code generation models, we evaluate five bias mitigation prompt strategies, i. e., utilizing bias testing results to refine the code (zero-shot), one-, few-shot, and two Chain-of-Thought (CoT) prompts.

Code Generation Fairness +1

CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generation

no code implementations • 17 Aug 2023 • Dong Huang, Qingwen Bu, Yuhao QING, Heming Cui

However, its application in code generation faces a distinct challenge, i. e., although the code generated with CoT reasoning is logically correct, it faces the problem of syntax error (e. g., invalid syntax error report) during code execution, which causes the CoT result's pass@1 in HumanEval even lower than the zero-shot result.

Code Generation Few-Shot Learning +1

Feature Map Testing for Deep Neural Networks

1 code implementation • 21 Jul 2023 • Dong Huang, Qingwen Bu, Yahao Qing, Yichao Fu, Heming Cui

Current test metrics, however, are primarily concerned with the neurons, which means that test cases that are discovered either by guided fuzzing or selection with these metrics focus on detecting fault-inducing neurons while failing to detect fault-inducing feature maps.

Fault Detection

Neuron Sensitivity Guided Test Case Selection for Deep Learning Testing

no code implementations • 20 Jul 2023 • Dong Huang, Qingwen Bu, Yichao Fu, Yuhao QING, Bocheng Xiao, Heming Cui

To address the above-mentioned problem, we propose NSS, Neuron Sensitivity guided test case Selection, which can reduce the labeling time by selecting valuable test cases from unlabeled datasets.

Autonomous Driving Fault Detection +1

Towards Building More Robust Models with Frequency Bias

no code implementations • ICCV 2023 • Qingwen Bu, Dong Huang, Heming Cui

The vulnerability of deep neural networks to adversarial samples has been a major impediment to their broad applications, despite their success in various fields.

Two Heads are Better than One: Robust Learning Meets Multi-branch Models

1 code implementation • 17 Aug 2022 • Dong Huang, Qingwen Bu, Yuhao QING, Haowen Pi, Sen Wang, Heming Cui

Compared to all methods that do not use additional data for training, our models achieve 67. 3% and 41. 5% robust accuracy on CIFAR-10 and CIFAR-100 (improving upon the state-of-the-art by +7. 23% and +9. 07%).

Adversarial Robustness Philosophy

Neural Architecture Search via Ensemble-based Knowledge Distillation

no code implementations • 29 Sep 2021 • Fanxin Li, Shixiong Zhao, Haowen Pi, Yuhao QING, Yichao Fu, Sen Wang, Heming Cui

Neural Architecture Search (NAS) automatically searches for well-performed network architectures from a given search space.

Knowledge Distillation Neural Architecture Search