Search Results for author:
Found 65 papers, 41 papers with code
Introducing v0.5 of the AI Safety Benchmark from MLCommons
We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0. 5 benchmark.
Machine Learning Robustness: A Primer
This chapter explores the foundational concept of robustness in Machine Learning (ML) and its integral role in establishing trustworthiness in Artificial Intelligence (AI) systems.
Bugs in Large Language Models Generated Code: An Empirical Study
The bug patterns are presented in the form of a taxonomy.
Trained Without My Consent: Detecting Code Inclusion In Language Models Trained on Code
Therefore, auditing code developed using LLMs is challenging, as it is difficult to reliably assert if an LLM used during development has been trained on specific copyrighted codes, given that we do not have access to the training datasets of these models.
ChatGPT vs LLaMA: Impact, Reliability, and Challenges in Stack Overflow Discussions
Since its release in November 2022, ChatGPT has shaken up Stack Overflow, the premier platform for developers' queries on programming and software development.
Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends
Based on our qualitative analysis, we present a taxonomy of the challenges and benefits associated with PTM reuse within this community.
Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Study
Conclusions: Our research addresses the critical issue of deep learning bug reproducibility.
Refining GPT-3 Embeddings with a Siamese Structure for Technical Post Duplicate Detection
When applied to the dataset we constructed with a recent Stack Overflow dump, our approach attains a Top-1, Top-5, and Top-30 accuracy of 23. 1%, 43. 9%, and 68. 9%, respectively.
Characterizing and Classifying Developer Forum Posts with their Intentions
The modeling of the intentions of posts can provide an extra dimension to the current tag taxonomy.
Studying the Practices of Testing Machine Learning Software in the Wild
Aims: To fill this gap, we perform the first fine-grained empirical study on ML testing practices in the wild, to identify the ML properties being tested, the followed testing strategies, and their implementation throughout the ML workflow.
GIST: Generated Inputs Sets Transferability in Deep Learning
As the demand for verifiability and testability of neural networks continues to rise, an increasing number of methods for generating test sets are being developed.
Detection and Evaluation of bias-inducing Features in Machine learning
In this study, we propose an approach for systematically identifying all bias-inducing features of a model to help support the decision-making of domain experts.
Data Cleaning and Machine Learning: A Systematic Literature Review
First, it aims to summarize the latest approaches for data cleaning for ML and ML for data cleaning.
Reinforcement learning informed evolutionary search for autonomous systems testing
Evolutionary search-based techniques are commonly used for testing autonomous robotic systems.
An Intentional Forgetting-Driven Self-Healing Method For Deep Reinforcement Learning Systems
Dr. DRL successfully helps agents to adapt to 19. 63% of drifted environments left unsolved by vanilla CL while maintaining and even enhancing by up to 45% the obtained rewards for drifted environments that are resolved by both approaches.
Deploying Deep Reinforcement Learning Systems: A Taxonomy of Challenges
In this paper, we propose an empirical study on Stack Overflow (SO), the most popular Q&A forum for developers, to uncover and understand the challenges practitioners faced when deploying DRL systems.
On the Effectiveness of Log Representation for Log-based Anomaly Detection
We believe our comprehensive comparison of log representation techniques can help researchers and practitioners better understand the characteristics of different log representation techniques and provide them with guidance for selecting the most suitable ones for their ML-based log analysis workflow.
Bug Characterization in Machine Learning-based Systems
Based on our results, fixing ML bugs are more costly and ML components are more error-prone, compared to non-ML bugs and non-ML components respectively.
An Empirical Study on Bugs Inside PyTorch: A Replication Study
Software systems are increasingly relying on deep learning components, due to their remarkable capability of identifying complex data patterns and powering intelligent behaviour.
Quality Issues in Machine Learning Software Systems
We validate the identified quality issues via a survey with ML practitioners.
Responsible Design Patterns for Machine Learning Pipelines
In this paper, we propose a comprehensive framework incorporating RDPs into ML pipelines to mitigate risks and ensure the ethical development of AI systems.
Leveraging Data Mining Algorithms to Recommend Source Code Changes
Objectives: This paper proposes an automatic method for recommending source code changes using four data mining algorithms.
On Codex Prompt Engineering for OCL Generation: An Empirical Study
We investigate the reliability of OCL constraints generated by Codex from natural language specifications.
Mutation Testing of Deep Reinforcement Learning Based on Real Faults
This allows us to compare different mutation killing definitions based on existing approaches, as well as to analyze the behavior of the obtained mutation operators and their potential combinations called Higher Order Mutation(s) (HOM).
AmbieGen: A Search-based Framework for Autonomous Systems Testing
To address this challenge, we introduce AmbieGen, a search-based test case generation framework for autonomous systems.
Can Ensembling Pre-processing Algorithms Lead to Better Machine Learning Fairness?
As machine learning (ML) systems get adopted in more critical areas, it has become increasingly crucial to address the bias that could occur in these systems.
An Empirical Study of Library Usage and Dependency in Deep Learning Frameworks
In this paper, we present a qualitative and quantitative analysis of the most frequent dl libraries combination, the distribution of dl library dependencies across the ml workflow, and formulate a set of recommendations to (i) hardware builders for more optimized accelerators and (ii) library builder for more refined future releases.
Reliable Malware Analysis and Detection using Topology Data Analysis
Next, we compare the different TDA techniques (i. e., persistence homology, tomato, TDA Mapper) and existing techniques (i. e., PCA, UMAP, t-SNE) using different classifiers including random forest, decision tree, xgboost, and lightgbm.
SmOOD: Smoothness-based Out-of-Distribution Detection Approach for Surrogate Neural Networks in Aircraft Design
Hybrid surrogate optimization maintains high results quality while providing rapid design assessments when both the surrogate model and the switch mechanism for eventually transitioning to the HF model are calibrated properly.
Physics-Guided Adversarial Machine Learning for Aircraft Systems Simulation
In the context of aircraft system performance assessment, deep learning technologies allow to quickly infer models from experimental measurements, with less detailed system knowledge than usually required by physics-based modeling.
An Empirical Study on the Usage of Automated Machine Learning Tools
Machine learning (ML) practitioners use AutoML tools to automate and optimize the process of feature engineering, model training, and hyperparameter optimization and so on.
A Comparison of Reinforcement Learning Frameworks for Software Testing Tasks
In this paper, we empirically investigate the applications of carefully selected DRL algorithms on two important software testing tasks: test case prioritization in the context of Continuous Integration (CI) and game testing.
Quality issues in Machine Learning Software Systems
This empirical study aims to identify a catalog of bad-practices related to poor quality in MLSSs.
A Probabilistic Framework for Mutation Testing in Deep Neural Networks
Methods: In this work, we propose a Probabilistic Mutation Testing (PMT) approach that alleviates the inconsistency problem and allows for a more consistent decision on whether a mutant is killed or not.
DiverGet: A Search-Based Software Testing Approach for Deep Neural Network Quantization Assessment
Quantization is one of the most applied Deep Neural Network (DNN) compression strategies, when deploying a trained DNN model on an embedded system or a cell phone.
Dev2vec: Representing Domain Expertise of Developers in an Embedding Space
Moreover, our findings suggest that ``issue resolving history'' of developers is the most informative source of information to represent the domain expertise of developers in embedding spaces.
Threat Assessment in Machine Learning based Systems
Attacks from the AI Incident Database and the literature are used to identify vulnerabilities and new types of threats that were not documented in ATLAS.
GitHub Copilot AI pair programmer: Asset or Liability?
In this paper, we study the capabilities of Copilot in two different programming tasks: (i) generating (and reproducing) correct and efficient solutions for fundamental algorithmic problems, and (ii) comparing Copilot's proposed solutions with those of human programmers on a set of programming tasks.
An Empirical Study of Challenges in Converting Deep Learning Models
Usually DL models are developed and trained using DL frameworks that have their own internal mechanisms/formats to represent and train DL models, and usually those formats cannot be recognized by other frameworks.
Bugs in Machine Learning-based Systems: A Faultload Benchmark
Although most of these tools use bugs' lifecycle, there is no standard benchmark of bugs to assess their performance, compare them and discuss their advantages and weaknesses.
Never trust, always verify : a roadmap for Trustworthy AI?
In this paper, we examine trust in the context of AI-based systems to understand what it means for an AI system to be trustworthy and identify actions that need to be undertaken to ensure that AI systems are trustworthy.
Studying the Practices of Deploying Machine Learning Projects on Docker
Studies have recently explored the use of Docker for deploying general software projects with no specific focus on how Docker is used to deploy ML-based projects.
Fool SHAP with Stealthily Biased Sampling
SHAP explanations aim at identifying which features contribute the most to the difference in model prediction at a specific input versus a background distribution.
The Different Faces of AI Ethics Across the World: A Principle-Implementation Gap Analysis
Next, we analyze the current level of AI readiness and current implementations of ethical AI principles in different countries, to identify gaps in the implementation of AI principles and their causes.
Testing Feedforward Neural Networks Training Programs
All these model training steps can be error-prone.
A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems
We compared three configurations of AmbieGen: based on a single objective genetic algorithm, multi objective, and random search.
Machine Learning Application Development: Practitioners' Insights
In this paper, we report about a survey that aimed to understand the challenges and best practices of ML application development.
Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow
This paper presents the first empirical study of Keras and TensorFlow silent bugs, and their impact on users' programs.
On Assessing The Safety of Reinforcement Learning algorithms Using Formal Methods
The increasing adoption of Reinforcement Learning in safety-critical systems domains such as autonomous vehicles, health, and aviation raises the need for ensuring their safety.
An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Datasets
We report the results of an empirical study that we conducted to determine the feasibility of developing an ensemble engine by combining the polarity labels of stand-alone SE-specific sentiment detectors.
Partial Order in Chaos: Consensus on Feature Attributions in the Rashomon Set
In this work, instead of aiming at reducing the under-specification of model explanations, we fully embrace it and extract logical statements about feature attributions that are consistent across all models with good empirical performance (i. e. all models in the Rashomon Set).
The challenge of reproducible ML: an empirical study on the impact of bugs
This study attempts to quantify the impact that the occurrence of bugs in a popular ML framework, PyTorch, has on the performance of trained models.
The Forgotten Role of Search Queries in IR-based Bug Localization: An Empirical Study
We confirmed that the state-of-the-art query construction approaches are indeed not sufficient for constructing appropriate queries (for bug localization) from certain natural language-only bug reports although they contain such queries.
Models of Computational Profiles to Study the Likelihood of DNN Metamorphic Test Cases
We investigate the distribution of computational profile likelihood of metamorphic test cases with respect to the likelihood distributions of training, test and error control cases.
How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
Method: We conduct a Systematic Literature Review (SLR) of research papers published between 2015 to 2020, covering topics related to the certification of ML systems.
HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks
Deep Neural Networks (DNN) applications are increasingly becoming a part of our everyday life, from medical applications to autonomous cars.
Data Driven Testing of Cyber Physical Systems
The most common approach for pre-deployment testing is to model the system and run simulations with models or software in the loop.
Mining API Usage Scenarios from Stack Overflow
Each task consists of a code example, the task description, and the reactions of developers towards the code example.
Software Engineering
Faults in Deep Reinforcement Learning Programs: A Taxonomy and A Detection Approach
To allow for the automatic detection of faults in DRL programs, we have defined a meta-model of DRL programs and developed DRLinter, a model-based fault detection approach that leverages static analysis and graph transformations.
SIGMA : Strengthening IDS with GAN and Metaheuristics Attacks
More specifically, we propose a new method named SIGMA, that leverages adversarial examples to strengthen IDS against new types of attacks.
Studying Software Engineering Patterns for Designing Machine Learning Systems
Machine-learning (ML) techniques have become popular in the recent years.
DeepEvolution: A Search-Based Testing Approach for Deep Neural Networks
To overcome these limitations, we propose, DeepEvolution, a novel search-based approach for testing DL models that relies on metaheuristics to ensure a maximum diversity in generated test cases.
TFCheck : A TensorFlow Library for Detecting Training Issues in Neural Network Programs
In this paper, we examine training issues in ML programs and propose a catalog of verification routines that can be used to detect the identified issues, automatically.
Machine Learning Software Engineering in Practice: An Industrial Case Study
Especially, retail customers of SAP deal with millions of sales transactions for their day-to-day business.
A Machine-learning Based Ensemble Method For Anti-patterns Detection
In this paper, we present SMAD (SMart Aggregation of Anti-patterns Detectors), a machine-learning based ensemble method to aggregate various anti-patterns detection approaches on the basis of their internal detection rules.
