1 | Annibale Panichella

Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Tasks

Tue, 18 Aug 2026 00:00:00 +0200

Test Case Selection for Deep Neural Networks: A Replication Study on LLMs for Code (Replicability Study)

Sat, 20 Jun 2026 00:00:00 +0200

A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair

Fri, 15 May 2026 00:00:00 +0200

Agentic Based Python Dependency Resolution

Wed, 15 Apr 2026 00:00:00 +0200

Observability and Fault Injection for LLM-Based Multi-Agent Systems in Software Engineering

Sun, 15 Mar 2026 00:00:01 +0100

Real-World Fault Detection for C-Extended Python Projects with Automated Unit Test Generation

Sun, 15 Mar 2026 00:00:00 +0100

BreachT5: Ensembling CodeT5+ Models for Multi-Label Vulnerability Detection in Smart Contracts

Mon, 02 Feb 2026 09:16:23 +0200

Efficient Exploration of Autonomous Driving System Safety Boundaries

Mon, 02 Feb 2026 09:16:23 +0200

The Last Dependency Crusade: Solving Python Dependency Conflicts with LLMs

Tue, 09 Sep 2025 09:16:23 +0200

The Pursuit of Diversity: Multi-Objective Testing of Deep Reinforcement Learning Agents

Tue, 09 Sep 2025 09:16:23 +0200

The Future of Software Transparency: Bridging Understanding, Measurement, and Practice

Fri, 08 Aug 2025 14:29:50 +0200

Abstract

Although the study of software transparency has deep roots in software engineering, a shared definition and practical application in real-world development contexts remain elusive. Through an in-depth analysis of the academic and industrial landscape, this article provides an overview of the current state of knowledge on software transparency, outlining a path to a deeper understanding of the subject for both developers and researchers. The challenge of software transparency involves not only establishing a formal, widely accepted understanding within the community, but also measuring and quantifying it in production environments. To this end, we survey academics and developers to evaluate an innovative approach to defining transparency and present a vision of a new framework for its quantification.

Automated Test-Case Generation for REST APIs Using Model Inference Search Heuristic

Mon, 05 May 2025 14:29:50 +0200

Abstract

The rising popularity of the microservice architectural style has led to a growing demand for automated testing approaches tailored to these systems. EvoMaster is a state-of-the-art tool that uses Evolutionary Algorithms (EAs) to automatically generate test cases for microservices’ REST APIs. One limitation of these EAs is the use of unit-level search heuristics, such as branch distances, which focus on fine-grained code coverage and may not effectively capture the complex, interconnected behaviors characteristic of system-level testing. To address this limitation, we propose a new search heuristic (MISH) that uses real-time automaton learning to guide the test case generation process. We capture the sequential call patterns exhibited by a test case by learning an automaton from the stream of log events outputted by different microservices within the same system. Therefore, MISH learns a representation of the systemwide behavior, allowing us to define the fitness of a test case based on the path it traverses within the inferred automaton. We empirically evaluate MISH’s effectiveness on six real-world benchmark microservice applications and compare it against a state-of-the-art technique, MOSA, for testing REST APIs. Our evaluation shows promising results for using MISH to guide the automated test case generation within EvoMaster

Metamorphic-Based Many-Objective Distillation of LLMs for Code-related Tasks

Sun, 19 Jan 2025 09:32:39 +0100

DRVN at the ICST 2025 Tool Competition – Self-Driving Car Testing Track

Wed, 01 Jan 2025 00:03:02 +0100

Abstract

DRVN is a regression testing tool that aims to diversify the test scenarios (road maps) to execute for testing and validating self-driving cars. DRVN harnesses the power of convolutional neural networks to identify possible failing roads in a set of generated examples before applying a greedy algorithm that selects and prioritizes the most diverse roads during regression testing. Initial testing discovered that DRVN performed well against random-based test selection.

Rocket: A System-Level Fuzz-Testing Framework for the XRPL Consensus Algorithm

Wed, 01 Jan 2025 00:03:02 +0100

Abstract

Byzantine fault tolerant algorithms are critical for achieving consistency and reliability in distributed systems, especially in the presence of faults or adversarial behavior. The consensus algorithm used by the XRP Ledger falls into this category. In practice, the implementation of these algorithms is prone to errors, which can lead to undesired behavior in the system. This paper introduces Rocket, a fuzz-testing framework designed for the XRPL consensus algorithm. Rocket enables researchers and developers to automatically inject network and process faults into a locally simulated network of XRPL validator nodes to test if the system behaves as expected. This technique has previously been shown to be effective in finding implementation errors. Rocket has been designed to focus on extensibility and ease of use, enabling users to run complex test scenarios with minimal setup. Video: https://www.youtube.com/watch?v=07Z3ufRa51Y

Suspicious Types and Bad Neighborhoods - Filtering Spectra with Compiler Information

Wed, 01 Jan 2025 00:03:02 +0100

Test Wars: A Comparative Study of SBST, Symbolic Execution, and LLM-Based Approaches to Unit Test Generation

Wed, 01 Jan 2025 00:03:02 +0200

Danger is My Middle Lane: Simulations from Real-World Dangerous Roads

Thu, 06 Jun 2024 09:16:23 +0200

Higher Fault Detection Through Novel Density Estimators in Unit Test Generation

Thu, 06 Jun 2024 09:16:23 +0200

Evolutionary Generative Fuzzing for Differential Testing of the Kotlin Compiler

Thu, 04 Apr 2024 17:56:40 +0200

Syntest-JavaScript: Automated Unit-Level Test Case Generation for JavaScript

Fri, 02 Feb 2024 17:56:40 +0200

CATMA: Conformance Analysis Tool For Microservice Applications

Mon, 01 Jan 2024 19:37:11 +0200

Abstract:

The microservice architecture allows developers to divide the core functionality of their software system into multiple smaller services. However, this architectural style also makes it harder for them to debug and assess whether the system’s deployment conforms to its implementation. We present CATMA, an automated tool that detects non-conformances between the system’s deployment and implementation. It automatically visualizes and generates potential interpretations for the detected discrepancies. Our evaluation of CATMA shows promising results in terms of performance and providing useful insights. CATMA is available at https://cyber-analytics.nl/catma.github.io/, and a demonstration video is available at https://youtu.be/WKP1hG-TDKc.

TestSpark: IntelliJ IDEA’s Ultimate Test Generation Companion

Mon, 01 Jan 2024 19:37:11 +0200

Abstract:

Writing software tests is laborious and time-consuming. To address this, prior studies introduced various automated test-generation techniques. A well-explored research direction in this field is unit test generation, wherein artificial intelligence (AI) techniques create tests for a method/class under test. While many of these techniques have primarily found applications in a research context, existing tools (e.g., EvoSuite, Randoop, and AthenaTest) are not user-friendly and are tailored to a single technique. This paper introduces Test- Spark, a plugin for IntelliJ IDEA that enables users to generate unit tests with only a few clicks directly within their Integrated De- velopment Environment (IDE). Furthermore, TestSpark also allows users to easily modify and run each generated test and integrate them into the project workflow. TestSpark leverages the advances of search-based test generation tools, and it introduces a technique to generate unit tests using Large Language Models (LLMs) by creating a feedback cycle between the IDE and the LLM. Since TestSpark is an open-source (https://github.com/JetBrains-Research/TestSpark), extendable, and well-documented tool, it is possible to add new test generation methods into the plugin with the minimum effort. This paper also explains our future studies related to TestSpark and our preliminary results.

Breaking the Silence: the Threats of Using LLMs in Software Engineering

Mon, 01 Jan 2024 14:29:50 +0200

Abstract

Large Language Models (LLMs) have gained considerable traction within the Software Engineering (SE) community, impacting various SE tasks from code completion to test generation, from program repair to code summarization. Despite their promise, researchers must still be careful as numerous intricate factors can influence the outcomes of experiments involving LLMs. This paper initiates an open discussion on potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings. In response, this paper proposes a set of guidelines tailored for SE researchers and Language Model (LM) providers to mitigate these concerns. The implications of the guidelines are illustrated using existing good practices followed by LLM providers and a practical example for SE researchers in the context of test case generation.

Multi-objective Black-Box Test Case Prioritization Based on Wordnet Distances

Wed, 18 Oct 2023 09:16:23 +0200

Abstract

Test case prioritization techniques have emerged as effective strategies to optimize this process and mitigate the regression testing costs. Commonly, black-box heuristics guide optimal test ordering, leveraging information retrieval (e.g., cosine distance) to measure the test case distance and sort them accordingly. However, a challenge arises when dealing with tests of varying granularity levels, as they may employ distinct vocabularies (e.g., name identifiers). In this paper, we propose to measure the distance between test cases based on the shortest path between their identifiers within the WordNet lexical database. This additional heuristic is combined with the traditional cosine distance to prioritize test cases in a multi-objective fashion. Our preliminary study conducted with two different Java projects shows that test cases prioritized with WordNet achieve larger fault detection capability (APFD) compared to the traditional cosine distance used in the literature.

A Fast Multi-objective Evolutionary Approach for Designing Large-Scale Optical Mode Sorter

Tue, 25 Apr 2023 10:56:44 +0200

Searching for Quality: Genetic Algorithms and Metamorphic Testing for Software Engineering ML

Tue, 25 Apr 2023 10:56:44 +0200

Evolutionary Approach for Concurrency Testing of Ripple Blockchain Consensus Algorithm

Sun, 01 Jan 2023 22:27:51 +0200

HasBugs - Handpicked Haskell Bugs

Sun, 01 Jan 2023 17:08:37 +0100

Grammar-Based Evolutionary Fuzzing for JSON-RPC APIs

Sun, 01 Jan 2023 17:56:40 +0200

On the Strengths of Pure Evolutionary Algorithms in Generating Adversarial Examples

Sun, 01 Jan 2023 17:56:40 +0200

Guess What: Test Case Generation for Javascript with Unsupervised Probabilistic Type Inference

Thu, 18 Aug 2022 09:16:23 +0200

Abstract

Search-based test case generation approaches make use of static type information to determine which data types should be used for the creation of new test cases. Dynamically typed languages like JavaScript, however, do not have this type information. In this paper, we propose an unsupervised probabilistic type inference approach to infer data types within the test case generation process. We evaluated the proposed approach on a benchmark of 98~units under test (i.e., exported classes and functions) compared to random type sampling w.r.t. branch coverage. Our results show that our type inference approach achieves a statistically significant increase in 56% of the test files with up to 71% of branch coverage compared to the baseline.

Guiding Automated Test Case Generation for Transaction-Reverting Statements in Smart Contracts

Sat, 11 Jun 2022 18:43:48 +0200

An Improved Pareto Front Modeling Algorithm for Large-scale Many-Objective Optimization

Sun, 27 Mar 2022 10:56:44 +0200

SynTest-Solidity: Automated Test Case Generation and Fuzzing for Smart Contracts

Tue, 18 Jan 2022 14:29:50 +0200

PropR: Property-Based Automatic Program Repair

Sat, 01 Jan 2022 09:32:39 +0100

Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations

Sat, 07 Aug 2021 22:27:51 +0200

Improving Test Case Generation for REST APIs Through Hierarchical Clustering

Wed, 07 Jul 2021 22:27:51 +0200

Hybrid Multi-level Crossover for Unit Test Case Generation

Tue, 29 Jun 2021 09:16:23 +0200

Abstract

State-of-the-art search-based approaches for test case generation work at test case level, where tests are represented as sequences of statements. These approaches make use of genetic operators (i.e., mutation and crossover) that create test variants by adding, altering, and removing statements from existing tests. While this encoding schema has been shown to be very effective for many-objective test case generation, the standard crossover operator (single-point) only alters the structure of the test cases but not the input data. In this paper, we argue that changing both the test case structure and the input data is necessary to increase the genetic variation and improve the search process. Hence, we propose a hybrid multi-level crossover (HMX) operator that combines the traditional test-level crossover with data-level recombination. The former evolves and alters the test case structures, while the latter evolves the input data using numeric and string-based recombinational operators. We evaluate our new crossover operator by performing an empirical study on more than 100 classes selected from open-source Java libraries for numerical operations and string manipulation. We compare HMX with the single-point crossover that is used in EvoSuite w.r.t structural coverage and fault detection capability. Our results show that HMX achieves a statistically significant increase in 30% of the classes up to 19% in structural coverage compared to the single-point crossover. Moreover, the fault detection capability improved up to 12% measured using strong mutation score.

Multi-objective Test Case Selection Through Linkage Learning-driven Crossover

Tue, 29 Jun 2021 09:16:23 +0200

Abstract

Test case selection (TCS) aims to select a subset of the test suite to run for regression testing. The selection is typically based on past coverage and execution cost data. Researchers have successfully used multi-objective evolutionary algorithms (MOEAs), such as NSGA-II and its variants, to solve the problem. These MOEAs use traditional crossovers to create new candidate solutions during the search. Recent studies in evolutionary computation showed that more effective recombinations can be made by using linkage learning. Inspired by these recent advances in this field, we propose a new variant of NSGA-II, called L2-NSGA, that uses linkage learning to optimize test case selection. In particular, we use an unsupervised clustering algorithm to infer promising patterns among the solutions (sub-test suites). Then, these patterns are used in the next iterations of L2-NSGA to create solutions that contain/preserve these inferred patterns. Our results show that our customizations make NSGA-II more effective for test case selection. Furthermore, the test suite sub-sets generated by L2-NSGA are less expensive and more effective (detect more faults) than those generated by MOEAs used in the literature for regression testing.

Log-based Slicing for System-level Test Cases

Mon, 19 Apr 2021 09:03:56 +0200

EvoSuite at the SBST 2021 Tool Competition

Thu, 01 Apr 2021 17:56:40 +0200

What Are We Really Testing in Mutation Testing for Machine Learning? A Critical Reflection

Mon, 18 Jan 2021 14:29:50 +0200

Search-Based Software Re-Modularization: A Case Study at Adyen

Fri, 01 Jan 2021 22:27:51 +0200

ReproducedPaper.org: Openly teaching and structuring machine learning reproducibility

Fri, 01 Jan 2021 17:32:54 +0200

Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities

Tue, 04 Aug 2020 16:05:57 +0200

DeepTC-Enhancer: Improving the Readability of Automatically Generated Tests

Thu, 30 Jul 2020 22:27:51 +0200

Good Things Come In Threes: Improving Search-based Crash Reproduction With Helper Objectives

Thu, 30 Jul 2020 22:27:46 +0200

Generating Highly-structured Input Data by Combining Search-based Testing and Grammar-based Fuzzing

Mon, 06 Jul 2020 14:29:50 +0200

Botsing, a Search-based Crash Reproduction Framework for Java

Sun, 05 Jul 2020 14:19:53 +0200

Crash Reproduction Using Helper Objectives

Sat, 04 Jul 2020 18:56:54 +0200

EvoSuite at the SBST 2020 Tool Competition

Fri, 17 Apr 2020 17:56:40 +0200

Automated Repair of Feature Interaction Failures in Automated Driving Systems

Fri, 17 Apr 2020 17:32:54 +0200

LogChunks: A Data Set for Build Log Analysis

Tue, 03 Mar 2020 17:08:37 +0100

Oracle Issues in Machine Learning and Where to Find Them

Tue, 03 Mar 2020 16:20:15 +0100

Abstract: The rise in popularity of machine learning (ML), and deep learning in particular, has both led to optimism about achievements of artificial intelligence, as well as concerns about possible weaknesses and vulnerabilities of ML pipelines. Within the software engineering community, this has led to a considerable body of work on ML testing techniques, including white- and black-box testing for ML models. This means the oracle problem needs to be addressed; for supervised ML applications, oracle information is indeed available in the form of dataset “ground truth”, that encodes input data with corresponding desired output labels. However, while ground truth forms a gold standard, there still is no guarantee it is truly correct. Indeed, syntactic, semantic, and conceptual framing issues in the oracle may negatively affect the ML system integrity. While syntactic issues may be automatically verified and corrected, the higher-level issues traditionally need human judgment and manual analysis. In this paper, we employ two heuristics based on information entropy and semantic analysis on well-known computer vision models and benchmark data from ImageNet. The heuristics are used to semi-automatically uncover potential higher-level issues in (i) the label taxonomy used to define the ground truth oracle (labels), and (ii) data encoding and representation. In doing this, beyond existing ML testing efforts, we illustrate the need for SE strategies that especially target and assess the oracle.

JCOMIX: A Search-Based Tool to Detect XML Injection Vulnerabilities in Web Applications

Wed, 26 Jun 2019 10:41:41 +0200

Effective and Efficient API Misuse Detection via Exception Propagation and Search-based Testing

Wed, 01 May 2019 20:22:51 +0200

Abstract : Application Programming Interfaces (APIs) typically come with (implicit) usage constraints. The violations of these constraints (API misuses) can lead to software crashes. Even though there are several tools that can detect API misuses, most of them suffer from a very high rate of false positives. We introduce Catcher, a novel API misuse detection approach that combines static exception propagation analysis with automatic search-based test case generation to effectively and efficiently pinpoint crash-prone API misuses in client applications. We validate Catcher against 21 Java applications, targeting misuses of the Java platform’s API. Our results indicate that Catcher is able to generate test cases that uncover 243 (unique) API misuses that result in crashes. Our empirical evaluation shows that Catcher can detect a large number of misuses (77 cases) that would remain undetected by the traditional coverage-based test case generator EvoSuite. Additionally, Catcher is on average eight times faster than EvoSuite in generating test cases for the identified misuses. Finally, we find that the majority of the exceptions triggered by Catcher are unexpected to developers i.e., not only unhandled in the source code but also not listed in the documentation of the client applications.

A Systematic Comparison of Search Algorithms for Topic Modelling - A Study on Duplicate Bug Report Identification

Mon, 29 Apr 2019 23:16:23 +0200

Abstract

Latent Dirichlet Allocation (LDA) has been used to support many software engineering tasks. Previous studies showed that default settings lead to sub-optimal topic modeling with a dramatic impact on the performance of such approaches in terms of precision and recall. For this reason, researchers used search algorithms (e.g., genetic algorithms) to automatically configure topic models in an unsupervised fashion. While previous work showed the ability of individual search algorithms in finding near-optimal configurations, it is not clear to what extent the choice of the meta-heuristic matters for SE tasks. In this paper, we present a systematic comparison of five different meta-heuristics to configure LDA in the context of duplicate bug reports identification. The results show that (1) no master algorithm outperforms the others for all software projects, (2) random search and PSO are the least effective meta-heuristics. Finally, the running time strongly depends on the computational complexity of LDA while the internal complexity of the search algorithms plays a negligible role.

An Adaptive Evolutionary Algorithm based on Non-Euclidean Geometry for Many-objective Optimization

Mon, 08 Apr 2019 10:45:30 +0200

EvoSuite at the SBST 2019 Tool Competition

Mon, 08 Apr 2019 10:41:50 +0200

Beyond Unit-Testing in Search-based Test Case Generation: Challenges and Opportunities

Mon, 08 Apr 2019 10:39:03 +0200

Incremental Control Dependency Frontier Exploration for Many-Criteria Test Case Generation

Fri, 20 Jul 2018 00:07:54 +0200

Single-objective versus Multi-Objectivized Optimization for Evolutionary Crash Reproduction

Fri, 20 Jul 2018 00:06:34 +0200

A Search-based Approach for Accurate Identification of Log Message Formats

Fri, 20 Jul 2018 00:05:13 +0200

Search-Based Test Data Generation for SQL Queries

Fri, 20 Jul 2018 00:04:04 +0200

An Investigation of Compression Techniques to Speed up Mutation Testing

Fri, 20 Jul 2018 00:03:02 +0200

Testing Autonomous Cars for Feature Interaction Failures using Many-Objective Search

Fri, 20 Jul 2018 00:01:56 +0200

Java Unit Testing Tool Competition - Sixth Round

Fri, 20 Jul 2018 00:01:14 +0200

Lightweight Detection of Android-specific Code Smells: the aDoctor Project

Sun, 13 Aug 2017 20:10:28 +0200

Software-Based Energy Profiling of Android Apps: Simple, Efficient and Reliable

Sun, 13 Aug 2017 20:08:32 +0200

Speeding-Up Mutation Testing via Data Compression and State Infection

Sun, 13 Aug 2017 20:06:48 +0200

Java Unit Testing Tool Competition - Fifth Round

Sun, 13 Aug 2017 20:05:07 +0200

PETrA: a Software-Based Tool for Estimating the Energy Profile of Android Applications

Sun, 13 Aug 2017 19:37:11 +0200

Guided Genetic Algorithm for Automated Crash Reproduction

Sun, 13 Aug 2017 19:34:17 +0200

Automatically Repairing Web Application Firewalls Based on Successful SQL Injection Attacks

Sun, 13 Aug 2017 19:28:13 +0200

LIPS vs MOSA: a Replicated Empirical Study on Automated Test Case Generation

Thu, 13 Jul 2017 20:03:02 +0200

Parameterizing and Assembling IR-based Solutions for Software Engineering Tasks using Genetic Algorithms

Thu, 18 Aug 2016 08:00:46 +0200

On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study

Thu, 18 Aug 2016 07:59:02 +0200

Evolutionary Testing for Crash Reproduction

Thu, 18 Aug 2016 07:56:19 +0200

How to Catch 'Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing

Thu, 18 Aug 2016 07:54:15 +0200

A Textual-based Technique for Smell Detection

Thu, 18 Aug 2016 07:52:50 +0200

Continuous Delivery Practices in a Large Financial Organization

Thu, 18 Aug 2016 07:51:10 +0200

A Search-based Training Algorithm for Cost-aware Prediction

Thu, 18 Aug 2016 07:47:34 +0200

Automatic Test Case Generation: What If Test Code Quality Matters?

Thu, 18 Aug 2016 07:45:36 +0200

The impact of test case summaries on bug fixing performance: An empirical investigation

Thu, 18 Aug 2016 07:43:26 +0200