Large Language Models

AI Agents have rapidly gained prominence in both research and industry as systems that extend large language models with planning, tool …

Background: Large Language Models (LLMs) are increasingly being applied to Software Engineering (SE) tasks, achieving high accuracy …

Recently, test case selection (TCS) techniques have been explored to support the operational evaluation of deep neural networks (DNNs) …

LLM-based automated program repair (APR) techniques have shown promising results in reducing debugging costs. However, prior results …

Knowledge distillation compresses large language models (LLMs) into more compact and efficient versions that achieve similar accuracy …

Abstract: Writing software tests is laborious and time-consuming. To address this, prior studies introduced various automated test-generation techniques. A well-explored research direction in this field is unit test generation, wherein artificial intelligence (AI) techniques create tests for a method/class under test.

Large Language Models (LLMs) have gained considerable traction within the Software Engineering (SE) community, impacting various SE tasks from code completion to test generation, from program repair to code summarization. Despite their promise, researchers must still be careful as numerous intricate factors can influence the outcomes of experiments involving LLMs. This paper initiates an open discussion on potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings. In response, this paper proposes a set of guidelines tailored for SE researchers and Language Model (LM) providers to mitigate these concerns. The implications of the guidelines are illustrated using existing good practices followed by LLM providers and a practical example for SE researchers in the context of test case generation.