Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests

Junda Zhao, Shurui Zhou, Eldan Cohen. ISSTA 2026.

LLMs show great promise for automating unit test generation, but the quality of the generated tests can suffer when the model is prompted with buggy code. This paper presents a new metric to quantify the misguidance effect, a phenomenon where buggy code steers LLMs toward generating tests that validate its erroneous behavior rather than expose it. Across 318 focal methods from Defects4J and 11 LLMs (13 configurations), prompting with buggy code has a twofold impact: it significantly increases misguided tests that assert the incorrect behavior, while suppressing the generation of effective, bug-finding tests. We corroborate the effect from a model-internal perspective, showing that buggy code skews an LLM’s preference toward tests that assert the same erroneous behavior.

To counter this, we introduce a specification-based unit test generation paradigm that replaces the code under test in the prompt with an LLM-generated specification docstring. This paradigm reduces misguided tests while substantially increasing effective tests, improves multi-round, feedback-driven test generation pipelines, and remains applicable to both buggy and bug-free code.

• arXiv
• Replication package on GitHub and archived on Zenodo
• Slides and poster from ISSTA 2026