Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)

Junda Zhao, Shurui Zhou, Eldan Cohen. ISSTA 2026. ACM SIGSOFT Distinguished Paper Award.

Coverage and mutation score are the standard proxy metrics for evaluating LLM-generated test suites. For human-written tests, however, prior studies by Inozemtseva et al. and Papadakis et al. showed that the correlations among coverage, mutation, and real-bug detection can largely vanish once test-suite size is controlled, and it is unclear whether those conclusions carry over to LLM-generated tests. We conduct a large-scale replication of these two studies on more than 100,000 test cases generated by 11 state-of-the-art LLMs for Defects4J, spanning statement, branch, and modified condition coverage, and re-examine the relationships among coverage, mutation score, and real-bug detection effectiveness.

Our findings diverge substantially from the prior results. The usefulness of coverage and mutation is highly context-dependent. In regression-style settings, where the code given to the LLM can reasonably be assumed bug-free, these metrics can provide meaningful signals when comparing across models. In the other common scenario, where the code under test may already be buggy and the goal is to expose that bug, they no longer serve as reliable indicators. We also find little evidence that test-suite size is a dominant confounder of these correlations for LLM-generated tests. Based on these findings, we discuss how to interpret results from prior studies and provide actionable guidance for evaluating LLM-based test generation.

• arXiv
• Replication package on GitHub and archived on Zenodo
• Slides and poster from ISSTA 2026