In the realm of code editing, there is a growing recognition that correctness alone is not sufficient. The focus has traditionally been on functional success metrics such as Pass@1 or Pass@k, which assess whether a model can successfully fix bugs in code. However, repairs should also be minimal, reviewable, and faithful to the original implementation. This concept of edit fidelity is crucial in ensuring that code edits are not only correct but also maintain the integrity and coherence of the existing codebase. A recent study delves into the phenomenon of over-editing where models tend to make unnecessary changes to code beyond what is required to address a bug. By analyzing over 400 BigCodeBench problems and injecting controlled AST-level corruptions into reference solutions, researchers were able to quantify the extent of over-editing in large language models (LLMs), even among advanced models like GPT-5.5. Despite achieving high Pass@1 scores, these models exhibited a tendency towards unnecessarily large edits and increased cognitive complexity. To mitigate over-editing behavior, a preservation instruction was introduced resulting in significant reduction in excess Levenshtein distance and added cognitive complexity while improving Pass@1 performance. Interestingly, these improvements were not solely attributed to larger reasoning budgets or model sizes but rather stemmed from deliberate focus on edit fidelity during post-training. Further exploration into learning minimal editing directly post-training revealed that supervised fine-tuning tended to overfit to known corruption patterns. In contrast,reinforcement learning offered a more balanced trade-off between out-of-domain edit fidelity and performance retention. These findings underscore the importance of edit fidelity as a distinct dimension of code-repair quality that can be both measured and learned systematically.By shifting the emphasis from mere functional success to encompassing edit fidelity as an essential criterion for evaluating code repairs,this research opens up new avenues for enhancing the overall quality and maintainability of software systems through more precise and faithful editing practices.
Summary- Making sure code is right is not enough when editing.
- Usually, we care about how well the code works, like if it passes certain tests.
- Fixes should be small, easy to check, and stay true to the original code (edit fidelity).
- Big language models sometimes make too many changes (over-editing), like GPT-5.5.
- A new rule called preservation instruction helps stop over-editing.
Definitions1. Correctness: Being right or accurate.
- Functional success metrics: Ways to measure how well something works based on specific criteria.
- Edit fidelity: Making changes that are true to the original version.
- Large language models (LLMs): Advanced computer programs that can understand and generate human-like text.
- Over-editing: Making too many unnecessary changes.
- Preservation instruction: A rule or guideline to prevent making excessive edits.
- Code-repair quality: How well fixes improve or maintain the code's functionality and structure.
In the world of code editing, there is a growing recognition that correctness alone is not enough. While functional success metrics such as Pass@1 or Pass@k have traditionally been used to evaluate whether a model can successfully fix bugs in code, it is becoming increasingly clear that repairs should also be minimal, reviewable, and faithful to the original implementation. This concept of "edit fidelity" is crucial in ensuring that code edits not only address bugs but also maintain the integrity and coherence of the existing codebase.
A recent study delves into this phenomenon of over-editing, where models tend to make unnecessary changes to code beyond what is required to fix a bug. By analyzing over 400 BigCodeBench problems and injecting controlled AST-level corruptions into reference solutions, researchers were able to quantify the extent of over-editing in large language models (LLMs), even among advanced models like GPT-5.5. Despite achieving high Pass@1 scores, these models exhibited a tendency towards unnecessarily large edits and increased cognitive complexity.
To mitigate this behavior, the researchers introduced a preservation instruction which resulted in significant reduction in excess Levenshtein distance (a measure of edit distance) and added cognitive complexity while improving Pass@1 performance. Interestingly, these improvements were not solely attributed to larger reasoning budgets or model sizes but rather stemmed from deliberate focus on edit fidelity during post-training.
Further exploration into learning minimal editing directly post-training revealed that supervised fine-tuning tended to overfit to known corruption patterns. In contrast,reinforcement learning offered a more balanced trade-off between out-of-domain edit fidelity and performance retention. These findings underscore the importance of edit fidelity as a distinct dimension of code-repair quality that can be both measured and learned systematically.
By shifting the emphasis from mere functional success metrics to encompassing edit fidelity as an essential criterion for evaluating code repairs,this research opens up new avenues for enhancing the overall quality and maintainability of software systems through more precise and faithful editing practices. This not only benefits developers by reducing the cognitive load of reviewing large edits, but also improves the overall stability and reliability of software systems.
One key takeaway from this research is that edit fidelity should be considered as a separate dimension in evaluating code repairs, rather than solely relying on functional success metrics. This highlights the need for more comprehensive evaluation methods that take into account both correctness and edit fidelity. Additionally, it emphasizes the importance of post-training techniques such as reinforcement learning in improving edit fidelity without sacrificing performance.
In conclusion, this study sheds light on the phenomenon of over-editing in large language models and highlights the significance of edit fidelity in code repairs. By addressing this issue, we can improve the overall quality and maintainability of software systems, making them more reliable and efficient for developers to work with. As technology continues to advance, it is crucial that we consider all aspects of code editing to ensure that our software remains robust and effective.
Large language models (LLMs) are increasingly used to edit existing code, but correctness alone is not enough: useful repairs should also be minimal, reviewable, and faithful to the original implementation. We study over-editing, the tendency of a model to rewrite code beyond what is required to fix a bug. We construct an evaluation framework from 400 BigCodeBench problems by injecting controlled AST-level corruptions into reference solutions, giving each repair task a known minimal patch. Across frontier LLMs, over-editing is widespread even among strong models like GPT-5.5: high Pass@1 can coexist with unnecessarily large edits and added cognitive complexity. A preservation instruction substantially reduces this behavior, lowering average excess Levenshtein distance from 0.195 to 0.131, reducing added cognitive complexity by 26.6%, and increasing Pass@1 by 2.3 points. However, these gains do not simply follow from a larger reasoning budget or larger models. We next ask whether minimal editing can be learned directly during post-training. We observe that supervised fine-tuning overfits to seen corruption patterns, whereas reinforcement learning gives the best out-of-domain edit-fidelity and performance-retention trade-off. These results position edit fidelity as a distinct axis of code-repair quality and show that it can be measured and learned.
