Large language models used for code editing can be trained and deployed in at least two output regimes: direct generation, where the model emits the entire modified file in one shot, and iterative diff-based generation ("steps"), where the model emits a sequence of localized search/replace edits applied one at a time until it signals completion or a step budget is exhausted. The diff-based regime is attractive because it mirrors how developers edit code and should require far fewer generated tokens per turn. We train two code models - a 100M-parameter model trained from scratch (Rainbow-Pony-100M) and a fine-tuned Qwen2.5-Coder-0.5B - in both regimes on a shared Flutter/Dart dataset, and evaluate all four resulting models on a held-out set of approx 1,790 tasks per model. Direct generation substantially outperforms diff-based generation on every metric we measure - compilation/static-analysis pass rate, bits-per-byte, character-level similarity to the reference, and blinded LLM-judge ratings of goal fulfillment, correctness, and code quality - and the gap persists after controlling for task difficulty via a matched-ID comparison and when restricting to code that compiles on both sides. We then identify a single, architecture-independent mechanism behind the conditions where diff-based generation does win: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in our dataset. We term this task locality and discuss its implications for when an edit-based training regime is and is not the right choice for a code-editing model.
直接生成 vs. 迭代 diff:Flutter/Dart 代码模型编辑方式的实证对比
AI 导读
研究在共享 Flutter/Dart 数据集上训练 Rainbow-Pony-100M 和微调 Qwen2.5-Coder-0.5B 两种代码模型,对比直接生成整文件与迭代 diff 编辑两种输出方式,在约 1,790 个任务上直接生成在编译通过率、bits-per-byte、字符相似度和 LLM 评审等所有指标上均显著优于 diff 方式。
HuggingFace Daily Papers(社区热门论文)
41
AI 编辑部评分,满分 100直接生成 vs. 迭代 diff:Flutter/Dart 代码模型编辑方式的实证对比
研究在共享 Flutter/Dart 数据集上训练 Rainbow-Pony-100M 和微调 Qwen2.5-Coder-0.5B 两种代码模型,对比直接生成整文件与迭代 diff 编辑两种输出方式,在约 1,790 个任务上直接生成在编译通过率、bits-per-byte、字符相似度和 LLM 评审等所有指标上均显著优于 diff 方式。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org