{"id":"5067650f-0864-4a91-b427-72ac9045139e","arxiv_id":"2608.07617","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TeXFix-Bench grounds document-repair evaluation in a mined 18-category fault taxonomy, and shows that compile success alone misses content-destroying repairs in 13.6-18.5 percent of cases.","lead":"The paper introduces TeXFix-Bench, a benchmark for measuring how well LLMs repair broken LaTeX, Typst, and Markdown documents, with faults derived from a mined taxonomy of real human errors. It reports that 13.6-18.5 percent of repairs that compile still materially alter the document text, so compile success alone overstates repair quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Typst/Markdown operators are extended from the LaTeX taxonomy by analogy, so the multi-format claims—especially the Typst difficulty gap—rest on an untested transfer assumption.","rationale":"The reader identified the by-analogy extension as the weakest assumption; I agree. This is the most load-bearing concern because the paper's central novelty is empirical grounding, and the 'multi-format' claim in the title and abstract depends on the LaTeX taxonomy transferring to Typst and Markdown. The low kappa (0.34) and the in-sample real-error case study are real but secondary: the kappa is disclosed with a granularity diagnosis and an operationalizability check, and the case study is explicitly labeled as a case study. The by-analogy extension is a structural gap in the evidence for the benchmark's cross-format validity, and the Typst difficulty headline would be misleading if the Typst operators are not representative. My recommendation is unchanged: conditional acceptance, with the condition being an external validation of the Typst and Markdown fault structure, consistent with the reader's verdict. The proposed test is concrete and feasible, and would directly settle whether the analogy holds.","tokens_in":13365,"tokens_out":6600,"duration_ms":63832,"concrete_test":"Mine a corpus of real Typst and Markdown hard failures from comparable sources (Typst forum/GitHub issues; pandoc and CommonMark issue trackers), verify them under the pinned engines (Typst 0.15.1, pandoc 3.10.1), and dual-code them with the 18-category LaTeX codebook plus a 'not covered' category. If the LaTeX categories cover less than ~80% of verified Typst/Markdown faults, or if a substantial 'Other' category emerges, the analogy fails and cross-format claims need re-scoping. Alternatively, run the Section 5.1 cross-benchmark on Typst seeds with mined human Typst crashes vs. DocMut Typst operators; if DocMut Typst faults are not bracketed from below by real Typst faults (as in Table 3), the transfer is not validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The taxonomy is mined exclusively from LaTeX hard crashes (Section 3: TeX.SE, GitHub, package docs; 168 faults). Yet DocMut includes 15 Typst and 8 Markdown operators that are explicitly \"designed by analogy\" (Section 4; Section 11 external-validity note). All cross-format conclusions, including the headline finding that Typst is markedly harder than LaTeX and Markdown (Section 8.2), assume that the 18 LaTeX-derived fault categories transfer to Typst and Markdown fault distributions. This is not a minor caveat: the benchmark's identity as a multi-format standard depends on those operators sampling realistic repair tasks. If real Typst or Markdown hard failures concentrate in categories not covered by the LaTeX taxonomy—e.g., Typst's different module system and Markdown's mostly structural HTML/image failures—then DocMut's Typst/Markdown instances are not representative samples, and the reported format difficulty rankings and restoration-oracle results for those formats are properties of the artificial operator distribution, not of real repair workloads. The paper is transparent about the analogy (Section 11), but the abstract and contribution list present the benchmark as multi-format and empirically grounded without the same evidentiary basis for Typst/Markdown that exists for LaTeX.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TeXFix-Bench, a benchmark for full-source document repair by LLMs across LaTeX, Typst, and Markdown. It first mines 168 real localized LaTeX hard crashes from TeX.SE, GitHub commits, and package documentation, and derives an 18-category fault taxonomy through dual open coding. The taxonomy is instantiated as DocMut, a 48-operator AST-aware mutation library (25 LaTeX, 15 Typst, 8 Markdown). Using a three-model cross-benchmark on matched seeds, the paper reports that DocMut faults are 5.6–9.2 percentage points harder to repair than pattern-based mutations, and a real-error case study on 88 human crashes reports 67.0% repair success, bracketing the synthetic sets from below. The paper then constructs a 10,437-instance benchmark from 743 openly licensed seeds, evaluates seven LLMs under a fixed zero-shot protocol with provider-pinned routing, and reports a 27.5-point intention-to-treat compile spread. A restoration oracle over 28,129 compiling repairs shows that 13.6–18.5% of compiling repairs materially alter document text, and that restoration rank diverges from compile rank. The benchmark, taxonomy, operator library, and campaign artifacts are released.","tokens_in":13642,"tokens_out":10897,"duration_ms":92782,"significance":"If the central claims hold, the paper supplies a valuable and much-needed evaluation standard for document-repair LLMs. Its strengths include unusually transparent reporting: the low inter-rater kappa is disclosed and analyzed, intention-to-treat accounting is retained, provider-pinned routing is documented with a per-request ledger, the balanced 6,613×7 model matrix is used for primary claims, and the 14.3% unscorable restoration rows are explicitly accounted for rather than hidden. The restoration oracle is a genuinely useful methodological contribution to the compile-only leaderboard debate. The release of the taxonomy, DocMut, and campaign artifacts makes the work reproducible. However, the multi-format identity of the benchmark rests on an untested analogy transfer from LaTeX to Typst and Markdown, and the real-error case study is partly circular because it reuses the corpus from which the taxonomy was derived. These issues temper the strength of the headline cross-format and validation claims.","major_comments":[{"comment":"The Typst and Markdown operators are not derived from mined human faults: Section 4 states they 'share the adequacy framework and are designed by analogy' from the LaTeX-mined taxonomy, and Section 11 lists this as an external-validity threat. Yet the abstract and contributions present the benchmark as 'multi-format' and 'empirically grounded,' and Section 8.2's headline that 'Typst is markedly harder' is computed on instances generated by these analogue operators. If real Typst or Markdown hard failures concentrate in categories outside the 18 LaTeX categories, the format-difficulty ranking and the restoration-oracle results for those formats are properties of the artificial operator distribution, not of real repair workloads. This is load-bearing for the benchmark's identity as a multi-format standard. A concrete fix would be to mine a modest sample of real Typst and Markdown hard crashes, compare their category coverage against the 18 LaTeX categories, and either validate the transfer or restrict the 'empirically grounded' claim to LaTeX.","section":"§4, §8.2, §11"},{"comment":"The real-error case study is used as evidence that 'human faults are harder still' and that taxonomy-guided injection moves synthetic evaluation toward real difficulty. However, the 88 evaluated cases are a stratified sample of the same 168 faults from which the taxonomy and, through the taxonomy-refinement loop in Section 4, the DocMut operators were derived. The ordering pattern > DocMut > real is therefore not an independent validation of the taxonomy's adequacy: the operators were refined precisely on these faults. An independent test would apply DocMut to a held-out set of newly mined hard crashes (or at least to faults excluded from the taxonomy derivation) and compare repair success on those. The current framing overstates the evidential value of the case study.","section":"§5.3"},{"comment":"The inter-rater reliability of the taxonomy is kappa=0.34 under free-form open coding, which is 'fair' and below conventional thresholds for instrument reliability. The paper explains the granularity asymmetry and adds an operationalizability check (kappa=0.64) against Annotator 1's labels, but that check is a deterministic classifier encoding the resolution rules, not a second human annotation with the codebook. Because the entire benchmark rests on the claim that the taxonomy is 'empirically grounded,' the absence of a closed-codebook human reliability round remains a load-bearing gap. A second-round human annotation on a subset of the 168 faults would materially strengthen the taxonomy's credibility.","section":"§3.2"},{"comment":"The claim that 'the model with the lowest compile rate restores content best among its successes' (Qwen) is based on mean S_pdf (0.971 vs. Grok's 0.966), a difference of 0.005. The <0.95 tail share, however, is 13.9% for Qwen and 13.6% for Grok, so on the tail metric Qwen is not best. The paper should report both moments with uncertainty (e.g., bootstrapped CIs) or with a pre-specified restoration ranking metric, and should soften the claim if the divergence is not consistent across summary statistics. The broader point that 13.6–18.5% of compiling repairs materially alter text is unaffected.","section":"§9.1"},{"comment":"The format-difficulty results in Section 8.2 are computed on the 'full recorded set,' which has uneven per-model coverage (6,613–7,007 instances per model), while the stated design is that all primary claims are verified on the balanced 6,613×7 matrix. The text does not show the format ranking on the balanced matrix, so the claim that the balanced matrix 'confirms all rankings' is not demonstrated for RQ2. Since the full recorded set mixes different model samples across instances, the aggregate format rates could be sensitive to which models attempted which instances. Please report the format ranking on the balanced matrix.","section":"§7.3, §8.2"}],"minor_comments":[{"comment":"Please state explicitly that the cross-benchmark is LaTeX-only (the fault patterns listed are LaTeX-specific), because the current text and Table 2 do not identify the format and the conclusion is subsequently generalized to all DocMut operators.","section":"§5.1"},{"comment":"The text says the engine-gate ledger covers 38 operators and 39 have nonzero instance support, but Table 9 lists 36 codes and omits the mentioned TEX-MTH-OPS single-instance operator; please reconcile these counts.","section":"§6.2, Appendix A"},{"comment":"The parenthetical qualification about Typst/Markdown being extended by analogy is good, but it appears only in the abstract; consider stating it prominently in Section 1 and in the contributions list, where the multi-format claim is made.","section":"Abstract, §1"},{"comment":"The statement that the Typst gap appears across nearly all models would be easier to verify if the per-model Typst CRitt rates were tabulated alongside the span.","section":"§8.2"},{"comment":"Reporting the fraction of candidates with S_pdf exactly 1.0 would help quantify how the median of 1.0 coexists with the 13.6–18.5% tail below 0.95.","section":"§9"},{"comment":"Reference [10] appears malformed ('ACM Comput. Surv.. Extended bibliography'); please clean up the reference entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually transparent and the released artifacts are a genuine contribution. My main concern is that the multi-format identity leans on an analogy transfer that is not validated; if the authors can either validate it with a small Typst/Markdown mining study or re-scope the claims, I would be comfortable. The kappa and circularity issues are addressable. This is not a reject, but the cross-format and validation claims need work before the paper meets the bar for a benchmark that is claimed to be empirically grounded across formats."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read TeXFix-Bench. The real contribution is the LaTeX fault taxonomy and the two-oracle evaluation protocol. Mined 168 verified hard-crash faults, coded them into 18 categories, built DocMut operators from them, and then showed those operators produce harder repair tasks than pattern-based injections (5.6-9.2 pp) and that real human faults are harder still (67% vs 81.3%). The restoration oracle result—13.6-18.5% of compiling repairs materially alter document text, and the model with the lowest compile rate restores best—is practically important and should survive scrutiny.\n\nThe paper is admirably transparent: kappa=0.34 reported as a finding about open-coding granularity, 14.3% unscorable restoration rows, unbalanced cells disclosed, provider pinning and ledger, ~$200 cost. The balanced 6,613x7 matrix anchors the rankings. That is real evidence.\n\nThe soft spots are real but mostly scoped. The Typst and Markdown operators are not mined; they extend the LaTeX taxonomy by analogy. So the headline \"Typst is markedly harder\" is a claim about DocMut's operators, not about real Typst failure distributions. The stress-test note is right: the benchmark's multi-format identity rests on an untested transfer assumption. The authors disclose it in Section 11, but the abstract and title present the multi-format claim more strongly than the evidence supports. The real-error case study is also partially circular—it stratifies the same 168 faults used to build the taxonomy—so the \"human faults are harder\" ordering is not an independent validation of taxonomy adequacy. Kappa 0.34 is low, though the operationalizability check at 0.64 gives some confidence the categories are codable.\n\nNone of this breaks the core LaTeX contribution. The fix is to reframe the Typst/Markdown results as exploratory or analogue-based, and to present the case study as a sanity check rather than validation. If they can ground even one of the other formats empirically—even a smaller taxonomy—the multi-format standard claim becomes solid.\n\nThis deserves a serious referee and conditional acceptance. I would bring it to reading group, and I'd cite it if I were working on document repair or LLM benchmark methodology.","headline":"A genuinely useful LaTeX repair benchmark with a strong two-oracle protocol; the multi-format extension is thinner than the title suggests but openly scoped.","tokens_in":14173,"tokens_out":2027,"would_cite":true,"duration_ms":19472,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that evaluating LLM document repair on arbitrary injected typos overstates performance, and that repair benchmarks should be grounded in a mined taxonomy of real human crashes and judged under both compile and…","keywords":["automated program repair","LaTeX","Typst","Markdown","mutation testing","fault taxonomy","LLM evaluation","two-oracle benchmark"],"falsifier":"Collect a separate set of real human Typst compilation crashes from public forums and issue trackers, classify them against the 18 LaTeX-derived categories, and repair them with the same models under the same protocol; if the real Typst fault distribution is dominated by categories outside the analogical mapping, or if models repair real Typst crashes at rates far above or below DocMut-generated Typst instances, the benchmark's cross-format validity claim would be falsified.","tokens_in":1668,"feed_emoji":"📜","tokens_out":2027,"duration_ms":63326,"temperature":0.7,"pith_summary":"This paper argues that evaluating LLM document repair on arbitrary injected typos overstates performance, and that repair benchmarks should be grounded in a mined taxonomy of real human crashes and judged under both compile and content-restoration oracles. It mines 168 real LaTeX compilation crashes from a Q&A forum, commit histories, and package documentation, codes them into 18 fault categories, and turns those categories into a 48-operator mutation library called DocMut. On matched seeds, repairs of DocMut faults are 5.6–9.2 percentage points harder than repairs of pattern-based mutations across three model families, and 88 real human crashes are harder still, with a 67.0% repair success rate. The resulting 10,437-instance benchmark is scored twice: engine compile success, and a text-restoration similarity score. The restoration oracle shows that among repairs that compile, 13.6–18.5% materially alter document text, so compile-only leaderboards overstate repair quality.","feed_headline":"Mining real LaTeX crashes yields harder, fairer LLM repair tests","feed_subtitle":"Compile-only leaderboards overstate repair quality; a restoration oracle shows 13.6–18.5% of fixes alter text.","key_machinery":"The load-bearing machinery is the mined fault taxonomy plus its instantiation and scoring. The taxonomy is an 18-category scheme derived by dual open coding of 168 verified localized hard crashes in LaTeX, with a documented reliability analysis. DocMut is the 48-operator, format-aware mutation library (25 LaTeX, 15 Typst, 8 Markdown) that instantiates the taxonomy into concrete broken sources. The two-oracle protocol carries the evaluation: a pinned engine gate (Tectonic for LaTeX, Typst engine for Typst, pandoc for Markdown) decides compile success, and a restoration oracle computes token-LCS Dice similarity between the extracted text of the candidate and the golden document to measure whether content survived.","core_discovery":"The central claim is that how faults are injected into a document-repair benchmark changes what the benchmark measures: pattern-based edits inflated repair scores by up to ten points, while faults modeled on real human crashes were harder for every model family tested, and genuinely mined human crashes were harder still. The paper further claims that compile success is an insufficient quality oracle, since a large fraction of compiling repairs delete or rewrite substantial document content, and that the model ranking by content restoration diverges from the ranking by compile rate. It presents this as an end-to-end loop: mine human faults, derive a taxonomy, instantiate it as a mutation library, validate that the library produces realistic difficulty, then run a large multi-model evaluation under two oracles.","pith_inferences":["If the restoration-oracle result generalizes, compile-only leaderboards may systematically favor models that patch by deletion, so product builders should weight content fidelity when choosing a repair model.","Mining Typst and Markdown human crashes directly would test the analogical extension of the LaTeX taxonomy; the 18 LaTeX categories may not cover Typst's scripting and set-rule faults or Markdown's rendering semantics.","The 67.0% success rate on real human crashes suggests realistic repair accuracy is well below the compile rates seen on synthetic sets, so future benchmark versions could include permanently held-out real-crash instances as a calibration check.","Extending the restoration oracle from extracted-text similarity to layout or visual similarity could sharpen the measurement of content fidelity, particularly for Typst and Markdown where text extraction is only a proxy."],"forward_implications":["Document-repair leaderboards should report a content-fidelity statistic alongside compile success, because compile-only rankings can reward aggressive full-rewrite behavior that destroys content.","Benchmark builders for new markup or programming domains should mine human faults before designing mutation operators, since operator selection changes measured difficulty by roughly five to ten points under fixed models and seeds.","The Typst gap, which appears across nearly all models, indicates that newer syntax and less training exposure make repairs markedly harder, so cross-language difficulty should be reported per format rather than averaged away.","Provider-pinned routing and a per-request provider ledger make delivery failures attributable to serving stacks, meaning deployment-facing rates should be separated from conditional model-skill rates.","The zero-shot protocol without diagnostics gives lower bounds on assisted repair performance; supplying compiler errors as input is a planned extension that would likely raise the reported success rates."],"supporting_citations":[{"why":"Establishes that mutation operator selection should be grounded in empirically sufficient fault categories, which the paper uses to justify mining a taxonomy before designing DocMut.","marker":"[11]"},{"why":"Supplies the mutation-testing background on operator design and equivalent mutants that informs DocMut's hard/soft tracks and render-diff equivalence guard.","marker":"[5]"},{"why":"Provides the APR benchmark contract of test-based oracles that the paper adapts into engine-gated compile oracles for document sources.","marker":"[7]"},{"why":"Warns that patches passing a test oracle can overfit, which directly motivates the paper's second oracle measuring content restoration.","marker":"[14]"},{"why":"The paper's own earlier pattern-based pilot benchmark is what DocMut is compared against, making it the baseline for the difficulty-gap validation.","marker":"[17]"},{"why":"Documents cross-engine inconsistencies in TeX-produced documents, which the paper cites to justify pinning engine versions in its gate oracle.","marker":"[15]"}],"fun_headline_variants":["Real crash faults beat synthetic edits for LLM repair benchmarks","Compile success overstates LLM repair quality; content restoration matters","Fault injection realism changes LLM repair rankings by up to 10 points","Real-world crash mining makes document repair LLM tests tougher","From real LaTeX crashes to harder LLM repair benchmarks"],"cache_read_input_tokens":16256,"weakest_assumption_plain":"The cross-format claims assume that the 18 fault categories mined from LaTeX crashes transfer to Typst and Markdown, but the Typst and Markdown DocMut operators were designed by analogy rather than mined; if real faults in those formats follow a different distribution, the benchmark's cross-format difficulty rankings are not representative.","fun_headline_variants_meta":{"raw":{"variants":["Real crash faults beat synthetic edits for LLM repair benchmarks","Compile success overstates LLM repair quality; content restoration matters","Fault injection realism changes LLM repair rankings by up to 10 points","Real-world crash mining makes document repair LLM tests tougher","From real LaTeX crashes to harder LLM repair benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000863,"raw_usage":{"total_tokens":3802,"prompt_tokens":1063,"completion_tokens":2739,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":2652}},"tokens_in":679,"tokens_out":2739,"duration_ms":19977,"temperature":1.0,"reasoning_tokens":2652,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:28:58.969188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a separate set of real human Typst compilation crashes from public forums and issue trackers, classify them against the 18 LaTeX-derived categories, and repair them with the same models under the same protocol; if the real Typst fault distribution is dominated by categories outside the analogical mapping, or if models repair real Typst crashes at rates far above or below DocMut-generated Typst instances, the benchmark's cross-format validity claim would be falsified.","supporting_citations":[{"cited_title":"Jefferson Offutt, Ammei Lee, Gregg Rothermel, Roland H","cited_arxiv_id":null,"evidence_quote":"Establishes that mutation operator selection should be grounded in empirically sufficient fault categories, which the paper uses to justify mining a taxonomy before designing DocMut."},{"cited_title":"Venkateshmurthy","cited_arxiv_id":null,"evidence_quote":"The paper's own earlier pattern-based pilot benchmark is what DocMut is compared against, making it the baseline for the difficulty-gap validation."}],"review_version":1}