{"id":"e65749a0-9b16-4496-a0ed-2424667d7c8a","arxiv_id":"2412.19770","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper introduces a dialogue-based fine-tuning dataset for Fortran-to-C++ translation and reports up to 3.31x CodeBLEU improvement and up to 92 percent compilation success on fine-tuned LLMs.","lead":"Fortran2CPP generates a multi-turn dialogue dataset using two LLM agents, a Questioner and a Solver, to translate Fortran code into C++ with compile and test feedback. Fine-tuning open-weight LLMs on this dataset improves their Fortran-to-C++ translation scores, though the gains are moderate in absolute terms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The functional-equivalence claim rests on LLM-generated unit tests as the oracle in both dataset creation and evaluation; an independent human-test audit is needed before the execution-test gains can be trusted.","rationale":"The reader's weakest-assumption analysis identifies exactly the concern I find most load-bearing: the LLM-generated unit-test oracle is used both during data generation and during evaluation, creating a circularity that can inflate functional-reliability metrics. I see no additional concern that is more fundamental. The CodeBLEU and compilation improvements are based on deterministic tools (n-gram/syntax/dataflow similarity and the GNU compiler), so those parts of the central claim are comparatively secure. The execution-test results and the claim of 'functional reliability' are the vulnerable piece, and the paper's own Limitations section concedes the point without providing a mitigating measurement. Because the manuscript is otherwise transparent, open-sources the dataset and models, and includes a manual investigation of 50 samples, the appropriate outcome is the reader's original CONDITIONAL verdict, pending an independent human-written test audit. I therefore set verdict_should_be to UNCHANGED, meaning no change to the reader's conditional recommendation, while emphasizing that the specific audit above should be a condition for full acceptance.","tokens_in":13561,"tokens_out":3433,"duration_ms":36506,"concrete_test":"Audit a random sample of 50 accepted pairs from the Fortran2CPP dataset and 50 samples from each evaluation benchmark (HPC-Fortran-Cpp and HumanEval-Fortran2Cpp). Have independent human Fortran and C++ programmers, blind to the LLM-generated tests, write unit tests for each pair, compile both versions with gfortran/g++, and compare pass/fail outcomes. Recompute the Execution Test metric using these human-written tests rather than GPT-4-generated tests. If execution success drops materially (e.g., more than 10 percentage points) or any accepted dataset pair fails the human tests, the functional-equivalence claim is inflated and the paper should be conditionally revised rather than accepted as-is.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim of improved functional reliability ('92% increase in compilation success rate', execution-test ratios up to 0.92) depends on the validity of LLM-generated unit tests as an equivalence oracle. This assumption is load-bearing twice. First, in dataset generation (Section 3.3.3), the Solver generates unit tests for both the original Fortran and translated C++ code in the same dialogue context as the translation itself, and the final verification (Section 3.3.5) is an LLM binary 'Yes/No' judgment over those test results. A semantically wrong translation that passes weak or self-consistent tests is therefore stored as a verified pair, corrupting the fine-tuning data. Second, in evaluation (Section 4.1), the Execution Test metric runs translated code against unit tests generated by GPT-4, and the HumanEval-Fortran2Cpp benchmark itself was validated through a GPT-4-based pipeline. The same class of LLM oracle is thus used to generate, filter, and measure, so reported execution-test gains can be inflated even when CodeBLEU and compilation improvements are real. The Limitations section acknowledges that 'adopting a stricter validation process for this step would be beneficial,' but no such validation is supplied, and without it the functional-reliability component of the central claim is not independently established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Fortran2CPP, a pipeline that uses a dual-agent Questioner-Solver LLM module to translate Fortran code to C++, producing both a code-pair dataset and a multi-turn dialogue dataset (11.7k prompt-response pairs derived from 1.2k dialogues). The authors fine-tune DeepSeek-Coder 6.7B, CodeLlama 13B, and StarCoder 15.5B on this dialogue data and report substantial improvements over base models and over models fine-tuned on code pairs only, on two benchmarks: HPC-Fortran-Cpp (296 pairs) and a newly created HumanEval-Fortran2Cpp (126 pairs). Metrics include CodeBLEU, compilation success rate, execution-test pass ratio, and a small manual investigation. The core claim is that process-level dialogue data improves syntactic and functional translation quality more than static code-pair fine-tuning.","tokens_in":13802,"tokens_out":3914,"duration_ms":39783,"significance":"If the reported execution-test gains are valid, this is a well-timed contribution to low-resource code translation: it provides an open dataset, open model artifacts, and a concrete demonstration that dialogue-style agent trajectories can be exploited for fine-tuning. The ablation against code-pair fine-tuning is a valuable and relatively clean comparison, and the consistent trends across three model families strengthen the qualitative finding that dialogue data helps. However, the significance is tempered by the fact that the functional-reliability claims rest on LLM-generated unit tests used both as the dataset filter and as the evaluation oracle; independent validation is needed before the execution-test component of the claims can be accepted.","major_comments":[{"comment":"The execution-test metric is not independent of the training-data filter. In §3.3.5, the final verification asks an LLM to answer 'Yes' or 'No' on unit-test outputs before a pair is kept, and in §4.1 the Execution Test runs translated code against unit tests generated by GPT-4. Because the same class of oracle is used to filter the training data and to measure success, the reported execution-test improvements (e.g., CodeLlama 13B rising from 0.0 to 0.92 on HumanEval-Fortran2Cpp in Table 4) may partly reflect the model learning to satisfy the test-generation style rather than true functional equivalence. The Limitations section acknowledges this, but no independent validation is supplied; please add a human-audited test suite (at least on a stratified sample) or an alternative differential-testing oracle to support the functional-reliability component of the central claim.","section":"§3.3.5 and §4.1"},{"comment":"The paper does not state which LLM (or which version) plays the roles of Questioner and Solver in the Fortran2CPP generation pipeline. This matters for interpreting the results: if GPT-4 powers the pipeline, the fine-tuning results are essentially distillation from GPT-4 and should be framed and analyzed as such; if an open-weight model is used, the reproducibility and cost claims change. Please specify the backbone model, its version, and the prompting protocol, and report the dialogue-level success rate (the 29.6% figure is described as a data 'conversation rate,' which appears to be a conversion rate for code pairs).","section":"§3.3 and §3.4"},{"comment":"The HumanEval-Fortran2Cpp benchmark was created by a GPT-4-based pipeline with iterative compilation, execution, and refinement, but the correctness of the resulting Fortran reference implementations is not independently verified. Since this benchmark is used to claim functional improvements (e.g., execution ratios up to 0.92), an incorrect reference implementation could inflate or deflate measured performance. Please provide a human validation of at least a sample of the 126 Fortran references or release the verification artifacts so that correctness can be checked.","section":"§4.1 (HumanEval-Fortran2Cpp construction)"},{"comment":"No variance or repeated-run information is reported for fine-tuning or evaluation. The evaluation sets are small (296 and 126), and the differences between dialogue and code-pair fine-tuning, while consistent in direction, are not quantified with error bars or significance tests. Please report results across at least three fine-tuning seeds and, if possible, multiple evaluation samples, so that the claimed advantage of dialogue over code-pair data is not an artifact of a single run.","section":"§4.2, Tables 3 and 4"}],"minor_comments":[{"comment":"There is a duplicated word in the Introduction: 'achieving achieving' should be 'achieving'.","section":"Abstract and §1"},{"comment":"The phrase 'with a successful data conversation rate at 29.6%' should read 'conversion rate'; the current wording is confusing.","section":"§3.4"},{"comment":"The figure contains the typo 'Fotran' for 'Fortran', and the model name is written inconsistently as 'Codellama' in some places and 'CodeLlama' in others; please standardize.","section":"Figure 1 and throughout"},{"comment":"The CodeParrot reference contains the placeholder 'Accessed: [Insert date here]'; the access date should be completed.","section":"Reference list"},{"comment":"The dialogue-splitting example uses a trivial 'Hi/Hello' exchange and does not illustrate a Fortran translation dialogue; consider replacing it with a real excerpt from the dataset.","section":"Appendix A.3"},{"comment":"The description of the Execution Test metric is underspecified: it is not clear whether the unit tests are generated from the Fortran or the C++ side, how many tests are used per pair, and how 'proportion of matching outputs' is computed when multiple tests exist; please clarify.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely a useful contribution to the code-translation and HPC-software-modernization community, and the open release of data and models is commendable. The main risk is that the functional-reliability claim depends on an LLM-oracle circularity, and the paper's current limitations statement does not resolve it. I recommend major revision with a request for an independent, human-validated test suite (on a sample) and explicit specification of the LLM used in dataset generation. If the authors provide this, the paper could become a strong systems contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper gives us something concrete: a new dialogue dataset for Fortran-to-C++ translation, generated by a two-agent Questioner-Solver pipeline, and a clean ablation showing that fine-tuning on dialogue data beats fine-tuning on simple code pairs across three open-weight models and two benchmarks. That ablation is the real find. It holds for CodeBLEU, compilation rate, and even the execution-test numbers, and the gains are consistent enough that I believe the direction is right.\n\nThe dataset is open-sourced, the pipeline is described in enough detail to reproduce, and the manual investigation scores (though small samples) line up with the automatic metrics. For anyone working on low-resource code translation, this is a useful contribution.\n\nNow the soft spots. The reader's circularity concern is legitimate: the same LLM-generated unit tests are used to filter training pairs and to measure execution success. That means the absolute execution-test numbers are probably over-optimistic. But it's not fatal, because CodeBLEU and compilation checks do not depend on that oracle, and the dialogue-vs-code-pair comparison uses the same biased filtering for both arms, so the relative gain is still informative. The paper explicitly acknowledges the limitation, which is honest.\n\nTwo things in the abstract are overstated. The 11.7k is prompt-response pairs, not dialogues (they come from 1.2k dialogues), and '92% increase in compilation success rate' is odd because the baseline is zero; it's a jump from 0 to 0.92. Minor, but worth fixing.\n\nI'd send this to review. The central claim—dialogue fine-tuning improves translation—is supported by independent metrics and a reasonable ablation. The reviewers should ask for an independent test oracle (human-written or at least a random audit) and error bars on the execution-test numbers. For me, it's a solid application-and-dataset paper, not a paradigm shift.","headline":"A genuinely useful dialogue dataset for Fortran-to-C++ translation, with a real ablation showing dialogue data beats code pairs—but the execution-test metric is weaker than it looks because the oracle is the same LLM that generated the data.","tokens_in":14371,"tokens_out":2675,"would_cite":true,"duration_ms":25763,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning open-weight LLMs on multi-turn dialogue data achieves up to a 3.31x CodeBLEU improvement and a 92% increase in compilation success for Fortran-to-C++ translation.","keywords":["Fortran-to-C++ translation","multi-turn dialogue dataset","LLM agent","Questioner-Solver","fine-tuning open-weight LLMs","CodeBLEU","low-resource language translation","high-performance computing"],"falsifier":"Run a random sample of the Fortran2CPP-verified translation pairs against an independent, hand-written test suite that probes array-index boundaries, integer overflow, floating-point rounding, and I/O ordering; if a substantial fraction of pairs that passed the LLM-generated tests fail the independent suite, the functional-equivalence oracle collapses and the reported reliability gains overstate true correctness.","tokens_in":13320,"feed_emoji":"🤖","tokens_out":15223,"duration_ms":107976,"temperature":0.7,"pith_summary":"The paper claims that fine-tuning open-weight large language models on a multi-turn dialogue dataset improves automatic Fortran-to-C++ translation, the kind of low-resource code migration needed to modernize high-performance computing applications. The Fortran2CPP dataset records every step of an LLM-agent pipeline—initial translation, unit test generation, compilation, execution, error fixing, and final verification—mediated by a Questioner-Solver pair of LLMs, and is released publicly. When three open-weight models (DeepSeek-Coder 6.7B, CodeLlama 13B, StarCoder 15.5B) are fine-tuned on the resulting 11.7k dialogue turns, the paper reports up to a 3.31x increase in CodeBLEU and large gains in compilation and execution pass rates across two evaluation benchmarks. If true, this means translation quality can be improved by training on the process of translation rather than only on verified code pairs, a result that could extend to other low-resource programming languages.","feed_headline":"Dialogue fine-tuning boosts Fortran-to-C++ CodeBLEU up to 3.31x","feed_subtitle":"Open-weight models trained on 11.7k translation dialogues reach 0.92 compilation success and surpass GPT-4 on CodeBLEU.","key_machinery":"The central machinery is the Questioner-Solver module, a dual-LLM agent core in which one LLM (the Questioner) inspects the current memory and environmental tool feedback and decides what to ask, while the other (the Solver) plans and executes translation, error repair, test generation, and tool calls. This division of labor produces a multi-turn dialogue that is saved and then split into cumulative prompt-response pairs to form the Fortran2CPP training dataset. The module's iterative feedback loop—compilation, execution, unit testing, and retry—is what embeds compiler diagnostics and error-fixing steps into the training data, and the paper's ablation shows that this dialogue version outperforms fine-tuning on the same underlying code pairs alone.","core_discovery":"The central discovery is that process-level dialogue data generated by a dual-LLM agent pipeline carries training signal that plain Fortran-C++ code pairs do not. The pipeline saves the full interaction history between a Questioner, which reads the current memory and tool feedback and asks the next question, and a Solver, which plans and executes translation, compilation, test generation, and repair actions. Those histories are split into cumulative prompt-response pairs, turning 1.2k dialogues into 11.7k training turns. Fine-tuning DeepSeek-Coder 6.7B, CodeLlama 13B, and StarCoder 15.5B on these dialogues yields consistent improvements over both their untuned versions and versions fine-tuned only on code pairs, with the largest CodeBLEU gain reaching 3.31x on the HumanEval-Fortran2Cpp benchmark and compilation success rising to 0.92 for CodeLlama. The paper interprets this as evidence that dialogue-based training embeds compiler diagnostics, error-resolution strategies, and iterative refinement behavior into the model.","pith_inferences":["If the dialogue-training effect is as general as claimed, the same Questioner-Solver pipeline could be applied to other low-resource translation pairs, such as legacy COBOL to Java or MPI-to-OpenMP refactoring, wherever compilation and execution feedback loops are available.","The reported gains may partly reflect the dataset's emphasis on the specific compiler and runtime errors common in Fortran-to-C++ translation; whether the trained models handle genuinely novel error types is not established by the paper.","A stricter oracle, such as property-based differential testing between the Fortran and C++ programs, would likely change the pipeline's acceptance decisions and the measured pass rates, and the direction and size of that change remain open empirical questions.","A token-matched comparison of dialogue fine-tuning versus simply adding more verified code pairs would isolate whether the process signal, rather than the extra training data itself, drives the improvement."],"forward_implications":["Fine-tuning on the dialogue dataset yields better translation quality than fine-tuning on the same underlying code pairs alone, as shown by the ablation across all metrics and both benchmarks.","Gains transfer from the HPC-Fortran-Cpp benchmark to the more diverse HumanEval-Fortran2Cpp benchmark, indicating the dataset teaches transferable Fortran-to-C++ patterns rather than overfitting one corpus.","A 13B open-weight model (CodeLlama) can match or exceed GPT-4 Turbo on CodeBLEU and compilation success while remaining open-weight and much smaller.","The dialogue generation pipeline scales with seed code size and iteration count, so larger datasets can be produced with the same Questioner-Solver workflow subject to compute and time constraints.","The released dataset and models allow independent reproduction and further fine-tuning for other code translation or migration tasks."],"supporting_citations":[{"why":"Supplies the HPC-Fortran-Cpp benchmark of 315 manually curated OpenMP Fortran-C++ pairs used as the primary evaluation dataset.","marker":"Lei et al. (2023)"},{"why":"Provides the HumanEval-X benchmark that the paper extends with Fortran implementations to create the HumanEval-Fortran2Cpp evaluation set.","marker":"Zheng et al. (2023)"},{"why":"Supplies the 80,000 seed Fortran files from which the pipeline generates the Fortran2CPP code pairs and dialogue dataset.","marker":"CodeParrot (2024)"},{"why":"DeepSeek-Coder is one of the three open-weight base models fine-tuned and evaluated.","marker":"Guo et al. (2024a)"},{"why":"CodeLlama is one of the three open-weight base models fine-tuned and evaluated.","marker":"Roziere et al. (2023)"},{"why":"StarCoder is one of the three open-weight base models fine-tuned and evaluated.","marker":"Li et al. (2023)"},{"why":"GPT-4 Turbo is the proprietary state-of-the-art baseline and generates the unit tests used in the Execution Test metric.","marker":"OpenAI (2024)"},{"why":"Supports the premise that LLM-generated unit tests can serve as a validation oracle, which the pipeline and evaluation rely on.","marker":"Chen et al. (2024b)"}],"fun_headline_variants":["Dialogue fine-tuning yields 3.31x CodeBLEU gain in Fortran-to-C++ translation","Fortran-to-C++ translation improved 3.31x with agent-crafted dialogue data","LLM dialogue training boosts Fortran-to-C++ CodeBLEU and compilation success","Multi-turn dialogue makes LLMs 3.31x better at Fortran-to-C++ translation","Fortran2CPP: Dialogue-driven LLM translation achieves 3.31x CodeBLEU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the unit tests generated by LLMs correctly determine whether the translated C++ code is functionally equivalent to the original Fortran program; if those tests miss important behaviors, the dataset's acceptance decisions and the reported execution-test improvements are both inflated.","fun_headline_variants_meta":{"raw":{"variants":["Dialogue fine-tuning yields 3.31x CodeBLEU gain in Fortran-to-C++ translation","Fortran-to-C++ translation improved 3.31x with agent-crafted dialogue data","LLM dialogue training boosts Fortran-to-C++ CodeBLEU and compilation success","Multi-turn dialogue makes LLMs 3.31x better at Fortran-to-C++ translation","Fortran2CPP: Dialogue-driven LLM translation achieves 3.31x CodeBLEU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000377,"raw_usage":{"total_tokens":2037,"prompt_tokens":1004,"completion_tokens":1033,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":913}},"tokens_in":620,"tokens_out":1033,"duration_ms":8244,"temperature":1.0,"reasoning_tokens":913,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:51:49.207282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a random sample of the Fortran2CPP-verified translation pairs against an independent, hand-written test suite that probes array-index boundaries, integer overflow, floating-point rounding, and I/O ordering; if a substantial fraction of pairs that passed the LLM-generated tests fail the independent suite, the functional-equivalence oracle collapses and the reported reliability gains overstate true correctness.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the HumanEval-X benchmark that the paper extends with Fortran implementations to create the HumanEval-Fortran2Cpp evaluation set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 80,000 seed Fortran files from which the pipeline generates the Fortran2CPP code pairs and dialogue dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4 Turbo is the proprietary state-of-the-art baseline and generates the unit tests used in the Execution Test metric."}],"review_version":1}