{"id":"e4b958f1-4e9b-4761-90d2-e242a9d79162","arxiv_id":"2412.08035","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A modular LLM pipeline with feature-mapping rules and type-compatibility checks reports 73% of functions from seven Go projects validated I/O equivalent in Rust, on projects up to 6,600 lines of code.","lead":"This paper presents Oxidizer, a tool that translates whole Go projects into Rust by splitting code into small pieces, translating each piece with an LLM, applying hand-written rules for tricky language differences, and checking behavior against I/O examples from the original test suite.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 73% I/O-equivalence rate is measured only against execution snapshots from the unit tests, so it does not establish correctness on untested inputs; differential fuzzing would settle whether the validation generalizes.","rationale":"The reader's weakest assumption is precisely the load-bearing concern I would identify: the execution snapshots define both the type-compatibility universe (Definition 1) and the I/O-equivalence universe (Definition 5), and the paper reports statement coverage as low as 43.2%, with no per-function snapshot counts. The strongest claim in the abstract—'reliable Rust translations' with '73% of functions successfully validated for I/O equivalence'—depends entirely on those snapshots being representative. If the test suite is sparse or unrepresentative, the validation rate can be high while many translated functions are wrong on real inputs. This is not an internal inconsistency: the paper explicitly defines equivalence with respect to V and marks uncovered functions as failing. The concern is about how much weight the headline claim can bear, which is exactly what the reader's conditional verdict captures. A differential fuzzing check would settle whether the V-relative validation generalizes. I see no other concern that is more load-bearing: the architecture is clearly described, the feature-mapping and type-compatibility ideas are plausible, and the comparison with prior work, while not head-to-head, is appropriately caveated. Therefore the reader's CONDITIONAL verdict should remain unchanged.","tokens_in":23951,"tokens_out":7806,"duration_ms":86209,"concrete_test":"For each function reported as I/O equivalent in Table 2, generate a held-out set of inputs outside the execution snapshots—for example, 1,000 random inputs respecting the type-compatible domain via property-based generation or fuzzing—and compare the serialized outputs and errors of the original Go function against the Rust translation. If any mismatch occurs, the 73% equivalence rate overstates semantic reliability and the concern lands; if no mismatches occur across all seven benchmarks, the snapshot-based validation is stronger than coverage alone suggests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Definition 5 (§6.1), which defines I/O equivalence 'with respect to V', where V is the set of input/output/error tuples collected by running the source project's unit tests (§7.1.1). The same snapshots define the feasible value set used in the type-compatibility definition (Definition 1, §5.2). Because V is finite and test-derived, every reported equivalence rate is a pass rate on a potentially small sample. Section 7.1.1 states that functions without collected examples receive an automatically failing unit test, and Table 1 reports statement coverage as low as 43.2% (histogram). Thus a function can be wrong on any input outside V and still count as 'validated'. The abstract's phrase 'reliable Rust translations' is therefore stronger than what the validation actually establishes: the paper shows that 73% of functions match the original on their test-suite inputs, not that 73% are semantically equivalent on all inputs. The paper is internally consistent because it states the V-relative definition, but the headline metric is easy to over-read, and the per-function snapshot counts are not reported, so a function with one easy example contributes as much as a function with thousands. Additionally, Definition 5 compares serialized return values and errors; side effects are folded into an 'extended output' without a fully described collection mechanism, so even within V, observable state changes may be missed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Oxidizer, a tool for translating entire Go projects to Rust using LLMs. The approach partitions a project into fragments (functions, types, globals), orders them by a dependency graph, and translates each fragment with the help of (i) hand-written feature mapping rules that constrain the LLM's handling of Go/Rust language differences, and (ii) type-compatibility checks that compare function and type signatures against execution snapshots collected from the project's own unit tests. A second phase checks I/O equivalence on those snapshots and repairs failing functions while freezing signatures. The evaluation on seven open-source Go projects reports that, on average, 99% of the code compiles and 73% of functions are validated as I/O equivalent, with ablation experiments suggesting that feature mapping is essential for progress and that type-compatibility improves equivalence rates.","tokens_in":24268,"tokens_out":4106,"duration_ms":43450,"significance":"If the reported numbers are taken at face value, this is a strong result for whole-project LLM-based translation: it substantially exceeds the equivalence rates reported by parallel work and, unlike earlier snippet-level approaches, it scales to a 6.6K-line / 369-function project. The two proposed mechanisms, feature mapping and type-compatibility, are clearly described and the ablation supports their importance. The paper also ships concrete reproducibility aids: it logs LLM inputs and outputs and can replay translation runs deterministically, and the feature mapping rules are specified enough to be re-implemented. The main value is in showing that a hybrid of symbolic rules, type-level checks, and modular LLM translation can make whole-project translation practical. The evaluation is honest about the V-relative nature of the validation in the body of the paper, but the abstract and headline metrics are easy to over-read as semantic equivalence on all inputs, and the reported rates do not separate mocked functions from genuinely translated ones.","major_comments":[{"comment":"The I/O equivalence relation is defined 'with respect to V', where V is the set of input/output/error tuples collected by running the source project's unit tests, and the paper states that functions without collected examples receive an automatically failing unit test. Consequently, the headline '% Equivalent' is a pass rate on the project's own test-suite inputs, not a statement of semantic equivalence on all inputs. A function can be wrong on any input outside V and still be counted as validated. The abstract's phrase 'reliable Rust translations' is therefore stronger than what the validation establishes. Please either qualify the abstract and conclusion to state that validation is on test-suite-derived examples, or add a held-out evaluation (e.g., differential fuzzing or a second test set) to show that the 73% rate generalizes. In addition, Table 1's statement coverage ranges from 43.2% to 100%, so per-benchmark and per-function snapshot counts should be reported so the reader can see how many examples underpin each function's validation.","section":null},{"comment":"The reported '% Compiled' and '% Equivalent' figures do not disclose how many functions in each benchmark ended up as mocks, i.e., functions whose bodies are replaced by a call to the original Go function through the Go-Rust boundary. A mocked function compiles by construction and is trivially I/O equivalent to the original, so including mocks in both metrics inflates the headline numbers and conflates 'translated and validated' with 'not actually translated'. The text admits that the output 'may have some function/method bodies replaced with mocks' (§7.1.1), but Table 2 gives no mock counts. Please report for each benchmark the number of functions that were mocked, and give the % Equivalent computed both including and excluding mocks. Without this breakdown, the central claim that 73% of functions were successfully translated and validated cannot be assessed.","section":null},{"comment":"The I/O equivalence check for functions compares serialized return values and errors, but the definition overloads the output y' to be 'an extension of the actual output that accounts for possible side-effects'. The paper never specifies how side effects are collected in the execution snapshots or how they are compared between Go and Rust. If side effects (mutations to receiver fields, global variables, I/O, or other observable state) are not captured in the snapshots, then two functions that differ only in such state will be incorrectly reported as equivalent. Please provide the concrete collection mechanism for side effects, or explicitly state that the validity of the equivalence check is limited to return values and errors, and adjust the claims accordingly.","section":null},{"comment":"The benchmark selection is restricted to projects that 'only make use of Go standard libraries', and the paper notes that third-party libraries were deliberately excluded from the evaluation. This limits the generalizability of the claim that Oxidizer translates 'real-world Go codebases': a substantial fraction of real Go projects depend on third-party packages, and the authors even acknowledge that their approach supports such dependencies (§5.1) but do not demonstrate it. Please state explicitly that the reported results are for a curated subset of Go projects without third-party dependencies, and discuss what additional validation would be needed to support the broader 'entire project' claim.","section":null}],"minor_comments":[{"comment":"The formal rule notation in Figures 7 and 10 is difficult to read because of rendering artifacts (e.g., 'D/uni∈1A6.endl→code' and the ⇓/↝ symbols appear corrupted in the PDF). Please re-set these judgments in clean LaTeX so the premises and conclusions are legible.","section":null},{"comment":"The round-tripping property for JSON serialization is stated as an assumption, but some Go types (e.g., channels, function values, cyclic data structures, or fields with unexported components) are not naturally JSON-serializable. Please state which types are assumed to be serializable and how the presented benchmarks avoid these cases.","section":null},{"comment":"The logging-and-replay mechanism is a strong reproducibility feature, but the paper does not point to a public artifact or repository. Please include an artifact URL or a clear statement of availability, along with the exact prompt templates and version of Claude 3 Sonnet used.","section":null},{"comment":"The sentence 'in one case by 144%' is ambiguous: an increase from 29% to 71% can be described as a 144% relative improvement, but the reader may misread it as 144 percentage points. Please restate with the actual before/after numbers.","section":null},{"comment":"The notation D_go(S_go(x)) is used to describe input conversion, but the serialization/deserialization functions S and D are introduced only in §5.2; a forward reference or a brief restatement would help the reader.","section":null},{"comment":"The claim of being 'considerably higher than any existing work' is based on comparing reported numbers from parallel papers rather than running those tools on the same benchmarks. Please soften this to 'higher than previously reported' or add a direct comparison on a shared benchmark set.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems/SE contribution with a clear, plausible mechanism and a useful ablation. My main concern is not the mechanism but the transparency of the evaluation metrics: the reader cannot tell how many functions are mocks, how many snapshots back each equivalence decision, or how side effects are captured. I believe the authors can fix this with additional reporting and careful claim softening, and I would be willing to review a revised version. I would also encourage the editor to ask for the artifact to be made available, since the logged-replay design makes exact reproduction feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine advance in whole-project LLM translation, with a clear architecture and honest ablations, but the headline 73% I/O-equivalence rate is not fully interpretable because the paper never says how many functions are mocks, and \"validated\" means \"matches on the test-suite snapshots.\"\n\nThe new things that are real: feature mapping rules that constrain the LLM with pre-specified translation patterns, and type-compatibility checks that catch signature-level mismatches early using serialization round-trips. The interface-to-trait decomposition (splitting Go interfaces into subtraits to preserve structural subtyping) is a nice concrete trick. The whole-project scale (6.6K LoC, 369 functions) is beyond the 100-line snippets in most prior work. The ablation shows feature mapping is load-bearing: without it, translation aborts and nothing validates.\n\nThe soft spots, in order of severity. First, the mock problem: when type-driven translation fails after max tries, the tool replaces a function with a stub that calls the original Go code via FFI. Such mocks will trivially pass any I/O equivalence check. Table 2 does not report how many functions ended up as mocks, so the 73% average could include an unknown number of non-translations. This is a real confound, not a nitpick. Second, I/O equivalence is defined relative to the finite set of I/O examples collected from the project's unit tests (Definition 5, Section 6.1). Functions without examples automatically fail, which is conservative, but the examples are whatever the tests happen to exercise; a function can be wrong on untested inputs and still count as validated. The abstract's \"reliable Rust translations\" overclaims. Third, side effects are folded into an \"extended output\" that is never formally defined; I can't tell if mutations to globals or pointer-referenced state are checked. Fourth, the benchmarks are deliberately limited to projects using only Go standard libraries, and there is no released artifact or code, despite a claim that LLM logs can be replayed.\n\nNone of this kills the paper. The internal definitions are clear, the ablations support the importance of feature mapping, and the comparison to parallel work is honestly framed. The fix is reporting mock counts and providing the artifact, plus softening the abstract to say \"validated on unit-test inputs.\" This deserves a serious referee, but the referee should push for those numbers.","headline":"A real advance in whole-project LLM translation with a clear architecture, but the 73% validation rate is uninterpretable without knowing how many functions are mocks.","tokens_in":24769,"tokens_out":3640,"would_cite":true,"duration_ms":36169,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM translation scales to whole projects when fragments are translated in dependency order, guided by feature-mapping rules and signature-level type-compatibility checks.","keywords":["code translation","large language models","Go to Rust","I/O equivalence","type compatibility","feature mapping","modular translation","program repair"],"falsifier":"Take a function the pipeline reports as I/O-equivalent and run it on inputs outside the collected test-suite snapshots, for example randomly generated values of the same types or mutations of recorded inputs, then compare Go and Rust outputs; a divergence on any such input would show the validation is restricted to covered behavior rather than full equivalence.","tokens_in":23768,"feed_emoji":"🦀","tokens_out":6662,"duration_ms":67092,"temperature":0.7,"pith_summary":"The paper claims that LLM-based translation of entire software projects, not just small snippets, can be made reliable by cutting the project into small fragments, translating them in dependency order, and checking each fragment locally before moving on. To make those local checks effective, it introduces feature mapping, which pairs the LLM with predefined translation rules and static checks for language features that do not map directly, and type-compatibility, which verifies at function-signature level that every value the original program can actually produce can cross into the translated type and back. Applied to seven Go projects translated to Rust, the largest at 6,600 lines and 369 functions, the approach compiles about 99% of the translated code and validates I/O equivalence for an average of 73% of functions on input-output examples drawn from the original test suite. If correct, this means whole-repository translation with LLMs can be practical and mostly verified, rather than a source of compilable but semantically broken code.","feed_headline":"Whole-project Go-to-Rust translation passes 73% of functions","feed_subtitle":"Rule-guided translation and signature-level type checks compile 99% of code and verify 73% of functions.","key_machinery":"The carrying mechanism is a post-order traversal over a dependency graph of code fragments, where each fragment must pass feature-mapping checks and a type-compatibility check before the next fragment is translated. Feature-mapping rules are triples: a syntactic pattern that detects when a rule applies, a natural-language instruction given to the LLM, and static checks that the generated code uses the expected Rust construct. Type-compatibility is checked by serializing execution-snapshot values to JSON and requiring lossless round-trips through the target type, with function signatures checked the same way. This makes errors detectable at the place they are introduced, before they contaminate downstream fragments.","core_discovery":"The central claim is that the two obstacles to scaling LLM translation—unreliable mappings of source-language features and errors that cascade through interdependent fragments—can be handled by combining a small set of human-written translation rules with signature-level type-compatibility checks. The paper defines type-compatibility through execution snapshots: feasible values of a Go type are those observed when the project's own unit tests run, and a Rust type is compatible if each such value can be serialized to JSON, deserialized into the Rust type, serialized back, and deserialized into the original Go value unchanged. On top of this, feature-mapping rules tell the LLM how to render specific Go constructs such as global initialization, error returns, and structural interfaces in Rust, and the rules are enforced by static checks on the generated code. After a type-driven phase produces a compiling, type-compatible project, a semantics-driven phase checks each function for I/O equivalence on the same snapshots, mocking callees so failures are local. The reported outcome is that almost all source lines compile and, on average, 73% of functions are I/O-equivalent, with every failing test an assertion failure rather than a crash.","pith_inferences":["One implication the paper leaves implicit is that the equivalence rate is measured only on snapshots from the original test suite, so the 73% figure should be read as validated on covered behavior; the paper itself counts uncovered functions as automatically failing, and statement coverage ranges from 43.2% to 100%.","A natural strengthening would be to add differential fuzzing after the pipeline: the same JSON round-trip harness used for type-compatibility could feed random or mutated inputs to both versions, turning validation from example-based into property-based.","Because the tool logs LLM inputs and outputs and supports replaying them, the pipeline is deterministic for a fixed log, which makes the evaluation reproducible and makes future LLM improvements directly comparable on the same benchmarks."],"forward_implications":["Whole-repository translation becomes a viable strategy: the largest case has 6,600 lines and 369 functions, far beyond the roughly 100-line ceiling reported for direct LLM translation.","Because type-compatibility is checked before semantics, translation can proceed even when a function cannot be made to compile: it is mocked by calling the original Go function through a boundary, so the rest of the project is not blocked.","The same pipeline yields a regression test suite for the translated code, since every I/O-equivalent function has concrete input-output examples that can be replayed as Rust unit tests.","Failing unit tests in the translated projects are assertion failures rather than crashes, which lets the pipeline attribute each failure to a specific function and keep repairs local.","The approach is described as agnostic to the language pair, so the feature-mapping and type-compatibility machinery could be instantiated for other source and target languages, not only Go-to-Rust."],"supporting_citations":[{"why":"Documents the drop in LLM translation success beyond roughly 100 lines and motivates the modular, project-level approach.","marker":"[15]"},{"why":"Shows prior validated LLM translation was limited to small snippets, providing the baseline that whole-project translation extends.","marker":"[16]"},{"why":"Is the parallel whole-repository translation work reporting 25.8% average I/O equivalence, the main baseline the results are compared against.","marker":"[17]"},{"why":"Is the parallel C-to-Rust whole-project translation that reaches high compilation rates but whose translated tests mostly crash, motivating semantic validation.","marker":"[18]"},{"why":"Provides the multi-language boundary semantics that inspire the type-compatibility definition.","marker":"[19]"},{"why":"Supplies the unified Rust error type used by the error-handling feature mapping rules.","marker":"[22]"},{"why":"Supplies the compiler-error-driven repair approach reused for localized compilation repair of individual fragments.","marker":"[23]"}],"fun_headline_variants":["LLM code translation scales to whole projects with 73% validation","Feature mapping and type checks make LLM translation reliable at scale","73% of functions pass I/O checks in Go-to-Rust translation","Rule-guided translation verifies 73% of functions in large codebases","Whole-project Go-to-Rust: 73% functions verified"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument treats the input-output examples captured by the project's unit tests as defining the feasible values for type-compatibility and the universe for I/O equivalence; if those examples miss important inputs, a function counted as equivalent can still be wrong.","fun_headline_variants_meta":{"raw":{"variants":["LLM code translation scales to whole projects with 73% validation","Feature mapping and type checks make LLM translation reliable at scale","73% of functions pass I/O checks in Go-to-Rust translation","Rule-guided translation verifies 73% of functions in large codebases","Whole-project Go-to-Rust: 73% functions verified"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000762,"raw_usage":{"total_tokens":3429,"prompt_tokens":1036,"completion_tokens":2393,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":2301}},"tokens_in":652,"tokens_out":2393,"duration_ms":15194,"temperature":1.0,"reasoning_tokens":2301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:17:04.741190+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a function the pipeline reports as I/O-equivalent and run it on inputs outside the collected test-suite snapshots, for example randomly generated values of the same types or mutations of recorded inputs, then compare Go and Rust outputs; a divergence on any such input would show the validation is restricted to covered behavior rather than full equivalence.","supporting_citations":[{"cited_title":"Towards translating real-world code with LLMs: A study of translating to Rust,","cited_arxiv_id":null,"evidence_quote":"Documents the drop in LLM translation success beyond roughly 100 lines and motivates the modular, project-level approach."},{"cited_title":"VERT: Verified equivalent Rust transpilation with large language models as few-shot learners,","cited_arxiv_id":null,"evidence_quote":"Shows prior validated LLM translation was limited to small snippets, providing the baseline that whole-project translation extends."},{"cited_title":"Repository-level compositional code translation and validation,","cited_arxiv_id":null,"evidence_quote":"Is the parallel whole-repository translation work reporting 25.8% average I/O equivalence, the main baseline the results are compared against."},{"cited_title":"Context-aware code segmentatio n for C-to-Rust translation using large language models,","cited_arxiv_id":null,"evidence_quote":"Is the parallel C-to-Rust whole-project translation that reaches high compilation rates but whose translated tests mostly crash, motivating semantic validation."},{"cited_title":"Operational semantics for multi- language programs,","cited_arxiv_id":null,"evidence_quote":"Provides the multi-language boundary semantics that inspire the type-compatibility definition."},{"cited_title":"Anyhow","cited_arxiv_id":null,"evidence_quote":"Supplies the unified Rust error type used by the error-handling feature mapping rules."},{"cited_title":"RustAssistant: Using LLMs to fix compilation errors in Rust code,","cited_arxiv_id":null,"evidence_quote":"Supplies the compiler-error-driven repair approach reused for localized compilation repair of individual fragments."}],"review_version":1}