{"id":"6f2ee696-b36a-416d-876d-89a74b05e955","arxiv_id":"2506.10984","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A step-by-step framework combining LLM code reasoning and generation with human review is applied to modernize a legacy Java application, but its effectiveness is supported only anecdotally.","lead":"This paper proposes a human-in-the-loop framework that uses open-source LLMs to generate requirements from legacy Java/.NET applications and then build a modernized application step by step. A case study on Spring PetClinic demonstrates the workflow, though without quantitative evaluation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The framework's central claim rests on unvalidated LLM-generated requirements; no ground-truth comparison exists in the case study, and the only automated check is self-cited and conceded unreliable.","rationale":"The reader identifies reverse generation verification as the weakest assumption, and I agree that this self-cited LLM self-comparison is unvalidated. My concern goes one step earlier in the pipeline: the requirements generated in Section 3.1 are never checked against ground-truth behavior, so even an honest human review has no independent oracle to catch omissions or hallucinations. The paper's own limitations section (Section 5) confirms the fragility of automated verification and the ad hoc nature of error handling, which supports rather than repairs the evidentiary gap. I do not see a need to change the reader's verdict: the rejection stands because the central claim is unsupported by the evidence presented, not because the framework is demonstrated to be incorrect. A focused ground-truth study would be the natural next step and could, in principle, rehabilitate the claim.","tokens_in":6517,"tokens_out":3293,"duration_ms":35675,"concrete_test":"Select 5–10 open-source legacy Java/.NET applications with existing behavior-level test suites. Run the framework's requirements-generation phase; have two independent subject-matter experts classify each generated requirement as correct, incorrect, or hallucinated against the application's actual behavior, computing precision and recall. Then generate the modernized application and run a translated version of the original test suite against it, reporting pass rate. If requirement recall is not near-perfect or the translated-test pass rate is not comparable to a manual modernization baseline, the effectiveness claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 makes the framework's foundational move: derive functional requirements from legacy code and use them as the basis for the modernized application. Correctness of every later artifact—ORM, API, tests, UI—depends on these requirements. The paper never measures whether those requirements match actual system behavior. The case study (Section 4.1) uses Spring PetClinic, which has known behavior, but does not compare generated requirements against it; instead, the author reviews and edits them, and the only automated consistency check (Section 3.5, 'Reverse Generation Verification') is cited to the author's own prior work [12] and is an LLM self-comparison. Section 5(d) explicitly concedes that reliable automation of this check is challenging and that the method goes little beyond text-similarity scores. Human review is assumed to catch errors, but no inter-rater agreement or error-rate metric is reported. Therefore the central claim that the framework effectively addresses reliability, security, and quality is not established at the point where correctness is first determined; a hallucinated or incomplete requirement propagates through generation, testing, and security review. This is a load-bearing gap, not a stylistic one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a framework for modernizing legacy Java/.NET applications using large language models (LLMs) by combining code reasoning and code generation with human oversight. The framework generates functional requirements from the legacy code, reviews and refines them with subject-matter experts, and then generates the modernized application layer by layer (data, business logic, interaction), using progressive prompting and a reference implementation tool. A qualitative case study on the Spring PetClinic application is presented, along with a list of observations and limitations. The central claim is that this approach effectively addresses reliability, security, and quality challenges in application modernization.","tokens_in":6738,"tokens_out":3101,"duration_ms":31657,"significance":"If substantiated, the framework would offer a low-cost, locally deployable, standardized alternative to ad-hoc LLM-assisted modernization, with transparency-oriented model selection and built-in human review. The paper makes a concrete contribution by describing an opinionated, tool-embedded workflow and by releasing an open-source reference implementation, which lowers the barrier to experimentation. However, the current evidence is anecdotal: the case study reports no metrics, baselines, or error analysis, and the principal automated verification method is self-cited and, by the manuscript's own admission, not reliably automatable. The self-referential character of the reference implementation (generated by the framework it demonstrates) further limits the evidentiary weight. The framework is plausibly useful, but the paper does not yet establish the claimed effectiveness.","major_comments":[{"comment":"The foundational step of requirements generation is not validated against ground truth. The case study uses Spring PetClinic, whose actual behavior is publicly known, yet the paper never compares the LLM-generated functional requirements to the real behavior of the application. The only check is the author's review and editing, with no error-rate measurement, inter-rater agreement, or independent validation. Since every later artifact (data model, ORM, API, tests, UI) inherits the correctness of these requirements, a hallucinated or incomplete requirement propagates through the entire modernization. This directly undermines the central claim that the framework effectively addresses reliability.","section":"Section 4.1"},{"comment":"Reverse Generation Verification is the only automated consistency check proposed, but it is cited to the author's own prior work [12] and is not independently validated in this manuscript. Section 5(d) explicitly concedes that creating a reliably automated process is challenging and goes little beyond text-similarity scores. Using this method as evidence of output quality is therefore circular: the validity of the check is assumed rather than demonstrated. The manuscript should either provide independent evidence of the method's reliability or significantly weaken the claim that it addresses quality assurance.","section":"Sections 3.5 and 5(d)"},{"comment":"The case study reports no quantitative outcomes: no success rates, defect counts, time measurements, security scan results, or comparisons against direct code translation or other modernization baselines. The abstract promises 'assessing alternative approaches where applicable,' but the case study only documents the sequence of steps and observations. Moreover, Section 5(b) admits occasional random behavior (incomplete or misaligned responses) and Section 5(e) admits random syntax errors across all models. These limitations directly contradict the abstract's implication that the framework tackles reliability challenges effectively. The paper needs a measured evaluation to support that claim.","section":"Section 4 (case study overall)"},{"comment":"The reference implementation tool was itself generated using the framework with minimal human intervention, and is then used in the case study as evidence of the framework's capability. This is a self-referential demonstration: the tool both embodies the framework and is cited as proof of it. Without an independent implementation or external evaluation, the case study cannot distinguish whether observed successes are due to the framework's methodology, the specific LLM, the author's manual corrections, or the particular application chosen.","section":"Section 3.4"}],"minor_comments":[{"comment":"The heading contains a typo: 'The Framework and the Tooll' should be 'The Framework and the Tool.'","section":"Section 3 heading"},{"comment":"The phrase 'It's worth nothing' appears to be a typo; it should read 'It's worth noting.'","section":"Section 3.1"},{"comment":"The section header '6 V. CONCLUSION' includes an extra 'V.' and should simply be '6 Conclusion.'","section":"Section 6"},{"comment":"The figures referenced in the case study (Figures 5-7) are not described in the running text; please ensure each figure is legible, captioned, and accompanied by a brief explanation of what the reader should observe.","section":"Figures 5-7"},{"comment":"The scope statement excludes COBOL and Fortran, but reference [1] is about Fortran-to-Python transformation; clarifying how that reference relates to the stated scope would avoid confusion.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is more of an experience report than a validated research study. The strongest concerns are the absence of any quantitative evaluation and the double circularity: the verification method is self-cited, and the reference implementation is both the product of the framework and its demonstration. I would require a substantive evaluation with metrics and independent validation before considering publication. The manuscript's framing in the abstract and introduction overstates what the current evidence supports."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is an honest practitioner paper that overclaims in its abstract. What's actually new: not much algorithmically—progressive prompting and human-in-the-loop are known, and the reverse-generation verification is self-cited from the author's prior work. The value is in the packaging: a concrete step-by-step framework for modernizing Java/.NET apps with local open models, a reference tool on GitHub, and a worked Spring PetClinic case study. The paper is refreshingly candid about failures: random behavior, syntax errors, and the limits of reverse generation verification (Section 5d).\n\nThe soft spot is the one the stress-test names. The whole pipeline starts from LLM-generated requirements, and the paper never checks those requirements against the actual behavior of the legacy application. For PetClinic, known behavior exists, but no comparison is reported. Human review is assumed to catch problems, but no error rates or inter-rater checks are given. So the central claim—that the framework 'effectively addresses reliability, security, and quality'—is not established at the point where correctness is first determined. A hallucinated requirement propagates through ORM, API, tests, and UI. The limitations section concedes the verification method is unreliable, which undercuts the case.\n\nI don't think this is a takedown. The author is transparent about scope: the tool is not production-ready, and the paper reads more like an experience report than a validated research result. As a position paper for practitioners, it has some value. But as a research contribution, the evidence is missing.\n\nMy recommendation: if this lands at a research venue, I'd suggest major revision or rejection, with the main ask being an evaluation that compares generated requirements against ground truth and reports error rates at each step. If the venue accepts practitioner experience reports, I'd send it to review—it is clear, honest, and the tool is a real artifact.","headline":"Honest practitioner framework for LLM modernization, but the central validation gap makes the reliability claim unproven.","tokens_in":7231,"tokens_out":1550,"would_cite":false,"duration_ms":16730,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a requirements-first, human-checked workflow, built on LLM code reasoning and code generation with progressive prompting, can make modernizing legacy Java and .NET applications reliable, secure, and affordable.","keywords":["application modernization","large language models","progressive prompting","code reasoning","code generation","reverse generation verification","human-in-the-loop","legacy systems"],"falsifier":"A concrete test: run the framework on several legacy Java/.NET modules, plant known defects in the generated outputs, and measure how often reverse generation verification or the secondary verification LLM flags them while text-similarity scores remain high. If the checks miss planted defects, the central reliability claim fails.","tokens_in":6328,"feed_emoji":"🤖","tokens_out":8166,"duration_ms":79309,"temperature":0.7,"pith_summary":"The paper argues that reliability, security, and quality problems in LLM-assisted modernization are best solved by process structure, not by a stronger model. Its framework first uses an LLM to extract functional requirements from three layers of a legacy Java or .NET application, then rebuilds the application in reverse layer order, with fixed prompts and human approval at each step. Quality is checked incrementally by humans, by feeding generated code back into the LLM to see whether it regenerates the original requirements, and optionally by comparing outputs from a second verification LLM. If this works as claimed, enterprise teams could modernize legacy systems with open, locally deployable models, without fine-tuning, prompt-engineering skill, or sending code to cloud services.","feed_headline":"Requirements-first LLM workflow delivers reliable legacy modernization","feed_subtitle":"Rebuild apps layer by layer, verify each output by regenerating requirements, with human review at every step.","key_machinery":"The load-bearing mechanism is progressive prompting: a sequence of small generation steps in which each prompt consumes the previous artifact, such as requirements yielding a data model, the data model yielding ORM objects, those yielding APIs, and those yielding tests and user interfaces. Requirements extraction goes through the interaction layer, business-logic layer, and data layer in that order, while code generation runs the same layers in reverse so each layer builds on a stable foundation. Quality is enforced by a human-centered approach—step-by-step verification, detailed explanations, and manual integration—augmented by reverse generation verification, where the generated artifact is re-entered into the LLM to regenerate its intended input and the two are compared, and by an optional second verification LLM that replicates the generation for comparison.","core_discovery":"On the paper's own terms, the central claim is that an opinionated pipeline, rather than a better model alone, solves the reliability, security, and quality problems of LLM-based application modernization. The pipeline extracts functional requirements from the legacy code's interaction, business-logic, and data layers, then rebuilds the target application in reverse layer order, starting with data, continuing with business logic, and finishing with the interaction layer. Fixed, embedded prompts drive each incremental generation step, and humans review every artifact before it feeds the next step. Quality assurance relies on human-centered verification plus reverse generation verification, in which generated code is fed back into the LLM to see whether the original requirements reappear. The paper demonstrates the workflow on a Java sample application, adding a new feature during modernization, and reports that the approach is practical with open models on modest local hardware.","pith_inferences":["If the framework generalizes beyond the single case study, the most valuable next test is a controlled comparison: same model and target framework with and without the requirements-first pipeline, measuring defect density, security findings, and review time.","The paper's observations about output variation suggest that reverse generation verification could become an automated CI gate only after calibrating a reliable similarity threshold, a step the paper does not provide.","The embedded-prompt design predicts a measurable benefit: output consistency across users should be higher than with free-form AI chat assistants, since prompt phrasing is fixed; the paper does not report such a measurement.","Because the case study adds new features during modernization, the framework implicitly claims that requirements-first work is an opportunity to redesign rather than merely translate; this could be tested by tracking which legacy behaviors are preserved versus deliberately changed and whether regressions appear in changed areas."],"forward_implications":["Open, locally deployed code models can handle the entire modernization workflow, so organizations can avoid fine-tuning and cloud dependency while keeping code in air-gapped environments.","Fixed embedded prompts reduce user-dependent variability, making generated artifacts more consistent and maintainable than outputs from general AI assistant chat.","Because requirements are extracted from the actual code and reviewed before development, teams can fix latent bugs and add features during modernization rather than only translating code.","Reverse generation verification is the more economical quality check compared with running a second verification LLM, though automating it consistently remains difficult.","The generated requirements may not match standard requirements templates, but they can be converted manually or automatically, so the workflow can still feed existing enterprise processes."],"supporting_citations":[{"why":"supplies the reverse generation verification method that anchors the framework's quality assurance.","marker":"[12]"},{"why":"introduces the requirements-driven, progressive prompting style the framework builds on.","marker":"[3]"},{"why":"documents limitations of direct LLM code translation for legacy systems, motivating the requirements-first approach.","marker":"[1]"},{"why":"supports the idea that LLM code reasoning can produce useful documentation and modernization insights for legacy code.","marker":"[2]"},{"why":"provides security and efficiency best practices that the framework's human-centered and verification steps incorporate.","marker":"[4]"},{"why":"transparency about training data is the stated reason for choosing the open code model over alternatives.","marker":"[5]"},{"why":"supplies the architecture reasoning that decoder-only models are not preferred, informing model choice.","marker":"[7]"},{"why":"quantifies the security risk of AI assistants that the framework's human verification is meant to offset.","marker":"[11]"}],"fun_headline_variants":["Opinionated pipeline, not model size, fixes LLM modernization","Layer-by-layer rebuild with human checks yields reliable LLM code","Reverse generation verifies LLM modernization work","Human-reviewed LLM pipeline for trustworthy legacy modernization","Requirements-first workflow: LLMs modernize apps reliably"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quality story leans on reverse generation verification, the assumption that asking an LLM to regenerate requirements from generated code and comparing the two sets is a reliable check of correctness; the paper cites this check to the author's earlier work and does not independently validate it here.","fun_headline_variants_meta":{"raw":{"variants":["Opinionated pipeline, not model size, fixes LLM modernization","Layer-by-layer rebuild with human checks yields reliable LLM code","Reverse generation verifies LLM modernization work","Human-reviewed LLM pipeline for trustworthy legacy modernization","Requirements-first workflow: LLMs modernize apps reliably"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1522,"prompt_tokens":924,"completion_tokens":598,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":520}},"tokens_in":540,"tokens_out":598,"duration_ms":6582,"temperature":1.0,"reasoning_tokens":520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:19:57.062470+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: run the framework on several legacy Java/.NET modules, plant known defects in the generated outputs, and measure how often reverse generation verification or the secondary verification LLM flags them while text-similarity scores remain high. If the checks miss planted defects, the central reliability claim fails.","supporting_citations":[{"cited_title":"Requirements are all you need: From requirements to code with llms,","cited_arxiv_id":null,"evidence_quote":"introduces the requirements-driven, progressive prompting style the framework builds on."},{"cited_title":"Bridging eras: Transforming fortran legacies into python with the power of large language models,","cited_arxiv_id":null,"evidence_quote":"documents limitations of direct LLM code translation for legacy systems, motivating the requirements-first approach."},{"cited_title":"Leveraging llms for legacy code modernization: Challenges and opportunities for llm-generated documentation,","cited_arxiv_id":null,"evidence_quote":"supports the idea that LLM code reasoning can produce useful documentation and modernization insights for legacy code."},{"cited_title":"Optimizing ai-assisted code generation: En- hancing security, efficiency, and accessibility in software development,","cited_arxiv_id":null,"evidence_quote":"provides security and efficiency best practices that the framework's human-centered and verification steps incorporate."},{"cited_title":"Ibm granite transparency report,","cited_arxiv_id":null,"evidence_quote":"transparency about training data is the stated reason for choosing the open code model over alternatives."},{"cited_title":"Codet5: Identifier-aware uni- fied pre-trained encoder-decoder models for code understanding and gen- eration,","cited_arxiv_id":null,"evidence_quote":"supplies the architecture reasoning that decoder-only models are not preferred, informing model choice."},{"cited_title":"Lost at c: A user study on the security implications of large language model code assistants,","cited_arxiv_id":null,"evidence_quote":"quantifies the security risk of AI assistants that the framework's human verification is meant to offset."}],"review_version":1}