{"id":"914d78b6-3ef9-4d79-95c1-67eca4225d57","arxiv_id":"2509.05585","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding code dependencies, user feedback, and fine-grained text cues to a graph model or a large language model improves requirements-to-code traceability beyond text similarity alone.","lead":"Traceability link recovery, matching requirements to code, fails when systems compare only words, because requirements and code share little vocabulary. This paper tests that gap across 99 open-source projects and proposes two models that add structural and human hints to boost link recovery on 12 projects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation may be contaminated: user-feedback links sampled from ground truth are never excluded from test, so reported F1 gains may reflect label leakage.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the manuscript never describes a train/test split or states that the 10% user-feedback links are held out from evaluation. This is not a disagreement with the field's consensus; it is an internal evaluation-protocol risk. The paper's underlying idea, integrating auxiliary strategies such as code dependency, fine-grained similarity, and user feedback, is plausible, and the replication package and large project corpus are real supporting artifacts. However, the headline numbers are only meaningful if the feedback edges and prompt labels do not reveal test answers. Since the paper includes no explicit exclusion, the reported F1 improvements could be inflated by leakage. The table inconsistencies further weaken confidence that the exact reported numbers are reproducible, but they are secondary to the unresolved split question. If a clean check shows no leakage, the verdict could move toward acceptance; absent that check, the rejection stance is justified. No ad hominem is intended: the omission may be an oversight rather than deliberate, but it must be resolved before the central claim can be accepted.","tokens_in":41601,"tokens_out":4112,"duration_ms":40958,"concrete_test":"Inspect the released replication package (github.com/ZZYG0g0g0/NLPL-TLR) and verify the evaluation protocol: determine whether the 10% user-feedback pairs from Eq. 17 are excluded from the test labels for both HGT-All and Gemini-All, then rerun the experiments with feedback sampled only from a properly separated training portion and with all feedback pairs removed from the test set. Recompute the Table 7-9 F1 averages; if the gains over HGNNLink shrink by more than the claimed 3.68% and 8.84% margins, the headline claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is that multi-strategy integration raises F1 by 3.68% (HGT-All vs HGNNLink) and 8.84% (Gemini-All vs HGNNLink) across 12 projects. For that claim to be sound, the reported Precision, Recall, and F1 must be computed on links whose labels were not available to the model. The paper never establishes this. Section 4.3.1, Eq. 17, says HGT-All 'randomly sample[s] 10% of the links from the ground truth traceability set' and adds each sampled positive link as a user-feedback edge. Section 4.3.2 then inserts 'User feedback indicates label is 0 or 1' into the Gemini prompt as Additional_Information 2. No train/test split, no candidate set, and no statement that feedback links are held out from evaluation appear anywhere in Section 4. If a test positive is among the sampled 10%, HGT-All has a direct edge between exactly the pair it is asked to classify, and Gemini-All is handed the answer in its prompt. Because the sampled links are positive-only, the leak is one-directional and inflates recall and F1. Independent reproduction is also made difficult by table inconsistencies, e.g., HGT-All on Albergate is 0.5714 in Table 7 but 0.4914 in Table 9, and the Seam2/Pig rows of Table 9 are duplicated. The unresolved data-splitting question is the primary threat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that textual similarity alone is insufficient for natural language-to-programming language (NL-PL) traceability link recovery (TLR), and that auxiliary strategies are needed. To support this, the authors report a systematic literature review of 50 studies and 99 projects, propose a new metric called the Difference Ratio, and present a case study on requirements-to-code TLR. They integrate three auxiliary strategies (code dependency, user feedback, and fine-grained semantic similarity) into two models: a supervised Heterogeneous Graph Transformer (HGT-All) and a prompt-based Gemini 2.5 Pro model (Gemini-All). The abstract claims average F1-score improvements of 3.68% for HGT-All and 8.84% for Gemini-All over the state-of-the-art HGNNLink across twelve open-source projects. The paper also reports statistical significance tests, ablations, and includes a replication package.","tokens_in":41920,"tokens_out":6348,"duration_ms":56501,"significance":"If the empirical results were valid, the paper would make a useful contribution: it assembles a large project corpus, proposes a lightweight diagnostic metric, and demonstrates a principled way to inject domain knowledge into both graph-based and LLM-based TLR models. The replication package and the systematic organization of prior strategies are valuable resources for the community. However, the central quantitative claims are currently undermined by a likely label-leakage problem in the user-feedback strategy and by internal inconsistencies in the reported tables and observations. The paper's direction is plausible, but the evidence as presented does not support the headline numbers.","major_comments":[{"comment":"The evaluation protocol for the user-feedback strategy is not sound as described. Eq. (17) states that HGT-All randomly samples 10% of the links from the ground-truth traceability set and adds each sampled positive link as a feedback edge. For Gemini-All, Additional_Information 2 tells the model 'User feedback indicates label is 0 or 1'. The paper never specifies a train/test split, a held-out set, or a statement that feedback links are excluded from evaluation. If any evaluated positive pair is among the sampled 10%, HGT-All receives a direct edge between exactly that pair and Gemini-All receives the answer in its prompt. Because only positive links are sampled, the leak is one-directional and would inflate recall and F1. This directly affects the abstract's claimed 3.68% and 8.84% improvements over HGNNLink. The authors must either show that feedback links were held out or re-run the experiments with an explicit split that prevents the model from seeing test labels.","section":"Section 4.3.1 (Eq. 17) and Section 4.3.2 (Additional_Information 2)"},{"comment":"The headline results cannot be reconstructed from the reported tables because of inconsistencies. HGT-All on Albergate is reported as Precision 0.4444, Recall 0.8000, F1 0.5714 in Table 7, but Table 9 gives Precision 0.4189, Recall 0.6200, F1 0.4914 for the same configuration and project. The Table 9 Albergate values are identical to the Table 7 HGT-Fine-grained row, suggesting a copying error. Additionally, the Pig and Seam2 rows in Table 9 contain the same numerical values for all methods (GA-XWCoDe, HGNNLink, HGT-All, Gemini-All), which is implausible for two different projects. These discrepancies must be resolved before the average F1 improvements in the abstract can be trusted.","section":"Tables 7 and 9"},{"comment":"Observation 4 reverses the paper's own evidence. The RQ2 results in Table 3 show that NL-PL artifact pairs such as Requirements-Code have a low Difference Ratio (+93.86%) and low average F1 (0.3824), while NL-NL and PL-PL pairs such as Issue-Commit (+465.19%, F1 0.5602) and Test-Code (+885.19%, F1 0.5602) have higher ratios and better F1. Table 5 further shows that heterogeneous artifact pairs use enhancement strategies more often (67.08%) than homogeneous pairs (54.81%). Observation 4 states the opposite: that text similarity methods perform poorly for NL-NL/PL-PL tasks and that strategies are less necessary for NL-PL tasks. This contradiction also undermines the motivation in Section 4.2.1, which invokes Observation 4 to justify the case study. The observation should be rewritten to match the data.","section":"Section 3.3.3, Observation 4"}],"minor_comments":[{"comment":"The paragraph beginning 'Second, due to the modular nature of PL artifacts' is duplicated verbatim.","section":"Section 1"},{"comment":"The metric is inconsistently named: the text and Table 3 use both 'Difference Ratio' and 'Different Ratio'. Please use one term throughout.","section":"Section 3.2.2 and Table 3"},{"comment":"The sentence 'HGT-All includes three components... (explained in Section??)' contains an unresolved cross-reference placeholder.","section":"Section 4.3"},{"comment":"The final classifier is described inconsistently: Eq. (30) stops at f4, the text introduces f5, and Eq. (31) computes an inner product of r_i and c_j. Please clarify the exact architecture used in the experiments.","section":"Section 4.3.1, Eqs. (28)-(31)"},{"comment":"The text contains several typos in the strategy examples, including 'examp;e', 'folows', 'isEqalTo', and 'TurePositives' in Eq. (1).","section":"Section 3.3.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core empirical claim—that multi-strategy HGT and Gemini 2.5 Pro beat the state of the art by 3.68% and 8.84% F1—is very likely inflated by label leakage. The user-feedback strategy randomly samples 10% of ground-truth links, adds them as HGT edges or as “label is 0 or 1” statements in the Gemini prompt, and the paper never states that those sampled links are held out from evaluation. Since the feedback links are positive-only, the leak inflates recall and F1. That is a load-bearing flaw, not a minor one.\n\nSecond, the paper does have real value elsewhere. The systematic review of 50 studies and 99 projects is a useful aggregation, and the proposed Difference Ratio—the relative gap in co-occurrence word ratios between true and false link sets—is a cheap, sensible diagnostic. Showing that RoBERTa F1 correlates with that ratio across artifact types is a legitimate empirical observation, and the finding that NL-PL pairs have lower Difference Ratios than NL-NL or PL-PL pairs is plausible and worth reporting.\n\nBut the soft spots are serious. No train/test split is described anywhere in Section 4. Table 9 contradicts Table 7 for at least two HGT-All entries (Albergate is 0.5714 in Table 7 but 0.4914 in Table 9; Seam2/Pig rows are duplicated). Observation 4 reverses the paper’s own conclusion: it says text similarity performs poorly for NL-NL and PL-PL pairs and well for NL-PL pairs, which is the opposite of the data and the rest of the text. Also, calling Gemini-All “unsupervised” while feeding it ground-truth labels in the prompt is misleading.\n\nThe underlying idea—that auxiliary strategies beyond text similarity are needed for NL-PL traceability—is reasonable and consistent with prior work. The authors are clearly thinking seriously about the problem. But as written, the quantitative support for the headline claim is not reproducible, and the leakage issue undermines the main contribution.\n\nI would not cite this in its current form. But this is not a desk reject; the corpus, the Difference Ratio, and the multi-strategy integration idea are worth a serious referee. I’d suggest major revision: clarify the data split, hold out user-feedback links from evaluation, correct the tables, and fix Observation 4. If that is done, the paper could be a solid contribution to the traceability community.\n\nMy recommendation: send it to peer review, but the verdict should be reject-and-resubmit with the evaluation issues made the centerpiece of the revision request.","headline":"Potentially useful benchmark and Difference Ratio metric, but the central F1 claims are untrustworthy as written because user-feedback links sampled from ground truth are never excluded from evaluation.","tokens_in":42414,"tokens_out":1402,"would_cite":false,"duration_ms":14365,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that requirements-to-code traceability link recovery needs auxiliary strategies beyond textual similarity, with multi-strategy HGT and Gemini 2.5 Pro reporting average F1 gains of 3.68% and 8.84% over the previous best…","keywords":["software traceability","traceability link recovery","text similarity","Heterogeneous Graph Transformer","large language model","requirements-to-code","Difference Ratio","auxiliary strategies"],"falsifier":"Inspect the released replication package and compare the user-feedback link list used to build HGT-All edges and Gemini-All prompts against the labels used for the reported Precision, Recall, and F1; any overlap means the central performance claim is not supported. A cleaner check is to rerun the twelve-project evaluation with a strict split where feedback links are sampled only from the training set and never from the test set, and see whether the 3.68% and 8.84% average gains survive.","tokens_in":41431,"feed_emoji":"🔗","tokens_out":9834,"duration_ms":73268,"temperature":0.7,"pith_summary":"The paper sets out to show that textual similarity alone cannot reliably recover traceability links between natural-language requirements and code. Using a systematic review of 99 open-source projects and a new diagnostic metric, the Difference Ratio, it argues that NL-PL artifact pairs have far less lexical separation between true and false links than NL-NL or PL-PL pairs. To close that gap, it integrates three auxiliary strategies as heterogeneous-graph edge types in an HGT model and as additional prompt information in Gemini 2.5 Pro. On twelve requirements-to-code projects, the integrated models report average F1 gains of 3.68% and 8.84% over the previous best supervised baseline. A sympathetic reader would take the paper's contribution to be an empirical case that strategy integration, not better similarity alone, is what moves NL-PL traceability.","feed_headline":"Multi-strategy models beat text-similarity traceability by up to 8.84%","feed_subtitle":"Dependency, feedback, and fine-grained signals lift F1 by 3.68% and 8.84% over the prior best on 12 projects.","key_machinery":"The Difference Ratio, $(P_{\\text{true}} - P_{\\text{false}})/P_{\\text{false}}$, compares average co-occurrence word ratios of true versus false link pairs; a low value diagnoses a dataset where lexical similarity cannot separate links. The recovery machinery then encodes auxiliary strategies as graph structure: code artifacts become nodes connected by import, extend, and method-call edges; requirement-code pairs get user-feedback edges from a random 10% of ground-truth links; and fine-grained semantic edges mark pairs where all seven decomposed code components (class name, method name, comments, parameters, return types) rank in the top 20% by VSM similarity to the requirement. HGT's type-aware attention learns the contribution of each edge type, while Gemini-All translates the same three strategies into sentences placed in the prompt's Additional_Information block.","core_discovery":"The central claim is that NL-PL traceability link recovery is limited by a semantic gap that word-overlap measures cannot bridge, and that injecting domain-specific auxiliary strategies into powerful base models is an effective remedy. Concretely, the paper claims that HGT-All, which adds code-dependency, user-feedback, and fine-grained similarity edges to a Heterogeneous Graph Transformer, and Gemini-All, which feeds the same signals as text into a prompt-based LLM, both outperform their no-strategy counterparts and the previous best method HGNNLink. The paper reports average F1-score improvements of 3.68% for HGT-All and 8.84% for Gemini-All over HGNNLink across twelve open-source Java projects, with statistical significance at the 0.05 level.","pith_inferences":["If the user-feedback edges are sampled from ground-truth links without a held-out test set, a portion of the evaluation labels may reach HGT-All and Gemini-All through feedback edges or prompt text; checking the released package for overlap between feedback links and scored labels would settle whether the reported 3.68% and 8.84% gains are real.","The Difference Ratio could be used by other researchers as a dataset-suitability prior: projects with low or negative ratio are candidates for multi-strategy integration, and one could test that hypothesis by predicting per-project F1 gains from ratio alone.","The edge-type recipe should transfer to other languages: replace the Java dependency extraction with Python or C++ dependency analysis and keep the HGT/Gemini prompts unchanged, giving a testable cross-language extension.","A practical hybrid not explored in the paper would run HGT-All on large repositories, run Gemini-All on small ones, and have Gemini verify HGT's borderline candidates, combining the two models' complementary strengths."],"forward_implications":["For NL-PL datasets, text-similarity-only methods should be treated as weak baselines; the Difference Ratio can serve as a cheap pre-screening test for whether auxiliary strategies are needed.","In supervised settings, HGT-All significantly outperforms GA-XWCoDe and HGNNLink at the 0.05 level across the twelve evaluated projects.","In unsupervised settings, Gemini-All significantly outperforms TAROT, with larger gains on small projects, while HGT-All wins on the largest projects.","New auxiliary strategies can be added downstream: HGT-All accommodates them as new edge types, and Gemini-All as new sentences in Additional_Information.","The paper's own implications note that HGT-All requires labeled training data and is sensitive to data scale, whereas Gemini-All is costly and sensitive to noisy strategy input, so the two cover complementary deployment scenarios."],"supporting_citations":[{"why":"Supplies HGNNLink, the previous best supervised baseline that HGT-All must beat.","marker":"[72]"},{"why":"Supplies the Heterogeneous Graph Transformer architecture that HGT-All extends with strategy-specific edge types.","marker":"[31]"},{"why":"Supplies Gemini 2.5 Pro, the base model for the prompt-based Gemini-All approach.","marker":"[62]"},{"why":"Supplies TAROT, the IR baseline that Gemini-All is compared against in unsupervised settings.","marker":"[22]"},{"why":"Supplies GA-XWCoDe, the genetic-algorithm/XGBoost baseline using code-dependency features.","marker":"[87]"},{"why":"Supplies the code-dependency and user-feedback strategies that the paper integrates into both models.","marker":"[21]"},{"why":"Supplies RoBERTa, the encoder producing requirement node features for HGT-All.","marker":"[39]"},{"why":"Supplies GraphCodeBERT, the encoder producing code node features for HGT-All.","marker":"[25]"},{"why":"Supplies MPLinker, the NL-NL method whose high F1 highlights the NL-PL gap.","marker":"[66]"}],"fun_headline_variants":["Beyond text similarity: multi-strategy models lift traceability F1 by 8.84%","Semantic gap closed: hybrid models beat text-only traceability by up to 8.84%","Multi-strategy HGT and Gemini beat textual similarity for code traceability","Traceability links: adding domain signals beats pure text similarity by 8.84%","Text similarity isn't enough: multi-strategy models gain up to 8.84% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 10% of ground-truth links randomly sampled as user feedback (Section 4.3.1, Eq. (17), and prompt Additional_Information 2) are not also used as evaluation labels; the manuscript never describes a train/test split or a hold-out of feedback links, so if those labels are visible to the models the reported gains are inflated by leakage.","fun_headline_variants_meta":{"raw":{"variants":["Beyond text similarity: multi-strategy models lift traceability F1 by 8.84%","Semantic gap closed: hybrid models beat text-only traceability by up to 8.84%","Multi-strategy HGT and Gemini beat textual similarity for code traceability","Traceability links: adding domain signals beats pure text similarity by 8.84%","Text similarity isn't enough: multi-strategy models gain up to 8.84% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1482,"prompt_tokens":958,"completion_tokens":524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":408}},"tokens_in":574,"tokens_out":524,"duration_ms":4162,"temperature":1.0,"reasoning_tokens":408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:23:05.725379+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released replication package and compare the user-feedback link list used to build HGT-All edges and Gemini-All prompts against the labels used for the reported Precision, Recall, and F1; any overlap means the central performance claim is not supported. A cleaner check is to rerun the twelve-project evaluation with a strict split where feedback links are sampled only from the training set and never from the test set, and see whether the 3.68% and 8.84% average gains survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies HGNNLink, the previous best supervised baseline that HGT-All must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Heterogeneous Graph Transformer architecture that HGT-All extends with strategy-specific edge types."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies TAROT, the IR baseline that Gemini-All is compared against in unsupervised settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies GA-XWCoDe, the genetic-algorithm/XGBoost baseline using code-dependency features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the code-dependency and user-feedback strategies that the paper integrates into both models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MPLinker, the NL-NL method whose high F1 highlights the NL-PL gap."}],"review_version":2}