{"id":"7050923e-56ce-4b8a-94a3-f3d718104b22","arxiv_id":"2411.11646","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Token-level highlighting does not improve human accuracy in tracing vulnerability patches, though TfIdf-Highlight is more faithful to the retrieval model than LIME.","lead":"The authors built a system that retrieves GitHub commits fixing known security vulnerabilities and tested whether highlighting important words helps human reviewers pick the correct patch. They found highlighting does not improve accuracy, and a simple word-frequency highlight matched the model's reasoning better than the widely used LIME method.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TfIdf-Highlight's 15% sufficiency gain likely inherits from the Tf-Idf-based diff-chunk selection that feeds the retrieval model, a confound the paper does not ablate.","rationale":"The reader's weakest assumption concerns external validity: the true-version-tag oracle used to build candidate pools may not transfer to real NVD usage. That is a real limitation, and the paper discloses it. However, the faithfulness comparison between TfIdf-Highlight and LIME is internal to the constructed candidate pools, so the oracle does not invalidate the relative RQ2 comparison. The more load-bearing concern is internal: the retrieval model's code input is the top chunk chosen by Tf-Idf similarity, and TfIdf-Highlight selects tokens from that same chunk using the same Tf-Idf statistics. This makes the model-agnostic explanation method informationally privileged relative to LIME, so the 15% sufficiency gain may be an artifact of the pipeline rather than evidence that TfIdf-Highlight better reflects model reasoning. This concern directly targets the paper's central quantitative claim, and it can be settled by a straightforward ablation. I do not think it requires rejection: the paper is honest about its scope, the human study is a useful negative result, and the dataset has value. But the condition for acceptance should include re-running the RQ2 comparison under a non-Tf-Idf chunk-selection baseline, or substantially qualifying the faithfulness claim. Since the reader already gave CONDITIONAL, my read leaves that verdict unchanged, with the added condition targeting the chunk-selection confound.","tokens_in":17773,"tokens_out":6555,"duration_ms":66050,"concrete_test":"Re-run the faithfulness evaluation (Table IV) with the retrieval model retrained or at least evaluated on diff chunks selected by position (e.g., the first chunk of each diff) instead of by Tf-Idf cosine similarity, keeping all other hyperparameters and the LIME/TfIdf-Highlight comparison identical. If the sufficiency gap between TfIdf-Highlight and LIME shrinks or loses statistical significance, the central RQ2 claim is an artifact of the Tf-Idf-based chunk selection. A complementary check is to retrain the model with the full diff truncated to 512 tokens without chunk selection and compare the two highlighters; if TfIdf-Highlight's advantage persists in that setting, the confound is not the sole driver.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central RQ2 claim is that TfIdf-Highlight is significantly more faithful to the trained retrieval model than LIME. That comparison is confounded by the input-construction pipeline. In Section V-A, each diff is split into 64-token chunks and only the chunk with the highest Tf-Idf cosine similarity to the CVE description is fed to CodeBERT/UnixCoder. TfIdf-Highlight (Section V-C) then selects, from this same chunk, the tokens that overlap with the CVE and have high Tf-Idf discriminativeness. Thus TfIdf-Highlight has privileged access to exactly the lexical-overlap signal that determined the model's input, while LIME must discover important tokens from black-box perturbations. The faithfulness metric in Section VI-B rewards methods that preserve the patch's rank when only highlighted tokens remain, so TfIdf-Highlight's advantage in Table IV may only reflect that it re-selects the trivial overlap tokens that the chunking procedure already made salient. The text-only rows are also affected because the model is trained on a representation whose code part was chosen by Tf-Idf overlap, biasing it toward lexical matching across modalities. The paper never trains or evaluates the model with a non-Tf-Idf-selected chunk, so the reported 15% sufficiency improvement cannot be attributed to a general explanation-quality advantage. The human result that highlighting does not improve accuracy is consistent with this shortcut interpretation, but the faithfulness claim is the quantitative headline and should be tested against this confound.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether token-level highlighting helps maintainers trace the commit that fixes a CVE. The authors construct a dataset of 3,573 (CVE, patch) pairs with candidate commit pools derived from version tags, fine-tune CodeBERT and UnixCoder as multi-modal retrieval models, and compare two explanation methods: LIME and a proposed model-agnostic TfIdf-Highlight. The evaluation uses a faithfulness metric (sufficiency and comprehensiveness) defined for ranking, plus blind human labeling experiments with a no-highlight control. The reported findings are that TfIdf-Highlight significantly improves sufficiency over LIME by about 15% in most settings, that TfIdf-Highlight receives higher helpfulness ratings than LIME, and that highlighting does not improve human labeling accuracy over no highlighting, sometimes reducing it.","tokens_in":18128,"tokens_out":6487,"duration_ms":65103,"significance":"If the results hold, the paper makes a useful empirical contribution to security database maintenance: it provides a large patch-tracing dataset, shows that dual-encoder retrieval models can rank candidate commits, and, importantly, offers a negative human result that token-overlap highlighting does not help and can even hurt maintainers' decisions. The blind user study with a no-highlight control is a strength, as is the attempt to define faithfulness for ranking rather than classification. The main concerns are that the candidate pools are built with an oracle version tag and that the central faithfulness comparison is confounded by the TfIdf-based input-chunk selection; these issues make the quantitative claims conditional in ways the current paper does not fully address.","major_comments":[{"comment":"The evaluation of the retrieval system and of both highlighting studies is built on candidate-commit pools constructed with the true patch version tag (Method 2), which is not available in the deployment scenario described in the introduction. Table I shows that using NVD version tags alone reaches only 60.2% coverage at range 0 and 76.7% at ±5, so the reported retrieval and faithfulness numbers are conditional on an oracle that a maintainer would not have. The paper explicitly acknowledges this at the end of Section IV-B, but the consequence is that RQ1 and RQ2 do not currently demonstrate performance for the actual NVD-assisted setting; please report the main results under Method 1 or under a noisy-version-tag protocol (e.g., ranging over ±1 and ±5) to show what a real deployment would achieve.","section":"Section IV-B, Table I"},{"comment":"The central RQ2 claim that TfIdf-Highlight is more faithful than LIME is confounded by the input-construction pipeline. Section V-A selects, for each diff, only the top-1 64-token chunk by TfIdf cosine similarity to the CVE description, and Section V-C's TfIdf-Highlight then highlights tokens with high TfIdf overlap from that same chunk. The sufficiency metric in Section VI-B replaces the explained component with only the highlighted tokens and measures rank preservation, so TfIdf-Highlight is being rewarded for re-selecting exactly the lexical-overlap signal that determined the model's input, while LIME is evaluated through black-box perturbations without that privileged access. Because no model is trained or evaluated with a non-TfIdf-selected chunk, the reported ~15% sufficiency gain cannot be attributed to a general explanation-quality advantage; please add an ablation using randomly chosen chunks, full-diff inputs, or a non-TfIdf chunk selector, and/or restrict both explainers to the same candidate-token vocabulary.","section":"Section V-A, Section V-C, Section VI-B"},{"comment":"The human labeling experiment is the only external grounding for the faithfulness comparison, but it is small (100 easy groups and 50 hard groups), all annotators are authors of the paper, and the hard experiment's accuracy differences (e.g., overall 0.76 with highlighting vs 0.86 without, with commit message revealed in Table VI) are reported without confidence intervals or significance tests. The paper's Section IX acknowledges the sample-size limitation, but the conclusion that highlighting does not improve labeling accuracy is a central negative result; please report per-annotator intervals, inter-annotator agreement, and a test such as McNemar or a bootstrap so the reader can assess the strength of the null result.","section":"Section VI-C, Tables V-VI"},{"comment":"The user study does not evaluate the trained retrieval model's ranked list: in the first experiment the distractor commit is the top-2 commit under TF-IDF, and the second experiment's groups are only filtered by file type rather than being sampled from the CodeBERT/UnixCoder ranking. Since the highlights are explanations of a neural model's decisions, but the candidate sets are generated by a different TF-IDF ranking, the human results measure how highlighting helps in a small ad hoc candidate pool rather than in the explainable retrieval system the paper proposes. Please clarify why the candidate selection in the user study is not tied to the trained model's ranking, or rerun the study on candidates that are near-misses of the neural model.","section":"Section VI-C, Figure 5"}],"minor_comments":[{"comment":"The sentence 'The results in Table IV indicate that TfIdf-highlight receives higher average helpfulness ratings than LIME' should refer to Table V, not Table IV; Table IV reports faithfulness scores.","section":"Section VI-C"},{"comment":"The summary says 'TfIdf-Highlight significantly outperforms BERT in the sufficiency score', but the comparison is against LIME; 'BERT' should be 'LIME'.","section":"Section VI-B, Summary of Findings for RQ2"},{"comment":"The formula for the message modality uses logtf(w,d)=0.5 log2(tf(w,d_msg))+0.5 log2(tf(w,d_diff)), whereas the code modality uses tf(w,d) without the log transform; please clarify whether this asymmetry is intentional and how it affects the highlighted-token counts.","section":"Section V-C, TfIdf-Highlight equations"},{"comment":"The definition of dfCVE(w,q) says 'the number of comments under the CVE q' but appears to mean the number of commits under that CVE; please correct the wording.","section":"Section V-C"},{"comment":"There is a typo in the text: 'TfIdf-Highlight has aa worse accuracy than LIME' should read 'a worse accuracy'.","section":"Section VI-C, Table VI"},{"comment":"The abbreviation XML is used both for 'explainable machine learning' in the introduction and for 'extreme multi-class classification' in Section VII, which can confuse readers; please disambiguate or use different abbreviations.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important practical problem and the negative human result is interesting, but the evaluation currently relies on an oracle candidate pool and the headline faithfulness comparison is confounded by the TfIdf-based chunk selection. Both issues are addressable with additional experiments, so I would not reject on scope or novelty; the authors should be asked to ablate the chunk-selection confound and to evaluate under realistic version-tag noise before the quantitative claims are accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the dataset and the negative human result, not for the faithfulness comparison. The paper is a careful, scoped empirical study: it builds a CVE-to-patch retrieval dataset (3,573 CVEs, 114k candidate commits), fine-tunes CodeBERT and UnixCoder for commit ranking, and compares LIME against a Tf-Idf overlap highlighter. The headline results—TfIdf-Highlight beats LIME by ~15% on sufficiency, while highlighting doesn't help human annotators—are reported with significance tests, a blind setup, and a no-highlight control. The negative human result is plausible and worth knowing.\n\nThe soft spot is the faithfulness claim. The retrieval model's input is the single 64-token diff chunk with the highest Tf-Idf cosine similarity to the CVE (Section V-A). TfIdf-Highlight then selects tokens from that same chunk using the same overlap statistics (Section V-C). TfIdf-Highlight thus has privileged access to exactly the signal that chose the model's input, while LIME must infer importance from black-box perturbations. The 15% sufficiency gain may only re-select the trivial overlap tokens that the chunker made salient. The paper never trains or evaluates with a different chunk selection, so RQ2's quantitative headline is not yet established as a general explanation-quality advantage. This isn't hidden; the authors describe the pipeline clearly, and they also disclose an oracle limitation: candidate commits come from the true patch version tag, which is often unavailable in practice (Table I: 60% coverage at range 0). That disclosure is honest, but the retrieval numbers are oracle-bound.\n\nThe human study is less affected: it compares actual highlighting conditions and finds no accuracy gain. Even if the highlight tokens are trivial, the system they built doesn't help, which is a legitimate result. The paper would be stronger with an ablation—random chunk selection, or a model trained on full truncated diffs—to see if the faithfulness advantage survives. Without that, I'd treat the sufficiency claim as unproven, not false.\n\nCitations look appropriate; the related work is on point. If the dataset ships, that alone is citable. Bottom line: it deserves a serious referee. The dataset and negative human result are valuable; the faithfulness claim needs a major revision and an ablation. I'd bring it to reading group to discuss evaluation confounds in explainable retrieval.","headline":"Useful dataset and a clean negative human result; the headline faithfulness claim is likely confounded by the Tf-Idf chunk selection and needs an ablation.","tokens_in":18642,"tokens_out":3421,"would_cite":true,"duration_ms":30518,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a simple TfIdf-based token highlight is more faithful to a trained patch-retrieval model than LIME, yet highlighting does not improve humans' ability to identify the correct security patch.","keywords":["security vulnerability patch tracing","explainable information retrieval","TfIdf-Highlight","LIME","faithfulness","CodeBERT","CVE","NVD delays"],"falsifier":"Run the same retrieval and highlighting pipeline on a candidate set filtered by NVD's published version tags instead of the true patched version, and measure whether retrieval recall and the faithfulness advantage of TfIdf-Highlight over LIME persist; Table I's coverage figures (60.2% at range 0, 76.7% at ±5) predict a substantial drop. Alternatively, a larger human study (more annotators, more CVE groups, and a preregistered analysis) could test whether the observed null or negative effect of highlighting on labeling accuracy is real or an artifact of three annotators and 50–100 examples.","tokens_in":17579,"feed_emoji":"🔍","tokens_out":7798,"duration_ms":62665,"temperature":0.7,"pith_summary":"The paper asks whether highlighting tokens in a commit message and diff can help security maintainers find the commit that fixes a known vulnerability. The authors build a retrieval system that ranks candidate commits for a CVE using CodeBERT or UnixCoder, then compare two ways of explaining the ranking: LIME, a standard model-agnostic explainer, and their own TfIdf-Highlight, which scores tokens by term frequency weighted by how rare they are within the CVE's candidates and across the whole dataset. They report that TfIdf-Highlight is significantly more faithful to the trained model than LIME, improving sufficiency by about 15% in most settings. But in a blind human-labeling study, neither highlight method improved annotators' accuracy at picking the patch, and with the commit message revealed the no-highlight condition was actually more accurate (0.86 vs 0.76). If true, this means token-level highlighting can reveal a black-box model's reasoning without hurting ranking performance, yet it is not enough to support the semantic judgments humans need to verify a patch.","feed_headline":"TfIdf highlights beat LIME, yet no highlight lifts human patch picking","feed_subtitle":"Token highlighting is more faithful to patch-retrieval models than LIME, yet it doesn't improve human labeling accuracy.","key_machinery":"The load-bearing object is TfIdf-Highlight, a parameter-free scoring rule that ranks tokens by a product of three factors: log term frequency in the commit message and diff, an idf computed within the candidate commits of the same CVE ($\\mathrm{idf}_{\\mathrm{CVE}}$), and an idf computed across the entire dataset. This score selects the top-$k$ tokens to highlight. The argument also relies on a faithfulness metric defined for ranking—the change in precision@1 between the original model probabilities and probabilities computed from only the highlighted tokens (sufficiency) or from the input with highlighted tokens removed (comprehensiveness)—which lets the authors compare explainers without ground-truth token labels.","core_discovery":"The central claim is that a model-agnostic token-overlap score, TfIdf-Highlight, is more faithful to a trained patch-retrieval model than LIME's locally weighted regression, and that this faithfulness does not translate into better human decision-making. Concretely, the authors define faithfulness for ranking as the change in precision@1 when the model is given only the highlighted tokens instead of the full input (sufficiency) or when the highlighted tokens are removed (comprehensiveness). Across CodeBERT and UnixCoder, on validation and test folds, TfIdf-Highlight lowers sufficiency scores by roughly 15% relative to LIME (e.g., from 0.398 to 0.286 for CodeBERT text-only validation), meaning the highlighted tokens better reproduce the model's ranking decision. A blind human experiment with three annotators and 50–100 groups of commits finds that TfIdf-Highlight receives higher helpfulness ratings than LIME, but accuracy is similar, and both are no better than no highlighting; with commit messages revealed, no-highlight accuracy is 0.86 versus 0.76 with highlighting. The authors conclude that token-level matching is insufficient for patch tracing, which requires understanding semantic relatedness between the CVE description and the commit.","pith_inferences":["The reliance on the true patched version tag as an oracle means the reported retrieval and faithfulness numbers are an upper bound for what a deployable system would achieve; integrating automatic version estimation (as in Table I's 60–77% coverage) would directly discount the headline gains.","The negative human result suggests a testable hypothesis: explanations that convey semantic relatedness (e.g., generated natural-language rationales or CWE-aware links) would outperform token highlighting, a direction the authors flag as future work.","TfIdf-Highlight's within-CVE idf term is a transferable trick for any retrieval explanation task: penalize words that are frequent among the candidates for a single query, since those cannot discriminate among them.","Because the authors find modality bias in multi-modal training (the model initially ignored code and relied on commit messages), their data augmentation forcing the model to learn from code is a reusable recipe for other code-plus-text retrieval models."],"forward_implications":["A retrieval model fine-tuned on commit messages and diff chunks can rank the true patch near the top: CodeBERT and UnixCoder reach precision@1 around 0.66, roughly 15 points above pure TfIdf.","TfIdf-Highlight, despite being model-agnostic and simpler than LIME, more faithfully reproduces the model's ranking decisions in almost all settings tested.","Because humans do not label patches more accurately with highlighting, token-level highlights should not be expected to replace maintainer review; at best they shorten the search.","For humans, the commit message carries more signal than diff code: revealing it raised accuracy, and the helpfulness of highlighting was judged higher when the message was present.","The proposed faithfulness score for ranking (change in precision@1 with highlighted tokens only or excluded) can be reused to evaluate other explainable retrieval systems."],"supporting_citations":[{"why":"Provides LIME, the baseline explanation method, and the prior claim that highlighted tokens help human labelers, which the paper tests against.","marker":"[2]"},{"why":"Defines the file-level patch-tracing task and shows CodeBERT as the best retrieval model, which the paper extends to commit-level retrieval.","marker":"[10]"},{"why":"Supplies CodeBERT, one of the two multi-modal retrieval models fine-tuned, along with its tokenizer and 512-token context limit.","marker":"[11]"},{"why":"Supplies UnixCoder, the second retrieval model used to test whether faithfulness results generalize across model architectures.","marker":"[35]"},{"why":"Introduces the ERASER benchmark and sufficiency/comprehensiveness definitions that the paper adapts into a ranking faithfulness score.","marker":"[3]"},{"why":"Provides the faithfulness definitions and caveats the paper relies on to justify its model-free evaluation of highlighting.","marker":"[15]"},{"why":"Contributes the BigVul dataset of labeled CVE-patch links, a primary source for the paper's training and test pairs.","marker":"[27]"},{"why":"Contributes PatchDB, another labeled patch dataset used to build the candidate CVE-patch pairs.","marker":"[28]"},{"why":"Shows a faithfulness evaluation based on probability changes without ground-truth rationales, which the paper reinterprets for ranking.","marker":"[44]"}],"fun_headline_variants":["TfIdf highlights beat LIME but don't aid human patch spotting","Highlighting tokens helps models but not humans in patch tracing","TfIdf highlighting more faithful than LIME, yet human accuracy flat","Token highlight: more faithful to model, but humans still fare the same"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The candidate commit pool for every CVE is built using the true first patched version tag, which the system would not know in real deployment; NVD's own version tags cover only about 60% of CVEs at the exact version and 77% within five versions.","fun_headline_variants_meta":{"raw":{"variants":["TfIdf highlights beat LIME but don't aid human patch spotting","Highlighting tokens helps models but not humans in patch tracing","TfIdf highlighting more faithful than LIME, yet human accuracy flat","Token highlight: more faithful to model, but humans still fare the same"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000559,"raw_usage":{"total_tokens":2753,"prompt_tokens":1135,"completion_tokens":1618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":751,"completion_tokens_details":{"reasoning_tokens":1540}},"tokens_in":751,"tokens_out":1618,"duration_ms":10932,"temperature":1.0,"reasoning_tokens":1540,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:17:03.429421+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same retrieval and highlighting pipeline on a candidate set filtered by NVD's published version tags instead of the true patched version, and measure whether retrieval recall and the faithfulness advantage of TfIdf-Highlight over LIME persist; Table I's coverage figures (60.2% at range 0, 76.7% at ±5) predict a substantial drop. Alternatively, a larger human study (more annotators, more CVE groups, and a preregistered analysis) could test whether the observed null or negative effect of highlighting on labeling accuracy is real or an artifact of three annotators and 50–100 examples.","supporting_citations":[{"cited_title":"Where is it? tracing the vulnerability-relevant files from vulnerability reports,","cited_arxiv_id":null,"evidence_quote":"Defines the file-level patch-tracing task and shows CodeBERT as the best retrieval model, which the paper extends to commit-level retrieval."},{"cited_title":"Patchdb: A large-scale security patch dataset,","cited_arxiv_id":null,"evidence_quote":"Contributes PatchDB, another labeled patch dataset used to build the candidate CVE-patch pairs."}],"review_version":1}