{"id":"df52203f-0978-47fe-a32b-27a60ea72120","arxiv_id":"2502.08050","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On one SLR update dataset, ML classifiers achieved an F-score of 0.33 and could discard 33.9% of studies at 100% recall, but human-human reviewer pairs still outperformed human-ML pairs.","lead":"This paper tests whether machine learning can help pick relevant studies when updating a systematic literature review. It finds the models are too inaccurate to replace human reviewers, but can cut screening effort by about 34% without missing relevant studies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported F-score 0.33 and 33.9% effort-reduction are the best results selected from many pipeline configurations evaluated on the test set, so they are optimistic, not unbiased, for the studied update.","rationale":"The paper is a transparent, small-scale empirical study with open artifacts and a clean holdout evaluation for each individual configuration. The reader's verdict of CONDITIONAL is appropriate. However, the reader's weakest_assumption focuses on the backward-snowballed training negatives, which is a valid external-validity threat, but the more load-bearing issue is internal: the authors selected the best-performing configuration after seeing test-set results, across many feature-selection methods, feature counts, algorithms, and tuning objectives. This makes the reported F-score of 0.33 and the 33.9% safe-discard figure optimistic upper bounds, not impartial estimates, even for the studied update. The negative conclusion (ML is not reliable for full automation) is robust because even an inflated F-score of 0.33 is low, but the positive claim (33.9% effort reduction at 100% recall) could be substantially weaker if the configuration were fixed a priori. The proposed concrete test—re-running with a pre-specified configuration or nested validation—would settle this. If the numbers decrease materially, the paper's practical recommendation would need to be tempered. I therefore recommend retaining CONDITIONAL, with the added condition that the configuration-selection procedure be justified or corrected.","tokens_in":11783,"tokens_out":6317,"duration_ms":68577,"concrete_test":"Fix the pipeline configuration before evaluating on the 551-paper holdout (e.g., pre-specify Anova-F with 1200 features for F-score target and Pearson with 1200 features for recall target, or use nested cross-validation on the training set to select feature-selection method and feature count). Re-run the evaluation. If the specificity at 100% recall drops substantially below 36.5% (187/513 true negatives) or the F-score drops below 0.33, the reported effort reduction is inflated by test-set selection.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline numbers are obtained by selecting the best among many pipeline configurations directly on the holdout test set. Section IV-B describes experiments varying feature-selection method (Chi2, Pearson, Anova-F), feature count (900–1500), algorithm (SVM, RF), and tuning target (F-score, recall), and then reports 'the best result' for RQ1 (RF, Anova-F, 1200 features, F-score 0.33) and RQ2 (SVM, Pearson, 1200 features, recall 1.0, 33.9% reduction). Because the test set is used for this outer model selection, the reported figures are maxima over a grid of configurations, not the performance of a pre-specified model. This selection bias can materially inflate both the F-score and specificity at 100% recall; with only 38 positives, many configurations can achieve high recall, and the one that discards the most negatives is likely an outlier. The central claim that ML can reduce effort by 33.9% without evidence loss is therefore not supported by an unbiased estimate even for the studied update. This is distinct from the reader's concern about backward-snowballed training negatives, which affects external validity; the test-set selection threatens the internal validity of the reported numbers themselves. The paper correctly criticizes Watanabe et al. for leakage, but this test-set selection is a similar optimistic bias that should be addressed before the quantitative conclusions are accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical evaluation of supervised machine learning models (Random Forest and Support Vector Machines) for supporting study selection in an update of a specific systematic literature review in software engineering. Using 128 training studies (45 included, 83 excluded via backward snowballing) and a holdout test set of 551 studies (38 included, 513 excluded) from the update, the authors train classifiers with various feature-selection methods and feature counts, reporting a best F-score of 0.33 for RF (RQ1) and an SVM configuration achieving 100% recall while excluding 33.9% of studies (RQ2). For RQ3, they compare human-only and human-ML reviewer pairs using Euclidean distance to the final consensus. The main conclusions are that ML cannot reliably automate selection but can reduce screening effort without loss of evidence, and that human-only pairs outperform human-ML pairs.","tokens_in":12045,"tokens_out":4879,"duration_ms":71358,"significance":"The study provides a rare, openly available dataset and a reproducible pipeline, and its holdout evaluation is a useful corrective to earlier work (e.g., Watanabe et al.) that used leakage-prone cross-validation. If the quantitative results were unbiased, the 33.9% effort-reduction figure would be a practically meaningful benchmark. However, the strength of the conclusions is limited by the single-dataset design and by the selection-bias and representativeness issues detailed below.","major_comments":[{"comment":"The reported best results are selected from a large grid of configurations evaluated on the same holdout test set. The paper experiments with two algorithms, three feature-selection methods, feature counts from 900 to 1500, and two tuning targets, and then reports 'the best result' for RQ1 (RF, Anova-F, 1200 features, F-score 0.33) and RQ2 (SVM, Pearson, 1200 features, recall 1.0, 33.9% reduction). Because the test set is used for this outer model selection, these numbers are maxima over the grid rather than unbiased estimates for a pre-specified model. With only 38 positive test instances, many configurations can attain 100% recall, and the configuration that discards the most negatives is likely an outlier. This threatens the internal validity of the headline quantitative claims. The authors should either select the configuration using only the training set (e.g., via nested cross-validation) or report the complete distribution of results across all configurations and provide a corrected (e.g., best-vs-typical) analysis.","section":"IV-B, V-A, V-B"},{"comment":"The 83 negative training examples were obtained by backward snowballing from the original SLR's included references, rather than from the original SLR's actually excluded studies. This is a non-representative sample: backward-snowballed references may be more topically related to included studies, older, or otherwise different from the true negatives in the update. If these differences affect classification, the measured F-score and the 33.9% safe-discard figure may not generalize, even to the studied update. The authors acknowledge the lack of access to the original excluded list, but this is a load-bearing assumption for the training data. Please provide evidence of representativeness (e.g., topic distributions, temporal overlap, or a sensitivity analysis retraining on different negative samples) or accordingly temper the quantitative conclusions.","section":"IV-A"},{"comment":"The claim of a 33.9% effort reduction 'without loss of evidence' rests on a single test set with only 38 positive instances. With 38/38 recalled, the lower 95% confidence bound on recall is about 90.7%, so the sample size provides limited evidence for the 'no evidence loss' guarantee. Moreover, the 33.9% figure is a point estimate with no measure of uncertainty; the paper should report confidence intervals or a sensitivity analysis over thresholds (Table III already shows the trade-off). As written, the guarantee-like phrasing overstates what can be concluded from one dataset.","section":"V-B"}],"minor_comments":[{"comment":"The preliminary tests that led to selecting SVM and RF out of the five classifiers are not reported. Since classifier selection is part of the pipeline, please provide the results of these initial tests or state that they were performed on the training set only.","section":"IV-B"},{"comment":"The mapping of RF probabilities to the 0-2 scale uses thresholds (0.50, 0.60) that are adjusted to match the reviewers' assessment distribution. This adjustment appears to be done on the test set; please clarify and discuss the potential for bias.","section":"V-C"},{"comment":"The Euclidean distances are small (e.g., 8 to 17 on a scale where the maximum distance per study is 2), and the claim that human-only pairs are 'much better aligned' would benefit from an effect-size measure or a test of whether the differences exceed what would be expected given that the oracle is derived from the trio itself.","section":"Table IV"},{"comment":"In Figure 1, the label 'ML Training Set' appears twice, which may confuse readers about the direction of the training and testing splits; please correct the labeling.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The test-set selection issue is serious and could be seen as a reproducibility concern similar to the leakage criticized in the paper. However, it is fixable: the authors have the full grid results in their repository and can re-analyze the data with nested cross-validation or by reporting the entire distribution. The single-dataset nature limits the contribution, but the paper's honest negative result (F-score 0.33) is valuable for the community. I would not reject outright, but the authors need to address the selection-bias point before the quantitative claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful, honest negative result for ML-assisted SLR updates, but the headline numbers are selected on the test set and therefore optimistic. The paper deserves a serious look, but the evaluation protocol needs work before the numbers are taken at face value.\n\nWhat's actually new: they ran a leakage-free holdout evaluation on a real SLR update, with open artifacts, and they explicitly document a data-leakage problem in Watanabe et al.'s earlier high-F-score result. That alone is a service to the community. The finding that a Random Forest reaches only 0.33 F-score and that SVM can discard 33.9% of studies at 100% recall on this dataset is a concrete data point that is more credible than the 0.92 F-score previously reported. The human-versus-ML pair comparison via Euclidean distance is also a reasonable, if simple, way to ask whether ML can substitute for a second reviewer.\n\nWhere it's soft. Most importantly, the reported RQ1 and RQ2 numbers are not the performance of a pre-specified model. They ran a grid over feature-selection methods, feature counts, algorithms, and tuning targets, then reported the best configuration. Because the test set was used for that outer selection, 0.33 and 33.9% are maxima over a grid, not unbiased estimates. On a test set with 38 positives, a configuration tuned to achieve 100% recall will often be an outlier in how many negatives it happens to discard. So the 33.9% effort-reduction figure is likely an upper bound; a fair estimate requires a separate validation set or a report of the full distribution of results. This is not a showstopper for the qualitative conclusion — even a 20% reduction would support 'can help reduce effort but not replace humans' — but the paper should not present these as unbiased point estimates.\n\nThe second concern is the training negatives. The 83 excluded training papers were obtained by backward snowballing from the original SLR's included references, not from the actually-excluded papers of the original review. That makes the training set a convenience sample that may not reflect the difficulty of the update's exclusion decisions. It's a threat to external validity, as the authors concede, but it also means the measured F-score and reduction rate may not generalize even to a similar update.\n\nMinor: no error bars, single dataset, and the RQ3 probability-to-vote mapping is fit to the test distribution. These are secondary.\n\nBottom line: who is this for? People working on automating SLR updates, and anyone who wants to see a careful critique of a prior high-performance claim. It's not a transformative result, but it's a solid, auditable negative finding. I'd send it to peer review after requiring the authors to fix the test-set selection issue — either by reporting the performance of the full grid (not just the best) or, better, by using the test set only once. The core conclusion will likely survive, but the headline numbers need to be honest.\n\nRecommendation: engage with it. I would accept for peer review with major revision.","headline":"Honest negative result with open artifacts, but headline numbers are best-of-grid selected on the test set, so treat them as optimistic.","tokens_in":12608,"tokens_out":3559,"would_cite":true,"duration_ms":29730,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Machine learning can trim a systematic-review update's screening load by a third, but cannot replace human reviewers.","keywords":["systematic literature review","SLR update","study selection","machine learning","text classification","Random Forest","Support Vector Machine","effort reduction"],"falsifier":"Train the same SVM and Random Forest pipelines on a second SLR update for which the original review's full list of excluded papers is available, using those true exclusions as the negative training set; if the 100%-recall discard fraction is not close to 33.9% or the F-score is not close to 0.33 on the holdout, the reported figures are an artifact of the surrogate negatives rather than a general property of ML for SLR updates.","tokens_in":11555,"feed_emoji":"🤖","tokens_out":8286,"duration_ms":58154,"temperature":0.7,"pith_summary":"The paper asks whether machine-learning text classifiers can take over study selection when a systematic literature review is updated. Using a real update and a consensus oracle from three experienced reviewers, it finds the best model scores only 0.33 F-score, too low for automation. The same models, tuned for recall, can discard about a third of candidate studies without missing any included study, cutting manual screening effort. Treating the model as an additional reviewer worsens agreement with the final result relative to human pairs. The authors conclude that ML can support effort reduction but that experienced human reviewers remain necessary for rigorous selection.","feed_headline":"ML cuts screening effort by a third, not replace reviewers","feed_subtitle":"100% recall while discarding 34% of papers, but human pairs still beat human-ML pairs.","key_machinery":"The central object is a supervised text-classification pipeline: candidate studies are converted with TF-IDF into numerical feature vectors, the most informative 1,200 features are selected by statistical tests (Chi-squared, Pearson correlation, or ANOVA F), and Random Forest or SVM classifiers are tuned with cross-validated hyperparameter search and then applied to a holdout testing set. The oracle is the consensus of three experienced researchers after full-text reading; for reviewer-pair analysis, the model's inclusion probabilities are mapped to the same 0/1/2 scale used by human reviewers and agreement is measured by Euclidean distance.","core_discovery":"The paper establishes that, for the studied SLR update, supervised ML text classifiers trained on the original review's data cannot reliably automate the study-selection step: the best configuration (Random Forest with ANOVA-F feature selection and 1,200 features) reaches an F-score of 0.33 at the default inclusion threshold. It also establishes that a recall-optimized SVM can act as a safe pre-filter, excluding 33.9% of the candidate pool while keeping recall at 100% against the three-reviewer oracle. When the model is paired with a human as if it were an additional reviewer, agreement with the final curated result is worse than for any human-human pair, so ML support does not substitute for a human reviewer in the initial screening.","pith_inferences":["Because the negative training examples were drawn by backward snowballing rather than from the original SLR's actual exclusion list, the 33.9% safe-discard figure and the 0.33 F-score could shift if real exclusions differ systematically from snowballed references.","The safe-discard fraction and F-score are measured on one update with only 38 included papers among 551 candidates; on reviews with different inclusion rates or topic breadth, the same pipeline would likely produce different numbers.","The human-ML pairing result may depend on how the model's probability output is converted to the reviewer scale; using the model only as a flag to discard low-probability papers, rather than averaging its score with a human's, might preserve more of the human signal."],"forward_implications":["A recall-tuned ML classifier can be used as a pre-filter that removes roughly one-third of candidate studies while keeping recall at 100% relative to the final curated list.","Using the F-score-tuned model as a replacement reviewer degrades agreement with the final consensus; human-only pairs gave Euclidean distances around 8.9–9.2 versus 11.5–11.8 for human-ML pairs.","Raising the inclusion threshold trades recall for effort: at 97.4% recall the model could discard 48.3% of studies, but one included study would be lost.","For serious SLR updates, at least two experienced human reviewers should perform the initial screening.","The reported results come from a holdout testing set with no data leakage, unlike earlier work that reported much higher F-scores but applied feature selection before the train-test split."],"supporting_citations":[{"why":"The original systematic review whose update is reproduced; its included studies become the positive training examples and its update process yields the oracle.","marker":"[25]"},{"why":"Reports much higher F-scores for text-classification-based SLR updates; this paper argues those results suffer from data leakage and uses a clean holdout design instead.","marker":"[23]"},{"why":"Cross-domain systematic mapping that identifies SVM and the Precision/Recall/F-measure metrics as standard for study-selection automation, guiding the experimental setup.","marker":"[13]"},{"why":"Systematic review of feature-selection methods and text classifiers that supplies the candidate algorithms (including RF and SVM) and feature-selection techniques tested here.","marker":"[17]"},{"why":"Defines the snowballing procedure used to build the negative training set by backward snowballing the original SLR's references.","marker":"[24]"}],"fun_headline_variants":["ML pre-filter cuts review effort by 34%, but can't replace humans","AI screens out third of papers with perfect recall, yet human pairs still reign","Machine learning reduces SLR workload, not reviewer judgment","ML slashes screening effort, but human review still wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The negative training examples are backward-snowballed references from the original SLR rather than the studies the original review actually excluded, so the measured F-score and the 33.9% safe-discard figure assume those negatives represent the real excluded pool.","fun_headline_variants_meta":{"raw":{"variants":["ML pre-filter cuts review effort by 34%, but can't replace humans","AI screens out third of papers with perfect recall, yet human pairs still reign","Machine learning reduces SLR workload, not reviewer judgment","ML slashes screening effort, but human review still wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1740,"prompt_tokens":968,"completion_tokens":772,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":698}},"tokens_in":584,"tokens_out":772,"duration_ms":6165,"temperature":1.0,"reasoning_tokens":698,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:00:25.865668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same SVM and Random Forest pipelines on a second SLR update for which the original review's full list of excluded papers is available, using those true exclusions as the negative training set; if the 100%-recall discard fraction is not close to 33.9% or the F-score is not close to 0.33 on the holdout, the reported figures are an artifact of the surrogate negatives rather than a general property of ML for SLR updates.","supporting_citations":[{"cited_title":"Suc- cessful combination of database search and snowballing for identification of primary studies in systematic literature studies,","cited_arxiv_id":null,"evidence_quote":"The original systematic review whose update is reproduced; its included studies become the positive training examples and its update process yields the oracle."},{"cited_title":"Reducing efforts of software engineering systematic literature reviews updates using text classification,","cited_arxiv_id":null,"evidence_quote":"Reports much higher F-scores for text-classification-based SLR updates; this paper argues those results suffer from data leakage and uses a clean holdout design instead."},{"cited_title":"Automated support for searching and selecting evidence in software engineering: A cross- domain systematic mapping,","cited_arxiv_id":null,"evidence_quote":"Cross-domain systematic mapping that identifies SVM and the Precision/Recall/F-measure metrics as standard for study-selection automation, guiding the experimental setup."},{"cited_title":"Feature selection methods for text classification: a systematic literature review,","cited_arxiv_id":null,"evidence_quote":"Systematic review of feature-selection methods and text classifiers that supplies the candidate algorithms (including RF and SVM) and feature-selection techniques tested here."},{"cited_title":"A snowballing procedure for systematic literature studies and a replication,","cited_arxiv_id":null,"evidence_quote":"Defines the snowballing procedure used to build the negative training set by backward snowballing the original SLR's references."}],"review_version":1}