{"id":"3c5914d0-b52b-405f-a27c-1b190e979523","arxiv_id":"2506.14843","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An expanded knowledge-graph model, CACTUS, classifies five AMD stages from non-imaging data with balanced accuracy of 0.27 to 0.34, outperforming standard ML baselines on the EYE-RISK last-visit dataset.","lead":"This paper applies an expanded, explainable knowledge-graph classifier called CACTUS to a large European age-related macular degeneration (AMD) dataset, using only non-imaging data such as age, genetics, diet, supplements, and clinical measurements. The authors report that CACTUS's PageRank variant has higher balanced accuracy than standard machine learning models across several levels of missing data, and that its top-ranked features match established AMD risk factors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CACTUS may be evaluated in-sample: no evidence that abstraction cut-offs and knowledge graphs are fit inside the CV folds, so the outperformance over ML baselines is not yet established.","rationale":"The paper's central empirical claim is the Table 2 comparison. The last-visit sampling identified by the reader threatens external validity for early screening, but even on a biased sample the comparison between algorithms could still support 'higher balanced accuracy than standard ML models.' The more load-bearing threat is evaluation leakage: if CACTUS's fitted abstraction thresholds and graphs see test labels, or if the whole model is scored in-sample, the comparison itself is invalid. The text's 'classification is then applied to all the available individuals to assess the performance' and the absence of a nested-protocol description make this a concrete, testable gap. The refinement/9-rank analysis adds a second leakage path. A single re-analysis with threshold and graph fitting nested in CV would settle the point, so I keep the reader's conditional verdict but shift the binding condition from dataset representativeness to evaluation integrity. Since no code or data artifacts are provided, this re-analysis requires author action or independent implementation.","tokens_in":22459,"tokens_out":7284,"duration_ms":77449,"concrete_test":"Recompute Table 2 with CACTUS nested inside the same 10 folds used for the baselines: for each fold, fit the ROC-derived U/D cut-offs for every continuous feature and build the class-specific graphs on the training fold only, then predict the test fold, and never use test labels or test values when choosing thresholds. If CPR's balanced accuracy at 0% missing falls to 0.31 or below (the XGB level), the reported outperformance is not established; if it stays near 0.34, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CACTUS/CPR achieves higher balanced accuracy than the ML baselines rests on a fair comparison. The Methods never state that CACTUS's fitted components — the exhaustive ROC cut-off search per continuous feature in 'Abstractions' and the class-specific conditional-probability graphs in 'The Knowledge Graphs' — are recomputed inside each training fold. Instead, Figure 1 and the 'Classification' section say the model is built and 'the classification is then applied to all the available individuals to assess the performance', which reads as in-sample evaluation. The ML baselines are described as using an 80/20 split and 10-fold CV, with imputation/scaling applied before input, but no equivalent statement is made for CACTUS. If the same full-data flips and graph parameters are used to score the test folds, the test labels have already influenced the feature discretization and graph weights, so CACTUS's Table 2 advantage (e.g., CPR 0.34 vs XGB 0.31 at 0% missing) could be an artifact. The same problem recurs in the refinement stage: the 9 highest ranks are selected from full-data rankings and Table 3's balanced accuracy is then computed on the same data, so the feature-importance results are also at leakage risk. Without code or a written nested protocol, the main comparison cannot be verified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript applies CACTUS, a knowledge-graph-based explainable classifier, to stage age-related macular degeneration (AMD stages 0–4) using non-imaging data from the EYE-RISK/E3 consortium (29,908 patients, 218 features). The method discretizes continuous features via an exhaustive ROC-based abstraction, builds class-specific weighted directed graphs, and assigns labels through three variants: Probabilistic (CPB), Degree (CDG), and PageRank (CPR). The authors compare these variants against six standard ML algorithms (Ridge, RF, LR, SGD, SVM, XGB) at 0%, 20%, 40%, 60%, and 80% artificially induced missing values, reporting that CACTUS variants achieve higher balanced accuracy than all baselines at every fragmentation level (Table 2). They introduce a confidence metric, rank features, and then refine the feature pool and evaluate the top-9 features (Table 3, Figures 4–7). The paper claims that CACTUS, especially CPR, is a reliable explainable tool for early AMD classification.","tokens_in":22685,"tokens_out":6383,"duration_ms":59910,"significance":"If the evaluation is unbiased, the paper demonstrates a transparent, missingness-robust alternative to image-based AMD staging: CACTUS is explainable, provides a confidence metric, and its feature rankings recover established AMD risk factors such as age, ARMS2, CFH, and genetic scores. Comparing against XGBoost is a useful addition. However, absolute balanced accuracy is modest (at most 0.34 versus a 0.20 chance level), the study is purely retrospective with substantial dropout, there is no external validation, and the reported comparisons carry a material risk of in-sample evaluation. The practical utility for early screening is therefore not established by this manuscript alone.","major_comments":[{"comment":"The paper does not state whether CACTUS's fitted components—the exhaustive ROC-based cut-off search per continuous feature, the class-partition selection, and the conditional-probability graph weights—are recomputed inside each training fold. The Classification section and Figure 1 say the model is built and 'the classification is then applied to all the available individuals to assess the performance,' while the ML baselines are described with an 80/20 split and 10-fold cross-validation. If the flips and graph parameters are estimated on the full data before scoring, the test labels have influenced the feature discretization and graph structure, so the Table 2 advantage (e.g., CPR 0.34 vs XGB 0.31 at 0% missingness) could be leakage-driven. The authors must either provide a nested evaluation in which all fitted parameters are estimated on training folds only and applied to held-out folds, or otherwise demonstrate that the comparison is not in-sample.","section":"Methods: 'Abstractions', 'The Knowledge Graphs', and 'Classification'; Figure 1"},{"comment":"The refinement step selects the nine highest-ranked features from ranks computed on the full dataset and then reports balanced accuracy on the same data. Because feature selection and evaluation share the same samples, the Table 3 results (CDG 31%, CPB 29%, CPR 29% on the nine highest ranks) are at risk of selection bias and cannot be interpreted as unbiased performance estimates. The authors should perform feature selection inside an inner cross-validation loop or on a separate validation cohort, and then evaluate on an untouched test portion.","section":"Results: 'CACTUS ranks'; Table 3"},{"comment":"The dataset is constructed from the last available visit per patient ('To have both a populous and balanced dataset, we considered the last visit available for each person'), and Table 1 shows heavy dropout over the study (from 28,910 participants at visit 1 to 1,538 at visit 5, with 17,526 dropped by visit 2). If dropout is correlated with disease severity, health status, or age, the resulting cohort may not represent an early-screening population, and the reported accuracy and the top ranking of age as a feature may be inflated. The manuscript should discuss this selection mechanism and its possible directional bias, or provide sensitivity analyses stratified by visit completeness.","section":"Methods: 'The defined dataset' paragraph and Table 1"},{"comment":"The central performance comparison is reported only as mean ± standard deviation, with no statistical significance tests, confidence intervals, or paired comparisons across cross-validation folds. Given the small absolute differences (e.g., 0.34±0.01 vs 0.31±0.01 at 0% missingness) and overlapping standard deviations, the claim that CACTUS 'outperforms' the baselines is not statistically supported by the evidence as presented. Please add appropriate significance testing, such as paired tests on the cross-validation folds or bootstrap confidence intervals for the differences.","section":"Results: Table 2 and Discussion"},{"comment":"The text states 'This filtration decreased the balanced accuracy by 0% to 2% for CDG, CPR, and CPB, as shown in Table 3,' but Table 3 reports CDG dropping from 33% (whole dataset) to 26% (refined dataset), a seven-percentage-point decrease. This internal inconsistency between the stated claim and the displayed results must be corrected and, if the refined-dataset CDG value is accurate, the discussion of the refinement step should be revised accordingly.","section":"Results: 'CACTUS ranks' versus Table 3"}],"minor_comments":[{"comment":"The figure legend labels the model 'Linear Regression (LR),' while the Methods section refers to 'Logistic Regression (LR);' the labels should be made consistent.","section":"Figure 2 legend"},{"comment":"A stray character 'Z' appears after 'Logistic Regression (LR)' in the text; please remove it.","section":"Methods: 'Comparison against standard Machine Learning algorithms'"},{"comment":"The caption says the balanced accuracy is reported 'in percentage,' but the table entries are decimals (e.g., 0.33, not 33); either the caption or the entries should be adjusted for consistency.","section":"Table 2 caption"},{"comment":"The phrase 'The more inflated the cumulative lines for confidence and populations, the better the corresponding model' is difficult to parse and should be rewritten for clarity.","section":"Results: 'CACTUS confidence'"},{"comment":"The text mentions a 'confidence threshold of 15%' that 'enables us to confidently apply CPR on 50% of the population,' but Figure 3 annotates vertical lines as the confidence levels covering 90%, 80%, 70%, 60%, and 50% of the population; please clarify which line corresponds to the 15% threshold or adjust the text.","section":"Results: 'CACTUS confidence' and Figure 3"},{"comment":"The statement that CACTUS 'does not make any assumptions (weak model) on how the real process works' is vague, and the citations to White and Hellström do not directly justify the overfitting argument; please provide a more precise complexity comparison (e.g., number of effective parameters versus training sample size).","section":"Methods: 'Comparison against standard Machine Learning algorithms'"},{"comment":"No data or code availability statement is provided; a clear statement on code and trained model availability would be important for reproducibility of the main comparison.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central claim of the paper hinges on the fairness of the CACTUS-versus-ML comparison. If the authors cannot provide code or a clearly specified nested evaluation protocol, the Table 2 comparison should not be accepted as evidence of superiority. The title's phrase 'Reliable Tool' overstates what an internal, retrospective evaluation can support; I would ask for external validation or a strong, unbiased internal validation before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real extension of the authors' CACTUS method to five-class AMD staging using only non-imaging data. The new pieces — exhaustive class-partition ROC cut-offs, distance-from-0.5 edge weights, a confidence metric, and feature ranking — are legitimate additions, and the paper is the first to compare CACTUS against XGBoost. On its face, the Table 2 results are consistent: CACTUS variants beat Ridge, RF, LR, SGD, SVM, and XGB at every fragmentation level, with balanced accuracy in the 0.27–0.34 range. That is above random chance (0.20) but far from clinically diagnostic.\n\nWhat is genuinely good: the method is explainable, handles missing values without imputation, and the authors are transparent about the confidence metric's subjective threshold. The biological plausibility of top features (age, genetic score, visual acuity decline) is a nice sanity check. The comparison against six baselines, including XGBoost which natively handles missing data, is the right thing to do.\n\nThe soft spot is real and load-bearing. The Methods never state that the ROC cut-off search and the knowledge-graph conditional probabilities are recomputed inside each cross-validation fold for CACTUS. The ML baselines get a clear 80/20 split and 10-fold CV; for CACTUS, Figure 1 says the model is built and then \"the classification is applied to all the available individuals to assess the performance.\" If the cut-offs and graph weights are fit on the full dataset and then scored on the same individuals, the test labels have already influenced the discretization and graph weights. That would put the Table 2 advantage on shaky ground. The same problem recurs in the refinement stage: the 9 highest-ranked features are chosen from full-data rankings, and Table 3's accuracy is computed on the same data. Without code or a written nested protocol, the central comparison cannot be verified. I think the stress-test note is right, and this is not a minor caveat.\n\nThere are secondary issues too: the last-visit sampling design biases the early-screening narrative, there is no external validation cohort, and no significance tests accompany the accuracy differences. The title's \"Reliable Tool\" overstates what 0.34 balanced accuracy supports. These are addressable.\n\nVerdict: the paper deserves a serious referee, but only with the explicit task of demanding a nested-CV protocol or code. If the authors can show the abstraction and graph parameters are fit per fold and the advantage persists, this becomes a useful contribution for fragmented health datasets. As written, the main performance claim is not yet established.","headline":"A genuine extension of the authors' CACTUS method with an internally consistent head-to-head win over six ML baselines, but the central comparison may be in-sample because the abstraction and graph parameters are never described as cross-validated per fold.","tokens_in":23291,"tokens_out":2141,"would_cite":false,"duration_ms":20403,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that CACTUS, a graph-based explainable classifier, stages age-related macular degeneration from non-imaging data with higher balanced accuracy than six standard machine-learning models across every tested level of…","keywords":["Age-related macular degeneration","Explainable AI","Knowledge graphs","Classification","Missing data","Balanced accuracy","CACTUS","PageRank"],"falsifier":"Train CPR and the same baselines on a dataset built from each patient's first visit, or on a simulated cohort with no dropout, and compare balanced accuracy and top-ranked features; if CPR's advantage over XGBoost shrinks or age loses its top rank, the paper's reliability claim for early screening is not supported. Separately, check whether CPR's confidence values actually order patients by true accuracy within each stage; if they do not, the trust-from-confidence argument fails.","tokens_in":22249,"feed_emoji":"👁️","tokens_out":3947,"duration_ms":39531,"temperature":0.7,"pith_summary":"This paper tries to establish that a knowledge-graph classifier called CACTUS can stage age-related macular degeneration (AMD) using ordinary clinical variables, such as age, genetics, diet, and comorbidity, without retinal images. The authors argue that CACTUS beats six standard machine-learning baselines in balanced accuracy at every level of induced missing data, and that its PageRank variant offers the most usable confidence estimates for real screening. If true, that would matter because AMD progression is slow but irreversible, and early screening on fragmented primary-care data could flag patients at risk before sight loss. The paper also claims the features CACTUS relies on, especially age and genetic risk score, align with established medical knowledge, which supports its transparency story.","feed_headline":"Graph-based CACTUS tops six ML models at AMD staging","feed_subtitle":"It handles missing clinical data, stays transparent, and flags age and genetics as top AMD factors.","key_machinery":"The load-bearing object is the per-class knowledge graph: a weighted directed graph whose nodes are abstracted flips (Up and Down for continuous features, allele categories for SNPs) and whose edges are the absolute difference between a conditional flip probability and 0.5, encoding how much more likely two flips are to co-occur than chance. Graph centrality, computed with PageRank or total degree, is multiplied by each flip's class-specific probability to give a significance score; a patient's similarity to a class is the sum of those significances over the flips they show. The new confidence measure, the average absolute difference between the winning class score and the other class scores, is what the paper uses to claim that the PageRank variant is the most trustworthy of the three CACTUS modes.","core_discovery":"On the paper's own terms: CACTUS abstracts each continuous feature into Up and Down flips using per-feature ROC cut-offs over all class partitions, builds one weighted directed knowledge graph per AMD stage whose edges are the absolute distance of conditional flip probabilities from 0.5, and scores new patients by combining flip probabilities with graph centrality (Probabilistic, Degree, or PageRank). The central result is that on 29,908 last-visit records from the harmonised EYE-RISK dataset, CACTUS's PageRank variant reaches a balanced accuracy of 0.34 for the five AMD stages, the highest of any model and the best in 3 of 5 fragmentation experiments, while the standard baselines stay between 0.27 and 0.31. Because random guessing is 0.20, the paper interprets these numbers as evidence that the graph representations capture genuine stage-specific structure in non-imaging data, and that the accompanying confidence score lets clinicians decide when to trust a decision.","pith_inferences":["CPR's edge over XGBoost at higher fragmentation rates suggests the abstraction step may be acting as a noise filter; a direct test would be to compare CACTUS against an XGBoost model trained on discretized features to isolate the contribution of graph centrality.","The confidence-population curves could be repurposed as a general calibration tool: any classifier can be evaluated by how its accuracy changes when you restrict to high-confidence cases, with CPR serving as a baseline rather than a special case.","Because the dataset is built from each patient's last visit, the reported 'early classification' performance is measured on the most advanced visit per person; a direct test for early diagnosis would use first visits or time-to-conversion to late AMD as the outcome."],"forward_implications":["If CACTUS's PageRank classification holds up, AMD staging can be performed from longitudinal questionnaire and lab data alone, reducing dependence on specialist image grading for screening.","The confidence metric allows clinicians to defer decisions when confidence is low, concentrating scarce resources on the patients the model is least sure about.","The feature ranks provide a transparent, testable bridge to medical literature: age topping the ranks and genetic scores appearing near the top are the kinds of outputs clinicians can verify.","The finding that nine features capture most of the model's performance suggests that a minimal data-collection protocol could be sufficient for early screening.","Because balanced accuracy stays modest (0.34) even though it beats baselines, the intended use is risk triage to aid a clinician, not autonomous diagnosis."],"supporting_citations":[{"why":"Supplies the original CACTUS algorithm and design that this paper extends with multi-class abstraction, confidence, and the new classification function.","marker":"[23]"},{"why":"Supplies the EYE-RISK/E3 harmonised AMD dataset, including the genetic, dietary, clinical, and demographic features used in all experiments.","marker":"[36]"},{"why":"Provides XGBoost, the baseline that natively handles missing values and is compared against CACTUS for the first time.","marker":"[45]"},{"why":"Provides the graph-tool library used to build the weighted directed knowledge graphs that carry the classification mechanism.","marker":"[37]"},{"why":"Documents age as a major AMD risk factor, the medical knowledge used to validate CACTUS's top-ranked feature.","marker":"[25]"},{"why":"Establishes global AMD burden and the projection of rising prevalence, motivating early diagnosis as the clinical goal.","marker":"[18]"},{"why":"Defines the explainability pillars and knowledge limits that justify the confidence and rank metrics as trust aids.","marker":"[13]"}],"fun_headline_variants":["CACTUS graph method edges out ML rivals in AMD staging","CACTUS's PageRank variant leads AMD classification accuracy","Graph-based CACTUS outperforms six ML models on AMD data","CACTUS: transparent knowledge graph wins AMD staging"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that picking each patient's last recorded visit produces a representative and balanced sample of a real early-screening population, so 25 years of dropout does not systematically distort which features predict AMD stage.","fun_headline_variants_meta":{"raw":{"variants":["CACTUS graph method edges out ML rivals in AMD staging","CACTUS's PageRank variant leads AMD classification accuracy","Graph-based CACTUS outperforms six ML models on AMD data","CACTUS: transparent knowledge graph wins AMD staging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000994,"raw_usage":{"total_tokens":4223,"prompt_tokens":970,"completion_tokens":3253,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":3184}},"tokens_in":586,"tokens_out":3253,"duration_ms":24431,"temperature":1.0,"reasoning_tokens":3184,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:01:40.930704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train CPR and the same baselines on a dataset built from each patient's first visit, or on a simulated cohort with no dropout, and compare balanced accuracy and top-ranked features; if CPR's advantage over XGBoost shrinks or age loses its top rank, the paper's reliability claim for early screening is not supported. Separately, check whether CPR's confidence values actually order patients by true accuracy within each stage; if they do not, the trust-from-confidence argument fails.","supporting_citations":[{"cited_title":"CACTUS: a Comprehensive Abstraction and Classification Tool for Uncovering Structures","cited_arxiv_id":"2308.12031","evidence_quote":"Supplies the original CACTUS algorithm and design that this paper extends with multi-class abstraction, confidence, and the new classification function."},{"cited_title":"European Eye Epidemiology","cited_arxiv_id":null,"evidence_quote":"Supplies the EYE-RISK/E3 harmonised AMD dataset, including the genetic, dietary, clinical, and demographic features used in all experiments."},{"cited_title":"Salimiaghdam, M","cited_arxiv_id":null,"evidence_quote":"Documents age as a major AMD risk factor, the medical knowledge used to validate CACTUS's top-ranked feature."},{"cited_title":"Guidotti, A","cited_arxiv_id":null,"evidence_quote":"Defines the explainability pillars and knowledge limits that justify the confidence and rank metrics as trust aids."}],"review_version":2}