{"id":"8ac62c48-745b-482d-9356-dad5d4da5157","arxiv_id":"2502.08772","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A protein flexibility predictor based on persistent sheaf Laplacians claims a 32% improvement over GNM, but the gain is an artifact of in-sample regression and disappears in the authors' own blind tests.","lead":"The authors apply a new topological data analysis method, the persistent sheaf Laplacian, to predict protein flexibility from protein structures. They claim a 32% improvement over a standard model, but the result depends on fitting the model to the same proteins it is tested on.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 32% improvement over GNM rests on in-sample regression with radii tuned on the full 364-protein set; the paper's own protein-level cross-validation (PCC 0.452) falls below the GNM baseline (0.565), so the central performance claim is unsupported.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the headline 32% improvement is based on in-sample regression with parameter selection on the full 364-protein set, compared against a literature GNM baseline. The full text confirms this in Section 2.1.2, and the paper's own blind validation results in Tables 4 and 5 contradict the headline. Protein-level 10-fold CV yields PCC 0.452, below GNM's 0.565, and leave-one-protein-out GBDT yields 0.588, only marginally above GNM despite using additional global and local features. The Cα-level 10-fold CV value of 0.840 in Table 6 is not evidence for cross-protein generalization because atoms from the same protein appear in both training and test folds. The mathematical PSL construction and the biological motivation are not themselves called into question; the central performance claim, however, is not supported by a valid evaluation protocol. Therefore the reader's REJECT verdict stands unchanged.","tokens_in":13014,"tokens_out":2420,"duration_ms":21680,"concrete_test":"Recompute the Table 1 comparison with protein-level cross-validation: for each of 10 folds (or leave-one-protein-out), fit the linear regression on PSL features using the fixed radii 6, 9, 12 Å (no selection on held-out proteins), predict B-factors for held-out proteins, and pool predictions to compute Pearson correlation. Compute GNM predictions (cutoff 7 Å) for the same proteins and evaluate them on the same folds. If the held-out PSL PCC does not exceed the matched GNM PCC on the identical split, the 32% claim fails; report the mean and standard deviation across folds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 2.1.2, the authors state that for the Table 1 results they 'conducted experiments on the set of 364 proteins with a few sets of parameters and chose those that yielded a good average Pearson correlation coefficient over the entire set,' then fit linear regression on those same 364 proteins. Thus the PSL Pearson correlation of 0.751 in Table 1 is an in-sample correlation of a model whose filtration radii (6, 9, 12 Å) and eigenvalue statistics were selected on the test set. The GNM baseline of 0.565 is taken from Park et al. (2013), so the comparison mixes a fitted regression with a fixed, parameter-free baseline and reports no error bars or cross-validation. The paper's own blind validation undermines the claim: leave-one-protein-out GBDT with additional features gives PCC 0.588 on the superset (Table 4), and protein-level 10-fold CV gives 0.452 (Table 5), below GNM. Consequently, the 32% improvement claimed in the abstract and Section 2.1.2 is not supported as a predictive performance gain; at best it is an in-sample descriptive fit. The PSL construction may be a useful feature, but the central claim requires a fair, held-out comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces persistent sheaf Laplacian (PSL) features for protein B-factor prediction. The authors construct local sheaf Laplacians on alpha-complex filtrations around each Cα atom, extract spectral statistics at three filtration radii, and use these as features in linear regression and in machine-learning pipelines with additional structural features. The headline claim is that the PSL model improves B-factor prediction accuracy by 32% over the Gaussian network model (GNM) on a 364-protein dataset, and the paper also presents case studies and a 'blind' machine-learning validation section.","tokens_in":13356,"tokens_out":4601,"duration_ms":42850,"significance":"The construction of a persistent sheaf Laplacian for per-atom, multiscale protein descriptors is mathematically interesting, and the authors share code and data, which is commendable. If the reported 32% improvement over GNM were obtained under a fair predictive protocol, the method would be a valuable addition to protein flexibility analysis. However, the central evidence does not support that claim: the Table 1 result is an in-sample linear regression whose radii were selected on the same proteins used for evaluation, and the paper's own held-out validation shows a much smaller gain (PCC 0.588 leave-one-protein-out GBDT) or a loss relative to GNM (PCC 0.452 protein-level 10-fold CV). The significance of the work as presented is therefore not established.","major_comments":[{"comment":"The 'predictions' in Table 1 are in-sample fits, not predictions. The authors state that they conducted experiments on the 364-protein set with a few sets of parameters and chose those that gave a good average Pearson correlation over the entire set, and the linear regression coefficients were then fit on the same proteins. The reported PSL PCC of 0.751 and the resulting '32% improvement over GNM' are therefore descriptive in-sample correlations. The GNM baseline is taken from Park et al. (2013) rather than recomputed under the same pipeline, so the comparison mixes a fitted model with a fixed baseline and no error bars. The abstract's wording 'in predicting B-factors' is not supported by this experiment; a held-out comparison using an identical evaluation protocol is required.","section":"§2.1.2, Table 1"},{"comment":"The paper's own blind validation contradicts the headline claim. Leave-one-protein-out GBDT achieves PCC 0.588 on the superset (Table 4), only slightly above the literature GNM value of 0.565 cited in Table 1, while protein-level 10-fold cross-validation gives PCC 0.452 (Table 5), below the GNM baseline. Unless this discrepancy is explained—for example by protocol differences or by isolating the contribution of the PSL features—the 32% improvement claim cannot be maintained. These results need to be discussed honestly in the abstract and conclusions.","section":"§2.3.6, Tables 4 and 5"},{"comment":"The Cα-level 10-fold cross-validation reported in Table 6 splits individual atoms randomly rather than by protein. Because atoms from the same protein appear in both training and test sets, and B-factors within a protein are strongly correlated, the high PCC values (0.839 and 0.840) mostly reflect within-protein interpolation and do not measure generalization to unseen proteins. This experiment should not be presented as evidence for cross-protein 'blind' prediction; protein-level splits, such as those in Tables 4 and 5, are the relevant evaluation.","section":"§2.3.6, Table 6"}],"minor_comments":[{"comment":"The phrase 'increase in accuracy of 32%' should be qualified as an in-sample Pearson correlation improvement relative to a literature GNM baseline, not as a predictive accuracy gain, unless a held-out comparison supports it.","section":"Abstract"},{"comment":"The sentence describing the leave-one-protein-out experiment says the authors trained models five times and calculated the average from 'ten sets of modeling predictions'; this is confusing—presumably ten folds were used, so the wording should be clarified.","section":"§2.3.6"},{"comment":"The random forest parameter 'min samples leaf = 0.8' is unusual; if it is a fraction, the denominator should be stated, and if it is an integer, 0.8 is not valid. Please clarify.","section":"Table 3"},{"comment":"Section 2.1.2 uses L0 sheaf Laplacians and eigenvalue statistics including the median, while Section 2.3.2 uses L1 matrices with the standard deviation; the reasons for these different choices should be stated.","section":"§2.3.2 and §2.1.2"},{"comment":"For the 1WHI case study the PSL RMSE is not reported numerically; the text says only that it is nearly half of the GNM RMSE of 6.59. Please give the exact value.","section":"§2.2, Figure 5"}],"recommendation":"reject","confidential_remarks":"The main quantitative claim of the paper appears to be an artifact of in-sample evaluation, and the authors' own blinded experiments argue against the central conclusion. I recommend rejection. A future manuscript that reframes the contribution as a descriptor study, removes or heavily qualifies the 32% claim, and provides protocol-matched comparisons with GNM on held-out proteins could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real application of a tool the same group previously introduced, and the per-atom sheaf construction is concrete and reproducible (code and data are up). But the headline 32% gain over GNM is not a predictive result. Section 2.1.2 openly says the filtration radii were chosen by maximizing average PCC on the full 364-protein set, and the linear regression is trained on the same B-factors it is scored against. So Table 1's 0.751 vs 0.565 is an in-sample description, not a forecast. The GNM number, meanwhile, is taken from Park et al., so you are mixing a fitted model with a fixed baseline.\n\nThe paper does better in the blind section. Leave-one-protein-out GBDT gets 0.588 on the superset, essentially tied with GNM's 0.565; protein-level 10-fold CV drops to 0.452, below GNM. Atom-level 10-fold CV gives 0.84, but that splits the same proteins across train/test, so it inflates the number. The authors report all these numbers honestly, which I credit, but they don't reconcile them with the abstract's 32%.\n\nThe case studies (calmodulin, etc.) show large RMSE reductions, but again the PSL model is likely fitted on the same protein, so those are less convincing than they look.\n\nWhat is genuinely useful: the construction of per-residue sheaf features via a target-atom label and distance-weighted restriction maps is a reasonable way to localize a topological descriptor, and the eigenspectrum statistics are natural. The comparison set is standard in this niche. As a descriptor for downstream ML, PSL is worth a look.\n\nBottom line: the mathematical content is fine and the code is available, but the central performance claim is not supported. A fair version of this paper would report the protein-level CV as the headline result, and then the contribution is 'modest improvement at best.' The authors need to redo the comparison with held-out parameter selection and matched baselines before the 32% claim can be taken seriously.\n\nFor peer review: yes, this deserves referee time, because the method is concrete and the flaws are fixable. But any referee should insist on a cross-validated, protocol-matched comparison. I'd probably reject the current version, but the work is salvageable.","headline":"The PSL features are a plausible new descriptor, but the 32% claim is an in-sample artifact and the paper's own cross-validation says the real gain is small or negative.","tokens_in":13822,"tokens_out":2915,"would_cite":false,"duration_ms":27484,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["55N31","92C40"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that persistent sheaf Laplacian spectra of Cα atoms predict protein B-factors with a Pearson correlation of 0.751, a 32% improvement over the Gaussian network model's 0.565.","keywords":["protein flexibility","B-factor prediction","persistent sheaf Laplacian","topological data analysis","Gaussian network model","Debye-Waller factor","multiscale spectra","machine learning"],"falsifier":"Run protein-level 10-fold cross-validation with the same 15 PSL features and linear regression used in Table 1, and average the Pearson correlation over held-out proteins. If that average does not exceed the 0.565 reported for GNM, the 32% improvement is an in-sample artifact; the paper's own leave-one-protein-out blind runs (0.588 with GBDT, 0.452 with protein-level 10-fold CV) already point in this direction.","tokens_in":12833,"feed_emoji":"🧬","tokens_out":7660,"duration_ms":65833,"temperature":0.7,"pith_summary":"Protein flexibility, measured by the B-factor, governs how proteins bind, catalyze, and signal, but classical elastic network models capture it only coarsely. This paper introduces the persistent sheaf Laplacian (PSL), a localized topological data analysis tool, to encode the local geometry and topology around each Cα atom. The authors report that a linear regression on PSL eigenvalue statistics reaches a Pearson correlation of 0.751 on a 364-protein superset, a 32% improvement over the Gaussian network model's 0.565. They also show that combining PSL features with global and local structural features supports blind machine-learning B-factor prediction.","feed_headline":"Sheaf Laplacian features beat GNM by 32% on protein B-factors","feed_subtitle":"Persistent sheaf Laplacian spectra capture local flexibility, reaching 0.751 Pearson correlation on 364 proteins.","key_machinery":"A cellular sheaf assigns a vector space (stalk) to each simplex and a linear map to each face relation; the persistent sheaf Laplacian generalizes the persistent Laplacian by incorporating these stalks and maps. Here the stalk over every simplex is R, and for a fixed atom A the restriction map along edge vivj is scalar multiplication by qj/rij, with qA = 0 and qj = 1 for all other atoms, so the resulting operator emphasizes the local neighborhood of A. The 0th persistent sheaf Laplacian is built from alpha complexes of the surrounding Cα atoms at radii 6, 9, and 12 angstroms; its nonzero eigenvalues supply five per-atom statistics per radius, giving a multiscale local descriptor that linear regression or tree-based models can use directly.","core_discovery":"The central discovery is that the harmonic and non-harmonic spectra of 0th persistent sheaf Laplacians, computed separately for each Cα atom over three filtration radii, carry enough local structural information for accurate B-factor prediction. By labeling the target atom with a zero stalk weight and its neighbors with unit weights, the construction makes the Laplacian spectrum sensitive to the atom's immediate environment. Five statistics per radius — zero-eigenvalue count and the maximum, minimum, mean, and median (or standard deviation) of the nonzero eigenvalues — form 15 per-atom features, on which linear regression attains an average Pearson correlation of 0.751 across the 364-protein superset, compared with 0.565 for GNM. On the small, medium, and large subsets the same features reach 0.927, 0.728, and 0.643 respectively.","pith_inferences":["Because the headline 32% improvement is computed on the same proteins used to fit the regression, the fair generalization test is the paper's own leave-one-protein-out result; at 0.588 PCC the PSL+GBDT blind model barely clears GNM's 0.565, suggesting the in-sample gain does not carry over to unseen proteins.","The atom-level 10-fold cross-validation (0.84 PCC) pools Cα atoms from the same proteins into training and test folds, so its high correlation likely reflects within-protein memorization rather than cross-protein generalization.","The per-atom labeling trick — one distinguished zero label among unit neighbors — could be transferred to other local prediction tasks, such as solvent accessibility or mutation stability, by redefining the stalk labels."],"forward_implications":["PSL features can be generated for any Cα atom from coordinates alone and plugged into linear regression to achieve 0.751 Pearson correlation on the 364-protein superset.","The method outperforms GNM, NMA, ASPH, opFRI, and pfFRI on the small, medium, and large protein subsets in the paper's comparisons.","On proteins identified as GNM failures (calmodulin, mTFP1, 1V70, and L14), PSL reduces RMSE substantially, for example from 17.9 to 2.78 on 1V70.","Combining PSL features with secondary-structure assignments, packing density, and global PDB metadata enables blind prediction of B-factors for proteins without experimental B-factors."],"supporting_citations":[{"why":"Supplies the small, medium, and large protein datasets and the GNM and NMA baseline correlation values used throughout the comparison.","marker":"[14]"},{"why":"Constructs the 364-protein superset and provides the opFRI and pfFRI baseline results that PSL is compared against.","marker":"[25]"},{"why":"Updates the superset to its 364-protein form and identifies specific proteins where GNM fails, which the case studies rely on.","marker":"[30]"},{"why":"Contributes 263 high-resolution proteins to the superset and motivates the flexibility-rigidity index baseline used for B-factor prediction.","marker":"[13]"},{"why":"Provides the atom-specific persistent homology (ASPH) method whose Bottleneck and Wasserstein results are compared with PSL.","marker":"[5]"},{"why":"Defines the Gaussian network model, the classical baseline that the paper claims to improve by 32%.","marker":"[21]"},{"why":"Introduces the persistent sheaf Laplacian construction that the feature-generation pipeline is built on.","marker":"[41]"},{"why":"Supplies the secondary-structure features used in the blind machine-learning prediction setup.","marker":"[45]"}],"fun_headline_variants":["Persistent sheaf Laplacian lifts B-factor prediction 32% over GNM","Sheaf Laplacian spectra match B-factors at 0.751 correlation","Sheaf Laplacian outperforms Gaussian network model by 32%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the Pearson correlation of a linear regression fit on the full 364-protein set, with filtration radii selected on that same set, is a valid measure of predictive performance, and that literature GNM values are directly comparable.","fun_headline_variants_meta":{"raw":{"variants":["Persistent sheaf Laplacian lifts B-factor prediction 32% over GNM","Sheaf Laplacian spectra match B-factors at 0.751 correlation","Sheaf Laplacian outperforms Gaussian network model by 32%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001152,"raw_usage":{"total_tokens":4751,"prompt_tokens":900,"completion_tokens":3851,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":3784}},"tokens_in":516,"tokens_out":3851,"duration_ms":24174,"temperature":1.0,"reasoning_tokens":3784,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T23:43:23.605106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run protein-level 10-fold cross-validation with the same 15 PSL features and linear regression used in Table 1, and average the Pearson correlation over held-out proteins. If that average does not exceed the 0.565 reported for GNM, the 32% improvement is an in-sample artifact; the paper's own leave-one-protein-out blind runs (0.588 with GBDT, 0.452 with protein-level 10-fold CV) already point in this direction.","supporting_citations":[{"cited_title":"Coarse grained normal mode analysis vs","cited_arxiv_id":null,"evidence_quote":"Supplies the small, medium, and large protein datasets and the GNM and NMA baseline correlation values used throughout the comparison."},{"cited_title":"Fast and anisotropic flexibility-rigidity index for protein flexibility and fluctuation analysis","cited_arxiv_id":null,"evidence_quote":"Constructs the 364-protein superset and provides the opFRI and pfFRI baseline results that PSL is compared against."},{"cited_title":"Communication: Capturing protein multiscale thermal fluctu- ations","cited_arxiv_id":null,"evidence_quote":"Updates the superset to its 364-protein form and identifies specific proteins where GNM fails, which the case studies rely on."},{"cited_title":"Multiscale multiphysics and multidomain models—flexibility and rigidity","cited_arxiv_id":null,"evidence_quote":"Contributes 263 high-resolution proteins to the superset and motivates the flexibility-rigidity index baseline used for B-factor prediction."},{"cited_title":"Atom-specific persistent homology and its application to protein flexibility analysis","cited_arxiv_id":null,"evidence_quote":"Provides the atom-specific persistent homology (ASPH) method whose Bottleneck and Wasserstein results are compared with PSL."},{"cited_title":"Direct evaluation of thermal fluctuations in proteins using a single-parameter harmonic potential","cited_arxiv_id":null,"evidence_quote":"Defines the Gaussian network model, the classical baseline that the paper claims to improve by 32%."},{"cited_title":"Persistent sheaf Laplacians","cited_arxiv_id":null,"evidence_quote":"Introduces the persistent sheaf Laplacian construction that the feature-generation pipeline is built on."},{"cited_title":"STRIDE: a web server for secondary structure assignment from known atomic coordinates of proteins","cited_arxiv_id":null,"evidence_quote":"Supplies the secondary-structure features used in the blind machine-learning prediction setup."}],"review_version":1}