{"id":"1ab44b3c-2675-4f31-be36-e544c46595f7","arxiv_id":"2504.20112","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Supervised pretraining with surrogate class labels improves material property prediction on six Materials Project properties over self-supervised baselines, by 2 to 6.67 percent MAE.","lead":"This paper tests whether pretraining a crystal-structure model with broad class labels such as metal or magnetic improves later predictions of material properties. Across six properties from the Materials Project, the supervised pretraining step lowers average errors compared with self-supervised baselines, though the results have no error bars and no released code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2–6.67% MAE gains are attributed to supervised pretraining, but SPMat differs from baselines in pretraining data, augmentation, and tuning, so the causal claim is not yet isolated.","rationale":"The reader's weakest assumption—baseline tuning fairness and absent error bars—is essentially the same load-bearing concern I identify, and I agree with it. My reading sharpens the point: the paper's own Table 3 shows GNDN alone improves SSL baselines in 11 of 12 settings, so augmentation is a real confound, and the pretraining corpus used for the baselines is not explicitly matched. The central claim is causal ('supervised pretraining yields gains'), so a matched ablation is the minimum requirement. The paper's 'unrelated labels' framing is also internally weakened because the bandgap surrogate is used for the bandgap target, though the improvement pattern across other properties and surrogate labels suggests the method has broader value. Given the consistently positive direction of the tables and the partial GNDN control in Table 3, the evidence is suggestive but provisional; the reader's CONDITIONAL verdict remains appropriate.","tokens_in":20403,"tokens_out":5894,"duration_ms":61977,"concrete_test":"Retrain SSL-BT and SSL-SC baselines on the same 121,371 unstable structures with the same augmentation pipeline (AM+EM+GNDN), batch sizes (128 for BT, 256 for SC), optimizer settings, 15 pretraining epochs, and the same fine-tuning head, split, and 200-epoch schedule as SPMat, for at least 5 random seeds. Report mean ± standard deviation of MAE for all six properties and compare directly with SPMat under identical data and augmentation. If SPMat still beats these matched baselines beyond one standard error on most properties, the supervision claim is supported; if the gap closes or reverses, the reported 2–6.67% gains are confounded by corpus scale or GNDN.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that surrogate-label supervision during pretraining produces the reported 2–6.67% MAE gains—requires comparisons in which SPMat and the SSL baselines differ only in the presence of that supervision. The paper does not enforce this. In Tables 1 and 2, SSL-BT and SSL-SC are described as Crystal-Twins baselines that use only atom and edge masking, while every SPMat model additionally uses the new GNDN augmentation, different batch sizes (128 for BT, 256 for SC), and different loss hyperparameters. Table 3 ablates GNDN, but only for the bandgap surrogate label, and it does not abate the pretraining corpus or the tuning budget. The Training Details state that pretraining uses 121,371 unstable structures, but the paper never states that the SSL baselines were pretrained on the same corpus; if released Crystal-Twins checkpoints were used, the comparison conflates supervision with pretraining-data scale. With no repeated seeds and no error bars, reported differences such as 0.044 vs 0.043 for Formation Energy and 0.502 vs 0.481 for Fermi Energy are within plausible run-to-run variability. The improvements are therefore not yet cleanly attributable to supervised pretraining rather than to augmentation, corpus scale, or tuning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPMat, a supervised pretraining framework for crystalline material property prediction. SPMat extends two SSL baselines (SimCLR and Barlow Twins) by adding surrogate labels—bandgap class, metallicity, magnetism, and gap directness—to the pretraining loss, and introduces a graph-level augmentation called Graph-level Neighbor Distance Noising (GNDN) that adds uniform noise to neighbor distances in the crystal graph. The authors report that fine-tuning SPMat-pretrained CGCNN encoders on six downstream regression tasks (formation energy, bandgap, density, Fermi energy, energy per atom, atomic density) yields MAE improvements between 2% and 6.67% over CGCNN without pretraining and over the Crystal-Twins SSL baselines. They also provide t-SNE visualizations, an ablation of the GNDN augmentation, and an analysis of loss components.","tokens_in":20645,"tokens_out":5751,"duration_ms":58689,"significance":"If the central claim is valid, the paper would make a useful contribution by showing that inexpensive class labels (e.g., metal/non-metal) can guide SSL pretraining for materials, reducing the need for per-property labeled data. The proposed supervised Barlow-Twin loss and the GNDN augmentation are sensible ideas that could generalize to other graph-based scientific domains. The paper explicitly builds on Crystal-Twins and compares against strong SSL baselines, which is commendable. However, the experimental evidence as presented does not yet isolate the effect of supervision from the effect of augmentation and tuning, and the lack of repeated runs means the reported gains are not statistically grounded. The strengths are the breadth of downstream tasks, the inclusion of an augmentation ablation, and the qualitative embedding analysis; the weaknesses are the confounded comparison protocol and the absence of uncertainty quantification.","major_comments":[{"comment":"The central claim that surrogate-label supervision produces the reported 2-6.67% MAE gains is not isolated from the GNDN augmentation. In Tables 1 and 2, every SPMat model uses atom masking, edge masking, and GNDN, while the SSL baselines are described as using only atom and edge masking. Table 3 shows that adding GNDN to the SSL baselines improves their MAE in 11 of 12 settings (e.g., bandgap from 0.281 to 0.276 for BT and from 0.287 to 0.274 for SimCLR). Moreover, a comparison of SPMat without GNDN (Table 3, 'Atom+Edge' rows) against SSL with GNDN ('Atom+Edge+GNDN' rows) shows the unsupervised model with GNDN is better or tied in most cases, e.g., bandgap 0.276 vs 0.283 (BT) and 0.274 vs 0.283 (SimCLR). The paper therefore confounds the supervision signal with the augmentation. To support the stated claim, the authors must vary only the presence of supervision while holding augmentation, batch size, and pretraining corpus fixed (e.g., SPMat-Atom+Edge vs SSL-Atom+Edge and SPMat-GNDN vs SSL-GNDN), and report results from multiple runs.","section":"Results (Tables 1-3) and Training details"},{"comment":"No uncertainty quantification is provided. All MAE values are single deterministic numbers, and differences as small as 0.001 (e.g., Table 1, Formation Energy: 0.043 vs 0.044) are treated as meaningful. GNN training on 33,990 materials is susceptible to run-to-run variance from random initialization, data splits, and augmentation sampling; without repeated seeds and standard deviations, the claimed improvements cannot be distinguished from noise. The authors should report mean ± std over at least 3-5 random seeds and, ideally, paired significance tests or confidence intervals for the key comparisons. This is particularly important because several of the reported improvements are within the range typically seen from random seed variation.","section":"Results (Tables 1-6)"},{"comment":"The framing that surrogate labels are 'unrelated' to downstream tasks is contradicted by the bandgap surrogate label. In the Results, the BG label is defined as a discretization of the same bandgap property that is later used as a regression target (conductor/semiconductor/insulator based on bandgap values), and in Tables 1 and 3 the bandgap surrogate yields the best or second-best result on bandgap prediction. Using a coarse label of the target property as a 'surrogate' does not demonstrate that supervision on unrelated labels transfers to the target. The paper should either drop the 'unrelated' claim, exclude the BG surrogate when evaluating the transfer argument, or provide a direct analysis showing that the gains persist when only clearly orthogonal labels (e.g., is-metal, is-magnetic) are used for a given target.","section":"Abstract/Results ('unrelated labels')"},{"comment":"The paper states that SPMat is pretrained on up to 121,371 unstable structures from the Materials Project, but it never states whether the SSL baselines (SSL-BT and SSL-SC) were pretrained on the same corpus and with the same training budget. If the baselines are taken from the released Crystal-Twins checkpoints, then the comparison conflates supervision with pretraining-data scale, number of epochs, and hyperparameter tuning. The authors must disclose the exact pretraining setup for the baselines and, if necessary, retrain them under identical data and budget conditions to make the comparison fair. This is a load-bearing condition for attributing the observed gains to supervised pretraining.","section":"Training details"}],"minor_comments":[{"comment":"The noise magnitude δ is mentioned in the Algorithm 1 description and in the Methods text ('range of −.5 to .5'), but the value of δ is not listed in the Training Details hyperparameters. Please specify δ explicitly and, if possible, discuss its sensitivity.","section":"Methods (GNDN)"},{"comment":"The loss definitions in Eq. (1) and Eq. (2) use symbols L_Attract, L_Repel, L_Corr., and L_Decorr. that are not formally defined in the main text. The Methods section later gives concrete forms (Eq. 7 for SupCon and Eq. 14 for supervised Barlow Twins), but the main-text equations should either be defined or explicitly cross-referenced to avoid ambiguity.","section":"Results (Eq. 1-2)"},{"comment":"There are repeated typos: 'pertaining' should be 'pretraining' in the abstract and in the Introduction ('first exploration of supervised pertaining with surrogate labels'). These should be corrected.","section":"Abstract and Introduction"},{"comment":"The statement 'code will be shared on a public GitHub repository after the work is accepted' limits reproducibility. Providing code, a configuration file with all hyperparameters, and random seeds at submission would strengthen the paper.","section":"Code Availability"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely topic and the proposed ideas are worth pursuing, but the experimental protocol as written does not support the causal claim that supervised pretraining causes the 2-6.67% improvements. The main fix—running controlled comparisons with and without supervision while keeping augmentation and pretraining data identical, plus repeated seeds—is feasible and should be required before further consideration. I would also suggest the editors ask the authors to temper the 'new benchmark' language unless the controlled results hold up. The issue is not a fundamental flaw in the method; it is a matter of evidence quality."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a legitimate incremental step: it takes supervised contrastive learning and a sample-wise supervised Barlow Twins loss into crystal property prediction, adds GNDN, a simple neighbor-distance-noise augmentation, and reports consistent MAE improvements over CGCNN and Crystal-Twins baselines across six properties. The GNDN rule is clean, the loss ablations in Table 5 are the right experiments, and the pattern of gains being stable across four surrogate labels is more convincing than any single number. I buy that supervision during pretraining helps here. I do not buy the exact magnitude yet. There are no error bars or repeated runs anywhere. Differences like 0.044 vs 0.043 for formation energy or 0.502 vs 0.481 for Fermi energy are within plausible run-to-run noise for this kind of model. The comparison to SSL baselines is also confounded: SPMat uses GNDN plus different batch sizes and loss hyperparameters, and the paper never states that the SSL baselines were pretrained on the same 121k unstable corpus. Table 3 ablates GNDN for the bandgap surrogate and shows supervision helps on top of augmentation, which partially addresses the concern, but it does not control the corpus or tuning budget. The 'unrelated labels' framing is also overstated: the bandgap surrogate is a discretized version of the bandgap target, which is one of the six downstream properties. That does not sink the method, because the other surrogate labels (metal, magnetic, direct gap) still improve other properties, but it means the cleanest demonstration is weaker than advertised. Code is promised only after acceptance, which is standard but a shame for a methods paper. On the whole: the core idea is sound and the experiments are mostly well designed; what is missing is statistical rigor and one clean controlled comparison. This deserves a serious peer-review slot, not a desk reject. I would ask the authors to add repeated runs with standard deviations, state the pretraining corpus for baselines explicitly, and either avoid the bandgap surrogate for the bandgap target or move it to a supplementary. If those come back, I would cite this.","headline":"A solid incremental method for supervised pretraining of crystal property predictors, with a real confound problem in the headline numbers.","tokens_in":21198,"tokens_out":2573,"would_cite":true,"duration_ms":25922,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Supervised pretraining with broad class labels improves material property prediction by 2% to 6.67%.","keywords":["material property prediction","self-supervised learning","surrogate labels","supervised pretraining","crystal graph neural networks","contrastive learning","Barlow Twins","graph augmentation"],"falsifier":"Run all compared methods with the same backbone, optimizer, batch size, and augmentation set for 10 random seeds; if the best SPMat variant’s average MAE does not exceed the best baseline by more than the seed standard deviation on every one of the six properties, the claimed advantage is not established. Separately, fix the surrogate labels and compare SPMat with and without GNDN: the paper’s Table 3 predicts MAE should rise in every property when GNDN is removed.","tokens_in":20134,"feed_emoji":"🧪","tokens_out":8876,"duration_ms":77859,"temperature":0.7,"pith_summary":"This paper claims that a material-property model can be pretrained more effectively with broad, inexpensive class labels—metal vs nonmetal, magnetic vs nonmagnetic, conductor/semiconductor/insulator, direct vs indirect gap—than with purely self-supervised objectives, even when the downstream job is a regression on an unrelated property. It introduces SPMat, a supervised-pretraining framework built on a CGCNN encoder, with two loss variants that pull same-class embeddings together and push different-class embeddings apart, plus a graph-level augmentation (GNDN) that adds noise to neighbor distances instead of deforming atomic positions. After fine-tuning on six DFT-derived properties, the pretrained models consistently beat an untrained CGCNN and two self-supervised baselines, with mean-absolute-error improvements of 2% to 6.67%. If the claim holds, foundation models for materials become cheaper to build, because surrogate class labels are already present in large crystallographic databases, and high-throughput screening gains a more accurate first-pass filter.","feed_headline":"Surrogate labels cut material prediction error by up to 6.67%","feed_subtitle":"Broad class labels guide pretraining; fine-tuned crystal graphs beat SSL baselines on all six properties.","key_machinery":"The load-bearing mechanism is the class-masked, sample-wise similarity matrix in the supervised Barlow-Twins loss (Eq. 14) and, for the SimCLR variant, a supervised contrastive objective (Eq. 7) that treats every same-class sample as a positive. The supervised Barlow-Twins loss replaces the original feature-wise cross-correlation with $C = \\frac{1}{D} \\tilde z_1 \\tilde z_2^T \\in \\mathbb{R}^{N \\times N}$, then applies a mask $M_{ij}$ with $M_{ij}=1$ when surrogate labels $y_i=y_j$ and $M_{ij}=0$ otherwise, driving same-class pairs toward similarity $+1$ and different-class pairs toward $-1$. The companion augmentation GNDN samples $\\varepsilon_{ij} \\sim U(-\\delta, \\delta)$ and forms perturbed distances $d'_{ij} = d_{ij} + \\varepsilon_{ij}$, which are then expanded in a Gaussian basis; this preserves the crystal structure while providing the varied views contrastive pretraining needs.","core_discovery":"The central claim is that crude class labels are a useful pretraining signal for material representations even when they carry no direct information about the regression target. The paper adapts the SimCLR and Barlow Twins objectives so that same-class augmented views are attracted or correlated toward +1 and different-class views are repelled or decorrelated toward -1; for Barlow Twins this is done by replacing the feature-wise cross-correlation with a sample-wise similarity matrix $C = \\frac{1}{D} \\tilde z_1 \\tilde z_2^T$ masked by surrogate-label equality. After fine-tuning for formation energy, bandgap, density, Fermi energy, energy per atom, and atomic density, every SPMat configuration outperforms the corresponding baseline, with best MAE gains from 2% to 6.67%; the bandgap label is the most consistently good surrogate for the Barlow-Twins variant. The paper also reports that GNDN, which adds uniform noise in the range $\\pm 0.5$ Å to neighbor distances before Gaussian expansion, improves results in 11 of 12 standard SSL settings and in every SPMat setting.","pith_inferences":["If the gains replicate, the loss design is backbone-agnostic: it only needs embeddings and labels, so SPMat should transfer to other graph or transformer encoders without changing the pretraining objective.","A natural ablation is to replace curated surrogate labels with unsupervised cluster assignments of DFT descriptors; if gains persist, the benefit comes from class-conditional grouping rather than the specific physics of metal/magnet/bandgap classes.","The paper reports no repeated seeds or confidence intervals, so the smallest claimed gains (around 2% on formation energy) could be within run-to-run noise; a multi-seed benchmark is the cheapest way to separate the stable part of the result from the noisy part.","Because the GNDN noise level is a single value ($\\delta=0.5$ Å), a per-property sensitivity sweep is untested; density and formation energy may tolerate or prefer different noise scales than bandgap."],"forward_implications":["A single pretrained CGCNN encoder can serve several downstream regression tasks: all six tested properties improve without any change to the pretraining stage.","The method is not tied to a particular surrogate label; bandgap, metallicity, magnetism, and gap-directness each produce gains, so whatever class information a database already carries can be used.","GNDN is claimed to be a drop-in augmentation: adding it to atom and edge masking improves MAE in 11 of 12 SSL baseline settings and in all SPMat settings reported in Table 3.","Guided pretraining also sharpens classification: SPMat-BT raises accuracy over standard SSL by 1.26% to 2.00% on the three binary material-class tasks, and t-SNE visualizations show tighter, better-separated clusters."],"supporting_citations":[{"why":"Supplies the SimCLR contrastive objective and the two-augmented-views setup that SPMat-SimCLR augments with surrogate labels.","marker":"[14]"},{"why":"Supplies the Barlow Twins redundancy-reduction loss that SPMat reworks into a sample-wise, label-masked objective.","marker":"[15]"},{"why":"Supplies the DFT-computed crystal structures, surrogate class labels, and six downstream property labels used for pretraining and fine-tuning.","marker":"[18]"},{"why":"Provides the SSL baselines (SimCLR and Barlow Twins on CGCNN) that SPMat must beat, along with the atom/edge masking recipe GNDN extends.","marker":"[27]"},{"why":"Provides the CGCNN encoder architecture and its hyperparameters, which serve as the shared backbone for all models in the comparison.","marker":"[30]"},{"why":"Supplies the supervised contrastive (SupCon) loss formulation that SPMat-SimCLR adapts to surrogate labels.","marker":"[32]"},{"why":"Supplies the negative-sample-free contrastive perspective used to design the different-class dissimilarity term in the supervised Barlow-Twins loss.","marker":"[35]"}],"fun_headline_variants":["Surrogate class labels improve material property prediction by 6.7%","Unrelated class labels still help pretrain material property models","Class-label pretraining outperforms self-supervision on six properties","Surrogate labels give up to 6.7% error reduction for material properties","Class info as surrogate signal: 6.7% better material predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that CGCNN, SSL-SimCLR, and SSL-Barlow-Twins baselines are comparably tuned and that the 2% to 6.67% MAE gaps are larger than seed-to-seed variation, since the paper reports no standard deviations, repeated seeds, or hyperparameter searches.","fun_headline_variants_meta":{"raw":{"variants":["Surrogate class labels improve material property prediction by 6.7%","Unrelated class labels still help pretrain material property models","Class-label pretraining outperforms self-supervision on six properties","Surrogate labels give up to 6.7% error reduction for material properties","Class info as surrogate signal: 6.7% better material predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002037,"raw_usage":{"total_tokens":7960,"prompt_tokens":996,"completion_tokens":6964,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":6870}},"tokens_in":612,"tokens_out":6964,"duration_ms":49317,"temperature":1.0,"reasoning_tokens":6870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:54:31.781269+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run all compared methods with the same backbone, optimizer, batch size, and augmentation set for 10 random seeds; if the best SPMat variant’s average MAE does not exceed the best baseline by more than the seed standard deviation on every one of the six properties, the claimed advantage is not established. Separately, fix the surrogate labels and compare SPMat with and without GNDN: the paper’s Table 3 predicts MAE should rise in every property when GNDN is removed.","supporting_citations":[{"cited_title":"& Hinton, G","cited_arxiv_id":null,"evidence_quote":"Supplies the SimCLR contrastive objective and the two-augmented-views setup that SPMat-SimCLR augments with surrogate labels."},{"cited_title":"& Deny, S","cited_arxiv_id":null,"evidence_quote":"Supplies the Barlow Twins redundancy-reduction loss that SPMat reworks into a sample-wise, label-masked objective."},{"cited_title":"& Barati Farimani, A","cited_arxiv_id":null,"evidence_quote":"Provides the SSL baselines (SimCLR and Barlow Twins on CGCNN) that SPMat must beat, along with the atom/edge masking recipe GNDN extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the supervised contrastive (SupCon) loss formulation that SPMat-SimCLR adapts to surrogate labels."}],"review_version":1}