{"id":"176b9240-386c-460e-aed9-a29ab00ef552","arxiv_id":"2608.06448","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ED-CSP generates lattice parameters and atomic coordinates from multi-view electron diffraction spot lists, achieving 57.49 percent MR@5 on held-out CHILI-100K materials and 66.27 percent with larger pretraining.","lead":"This paper trains a neural network, ED-CSP, to reconstruct 3D crystal structures from sparse electron diffraction spot patterns plus the chemical composition. On a standard benchmark of 2,075 materials, it beats a state-of-the-art model that conditions on X-ray powder diffraction, and the authors show the diffraction input matters beyond composition.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Registry-1M pretraining pool is not certified CHILI-disjoint; the 66.27% headline and the 53.52% absent-formula generative claim could be inflated by pretraining near-duplicate memorization, and the post-hoc check does not cover the absent-formula subset.","rationale":"The reader's weakest-assumption analysis identifies exactly the same vulnerability: the registry-1M precursor was assembled before the final CHILI-exclusion filtering, and the only leakage check is a post-hoc sensitivity analysis on a subset. My independent reading of the full text confirms this is the single most load-bearing concern. The paper's own Methods section explicitly separates the registry-1M precursor from the 4.85-million-structure ED-CS corpus precisely because the precursor predates the filtering, and the sensitivity analysis is limited to the full 2,075-query split; it does not isolate the 1,024 absent-formula queries used to argue for 'true generative capability beyond exact-formula retrieval.' This matters because the absent-formula subset is the cleanest evidence that the model is not performing library lookup, but if the registry pool contains a structural near-duplicate of a test query, the model does not need to generate that structure; it can retrieve a memorized prototype. The same logic threatens the headline 66.27% number, though the comparative claim against PXRDGen is less vulnerable because both systems are evaluated on the same test set and the registry benefit would be the main casualty. The paper is otherwise methodologically strong: external held-out benchmark, paired confidence intervals, input interventions (zero-ED drops MR@1 from 51.66% to 17.35%), and a converged composition-only reference. These strengths support the CHILI-only results and the signal-use argument, but they do not resolve the pretraining-pool provenance issue. My proposed check is concrete and feasible: apply the same strict matcher to compute registry-pool/test overlap, then re-evaluate on the clean subset, both for the full split and the absent-formula stratum. This directly settles whether the registry-1M gains and the generative claim survive leakage exclusion. Because the paper already discloses the provenance gap and provides a partial post-hoc check, the appropriate disposition is unchanged conditional acceptance pending this verification, rather than rejection.","tokens_in":12199,"tokens_out":4905,"duration_ms":39751,"concrete_test":"Run the ED-CS CHILI-exclusion manifest (same strict StructureMatcher settings) against the registry-1M pretraining pool to identify all CHILI test queries with a structural near-duplicate in that pool; then recompute MR@1/MR@5 for registry-1M ED-CSP, CHILI-only ED-CSP, and PXRDGen on the clean subset (queries with no near-duplicate), and separately on the clean subset of the 1,024 absent-formula queries. If the registry-1M advantage and the absent-formula 53.52% persist with non-overlapping confidence intervals, the leakage concern is resolved; if they shrink substantially, the headline requires a re-run with a filtered pretraining pool.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the provenance of the registry-1M warm-start pool relative to the 2,075 CHILI-100K test queries. The paper openly states (Method, 'Training and Initialization' and 'Datasets and ED Simulation') that this one-million-structure precursor predates the final ED-CS CHILI-exclusion filtering, so it is not certified disjoint from the test set. The headline 66.27% MR@5 (Table 2) and the separate claim of 53.52% MR@5 on 1,024 compositions absent from the train/validation retrieval library both come from this registry-initialized model. The only leakage check reported is a post-hoc sensitivity analysis excluding 888 queries with a strict StructureMatcher near-duplicate, and it is reported only for the full split; no analogous overlap analysis is given for the 1,024 absent-formula subset, even though that subset is the basis of the 'true generative capability beyond exact-formula retrieval' claim. If the registry pool contains structural near-duplicates of those 1,024 test structures, the model could be recalling pretraining memoranda rather than generating, and both the 66.27% and 53.52% figures would be inflated. Additional ambiguity: the text says 'with a strict StructureMatcher near-duplicate' without specifying the pool against which matches were found, so the sensitivity check is not precisely reproducible. This concern is load-bearing because the paper's own central contribution is the demonstration of a scaling benefit from the registry-1M precursor and of generative recovery beyond library lookup; both claims lose their force if the pool leaks test structures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ED-CSP, a generative model that predicts lattice parameters and fractional atomic coordinates from a known composition, atom count, and multiple detector-plane electron diffraction (ED) spot lists. The authors also construct ED-CS, a 4.85-million-structure dataset with simulated multi-view ED, deduplicated across seven materials repositories and filtered to exclude CHILI-100K overlaps. On 2,075 held-out CHILI-100K materials, CHILI-only ED-CSP achieves 57.49% MR@5 versus 52.92% for PXRDGen, and warm-starting from a one-million-structure precursor raises MR@5 to 66.27%. On 1,024 compositions absent from the train/validation retrieval library, the registry-initialized model achieves 53.52% MR@5, which the paper presents as evidence of true generative capability beyond exact-formula retrieval. The evaluation also includes input interventions, coverage-stratified retrieval controls, and comparisons to indexed-reflection reconstruction.","tokens_in":12531,"tokens_out":4758,"duration_ms":39129,"significance":"The evaluation design is a clear strength: it uses a fixed external benchmark, the training objective (flow MSE on lattice and coordinates) is separate from the evaluation metric (StructureMatcher MR@5), and the paper reports bootstrap confidence intervals, paired significance tests, input interventions, and coverage-stratified retrieval controls. The public release of code, frozen protocols, provenance certificates, and a registry checkpoint supports reproducibility. If the registry-1M overlap concern is resolved, the contribution is significant: ED-CSP is, to my knowledge, the first generative crystal structure prediction method from unindexed sparse multi-view ED spot lists, and ED-CS is a large, carefully documented simulated dataset that can serve as a benchmark for future work.","major_comments":[{"comment":"The registry-1M warm-start pool is not certified disjoint from the CHILI-100K test queries. The paper states in Datasets and ED Simulation that the completed one-million-structure precursor predates the final ED-CS filtering, and the headline 66.27% MR@5 and the 53.52% absent-formula MR@5 are both produced by the registry-initialized model. The reported post-hoc sensitivity analysis excludes 888 queries with a strict StructureMatcher near-duplicate and is reported only for the full split; it does not cover the 1,024-query absent-formula subset that is the basis of the generative-capability claim, and it does not state against which pool the near-duplicate matches were found. To support the scaling and out-of-library claims, the authors should either certify the registry pool disjoint from the 2,075 test structures, or report a near-duplicate analysis for both the absent-formula subset and the full split against the registry-1M pool, or retrain the registry model on a certified-disjoint pool and report the revised numbers.","section":"Datasets and ED Simulation; Results, Registry Scaling and Signal Use"},{"comment":"The sensitivity analysis is not described precisely enough to be reproducible. The sentence 'excluding 888 queries with a strict StructureMatcher near-duplicate' does not specify the reference pool (registry-1M, ED-CS, or the CHILI train/validation library), the direction of matching, or how many of the 1,024 absent-formula queries are among the 888. Since the reported 38.84%/53.92% figures are the only leakage control for the registry-initialized model, this ambiguity is load-bearing for the paper's main scaling result.","section":"Results, Registry Scaling and Signal Use"}],"minor_comments":[{"comment":"The sentence beginning 'WeretainthereferenceCHILI-100Ksplitforcomparability' appears to have a typo; it should read 'We retain' or 'We retrain' the reference CHILI-100K split, and the intended meaning should be clarified.","section":"Results, Registry Scaling and Signal Use"},{"comment":"In the 'Zero ED spots' row, only MR@1 is reported, while MR@5 is a dash. Clarifying why MR@5 is not reported for this ablation would make the table easier to interpret.","section":"Table 2"},{"comment":"The throughput-oriented online policy that extends a case to at most 100 views uses a 'break-even threshold' that is described only in the supplementary material; a one-sentence summary in the main text would help readers assess the simulation cost and dataset construction choices.","section":"Datasets and ED Simulation"}],"recommendation":"major_revision","confidential_remarks":"The main uncertainty is the registry-1M overlap with the CHILI-100K test set. If the authors can provide a certified exclusion or an overlap analysis for the absent-formula subset, the paper's central claims would be substantially strengthened. The rest of the evaluation is careful and the contribution is timely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper introduces a genuinely new problem — generative CSP from unindexed multi-view ED spot lists plus composition — and supports it with a large simulated dataset and unusually careful evaluation. It deserves a serious referee, but the headline results rest on a pretraining pool that is not certified disjoint from the test set, and that needs to be fixed before I'd trust the absolute numbers.\n\nWhat's new: the task itself. Prior ED machine learning covered label prediction, retrieval, and indexed-reflection refinement; nobody was generating full lattices and fractional coordinates from unindexed spots. ED-CSP does that with a relational set encoder over spot lists, mean–max pooling across views, and a periodic flow generator. The dataset, ED-CS v1, is a 4.85-million-structure curated snapshot with simulated multi-view ED, deduplicated and CHILI-excluded, and the provenance machinery (hashes, manifests, source counts) is genuinely reproducible.\n\nWhat's good: the evaluation is a model of how to run a fair ML benchmark. Paired bootstrap CIs, sign-flip tests, input ablations, a coverage-stratified retrieval control that separates analogue lookup from generation, and a converged composition-only reference. The signal-use evidence is strong: removing ED spots drops MR@1 by 34 points, and swapping in same-formula donor ED drops MR@5 by 22 points on a stricter subset. The comparison to PXRDGen under the same generator is a system-level comparison, and the authors say so. No circularity: the training loss (flow MSE on lattice/coordinates) is different from the StructureMatcher evaluation.\n\nThe soft spot: the registry-1M warm-start pool. The paper itself states it predates the final CHILI-exclusion filtering, so it is not proven disjoint from the 2,075 test queries. Both the 66.27% headline and the 53.52% absent-formula result come from that model. The post-hoc sensitivity analysis removing 888 near-duplicates still shows registry-1M at 53.92% vs 42.46% for CHILI-only, which is heartening, but the paper never specifies which pool those near-duplicates came from, and does not report an analogous check for the absent-formula subset. That's an open question, not a demonstrated failure. If the absent-formula subset is also contaminated, the 'true generative' claim weakens. A clean re-run with a fully filtered pool would close it.\n\nMinor: everything is simulated with known composition; experimental transfer is untested (stated). The composition-only reference is descriptive, not paired (acknowledged). Those are fine.\n\nI'd send it to peer review with a request to resolve the registry leakage explicitly. I'd bring this to the reading group as a cautionary tale in benchmarking. I probably wouldn't cite it directly unless I work in this exact area, which I don't.","headline":"A new task and a strong simulated benchmark for ED-to-structure generation, but the headline registry-pretraining numbers carry an unresolved leakage risk that deserves a clean re-run.","tokens_in":13065,"tokens_out":5537,"would_cite":false,"duration_ms":44427,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ED-CSP predicts a crystal's lattice and fractional atomic coordinates from a known composition and multiple unindexed electron-diffraction spot sets, reaching 57.49% MR@5 on 2,075 held-out CHILI-100K materials with CHILI-only training and…","keywords":["crystal structure prediction","electron diffraction","generative model","periodic flow","multi-view representation learning","lattice parameters","fractional coordinates","structure matching"],"falsifier":"Run the same strict structure matcher used to build ED-CS against the registry-1M precursor and count how many of the 2,075 held-out queries (especially the 1,024 'absent-formula' ones) have a matcher-level near-duplicate in that precursor. If a sizable share do, the 66.27% and 53.52% figures would be explained substantially by near-copy retrieval; if none do, the generative interpretation survives.","tokens_in":12021,"feed_emoji":"🔬","tokens_out":11241,"duration_ms":82190,"temperature":0.7,"pith_summary":"The paper claims that crystal structure prediction can be driven by sparse, unindexed electron-diffraction (ED) spot lists rather than by X-ray powder profiles, indexed reflections, or library lookup. For a known chemical composition and atom count, ED-CSP generates the full periodic structure — lattice parameters and fractional atomic coordinates — by encoding each detector-plane view as a relational set, pooling across views, and decoding with a periodic flow generator. On 2,075 held-out CHILI-100K materials, CHILI-only training reaches 57.49% MR@5 and beats the PXRD-conditioned PXRDGen baseline (52.92%); warm-starting from a one-million-structure precursor raises MR@5 to 66.27%. The model still reaches 53.52% MR@5 on 1,024 queries with no same-formula candidate in the retrieval library, and swapping in diffraction from a non-isomorphic same-formula donor lowers MR@5 by 22.09 percentage points. These numbers matter because electron diffraction is often the only viable signal for nanocrystalline or beam-sensitive materials, so a generative route from raw spot sets to atomic structure would let more real materials be solved automatically.","feed_headline":"Diffraction spots plus composition recover 66% of structures","feed_subtitle":"A generative model recovers lattice and atomic coordinates from sparse electron-diffraction spot lists, beating an X-ray baseline.","key_machinery":"The load-bearing machinery is the pairing of a permutation-invariant sparse-ED encoder with a periodic flow generator. Each detector-plane spot list is processed by a shared relational graph-attention encoder (a PE-AG-GMoE-style backbone) that consumes only two-dimensional spot coordinates and log-intensities; a mean–max pooling head merges the per-view representations so the multi-view input can be unordered. The pooled representation conditions a six-layer CSPNet-style periodic flow decoder that outputs vector fields for the lattice and periodic atomic coordinates, trained with a weighted mean-squared-error loss ($L = L_{\\text{lat}} + 100L_{\\text{coord}}$). Contrastive ED–structure pretraining initializes the encoder, and the registry-1M experiment warm-starts the entire model before CHILI fine-tuning. Because PXRDGen shares the same generator architecture, the comparison isolates the ED conditioning modality.","core_discovery":"ED-CSP establishes that composition-conditioned, generative crystal structure prediction is possible from unindexed multi-view ED spot lists. Given a composition and $K$ detector-plane spot sets $S_{1:K}$, the model models $p_\\theta(L,F \\mid A,S_{1:K})$ and jointly predicts the lattice $L$ and fractional coordinates $F$ without Miller indices, zone axes, or crystallographic labels. The benchmark on 2,075 held-out CHILI-100K materials reports 42.12% MR@1 and 57.49% MR@5 under CHILI-only training, versus 34.02% and 52.92% for PXRDGen; full-model warm-start from the registry-1M precursor gives 51.66% MR@1 and 66.27% MR@5. Signal-use ablations show MR@1 falls from 51.66% to 17.35% when ED spots are removed, and donor-swap ED from a non-isomorphic same-formula structure lowers mean MR@5 by 22.09 percentage points. Coverage stratification separates generation from retrieval: with an exact-formula analogue available, library lookup reaches 85.82% MR@5 versus 78.69% for the registry-initialized model, while for the 1,024 queries with no same-formula candidate, ED-CSP retains 53.52% MR@5 and lookup has no candidate.","pith_inferences":["A direct experimental-transfer test is the natural next step: fine-tuning on even a few hundred measured 3D-ED structures with matched detector conditions would show whether the simulated spot statistics survive contact with real detector calibration, background, and missing reflections.","The view-count trend (Top-5 encoder retrieval rising from 7.81% at 10 views to 13.38% at 20) suggests the 10-view limit used in generator training under-samples the available orientation information, so larger view sets may yield further gains.","The composition-only reference at 50.94% MR@5 implies that a sizable share of recoveries may come from learned chemical priors; a paired, identical-layout comparison against composition-only generation would calibrate how much ED itself contributes on the full split.","The same-formula donor-swap result indicates the model has learned some orientation-dependent geometry; an extension would be to vary simulated thickness and see whether the predictions track dynamical scattering effects or only kinematic spot geometry."],"forward_implications":["Electron diffraction becomes a usable conditioning signal for full structure generation, not just for classifying spot patterns or retrieving library entries.","Scaling the training corpus helps: switching from CHILI-only initialization to a one-million-structure warm start raises MR@1 by about 9.5 points and MR@5 by about 8.8 points.","Generation and exact-formula lookup can be combined: using retrieval when a same-formula analogue exists and ED-CSP otherwise yields 69.88% MR@5, a paired gain of 3.61 points over ED-CSP alone.","Predicted candidates remain improvable by energy-based post-processing: relaxing and re-ranking with a learned interatomic potential raises top-1 recovery by about 13 points, with the relaxed-pool ceiling at 61.69% MR@5 pointing to room for ED-aware refinement."],"supporting_citations":[{"why":"It supplies the PXRDGen baseline and the contrastive-pretraining design; the central comparison measures ED-CSP against this PXRD-conditioned generator on the same CHILI split and decoder.","marker":"(Li et al. 2025)"},{"why":"It supplies the CHILI-100K benchmark and the 2,075 held-out queries used for all reported match rates.","marker":"(Friis-Jensen et al. 2024)"},{"why":"It supplies the PE-AG-GMoE-style relational set encoder that turns each detector-plane ED spot list into a representation.","marker":"(Nathani et al. 2026)"},{"why":"It supplies the py4DSTEM simulation pipeline used to generate the multi-view ED spot patterns for ED-CS.","marker":"(Savitzky et al. 2021)"},{"why":"It supplies the pymatgen StructureMatcher that defines a structural match, hence the MR@1 and MR@5 metric.","marker":"(Ong et al. 2013)"},{"why":"It supplies the CSPNet-style periodic graph decoder used as the flow generator for both ED-CSP and PXRDGen.","marker":"(Jiao et al. 2023)"}],"fun_headline_variants":["ED-CSP turns electron diffraction spots into 66% of crystal structures","Generative model recovers crystals from sparse ED spots, beats X-ray baseline","Without exact-formula retrieval: 53.5% MR@5 from electron diffraction","From spot lists to structures: ED-CSP hits 66% MR@5, beating PXRDGen"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 66.27% registry-1M result assumes that the earlier one-million-structure precursor, assembled before the final CHILI-exclusion filtering, contains no structural near-duplicates of the 2,075 held-out test materials; the authors' protection is a post-hoc sensitivity analysis on a subset rather than a guaranteed exclusion.","fun_headline_variants_meta":{"raw":{"variants":["ED-CSP turns electron diffraction spots into 66% of crystal structures","Generative model recovers crystals from sparse ED spots, beats X-ray baseline","Without exact-formula retrieval: 53.5% MR@5 from electron diffraction","From spot lists to structures: ED-CSP hits 66% MR@5, beating PXRDGen"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001001,"raw_usage":{"total_tokens":4346,"prompt_tokens":1167,"completion_tokens":3179,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":783,"completion_tokens_details":{"reasoning_tokens":3100}},"tokens_in":783,"tokens_out":3179,"duration_ms":19536,"temperature":1.0,"reasoning_tokens":3100,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:33:42.266890+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same strict structure matcher used to build ED-CS against the registry-1M precursor and count how many of the 2,075 held-out queries (especially the 1,024 'absent-formula' ones) have a matcher-level near-duplicate in that precursor. If a sizable share do, the 66.27% and 53.52% figures would be explained substantially by near-copy retrieval; if none do, the generative interpretation survives.","supporting_citations":[{"cited_title":"and Anker, Andy S","cited_arxiv_id":null,"evidence_quote":"It supplies the CHILI-100K benchmark and the 2,075 held-out queries used for all reported match rates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the PE-AG-GMoE-style relational set encoder that turns each detector-plane ED spot list into a representation."},{"cited_title":"and Zeltmann, Steven E","cited_arxiv_id":null,"evidence_quote":"It supplies the py4DSTEM simulation pipeline used to generate the multi-view ED spot patterns for ED-CS."},{"cited_title":"Advances in Neural Information Processing Systems , volume =","cited_arxiv_id":null,"evidence_quote":"It supplies the CSPNet-style periodic graph decoder used as the flow generator for both ED-CSP and PXRDGen."}],"review_version":1}