REVIEW 4 major objections 5 minor 29 references
CrystalICL: Enabling In-Context Learning for Crystal Generation
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read CrystalICL claims that giving LLMs a compact space-group/Wyckoff representation plus few-shot, condition-aware prompts lets them carry over in-context learning to crystal generation, outperforming zero-shot baselines on four benchmarks.
desk verdict A genuinely new few-shot ICL + Wyckoff-tokenization framework for crystal generation, but the central superiority claim is undercut by its own Table 2; worth refereeing after reframing and fuller data. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the space-group-based crystal tokenization (SGS), which uses Wyckoff positions: points whose site-symmetry groups are conjugate, so atoms sharing a Wyckoff position can be collapsed into one representative atom. This cuts the number of coordinates to generate and removes the need for the model to enforce symmetry. The second mechanism is the condition-structure aware hybrid instruction set, which supplies few-shot demonstrations chosen by property matching, CrystalNN fingerprint distance, or both; the third is a multi-task property-prediction instruction set that masks a property in the text and trains the model to fill it in. Together they are meant to make a fine
What would settle it
Run chemical-formula-conditioned generation on MP20 with SGS at several demonstration counts (e.g., 8 or 16) and compare with XYZ-format CrystalLLM/CrystalICL; if SGS cannot reach the 0.9394 pretty-formula success of the XYZ baseline (or its own 3-shot XYZ 0.9906), the compact representation is discarding composition information that examples cannot restore.
Extended reading notes
Core claim
The paper claims that in-context learning is not lost when an LLM is specialized to crystal generation, provided the crystal is written in a form the LLM can reason about. Its space-group-based tokenization (SGS) replaces all atoms in equivalent Wyckoff positions with one representative atom, so the model predicts space-group identity, lattice parameters, and Wyckoff sites rather than every fractional coordinate. A hybrid instruction set mixes zero-shot prompts with few-shot prompts whose examples are chosen by property similarity, structural similarity via CrystalNN fingerprints, or both; a multi-task instruction set also asks the model to predict masked properties from structure text. The
Load-bearing premise
The compact space-group/Wyckoff representation must preserve the compositionally relevant details of a crystal; the paper's own Table 1 shows a clear drop in chemical-formula success under SGS, so if that drop is not recoverable by demonstrations, the method's generality is only partial.
Editorial extensions
If this is right
- If CrystalICL holds, few-shot prompting becomes a viable interface for materials generation: a user supplies a target property plus a handful of similar crystals and gets candidate structures without task-specific retraining.
- Space-group-conditioned generation becomes practical in LLMs; the paper reports a jump in space-group success rate from XYZ text (around 6-11%) to SGS text (above 98% on MP20/MP30), with symmetry adherence above 96%.
- The auxiliary property-prediction task can serve as a general recipe: training an LLM to invert structure-property mappings improves its generative fidelity even when generation is the primary task.
- Unconditional generation over narrow domains, such as the all-carbon C24 set, is made tractable for LLMs: SGS reduces Wasserstein distances for density and formation energy versus the XYZ-format baseline.
- Condition-based demonstration selection matters more than demonstration count; the paper finds shuffling the order of examples hurts, while varying the number of shots does not significantly change success.
Reading between the lines
- The SGS tokenization trades chemical-composition fidelity for symmetry fidelity: on MP20 the 3-shot SGS chemical-formula success is 0.8868 versus 0.9394 for XYZ, so a hybrid representation or a second decoding path for composition might recover both strengths; this is an extension the paper does not test.
- The finding that shot count barely matters while example ordering does suggests the demonstrations may act more as format anchors than as a source of transferable chemical knowledge; testing with intentionally misleading but well-formatted examples would separate the two mechanisms.
- CrystalNN-based structure retrieval is a natural candidate for replacing hand-designed property filters; a learned retriever or embedding-based scorer could make the few-shot selection cheaper and more effective on large databases, but that is outside the paper's experiments.
- Because the model is trained and evaluated on DFT-relaxed structures, few-shot ICL could plausibly extend to other crystal-property mappings, such as synthesizability or ionic conductivity, if structure text for those properties is available in the prompt; the paper leaves this unexplored.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CrystalICL, an LLM-based crystal generation method. It introduces a space-group based tokenization (SGS) that represents crystals by space group, Wyckoff positions, and representative atoms; a condition-structure aware instruction-tuning framework with three example-selection strategies; and a multi-task auxiliary property-prediction objective. The method is evaluated on MP20, MP30, P5, and C24 for conditional and unconditional generation, with CrystalLLM, CDVAE, and DiffCSP as baselines. The paper claims that CrystalICL is the first method to leverage LLM in-context learning for crystal generation and that it outperforms leading baselines on conditional and unconditional tasks.
Significance. The SGS representation is a sensible attempt to reduce the burden of learning crystallographic symmetry in autoregressive text generation, and the retrieval-augmented prompt construction is a plausible way to inject few-shot structure–property priors. The ablation study and the preprocessing-cost appendix are useful. However, the central broad claim of superiority over leading baselines is not supported by the reported numbers: unconditional property-distribution metrics favor DiffCSP on several datasets, and the conditional comparison is limited to CrystalLLM. The contribution can still be valuable if the claims are re-scoped and the evaluation protocol is made fairer, but the current framing overstates the evidence.
major comments (4)
- [Abstract; §4.4 (Table 2)] The abstract claims 'superiority of CrystalICL over the leading baseline methods on conditional and unconditional generation tasks.' Table 2 does not support this for unconditional generation. On MP20, DiffCSP has wdist(rho)=0.1907 and wdist(E)=0.1394, while CrystalICL(SGS) has 0.6039 and 0.2568; on C24, DiffCSP has wdist(E)=0.0415 versus CrystalICL(SGS)=1.3061, and CDVAE also beats CrystalICL on wdist(E). CrystalICL is consistently better than CrystalLLM, but not than the diffusion/VAE baselines. The abstract and conclusion should be restricted to the comparisons actually supported.
- [§4.1–4.2 (Table 1)] For conditional generation the only baseline is CrystalLLM (§4.1), so 'leading baseline methods' is unsupported. Moreover, the reported advantage is format-dependent: on MP20 Pretty Formula, CrystalICL(SGS) 3-Shot reaches 0.8868 versus CrystalLLM(XYZ) 0.9394; on MP30 the corresponding numbers are 0.9641 versus 0.9699. The text in §4.2 calls this 'slightly reduced,' but the drop exceeds 5 percentage points on MP20. Since SGS is a main contribution, the paper should explicitly frame the tradeoff between symmetry-conditioning gains and composition-generation losses, and compare within the same representation.
- [§4.4, unconditional generation protocol] The paper states that when sampling for unconditional generation, 'If a sampled string cannot be parsed as a valid CIF, the sample is rejected and re-sampled.' This makes the reported validity and coverage numbers conditional on a retry loop that diffusion/VAE baselines do not have. As a result, the validity comparisons in Table 2 are not apples-to-apples. The authors should report the raw parse/validity rate before re-sampling and either apply the same rejection criterion to all baselines or drop the validity comparison.
- [§4.5 (Table 3)] The conclusion that 'hybrid instruction tuning effectively enhances the capabilities of CrystalICL across various scenarios' is not consistently supported by Table 3. In the 3-Shot setting on MP20, the full CrystalICL achieves Pretty Formula 0.9214, while C-only achieves 0.9340 and noAux achieves 0.9347; for Space Group the full model scores 0.9948 versus 0.9954 (C) and 0.9959 (noAux). The full hybrid set is thus not clearly superior in few-shot inference. This needs a more careful analysis or a softened conclusion.
minor comments (5)
- [Figure 1] The bar chart is difficult to interpret: the percentages above the bars are not clearly tied to the four metric categories, and the 0%/100% labels are unexplained. Also, GPT-3.5 Turbo is used as motivation but is not included as a baseline in the main experiments.
- [§4.5] The sentence 'demonstrating that randomly example selection strategy leads to failing to derive task-relevant information from the demonstrations, thus losing ICL capability' is grammatically unclear and does not follow from the preceding comparison of F, CF, and C. Please rewrite.
- [Table 2 / Appendix C] The table header mixes 'Validity Check' with separate 'Composition' and 'Structural' columns. Please clarify which numbers correspond to the definitions in Appendix C and whether 'Validity Check' is a combined score.
- [Appendix H] The physical-realism metrics (atomic overlap, symmetry adherence, energy feasibility) are reported only in the appendix and are not referenced in the main evaluation. This evidence is relevant to the core claims and should be summarized in the main text.
- [General] No code or data splits are provided. Given the many hyperparameters in Appendix E (LoRA rank, alpha, dropout, learning rate, temperature, top-p, number of shots), a public implementation would substantially improve reproducibility.
Circularity Check
Partial circularity: SGS space-group success rate is guaranteed by the output format; remaining evaluation is independent.
-
self definitional
[Sec. 3.1 (Space-group based Crystal Tokenization), Sec. 4.2 (Conditional Generation Evaluation), Table 1, Appendix C]
"given the space group, atoms of the same element occupying the same Wyckoff position can be represented by a single atom. This decomposes the task of predicting the fractional 3D coordinates of all atoms in the unit cell into two components: modeling the correspondence between the space group and Wyckoff positions, and predicting the Wyckoff positions of atoms within the unit cell."
The SGS output format begins with the space-group symbol and lists atoms by Wyckoff position. Expanding those Wyckoff positions for the declared space group produces a structure whose pymatgen-determined space group matches the declared label by construction. Since the conditional input is the target space group and Appendix C scores success via SpacegroupAnalyzer, the near-perfect SGS space-group success rates (e.g., MP20 3-Shot 0.9908 vs 0.0886 for XYZ in Table 1) measure the model's ability to copy the condition into the label field, not an independent ability to realize the symmetry from unscaffolded coordinates. The paper's conclusion that SGS significantly enhances space-group-conditioned generation is therefore built into the tokenization definition, not demonstrated independently.
full rationale
The paper is an empirical systems contribution with no fitted-parameter prediction chain. Few-shot demonstrations are drawn from the training set and test conditions from the test set, which is standard supervised evaluation; property-conditioned success is scored with an external MEGNet estimator, so no self-evaluation loop occurs. There are no load-bearing self-citations or imported uniqueness theorems. The one localized circular step is the SGS space-group success metric: because the representation explicitly includes the space-group label and Wyckoff positions, a parseable SGS output already determines the target symmetry, making the SGS-vs-XYZ space-group comparison largely a format artifact. However, the central claims also rest on independent metrics (chemical formula success, MEGNet-estimated formation energy/band gap conditional accuracy, and unconditional coverage/property-distribution distances), and those comparisons are not forced by construction. The abstract's overstatement relative to Table 2 and the limited conditional baseline set are correctness/scope concerns, not circularity.
Assumptions & free parameters
free parameters (7)
- LoRA rank =
8
- LoRA alpha =
32
- LoRA dropout =
0.05
- learning rate =
5e-4
- temperature =
0.9
- top_p =
0.9
- number of shots =
3
assumptions (4)
- domain assumption Llama-2-7b-chat provides a strong enough base for instruction tuning to learn structure-property mappings.
- domain assumption The space-group based representation with Wyckoff positions is a faithful and sufficient descriptor of crystal structures.
- domain assumption MEGNet models trained on Materials Project provide reliable estimates for formation energy and band gap of generated crystals.
- domain assumption The success-rate thresholds (sign-based for formation energy, plus or minus 0.5 eV for band gap) are valid indicators of generation quality.
Cite this review
Pith. "Pith review of CrystalICL: Enabling In-Context Learning for Crystal Generation." pith.science (2026). https://pith.science/paper/7CRYI2AO
@misc{pith2026250820143,
author = {Pith},
title = {Pith review of: CrystalICL: Enabling In-Context Learning for Crystal Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/7CRYI2AO}},
note = {Machine review of arXiv:2508.20143}
}
read the original abstract
Designing crystal materials with desired physicochemical properties remains a fundamental challenge in materials science. While large language models (LLMs) have demonstrated strong in-context learning (ICL) capabilities, existing LLM-based crystal generation approaches are limited to zero-shot scenarios and are unable to benefit from few-shot scenarios. In contrast, human experts typically design new materials by modifying relevant known structures which aligns closely with the few-shot ICL paradigm. Motivated by this, we propose CrystalICL, a novel model designed for few-shot crystal generation. Specifically, we introduce a space-group based crystal tokenization method, which effectively reduces the complexity of modeling crystal symmetry in LLMs. We further introduce a condition-structure aware hybrid instruction tuning framework and a multi-task instruction tuning strategy, enabling the model to better exploit ICL by capturing structure-property relationships from limited data. Extensive experiments on four crystal generation benchmarks demonstrate the superiority of CrystalICL over the leading baseline methods on conditional and unconditional generation tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Luis M Antunes, Keith T Butler, and Ricardo Grau-Crespo. 2024. Crystal structure generation with autoregressive large language modeling. Nature Communications, 15(1):1--16
work page 2024
-
[4]
Kohei Shinohara Atsushi Togo and Isao Tanaka. 2024. https://doi.org/10.1080/27660400.2024.2384822 Spglib: a software library for crystal symmetry search . Sci. Technol. Adv. Mater., Meth., 4(1):2384822--2384836
arXiv 2024
-
[5]
Keith T Butler, Daniel W Davies, Hugh Cartwright, Olexandr Isayev, and Aron Walsh. 2018. Machine learning for molecular and materials science. Nature, 559(7715):547--555
work page 2018
-
[6]
Ivano E Castelli, David D Landis, Kristian S Thygesen, S ren Dahl, Ib Chorkendorff, Thomas F Jaramillo, and Karsten W Jacobsen. 2012. New cubic perovskites for one-and two-photon water splitting using the computational materials repository. Energy & Environmental Science, 5(10):9034--9043
work page 2012
-
[7]
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. 2019. Graph networks as a universal machine learning framework for molecules and crystals. Chemistry of Materials, 31(9):3564--3572
work page 2019
-
[8]
Kishalay Das, Subhojyoti Khastagir, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. 2025. https://openreview.net/forum?id=AkBrb7yQ0G Periodic materials generation using text guided joint diffusion model . In The Thirteenth International Conference on Learning Representations, ICLR . OpenReview.net
work page 2025
Show all 29 references
-
[9]
Gautam R Desiraju. 2002. Cryptic crystallography. Nature materials, 1(2):77--79
2002
- [10]
-
[11]
Lawrence Zitnick, and Zachary W
Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C. Lawrence Zitnick, and Zachary W. Ulissi. 2024. Fine-tuned language models generate stable inorganic materials as text. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, ...
2024
-
[12]
Sydney R Hall, Frank H Allen, and I David Brown. 1991. The crystallographic information file (cif): a new standard archive file for crystallography. Foundations of Crystallography, 47(6):655--685
1991
- [13]
-
[14]
Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representat...
2022
-
[15]
Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. 2013. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL ...
2013
-
[16]
Rui Jiao, Wenbing Huang, Peijia Lin, Jiaqi Han, Pin Chen, Yutong Lu, and Yang Liu. 2023. Crystal structure prediction by joint equivariant diffusion. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurI...
2023
-
[17]
Rui Jiao, Wenbing Huang, Yu Liu, Deli Zhao, and Yang Liu. 2024. Space group constrained crystal generation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net
2024
-
[18]
Walter Kohn and Lu Jeu Sham. 1965. Self-consistent equations including exchange and correlation effects. Physical review, 140(4A):A1133
1965
-
[19]
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. 2023. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123--1130
2023
-
[20]
H. LIPSON. 1949. https://doi.org/10.1038/163622a0 Crystal structures . Nature, 163(4147):622--622
1949 doi
-
[21]
Chevrier, Kristin A
Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L. Chevrier, Kristin A. Persson, and Gerbrand Ceder. 2013. https://doi.org/10.1016/j.commatsci.2012.10.028 Python materials genomics (pymatgen): A rob...
2013 doi
-
[22]
Chris J Pickard. 2020. Airss data for carbon at 10gpa and the c+ n+ h+ o system at 1gpa. (No Title)
2020
- [23]
- [24]
-
[25]
Logan Ward, Ankit Agrawal, Alok Choudhary, and Christopher Wolverton. 2016. A general-purpose machine learning framework for predicting properties of inorganic materials. npj Computational Materials, 2(1):1--7
2016
-
[26]
Jaakkola
Tian Xie, Xiang Fu, Octavian - Eugen Ganea, Regina Barzilay, and Tommi S. Jaakkola. 2022. Crystal diffusion variational autoencoder for periodic material generation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 ...
2022
-
[27]
Weitong Zhang, Xiaoyun Wang, Weili Nie, Joe Eaton, Brad Rees, and Quanquan Gu. 2023. Moleculegpt: Instruction following large language models for molecular property prediction. In NeurIPS 2023 Workshop on New Frontiers of AI for Drug Discovery and Development
2023
-
[28]
May, Geoffrey I
Yizhen Zheng, Huan Yee Koh, Maddie Yang, Li Li, Lauren T. May, Geoffrey I. Webb, Shirui Pan, and George Church. 2024. https://doi.org/10.48550/ARXIV.2409.04481 Large language models in drug discovery and development: From disease mechanisms to clinical trials . CoRR, abs/2409.04481
-
[29]
Nils ER Zimmermann and Anubhav Jain. 2020. Local structure order parameters and site fingerprints for quantification of coordination environment and crystal structure similarity. RSC advances, 10(10):6063--6081
2020
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.