REVIEW 3 major objections 5 minor 30 references
How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performance
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Model-size scaling laws generally extend to knowledge graph engineering tasks, but plateau effects and occasional within-family drops mean the largest model is not always the best choice.
desk verdict Useful empirical study on model size vs. RDF/SPARQL performance, but the significance tests overstate the case and the size effect is partly confounded with family/generation; still worth a referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the LLM-KG-Bench automated evaluation framework, run over 26 open instruction-tuned LLMs with 50 repetitions per task variation, because it produces the repeated, automatically checked RDF and SPARQL scores that make statistical comparison possible. Scores are grouped into central measures (format-sensitive F1 scores such as listTrimF1, f1, strSimilarity, and trimF1), fragment measures, and syntax-parsability measures, and the analysis focuses on central measures as the strictest indicator of correctness. On top of these scores, the statistical machinery is comparison of four size categories (tiny [0–3B], small (3B–8B], medium (8B–33B], large (33B–72B]) using Kruskal–Wallis tests followed by Dunn post-hoc tests with Bonferroni correction, complemented by scatter plots that connect models of the same family.
What would settle it
Rerun the same LLM-KG-Bench evaluation on the excluded Text2Sparql variations and on all eight unaggregated RdfFriendCount variations; if those variants show flat or declining score-versus-size relationships while the 23 analysed variations rise, the generalisation is narrower than claimed. A complementary numeric check is to compute the rank correlation between parameter count and central-measure score per task across the 26 models and count violations of monotonicity; finding many tasks with near-zero or negative correlations would directly contradict the claim that scaling laws generally apply.
Extended reading notes
Core claim
The central claim is that scaling laws by model size, previously established for general language modeling, also apply to knowledge graph engineering tasks, with qualifications. Analysing central (format-sensitive, F1-based) measures for 23 task variations from seven task classes, the paper reports that Kruskal–Wallis tests reject equal score distributions across the tiny, small, medium, and large size categories for every task variation, and that most significant pairwise differences go in the direction of larger models scoring higher. The exceptions are RdfFriendCount variations where scores decrease from smaller to larger groups on some input formats, and numerous plateau or ceiling cases where adjacent size groups are statistically indistinguishable. Within model families, the largest member is usually the best performer, but the paper documents local drops where a larger sibling performs worse than a smaller one, with recovery usually at the next size. The paper concludes that model size generally predicts KGE performance but should be supplemented by testing neighbouring sizes in the same family.
Load-bearing premise
The analysis assumes that the 23 selected task variations, after dropping three Text2Sparql variations and aggregating eight RdfFriendCount variations 'for consistency reasons', represent knowledge graph engineering tasks generally; if the excluded or aggregated variations behave differently, the conclusion that scaling laws generally apply to KGE may not transfer.
Editorial extensions
If this is right
- Practitioners can use model size as an initial screening criterion when choosing an open LLM for RDF or SPARQL engineering, because larger models usually score higher on the tested task classes.
- For tasks that show ceiling effects, medium-sized models around 8–14B parameters reach scores comparable to far larger models, so choosing them can reduce memory and inference costs without much accuracy loss.
- Within a model family, a larger sibling is not guaranteed to outperform a smaller one, so the paper's practical recommendation is to test the next smallest and next largest model in the family.
- Difficult tasks such as Text2Sparql plateau at low scores even for the largest open models up to 72B, and the included proprietary models also plateau on that task, so model size alone will not unlock those capabilities.
- Mixture-of-Experts models in the benchmark did not systematically beat dense models with comparable active-parameter counts, indicating that cost-effectiveness should be judged on active parameters rather than total size.
Reading between the lines
- Beyond this benchmark, the plateau around 8–14B parameters on syntax-heavy tasks suggests that current RDF and SPARQL task variations are becoming saturated for medium models; new, harder variations would be needed to reveal whether the scaling law continues at larger sizes.
- If the local within-family drops are not noise, they imply that parameter count is a noisy proxy even within one family, so explaining them would require training data, training steps, or architectural factors that the paper explicitly leaves to future work.
- The uniform failure on Text2Sparql and RdfFriendCount suggests a distinct capability gap—translating natural language into precise query or graph semantics—that may respond more to targeted fine-tuning or reasoning training than to raw parameter scaling.
- One caveat the paper does not itself stress is that the RdfFriendCount task variations were aggregated before analysis; inspecting the eight raw variations separately would show whether the decreasing or plateau patterns are uniform across serialization formats.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes a dataset of 26 open LLMs evaluated with the LLM-KG-Bench framework on 23 RDF- and SPARQL-related task variations. The authors partition models into four size categories, run Kruskal-Wallis and Dunn tests on central measures, and visually inspect score-versus-size trends within model families. They report that larger models generally achieve higher scores, with plateau and ceiling effects on some tasks and occasional local performance drops within families, and conclude that model-size scaling laws broadly apply to the selected KGE tasks. They also discuss cost-effectiveness implications, especially for MoE and medium-sized models.
Significance. If the central claim holds, the paper provides useful, practically relevant guidance for choosing LLMs in knowledge graph engineering tasks, a domain where systematic LLM evaluation is still scarce. The main strengths are the automated, repeatable benchmark infrastructure, the public availability of the dataset and analysis code, and the explicit focus on semantic and syntactic correctness rather than multiple-choice accuracy. However, the significance is conditional: the statistical evidence for size effects is weakened by pseudoreplication in the significance tests, and the cross-family comparisons are observationally confounded with family, generation, and architecture. The paper's contribution is better described as a descriptive observational study of one benchmark run than as an established scaling-law result for KGE.
major comments (3)
- [Section 4.2, Table 3] The Kruskal-Wallis and Dunn tests treat the 50 repetitions per model as independent observations. These repetitions are repeated draws from the same model, not independent samples from the size group, so the tests inflate the effective sample size by a factor of about 50 and produce artificially extreme p-values (e.g., p≈7e-122 for Text2Sparql turtle subschema). The analysis should be performed at the model level (e.g., model-mean scores or a mixed model with model as a random effect), and the conclusions should be based on effect sizes rather than on these pseudoreplicated p-values.
- [Section 4.2, Table 1] The cross-family size-group comparisons are confounded with model family, release generation, and architecture. The tiny category consists only of Qwen2/Qwen2.5 and Llama-3.2 models, while the large category includes Llama-3.3, Qwen2.5-72B, and two MoE models binned by total parameter count (57B and 42B) whose active parameter counts (14B and 6.6B) would place them in the medium/small range. Therefore, significant group differences in Table 3 cannot be attributed to parameter count alone. The authors should either control for family/generation in the analysis or explicitly reframe the results as descriptive associations between size category and score in this particular convenience sample.
- [Section 4.3, Figure 1, Section 6] The conclusion that 'scaling laws generally also apply to the selected KGE tasks' is stronger than the evidence supports. The within-family analysis is visual only, and the paper itself reports local drops where larger models underperform smaller family members. Without a quantitative family-controlled analysis or a fitted scaling-law model (even a simple per-task, per-family slope), the apparent general trend could be driven by family identity and generation rather than by parameter count. The authors should either provide such an analysis or soften the central claim to say that, in this benchmark run, larger open models tended to score higher on many task variations, with notable exceptions.
minor comments (5)
- [Figure 1 caption] The caption contains the typo 'aösp' (likely 'also') and should be corrected.
- [Section 3] The text says the dataset covers 'over 30 open and proprietary LLMs' and later that 26 open LLMs are analyzed; please clarify the exact total count and the number excluded.
- [Section 3] The exclusion of three Text2Sparql variations and the aggregation of eight RdfFriendCount variations 'for consistency reasons' deserves a more explicit justification, since the representativeness of the 23 analyzed variations depends on this choice.
- [Table 3] Using blank cells to indicate significant differences is difficult to read; reporting explicit p-values or significance symbols (e.g., '***') would improve clarity.
- [Throughout] There are minor notation inconsistencies such as 'LLM-KG Bench' versus 'LLM-KG-Bench' and 'Mixture-of-Expert' versus 'Mixture-of-Experts'; please standardize.
Circularity Check
No circularity: benchmark scores are external measurements; the scaling-law conclusion is an empirical observation, not an equation fitted to its own output.
full rationale
The paper's central claim is that model-size scaling laws generally hold for the selected KGE tasks, based on external benchmark scores from LLM-KG-Bench runs on 26 open LLMs. The scores are measured outcomes, not derived from model size; size categories are defined by parameter counts; and no parameter is fitted to the scores and then renamed as a prediction. The Kruskal-Wallis and Dunn tests compare score distributions across independently defined size bins, so a significant group difference is not an artifact of construction. The paper does rely on the authors' own prior LLM-KG-Bench framework and dataset [9,10], but that prior work is a reproducible measurement infrastructure with open code and results, and the current paper does not invoke any uniqueness theorem or ansatz from those citations to force its conclusions. The statistical concerns raised by a skeptic—family/generation/MoE confounds and absence of a fitted scaling curve—are threats to validity and generalizability, not circularity. No equation or claim reduces by definition to its own inputs.
Assumptions & free parameters
free parameters (1)
- Size category boundaries =
[0-3], (3-8], (8-33], (33-72] billion parameters
assumptions (5)
- domain assumption LLM-KG-Bench v3 scores are valid automated measures of KGE capabilities.
- domain assumption The 23 analyzed task variations, after exclusions and aggregation, represent the broader space of KGE tasks.
- domain assumption Reported model parameter counts are accurate and comparable across families.
- domain assumption Repeated benchmark runs from the same model are independent observations for the Kruskal-Wallis tests.
- domain assumption Differences in training data, compute, and fine-tuning between families do not dominate the size effect.
Cite this review
Pith. "Pith review of How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performance." pith.science (2026). https://pith.science/paper/FWWZJ6PM
@misc{pith2026250516276,
author = {Pith},
title = {Pith review of: How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performance},
year = {2026},
howpublished = {\url{https://pith.science/paper/FWWZJ6PM}},
note = {Machine review of arXiv:2505.16276}
}
read the original abstract
When using Large Language Models (LLMs) to support Knowledge Graph Engineering (KGE), one of the first indications when searching for an appropriate model is its size. According to the scaling laws, larger models typically show higher capabilities. However, in practice, resource costs are also an important factor and thus it makes sense to consider the ratio between model performance and costs. The LLM-KG-Bench framework enables the comparison of LLMs in the context of KGE tasks and assesses their capabilities of understanding and producing KGs and KG queries. Based on a dataset created in an LLM-KG-Bench run covering 26 open state-of-the-art LLMs, we explore the model size scaling laws specific to KGE tasks. In our analyses, we assess how benchmark scores evolve between different model size categories. Additionally, we inspect how the general score development of single models and families of models correlates to their size. Our analyses revealed that, with a few exceptions, the model size scaling laws generally also apply to the selected KGE tasks. However, in some cases, plateau or ceiling effects occurred, i.e., the task performance did not change much between a model and the next larger model. In these cases, smaller models could be considered to achieve high cost-effectiveness. Regarding models of the same family, sometimes larger models performed worse than smaller models of the same family. These effects occurred only locally. Hence it is advisable to additionally test the next smallest and largest model of the same family.
Figures
Reference graph
Works this paper leans on
-
[1]
A. Hogan, E. Blomqvist, M. Cochez, C. d’Amato, G. de Melo, C. Gutierrez, J. E. L. Gayo, S. Kirrane, S. Neumaier, A. Polleres, R. Navigli, A.-C. N. Ngomo, S. M. Rashid, A. Rula, L. Schmelzeisen, J. Sequeda, S. Staab, A. Zimmermann, Knowledge graphs, ACM Computing Surveys (CSUR) 54 (2020) 1 – 37. doi:10.1145/3447772
doi:10.1145/3447772 2020
-
[2]
S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, X. Wu, Unifying large language models and knowledge graphs: A roadmap, IEEE Transactions on Knowledge and Data Engineering (TKDE) (2024). doi:10.1109/TKDE.2024.3352100
arXiv 2024
-
[3]
B. P. Allen, L. Stork, P. Groth, Knowledge engineering using large language models (2023). doi:10.4230/TGDK.1.1.3
-
[4]
R. Buchmann, J. Eder, H.-G. Fill, U. Frank, D. Karagiannis, E. Laurenzi, J. Mylopoulos, D. Plexousakis, M. Y. Santos, Large language models: Expectations for semantics-driven systems engineering, Data and Knowledge Engineering 152 (2024) 102324. doi:10.1016/j.datak.2024.102324
arXiv 2024
-
[5]
T. A. Taffa, R. Usbeck, Leveraging LLMs in scholarly knowledge graph question answering (2023). doi:10.48550/ARXIV.2311.09841
-
[6]
M. Hofer, J. Frey, E. Rahm, Towards self-configuring knowledge graph construction pipelines using llms - a case study with rml, in: Fifth International Workshop on Knowledge Graph Construction @ ESWC2024, volume 3718 of CEUR Workshop Proceedings, CEUR-WS.org, 2024. URL: https://ceur-ws.org/Vol-3718/paper6.pdf
work page 2024
-
[7]
L. Kovriguina, R. Teucher, D. Radyush, D. Mouromtsev, Sparqlgen: One-shot prompt-based approach for sparql query generation, in: International Conference on Semantic Systems, volume 3526 of CEUR Workshop Proceedings, CEUR-WS.org, 2023. URL: https://ceur-ws.org/Vol-3526/ paper-08.pdf
work page 2023
-
[8]
H. Babaei Giglou, J. D’Souza, S. Auer, LLMs4OL: Large Language Models for Ontology Learning, Springer Nature Switzerland, 2023, pp. 408–427. doi:10.1007/978-3-031-47240-4_22
Show all 30 references
-
[9]
Meyer, J
L.-P. Meyer, J. Frey, K. Junghanns, F. Brei, K. Bulert, S. Gründer-Fahrer, M. Martin, Developing a scalable benchmark for assessing large language models in knowledge graph engineering, in: N. Keshan, S. Neumaier, A. L. Gentile, S. Vahdati (Eds.), Proceedings of the Posters an...
2023
-
[10]
Meyer, J
L.-P. Meyer, J. Frey, D. Heim, F. Brei, C. Stadler, K. Junghanns, M. Martin, LLM-KG-Bench 3.0: A compass for semantic technology capabilities in the ocean of LLMs, in: Proceedings of ESWC 2025 Resources Track, 2025. Accepted for publication
2025
-
[11]
J. Frey, L. Meyer, N. Arndt, F. Brei, K. Bulert, Benchmarking the abilities of large language models for RDF knowledge graph creation and comprehension: How well do llms speak turtle?, in: M. Alam, M. Cochez (Eds.), Proceedings of the Workshop on Deep Learning for Knowledge Gr...
2023
-
[12]
Frey, L.-P
J. Frey, L.-P. Meyer, F. Brei, S. Gruender, M. Martin, Assessing the evolution of llm capabilities for knowledge graph engineering in 2023, in: Proceedings of Special Track Large Language Models for Knowledge Engineering at Extended Semantic Web Conference 2024 (ESWC24), 2024....
2023 doi
-
[13]
Meyer, J
L.-P. Meyer, J. Frey, F. Brei, N. Arndt, Assessing sparql capabilities of large language models, in: E. Vakaj, S. Iranmanesh, R. Stamartina, N. Mihindukulasooriya, S. Tiwari, F. Ortiz-Rodríguez, R. Mcgranaghan (Eds.), Proceedings of the 3rd International Workshop on Natural La...
2024
-
[14]
Kaplan, S
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models (2020). arXiv:2001.08361
2020 arXiv
-
[15]
Chiang, L
W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, I. Stoica, Chatbot arena: An open platform for evaluating llms by human preference,
-
[16]
Fourrier, N
C. Fourrier, N. Habib, A. Lozovskaya, K. Szafer, T. Wolf, Open llm leaderboard v2, https: //huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard, 2024
2024
-
[17]
Liang, R
P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta- Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. W...
2023 arXiv
-
[18]
J. Liu, C. S. Xia, Y. Wang, L. Zhang, Is your code generated by chatgpt really correct? rigorous eval- uation of large language models for code generation, Advances in Neural Information Processing Systems 36 (2024)
2024
-
[19]
Meyer, C
L.-P. Meyer, C. Stadler, J. Frey, N. Radtke, K. Junghanns, R. Meissner, G. Dziwis, K. Bulert, M. Martin, Llm-assisted knowledge graph engineering: Experiments with chatgpt, in: C. Zinke-Wehlmann, J. Friedrich (Eds.), First Working Conference on Artificial Intelligence Developm...
2023
-
[20]
Mihindukulasooriya, S
N. Mihindukulasooriya, S. Tiwari, C. F. Enguix, K. Lata, Text2KGBench: A benchmark for ontology- driven knowledge graph generation from text, in: T. R. Payne, V. Presutti, G. Qi, M. Poveda-Villalón, G. Stoilos, L. Hollink, Z. Kaoudi, G. Cheng, J. Li (Eds.), The Semantic Web – ...
2023
-
[21]
Y. Zhu, X. Wang, J. Chen, S. Qiao, Y. Ou, Y. Yao, S. Deng, H. Chen, N. Zhang, Llms for knowl- edge graph construction and reasoning: Recent capabilities and future opportunities, 2023. arXiv:2305.13168
2023 arXiv
-
[22]
Usbeck, M
R. Usbeck, M. Röder, M. Hoffmann, F. Conrads, J. Huthmann, A.-C. Ngonga-Ngomo, C. Demmler, C. Unger, Benchmarking question answering systems, Semantic Web 10 (2019) 293–304. doi:10. 3233/sw-180312
2019
-
[23]
H. M. Zahera, M. Ali, M. A. Sherif, D. Moussallem, A. N. Ngomo, Generating SPARQL from natural language using chain-of-thoughts prompting, in: A. A. Salatino, M. Alam, F. Ongenae, S. Vahdati, A. L. Gentile, T. Pellegrini, S. Jiang (Eds.), Knowledge Graphs in the Age of Languag...
2024
-
[24]
Shorten, C
C. Shorten, C. Pierse, T. B. Smith, E. Cardenas, A. Sharma, J. Trengrove, B. van Luijt, Structuredrag: Json response formatting with large language models, 2024. arXiv:2408.11061
2024 arXiv
-
[25]
Srivastava, et al., Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, Transactions on Machine Learning Research (2023)
A. Srivastava, et al., Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, Transactions on Machine Learning Research (2023). arXiv:2206.04615
2023 arXiv
-
[26]
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, Q. V. Le, Emergent abilities of large language models, Transactions on Machine Learning Research (2022). arXiv:2206.07682
2022 arXiv
-
[27]
Hoffmann, S
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, L. Sifre, T...
2022 arXiv
-
[28]
W. H. Kruskal, W. A. Wallis, Use of ranks in one-criterion variance analysis, Journal of the American Statistical Association 47 (1952) 583–621
1952
-
[29]
O. J. Dunn, Multiple comparisons using rank sums, Technometrics 6 (1964) 241–252
1964
-
[30]
C. E. Bonferroni, Il calcolo delle assicurazioni su gruppi di teste, 1935, pp. 13–60. A. Online Resources The LLM-KG-Bench framework is available here: LLM-KG-Bench. The raw benchmark run results and further figures are available here: Results LLM-KG-Bench v3. The code written...
1935
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.