SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain

Chao Qu; Dakuan Lu; Rui Xu; Tianchu Yao; Wei Chu; Xiaoyu Tan; Yinghui Xu; Yuan Qi

REVIEW 3 major objections 5 minor 2 cited by

SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper presents SCP-116K, a dataset of 116,756 higher-education science problem-solution pairs, extracted automatically from heterogeneous sources by a six-stage pipeline.

desk verdict A genuinely useful extraction pipeline and a new dataset for higher-ed science problems, but the paper's core pair-count claim is contradicted by its own pipeline, and the main distillation result is compromised by test-set similarity selection. read the letter →

arxiv 2501.15587 v2 pith:XPKDJ6D2 submitted 2025-01-26 cs.CL cs.AIcs.IR

Dakuan Lu , Xiaoyu Tan , Rui Xu , Tianchu Yao , Chao Qu , Wei Chu , Yinghui Xu , Yuan Qi This is my paper · ORCID

classification cs.CLcs.AIcs.IR

keywords problem-solutiondatasethighereducationscienceautomatedextractionpipelineGPT-4oknowledgedistillationscientificreasoningGPQALaTeXtranscription

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single automated pipeline can convert heterogeneous textbook sources, such as PDFs, EPUBs, and slide decks from digital libraries, into a large dataset of higher-education physics, chemistry, and biology problem-solution pairs. It introduces SCP-116K, advertised as 116,756 high-quality pairs, and a six-stage extraction workflow built on GPT-4o for rendering, parsing, segmenting, extracting, filtering, and matching. The motivation is that advanced reasoning models like o1 show what high-quality training data can do for mathematics, and science lacks an equivalent public resource. The paper also reports that only 43,000 pairs survive the final verification stage, and it uses the dataset in knowledge-distillation experiments that improve a 32B model's GPQA-diamond accuracy by about 11 percentage points.

What carries the argument

The load-bearing mechanism is the six-stage extraction pipeline: keyword retrieval from 6.69 million documents, uniform image rendering and GPT-4o image-to-markdown transcription with LaTeX, GPT-4o structural segmentation, structured extraction of problems, solutions, and numeric identifiers, two-stage quality filtering, and dual-pathway matching using numeric identifiers and the Stella embedding model with GPT-4o verification of each candidate pair. The dual-pathway matching is what lets the pipeline pair problems with solutions even when they sit in distant chapters or separate solution manuals.

What would settle it

Grade a random sample of 200 of the 43,000 verified pairs with two independent human graders, checking whether the LaTeX transcription matches the source image, whether the solution actually addresses the problem, and whether the final answer is correct; if a meaningful fraction fail, the 'high-quality' claim is unsupported.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that SCP-116K is the first large-scale, authentic higher-education science problem-solution dataset, created without synthetic generation, and that its extraction pipeline generalizes across document formats and domains. The pipeline works by rendering every document to images, using GPT-4o to transcribe the images to markdown with LaTeX formulas, segmenting on structural boundaries, extracting problems and solutions with their numeric identifiers, removing incomplete or self-referential entries, and then matching problems to solutions through a dual pathway: numeric identifier matching plus semantic similarity from the Stella embedding model. Each candidate match is verified by GPT-4o for correctness and completeness. This produces, by the paper's count in Section 3.6, 43,000 verified pairs; the abstract's 116,756 figure describes the number of problems after quality filtering, before final pair matching.

Load-bearing premise

The dataset's 'high-quality' label rests on GPT-4o being reliable at every step, converting textbook images into markdown with correct LaTeX, detecting boundaries, extracting complete problems, and judging whether a solution is correct and complete, yet the paper provides no human-annotated sample to check that reliability.

Editorial extensions

If this is right

Researchers can train or fine-tune models on authentic graduate-level physics, chemistry, and biology problems without synthetic generation, filling a gap left by datasets like CAMEL and ScienceQA.
The six-stage pipeline becomes a reusable recipe for turning other document collections into question-answering or reasoning datasets in specialized domains.
A 32B model fine-tuned on 1,000 SCP-116K examples plus STILL-2 data reaches 58.59% on GPQA-diamond, within about two points of o1-mini's 60.61%, so the dataset's content transfers to graduate-level reasoning benchmarks.
The dataset supplies baselines: o1-mini scores 58.40% and QwQ-32B-preview 55.79% on SCP-116K, leaving room for improvement.
Expanding collection to more fields and larger scale is the stated next step, which the pipeline's format-agnostic design is intended to support.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The title number 116,756 counts filtered problems, not matched pairs; the usable verified resource is the 43,000 pairs, so benchmark and training claims should be read against that smaller number.
Because every extraction and verification decision runs through GPT-4o, the dataset quality is upper-bounded by GPT-4o's transcription and reasoning accuracy; an independent open-weight parser run through the same pipeline would test that ceiling.
The same dual-pathway matching design should transfer to any field whose textbooks number exercises and provide answer keys, such as engineering or quantitative economics.
A human error analysis on even a few hundred pairs would turn 'high-quality' from an assertion into a measured property; the paper currently leaves that measurement undone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper builds a useful-looking pipeline for extracting higher-ed science problem-solution pairs from heterogeneous textbook sources, and the dataset is genuinely new. But the headline number—116,756 pairs—contradicts the paper's own pipeline counts, and the main distillation result is partly explained by selecting training examples similar to the test benchmark. Both are fixable, but they need to be fixed before this can be taken at face value.

On the positive side: the pipeline itself is a sensible integration of off-the-shelf components (document-to-image rendering, GPT-4o for markdown/LaTeX conversion, segmentation, structured extraction, then numerical and semantic matching with Stella embeddings). The authors are unusually transparent about counts at each stage, and they release the dataset and code. The target domain—undergraduate-to-doctoral science problems—is genuinely under-served compared to math datasets.

Now the soft spots. The abstract and conclusion claim "116,756 high-quality problem-solution pairs," but Section 3.5 reports 116,756 problems and 70,000 solutions after filtering, and Section 3.6 reports only 43,000 verified matched pairs. Since a pair requires one problem and one solution, the maximum possible pair count after filtering is 70,000. The 116K figure seems to count problems only. That's a 2.7x overstatement of the verified resource, and it undermines the dataset-scale comparisons in Sections 1 and 2. This is a load-bearing flaw in the paper's central claim.

The second issue is the distillation experiment. The "hybrid" model selects 1,000 SCP examples based on semantic similarity to GPQA-diamond, then evaluates on GPQA-diamond. That's essentially training on the test distribution. The GPQA improvement of about 11 points is likely inflated. The authors should evaluate on a held-out benchmark or on SCP itself.

A third, softer issue: the "high-quality" label rests almost entirely on GPT-4o—it parses, extracts, matches, and verifies. There's no human-annotated sample or error analysis. That leaves the quality claim unverified, though not necessarily wrong. Accuracies in Table 1 are also reported without variance or a clear scoring metric—minor.

Who is this for? People building scientific-reasoning training data and anyone interested in automated dataset construction. It would make a good reading-group discussion on the gap between dataset claims and dataset reality.

Recommendation: send it to peer review, but with a firm request to align the abstract with the pipeline counts, redo the distillation evaluation without test-set similarity selection, and add a human quality audit. If those are addressed, the dataset could be a solid contribution.

Referee Report

3 major / 5 minor

Summary. The paper presents SCP-116K, an automated pipeline for extracting science problem-solution pairs from heterogeneous documents, and claims a dataset of 116,756 high-quality pairs. The pipeline uses GPT-4o for document filtering, image-to-markdown conversion, segmentation, structured extraction, and verification, while matching combines numerical identifiers and semantic similarity. The authors also report baseline accuracy of o1-mini and QwQ-32B-preview on the dataset and distillation experiments with Qwen2.5-32B-Instruct evaluated on GPQA-diamond. The manuscript releases both the dataset and the extraction pipeline.

Significance. If the claims held, SCP-116K would be a valuable resource: it targets a real gap in higher-education science QA data, is larger than existing public resources, and the pipeline could transfer to other domains. The authors openly release the dataset and pipeline, and they provide baseline evaluations and distillation results, which supports reproducibility. However, the scale and quality claims must be reconciled with the pipeline's own counts and with the absence of human verification.

major comments (3)

[Abstract; Section 3.5; Section 3.6] The central scale claim is not supported by the reported pipeline counts. The abstract and Section 1 describe '116,756 high-quality problem-solution pairs,' and Section 2 compares against prior datasets on this basis. Section 3.5, however, reports 116,756 problems and 70,000 solutions after quality filtering, and Section 3.6 reports 43,000 verified matched pairs. Since a pair requires at least one problem and one solution, the maximum possible number of pairs after Section 3.5 is min(116,756, 70,000) = 70,000, and the verified output is 43,000. The paper should either redefine the dataset as '116,756 problems with 43,000 verified paired solutions' or provide a count of released pairs that is consistent with the pipeline. As written, the headline scale overstates the available paired resource by a factor of about 2.7.
[Section 4.2; Section 4.3; Table 1] The distillation experiments select training examples based on semantic similarity to GPQA-diamond questions and then evaluate on GPQA-diamond, which is a form of test-set selection bias. In Section 4.2, the hybrid approach uses '1,000 selected problem-solution pairs from SCP-116K, chosen based on semantic similarity to GPQA-diamond questions,' and Table 1 reports its GPQA-diamond accuracy. This overlap between training selection and evaluation makes the reported improvement (58.59%) an optimistic estimate of general scientific reasoning gains. The authors should evaluate on a held-out benchmark not used for data selection, or at least analyze sensitivity to the selection criterion.
[Section 3.2; Section 3.4; Section 3.6; Appendix A.5] The 'high-quality' property is asserted without a human check. GPT-4o performs the image-to-markdown conversion, extraction, matching verification, and correctness judgment; no human-annotated sample, error analysis, or inter-annotator agreement is reported. The verification prompt in Appendix A.5 asks GPT-4o to judge whether a solution is 'correct' and 'complete' via its own reasoning, which is the same model family that produced the extraction. This does not establish the dataset's quality independent of the model being assessed. The authors should include a human-evaluated random sample (e.g., 100-200 pairs) with per-stage error rates, particularly for formula rendering and solution correctness.

minor comments (5)

[Figure 1] Figure 1 contains a typo: 'Probelm' should be 'Problem.'
[Section 3.7] Section 3.7 says the models generate solutions 'for all problems in our dataset,' but it is unclear whether this refers to all 116,756 problems or the 43,000 verified pairs; please clarify.
[General dataset documentation] The paper does not report the number of source documents per discipline, the distribution of educational levels, or the release license and format; these details are needed for dataset usability.
[Section 4.1] Section 4.1 reports accuracy on SCP-116K (58.40% and 55.79%) but does not describe the evaluation protocol, such as exact-match versus model-based grading or the sample size; please specify.
[Section 2] The related-work comparison in Section 2 would benefit from a quantitative statement of how many of the 116,756 entries are complete pairs after matching, to avoid overclaiming relative to CAMEL's 20,000 pairs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; count inconsistency and same-model quality control are correctness risks, not circularity.

full rationale

The paper contains no mathematical derivation loop. The pipeline stages (Section 3) are sequential transformations: document filtering, image-to-markdown conversion, segmentation, extraction, quality filtering, and matching. Each stage reports independent counts (190K problems/80K solutions; 116,756 problems/70,000 solutions; 43,000 verified pairs), so no later quantity is defined solely in terms of an earlier claim. The abstract's '116,756 problem-solution pairs' conflicts with Section 3.6's '43,000 high-quality verified problem-solution pairs,' but this is an internal-count overstatement rather than a circular step: the pair count is not derived from a fitted assumption or self-citation. The use of GPT-4o both to extract (Sections 3.2-3.4) and to verify matches (Section 3.6) is a same-model quality-control concern, not a logical equivalence, since verification is a separate task with a separate prompt (Appendix A.1.5). The distillation experiment selects 1,000 SCP-116K examples by semantic similarity to GPQA-diamond and then evaluates on GPQA-diamond (Section 4.2); this is disclosed selection bias that can inflate the measured gain, but the evaluation is an external benchmark measurement rather than a prediction forced by construction. No load-bearing self-citations are present, so the paper is self-contained in the circularity sense; its weaknesses belong to data-quality verification and internal consistency, not circular reasoning.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ledger contains no invented physical or mathematical entities. The free parameters are design choices in the pipeline. The main burden is that GPT-4o and Stella are treated as trusted, error-free components without independent validation.

free parameters (3)

candidate solutions per problem (k) = 4
Section 3.6 fixes up to four ranked candidate solutions per problem with no sensitivity analysis.
SCP examples used in hybrid distillation (N) = 1,000
Section 4.2 selects 1,000 examples by hand based on similarity to GPQA-diamond; the reported GPQA improvement depends on this choice.
semantic matching threshold = not reported
Section 3.6 uses Stella similarity scores to rank matches but does not state the cutoff, so the 43,000 count is not exactly reproducible from the paper.

assumptions (4)

domain assumption GPT-4o converts rendered book images to markdown with correct LaTeX for formulas and chemical structures
Invoked in Section 3.2 and Appendix A.2; parsing errors would corrupt every downstream problem and solution.
domain assumption Text-embedding similarity from Stella reliably pairs problems with their solutions
Invoked in Section 3.6; when numerical IDs are absent or inconsistent, matching quality depends entirely on this proxy.
domain assumption GPT-4o can judge correctness and completeness of a solution without external reference
Invoked in Appendix A.5; no human verification is reported, so the high-quality label rests on this assumption.
domain assumption Title keyword search and GPT-4o metadata filtering recover a representative corpus of higher-education science textbooks
Invoked in Section 3.1; if the 467 selected books are skewed, the dataset coverage is skewed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain." pith.science (2026). https://pith.science/paper/XPKDJ6D2

@misc{pith2026250115587,
  author       = {Pith},
  title        = {Pith review of: SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPKDJ6D2}},
  note         = {Machine review of arXiv:2501.15587}
}

read the original abstract

Recent breakthroughs in large language models (LLMs) exemplified by the impressive mathematical and scientific reasoning capabilities of the o1 model have spotlighted the critical importance of high-quality training data in advancing LLM performance across STEM disciplines. While the mathematics community has benefited from a growing body of curated datasets, the scientific domain at the higher education level has long suffered from a scarcity of comparable resources. To address this gap, we present SCP-116K, a new large-scale dataset of 116,756 high-quality problem-solution pairs, automatically extracted from heterogeneous sources using a streamlined and highly generalizable pipeline. Our approach involves stringent filtering to ensure the scientific rigor and educational level of the extracted materials, while maintaining adaptability for future expansions or domain transfers. By openly releasing both the dataset and the extraction pipeline, we seek to foster research on scientific reasoning, enable comprehensive performance evaluations of new LLMs, and lower the barrier to replicating the successes of advanced models like o1 in the broader science community. We believe SCP-116K will serve as a critical resource, catalyzing progress in high-level scientific reasoning tasks and promoting further innovations in LLM development. The dataset and code are publicly available at https://github.com/AQA6666/SCP-116K-open.

Figures

Figures reproduced from arXiv: 2501.15587 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

BlueLM-2.5-3B Technical Report
cs.AI 2025-07 conditional novelty 5.0 of 10

BlueLM-2.5-3B is a small multimodal model with a switchable thinking mode that reportedly matches larger models like Qwen3-4B and comes close to Kimi-VL-A3B-16B on many benchmarks.
SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
cs.LG 2025-06 conditional novelty 5.0 of 10

SwS uses failures during RL training to synthesize targeted math problems, improving reasoning accuracy on eight benchmarks.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages · cited by 2 Pith papers

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov, and Sergei Tilga. 2025. https://arxiv.org/abs/2412.03205 U-math: A university-level benchmark for evaluating mathematical skills in llms . Preprint, arXiv:2412.03205

arXiv 2025
[4]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

arXiv 2025
[5]

Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, S...

arXiv 2024
[6]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

2021
[7]

Kaggle and Google. 2023. Kaggle llm science exam competition. https://www.kaggle.com/competitions/kaggle-llm-science-exam. Online; accessed 2024

work page 2023
[8]

Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. https://arxiv.org/abs/2303.17760 Camel: Communicative agents for "mind" exploration of large scale language model society . Preprint, arXiv:2303.17760

arXiv 2023

Show all 16 references

[9]

Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. 2024. https://doi.org/10.18653/v1/2024.findings-acl.411 M ath B ench: Evaluating the theory and application proficiency of LLM s with a hierar...

2024 doi
[10]

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Sy...

2022
[11]

Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. 2024. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413

2024 arXiv
[12]

NovaSearch. 2024. Stella: A large language model for text embeddings. https://huggingface.co/NovaSearch/stella_en_400M_v5. Online; accessed 2024

2024
[13]

OpenAI. 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. Accessed: 2025-01-23

2024
[14]

OpenAI . 2024. O1: Optimizing the future of ai research. https://openai.com/index/openai-o1-system-card/. Online; accessed 2024

2024
[15]

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022

2023 arXiv
[16]

Qwen Team. 2024. https://qwenlm.github.io/blog/qwq-32b-preview/ Qwq: Reflect deeply on the boundaries of the unknown

2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov, and Sergei Tilga. 2025. https://arxiv.org/abs/2412.03205 U-math: A university-level benchmark for evaluating mathematical skills in llms . Preprint, arXiv:2412.03205

arXiv 2025

[4] [4]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

arXiv 2025

[5] [5]

Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, S...

arXiv 2024

[6] [6]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

2021

[7] [7]

Kaggle and Google. 2023. Kaggle llm science exam competition. https://www.kaggle.com/competitions/kaggle-llm-science-exam. Online; accessed 2024

work page 2023

[8] [8]

Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. https://arxiv.org/abs/2303.17760 Camel: Communicative agents for "mind" exploration of large scale language model society . Preprint, arXiv:2303.17760

arXiv 2023