{"id":"85c47d67-6e64-45c5-941c-97e1b1c4d598","arxiv_id":"2501.15587","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"SCP-116K is a proposed 116,756-item science problem-solution dataset, but the paper's own matching stage reports only 43,000 verified problem-solution pairs.","lead":"SCP-116K introduces a large dataset of college-level science problems and solutions, extracted from textbooks with a GPT-4o based pipeline. The paper also shows that training a smaller model on a slice of this data can improve scores on the GPQA science benchmark, though the reported scale is inconsistent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 116,756 problem-solution pairs contradict Section 3.6's 43,000 verified pairs; the central dataset-size claim is unsupported by the paper's own pipeline counts.","rationale":"The reader's weakest_assumption focuses on GPT-4o reliability, but the most load-bearing issue is the numerical inconsistency in the paper's own pipeline description. The abstract and conclusion assert 116,756 pairs; Section 3.6 reports 43,000 verified pairs. The discrepancy is not resolved anywhere in the manuscript, and no appendix explains a different counting convention. This matters because the paper's novelty claim is precisely the scale and authenticity of the paired resource. A dataset of 43,000 pairs may still be useful, but it is not the contribution advertised. The proposed check is cheap: count the released data. If the released file contains 116,756 complete pairs but Section 3.6's 43,000 refers to something else, the paper needs to clarify that number. If it contains 43,000, the abstract and Section 1 must be corrected. Because the same revision path should also address the missing human quality evaluation and the GPQA-selection confound, the CONDITIONAL verdict remains appropriate; no change to the reader's verdict is needed.","tokens_in":9023,"tokens_out":3406,"duration_ms":29529,"concrete_test":"Download the released SCP-116K dataset from the provided GitHub URL; parse all entries; count entries that contain both a non-empty problem and a non-empty solution. Compare that count to 116,756 and to 43,000. If the number of complete pairs is 43,000 (or any number below 70,000), the abstract's '116,756 problem-solution pairs' is false and the central claim must be corrected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the release of 116,756 high-quality problem-solution pairs (Abstract; Section 1). The pipeline's own counts do not support this. Section 3.4 reports roughly 190,000 problems and 80,000 solutions before filtering; Section 3.5 reports 116,756 problems and 70,000 solutions after filtering; Section 3.6 reports only 43,000 verified matches after numerical/semantic matching and GPT-4o verification. Since a complete pair requires one problem and one solution, the maximum possible pair count after Section 3.5 is min(116,756, 70,000) = 70,000, and the pipeline's verified output is 43,000. The abstract and conclusion describe 116,756 pairs, overstating the verified dataset by a factor of roughly 2.7. This is not a semantic quibble: the dataset's headline scale, the comparison to prior datasets in Section 2, and the stated 'comprehensive resource' are all premised on the larger number. If the released data actually contains 116,756 problems but only 43,000 paired solutions, the central contribution is a much smaller resource. The correctness of every downstream claim about dataset scale depends on this count.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SCP-116K, an automated pipeline for extracting science problem-solution pairs from heterogeneous documents, and claims a dataset of 116,756 high-quality pairs. The pipeline uses GPT-4o for document filtering, image-to-markdown conversion, segmentation, structured extraction, and verification, while matching combines numerical identifiers and semantic similarity. The authors also report baseline accuracy of o1-mini and QwQ-32B-preview on the dataset and distillation experiments with Qwen2.5-32B-Instruct evaluated on GPQA-diamond. The manuscript releases both the dataset and the extraction pipeline.","tokens_in":9206,"tokens_out":3016,"duration_ms":24080,"significance":"If the claims held, SCP-116K would be a valuable resource: it targets a real gap in higher-education science QA data, is larger than existing public resources, and the pipeline could transfer to other domains. The authors openly release the dataset and pipeline, and they provide baseline evaluations and distillation results, which supports reproducibility. However, the scale and quality claims must be reconciled with the pipeline's own counts and with the absence of human verification.","major_comments":[{"comment":"The central scale claim is not supported by the reported pipeline counts. The abstract and Section 1 describe '116,756 high-quality problem-solution pairs,' and Section 2 compares against prior datasets on this basis. Section 3.5, however, reports 116,756 problems and 70,000 solutions after quality filtering, and Section 3.6 reports 43,000 verified matched pairs. Since a pair requires at least one problem and one solution, the maximum possible number of pairs after Section 3.5 is min(116,756, 70,000) = 70,000, and the verified output is 43,000. The paper should either redefine the dataset as '116,756 problems with 43,000 verified paired solutions' or provide a count of released pairs that is consistent with the pipeline. As written, the headline scale overstates the available paired resource by a factor of about 2.7.","section":"Abstract; Section 3.5; Section 3.6"},{"comment":"The distillation experiments select training examples based on semantic similarity to GPQA-diamond questions and then evaluate on GPQA-diamond, which is a form of test-set selection bias. In Section 4.2, the hybrid approach uses '1,000 selected problem-solution pairs from SCP-116K, chosen based on semantic similarity to GPQA-diamond questions,' and Table 1 reports its GPQA-diamond accuracy. This overlap between training selection and evaluation makes the reported improvement (58.59%) an optimistic estimate of general scientific reasoning gains. The authors should evaluate on a held-out benchmark not used for data selection, or at least analyze sensitivity to the selection criterion.","section":"Section 4.2; Section 4.3; Table 1"},{"comment":"The 'high-quality' property is asserted without a human check. GPT-4o performs the image-to-markdown conversion, extraction, matching verification, and correctness judgment; no human-annotated sample, error analysis, or inter-annotator agreement is reported. The verification prompt in Appendix A.5 asks GPT-4o to judge whether a solution is 'correct' and 'complete' via its own reasoning, which is the same model family that produced the extraction. This does not establish the dataset's quality independent of the model being assessed. The authors should include a human-evaluated random sample (e.g., 100-200 pairs) with per-stage error rates, particularly for formula rendering and solution correctness.","section":"Section 3.2; Section 3.4; Section 3.6; Appendix A.5"}],"minor_comments":[{"comment":"Figure 1 contains a typo: 'Probelm' should be 'Problem.'","section":"Figure 1"},{"comment":"Section 3.7 says the models generate solutions 'for all problems in our dataset,' but it is unclear whether this refers to all 116,756 problems or the 43,000 verified pairs; please clarify.","section":"Section 3.7"},{"comment":"The paper does not report the number of source documents per discipline, the distribution of educational levels, or the release license and format; these details are needed for dataset usability.","section":"General dataset documentation"},{"comment":"Section 4.1 reports accuracy on SCP-116K (58.40% and 55.79%) but does not describe the evaluation protocol, such as exact-match versus model-based grading or the sample size; please specify.","section":"Section 4.1"},{"comment":"The related-work comparison in Section 2 would benefit from a quantitative statement of how many of the 116,756 entries are complete pairs after matching, to avoid overclaiming relative to CAMEL's 20,000 pairs.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The count discrepancy is severe enough that the paper should not be accepted in current form. If the dataset indeed contains only 43,000 verified pairs, the authors should revise the title and abstract accordingly; the current framing overstates the resource. The lack of human verification means 'high-quality' remains an assertion about an LLM pipeline rather than a demonstrated property. The fit to cs.CL is acceptable, but the claims need tightening before the paper can be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper builds a useful-looking pipeline for extracting higher-ed science problem-solution pairs from heterogeneous textbook sources, and the dataset is genuinely new. But the headline number—116,756 pairs—contradicts the paper's own pipeline counts, and the main distillation result is partly explained by selecting training examples similar to the test benchmark. Both are fixable, but they need to be fixed before this can be taken at face value.\n\nOn the positive side: the pipeline itself is a sensible integration of off-the-shelf components (document-to-image rendering, GPT-4o for markdown/LaTeX conversion, segmentation, structured extraction, then numerical and semantic matching with Stella embeddings). The authors are unusually transparent about counts at each stage, and they release the dataset and code. The target domain—undergraduate-to-doctoral science problems—is genuinely under-served compared to math datasets.\n\nNow the soft spots. The abstract and conclusion claim \"116,756 high-quality problem-solution pairs,\" but Section 3.5 reports 116,756 problems and 70,000 solutions after filtering, and Section 3.6 reports only 43,000 verified matched pairs. Since a pair requires one problem and one solution, the maximum possible pair count after filtering is 70,000. The 116K figure seems to count problems only. That's a 2.7x overstatement of the verified resource, and it undermines the dataset-scale comparisons in Sections 1 and 2. This is a load-bearing flaw in the paper's central claim.\n\nThe second issue is the distillation experiment. The \"hybrid\" model selects 1,000 SCP examples based on semantic similarity to GPQA-diamond, then evaluates on GPQA-diamond. That's essentially training on the test distribution. The GPQA improvement of about 11 points is likely inflated. The authors should evaluate on a held-out benchmark or on SCP itself.\n\nA third, softer issue: the \"high-quality\" label rests almost entirely on GPT-4o—it parses, extracts, matches, and verifies. There's no human-annotated sample or error analysis. That leaves the quality claim unverified, though not necessarily wrong. Accuracies in Table 1 are also reported without variance or a clear scoring metric—minor.\n\nWho is this for? People building scientific-reasoning training data and anyone interested in automated dataset construction. It would make a good reading-group discussion on the gap between dataset claims and dataset reality.\n\nRecommendation: send it to peer review, but with a firm request to align the abstract with the pipeline counts, redo the distillation evaluation without test-set similarity selection, and add a human quality audit. If those are addressed, the dataset could be a solid contribution.","headline":"A genuinely useful extraction pipeline and a new dataset for higher-ed science problems, but the paper's core pair-count claim is contradicted by its own pipeline, and the main distillation result is compromised by test-set similarity selection.","tokens_in":9829,"tokens_out":3367,"would_cite":false,"duration_ms":29574,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents SCP-116K, a dataset of 116,756 higher-education science problem-solution pairs, extracted automatically from heterogeneous sources by a six-stage pipeline.","keywords":["problem-solution dataset","higher education science","automated extraction pipeline","GPT-4o","knowledge distillation","scientific reasoning","GPQA","LaTeX transcription"],"falsifier":"Grade a random sample of 200 of the 43,000 verified pairs with two independent human graders, checking whether the LaTeX transcription matches the source image, whether the solution actually addresses the problem, and whether the final answer is correct; if a meaningful fraction fail, the 'high-quality' claim is unsupported.","tokens_in":8758,"feed_emoji":"🧪","tokens_out":5666,"duration_ms":46663,"temperature":0.7,"pith_summary":"The paper claims that a single automated pipeline can convert heterogeneous textbook sources, such as PDFs, EPUBs, and slide decks from digital libraries, into a large dataset of higher-education physics, chemistry, and biology problem-solution pairs. It introduces SCP-116K, advertised as 116,756 high-quality pairs, and a six-stage extraction workflow built on GPT-4o for rendering, parsing, segmenting, extracting, filtering, and matching. The motivation is that advanced reasoning models like o1 show what high-quality training data can do for mathematics, and science lacks an equivalent public resource. The paper also reports that only 43,000 pairs survive the final verification stage, and it uses the dataset in knowledge-distillation experiments that improve a 32B model's GPQA-diamond accuracy by about 11 percentage points.","feed_headline":"Pipeline extracts 116,756 university science problem-solution pairs","feed_subtitle":"Open dataset and reusable pipeline target STEM reasoning models; 43,000 pairs pass final verification.","key_machinery":"The load-bearing mechanism is the six-stage extraction pipeline: keyword retrieval from 6.69 million documents, uniform image rendering and GPT-4o image-to-markdown transcription with LaTeX, GPT-4o structural segmentation, structured extraction of problems, solutions, and numeric identifiers, two-stage quality filtering, and dual-pathway matching using numeric identifiers and the Stella embedding model with GPT-4o verification of each candidate pair. The dual-pathway matching is what lets the pipeline pair problems with solutions even when they sit in distant chapters or separate solution manuals.","core_discovery":"On its own terms, the central claim is that SCP-116K is the first large-scale, authentic higher-education science problem-solution dataset, created without synthetic generation, and that its extraction pipeline generalizes across document formats and domains. The pipeline works by rendering every document to images, using GPT-4o to transcribe the images to markdown with LaTeX formulas, segmenting on structural boundaries, extracting problems and solutions with their numeric identifiers, removing incomplete or self-referential entries, and then matching problems to solutions through a dual pathway: numeric identifier matching plus semantic similarity from the Stella embedding model. Each candidate match is verified by GPT-4o for correctness and completeness. This produces, by the paper's count in Section 3.6, 43,000 verified pairs; the abstract's 116,756 figure describes the number of problems after quality filtering, before final pair matching.","pith_inferences":["The title number 116,756 counts filtered problems, not matched pairs; the usable verified resource is the 43,000 pairs, so benchmark and training claims should be read against that smaller number.","Because every extraction and verification decision runs through GPT-4o, the dataset quality is upper-bounded by GPT-4o's transcription and reasoning accuracy; an independent open-weight parser run through the same pipeline would test that ceiling.","The same dual-pathway matching design should transfer to any field whose textbooks number exercises and provide answer keys, such as engineering or quantitative economics.","A human error analysis on even a few hundred pairs would turn 'high-quality' from an assertion into a measured property; the paper currently leaves that measurement undone."],"forward_implications":["Researchers can train or fine-tune models on authentic graduate-level physics, chemistry, and biology problems without synthetic generation, filling a gap left by datasets like CAMEL and ScienceQA.","The six-stage pipeline becomes a reusable recipe for turning other document collections into question-answering or reasoning datasets in specialized domains.","A 32B model fine-tuned on 1,000 SCP-116K examples plus STILL-2 data reaches 58.59% on GPQA-diamond, within about two points of o1-mini's 60.61%, so the dataset's content transfers to graduate-level reasoning benchmarks.","The dataset supplies baselines: o1-mini scores 58.40% and QwQ-32B-preview 55.79% on SCP-116K, leaving room for improvement.","Expanding collection to more fields and larger scale is the stated next step, which the pipeline's format-agnostic design is intended to support."],"supporting_citations":[{"why":"Supplies both the GPT-4o transcription, extraction, and verification engine and the o1-mini model whose distilled solutions are used in the experiments.","marker":"OpenAI, 2024"},{"why":"Provides the GPQA-diamond benchmark on which the distillation gains are measured and defines the graduate-level reasoning standard the dataset complements.","marker":"Rein et al., 2023"},{"why":"Provides the STILL-2 distillation framework and baseline data that SCP-116K examples replace or augment in the experiments.","marker":"Min et al., 2024"},{"why":"Provides the CAMEL physics problem-solution dataset, the synthetic-data baseline that SCP-116K is compared against and whose science QA is replaced in the STILL-2 variant.","marker":"Li et al., 2023"},{"why":"Supplies the Stella embedding model used in the semantic matching pathway to pair problems with candidate solutions.","marker":"NovaSearch, 2024"},{"why":"Provides QwQ-32B-preview, one of the two reasoning models used to collect model-generated solutions and benchmarked on SCP-116K and GPQA.","marker":"Team, 2024"}],"fun_headline_variants":["SCP-116K: 116K university science problem-solution pairs","Automated pipeline extracts 116K science problem-solution pairs","New dataset: 116K science problem-solution pairs from university PDFs","116K real university science Q&A pairs, no synthetic data","43K verified science problem-solution pairs via automated pipeline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's 'high-quality' label rests on GPT-4o being reliable at every step, converting textbook images into markdown with correct LaTeX, detecting boundaries, extracting complete problems, and judging whether a solution is correct and complete, yet the paper provides no human-annotated sample to check that reliability.","fun_headline_variants_meta":{"raw":{"variants":["SCP-116K: 116K university science problem-solution pairs","Automated pipeline extracts 116K science problem-solution pairs","New dataset: 116K science problem-solution pairs from university PDFs","116K real university science Q&A pairs, no synthetic data","43K verified science problem-solution pairs via automated pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00097,"raw_usage":{"total_tokens":4130,"prompt_tokens":958,"completion_tokens":3172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":3081}},"tokens_in":574,"tokens_out":3172,"duration_ms":18886,"temperature":1.0,"reasoning_tokens":3081,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:08:33.889417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Grade a random sample of 200 of the 43,000 verified pairs with two independent human graders, checking whether the LaTeX transcription matches the source image, whether the solution actually addresses the problem, and whether the final answer is correct; if a meaningful fraction fail, the 'high-quality' claim is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Stella embedding model used in the semantic matching pathway to pair problems with candidate solutions."}],"review_version":1}