{"id":"1e827084-8a4a-4de0-9bce-94b2aef723bf","arxiv_id":"2412.14436","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-stage data curation pipeline (GloVe similarity plus BERT educational-value scoring) produces an astronomy dataset that, after fine-tuning LLaMA-3-8B on 1B tokens, lifts MMLU astronomy from 69.08% to 76.3%.","lead":"ORBIT filters a 1.3-trillion-token web corpus into a 10-billion-token astronomy dataset using embedding similarity plus an educational-value classifier, then fine-tunes LLaMA-3-8B on a 1B-token slice. The resulting model improves MMLU astronomy accuracy from 69% to 76% and beats the AstroLLaMA baseline, but the paper lacks a control fine-tune on unfiltered data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No decontamination check: AstroBench's source articles are exactly the kind of content ORBIT retains, so the reported gain may be test-set memorization rather than domain adaptation.","rationale":"I endorse the reader's conditional verdict. The paper's released code, dataset, and model are real contributions, and the AstroBench gain is directionally credible if the benchmark is clean. However, the reader's weakest assumption (validity of the GPT-4o-labeled BERT regressor) is downstream of a more basic threat: the benchmark may overlap the training corpus. The paper describes AstroBench as derived from Annual Review articles, which are exactly the astronomy-relevant, high-educational-value documents ORBIT is designed to retain. In the absence of any contamination analysis, the quantitative headline cannot yet be read as evidence of domain adaptation. This is not a question of consensus or style; it is a falsifiable external-validity check. I would keep the CONDITIONAL verdict, adding decontamination as an explicit condition, and would not move to ACCEPT until the overlap analysis and deduplicated retraining are reported.","tokens_in":18567,"tokens_out":9609,"duration_ms":82053,"concrete_test":"Download the official AstroBench test questions and the 885 Annual Review source articles, and run a 13-gram near-duplicate overlap analysis between the 10B ORBIT corpus (especially the 1B-token training subset) and the benchmark. Remove every training document that shares a 13-gram with any AstroBench question or source article, retrain Orbit-LLaMA on the deduplicated 1B subset, and re-evaluate. If the AstroBench score drops materially toward the 61.5 baseline, the headline gain is memorization; if the score is unchanged after removal, the concern does not land. Repeat the overlap check for MMLU-astronomy as a secondary check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that AstroBench and MMLU measure transferable astronomy knowledge, not memorization of benchmark-related text. The paper's main quantitative evidence is the AstroBench gain (Meta-LLaMA 61.5 -> Orbit-LLaMA 69.7) on the official 4,425-question benchmark, which Section 4.4.1 states is built from 885 Annual Review of Astronomy and Astrophysics articles (1963-2023). ORBIT's source corpus is FineWeb-Edu, a CommonCrawl-derived educational corpus (Section 3.1), and the pipeline explicitly retains exactly the documents that are astronomy-relevant and educationally valuable (Sections 3.2.1-3.2.2). Annual Review astronomy articles are highly astronomy-relevant, educationally valuable, and plausibly present in CommonCrawl/FineWeb-Edu, so the 1B-token training sample could easily contain near-duplicates of AstroBench source material. The paper reports no decontamination step: no n-gram or near-duplicate overlap analysis, no exclusion of benchmark source documents, and no check of whether the base model already answers AstroBench from pretraining. Without such a check, the 8.2-point AstroBench improvement is consistent with test-set contamination. The MMLU astronomy gain (69.08 to 76.3 on 152 questions, roughly 11 questions) is also too small to carry the claim alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ORBIT, a two-stage pipeline that filters the 1.3T-token FineWeb-Edu corpus into a 10B-token astronomy-focused subset: Stage 1 retains documents whose average GloVe embedding has cosine similarity above τ=0.2 to the mean embedding of 101 astronomy terms, and Stage 2 retains documents whose educational value, scored by a BERT regressor trained on GPT-4o labels, is at least η=3. Fine-tuning LLaMA-3-8B on a randomly sampled 1B-token subset yields reported gains on MMLU astronomy (69.08% to 76.30%), AstroBench (61.5 to 69.7), and GPT-4o preference (73% vs. the base model). The authors also report GPT-4o-based educational-value improvements for law and medicine datasets, but without downstream task evaluation.","tokens_in":18869,"tokens_out":5009,"duration_ms":39084,"significance":"If the reported gains are causal, ORBIT is a low-cost, scalable recipe for domain adaptation from web corpora, and the open-sourced pipeline (code, data, model) would be a useful community resource. The astronomy case study is well-motivated, and the inclusion of expert qualitative evaluation is a strength. However, the central attribution claim is not yet supported: the paper lacks a same-size control fine-tune on random or keyword-filtered data, reports no seed variance or confidence intervals, and performs no decontamination check against the AstroBench source articles. The mathematical appendix formalizes the averaging heuristic, but its i.i.d. residual assumption is asserted rather than derived, and the empirical validation is only a small residual-distribution plot.","major_comments":[{"comment":"The paper reports that Orbit-LLaMA outperforms Meta-LLaMA and AstroLLaMA, but it never fine-tunes a control model on a same-size random sample of FineWeb-Edu or on a keyword-filtered sample. Without such a control, the observed gains could be due to continued pretraining on any 1B tokens of educational text rather than to ORBIT's specific filters. Please add at least one control fine-tune on 1B random FineWeb-Edu tokens and one on 1B keyword-filtered tokens, using identical training hyperparameters.","section":"Section 4.5 / Table 2"},{"comment":"All quantitative results come from a single fine-tuning run. The MMLU astronomy gain is 76.30 vs. 69.08 on 152 questions, which is about 11 questions, so even modest run-to-run variance could change the conclusion. Report results over at least three seeds, or provide bootstrap confidence intervals, for the main benchmark comparisons.","section":"Section 4.5 / Table 2"},{"comment":"AstroBench is constructed from 885 Annual Review of Astronomy and Astrophysics articles (1963–2023), and ORBIT's source corpus is FineWeb-Edu, a CommonCrawl-derived set that plausibly contains these articles or near-duplicates. The paper reports no decontamination analysis: no n-gram overlap measurement, no exclusion of benchmark source documents, and no check of whether the base model already answers AstroBench from pretraining. Without this, the 8.2-point AstroBench gain is consistent with test-set memorization rather than domain adaptation. Please report overlap statistics and, if needed, retrain after removing near-duplicates.","section":"Section 4.4.1 / AstroBench"},{"comment":"The law and medicine validation measures only GPT-4o educational-value scores of the filtered data; it does not fine-tune models or evaluate on any law/medicine benchmark. Moreover, the Stage 2 regressor is itself trained on GPT-4o labels, so using GPT-4o to score the output is partly circular. The abstract's claim of 'validated ORBIT's generalizability' is therefore overstated; either add downstream task evaluations for law and medicine or soften the claim to dataset-quality improvement only.","section":"Section 3.2.2 / Section 4.3"}],"minor_comments":[{"comment":"Appendix C and Appendix D contain essentially identical mathematical content; one should be removed.","section":"Appendix C and D"},{"comment":"The text says 'using Huggingface's HUGGINGFACEFW/FINEWEB-EDU-CLASSIFIER model' and also 'We developed a BERT-based regressor'; please clarify whether the regressor is the existing FineWeb-Edu classifier or a newly trained model on GPT-4o labels.","section":"Section 3.2.2"},{"comment":"The law lexicon promised in Section 3.2.3 is missing from the appendix; only the astronomy and medical term lists are shown.","section":"Appendix A.2 / Table 6"},{"comment":"Figure 4 does not specify what 'average score' refers to (dataset quality, downstream benchmark, or something else) or how many runs produce the standard error of the mean; please add a detailed caption and methodology.","section":"Figure 4"},{"comment":"Typos and formatting issues include 'qualtiy' and 'Further more' (Section 3.2.2), 'strength's' (Section 4.4.2), 'LL AMA' (abstract), and inconsistent capitalization of 'Orbit-LLaMA/Orbit-LLaMa'.","section":"Throughout"},{"comment":"The AstroLLaMA reference (Nguyen et al.) lacks a year and venue; the AstroBench reference (Ting et al.) should also be checked for completeness.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The methodology is promising, but the experimental section does not yet support the causal claims. The missing controls, the absence of variance reporting, and the lack of decontamination analysis are the critical gaps. These are fixable with additional experiments, so I do not recommend rejection, but the revisions are substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the good news. ORBIT ships something the community can use right away: a 10B-token astronomy corpus, the code, and a fine-tuned LLaMA-3-8B. The two-stage filtering idea is not conceptually new—GloVe similarity plus a BERT quality regressor is essentially FineWeb-Edu with a domain twist—but the cost breakdown is genuinely useful, and the open release makes it easy to test variants. That alone makes the paper worthwhile.\n\nThe soft spots are in the experimental design. The main quantitative claims—7 points on MMLU astronomy, 8.2 on AstroBench—come from a single run with no seed variance. There is no control fine-tuned on a random sample of FineWeb-Edu of the same size, nor a keyword-filtered sample, so the paper never shows that ORBIT's specific filters (thresholds, regressor) are what caused the gain. The small MMLU sample size (152 questions) makes the 7-point gain roughly 11 questions; without variance you can't tell if that's signal or noise.\n\nThe stress-test concern about decontamination is valid and load-bearing for the AstroBench result. AstroBench is built from 885 Annual Review of Astronomy and Astrophysics articles. FineWeb-Edu comes from CommonCrawl, and Annual Review content is exactly the kind of educational, astronomy-heavy text that ORBIT retains. The paper reports no overlap analysis, no n-gram decontamination, no exclusion of benchmark source documents. So the AstroBench gain could be memorization, not domain adaptation. MMLU is a separate benchmark, but it's also in the wild and should be checked for contamination.\n\nThe law/medicine validation is a side note, not evidence of generalizability: only GPT-4o scores on the filtered datasets, no fine-tuned models, no downstream benchmarks, and the scorer is the same model family that produced the labels for the Stage 2 regressor, so that part is circular. The GPT-4o preference win rate (73%) has the same circularity, though the graduate student evaluations give some independent signal.\n\nOn the citation pattern and framing, the paper is honest about limitations and related work. It doesn't oversell the novelty. The central argument—that careful curation helps domain fine-tuning—is plausible and probably true, but the evidence as presented doesn't yet support it.\n\nWho should read this: anyone working on domain adaptation or dataset curation will want the corpus and code. It deserves a serious referee, but the referee should send it back for a control fine-tuning, variance reporting, and a decontamination analysis. With those, the paper could become a solid reference point for the field.","headline":"Useful open-source dataset and pipeline, but the paper needs a proper control fine-tuning and a decontamination check before we can believe the benchmark gains come from ORBIT.","tokens_in":19411,"tokens_out":2657,"would_cite":true,"duration_ms":21319,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage filter turns noisy web text into specialist training data, lifting an astronomy model's MMLU score from 69% to 76%.","keywords":["dataset curation","domain adaptation","large language models","astronomy","embedding-based filtering","educational value","fine-tuning","FineWeb-Edu"],"falsifier":"Train the same LLaMA-3-8B on a matched 1B-token slice of the Stage-1-only astronomy corpus (no educational-value filter) and on a 1B-token slice of the lowest-scoring Stage-2 documents; if either matches or beats the ORBIT slice on MMLU astronomy and official AstroBench, the Stage 2 quality filter is not the cause of the reported gains.","tokens_in":18374,"feed_emoji":"🔭","tokens_out":5963,"duration_ms":41634,"temperature":0.7,"pith_summary":"This paper claims that a cheap two-stage filter can carve a high-quality, domain-specific dataset out of a huge general web corpus, and that fine-tuning on a fraction of that dataset makes a general large language model markedly better at astronomy. The method, called ORBIT, first keeps documents whose GloVe word-embedding cosine similarity to an averaged astronomy term vector exceeds a threshold, then keeps only documents a BERT-based regressor scores as educationally valuable. Fine-tuning LLaMA-3-8B on a 1B-token slice of the resulting 10B-token astronomy corpus raises MMLU astronomy accuracy from 69.08% to 76.3% and AstroBench from 61.5 to 69.7, beating an arXiv-only astronomy model. If true, the practical consequence is that specialist models can be built from open web data at low cost, without waiting for curated domain corpora.","feed_headline":"Filtered web data lifts astronomy LLM from 69% to 76%","feed_subtitle":"Two-stage curation squeezes 1.3T noisy tokens into a 10B astronomy set that beats arXiv-only baselines.","key_machinery":"The load-bearing object is the two-stage ORBIT pipeline. Stage 1's astronomy vector $A$ is the mean of normalized GloVe embeddings of domain terms; by the law of large numbers the per-term noise components average out, so $A$ approximates the shared domain component, and cosine similarity $A\\cdot B/(|A||B|)$ between $A$ and a document vector $B$ measures topical relevance at near-zero marginal cost. Stage 2's BERT-based regressor scores retained documents on a 0-5 educational-value scale, using the $\\eta=3$ cutoff to remove shallow or off-topic text. The pipeline works because cheap embedding filtering reduces the corpus roughly 100-fold before the expensive classifier runs, making the whole curation cost about $206 instead of an estimated $16,200 for classifier-only filtering.","core_discovery":"The central discovery is that semantic relevance and educational value, measured separately, identify the training documents that transfer to domain expertise. Stage 1 uses a single averaged GloVe vector built from 101 astronomy terms and keeps documents with cosine similarity above $\\tau=0.2$, shrinking the 1.3T-token FineWeb-Edu corpus to about 20B tokens. Stage 2 applies a BERT-based regressor trained on 50,000 GPT-4o educational-value ratings and keeps documents scoring at or above $\\eta=3$, yielding a 10B-token astronomy set. Fine-tuning LLaMA-3-8B on a random 1B-token subset of that set outperforms the base model and an arXiv-abstract-only astronomy model on MMLU astronomy, official AstroBench, and expert preference ratings; the same pipeline applied to law and medicine raises average GPT-4o educational-value scores from about 0.4 to above 2.9.","pith_inferences":["Inference (editorial, not in the paper): the performance gain may come mostly from Stage 1's relevance filter, since the authors report that Stage 1 alone already lifts educational-value scores in law and medicine; an ablation training on Stage-1-only data would separate the two stages' contributions.","Inference (editorial): the $\\eta=3$ cutoff and the GPT-4o-derived definition of 'educational value' are likely domain-sensitive; applying ORBIT to a new field may require re-calibrating both thresholds rather than reusing the astronomy values.","Inference (editorial): because fine-tuning uses only 1B tokens, ORBIT suggests a scaling route opposite to pretraining: targeted small curated slices, not larger corpora, may drive domain gains; testing on a 2B-5B token slice would show whether the curve is saturated."],"forward_implications":["Specialist language models can be built from open web data: the 10B-token ORBIT astronomy set and the fine-tuned model are released, so the result is directly testable and reusable.","A 1B-token subset is enough: fine-tuning on one-tenth of the curated set already produces the reported gains, suggesting large uncurated corpora are not necessary for domain adaptation.","The curation cost is small: the combined pipeline runs in roughly 297 hours on commodity hardware plus one GPU, versus an estimated 12,000 GPU-hours for Stage 2 alone, which lowers the barrier for smaller groups.","The method transfers across domains: applying the same two-stage recipe to law and medicine raises GPT-4o-rated educational value of retained documents from about 0.4 to 2.9-3.05, indicating the pipeline is not astronomy-specific."],"supporting_citations":[{"why":"Supplies the 1.3T-token FineWeb-Edu corpus that ORBIT filters down to the 10B-token astronomy set.","marker":"Penedo et al., 2024"},{"why":"Provides the GloVe word embeddings used in Stage 1 to build the astronomy vector and document vectors.","marker":"Pennington et al., 2014"},{"why":"Provides the BERT architecture behind the Stage 2 educational-value regressor.","marker":"Devlin et al., 2019"},{"why":"Provides GPT-4o, which generated the 50,000 educational-value labels used to train the Stage 2 regressor.","marker":"OpenAI et al., 2024"},{"why":"Provides LLaMA-3-8B, the base model fine-tuned on the ORBIT-curated subset.","marker":"Dubey et al., 2024"},{"why":"Supplies the MMLU astronomy benchmark on which the fine-tuned model gains 7 points.","marker":"Hendrycks et al., 2021"},{"why":"Supplies the official AstroBench benchmark and the AstroMLab evaluation protocol used for comparison.","marker":"Ting et al., 2024"},{"why":"Provides AstroLLaMA, the arXiv-abstract-only astronomy baseline that ORBIT outperforms.","marker":"Nguyen et al."}],"fun_headline_variants":["Filtering 1.3T tokens to 10B lifts astronomy LLM by 7 points","ORBIT: cost-efficient curation boosts astronomy LLM from 69% to 76%","Semantic relevance plus education score: recipe for astronomy LLM boost","Web noise to expert: 10B-token astronomy set beats arXiv baseline","LLaMA-3 gets 7-point astronomy jump with curated 1B tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the BERT regressor's GPT-4o-derived educational-value scores track what actually makes text good for domain fine-tuning, so the $\\eta=3$ cutoff keeps exactly the documents that improve the model rather than merely sounding educational.","fun_headline_variants_meta":{"raw":{"variants":["Filtering 1.3T tokens to 10B lifts astronomy LLM by 7 points","ORBIT: cost-efficient curation boosts astronomy LLM from 69% to 76%","Semantic relevance plus education score: recipe for astronomy LLM boost","Web noise to expert: 10B-token astronomy set beats arXiv baseline","LLaMA-3 gets 7-point astronomy jump with curated 1B tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000942,"raw_usage":{"total_tokens":4082,"prompt_tokens":1056,"completion_tokens":3026,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":2917}},"tokens_in":672,"tokens_out":3026,"duration_ms":18732,"temperature":1.0,"reasoning_tokens":2917,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:14:01.720009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same LLaMA-3-8B on a matched 1B-token slice of the Stage-1-only astronomy corpus (no educational-value filter) and on a 1B-token slice of the lowest-scoring Stage-2 documents; if either matches or beats the ORBIT slice on MMLU astronomy and official AstroBench, the Stage 2 quality filter is not the cause of the reported gains.","supporting_citations":[],"review_version":1}