ORBIT: Cost-Effective Dataset Curation for Large Language Model Domain Adaptation with an Astronomy Case Study

ChengXiang Zhai; Eric Modesitt; Ke Yang; Spencer Hulsey; Volodymyr Kindratenko

REVIEW 4 major objections 6 minor 19 references

ORBIT: Cost-Effective Dataset Curation for Large Language Model Domain Adaptation with an Astronomy Case Study

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-stage filter turns noisy web text into specialist training data, lifting an astronomy model's MMLU score from 69% to 76%.

desk verdict Useful open-source dataset and pipeline, but the paper needs a proper control fine-tuning and a decontamination check before we can believe the benchmark gains come from ORBIT. read the letter →

arxiv 2412.14436 v1 pith:UJCY3SCO submitted 2024-12-19 cs.CL cs.AI

Eric Modesitt , Ke Yang , Spencer Hulsey , Chengxiang Zhai , Volodymyr Kindratenko This is my paper · ORCID

classification cs.CLcs.AI

keywords datasetcurationdomainadaptationlargelanguagemodelsastronomyembedding-basedfilteringeducationalvaluefine-tuningFineWeb-Edu

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a cheap two-stage filter can carve a high-quality, domain-specific dataset out of a huge general web corpus, and that fine-tuning on a fraction of that dataset makes a general large language model markedly better at astronomy. The method, called ORBIT, first keeps documents whose GloVe word-embedding cosine similarity to an averaged astronomy term vector exceeds a threshold, then keeps only documents a BERT-based regressor scores as educationally valuable. Fine-tuning LLaMA-3-8B on a 1B-token slice of the resulting 10B-token astronomy corpus raises MMLU astronomy accuracy from 69.08% to 76.3% and AstroBench from 61.5 to 69.7, beating an arXiv-only astronomy model. If true, the practical consequence is that specialist models can be built from open web data at low cost, without waiting for curated domain corpora.

What carries the argument

The load-bearing object is the two-stage ORBIT pipeline. Stage 1's astronomy vector $A$ is the mean of normalized GloVe embeddings of domain terms; by the law of large numbers the per-term noise components average out, so $A$ approximates the shared domain component, and cosine similarity $A\cdot B/(|A||B|)$ between $A$ and a document vector $B$ measures topical relevance at near-zero marginal cost. Stage 2's BERT-based regressor scores retained documents on a 0-5 educational-value scale, using the $\eta=3$ cutoff to remove shallow or off-topic text. The pipeline works because cheap embedding filtering reduces the corpus roughly 100-fold before the expensive classifier runs, making the whole curation cost about $206 instead of an estimated $16,200 for classifier-only filtering.

What would settle it

Train the same LLaMA-3-8B on a matched 1B-token slice of the Stage-1-only astronomy corpus (no educational-value filter) and on a 1B-token slice of the lowest-scoring Stage-2 documents; if either matches or beats the ORBIT slice on MMLU astronomy and official AstroBench, the Stage 2 quality filter is not the cause of the reported gains.

Watch

Extended reading notes

Core claim

The central discovery is that semantic relevance and educational value, measured separately, identify the training documents that transfer to domain expertise. Stage 1 uses a single averaged GloVe vector built from 101 astronomy terms and keeps documents with cosine similarity above $\tau=0.2$, shrinking the 1.3T-token FineWeb-Edu corpus to about 20B tokens. Stage 2 applies a BERT-based regressor trained on 50,000 GPT-4o educational-value ratings and keeps documents scoring at or above $\eta=3$, yielding a 10B-token astronomy set. Fine-tuning LLaMA-3-8B on a random 1B-token subset of that set outperforms the base model and an arXiv-abstract-only astronomy model on MMLU astronomy, official AstroBench, and expert preference ratings; the same pipeline applied to law and medicine raises average GPT-4o educational-value scores from about 0.4 to above 2.9.

Load-bearing premise

The load-bearing premise is that the BERT regressor's GPT-4o-derived educational-value scores track what actually makes text good for domain fine-tuning, so the $\eta=3$ cutoff keeps exactly the documents that improve the model rather than merely sounding educational.

Editorial extensions

If this is right

Specialist language models can be built from open web data: the 10B-token ORBIT astronomy set and the fine-tuned model are released, so the result is directly testable and reusable.
A 1B-token subset is enough: fine-tuning on one-tenth of the curated set already produces the reported gains, suggesting large uncurated corpora are not necessary for domain adaptation.
The curation cost is small: the combined pipeline runs in roughly 297 hours on commodity hardware plus one GPU, versus an estimated 12,000 GPU-hours for Stage 2 alone, which lowers the barrier for smaller groups.
The method transfers across domains: applying the same two-stage recipe to law and medicine raises GPT-4o-rated educational value of retained documents from about 0.4 to 2.9-3.05, indicating the pipeline is not astronomy-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference (editorial, not in the paper): the performance gain may come mostly from Stage 1's relevance filter, since the authors report that Stage 1 alone already lifts educational-value scores in law and medicine; an ablation training on Stage-1-only data would separate the two stages' contributions.
Inference (editorial): the $\eta=3$ cutoff and the GPT-4o-derived definition of 'educational value' are likely domain-sensitive; applying ORBIT to a new field may require re-calibrating both thresholds rather than reusing the astronomy values.
Inference (editorial): because fine-tuning uses only 1B tokens, ORBIT suggests a scaling route opposite to pretraining: targeted small curated slices, not larger corpora, may drive domain gains; testing on a 2B-5B token slice would show whether the curve is saturated.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the good news. ORBIT ships something the community can use right away: a 10B-token astronomy corpus, the code, and a fine-tuned LLaMA-3-8B. The two-stage filtering idea is not conceptually new—GloVe similarity plus a BERT quality regressor is essentially FineWeb-Edu with a domain twist—but the cost breakdown is genuinely useful, and the open release makes it easy to test variants. That alone makes the paper worthwhile.

The soft spots are in the experimental design. The main quantitative claims—7 points on MMLU astronomy, 8.2 on AstroBench—come from a single run with no seed variance. There is no control fine-tuned on a random sample of FineWeb-Edu of the same size, nor a keyword-filtered sample, so the paper never shows that ORBIT's specific filters (thresholds, regressor) are what caused the gain. The small MMLU sample size (152 questions) makes the 7-point gain roughly 11 questions; without variance you can't tell if that's signal or noise.

The stress-test concern about decontamination is valid and load-bearing for the AstroBench result. AstroBench is built from 885 Annual Review of Astronomy and Astrophysics articles. FineWeb-Edu comes from CommonCrawl, and Annual Review content is exactly the kind of educational, astronomy-heavy text that ORBIT retains. The paper reports no overlap analysis, no n-gram decontamination, no exclusion of benchmark source documents. So the AstroBench gain could be memorization, not domain adaptation. MMLU is a separate benchmark, but it's also in the wild and should be checked for contamination.

The law/medicine validation is a side note, not evidence of generalizability: only GPT-4o scores on the filtered datasets, no fine-tuned models, no downstream benchmarks, and the scorer is the same model family that produced the labels for the Stage 2 regressor, so that part is circular. The GPT-4o preference win rate (73%) has the same circularity, though the graduate student evaluations give some independent signal.

On the citation pattern and framing, the paper is honest about limitations and related work. It doesn't oversell the novelty. The central argument—that careful curation helps domain fine-tuning—is plausible and probably true, but the evidence as presented doesn't yet support it.

Who should read this: anyone working on domain adaptation or dataset curation will want the corpus and code. It deserves a serious referee, but the referee should send it back for a control fine-tuning, variance reporting, and a decontamination analysis. With those, the paper could become a solid reference point for the field.

Referee Report

4 major / 6 minor

Summary. The paper introduces ORBIT, a two-stage pipeline that filters the 1.3T-token FineWeb-Edu corpus into a 10B-token astronomy-focused subset: Stage 1 retains documents whose average GloVe embedding has cosine similarity above τ=0.2 to the mean embedding of 101 astronomy terms, and Stage 2 retains documents whose educational value, scored by a BERT regressor trained on GPT-4o labels, is at least η=3. Fine-tuning LLaMA-3-8B on a randomly sampled 1B-token subset yields reported gains on MMLU astronomy (69.08% to 76.30%), AstroBench (61.5 to 69.7), and GPT-4o preference (73% vs. the base model). The authors also report GPT-4o-based educational-value improvements for law and medicine datasets, but without downstream task evaluation.

Significance. If the reported gains are causal, ORBIT is a low-cost, scalable recipe for domain adaptation from web corpora, and the open-sourced pipeline (code, data, model) would be a useful community resource. The astronomy case study is well-motivated, and the inclusion of expert qualitative evaluation is a strength. However, the central attribution claim is not yet supported: the paper lacks a same-size control fine-tune on random or keyword-filtered data, reports no seed variance or confidence intervals, and performs no decontamination check against the AstroBench source articles. The mathematical appendix formalizes the averaging heuristic, but its i.i.d. residual assumption is asserted rather than derived, and the empirical validation is only a small residual-distribution plot.

major comments (4)

[Section 4.5 / Table 2] The paper reports that Orbit-LLaMA outperforms Meta-LLaMA and AstroLLaMA, but it never fine-tunes a control model on a same-size random sample of FineWeb-Edu or on a keyword-filtered sample. Without such a control, the observed gains could be due to continued pretraining on any 1B tokens of educational text rather than to ORBIT's specific filters. Please add at least one control fine-tune on 1B random FineWeb-Edu tokens and one on 1B keyword-filtered tokens, using identical training hyperparameters.
[Section 4.5 / Table 2] All quantitative results come from a single fine-tuning run. The MMLU astronomy gain is 76.30 vs. 69.08 on 152 questions, which is about 11 questions, so even modest run-to-run variance could change the conclusion. Report results over at least three seeds, or provide bootstrap confidence intervals, for the main benchmark comparisons.
[Section 4.4.1 / AstroBench] AstroBench is constructed from 885 Annual Review of Astronomy and Astrophysics articles (1963–2023), and ORBIT's source corpus is FineWeb-Edu, a CommonCrawl-derived set that plausibly contains these articles or near-duplicates. The paper reports no decontamination analysis: no n-gram overlap measurement, no exclusion of benchmark source documents, and no check of whether the base model already answers AstroBench from pretraining. Without this, the 8.2-point AstroBench gain is consistent with test-set memorization rather than domain adaptation. Please report overlap statistics and, if needed, retrain after removing near-duplicates.
[Section 3.2.2 / Section 4.3] The law and medicine validation measures only GPT-4o educational-value scores of the filtered data; it does not fine-tune models or evaluate on any law/medicine benchmark. Moreover, the Stage 2 regressor is itself trained on GPT-4o labels, so using GPT-4o to score the output is partly circular. The abstract's claim of 'validated ORBIT's generalizability' is therefore overstated; either add downstream task evaluations for law and medicine or soften the claim to dataset-quality improvement only.

minor comments (6)

[Appendix C and D] Appendix C and Appendix D contain essentially identical mathematical content; one should be removed.
[Section 3.2.2] The text says 'using Huggingface's HUGGINGFACEFW/FINEWEB-EDU-CLASSIFIER model' and also 'We developed a BERT-based regressor'; please clarify whether the regressor is the existing FineWeb-Edu classifier or a newly trained model on GPT-4o labels.
[Appendix A.2 / Table 6] The law lexicon promised in Section 3.2.3 is missing from the appendix; only the astronomy and medical term lists are shown.
[Figure 4] Figure 4 does not specify what 'average score' refers to (dataset quality, downstream benchmark, or something else) or how many runs produce the standard error of the mean; please add a detailed caption and methodology.
[Throughout] Typos and formatting issues include 'qualtiy' and 'Further more' (Section 3.2.2), 'strength's' (Section 4.4.2), 'LL AMA' (abstract), and inconsistent capitalization of 'Orbit-LLaMA/Orbit-LLaMa'.
[References] The AstroLLaMA reference (Nguyen et al.) lacks a year and venue; the AstroBench reference (Ting et al.) should also be checked for completeness.

Circularity Check

2 steps flagged · score 2.0 of 10

Minor same-oracle evaluation in qualitative and cross-domain claims; main benchmark results are externally grounded.

other [Section 4.3 (Cross-Domain Validation) and Section 3.2.2]
"The average educational value scores, evaluated using GPT-4o, showed significant improvements over the unfiltered baseline (0.3), with 2.9 for medicine and 3.05 for law... indicating that Stage 1 filtering alone is sufficient to extract high-quality, domain-specific content across diverse fields."

The law/medicine 'data quality' improvement is measured with GPT-4o, the same oracle that defines the educational-value construct and supplies the labels for the Stage 2 BERT regressor. The filter's selection criterion (cosine similarity to domain lexicons) is designed to retain domain-relevant text, and the GPT-4o prompt rewards domain relevance, so the reported gain from 0.3 to 2.9/3.05 largely restates the selection criterion in the evaluation metric's own units. This is a re-measurement of the intended filtering effect rather than an independent validation, though it does not affect the external astronomy benchmarks.
other [Abstract; Section 3.2.2; Section 4.5, Table 3]
"with GPT-4o evaluations preferring it in 73% of cases across 1000 astronomy-specific questions."

The fine-tuning data were curated using GPT-4o-generated educational-value labels (Section 3.2.2), and the same model family then judges the fine-tuned model's outputs in the 73% preference result. The preference therefore partly measures how well Orbit-LLaMA conforms to GPT-4o's notion of good astronomy text rather than an independent quality signal. This is not load-bearing for the central claim because MMLU astronomy and official AstroBench scores are external, and the paper also reports independent human graduate-student preferences (66%) that corroborate the qualitative result.

full rationale

The central derivation chain is not circular: the ORBIT pipeline composes GloVe cosine thresholding with a BERT regressor, and the headline results are measured on external benchmarks (MMLU astronomy and the official AstroBench) that were not used to set the thresholds or train the regressor. The only self-referential parts are the law/medicine 'quality' validation and the GPT-4o preference evaluation, both of which use GPT-4o as the quality oracle that also produced the educational-value labels for Stage 2. These are minor and non-load-bearing for the main astronomy benchmark claim, which stands independently. The absence of a decontamination check (AstroBench's source Annual Review articles could plausibly appear in FineWeb-Edu) is a real correctness and contamination risk, but it is not a by-construction circularity and therefore does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims depend on hand-chosen thresholds (tau=0.2, eta=3) and a hand-built 101-term lexicon; the educational-value regressor inherits GPT-4o's judgment. No new physical or formal entities are invented.

free parameters (4)

similarity threshold tau = 0.2
Empirically chosen to balance dataset size and quality; retains approximately 20B tokens (Section 3.2.1).
educational value threshold eta = 3
Retains documents with regressor score greater than or equal to 3, yielding approximately 10B tokens (Section 3.2.2).
astronomy term lexicon = 101 hand-selected terms
Hand-curated list used to define the domain vector A; affects which documents are retained in Stage 1 (Appendix A.2).
embedding model and dimension = GloVe, fastText, 100d and 300d variants tested
Filtering results vary slightly by embedding choice (Figure 4), but the final configuration is not clearly specified in the text.

assumptions (5)

domain assumption GloVe word embeddings accurately represent semantic relatedness for astronomy domain filtering
Stage 1 relies on cosine similarity between averaged GloVe embeddings and the astronomy vector to identify relevant documents (Section 3.2.1).
domain assumption The 101-term astronomy lexicon adequately covers the astronomy domain
The aggregated vector A is the mean of these terms; coverage gaps would bias filtering away from relevant subtopics (Section 3.2.1, Appendix A.2).
domain assumption GPT-4o educational-value labels are a valid proxy for training-data quality
Stage 2 regressor is trained on GPT-4o scores; the threshold eta=3 assumes these scores separate valuable from worthless documents (Section 3.2.2).
ad hoc to paper Embedding residuals ri are i.i.d. with zero mean
Appendix C and D decompose term embeddings as a + r_i and invoke the law of large numbers; this is an idealized model, not empirically guaranteed for real word embeddings.
domain assumption FineWeb-Edu is a high-quality base corpus for domain adaptation
The pipeline starts from a 1.3T-token educational subset of FineWeb; if the base corpus is skewed or low-quality, ORBIT inherits the skew (Section 3.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of ORBIT: Cost-Effective Dataset Curation for Large Language Model Domain Adaptation with an Astronomy Case Study." pith.science (2026). https://pith.science/paper/UJCY3SCO

@misc{pith2026241214436,
  author       = {Pith},
  title        = {Pith review of: ORBIT: Cost-Effective Dataset Curation for Large Language Model Domain Adaptation with an Astronomy Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJCY3SCO}},
  note         = {Machine review of arXiv:2412.14436}
}

read the original abstract

Recent advances in language modeling demonstrate the need for high-quality domain-specific training data, especially for tasks that require specialized knowledge. General-purpose models, while versatile, often lack the depth needed for expert-level tasks because of limited domain-specific information. Domain adaptation training can enhance these models, but it demands substantial, high-quality data. To address this, we propose ORBIT, a cost-efficient methodology for curating massive, high-quality domain-specific datasets from noisy web sources, tailored for training specialist large language models. Using astronomy as a primary case study, we refined the 1.3T-token FineWeb-Edu dataset into a high-quality, 10B-token subset focused on astronomy. Fine-tuning \textsc{LLaMA-3-8B} on a 1B-token astronomy subset improved performance on the MMLU astronomy benchmark from 69\% to 76\% and achieved top results on AstroBench, an astronomy-specific benchmark. Moreover, our model (Orbit-LLaMA) outperformed \textsc{LLaMA-3-8B-base}, with GPT-4o evaluations preferring it in 73\% of cases across 1000 astronomy-specific questions. Additionally, we validated ORBIT's generalizability by applying it to law and medicine, achieving a significant improvement of data quality compared to an unfiltered baseline. We open-source the ORBIT methodology, including the curated datasets, the codebase, and the resulting model at \href{https://github.com/ModeEric/ORBIT-Llama}{https://github.com/ModeEric/ORBIT-Llama}.

Figures

Figures reproduced from arXiv: 2412.14436 by the authors.

**Figure 2.** Full Stage 2 pipeline visualized. the cosine similarity between its embedding vector and the domain-specific aggregated vector was calculated. Documents exceeding the similarity threshold of 0.2 were retained for further analysis. 4 Experiments To validate the effectiveness of the ORBIT methodology, we conducted a series of experiments focusing on the quality of the curated dataset, the impact of fine-tuning on m… view at source ↗

**Figure 4.** Average Score vs Percent Kept, comparing [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Distribution of residual components for the [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 12 canonical work pages

[1]

This approach allowed review- ers to account for cases where correctness was unclear or when responses were partially cor- rect

Preference Ratings: Four graduate students were asked to select their preferred response for each question, considering the aforemen- tioned criteria. This approach allowed review- ers to account for cases where correctness was unclear or when responses were partially cor- rect

work page
[2]

Detailed Feedback: Two astronomy gradu- ate students with Teaching Assistant grading experience analyzed the responses in-depth, identifying strengths, weaknesses, and id- iosyncrasies of each model. A.4.3 Model-Specific Observations Detailed grader comments highlighted the follow- ing characteristics: • Meta LLaMA-3 : Generated verbose and repetitive ans...

work page
[3]

Rank the Responses : Rank the responses from the three models in order of preference for each question, with justifications for their rankings

work page
[4]

B.3.2 Recruitment and Voluntary Participation All annotators were graduate students specializ- ing in astronomy

Provide Feedback: Offer detailed comments on strengths and weaknesses for each re- sponse, particularly highlighting areas where models excelled or failed. B.3.2 Recruitment and Voluntary Participation All annotators were graduate students specializ- ing in astronomy. They volunteered for the task, and their contributions were limited to evaluating a mana...

work page 2020
[6]

arXiv preprint

GPT-4o System Card. arXiv preprint . ArXiv:2410.21276. Rui Pan, Tuan Dung Nguyen, Hardik Arora, and et al

arXiv
[7]

arXiv preprint

AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy. arXiv preprint. ArXiv:2409.19750. Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben allal, and et al. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv preprint. ArXiv:2406.17557. Jeffrey Pennington, Richard Socher, and Christopher Manning...

arXiv 2024
[8]

Score : X

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv preprint. ArXiv:1910.10683. Karan Singhal, Shekoofeh Azizi, Tao Tu, and et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172–180. Yuan-Sen Ting, Tuan Dung Nguyen, Tirthankar Ghosal, and et al. 2024. AstroMLab 1: Who Wins Astronomy Jeopar...

arXiv 1910
[11]

Read and Assess : Carefully review the re- sponses generated by the three models for each test question

work page

Show all 19 references

[12]

• Clarity: Assess whether the response is well-structured, easy to read, and free of ambiguity

Evaluate Against Criteria: • Accuracy: Determine if the content of the response is factually correct and rele- vant to the question. • Clarity: Assess whether the response is well-structured, easy to read, and free of ambiguity. • Reasoning: Evaluate the logical struc- ture an...
[18]

Thresholding: Retain the document if the similarity exceeds τ. Runtime: O(1) Total Complexity per Document: O(s · d) Given N documents, each with s tokens on av- erage, the overall complexity for the filtering step is: O(N · s · d) Optimizations Implemented: • Precomputation o...

2020
[19]

Runtime: O(1)

Embedding Lookup: For each token wj in document D, retrieve its embedding ewj from a hashmap. Runtime: O(1)
[20]

Runtime: O(s · d)

Document Vector Computation: Calculate B = Ps j=1 ewj. Runtime: O(s · d)
[21]

Runtime: O(d)

Similarity Calculation: Compute cosine sim- ilarity between A and B. Runtime: O(d)
[22]

Thresholding: Retain the document if the similarity exceeds τ. Runtime: O(1) Total Complexity per Document: O(s · d) Given N documents, each with s tokens on av- erage, the overall complexity for the filtering step is: O(N · s · d) Optimizations Implemented: • Precomputation o...
[2019]

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long and Short Papers), pages 417...

2019 arXiv
[2021]

arXiv preprint

Measuring Massive Multitask Language Un- derstanding. arXiv preprint. ArXiv:2009.03300. Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Alek- sander M ˛ adry, Alex Baker-Whitcomb, Alex B...

2009 arXiv
[2022]

Transactions of the Associa- tion for Computational Linguistics , 10:50–72

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the Associa- tion for Computational Linguistics , 10:50–72. Hongyu Li, Liang Ding, Meng Fang, and et al. 2024. Re- visiting Catastrophic Forgetting in Large Language Model Tuning. In Findings o...

2024
[2023]

arXiv preprint

Data-Juicer: A One-Stop Data Processing System for Large Language Models. arXiv preprint. ArXiv:2309.02033. 10 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and et al

arXiv
[2024]

GPT-4 System Card. OpenAI. Jared Kaplan, Sam McCandlish, Tom Henighan, and et al. 2020. Scaling Laws for Neural Language Mod- els. arXiv preprint. ArXiv:2001.08361. Julia Kreutzer, Isaac Caswell, Lisa Wang, and et al

2020 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

This approach allowed review- ers to account for cases where correctness was unclear or when responses were partially cor- rect

Preference Ratings: Four graduate students were asked to select their preferred response for each question, considering the aforemen- tioned criteria. This approach allowed review- ers to account for cases where correctness was unclear or when responses were partially cor- rect

work page

[2] [2]

Detailed Feedback: Two astronomy gradu- ate students with Teaching Assistant grading experience analyzed the responses in-depth, identifying strengths, weaknesses, and id- iosyncrasies of each model. A.4.3 Model-Specific Observations Detailed grader comments highlighted the follow- ing characteristics: • Meta LLaMA-3 : Generated verbose and repetitive ans...

work page

[3] [3]

Rank the Responses : Rank the responses from the three models in order of preference for each question, with justifications for their rankings

work page

[4] [4]

B.3.2 Recruitment and Voluntary Participation All annotators were graduate students specializ- ing in astronomy

Provide Feedback: Offer detailed comments on strengths and weaknesses for each re- sponse, particularly highlighting areas where models excelled or failed. B.3.2 Recruitment and Voluntary Participation All annotators were graduate students specializ- ing in astronomy. They volunteered for the task, and their contributions were limited to evaluating a mana...

work page 2020

[5] [6]

arXiv preprint

GPT-4o System Card. arXiv preprint . ArXiv:2410.21276. Rui Pan, Tuan Dung Nguyen, Hardik Arora, and et al

arXiv

[6] [7]

arXiv preprint

AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy. arXiv preprint. ArXiv:2409.19750. Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben allal, and et al. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv preprint. ArXiv:2406.17557. Jeffrey Pennington, Richard Socher, and Christopher Manning...

arXiv 2024

[7] [8]

Score : X

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv preprint. ArXiv:1910.10683. Karan Singhal, Shekoofeh Azizi, Tao Tu, and et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172–180. Yuan-Sen Ting, Tuan Dung Nguyen, Tirthankar Ghosal, and et al. 2024. AstroMLab 1: Who Wins Astronomy Jeopar...

arXiv 1910

[8] [11]

Read and Assess : Carefully review the re- sponses generated by the three models for each test question

work page