Pith. sign in

REVIEW 4 major objections 4 minor 42 references

A data-centric multi-agent system can automate the full scientific research loop from raw data to paper, and a compact open-weights model rivals much larger proprietary systems.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:36 UTC pith:GXVWYFRF

load-bearing objection Credible orchestration, but the OpenSciDER numbers need a contamination audit before you trust them. the 4 major comments →

arxiv 2603.01421 v3 pith:GXVWYFRF submitted 2026-03-02 cs.AI cs.CL

SciDER: Scientific Data-centric End-to-end Researcher

Ke Lin , Owais Aijaz , Yilin Lu , Yiyang Luo , Xuehang Guo , Preslav Nakov This is my paper
classification cs.AI cs.CL
keywords autonomous scientific agentsdata-centric AImulti-agent systemsevolutionary idea searchopen-weights LLMscientific discoverymultimodal skill systemend-to-end research automation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SciDER argues that the main bottleneck in AI research agents is not reasoning but grounding: hypothesis generation, data analysis, experimentation, and critique all need to be tied to the actual raw data being studied. The paper builds a four-agent system that first structures raw files into a formal data report, then synthesizes executable code from that report, with an evolutionary idea search to propose hypotheses and a critic loop for refinement. It also releases an 8K-trajectory dataset and a 27-billion-parameter open-weights model fine-tuned on it. Across six benchmarks spanning idea generation, data analysis, ML engineering, scientific coding, end-to-end research, and multimodal visualization, the system reports competitive or leading results, with the open-weights variant beating several proprietary-model baselines. If these claims hold, a single modest open system could execute a meaningful fraction of the data-to-paper workflow in specialized scientific domains.

Core claim

SciDER's central claim is that automated scientific research should be organized around the data first: before hypotheses are finalized or code is written, a dedicated analysis agent parses the raw file tree into a structured report covering schemas, quality issues, field semantics, and dependencies, and that report grounds both idea generation and experiment code synthesis. The paper argues that this data-centric orchestration, combined with an evolutionary idea search that improves candidate hypotheses via rank-based crossover and mutation, and a critic loop that patches outputs against concrete revision instructions, lets one integrated multi-agent system cover the whole research lifecycl

What carries the argument

The load-bearing object is the structured data report R = (R_struct, R_qual, R_sem, R_dep): a typed schema and statistical fingerprint for each file, field-level quality metrics, query-conditioned semantic roles, and a dependency graph among fields. This report is what turns raw files into a form that can ground executable code, so the experiment agent addresses fields by role rather than by column name. Two supporting mechanisms carry the rest: Evolutionary Idea Search (EIS) converts an LLM judge's noisy absolute scores into K-way rank permutations across novelty, feasibility, impact, and specificity, then applies improve-and-combine operators to evolve a seed pool; and a multimodal skill s

Load-bearing premise

Everything hinges on the assertion that the 8K trajectories used to fine-tune OpenSciDER-27B are truly disjoint from the six evaluation benchmarks; the paper states the separation but gives no verifiable procedure or artifact, so a single leak would inflate the reported gains.

What would settle it

Search the released OpenSciDER-SFT-8K trajectories for near-duplicate problem statements, code, or data file names from the six evaluation benchmarks (DiscoveryBench, MLE-Bench, SciCode, AIRS-Bench, AstroVisBench, AI-Idea-Bench). If any evaluation instance matches a training trajectory even after normalization, the benchmark numbers are contaminated. Alternatively, rerun the OpenSciDER-27B agent on a fresh, non-overlapping scientific benchmark and compare its accuracy to the reported six-benchmark scores.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the results transfer beyond the six benchmarks, a single 27B open-weights model can carry out a meaningful fraction of the data-to-paper loop, lowering the infrastructure cost of autonomous research.
  • Grounding code synthesis in a structured data report—rather than in raw file names—appears to be a transferable design principle for any agent operating on unfamiliar, domain-specific data.
  • The OpenSciDER-SFT-8K trajectory dataset, if clean of benchmark overlap, becomes a reusable resource for instruction-tuning other open models toward scientific agent behavior.
  • Rank-based evolutionary search over LLM-judged ideas offers a practical response to the calibration noise of absolute LLM scores, making idea selection more reliable under a fixed call budget.
  • The iterative critic-revision loop, with concrete patch instructions and an approval subagent, provides a template for keeping long-horizon agentic runs stable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: if a future replay found evaluation instances inside the training trajectories despite the paper's separation claim, the reported gains would be inflated; that claim currently rests on the authors' assertion rather than on a published artifact or verifiable procedure.
  • Editorial: the appendix's generated manuscript, 'EvidentialHyperMAML,' carries its own limitations section (single dataset, no controlled ablations); the system can emit self-critical text, but the showcase artifact is not itself a fully verified scientific contribution.
  • Editorial: the same EIS ranking scheme could be lifted out of scientific ideation and applied to any small-pool selection problem where LLM judges are unreliable, such as hyperparameter search or dataset selection.
  • Editorial: a natural testable extension is to measure whether the structured data report alone—without the full agent loop—improves code accuracy on unseen raw-data tasks; that would isolate the data-centric effect the paper attributes to the whole pipeline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents SciDER, a multi-agent framework for end-to-end scientific research that couples ideation, data analysis, experimentation, and criticism, with a data-centric design and a dynamic multimodal skill system. The authors also release OpenSciDER-SFT-8K, a fine-tuning trajectory dataset, and OpenSciDER-27B, a LoRA-tuned Qwen-3.6-27B model. The central empirical claim is that across six benchmarks (AI-Idea-Bench, DiscoveryBench, MLE-Bench, SciCode, AIRS-Bench, AstroVisBench), SciDER and OpenSciDER achieve competitive or leading results, particularly on data-centric analysis, end-to-end research execution, and multimodal visualization. The evaluation compares against strong baselines, with reported headline gains such as 63.22 novelty on AI-Idea-Bench, 63.64% any-medal on MLE-Bench Lite, 0.780 on AIRS-Bench, and 38.6% correct on AstroVisBench. The paper additionally includes ablations of the Evolutionary Idea Search and the data-analysis module, qualitative expert feedback, and two case studies.

Significance. If the empirical claims hold, this is a useful contribution to the growing area of autonomous scientific agents. The framework's open release of both a trajectory dataset and a fine-tuned 27B model is a concrete community asset, and the formalization of the pipeline components (file-tree probing, dependency graph, skill system) is a step toward reproducible agent design. The central idea—grounding experiment generation in an explicit data-analysis report—is plausible and worth testing. However, the paper's strongest headline numbers come from OpenSciDER, the fine-tuned variant, and several of those numbers rely on unverified disjointness between the training trajectories and the evaluation benchmarks. The evaluation also relies heavily on leaderboard-sourced baselines and lacks error bars or shared-harness comparisons. These issues are load-bearing for the 'competitive or leading results' claim, though they appear addressable with additional transparency, re-runs, and sensitivity analyses. The ablations, while suggestive, currently validate the ideation module on a metric it directly optimizes, which weakens the causal interpretation.

major comments (4)
  1. [§III-C (OpenSciDER) and Tables III–VI] The claim that OpenSciDER's training trajectories are disjoint from the Section IV evaluation benchmarks is unsupported. The text states that the curators 'confirmed that no evaluation trajectories from Section IV were used in training by separating them into distinct subsets,' but it gives no benchmark list, no deduplication procedure, no hash or overlap analysis, and no released audit artifact. This is especially concerning because the training corpus is assembled from public benchmark trajectories, including 5,854 'successful benchmark trajectories for future use.' If any DiscoveryBench, MLE-Bench, SciCode, AIRS-Bench, AstroVisBench, or AI-Idea-Bench trajectory leaked into the 8,532 SFT examples, the OpenSciDER gains (e.g., 63.22 novelty in Table III, 38.6% correct in Fig. 3, 38.07% SciCode sub-problem success in Fig. 2) would partly reflect memorization rather than generalization. Th
  2. [§IV-A and Tables III–VI, Fig. 2] The evaluation compares against baselines 'sourced directly from their benchmarks and leaderboards' rather than re-run in a shared harness. This makes the claimed margins difficult to interpret. For example, Table V reports SciDER (Gemini-3-Pro) at 63.64% vs. AIRA (Gemini-3-Pro) at 56.89%, and Table VI reports SciDER at 0.780 vs. AIRA (GPT-5) at 0.550; if the baseline numbers come from different API versions, decoding strategies, or sampling budgets, the comparisons are not controlled. No error bars, standard deviations, or significance tests are reported for any main benchmark. At minimum, the key comparisons that support the headline claims should be re-run under the same harness and evaluation protocol, and the variance across runs or seeds should be reported.
  3. [§IV-G(a), Algorithm 1, Fig. 4(a)] The EIS ablation measures the lift in the same LLM-judge composite score that EIS is designed to optimize. The weights w = (0.30, 0.25, 0.25, 0.20) are hand-set, and no sensitivity or robustness analysis is provided. Because the selection procedure directly targets this composite, the observed Δ (e.g., +23.8% novelty) could reflect optimization on the evaluation metric rather than genuinely better research ideas. To support the claim that EIS is 'the critical driver,' the authors should evaluate the final ideas with an external or independent metric (e.g., a different judge, or the benchmark's own scoring protocol) and test robustness to the composite weights and ranking method.
  4. [§IV-G(b), Fig. 4(b)] The data-analysis ablation uses only two DiscoveryBench tasks (ML Req. Eng. and Meta-Regression) and reports a single score for each condition without error bars or statistical comparison. Given that the 'data-centric' design is a central novelty of the paper, the ablation is too thin to establish that the data-analysis module is the cause of the DiscoveryBench improvements shown in Table IV. A broader ablation across the full benchmark, or at least across more tasks with repeated runs, is needed.
minor comments (4)
  1. [Table III] Table III has no model column. The text says baseline results are 'reported using Gemini-2.5-Pro as well as SciDER,' but it is unclear which backbone is used for the SciDER and OpenSciDER rows. Please specify the model for each row, and make the naming consistent (OpenSciDER in Table III vs. SciDER (OpenSciDER-27B) elsewhere).
  2. [Table VI] The sentence 'The results are collected within five different benchmark iterations' is unclear. Does AIRS-Bench define five iterations, or are these repeated runs? Please clarify and report the per-iteration scores or variance.
  3. [§IV-H] The qualitative feedback section reports a mean score of 4.846/5.000 from 13 self-selected domain experts, but there is no description of how the experts were recruited, whether they were independent of the authors, or what tasks they attempted. This limits the generalizability of the 'practical utility' claim.
  4. [Fig. 2] The figure is hard to parse because the bars are not labeled in a readable order and the values in the text (e.g., 11.44% margin, 31.97% relative improvement) are not directly mapped to the bars. Please add a legend or reformat the figure so each bar can be identified with its method and metric.

Circularity Check

1 steps flagged

Partial circularity in the EIS ablation: the reported lift is measured with the same internal LLM-judge composite that EIS optimizes; the six-benchmark results remain externally grounded.

specific steps
  1. self definitional [Section III-A (S(x) formula / Algorithm 1) and Section IV-G-a / Fig. 4(a)]
    "EIS addresses this by prompting judges to rank K-way batches based on novelty, feasibility, impact, and specificity. These ranks are converted into a weighted composite score ... Ranks are converted to normalised scores score_d(x) = (n+ 1−rank_d(x))/n and combined into S(x) =P_d w_d ·score_d(x), with weights w = (0.30,0.25,0.25,0.20). ... Because all dimensions are scored simultaneously, we log the initial best composite score (σ⋆0) from the literature-conditioned seed pool as our natural baseline. We then measure the absolute lift (∆ =σ⋆T −σ⋆0) achieved after T evolutionary generations."

    The ablation's Δ = σ*_T − σ*_0 uses the same σ produced by BATCHRANK in Algorithm 1, i.e., the weighted LLM-judge composite S(x)=Σ_d w_d·score_d(x). Algorithm 1 selects survivors and the final output by arg max σ(x), so the 'lift' measures the optimizer's own fitness before and after optimization. Reporting this as evidence that 'EIS is the critical driver' of quality/novelty/feasibility makes the gain definitional: the quantity being optimized is also the quantity being reported as the outcome. If Fig. 4(a) instead reports external AI-Idea-Bench scores, the section's notation and composite-score language do not say so, and no external-scoring artifact is given.

full rationale

The paper's core benchmark claims are externally grounded: Tables III–VI and Figs. 2–3 report scores on AI-Idea-Bench, DiscoveryBench, MLE-Bench, SciCode, AIRS-Bench, and AstroVisBench, with benchmark implementations/leaderboards cited as external sources. There is no fitted parameter renamed as a benchmark result. The OpenSciDER contamination concern is a real empirical validity risk, not definitional circularity: the paper asserts separation of Section IV trajectories from training without an audit artifact, but no equation shows evaluation trajectories entering the training objective. The one clear construction-equivalence issue is the EIS ablation, where the reported improvement is written as the same internal selection score that EIS is designed to maximize; that is partial circularity in a supporting claim, not in the central six-benchmark comparison. No load-bearing self-citation or imported uniqueness theorem appears, and the qualitative/critic loops are not used as the primary external evidence.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central benchmark results depend on hand-set EIS weights, default algorithm budgets, data-leakage separation, and comparability of external leaderboard baselines. No derived physical or mathematical constants are involved; the ledger is dominated by hyperparameters and unverified domain assumptions.

free parameters (5)
  • EIS ranking weights = w=(0.30, 0.25, 0.25, 0.20)
    Hand-set weights for novelty, feasibility, impact, and specificity; no sensitivity analysis; they define the composite score that drives EIS selection and the ablation curves.
  • EIS population and budget defaults = n=8, k=4, T=3, budget=60, n_imp/n_com=3/1
    Algorithm defaults determine search depth and breadth; no ablation is reported for these choices.
  • Dependency graph overlap threshold = tunable (unspecified)
    Section III-A-b adds edges for value-set overlap "above a tunable threshold"; the threshold is not reported, affecting the data report structure.
  • Critic retry budget = N_max=2
    Per-agent revision budget; interacts with critic verdict gating and could affect final outputs.
  • OpenSciDER training hyperparameters = LoRA r=32, alpha=64, lr=1e-4, epochs=2, BF16
    Chosen for the 27B fine-tune; direct effect on model quality; no sweeps are reported.
axioms (4)
  • domain assumption LLM-as-judge rank signals are stable and valid enough to evolve research ideas
    EIS's entire search and its ablation rely on prompt-based K-way rankings; no human-validated reliability analysis is provided.
  • ad hoc to paper OpenSciDER-SFT-8K training trajectories do not overlap evaluation tasks
    Authors assert separation in Section III-C but give no checksums, task lists, or contamination tests.
  • domain assumption Leaderboard baseline numbers are directly comparable to this paper's runs
    Section IV-A says results for other methods are "sourced directly from their benchmarks and leaderboards" without a shared harness.
  • domain assumption SFT on benchmark-derived trajectories transfers to held-out research workflows
    Used to claim OpenSciDER's gains on AIRS-Bench and AstroVisBench; no generalization analysis is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 19650 in / 11950 out tokens · 113182 ms · 2026-08-02T19:36:59.029019+00:00 · methodology

0 comments
read the original abstract

While large language models accelerate scientific discovery, existing agents face severe limitations in adaptability, domain generalization, and multimodal scalability, often struggling to autonomously process raw, domain-specific experimental data. To overcome these barriers, we introduce SciDER, a multi-agent system designed to flexibly automate the entire research lifecycle. This framework employs a novel data-centric approach and integrates a dynamic multimodal skill system across four specialized sub-agents. Specifically, an ideation agent generates novel hypotheses via Evolutionary Idea Search, a data analysis agent systematically structures raw data, an experimentation agent synthesizes executable code grounded in dataset characteristics, and a critic agent drives iterative self-refinement. To democratize open-source scientific discovery, we release OpenSciDER-SFT-8K, a high-quality execution trajectory dataset, alongside the OpenSciDER-27B fine-tuned model. Across six benchmarks, SciDER and OpenSciDER obtain competitive or leading results, with especially strong gains on data-centric analysis, end-to-end research execution, and multimodal scientific visualization. By integrating data analysis with experimental execution, SciDER bridges the gap between abstract scientific reasoning and reproducible experimentation synthesis.

Figures

Figures reproduced from arXiv: 2603.01421 by Ke Lin, Owais Aijaz, Preslav Nakov, Xuehang Guo, Yilin Lu, Yiyang Luo.

Figure 1
Figure 1. Figure 1: The system architecture of SciDER. The pipeline automates the research lifecycle through four multimodal specialized agents: Ideation, Data Analysis, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Results of SciCode. Solve rates for main and subproblems are reported, with higher rates indicating greater domain-specific capability. that our iterative trial-critic-revision cycles ensure highly ro￾bust execution across diverse scientific domains. E. End-to-end Research We evaluated our framework’s cross-domain, end-to-end re￾search capabilities on AIRS-Bench [19], comparing it against standard ReAct [4… view at source ↗
Figure 3
Figure 3. Figure 3: illustrates a compelling dynamic in handling out￾of-distribution scientific tasks. Notably, OpenSciDER signifi￾cantly outperforms both the standard multimodal ReAct agent and, remarkably, the closed-source-backed SciDER frame￾work. While frontier proprietary models often struggle with the highly specialized, long-tail API syntax required in astro￾physics, our open-weights OpenSciDER successfully leverages … view at source ↗
Figure 4
Figure 4. Figure 4: Ablation Study of EIS and Data Analysis: (a) Impact of EIS on [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 2
Figure 2. Figure 2: AUC-ROC (higher is better) and ECE (lower [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Selected generated pages showcasing end-to-end scientific discovery evidence for [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative generated examples from AstroVisBench. Utilizing an iterative multimodal feedback loop, SciDER successfully synthesizes and refines complex astronomical visualizations, accurately handling long-tail APIs and domain-specific formatting. [11] A. D. White, G. M. Hocky, H. A. Gandhi, M. Ansari, S. Cox, G. P. Wellawatte, S. Sasmal, Z. Yang, K. Liu, Y. Singh, and W. J. Pena Ccoa, ˜ “Assessment of che… view at source ↗
Figure 7
Figure 7. Figure 7: Per-task scores of our method on DiscoveryBench. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Per-task scores of our method on AIRS-Bench. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 9 linked inside Pith

  1. [11]

    Assessment of chemistry knowledge in large language models that generate code,

    A. D. White, G. M. Hocky, H. A. Gandhi, M. Ansari, S. Cox, G. P. Wellawatte, S. Sasmal, Z. Yang, K. Liu, Y . Singh, and W. J. Pe˜na Ccoa, “Assessment of chemistry knowledge in large language models that generate code,”Digital Discovery, vol. 2, pp. 368–376, 2023. [Online]. Available: http://dx.doi.org/10.1039/D2DD00087C

    doi:10.1039/d2dd00087c 2023

  2. [12]

    Scicode: a research coding benchmark curated by scientists,

    M. Tian, L. Gao, S. D. Zhang, X. Chen, C. Fan, X. Guo, R. Haas et al., “Scicode: a research coding benchmark curated by scientists,” in Proceedings of the 38th International Conference on Neural Information Processing Systems, ser. NIPS ’24. Red Hook, NY , USA: Curran Associates Inc., 2024

    2024

  3. [13]

    Agent laboratory: Using LLM agents as research assistants,

    S. Schmidgall, Y . Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, M. Moor, Z. Liu, and E. Barsoum, “Agent laboratory: Using LLM agents as research assistants,” inFindings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov...

    2025

  4. [14]

    TinyScientist: An interactive, extensible, and controllable framework for building research agents,

    H. Yu, K. Xuan, F. Li, K. Zhu, Z. Lei, J. Zhang, Z. Qi, K. Richardson, and J. You, “TinyScientist: An interactive, extensible, and controllable framework for building research agents,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, I. Habernal, P. Schulam, and J. Tiedemann, Eds. Suzhou, Chi...

    2025

  5. [15]

    Internagent: When agent becomes the scientist–building closed-loop system from hypothesis to verification,

    I. Team, B. Zhang, S. Feng, X. Yan, J. Yuan, R. Ma, Y . Huet al., “Internagent: When agent becomes the scientist–building closed-loop system from hypothesis to verification,” 2025. [Online]. Available: https://arxiv.org/abs/2505.16938

    Pith/arXiv arXiv 2025

  6. [16]

    Internagent-1.5: A unified agentic framework for long- horizon autonomous scientific discovery,

    S. Feng, R. Ma, X. Yan, Y . Fan, Y . Hu, S. Huang, S. Zhang et al., “Internagent-1.5: A unified agentic framework for long- horizon autonomous scientific discovery,” 2026. [Online]. Available: https://arxiv.org/abs/2602.08990

    arXiv 2026

  7. [17]

    AI idea bench 2025: AI research idea generation benchmark,

    Y . Qiu, H. Zhang, Z. Xu, M. Li, D. Song, Z. Wang, and K. Zhang, “AI idea bench 2025: AI research idea generation benchmark,” 2025. [Online]. Available: https://arxiv.org/abs/2504.14191

    Pith/arXiv arXiv 2025

  8. [18]

    DiscoveryBench: Towards data-driven discovery with large language models,

    B. P. Majumder, H. Surana, D. Agarwal, B. D. Mishra, A. Meena, A. Prakhar, T. V ora, T. Khot, A. Sabharwal, and P. Clark, “DiscoveryBench: Towards data-driven discovery with large language models,” inInternational Conference on Learning Representations,

  9. [19]

    AIRS-Bench: A suite of tasks for frontier AI research science agents,

    A. Lupidi, B. Gauri, T. S. Foster, B. Al Omari, D. Magka, A. Pepe, A. Audran-Reisset al., “AIRS-Bench: A suite of tasks for frontier AI research science agents,” 2026. [Online]. Available: https://arxiv.org/abs/2602.06855

    arXiv 2026

  10. [20]

    AstroVisBench: A code benchmark for scientific computing and visualization in astronomy,

    S. Joseph, S. M. Husain, S. Offner, S. Juneau, P. Torrey, A. Bolton, J. Farias, N. Gaffney, G. Durrett, and J. J. Li, “AstroVisBench: A code benchmark for scientific computing and visualization in astronomy,” inAdvances in Neural Information Processing Systems, vol. 38, 2025. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2025/hash...

    2025

  11. [21]

    arxiv.org e-print archive,

    arXiv, “arxiv.org e-print archive,” https://arxiv.org, 2026, accessed: 2026-05-23. [Online]. Available: https://arxiv.org

    2026

  12. [22]

    Semantic scholar,

    Allen Institute for AI, “Semantic scholar,” https://www.semanticscholar. org, 2026, accessed: 2026-05-23. [Online]. Available: https://www. semanticscholar.org

    2026

  13. [23]

    [Online]

    National Center for Biotechnology Information, “Pubmed,” https: //pubmed.ncbi.nlm.nih.gov, 2026, accessed: 2026-05-23. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov

    2026

  14. [24]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS ’23. Red Hook, NY , USA: Curran Associates Inc., 2023

    2023

  15. [25]

    Teaching large language models to self-debug,

    X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou, “Teaching large language models to self-debug,” inInternational Conference on Learning Repre- sentations, 2024

    2024

  16. [26]

    ToolLLM: Facilitating large language models to master 16000+ real-world APIs,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qianet al., “ToolLLM: Facilitating large language models to master 16000+ real-world APIs,” inInternational Conference on Learning Representations, 2024

    2024

  17. [27]

    Paperorchestra: A multi-agent framework for automated ai research paper writing,

    Y . Song, Y . Song, T. Pfister, and J. Yoon, “Paperorchestra: A multi-agent framework for automated ai research paper writing,” 2026. [Online]. Available: https://arxiv.org/abs/2604.05018

    Pith/arXiv arXiv 2026

  18. [28]

    Datascibench: An llm agent benchmark for data science,

    D. Zhang, S. Zhoubian, M. Cai, F. Li, L. Yang, W. Wang, T. Dong, Z. Hu, J. Tang, and Y . Yue, “Datascibench: An llm agent benchmark for data science,” 2025. [Online]. Available: https://arxiv.org/abs/2502.13897

    Pith/arXiv arXiv 2025

  19. [29]

    Ds-1000: A natural and reliable benchmark for data science code generation,

    Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.- t. Yih, D. Fried, S. Wang, and T. Yu, “Ds-1000: A natural and reliable benchmark for data science code generation,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 18 319–18 345

    2023

  20. [30]

    Dsbench: How far are data science agents from becoming data science experts?

    L. Jing, Z. Huang, X. Wang, W. Yao, W. Yu, K. Ma, H. Zhang, X. Du, and D. Yu, “Dsbench: How far are data science agents from becoming data science experts?” inInternational Conference on Learning Repre- sentations, 2025

    2025

  21. [31]

    Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery,

    Z. Chen, S. Chen, Y . Ning, Q. Zhang, B. Wang, B. Yu, Y . Liet al., “Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery,” inInternational Conference on Learning Representations, 2025

    2025

  22. [32]

    Qwen3.6,

    Qwen Team, “Qwen3.6,” https://github.com/QwenLM/Qwen3.6, Apr. 2026, open-weight Qwen3.6-27B; accessed: 2026-06-03

    2026

  23. [33]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2022

    2022

  24. [34]

    Zero: Memory optimizations toward training trillion parameter models,

    S. Rajbhandari, J. Rasley, O. Ruwase, and Y . He, “Zero: Memory optimizations toward training trillion parameter models,” inSC20: international conference for high performance computing, networking, storage and analysis. IEEE, 2020, pp. 1–16

    2020

  25. [35]

    Yarn: Efficient context window extension of large language models,

    B. Peng, J. Quesnelle, H. Fan, and E. Shippole, “Yarn: Efficient context window extension of large language models,” inInternational Conference on Learning Representations, 2024

    2024

  26. [36]

    SciPIP: An LLM-based scientific paper idea proposer,

    W. Wang, L. R. Gu, L. Zhang, Y . Luo, Y . Dai, C. Shen, L. Xie, B. Lin, X. He, and J. Ye, “SciPIP: An LLM-based scientific paper idea proposer,” 2024. [Online]. Available: https://arxiv.org/abs/2410.23166

    Pith/arXiv arXiv 2024

  27. [37]

    Many heads are better than one: Improved scientific idea generation by a LLM-based multi-agent system,

    H. Su, R. Chen, S. Tang, Z. Yin, X. Zheng, J. Liet al., “Many heads are better than one: Improved scientific idea generation by a LLM-based multi-agent system,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics, Jul. 2025, pp. 28 201...

    2025

  28. [38]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosenet al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” 2025. [Online]. Available: https://arxiv.org/abs/2507.06261

    Pith/arXiv arXiv 2025

  29. [39]

    Data-driven discovery with large generative models,

    B. P. Majumder, H. Surana, D. Agarwal, S. Hazra, A. Sabharwal, and P. Clark, “Data-driven discovery with large generative models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.13610

    Pith/arXiv arXiv 2024

  30. [40]

    Asta agents: AI tools for scientific research,

    Allen Institute for AI, “Asta agents: AI tools for scientific research,” https://allenai.org/asta/agents, 2026, accessed: 2026-05-24. [Online]. Available: https://allenai.org/asta/agents

    2026

  31. [41]

    System card: Claude opus 4 & claude sonnet 4,

    Anthropic, “System card: Claude opus 4 & claude sonnet 4,” Anthropic, Tech. Rep., May 2025, accessed: 2026-05-23. [Online]. Available: https://www-cdn.anthropic.com/ 6d8a8055020700718b0c49369f60816ba2a7c285.pdf

    2025

  32. [42]

    AI research agents for machine learning: Search, exploration, and generalization in MLE-bench,

    E. Toledo, K. Hambardzumyan, M. Josifoski, R. Hazra, N. Baldwin, A. Audran-Reiss, M. Kuchniket al., “AI research agents for machine learning: Search, exploration, and generalization in MLE-bench,” in Advances in Neural Information Processing Systems, vol. 38, 2025. [On- line]. Available: https://proceedings.neurips.cc/paper files/paper/2025/ hash/328b8188...

    2025

  33. [43]

    ML-Master: Towards AI-for-AI via integration of exploration and reasoning,

    Z. Liu, Y . Cai, X. Zhu, Y . Zheng, R. Chen, Y . Wen, Y . Wang, W. E, and S. Chen, “ML-Master: Towards AI-for-AI via integration of exploration and reasoning,” 2025. [Online]. Available: https: //arxiv.org/abs/2506.16499

    Pith/arXiv arXiv 2025

  34. [44]

    AIDE: AI-driven exploration in the space of code,

    Z. Jiang, D. Schmidt, D. Srikanth, D. Xu, I. Kaplan, D. Jacenko, and Y . Wu, “AIDE: AI-driven exploration in the space of code,” 2025. [Online]. Available: https://arxiv.org/abs/2502.13138

    Pith/arXiv arXiv 2025

  35. [45]

    Gemini 3 pro model card,

    Google DeepMind, “Gemini 3 pro model card,” Google DeepMind, Tech. Rep., Nov. 2025, accessed: 2026-06-03. [Online]. Available: https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-Pro-Model-Card.pdf

    2025

  36. [46]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations, 2022

    2022

  37. [47]

    Addressing the assessment challenge with an online system that tutors as it assesses,

    M. Feng, N. Heffernan, and K. Koedinger, “Addressing the assessment challenge with an online system that tutors as it assesses,”User modeling and user-adapted interaction, vol. 19, no. 3, pp. 243–266,

  38. [2009]

    Available: https://doi.org/10.1007/s11257-009-9063-7 APPENDIX 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Score Archaeology Non-native Plants Meta-Reg

    [Online]. Available: https://doi.org/10.1007/s11257-009-9063-7 APPENDIX 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Score Archaeology Non-native Plants Meta-Reg. (Raw) NLS Incarceration NLS SES NLS (Raw) Meta-Regression ML Req. Eng. Average 0.248 0.253 0.258 0.290 0.321 0.346 0.482 0.635 0.227 0.174 0.274 0.357 0.312 0.190 0.397 0.586 0.354 0.315 SciDER (Claude-Sonne...

    doi:10.1007/s11257-009-9063-7

  39. [2018]

    Student-Conditioned Evidential Meta-Learning with Hyper- Conditioned Priors for Calibrated Few-Shot Knowledge Tracing

    On first-order meta-learning algorithms.arXiv preprint. Shalini Pandey and George Karypis. 2019. A self- attentive model for knowledge tracing.arXiv preprint. Shalini Pandey and Jaideep Srivastava. 2020. Rkt : Relation-aware self-attention for knowledge tracing. arXiv preprint. Chris Piech, Jonathan Spencer, Jonathan Huang, Surya Ganguli, Mehran Sahami, L...

    2019

  40. [2024]

    arXiv preprint

    Learning to love edge cases in formative math assessment: Using the ammore dataset and chain- of-thought prompting to improve grading accuracy. arXiv preprint. Fucai Ke, Weiqing Wang, Weicong Tan, Lan Du, Yuan Jin, Yujin Huang, and Hongzhi Yin. 2022. Hitskt: A hierarchical transformer model for session-aware knowledge tracing.arXiv preprint. Hyeondey Kim,...

    2022

  41. [2025]

    Available: https://openreview.net/forum?id=vyflgpwfJW

    [Online]. Available: https://openreview.net/forum?id=vyflgpwfJW

  42. [2026]

    Yu Lu, Deliang Wang, Qinggang Meng, and Penghe Chen

    Circuit complexity of hierarchical knowledge tracing and implications for log-precision transform- ers.arXiv preprint. Yu Lu, Deliang Wang, Qinggang Meng, and Penghe Chen. 2020. Towards interpretable deep learning models for knowledge tracing.arXiv preprint. Alex Nichol, Joshua Achiam, and John Schulman

    2020