Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

The paper claims that an LLM-agent pipeline combining heterogeneous online evidence retrieval, multi-persona deliberation, and decoupled multi-criteria scoring can reproduce human-level research idea evaluation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-02 23:13 UTC pith:VGJ74ZUG

load-bearing objection A well-built, genuinely useful idea-evaluation framework whose headline gains are undermined by an unaddressed temporal-leakage problem: the system searches with a default 'now' timestamp, so it can see the target paper and its post-acceptance record. the 4 major comments →

arxiv 2602.14367 v2 pith:VGJ74ZUG submitted 2026-02-16 cs.CL cs.AIcs.IRcs.LG

InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

Shuofei Qiao , Yunxiang Wei , Xuehai Wang , Bin Wu , Boyang Xue , Ningyu Zhang , Hossein A. Rahmani , Yanshan Wang
show 5 more authors
Qiang Zhang Keyan Ding Jeff Z. Pan Huajun Chen Emine Yilmaz
This is my paper
classification cs.CL cs.AIcs.IRcs.LG
keywords research idea evaluationLLM-as-a-judgemulti-perspective reasoningknowledge groundinginnovation review boardmulti-criteria decision makingretrieval-augmented evaluationscientific discovery
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that evaluating a research idea should be treated as a knowledge-grounded, multi-perspective reasoning problem rather than a single LLM judgment. To support that view, it builds InnoEval, a pipeline that searches literature, web discussion, and code repositories, grounds each part of the idea in retrieved evidence, convenes a board of simulated reviewers with distinct academic personas, and scores the idea separately on clarity, novelty, feasibility, validity, and significance before synthesizing a meta-review with a decision and revision suggestions. On datasets built from stratified samples of real conference submissions, the paper reports that InnoEval outperforms all baselines on single-idea classification, pairwise comparison, and group ranking, with three-class F1 of 74.56%, roughly 5 percentage points higher pairwise accuracy, and 7.56 percentage points higher group-ranking accuracy. The paper also reports that its scores correlate with human expert and peer-review judgments on all five dimensions and that its evaluation reports win over 70% of head-to-head comparisons on overall quality. A sympathetic reader would care because, if the claim holds, automated idea assessment can move toward an evidence-cited, consensus-based assistant for scientific review and research prioritization.

Core claim

The central discovery is that a fully agentic evaluation system—one that actively searches heterogeneous online sources, extracts fine-grained supporting or contradicting evidence, and lets multiple persona-based reviewers score each criterion independently before an area-chair-style meta-review—matches or exceeds both simpler LLM judges and specialized idea-evaluation systems. This is demonstrated on a new benchmark derived from real peer-reviewed submissions, where InnoEval outperforms the strongest baseline by 16.18% F1 on three-class point-wise prediction, by roughly 5% accuracy on pairwise comparison, and by 7.56% accuracy on group-wise ranking. The paper further claims its scores corre

What carries the argument

The carrying mechanism is a three-stage pipeline. First, a heterogeneous deep knowledge search engine iteratively refines queries and combines semantic similarity with LLM relevance scoring to select top evidence from literature, web, and code repositories. Second, a grounding agent aligns each structured part of the idea with atomized evidence that supports or contradicts it. Third, an innovation review board—a named central object defined as a pool of synthetic academic personas with different expertise and masking of accessible knowledge—feeds five dimension-specialized evaluators (clarity, novelty, feasibility, validity, significance), whose independent scores and narratives are synthesi

Load-bearing premise

The benchmark's ground truth is the acceptance decisions of two major machine-learning conferences, so the central claim would give way if those decisions mostly reflect reviewer bias, writing quality, or noise rather than the quality of the ideas themselves.

What would settle it

Apply InnoEval to a set of historical submissions with known long-term impact—for example, papers rejected by one venue that later became highly cited—and check whether high InnoEval scores predict eventual impact as well as or better than the original acceptance decisions. If InnoEval's correlation with acceptance is high but its correlation with impact is near zero, the claim that it measures idea quality would be refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If InnoEval is right, automated single-idea evaluation can reach 74.56% macro F1 on a three-way Reject/Poster/Highlight prediction task, where most baselines collapse onto one or two labels.
  • Pairwise idea comparison gains about 5 percentage points in accuracy, and group-wise ranking accuracy gains about 7.56 percentage points, suggesting the approach can support practical triage and ranking of research proposals.
  • The reports include cited evidence and concrete revision suggestions, and integrating the evaluation into an iterative idea-generation loop measurably improves the quality of generated ideas.
  • Ablations show the performance depends on all three design choices—fine-grained grounding, persona-based review consensus, and heterogeneous web/code/literature search—rather than on any one component alone.
  • Increasing the number of distinct personas improves accuracy in a scaling-like pattern, and personalized multi-reviewer consensus outperforms plain test-time scaling with a single reviewer.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the paper treats persona diversity as a scaling axis, a testable extension is to vary the composition of the persona pool—disciplinary mix, seniority, or methodological preferences—and measure whether it changes not just accuracy but the direction of errors.
  • The time-stamped evaluation design could be repurposed as a retrospective test: evaluate historical proposals at their original timestamp and compare InnoEval's verdicts with later real-world impact, which would be a stronger probe of whether the system measures genuine innovation rather than reviewer style.
  • If conference acceptance labels are noisy proxies, the paper's per-dimension scores offer a cleaner unit of analysis; one could use the novelty-versus-feasibility trade-off pattern as a signature of idea type for portfolio-level research allocation.
  • The evidence-grounding reports are themselves a potential product: the cited literature and code evidence surfaced during evaluation could be reused as a starting bibliography or related-work skeleton for the idea, an implication the paper does not develop.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper casts research-idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduces InnoEval, a pipeline with (i) a heterogeneous deep search engine over literature, web, and code; (ii) an innovation review board of persona-based reviewers; and (iii) multi-dimensional decoupled evaluation across Clarity, Novelty, Validity, Feasibility, and Significance. The authors construct benchmark datasets from NeurIPS 2025 and ICLR 2025 submissions: 217 point-wise samples, 372 pair-wise pairs, and 172 group-wise ranking instances. They report state-of-the-art results on all three settings (e.g., 75.74 macro-F1 for three-class point-wise prediction vs. 65.02 for the best baseline), qualitative win-rates above 70% against baselines, a 60-sample human study with correlations 0.55--0.78 on five dimensions, and ablations supporting grounding, personalization, and heterogeneous search.

Significance. If the benchmark and evaluation protocol are valid, this is a substantial engineering and empirical contribution: the framework is modular, the prompt design is unusually detailed and reproducible, and the task suite (point-wise, pair-wise, group-wise) is more comprehensive than most prior idea-evaluation work. The paper also makes a useful methodological statement: evaluation should be grounded in diverse evidence, deliberated by multiple perspectives, and decomposed into multiple criteria. However, the quantitative claims rest entirely on the validity of conference acceptance decisions as gold labels and on the search protocol's temporal controls. The current manuscript does not establish either, so the headline results should be treated as provisional.

major comments (4)
  1. [§3.1, Eq. (3); §4.1 / Appx. B; Appx. H.4] Temporal leakage is the most serious threat to the central quantitative claim. Section 2 states that each idea carries a timestamp t and 't will default to the latest time if not explicitly specified.' Appx. B describes dataset construction from NeurIPS25/ICLR25 papers but never sets t to the submission/review date. The search engine therefore queries the current web, arXiv, Semantic Scholar, GitHub, etc., and can retrieve the target paper itself, its OpenReview record, subsequent citations, and follow-up code. The novelty prompt in Appx. H.4 explicitly anticipates finding 'a preprint, repository, or website OF THIS EXACT IDEA' and instructs the reviewer not to penalize it, confirming that self-retrieval occurs. Since InnoEval is the only method that exploits web and code search, a large part of the reported gains over literature-only baselines may reflect hindsight about acceptance outc
  2. [§4.1, Table 1; Appx. B] The benchmark labels are conference acceptance decisions (Reject/Poster/Spotlight/Oral) from two venues and many different tracks. These are noisy, potentially biased, and not directly comparable across tracks. Moreover, no confidence intervals, bootstrap estimates, or significance tests are reported anywhere in Table 1 or the ablation figures. The point-wise dataset is only 217 samples, with extreme class imbalance (138 Reject, 66 Poster, 9 Spotlight, 4 Oral); macro-F1 on the three-class task is therefore highly sensitive to a handful of Highlight examples. The pair-wise and group-wise datasets are constructed by selecting one similar paper per label stratum, which may artificially amplify separability. The authors should report CIs or significance tests, and should justify or re-anchor the gold labels: for instance, by providing expert-scored labels on a held-out subset and showing tha
  3. [§5.1, Fig. 2; Appx. D] The claim that InnoEval's judgments are 'highly aligned with human experts' is based on 60 samples scored by five experts. No confidence intervals, per-expert agreement, or inter-rater reliability are reported. The 'peer-review judgments' are not direct scores but are extracted by an LLM from review texts, which introduces an additional measurement layer. The observed correlations (0.55--0.78) are moderate, and the statement 'highly aligned' overstates what can be concluded from n=60 with no uncertainty quantification. The authors should report bootstrap CIs, per-expert correlations, and ideally use held-out expert labels that were not already used to motivate the system design.
  4. [§5.3(d), Fig. 4] The metric-influence regression and the inter-metric pairwise correlations are computed on InnoEval's own predicted scores, and the paper then presents these as 'convergent evidence' that InnoEval 'can successfully capture the essence of idea evaluation.' This is not independent evidence of human alignment; it characterizes the internal scoring behavior of the model. For example, if InnoEval always rated Significance as a weighted average of Novelty and Validity, the correlations in Fig. 4 would emerge by construction. These analyses should be relabeled as descriptive properties of InnoEval, or be compared against the same correlations computed on human dimension scores before being used to support external validity.
minor comments (6)
  1. [Table 1] The column headers (Acc2, F12, Acc3, F13, Acceasy, Acchard, Best, Lis, Acc) are not defined in the main text; readers must infer them from Appx. B. A caption or a table note should spell out the binary/ternary split and the LIS definition.
  2. [§3.1 and Table 3] The search tool set in §3.1 and Table 3 is rendered as an empty tuple in the provided text. If this is not a PDF-extraction artifact, the notation needs to be completed; otherwise, the formal definition of Q_{p,u} is incomplete.
  3. [Appx. A] The limitations section lists discipline, efficiency, and modality, but does not acknowledge the two most consequential validity threats: temporal leakage in the search protocol and the use of noisy acceptance decisions as gold labels. These should be discussed explicitly.
  4. [Appx. D] The human study says experts were 'permitted to employ any search tools,' but it does not report whether they had access to the target paper's OpenReview page or acceptance decision. This matters because the same tools may have introduced hindsight into the human scores.
  5. [§2 / §3.4] The 'revision suggestions' use 'future information' about the idea (post-timestamp knowledge). This is a reasonable feature, but it should be clearly separated from the scoring protocol in the point-wise decision; otherwise the boundary between prospective evaluation and post-hoc advice is blurred.
  6. [General] The authors state they will open-source code and data, but no repository link or data access information appears in the manuscript. Given the centrality of the benchmark, providing the exact ideas, timestamps, and retrieval snapshots is important for reproducibility.

Circularity Check

0 steps flagged

No circularity: the central benchmark claim is grounded in external OpenReview acceptance labels and human judgments; no fitted parameter is renamed as a prediction.

full rationale

The paper's load-bearing chain is external. Ground-truth labels are conference decisions crawled from OpenReview (Appx. B: 'we crawl NeurIPS 2025 and ICLR 2025 papers from OpenReview... partitioned into four strata according to their final decisions'), and the reported gains (Tab. 1) measure prediction and ranking against those labels. InnoEval is not trained on the labels; its scoring rubrics map scores to decision bins (Appx. H.5) rather than consuming labels as inputs. Human evaluation compares against expert ratings on 60 samples (Appx. D), an independent external signal. Baselines share the backbone and some modules, but this is a controlled comparison, not a circular derivation. The self-citations in the paper (Qiao et al. 2023/2025, Ou et al. 2025) appear only in related work and are not load-bearing; no uniqueness theorem is invoked from the authors' prior work. The Sec. 5.3 'Metrics Influence' regression and inter-metric scatter plots are computed on InnoEval's own predicted scores, so they are weak internal validation when offered as 'convergent evidence' that the system captures the essence of idea evaluation; however, no prediction reduces to a fitted input and no central result depends on those analyses. The timestamp default ('t will default to the latest time if not explicitly specified', Sec. 2) combined with online search is a real temporal-leakage/validity threat, and the novelty prompt's 'Self-Discovery Check' confirms the search can surface the target paper; but leakage is an evaluation-protocol concern, not a definitional equivalence or fitted-input-as-prediction. Under the stated circularity criteria, no circular step can be exhibited.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The framework relies on a modest set of hand-set pipeline hyperparameters (α, m, N, five personas) and, more substantially, on the assumption that conference accept/reject labels are trustworthy ground truth. The most consequential ungrounded choice is the persona board: its biases and familiarity scores are authored by the paper's own team, so the 'consensus' is a construct of the system rather than an independent signal. No physical entities are invented; the only new artifacts are algorithmic modules and datasets.

free parameters (6)
  • Hybrid scoring weight α = 0.2
    Weight between semantic (S_sem) and LLM-judge (S_llm) relevance scores in Eq. (6). Chosen by hand; directly controls which knowledge pieces survive filtering.
  • Per-type retention count m = 10
    Maximum number of brief knowledge items retained for literature, web, and code separately (§3.1). Controls the amount of context available to reviewers.
  • Query refinement rounds N = 3
    Number of iterative search-and-refine cycles (§3.1). Balances search depth against cost; no validation split is reported.
  • Number of personas per idea = 5
    Randomly sampled from the 15-persona board (§3.3). Chosen for efficiency; authors state more personas improve performance but increase cost.
  • Persona base (15 profiles) = hand-authored profiles with familiarity scores (1–10), goals, constraints
    Personas define the simulated reviewer biases and knowledge masking rates (Appx. F). These hand-authored priors shape all scores.
  • Score distribution priors in prompts = e.g., 9–10 ≈10%, 7–8 ≈25%, 5–6 ≈45%, 0–4 ≈20%
    Each dimension-specific evaluation prompt instructs the LLM to follow an expected score distribution (Appx. H.4). This is an explicit prior forcing score spread.
axioms (6)
  • domain assumption Conference acceptance labels (Reject/Poster/Spotlight/Oral) are valid ground-truth measures of research idea quality.
    The benchmark labels all tasks and the F1/accuracy numbers are computed against these labels (§4.1). If these decisions are noisy or biased, the reported superiority may largely reflect reproducing reviewer behavior.
  • domain assumption The five dimensions Clarity, Novelty, Validity, Feasibility, Significance are an adequate decomposition of idea quality.
    The multi-dimensional decoupled evaluation and all human correlations rely on this decomposition (§3.3, §4.1). Other dimensions (e.g., ethics, reproducibility) are not measured.
  • ad hoc to paper Masking retrieved knowledge according to persona familiarity approximates human experts' differing backgrounds.
    The paper assumes that randomly hiding 20–100% of literature/web/code knowledge based on 1–10 familiarity scores simulates genuine diversity of human reviewers (Appx. F). No independent evidence connects this masking mechanism to human cognition.
  • domain assumption The online sources (arXiv, Semantic Scholar, Google Scholar, Google, GitHub, Kaggle) provide sufficient coverage and timestamps can be enforced at search time.
    The 'living knowledge' claim depends on these APIs returning relevant, non-stale, and correctly time-filtered results (§3.1). API limitations or paywalls would degrade grounding.
  • domain assumption LLM scores conditioned on real peer-review comments faithfully represent human reviewer judgments.
    In the human-evaluation protocol, a separate LLM converts textual peer-review comments into five dimension scores, and these 'Reviews' correlations are reported alongside human expert correlations (Appx. D). This proxy is used as a second validation signal.
  • domain assumption The self-retrieved identical paper should not be penalized as prior art.
    The Novelty prompt includes a Self-Discovery Check telling the reviewer to assume a retrieved identical paper is the idea itself (Appx. H.4). This prevents false novelty penalties but also assumes exact matches are always the idea rather than near-duplicate prior work.

reviewed 2026-08-02 · how reviews work

0 comments
read the original abstract

The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.

Figures

Figures reproduced from arXiv: 2602.14367 by Bin Wu, Boyang Xue, Emine Yilmaz, Hossein A. Rahmani, Huajun Chen, Jeff Z. Pan, Keyan Ding, Ningyu Zhang, Qiang Zhang, Shuofei Qiao, Xuehai Wang, Yanshan Wang, Yunxiang Wei.

Figure 1
Figure 1. Figure 1: Framework of InnoEval. Structured idea parts with dashed boxes are optional. Given the raw-text idea, the deep knowledge search engine (left-hand) iterates N times, ultimately yielding enriched knowledge reports categorized into three types: literature, web, and code. During evaluation, each metric is assessed by a dedicated evaluator agent, and users can freely register additional metrics. evaluation crit… view at source ↗
Figure 2
Figure 2. Figure 2: Left Heat Map: Human Evaluation. Correlations between InnoEval’s scores on the five dimensions and the scores assigned by human experts (Human), as well as by online peer-review comments (Reviews) on the same five dimensions. Right Bar Charts: Ablation Studies. -Grounding removes the grounding module and feeds raw search results directly into evaluation; -Personalized disables the persona, letting the agen… view at source ↗
Figure 3
Figure 3. Figure 3: (a) Multi-perspective Test-time Scaling. We compare the test-time scaling results with or without academic personas on point-wise and group-wise tasks. (b) Search Module Eval. We compare our heterogeneous deep knowledge search engine with the search modules of other baselines from four metrics. The specific definition of each metric can be found in Appx.E. (c) Idea Generation. We use the evaluation results… view at source ↗
Figure 4
Figure 4. Figure 4: Scatter Plots Between Metric Pairs. We perform linear regression fitting for each metric pair. The red dashed line is the fit after removing outliers, and its slope is reported. r denotes the Pearson coefficient, ρ is the Spearman coefficient, and R2 represents the fit goodness of all inliers explained by the fitted line. A complete version of all metric pairs’ correlation can be seen in Fig.5. be designed… view at source ↗
Figure 5
Figure 5. Figure 5: Scatter Plots Between Metric Pairs. We perform linear regression fitting for each metric pair. The red dashed line is the fit after removing outliers, and its slope is reported. r denotes the Pearson coefficient, ρ is the Spearman coefficient, and R2 represents the fit goodness of all inliers explained by the fitted line. the topic keywords covered by the retrieved resources overlap with those of the idea.… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SciAtlas: A Large-Scale Knowledge Graph for Automated Scientific Research

    cs.AI 2026-05 unverdicted novelty 4.0

    SciAtlas builds a large-scale multi-disciplinary academic knowledge graph and a neuro-symbolic retrieval system to support automated scientific research tasks such as literature review and idea positioning.

Reference graph

Works this paper leans on

55 extracted references · 8 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs) 24

    foundation or frontier models, including LLMs 15 1. Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs) 24

  2. [2]

    DeepSeek-AI

    URL https://openreview.net/forum? id=mZn2Xyh9Ec. DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large language models, 2025. URL https://arxiv. org/abs/2512.02556. Du Plessis, M. The role of knowledge management in innovation.Journal of knowledge management, 11(4): 20–29, 2007. Feng, T., Sun, Y ., and You, J. Grapheval: A lightweight graph-based...

    Pith/arXiv arXiv 2025

  3. [3]

    Machine learning for sciences (e.g

    applications to computer vision, audio, language, and other modalities 11 3. Machine learning for sciences (e.g. climate, health, life sciences, physics, social sciences) 7

  4. [4]

    Applications (e.g., vision, language, speech and audio, Creative AI) 7

    reinforcement learning 11 4. Applications (e.g., vision, language, speech and audio, Creative AI) 7

  5. [5]

    URL https: //doi.org/10.48550/arXiv.2408.06292

    doi: 10.48550/ARXIV .2408.06292. URL https: //doi.org/10.48550/arXiv.2408.06292. Majumder, M. Multi criteria decision making. InImpact of urbanization on water shortage in face of climatic aberrations, pp. 35–47. Springer, 2015. Moussa, H. N., Silva, P. Q. D., Adu-Ampratwum, D., East, A., Lu, Z., Puccetti, N., Xue, M., Sun, H., Majumder, B. P., and Kumar,...

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.06292 2015

  6. [6]

    URL https: //doi.org/10.48550/arXiv.2510.16234

    doi: 10.48550/ARXIV .2510.16234. URL https: //doi.org/10.48550/arXiv.2510.16234. Navajas, J., Niella, T., Garbulsky, G., Bahrami, B., and Sig- man, M. Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds.Nature Human Behaviour, 2(2):126–132, 2018. Nie, F., Wang, J., Hua, H., Bianchi, F., Kwon, Y ., Qi, Z., Queen, O., ...

    doi:10.48550/arxiv.2510.16234 2018

  7. [7]

    Ou, Y ., Luo, Y ., Zheng, J., Wei, L., Qiao, S., Zhang, J., Zheng, D., Chen, H., and Zhang, N

    https://openai.com/index/ introducing-o3-and-o4-mini/. Ou, Y ., Luo, Y ., Zheng, J., Wei, L., Qiao, S., Zhang, J., Zheng, D., Chen, H., and Zhang, N. Automind: Adap- tive knowledgeable agent for automated data science. CoRR, abs/2506.10974, 2025. doi: 10.48550/ARXIV . 2506.10974. URLhttps://doi.org/10.48550/ arXiv.2506.10974. Qiao, S., Ou, Y ., Zhang, N.,...

    doi:10.48550/arxiv.2506.10974 2025

  8. [8]

    URL https: //doi.org/10.48550/arXiv.2506.22026

    doi: 10.48550/ARXIV .2506.22026. URL https: //doi.org/10.48550/arXiv.2506.22026. Shao, C., Huang, D., Li, Y ., Zhao, K., Lin, W., Zhang, Y ., Zeng, Q., Chen, Z., Li, T., Huang, Y ., Wu, T., Liu, X., Zhao, R., Zhao, M., Zhang, X., Wang, Y ., Zhen, Y ., Xu, F., Li, Y ., and Liu, T.-Y . Omniscientist: Toward a co-evolving ecosystem of human and ai scientists...

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.22026 2025

  9. [9]

    URL https: //doi.org/10.48550/arXiv.2505.18705

    doi: 10.48550/ARXIV .2505.18705. URL https: //doi.org/10.48550/arXiv.2505.18705. Wang, W., Gu, L., Zhang, L., Luo, Y ., Dai, Y ., Shen, C., Xie, L., Lin, B., He, X., and Ye, J. Scipip: An llm-based scientific paper idea proposer.CoRR, abs/2410.23166, 2024a. doi: 10.48550/ARXIV .2410.23166. URLhttps: //doi.org/10.48550/arXiv.2410.23166. Wang, Y ., Guo, Q.,...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.18705 2024

  10. [10]

    URL https: //doi.org/10.48550/arXiv.2403.09733

    doi: 10.48550/ARXIV .2403.09733. URL https: //doi.org/10.48550/arXiv.2403.09733. Weng, Y ., Zhu, M., Xie, Q., Sun, Q., Lin, Z., Liu, S., and Zhang, Y . Deepscientist: Advancing frontier-pushing scientific findings progressively.CoRR, abs/2509.26603,

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.09733

  11. [11]

    URL https: //doi.org/10.48550/arXiv.2509.26603

    doi: 10.48550/ARXIV .2509.26603. URL https: //doi.org/10.48550/arXiv.2509.26603. Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., and Nie, J. C-pack: Packed resources for general chinese em- beddings. In Yang, G. H., Wang, H., Han, S., Hauff, C., Zuccon, G., and Zhang, Y . (eds.),Proceedings of the 47th International ACM SIGIR Conference on Resear...

    doi:10.48550/arxiv.2509.26603 2024

  12. [12]

    URL https: //doi.org/10.1145/3626772.3657878

    doi: 10.1145/3626772.3657878. URL https: //doi.org/10.1145/3626772.3657878. 11 InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem Xie, Q., Weng, Y ., Zhu, M., Shen, F., Huang, S., Lin, Z., Zhou, J., Mao, Z., Yang, Z., Yang, L., Wu, J., and Zhang, Y . How far are AI scientists from changing the world? CoRR, a...

    arXiv 2025

  13. [13]

    acl-long.609/

    URL https://aclanthology.org/2025. acl-long.609/. Yu, Z., Feng, K., Zhao, Y ., He, S., Zhang, X.-P., and Cohan, A. Alpharesearch: Accelerating new algorithm discovery with language models, 2025. URL https://arxiv. org/abs/2511.08522. Zhang, B., Feng, S., Yan, X., Yuan, J., Yu, Z., He, X., Huang, S., Hou, S., Nie, Z., Wang, Z., Liu, J., Ma, R., Peng, T., Y...

    arXiv 2025

  14. [14]

    other topics in machine learning (i.e., none of the above) 5

  15. [15]

    Theory (e.g., control theory, learning theory, algorithmic game theory) 11

    generative models 13 2. Theory (e.g., control theory, learning theory, algorithmic game theory) 11

  16. [16]

    applications to neuroscience & cognitive science 4

  17. [17]

    neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.) 3

  18. [18]

    Probabilistic methods (e.g., variational inference, causal inference, Gaussian processes) 6

    datasets and benchmarks 10 5. Probabilistic methods (e.g., variational inference, causal inference, Gaussian processes) 6

  19. [19]

    Social and economic aspects of machine learning (e.g., fairness, interpretability, human-AI interaction, privacy, safety, strategic behavior) 6

    interpretability and explainable AI 10 6. Social and economic aspects of machine learning (e.g., fairness, interpretability, human-AI interaction, privacy, safety, strategic behavior) 6

  20. [20]

    Reinforcement learning (e.g., decision and control, planning, hierarchical RL, robotics) 5

    unsupervised, self-supervised, semi-supervised, and supervised representation learning 10 7. Reinforcement learning (e.g., decision and control, planning, hierarchical RL, robotics) 5

  21. [21]

    Optimization (e.g., convex and non-convex, stochastic, robust) 4

    applications to physical sciences (physics, chemistry, biology, etc.) 7 8. Optimization (e.g., convex and non-convex, stochastic, robust) 4

  22. [22]

    Neuroscience and cognitive science (e.g., neural coding, brain-computer interfaces) 3

    learning theory 7 9. Neuroscience and cognitive science (e.g., neural coding, brain-computer interfaces) 3

  23. [23]

    General machine learning (supervised, unsupervised, online, active, etc.) 3

    applications to robotics, autonomy, planning 6 10. General machine learning (supervised, unsupervised, online, active, etc.) 3

  24. [24]

    Infrastructure (e.g., libraries, improved implementation and scalability, distributed solutions) 2

    learning on graphs and other geometries & topologies 6 11. Infrastructure (e.g., libraries, improved implementation and scalability, distributed solutions) 2

  25. [25]

    Others 2

    alignment, fairness, safety, privacy, and societal considerations 6 12. Others 2

  26. [26]

    Evaluation (e.g., methodology, meta studies, replicability and validity, human-in-the-loop) 1

    learning on time series and dynamical systems 5 13. Evaluation (e.g., methodology, meta studies, replicability and validity, human-in-the-loop) 1

  27. [30]

    infrastructure, software libraries, hardware, systems, etc. 1

  28. [31]

    miracle steps

    transfer learning, meta learning, and lifelong learning 1 labels (Poster vs. Reject, Highlight vs. Poster), and are therefore harder to distinguish. After screening and filtering, we construct a dataset Dpair comprising 372 samples, including 172 easy pairs and 200 difficult pairs. We directly use Accuracy to evaluate pair-wise tasks and report the result...

    2025

  29. [32]

    First, in your reasoning (not in output), deeply understand and condense the idea into: - One single core essence (usually 3–8 words that capture what this work is truly about) - One single most important motivation/pain point this idea is directly solving - One or (rarely) two truly decisive technical components that define the method

  30. [33]

    No secondary or peripheral concepts are allowed

    All generated queries MUST revolve only around these 2–4 ultra-core concepts identified above. No secondary or peripheral concepts are allowed

  31. [34]

    vision-language models

    For each core concept, expand 2–5 academic synonyms or alternative phrasings that commonly appear in real paper titles (e.g., “vision-language models”, “multimodal large language models”, “VLMs”)

  32. [35]

    Most queries should be single-concept with rich OR chains or at most one AND

    Generate queries using primarily OR within the same concept slot to maximize recall of different expressions, and use AND extremely sparingly — only when combining two truly indispensable core concepts (core problem + core method, or core method + core context). Most queries should be single-concept with rich OR chains or at most one AND

  33. [36]

    ” clauses - Only ti:“

    Final goal: every returned paper from these 6–10 queries should feel ”this is almost exactly our idea” to a human researcher. Precision ¿ breadth. ====================== STRICT OUTPUT FORMAT (UNCHANGED) ====================== Output ONLY: [QUERY1|QUERY2|...|QUERYN ] - 6≤N≤10 - Each QUERY contains 1 to 3 ti:“...” clauses - Only ti:“...” clauses + uppercase...

  34. [37]

    The true core of this idea is: X

    Core essence (one phrase): “The true core of this idea is: X”

  35. [38]

    The single most important pain point being solved is: Y

    Most direct motivation/gap: “The single most important pain point being solved is: Y”

  36. [39]

    The truly novel/enabling technique(s) are: Z (and W if any)

    Decisive technical component(s): “The truly novel/enabling technique(s) are: Z (and W if any)”

  37. [40]

    supervised fine-tuning

    For each of X, Y , Z, list 3–5 title-level synonyms/alternative phrasings Only after this analysis do you design the 6–10 queries. ====================== WHAT IS NOW FORBIDDEN ====================== - Using AND to combine two non-essential or loosely related concepts - Queries that would return>200–300 results on arXiv (too noisy) - Including minor techni...

  38. [41]

    - Notice terminology and synonyms that appear to be widely used and well-matched to the idea

    Analyze good queries and top paper titles: - Extract recurring, high-signal keywords/phrases and phrasings that characterize the core topic, tasks, methods, or domains. - Notice terminology and synonyms that appear to be widely used and well-matched to the idea

  39. [42]

    - Identify low-relevance or noisy keywords and avoid reusing them in new queries

    Analyze weak queries: - Identify over-specific fragments (very detailed or niche conditions) that likely prevent finding additional relevant papers; consider how they could be generalized or removed. - Identify low-relevance or noisy keywords and avoid reusing them in new queries

  40. [43]

    Reflect on coverage and gaps: - Determine which aspects of the idea full text are already well-covered by the current top papers (e.g., particular methods, datasets, problem settings). 35 InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem - Identify missing or under-explored perspectives, such as: alternativ...

  41. [44]

    ” clauses combined with uppercase AND / OR. - Each query must contain 1–3 ti:“

    Design refined queries: - Reuse and recombine high-signal keywords from good queries and from top paper titles. - Generalize over-specific fragments from weak queries (e.g., shorten overly detailed phrases, drop unnecessary constraints, or replace them with slightly broader terms). - Avoid low-relevance or noisy keywords observed in weak queries. - Introd...

  42. [45]

    Good queries + top source titles/summaries: extract recurring high-signal concepts, phrasings, and synonyms that align with the idea full text

  43. [46]

    36 InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

    Weak queries: spot over-specific fragments to generalize/remove, and noisy/low-relevance terms to avoid. 36 InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

  44. [47]

    Coverage check: note which aspects are already well-covered and which angles, methods, domains, or terminology are missing

  45. [48]

    FORMAT CONSTRAINTS: - Each query uses ONLY AND / OR (no NOT), with 1–3 keyword/phrase groups

    Design refined queries: recombine strong keywords, generalize over-specific bits, drop noisy terms, and introduce adjacent terminology that can surface complementary results while staying on-topic. FORMAT CONSTRAINTS: - Each query uses ONLY AND / OR (no NOT), with 1–3 keyword/phrase groups. - Multi-word concepts must be in double quotes; use OR in parenth...

  46. [49]

    Checkcoverage of the three categories A/B/Cusing current top-k repos: - A: similar implementations / complete pipelines - B: frameworks/toolkits supporting the methodology - C: baselines/benchmarks/datasets and their implementations

  47. [50]

    Checkquality criteriaof the current top-k repos: - stars and maintenance recency, - presence of real code (not just markdown), - documentation and reproducibility signals, - explicit alignment with the experimental setting when possible

  48. [51]

    Diffusion

    If certain categories (A/B/C) or quality aspects are under-covered: - Designmore general, less constrained follow-up queriesthat: - broaden over-specific patterns from weak queries, - drop redundant or noisy keywords, - reuse strong, high-signal terms from good queries and top repo titles. ====================== REFINEMENT STRATEGY ====================== ...

  49. [52]

    You MUST explicitly check for: (a) specific expected quantitative results, (b) specific baselines/compar- isons, (c) clear evaluation protocol, (d) concrete method mechanism

  50. [53]

    - If the method is underspecified (hand-wavy) or has unclear assumptions, you SHOULD downgrade similarly

    Missing-evidence is itself valid justification to DOWNGRADE: - If there are no specific datasets/metrics AND no clear experimental plan, you SHOULD downgrade ac score (typically -0.5 to -1.5) and set confidence to low/medium. - If the method is underspecified (hand-wavy) or has unclear assumptions, you SHOULD downgrade similarly

  51. [54]

    Strong-evidence is required to UPGRADE: - Upgrade only if concrete evidence is present (specific math formulations, comprehensive baseline lists, rigorous theoretical grounding)

  52. [55]

    Ideas”: - Since this is an idea evaluation (no full text), be extra critical of “vague promises

    Calibration on “Ideas”: - Since this is an idea evaluation (no full text), be extra critical of “vague promises”. A list of “we will improve accuracy” is NOT evidence. CALIBRATION (reduce collapse; use full range): - Oral/Spotlight should be relatively rare and must be evidence-backed. 46 InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Mult...

  53. [2023]

    URL https://openreview.net/forum? id=COZDy0WYGg. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces.CoRR, abs/2312.00752, 2023. doi: 10.48550/ARXIV .2312.00752. URL https:// doi.org/10.48550/arXiv.2312.00752. Gu, A., Goel, K., and R ´e, C. Efficiently modeling long sequences with structured state spaces. InThe Tenth Intern...

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.00752 2023

  54. [2024]

    URL https: //doi.org/10.48550/arXiv.2411.14199

    doi: 10.48550/ARXIV .2411.14199. URL https: //doi.org/10.48550/arXiv.2411.14199. Baek, J., Jauhar, S. K., Cucerzan, S., and Hwang, S. J. Re- searchagent: Iterative research idea generation over scien- tific literature with large language models. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.),Proceedings of the 2025 Conference of the Nations of the Ameri...

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.14199 2025

  55. [2025]

    URL https: //doi.org/10.48550/arXiv.2502.14776

    doi: 10.48550/ARXIV .2502.14776. URL https: //doi.org/10.48550/arXiv.2502.14776. Liu, Z., Cai, Y ., Zhu, X., Zheng, Y ., Chen, R., Wen, Y ., Wang, Y ., E, W., and Chen, S. Ml-master: Towards ai-for-ai via integration of exploration and reasoning. CoRR, abs/2506.16499, 2025. doi: 10.48550/ARXIV . 2506.16499. URLhttps://doi.org/10.48550/ arXiv.2506.16499. L...

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.14776 2025

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.