Pith. sign in

REVIEW 5 major objections 6 minor 92 references

ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A parse–generate–filter pipeline produces a 61,376-pair geoscience RAG dataset whose task-oriented questions reflect the information needs of professional researchers.

desk verdict A large, genuinely new geoscience RAG dataset with a clearly described pipeline; the dataset-paper matching step and the answer filter need more validation before the reliability claims hold. read the letter →

arxiv 2506.11117 v1 pith:BLJU46PP submitted 2025-06-09 cs.CL cs.AIcs.IR

Junyong Lin , Lu Dai , Ruiqian Han , Yijie Sui , Ruilin Wang , Xingliang Sun , Qinglin Wu , Min Feng
show 2 more authors
Hao Liu Hui Xiong
This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords Retrieval-AugmentedGenerationLargeLanguageModelsScientificWorkflowsDatasetSyntheticInformationRetrievalQuestionAnsweringGeoscience
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing scientific retrieval and QA datasets are dominated by simple lookup questions, while real researchers ask task-oriented questions embedded in a workflow. ScIRGen is a pipeline that builds a large-scale RAG dataset from geoscience dataset metadata plus the papers that use those datasets, generates questions under an 18-type cognitive taxonomy of question categories, and filters generated answers by measuring whether conditioning on the context raises an LLM's confidence in the answer. The resulting ScIRGen-Geo dataset contains 61,376 QA pairs, a corpus of 3,345 datasets and their linked papers, and a cognitive distribution that is more balanced than existing benchmarks. The paper's benchmark results show that current LLMs handle short verification questions well but score poorly on causal, procedural, and analytical reasoning, which is the gap the dataset is designed to expose.

What carries the argument

The machinery has three load-bearing parts. First, a Dataset-Paper Relevance Analyzer prompts an LLM to decide, from metadata and paper content, whether a publication substantively uses a given dataset; those links are what turn thin metadata into a grounded context. Second, question generation is driven by an 18-type pedagogical taxonomy (covering verification, quantification, causal antecedent, instrumental/procedural, and related categories), which controls the diversity and cognitive level of the synthesized questions. Third, SePer-Filter tests each candidate answer by computing $\Delta\mathrm{SePer} = P_M(a^*\mid q,d) - P_M(a^*\mid q)$ with $q$ the question, $d$ the context, and $a^*$ the answer, and accepts the triplet only when $\Delta\mathrm{SePer} > 0$; the paper reports this filter reaches 95% precision against human annotation on a 100-sample evaluation.

What would settle it

A human audit of a random sample of dataset–paper links is the direct test: if geoscience experts find that a substantial fraction (say over 10%) of the analyzer's 'used' links are false matches, then the questions and ground-truth answers built on those links are grounded in irrelevant evidence, and the dataset's realism claim would not survive for that slice.

Watch

Extended reading notes

Core claim

The paper's claim is that it is possible to synthesize a trustworthy, research-level RAG dataset by construction rather than by mining or crowdsourcing. ScIRGen enriches each dataset's thin metadata with six aspects parsed from relevant publications (background, research objective, methods, challenges, dataset usage, findings), uses those enriched contexts to generate QA pairs across 18 question types in a pedagogy-derived taxonomy, and keeps an answer only when the context shifts the generator's confidence upward ($\Delta\mathrm{SePer}>0$). The result is ScIRGen-Geo: 61,376 question-answer pairs over 3,345 geoscience datasets, whose cognitive profile spans six levels with a diversity index of $0.91$, higher than the compared benchmarks. The paper then shows that retrieval improves when the index includes paper-derived passages, while QA accuracy stays low for causal and procedural question types across all tested models.

Load-bearing premise

The whole dataset stands on the LLM-based relevance analyzer correctly deciding which papers substantively use each dataset, and the paper provides no human evaluation of that matching step.

Editorial extensions

If this is right

  • The dataset supports both dataset retrieval and open-domain question answering, so a single benchmark can evaluate retrievers and generators on the same research-oriented questions.
  • Adding parsed paper passages to the retrieval index improves recall for every tested retriever, with BGE-M3's Recall@100 rising by about 14%, so paper-augmented metadata is a practical way to enrich sparse scientific dataset descriptions.
  • All twelve evaluated LLMs score below 0.25 on causal reasoning question types, indicating that the benchmark isolates a specific weakness in current models rather than measuring overall language ability.
  • Because the generation pipeline is not geoscience-specific, the same parse-generate-filter design can be applied to other scientific domains to build comparable RAG datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The realism claim is only as strong as the unvalidated dataset–paper matching; a human audit of those links, or a comparison against real researcher query logs, would be the natural next experiment and is not reported in the paper.
  • The filter's human alignment rests on a single 100-triplet annotation study; a larger annotation set, or a stress test that feeds deliberately wrong answers and unrelated contexts, would show how the 95% precision figure scales.
  • One testable extension of the framework would be to generate a dataset for another discipline and measure whether the taxonomy's 18 question types still cover the questions researchers actually ask, or whether domain-specific question types are needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces ScIRGen, an end-to-end pipeline for synthesizing retrieval-augmented generation (RAG) datasets for scientific research, and applies it to construct ScIRGen-Geo, a 61,376-question-answer dataset for geoscience with 3,354 datasets and 943 matched papers. The pipeline has three stages: (1) corpus curation, which links datasets to relevant publications via an LLM-based relevance analyzer and parses papers into six aspects; (2) question generation, which uses Graesser and Pearson's 18-type cognitive taxonomy to generate QA pairs from dataset metadata and paper passages; and (3) answer filtering, which uses a perplexity-shift (SePer) statistic to accept or reject generated answers. The paper reports benchmark results for dataset retrieval (BM25, Contriever, BGE-M3, etc.) and QA (general and scientific LLMs, with and without retrieved context), and analyzes the cognitive-level distribution of the questions against existing benchmarks such as BEIR and BIRCO.

Significance. If the pipeline's outputs are reliable, ScIRGen-Geo would be a substantial resource: it is large-scale, domain-specific, publicly released, and aimed at task-oriented queries that existing QA/retrieval benchmarks lack. The paper's strengths include a detailed, reproducible description of the generation prompts (in the appendix), a released dataset and code, a filtering method that is compared against LLM-as-a-judge baselines and a 100-sample human annotation, and a thoughtful cognitive-level analysis using revised Bloom's taxonomy. The central claims, however, rest on two load-bearing but insufficiently validated components: the LLM-based dataset-paper matching (Section 2.1.1) and the SePer filter (Section 2.3). The matching step has no human evaluation and the prompt biases toward recall; the filter measures self-consistency of an answer with the context it was generated from, not correctness against an external ground truth. These issues directly affect the claimed reliability of the dataset as a benchmark.

major comments (5)
  1. [Section 2.1.1, Table 9] The Dataset-Paper Relevance Analyzer is the upstream step that determines which papers provide context for QA generation and which datasets are labeled as retrieval positives, yet no human evaluation of this matching is reported. Table 9 explicitly instructs the LLM to "err on the side of considering the paper relevant" when there is indirect evidence, which maximizes recall at the cost of precision. Since all QA contexts and retrieval labels inherit errors from this step, a false-positive link means many QA pairs are grounded in an irrelevant paper and the retrieval ground-truth dataset may not be the one the question is about. Please provide a human-precision evaluation on a random sample of predicted dataset-paper pairs (e.g., 100 pairs with expert judgment), and if precision is low, introduce a confidence threshold or a human-in-the-loop verification step.
  2. [Section 2.3, Eqs. (1)-(2)] The SePer filter is circular as defined: a* is the answer generated from the context d, so P_M(a*|q,d) is expected to exceed P_M(a*|q) for any answer that is a near-paraphrase of d, regardless of whether the answer is actually correct for the question or the dataset. The filter cannot detect an irrelevant context (e.g., a wrongly matched paper from Section 2.1.1) because the positive shift is driven by lexical/statistical overlap between d and a*. The 100-sample human validation shows that the filter agrees with human judgments of faithfulness for cases where the context is relevant, but it does not test the filter's behavior when the context is irrelevant or when a wrong but self-consistent answer is generated. This is load-bearing because the paper claims the filter is "highly aligned with human judgment" and uses it to present ScIRGen-Geo as reliable ground truth. Please report an experiment that feeds the filter with (q, d, a*) triples where d is an irrelevant context (e.g., from a different dataset) and show that ΔSePer remains negative, or restrict the claim to self-consistency filtering rather than external faithfulness.
  3. [Section 4.2, Table 4] The retrieval evaluation assumes exactly one relevant dataset per query: the dataset from which the QA pair was generated. Many of the task-oriented questions (e.g., "What methods are used for processing eddy covariance data to ensure quality and accuracy in arid regions?") could plausibly be answered by several datasets in the corpus, so the reported R@k and MRR understate true recall. Additionally, in the w/paper setting, the index includes the very paper passages used to generate the test questions, making the setting closer to memorization of the generating context than to generalizing to unseen scientific queries. Please either annotate additional relevant datasets for a sample of test queries and report recall with the expanded label set, or explicitly discuss this limitation in the benchmark interpretation.
  4. [Abstract and Section 2.2] The central claim that ScIRGen-Geo "more accurately reflects the information needs of professional science researchers" is not directly evidenced. The paper shows that questions follow a cognitive taxonomy and that the distribution spans Bloom's levels, but it does not compare the synthesized questions to real researcher queries, search logs, or expert-elicited information needs. Figure 2 mentions an "Expert Survey" but no details are given anywhere in the manuscript. Please provide an evaluation in which domain experts rate whether the generated questions resemble queries they would ask in their own research, or soften the claim to "designed to reflect task-oriented query patterns based on a pedagogical taxonomy."
  5. [Section 2.1.1] The dataset curation process from the original 6,819 datasets to the final 3,354 dataset entries is unexplained. The text states that 6,819 datasets were "systematically mapped to 6,755 domain-specific publications," and then reports that the workflow "ultimately yielded 3,345 dataset metadata entries" (the table later uses 3,354). The paper does not describe why roughly half of the datasets were removed, whether the removal was based on metadata completeness, absence of papers, or other criteria. This is relevant because a non-random exclusion could bias the resulting benchmark toward well-described datasets. Please describe the filtering criteria and reconcile the 3,345/3,354 inconsistency.
minor comments (6)
  1. [Section 4.3, Table 5] The text says "C3(Applying) and C4(Evaluating) are the hardest types," but in revised Bloom's taxonomy C4 is Analyzing and C5 is Evaluating. Please correct the label and also reconcile the sentence "C5(Evaluating) is the level that achieves the highest accuracy" with the preceding claim that C3 and C4 are the hardest; Table 5 indicates C5 has the highest accuracy, so the narrative needs rewording.
  2. [Section 2.3, Table 2] The GPT-3.5 and GPT-4 filtering baselines are not described anywhere in the text or appendix. Please state the prompt and scoring protocol used for these LLM-as-a-judge baselines so that the comparison in Table 2 is reproducible.
  3. [Section 4.1] The entailment-model threshold of 0.5 is asserted without justification or sensitivity analysis. Please either cite prior work that establishes this threshold or show accuracy results for a range of thresholds (e.g., 0.3, 0.5, 0.7).
  4. [Section 2.1.2, Table 12] The second-stage parse engine contains a conflicting instruction: it first says "EACH section ... MUST have AT LEAST ONE item retained" but later says "Keep at least one item per section if any exists." If a paper has no findings or challenges, the first instruction could force the LLM to produce spurious content. Please clarify how the parser handles empty sections.
  5. [Section 2.3 and Reference [8]] The relationship between the original SePer framework and the SePerFilter adaptation is described only briefly. Please clarify in the main text whether the difference is only in the input (answer vs. retrieved content) or whether the underlying computation is otherwise unchanged.
  6. [Table 3(b) and Section 2.1.1] The dataset count is inconsistent: Section 2.1.1 says "3,345 dataset metadata entries," while Table 3(b) sums to 3,354 and Section 4.2 says 3,354. Please correct the number.

Circularity Check

1 steps flagged · score 5.0 of 10

Answer filtering via ΔSePer is partly self-confirming because answers are generated from the same context used to compute the belief shift; the 100-sample human check provides only partial external grounding.

  1. self definitional [Section 2.3, Eq. (1)-(2); answer generation prompt in Appendix Table 14]
    "Let 𝑞∈ Q denote a question, 𝑑∈ D denote the associated context (or document), and 𝑎∗∈A denote the ground truth answer. ... ΔSePer=𝑃 𝑀(𝑎∗|𝑞,𝑑)−𝑃 𝑀(𝑎∗|𝑞). (1) ... The underlying hypothesis is that if the answer 𝑎∗ is both correct and faithful to the context 𝑑, then incorporating 𝑑 should increase the generator’s confidence in 𝑎∗, leading to ΔSePer> 0."

    The answers being filtered are generated from exactly this context: the generation prompt instructs the model to 'answer each query using the field above,' where the field contains the dataset metadata and parsed paper content. Therefore a* is, by construction, a continuation conditioned on d. Computing P_M(a*|q,d) for that same d measures whether the model recognizes its own conditioning input, not whether a human expert would judge the answer valid. Positive ΔSePer is thus largely a self-consistency check of the generation context, especially if M is the same as or similar to the generator.

full rationale

The paper has substantial non-circular content: the corpus is grounded in real dataset metadata and publications, question types come from an external pedagogical taxonomy, and benchmarks are evaluated against external baselines. The SePer self-citation [8] is not load-bearing because the paper redefines ΔSePer and supplies its own human-alignment test. The identified circularity is confined to the SePerFilter step: because ground-truth answers are generated from the context d and then validated by whether d raises the model's probability of those answers, the filter's large-scale accept/reject decision is partly determined by construction rather than by independent evidence of answer validity. This weakens but does not eliminate the central claim, since the 100-sample expert annotation and the downstream benchmark experiments provide some external checks. The unvalidated Dataset-Paper Relevance Analyzer (Section 2.1.1) is a correctness risk rather than a circularity: nothing in its construction makes the matching true by definition, and a wrong link would degrade quality without being logically forced.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework's main inputs are design choices and assumptions about LLM behavior, the taxonomy, and the reliability of automated relevance matching. Free parameters are limited to generation and evaluation thresholds. No new physical or conceptual entities are introduced.

free parameters (4)
  • QA pairs per question type per dataset (datasets with papers) = 3
    Chosen by hand; yields 54 pairs per dataset and shapes dataset size and balance (Section 2.2).
  • Question types per dataset without papers = 8
    Selected by LLM from taxonomy; determines coverage for metadata-only datasets (Section 2.2).
  • Entailment threshold for QA evaluation = 0.5
    Used to classify generated answers as semantically equivalent to the reference answer (Section 4.1).
  • Passage chunk size = 100 tokens
    Used for the retrieval-augmented QA benchmark (Section 4.3).
assumptions (5)
  • domain assumption Graesser and Pearson's 18-type taxonomy spans the space of scientific research questions.
    Question generation relies entirely on this taxonomy; if real research queries fall outside it, the dataset's realism is limited (Section 2.2).
  • ad hoc to paper An LLM's conditional probability shift indicates answer faithfulness.
    The SePer-Filter accepts an answer only if P_M(a*|q,d) > P_M(a*|q); this belief-shift hypothesis is central to answer quality (Section 2.3, Eq. 1-2).
  • domain assumption LLM-based Dataset-Paper Relevance Analyzer correctly identifies papers that use each dataset.
    No human evaluation is provided for this classifier; errors propagate to the contexts used for QA generation (Section 2.1.1).
  • domain assumption Extracted paper passages contain sufficient information to answer generated questions.
    The QA pairs are generated from metadata and parsed passages; if extraction misses key details, answers may be unanswerable from the corpus (Section 2.2).
  • domain assumption The 100-sample human annotation is representative of the full 61k dataset.
    Filter validation uses this sample; the small size may not reflect the diversity of question types and contexts (Section 2.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research." pith.science (2026). https://pith.science/paper/BLJU46PP

@misc{pith2026250611117,
  author       = {Pith},
  title        = {Pith review of: ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLJU46PP}},
  note         = {Machine review of arXiv:2506.11117}
}
read the original abstract

Scientific researchers need intensive information about datasets to effectively evaluate and develop theories and methodologies. The information needs regarding datasets are implicitly embedded in particular research tasks, rather than explicitly expressed in search queries. However, existing scientific retrieval and question-answering (QA) datasets typically address straightforward questions, which do not align with the distribution of real-world research inquiries. To bridge this gap, we developed ScIRGen, a dataset generation framework for scientific QA \& retrieval that more accurately reflects the information needs of professional science researchers, and uses it to create a large-scale scientific retrieval-augmented generation (RAG) dataset with realistic queries, datasets and papers. Technically, we designed a dataset-oriented information extraction method that leverages academic papers to augment the dataset representation. We then proposed a question generation framework by employing cognitive taxonomy to ensure the quality of synthesized questions. We also design a method to automatically filter synthetic answers based on the perplexity shift of LLMs, which is highly aligned with human judgment of answers' validity. Collectively, these methodologies culminated in the creation of the 61k QA dataset, ScIRGen-Geo. We benchmarked representative methods on the ScIRGen-Geo dataset for their question-answering and retrieval capabilities, finding out that current methods still suffer from reasoning from complex questions. This work advances the development of more sophisticated tools to support the intricate information needs of the scientific community.

Figures

Figures reproduced from arXiv: 2506.11117 by the authors.

Figure 1
Figure 1. Traditional queries fall short in task-oriented sce [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The QA data generation pipeline. (a) Gather dataset metadata along with a corpus of relevant papers, from which we [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Answer Filter Assessment: SePer-Filter vs. GPT-3.5 and GPT-4 In this section, we rigorously define our automatic QA data filtering method ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Cognitive Level Distribution Across Different QA Datasets. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Diagram mapping each Graesser question type in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Expert Annotation Interface [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 62 canonical work pages

  1. [1]

    Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. LitSearch: A Retrieval Benchmark for Scientific Literature Search. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 15068–15083

    2024

  2. [2]

    Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA Corpora Generation with Roundtrip Consistency. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics

    2019

  3. [3]

    Lorin W Anderson and David R Krathwohl. 2010. Kerangka landasan untuk pembelajaran, pengajaran, dan asesmen (terjemahan Agung Prihantoro).New York: Addition Wesley Longman.(buku asli diterbitkan tahun 2001)(2010)

    2010

  4. [4]

    Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A Pretrained Language Model for Scientific Text. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 3615–3620

    2019

  5. [5]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216 (2024)

    arXiv 2024

  6. [6]

    Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun

    Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. 2025. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discover...

    2025

  7. [7]

    Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld

  8. [8]

    Lu Dai, Yijie Xu, Jinhui Ye, Hao Liu, and Hui Xiong. 2025. SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/ forum?id=ixMBnOhFGd

    2025

Show all 92 references
  1. [9]

    Cheng Deng, Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi, Yi Xu, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al . 2024. K2: A foundation language model for geoscience knowledge understanding and utilization. InProceedings of the 17th ACM International Conf...

    2024

  2. [10]

    2019.Unified language model pre-training for natural language understanding and generation

    Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.Unified language model pre-training for natural language understanding and generation. Curran Associates Inc

    2019

  3. [11]

    Michael Färber and Ann-Kathrin Leisinger. 2021. Recommending datasets for scientific problem descriptions. InProceedings of the 30th ACM International Conference on Information & Knowledge Management. 3014–3018

    2021

  4. [12]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling Large Language Models to Generate Text with Citations. InEmpirical Methods in Natural Language Processing (EMNLP)

    2023

  5. [13]

    Arthur C Graesser and Natalie K Person. 1994. Question asking during tutoring. American educational research journal31, 1 (1994), 104–137

    1994

  6. [14]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    2025 arXiv

  7. [15]

    Weiyu Guo, Ziyue Qiao, Ying Sun, Yijie Xu, and Hui Xiong. 2025. Revisiting Noise Resilience Strategies in Gesture Recognition: Short-Term Enhancement in Surface Electromyographic Signal Analysis. InProceedings of the International Conference on Machine Learning (ICML)

    2025

  8. [16]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. InInternational conference on machine learning. PMLR, 3929–3938

    2020

  9. [17]

    Qi He, Jian Pei, Daniel Kifer, Prasenjit Mitra, and Lee Giles. 2010. Context-aware citation recommendation. InProceedings of the 19th international conference on World wide web. 421–430

    2010

  10. [18]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bo- janowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense in- formation retrieval with contrastive learning.arXiv preprint arXiv:2112.09118 (2021)

    2021 arXiv

  11. [19]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for C...

    2019

  12. [20]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing S...

    2020

  13. [21]

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jianguang Lou, Shifeng Chen, Yansong Tang, and Weizhu Chen. 2024. Arena learning: Build data flywheel for llms post-training via simulated chatbot arena.arXiv preprint arXiv:2407.10627(2024)

    2024 arXiv

  14. [22]

    Tengfei Lyu, Weijia Zhang, Jinliang Deng, and Hao Liu. 2025. AutoSTF: Decou- pled Neural Architecture Search for Cost-Effective Automated Spatio-Temporal Forecasting. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1. 985–996

    2025

  15. [23]

    Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021. Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation. InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Vol...

    2021

  16. [24]

    Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, et al. 2024. Agentinstruct: Toward generative teaching with agentic flows.arXiv preprint arXiv:2407.03502(2024)

    2024 arXiv

  17. [25]

    Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catan- zaro. 2020. Training Question Answering Models From Synthetic Data. InProceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    2020

  18. [26]

    Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, and Alice Oh. 2024. Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 20571–20590....

    2024 doi

  19. [27]

    Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O Arik, Danqi Chen, and Tao Yu. 2025. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive R...

    2025

  20. [28]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_ alpaca

    2023

  21. [29]

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Sto- jnic. 2022. Galactica: A large language model for science.arXiv preprint arXiv:2211.09085(2022)

    2022 arXiv

  22. [30]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. arXiv:2104.08663 [cs.IR] https://arxiv.org/abs/ 2104.08663

    2021 arXiv

  23. [31]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  24. [32]

    Vijay Viswanathan, Luyu Gao, Tongshuang Wu, Pengfei Liu, and Graham Neubig

  25. [33]

    Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang. 2023. Evaluating open-qa evaluation.Advances in Neural Information Processing Systems36 (2023), 77013–77042

    2023

  26. [34]

    Hao Wang, Jindong Han, Wei Fan, Leilei Sun, and Hao Liu. 2025. RePST: Lan- guage Model Empowered Spatio-Temporal Forecasting via Semantic-Oriented Reprogramming. InProceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence

    2025

  27. [35]

    Hao Wang, Jindong Han, Wei Fan, Weijia Zhang, and Hao Liu. 2025. PhyDA: Physics-Guided Diffusion Models for Data Assimilation in Atmospheric Systems. arXiv preprint arXiv:2505.12882(2025)

    2025 arXiv

  28. [36]

    Jianyou Andre Wang, Kaicheng Wang, Xiaoyue Wang, Prudhviraj Naidu, Leon Bergen, and Ramamohan Paturi. 2023. Scientific document retrieval using multi- level aspect-based queries.Advances in Neural Information Processing Systems36 (2023), 38404–38419

    2023

  29. [37]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training.arXiv preprint arXiv:2212.03533(2022)

    2022 arXiv

  30. [38]

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. Math-Shepherd: Verify and Reinforce LLMs Step-by- step without Human Annotations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistic...

    2024

  31. [39]

    Xiaoyue Wang, Jianyou Wang, Weili Cao, Kaicheng Wang, Ramamohan Paturi, and Leon Bergen. 2024. BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives. arXiv:2402.14151 [cs.IR] https://arxiv.org/abs/2402.14151

    2024 arXiv

  32. [40]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguis...

    2023

  33. [41]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language P...

    2018

  34. [42]

    Junqi Yin, Sajal Dash, Feiyi Wang, and Mallikarjun Shankar. 2023. Forge: Pre- training open foundation models for science. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1–13. A Appendix Table 8 presents the retr...

    2023

  35. [46]

    Carefully read the paper content and look for any direct mentions of the target dataset

  36. [47]

    Also consider indirect references, such as descriptions of a dataset with characteristics closely matching the target dataset

  37. [48]

    If there are no direct mentions but strong indirect evidence, err on the side of considering the paper relevant

  38. [49]

    Given a segment of text from an academic paper, classify it into one of these categories:

    Return your response in the following format: USED:[Yes/No] EXPLANATION: [Brief explanation of your decision with specific evidence from the text] Table 9: (a) Dataset Relevance Analyzer You are an expert in academic paper analysis. Given a segment of text from an academic pap...

  39. [50]

    abstract&introduction

  40. [51]

    none" - The abstract and introduction sections should be combined into

    conclusion Rules: - If the text doesn’t clearly belong to any category, classify it as "none" - The abstract and introduction sections should be combined into "abstract&introduction" - Method sections might also be called "methodology", "approach", "proposed method", etc. - Ex...

  41. [52]

    - Describe the current state of knowledge in the field of study, highlighting key findings and challenges

    Background: - Provide complete sentences that explain essential background knowledge related to the research topic. - Describe the current state of knowledge in the field of study, highlighting key findings and challenges. - Include explanations of key concepts, theories, or p...

  42. [53]

    Requirements: - State the research motivation, explaining why this study is important and what drives the inquiry

    Research Objective: - Provide complete sentences that state the primary purpose of the research, or, - Describe the main goal or target of the study and the specific problem the research seeks to solve or address, or, - Explain what the researchers hope to achieve through thei...

  43. [54]

    we", "our study

    Methods (ONLY include if present): - ONLY complete sentences describing experimental procedures ACTUALLY PERFORMED in THIS paper - ONLY complete sentences explaining methodological steps ACTUALLY IMPLEMENTED by THIS study’s authors Requirements: - Include ONLY methods that wer...

    2025

  44. [55]

    Challenges (ONLY include if present): - Complete sentences describing research gaps or inadequacies in current solutions - Complete sentences explaining limitations or difficulties in the research process - Complete sentences identifying technical barriers or obstacles - Compl...

  45. [56]

    Dataset (ONLY include if present): - Complete sentences describing the datasets used - Complete sentences detailing size, composition, or key characteristics Requirements: - Extract full sentences that provide complete information about the dataset - Do not split dataset descr...

  46. [57]

    Findings (ONLY include if present): - ONLY complete sentences stating PRIMARY research findings that directly address the core research question - ONLY complete sentences describing MAJOR discoveries or conclusions that answer the study’s main objectives Requirements: - Each f...

  47. [58]

    Never extract partial sentences or fragments

    ONLY extract COMPLETE SENTENCES. Never extract partial sentences or fragments

  48. [59]

    Each extracted text must be grammatically complete and semantically self-contained

  49. [60]

    If multiple sentences are needed to express a complete idea, include all relevant sentences together

  50. [61]

    If a sentence spans multiple categories, include it in all relevant categories

  51. [62]

    Maintain the exact wording from the original text - do not modify or summarize

  52. [63]

    If no complete sentences are found for a category, write "None" for that category. Format your response as key-value pairs, one per line: Background: [extracted complete sentences] Research Objective: [extracted complete sentences] Methods: [extracted complete sentences] Chall...

    2025

  53. [64]

    For sections with multiple items: - Keep the specific and informative items - Filter out redundant content Content Selection Principles:

  54. [65]

    Completeness: Content should form a complete, self-contained unit of information

  55. [66]

    Relevance: Content should directly relate to its category

  56. [67]

    Non-redundancy: Avoid repetitive information

  57. [68]

    Clarity: Prefer clear, well-structured expressions

  58. [69]

    Category Alignment: Content should strictly belong to its claimed category Detailed Evaluation Guidelines:

  59. [70]

    Each category must maintain at least one item

  60. [71]

    Selected content should be complete and self-contained

  61. [72]

    Avoid splitting related information across multiple items

  62. [73]

    Maintain proper categorization of content

  63. [74]

    Prefer comprehensive items over partial descriptions

  64. [75]

    Remove redundant information even if well-written

  65. [76]

    Ensure selected content actually belongs to its category

  66. [77]

    Consider the overall coherence of the selected content Here’s a concrete example of good evaluation: Original Content Example (partial): Background:

  67. [78]

    Energy and water vapor interactions between land surfaces and the atmosphere are the most crucial ecological processes

    "Energy and water vapor interactions between land surfaces and the atmosphere are the most crucial ecological processes... " [Complete EC and LAS background]

  68. [79]

    Experimental Research (W A TER)

    "Experimental Research (W A TER)", many observation sites were established... "

  69. [80]

    The relationship between𝐻+𝐿𝐸and𝑅 𝑛-𝐺0 can be expressed by

    "The relationship between𝐻+𝐿𝐸and𝑅 𝑛-𝐺0 can be expressed by... " Analysis and Decisions: - Keep item 1 because it provides comprehensive background about methods and limitations - Remove item 2 because it’s an incomplete sentence lacking context - Remove item 3 because it conta...

  70. [81]

    Analyze each section’s content carefully

  71. [82]

    Apply the selection principles consistently

  72. [83]

    Provide clear reasoning for your decisions

  73. [84]

    Maintain the essential meaning while removing redundancy

  74. [85]

    You are a researcher generating questions and answers to find relevant metadata, datasets, and papers within a specific domain

    Ensure proper categorization is maintained Return your response in this format: KEEP-INDICES: Background: [list of indices to keep] Methods: [list of indices to keep] Findings: [list of indices to keep] Challenges: [list of indices to keep] Dataset: [list of indices to keep] R...

    2025

  75. [86]

    Only questions and answers without any other information

  76. [87]

    Do not summarize the metadata or Content of relevant Papers

  77. [88]

    Focus on questions that are open-ended and explore a wide range of possibilities or methodologies related to the field

  78. [89]

    a dataset,

    Use neutral terms like "a dataset, " "data collection method, " or "research approach, " instead of specific references like "the study" or "this dataset. "

  79. [90]

    Aim for questions that encourage exploration of nuanced technological details, methodological rigor, and potential sources or strategies for dataset expansion or refinement within the field

  80. [91]

    The answer should come from the field I mentioned above

  81. [92]

    compare,

    Return the questions and answers in JSON format. Table 14: Main Prompt to generate question and answer Role:You are an expert evaluator and cognitive scientist specializing in revised Bloom’s Taxonomy. Task:Classify each question based on its cognitive complexity into one of t...

    2025

  82. [2020]

    InProceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics

    SPECTER: Document-level Representation Learning using Citation- informed Transformers. InProceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics. 2270–2282

  83. [2022]

    Transactions of the Association for Computational Linguistics10 (2022), 539–554

    MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics10 (2022), 539–554

    2022

  84. [2023]

    InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 10288–10303

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.