MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs

Guandong Wang; Jie Zhai; Jingping Liu; Qi Ye; Tong Ruan; Yating Wang; Yongqi Fan

REVIEW 3 major objections 5 minor 67 references

MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that an evaluation pipeline which classifies open-ended questions as factoid or non-factoid before scoring or ranking answers aligns better with human-annotated quality orders than one-size-fits-all automatic judges.

desk verdict A sensible two-stage routing idea for open-ended QA evaluation with solid experiments, but the gold rankings on the self-built datasets need reliability evidence before I'd fully trust the margin of victory. read the letter →

arxiv 2506.15215 v1 pith:CDYUXVWH submitted 2025-06-18 cs.CL

Yongqi Fan , Yating Wang , Guandong Wang , Jie Zhai , Jingping Liu , Qi Ye , Tong Ruan This is my paper · ORCID

classification cs.CL

keywords open-endedQAevaluationfactoidvsnon-factoidLLM-as-a-judgekey-pointscoringnaturallanguageinferencelistwiserankingsilverinstancesinterpretable

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MinosEval is a two-stage method for grading open-ended answers. It first classifies each question as factoid (an answer with a definite factual core) or non-factoid (creative or subjective). For factoid questions it extracts key points from the reference answer and scores each model response by how strongly it entails those points using an NLI model. For non-factoid questions it generates five quality-graded silver answers and asks an LLM to do listwise ranking against those anchors. Across four datasets the resulting orderings align more closely with human annotations than do lexical metrics, embedding similarity, and standard pointwise/pairwise/listwise LLM judges, and the key points and silver examples make the verdicts interpretable.

What carries the argument

The engine of MinosEval is the two-branch scoring design. The factoid branch, adaptive key-point scoring, extracts key points $K$ from the question and reference answer and scores each response $r_i$ by $\frac{1}{|K|}\sum_{k_j \in K} \mathrm{NLI}(r_i, k_j)$, where $\mathrm{NLI}(r_i, k_j) = s_{e,ij} - s_{c,ij}$ is entailment probability minus contradiction probability from a multilingual NLI model (mDeBERTa-v3-base-mnli-xnli). The non-factoid branch, instance-aware listwise ranking, generates five silver instances $I_i$ at quality levels excellent, good, fair, poor, and bad, then hands question, reference, instances, and candidate list to an LLM ranker that outputs an ordered list $R_{nf} = A_{nf}(q_i, a_i, I_i, R_i)$. The third component is the LLM-based fact detection module that routes each sample to one branch; the paper shows that this routing is what makes the tailored scoring strategies possible, and that using either branch on all questions degrades performance.

What would settle it

Take a new set of open-ended questions with high-quality human rankings from a large, independent pool of annotators with measured agreement; if MinosEval's correlation with those rankings is not stronger than plain listwise ranking when the factual/non-factual split is done automatically, the central claim fails. A second falsifier: if removing the fact/non-fact classifier (feeding all questions to one strategy) does not lower performance on at least one dataset, the distinction is not load-bearing.

Watch

Extended reading notes

Core claim

On its own terms, MinosEval claims that the factoid/non-factoid distinction is the missing ingredient in open-ended QA evaluation. The pipeline first runs an LLM-based fact-detection module (over 90% accuracy with five demonstrations) that assigns each question to a factoid cluster or a non-factoid cluster. For factoid questions it computes, for every candidate response, the average NLI score across key points extracted from the reference answer, where the NLI score is entailment probability minus contradiction probability; the responses are then sorted by that score. For non-factoid questions it generates five silver answer instances spanning excellent to bad quality and asks an LLM ranker to order the candidates with those instances as anchors. Across four datasets the resulting rankings show higher Kendall's Tau, Spearman's Rho, and Rank-Biased Overlap against human-annotated gold rankings than BLEU, ROUGE, BERTScore, BEM, PEDANTS, pointwise, pairwise, listwise, LINKAGE, FActScore, RefChecker, PROMETHEUS 2, and AUTO-J. The paper also reports that the gap widens when the question split is done by hand rather than by the automatic module, and that on small candidate sets with the automatic module it does not always beat the strongest baselines.

Load-bearing premise

The load-bearing premise is that the consensus human rankings used as ground truth are the correct quality order; they were produced by a small team with no measured agreement and no independent validation.

Editorial extensions

If this is right

Factoid answers can be graded transparently: the extracted key points show exactly which required facts a response entailed and which it contradicted.
Non-factoid answers are ranked with explicit quality anchors (excellent, good, fair, poor, bad silver instances), making the LLM judge's ordering easier to audit.
Automatic fact detection reaches over 90% accuracy with five demonstrations, so the full pipeline needs no manual question labeling.
On the larger self-built datasets the gap over pairwise/listwise baselines is largest, suggesting the benefit grows as candidate answers become more numerous and more similar in quality.
Benchmark results on 'non-factoid' datasets that actually contain factoid questions should be read with caution, since a one-size-fits-all judge may rank those subsets poorly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension: the same key points used for ranking could be fed back to an LLM as fine-grained revision hints, turning the evaluator into a grader that tells the generator which facts it missed.
The factoid/non-factoid boundary is likely a spectrum; a soft weighting between the two scoring strategies might handle borderline questions that the paper's hard split misclassifies.
Because the paper finds instance-aware listwise ranking also works reasonably on factoid questions, a cheaper coarse version could drop NLI key-point scoring for all but the most fact-critical evaluations.
The observed reclassification of many 'non-factoid' samples into factoid suggests existing open-ended QA benchmarks may have been evaluated under a mismatched scoring model; re-running past comparisons with the split could change reported model orderings.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What should you know about this paper: it proposes MinosEval, a two-stage evaluator that first classifies open-ended questions as factoid or non-factoid, then applies key-point scoring with an NLI model for factoid questions and instance-aware listwise ranking (with LLM-generated silver quality anchors) for non-factoid ones. The genuinely new pieces are the routing itself and the silver-anchor listwise step; the components are known from FActScore, RefChecker, and LINKAGE. They also release two self-built datasets with six candidate responses per question, which fills a real gap.

The paper does a lot right. The baseline selection is fair and broad, including specialized judges like PROMETHEUS 2 and AUTO-J. The gains on the larger self-built datasets are consistent across metrics, and the ablation convincingly shows why routing matters: adaptive key-point scoring alone collapses on non-factoid questions, while instance-aware listwise ranking generalizes but benefits from the split. The few-shot fact detection at over 90% accuracy is a practical plus. Cost analysis, error analysis, and the code/data release are all there.

The soft spots are real but not disqualifying. The biggest one is the gold rankings on AlignBench_Minos and GaokaoBench_Minos: three annotators plus GPT-4o, discussion-based consensus, and no reported inter-annotator agreement. The 98.39% modification rate for GPT-4o's initial rankings tells us GPT-4o alone wasn't trusted, but it doesn't establish that the consensus is a stable target. Because the method's largest wins appear on exactly these datasets, and Table 2 reports no variance, I'd want IAA and bootstrapped confidence intervals before leaning hard on those margins. That said, this is a weakness in evidence strength, not a fatal flaw: all baselines are measured against the same gold, so the relative ranking is still informative, just noisier than claimed.

Also worth flagging: Section 4.2 says GPT-4 was the base model for all LLM methods, while Section 3.2, Table C1, and the ablation use GPT-4o. That looks like a leftover from an earlier draft, but it needs fixing. The underperformance on TREC-DL-NF_S5 on Kendall and Spearman vs plain listwise is a caveat, not a kill shot; the paper's explanation about 2-4 candidates and misordering sensitivity is plausible but deserves more than a paragraph. Finally, the interpretability claim is qualitative — a small human check of whether key points and silver instances actually help would substantiate it.

There is no formal circularity: the evaluation is against human annotations, and the silver instances are generated by the same model family used for ranking, which is a mild self-confirmation risk but not a load-bearing one. Overall, this is a solid, honest empirical paper aimed at the LLM-evaluation subfield. It deserves a serious referee, and I would accept it conditionally, with the main requests being inter-annotator agreement, variance estimates on the self-built datasets, and a clarified base-model statement.

Referee Report

3 major / 5 minor

Summary. The paper proposes MinosEval, a two-stage LLM-based evaluation framework for open-ended QA. In the first stage, an LLM classifies each question as factoid or non-factoid. For factoid questions, the method extracts key points from the reference answer and computes an average NLI entailment-minus-contradiction score for each candidate response against those key points (Eqs. 2-3). For non-factoid questions, it generates five quality-graded silver instances and uses an LLM ranker to produce a listwise ordering informed by those instances (Eq. 4). The method is evaluated on four datasets: two sampled English datasets (ANTIQUE_S5, TREC-DL-NF_S5) and two newly constructed Chinese datasets (AlignBench_Minos, GaokaoBench_Minos) with six candidate responses per question and consensus gold rankings. The paper reports Kendall's tau, Spearman's rho, and RBO against the gold rankings, together with ablations, robustness checks, cost comparisons, and error analyses.

Significance. The central idea—that factoid and non-factoid open-ended questions require different evaluation protocols—is well motivated, and MinosEval gives a concrete, fully automated operationalization. The public release of the datasets, code, and prompt templates is a practical strength. The ablation study is useful, particularly the comparisons of key-point scoring against direct NLI, the directionality analysis for premise/hypothesis, and the sweep over different LLM and NLI backends. I do not find a formal circularity in the evaluation: the headline correlations are computed against human annotations, and no constants are fitted to the test data. The more concrete version of that concern is that GPT-4o participated in producing both the gold rankings and the evaluation artifacts (key points and silver instances), so the risk of model-family self-confirmation is real. The empirical support is weakened by the absence of inter-annotator agreement for the gold rankings and by the lack of variance or significance estimates on the two self-built datasets, so the reported gains should be treated as promising but not yet fully established.

major comments (3)

[4.1] The gold rankings for AlignBench_Minos and GaokaoBench_Minos are described as a consensus among one PhD student, two Master's students, and GPT-4o, reached after independent annotation and discussion, with 98.39% of GPT-4o's initial rankings modified by humans. No inter-annotator agreement is reported. This is load-bearing because every headline comparison in Table 2 is a correlation against this single consensus ranking. The 98.39% statistic shows that GPT-4o alone was not trusted, but it does not quantify agreement among the three human annotators or between the human consensus and GPT-4o. Without a reliability estimate (e.g., pairwise Kendall's tau among annotators or Krippendorff's alpha), it is impossible to tell whether the margins in Table 2 (e.g., Spearman 54.89 vs 51.46 on AlignBench_Minos) exceed the noise in the gold standard. Please report human-human and human-GPT-4o agreement and, if feasible, recompute the main comparisons against a human-only gold ranking or a version that excludes GPT-4o's contribution.
[Tables 2 and B5] Tables 2 and B5 report only point estimates for each method on AlignBench_Minos and GaokaoBench_Minos, with no confidence intervals, bootstrap errors, or significance tests. On GaokaoBench_Minos the gap between MinosEval and plain listwise is small (Spearman 65.77 vs 64.66), and several RBO differences are under one point. Because the gold ranking is itself a single consensus, the reader cannot tell whether these differences are meaningful. Please provide variability estimates across items (e.g., bootstrap over questions), report paired significance tests for Kendall and Spearman, or explain why such tests are not applicable.
[4.2 vs 3.2.1 and Table C2] Section 4.2 states that 'GPT-4 was used as the base model' for all LLM-based methods in the main experiments, while Section 3.2.1 and Table 6 specify GPT-4o for fact detection and Table C2 lists GPT-4o-2024-08-06 for the MinosEval instantiations. If the main experiments used GPT-4o for MinosEval but GPT-4 for the baselines, the comparisons in Tables 2 and 3 are not fully controlled. If GPT-4 was used for all methods, the descriptions of the fact-detection, key-point, and silver-instance modules are inconsistent with the implementation details. Please clarify the exact model used for each module in the main experiments and, if the models differ, rerun the baselines with the same model or report the per-model comparison explicitly.

minor comments (5)

[Table B3] In Table B3, the RBO(p=0.9) value for DeepSeekV2.5+mDeBERTa† is 52.28, far outside the range of the surrounding rows (all near 86), which appears to be a typographical or transcription error and should be corrected.
[Table B5] Table B5 has typos in the header and surrounding text: 'Fctoid / Non-Factoid' and 'GaokaoBnech_Minos', and the cell formatting makes several values difficult to read; please clean up the table and run-on cells.
[2.2] In Section 2.2, 'BERTScore (Zhang et al.) easuring semantic similarity' contains a typo ('easuring' should be 'measuring'); the same section uses 'golden standard' where 'gold standard' is more conventional.
[3.2.3 and Figure 2] The term 'sliver' appears instead of 'silver' in Section 3.2.3 and in the 'SliverInstances' label in Figure 2; this should be made consistent throughout, including in the prompts shown in the appendix.
[4.5] The typo 'AlginBench_Minos' appears in Sections 4.5, 4.7, and Figure 3; please standardize the dataset name to AlignBench_Minos.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; MinosEval's ranking claims are tested against independent human-annotated gold rankings.

full rationale

The paper's central claim is empirical: MinosEval rankings are compared with human-annotated gold rankings on four datasets, including externally sourced ANTIQUE_S5 and TREC-DL-NF_S5. No model parameter is fitted to those gold rankings, and no equation in Sections 3.2.2-3.2.3 defines the target ranking in terms of MinosEval's own components. Formulas 2 and 4 are operational scoring and ranking procedures, not constraints derived from the human labels used for evaluation. The key points and silver instances are generated from question/reference pairs independently of the gold rankings, and the baselines use the same LLM families. The fact-detection accuracy is measured against the authors' own annotation protocol, which is a normal labeling procedure rather than a derivation of the result from its inputs. I found no load-bearing self-citation, no fitted parameter renamed as a prediction, and no uniqueness theorem imported from the authors' prior work. Concerns about the small annotation team, the 98.39% human modification rate, and possible LLM self-preference are validity or robustness issues, not circularity, because the ground truth is human judgment external to the method.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric parameters are fitted to the evaluation data; the method uses prompts, an off-the-shelf NLI model, and LLM-generated scaffolds. The load-bearing assumptions concern the reliability of the reference answer as the source of evaluation criteria, the validity of the binary question taxonomy, and the quality of LLM-generated key points and silver instances. No new physical or formal entities are introduced.

assumptions (4)

domain assumption The binary factoid/non-factoid distinction is a valid and sufficient partition of open-ended questions for evaluation purposes.
Section 3.1 defines the taxonomy and all samples are routed into one of two clusters. The Limitations section admits that boundary cases can fit both categories, so the partition may be too coarse for mixed questions.
domain assumption LLM-extracted key points from the reference answer cover the criteria humans use to judge factoid answers.
Formula 2 computes scores only from the extracted key points K. If extraction misses important content or adds noise, the resulting ranking diverges from human judgment. No coverage validation against human annotations is reported.
domain assumption NLI entailment-minus-contradiction probabilities are a valid proxy for whether a response covers a key point.
Formulas 2 and 3 define the factoid score entirely through mDeBERTa-v3-base-mnli-xnli. The paper checks premise/hypothesis direction but does not calibrate NLI outputs against human coverage judgments, and the Limitations section notes the general NLI model may be limited in specialized scenarios.
domain assumption LLM-generated silver instances form a reliable quality ladder for listwise ranking of non-factoid answers.
Formula 4 feeds five generated quality levels into the ranker. If the generated Excellent to Bad ladder is miscalibrated, the ranking anchors are wrong. Error analysis lists low-quality instances as one error category but does not quantify it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs." pith.science (2026). https://pith.science/paper/CDYUXVWH

@misc{pith2026250615215,
  author       = {Pith},
  title        = {Pith review of: MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CDYUXVWH}},
  note         = {Machine review of arXiv:2506.15215}
}

read the original abstract

Open-ended question answering (QA) is a key task for evaluating the capabilities of large language models (LLMs). Compared to closed-ended QA, it demands longer answer statements, more nuanced reasoning processes, and diverse expressions, making refined and interpretable automatic evaluation both crucial and challenging. Traditional metrics like ROUGE and BERTScore struggle to capture semantic similarities due to different patterns between model responses and reference answers. Current LLM-based evaluation approaches, such as pairwise or listwise comparisons of candidate answers, lack intuitive interpretability. While pointwise scoring of each response provides some descriptions, it fails to adapt across different question contents. Most notably, existing methods overlook the distinction between factoid and non-factoid questions. To address these challenges, we propose \textbf{MinosEval}, a novel evaluation method that first distinguishes open-ended questions and then ranks candidate answers using different evaluation strategies. For factoid questions, it applies an adaptive key-point scoring strategy, while for non-factoid questions, it uses an instance-aware listwise ranking strategy. Experiments on multiple open-ended QA datasets, including self-built ones with more candidate responses to complement community resources, show that MinosEval better aligns with human annotations and offers more interpretable results.

Figures

Figures reproduced from arXiv: 2506.15215 by the authors.

**Figure 2.** The MinosEval Workflow: Distinguishing Factoid vs. Non-Factoid Questions, Adaptive Key-Point [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Comparison of Computational Cost (Scaled [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 29 canonical work pages

[1]

R Agustianingsih and Ali Mahmudi. 2019. How to design open-ended questions?: Literature review. In Journal of Physics: Conference Series, volume 1320, page 012003. IOP Publishing

work page 2019
[2]

Maryam Amirizaniani, Elias Martin, Maryna Sivachenko, Afra Mashhadi, and Chirag Shah. 2024. Can llms reason like humans? assessing theory of mind reasoning in llms for open-ended questions. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 34--44

2024
[3]

Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037

arXiv 2022
[4]

Valeriia Bolotova-Baranova, Vladislav Blinov, Sofya Filippova, Falk Scholer, and Mark Sanderson. 2023 a . https://doi.org/10.18653/v1/2023.acl-long.290 W iki H ow QA : A comprehensive benchmark for multi-document non-factoid question answering . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

work page doi:10.18653/v1/2023.acl-long.290 2023
[5]

Valeriia Bolotova-Baranova, Vladislav Blinov, Sofya Filippova, Falk Scholer, and Mark Sanderson. 2023 b . Wikihowqa: A comprehensive benchmark for multi-document non-factoid question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5291--5314

work page 2023
[6]

Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. 2023. Elo uncovered: Robustness and best practices in language model evaluation. In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 339--352

work page 2023
[7]

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 1877--1901

2020
[8]

Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, and Tal Schuster. 2022. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. arXiv preprint arXiv:2202.07654

arXiv 2022

Show all 67 references

[9]

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297

2024 arXiv
[10]

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

2024
[11]

OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass

2023
[12]

Jonathan Cook, Tim Rockt \"a schel, Jakob Foerster, Dennis Aumiller, and Alex Wang. 2024. Ticking all the boxes: Generated checklists improve llm evaluation and generation. arXiv preprint arXiv:2410.03608

2024 arXiv
[13]

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. https://arxiv.org/abs/2102.07662 Overview of the trec 2020 deep learning track . Preprint, arXiv:2102.07662

2021 arXiv
[14]

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820

2020 arXiv
[15]

Vittoria Dentella, Fritz G \"u nther, Elliot Murphy, Gary Marcus, and Evelina Leivada. 2024. Testing ai on language comprehension tasks reveals insensitivity to underlying meaning. Scientific Reports, 14(1):28083

2024
[16]

Alexander R Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391--409

2021
[17]

Vikas Gaikwad and Archana Patil. 2023. Factoid question answering system using knowledge graph. In 2023 7th International Conference On Computing, Communication, Control And Automation (ICCUBEA), pages 1--4. IEEE

2023
[18]

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

2024 arXiv
[19]

Google. 2023. https://gemini.google.com/ Hello gemini . Technical report, OpenAI

2023
[20]

Romain Harang. 2024. Beyond rouge: Applying an elo algorithm to rank model performances in summarization. In Proceedings of the 30th Annual Meeting of the Association for Natural Language Processing, pages 2799--2804

2024
[21]

Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W Bruce Croft. 2020. Antique: A non-factoid question answering benchmark. In Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14--17, 2020, Proceedings, Part II...

2020
[22]

Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. Refchecker: Reference-based fine-grained hallucination checker and benchmark for large language models. arXiv preprint arXiv:2405.14486

2024 arXiv
[23]

Kelvin Jiang, Dekun Wu, and Hui Jiang. 2019. Freebaseqa: A new factoid qa data set matching trivia-style question-answer pairs with freebase. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te...

2019
[24]

Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Leong, Jia Qing Tan, Enamul Hoque, and Shafiq Joty. 2022. Opencqa: Open-ended question answering with charts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11817--11837

2022
[25]

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535

2024 arXiv
[26]

Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. https://doi.org/10.18653/v1/2021.naacl-main.393 Hurdles to progress in long-form question answering . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...

2021 doi
[27]

K Lakshmi and S Sindhu. 2023. A study on various approaches towards non-factoid question answering systems. In 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), pages 780--788. IEEE

2023
[28]

Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023 a . Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470

2023 arXiv
[29]

Qintong Li, Leyang Cui, Lingpeng Kong, and Wei Bi. 2023 b . Collaborative evaluation: Exploring the synergy of large language models and humans for open-ended generation evaluation. arXiv preprint arXiv:2310.19740

2023 arXiv
[30]

Yu Li, Shenyu Zhang, Rui Wu, Xiutian Huang, Yongrui Chen, Wenhao Xu, Guilin Qi, and Dehai Min. 2024 a . Mateval: A multi-agent discussion framework for advancing open-ended text evaluation. In International Conference on Database Systems for Advanced Applications, pages 415--4...

2024
[31]

Zongxia Li, Ishani Mondal, Yijun Liang, Huy Nghiem, and Jordan Lee Boyd-Graber. 2024 b . Pedants: Cheap but effective and interpretable answer equivalence. arXiv preprint arXiv:2402.11161

2024 arXiv
[32]

Holistic evaluation of language models

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research
[33]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

2004
[34]

Pei-Ying Lin, Erick Chandra, and Jane Yung-jen Hsu. 2024. Asmr: Aggregated semantic matching retrieval unleashing commonsense ability of llm through open-ended question answering. In Proceedings of the AAAI Symposium Series, volume 3, pages 157--166

2024
[35]

Chuang Liu, Linhao Yu, Jiaxuan Li, Renren Jin, Yufei Huang, Ling Shi, Junhui Zhang, Xinmeng Ji, Tingting Cui, Tao Liu, et al. 2024 a . Openeval: Benchmarking chinese llms across capability, alignment and safety. arXiv preprint arXiv:2403.12316

2024 arXiv
[36]

Mianxin Liu, Weiguo Hu, Jinru Ding, Jie Xu, Xiaoyang Li, Lifeng Zhu, Zhian Bai, Xiaoming Shi, Benyou Wang, Haitao Song, et al. 2024 b . Medbench: A comprehensive, standardized, and reliable benchmarking system for evaluating chinese medical large language models. Big Data Mini...

2024
[37]

Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023 a . Webglm: Towards an efficient web-enhanced question answering system with human preferences. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery ...

2023
[38]

Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. 2023 b . Alignbench: Benchmarking chinese alignment of large language models. arXiv preprint arXiv:2311.18743

2023 arXiv
[39]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 c . G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511--2522

2023
[40]

Linhao Luo, Trang Vu, Dinh Phung, and Reza Haf. 2023. Systematic assessment of factual knowledge in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13272--13286

2023
[41]

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empiric...

2023
[42]

Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. 2024. Open-llm-leaderboard: From multi-choice to open-style questions for llms evaluation, benchmark, and arena. arXiv preprint arXiv:2406.07545

2024 arXiv
[43]

Thuy Thi-Nhu Ngo, Howard Hao-Jan Chen, and Kyle Kuo-Wei Lai. 2024. The effectiveness of automated writing evaluation in efl/esl writing: a three-level meta-analysis. Interactive learning environments, 32(2):727--744

2024
[44]

OpenAI. 2022. https://openai.com/blog/chatgpt Hello chatgpt . Technical report, OpenAI

2022
[45]

OpenAI. 2023. https://openai.com/blog/new-models-and-developer-products-announced-at-devday New models and developer products announced at devday . Technical report

2023
[46]

OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o . Technical report, OpenAI

2024
[47]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[48]

Lin Shi, Chiyu Ma, Wenhua Liang, Weicheng Ma, and Soroush Vosoughi. 2024. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. arXiv preprint arXiv:2406.07791

2024
[49]

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions o...

2023
[50]

Mingxu Tao, Dongyan Zhao, and Yansong Feng. 2024. Chain-of-discussion: A multi-model framework for complex evidence-based question answering. arXiv preprint arXiv:2402.16313

2024 arXiv
[51]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

2023 arXiv
[52]

Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. 2024. Foundational autoraters: Taming large language models for better automatic evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pag...

2024
[53]

Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. 2024 a . https://arxiv.org/abs/2310.03304 Learning personalized alignment for evaluating open-ended text generation . Preprint, arXiv:2310.03304

2024 arXiv
[54]

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926

2023 arXiv
[55]

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024 b . Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574

2024 arXiv
[56]

Yuqi Wang, Lyuhao Chen, Songcheng Cai, Zhijian Xu, and Yilun Zhao. 2024 c . https://doi.org/10.18653/v1/2024.emnlp-main.815 Revisiting automated evaluation for long-form table question answering . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

2024 doi
[57]

Yuqi Wang, Lyuhao Chen, Songcheng Cai, Zhijian Xu, and Yilun Zhao. 2024 d . Revisiting automated evaluation for long-form table question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14696--14706

2024
[58]

William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS), 28(4):1--38

2010
[59]

Zhichao Yan, Jiapu Wang, Jiaoyan Chen, Xiaoli Li, Ru Li, and Jeff Z Pan. 2024. Atomic fact decomposition helps attributed question answering. arXiv preprint arXiv:2410.16708

2024 arXiv
[60]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024 a . Qwen2 technical report. arXiv preprint arXiv:2407.10671

2024 arXiv
[61]

Sihui Yang, Keping Bi, Wanqing Cui, Jiafeng Guo, and Xueqi Cheng. 2024 b . Linkage: Listwise ranking among varied-quality references for non-factoid qa evaluation via llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6985--7000

2024
[62]

Peiran Yao and Denilson Barbosa. 2024. Accurate and nuanced open-qa evaluation through textual entailment. arXiv preprint arXiv:2405.16702

2024 arXiv
[63]

Ping Yu, Tianlu Wang, Olga Golovneva, Badr AlKhamissi, Siddharth Verma, Zhijing Jin, Gargi Ghosh, Mona Diab, and Asli Celikyilmaz. 2023. Alert: Adapt language models to reasoning tasks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

2023
[64]

Bertscore: Evaluating text generation with bert

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations
[65]

Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474

2023 arXiv
[66]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[67]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

R Agustianingsih and Ali Mahmudi. 2019. How to design open-ended questions?: Literature review. In Journal of Physics: Conference Series, volume 1320, page 012003. IOP Publishing

work page 2019

[2] [2]

Maryam Amirizaniani, Elias Martin, Maryna Sivachenko, Afra Mashhadi, and Chirag Shah. 2024. Can llms reason like humans? assessing theory of mind reasoning in llms for open-ended questions. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 34--44

2024

[3] [3]

Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037

arXiv 2022

[4] [4]

Valeriia Bolotova-Baranova, Vladislav Blinov, Sofya Filippova, Falk Scholer, and Mark Sanderson. 2023 a . https://doi.org/10.18653/v1/2023.acl-long.290 W iki H ow QA : A comprehensive benchmark for multi-document non-factoid question answering . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

work page doi:10.18653/v1/2023.acl-long.290 2023

[5] [5]

Valeriia Bolotova-Baranova, Vladislav Blinov, Sofya Filippova, Falk Scholer, and Mark Sanderson. 2023 b . Wikihowqa: A comprehensive benchmark for multi-document non-factoid question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5291--5314

work page 2023

[6] [6]

Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. 2023. Elo uncovered: Robustness and best practices in language model evaluation. In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 339--352

work page 2023

[7] [7]

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 1877--1901

2020

[8] [8]

Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, and Tal Schuster. 2022. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. arXiv preprint arXiv:2202.07654

arXiv 2022