Pith. sign in

REVIEW 1 major objections 1 minor 38 references

SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators

T0 review · 1 major / 1 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper converts 1,630 authentic peer-review reports into four quality dimensions and shows fine-tuning on them cuts an evaluator's average score error roughly in half.

desk verdict A valuable dataset resource whose headline evaluator claim collapses against a trivial majority-class baseline. read the letter →

arxiv 2608.07641 v1 pith:MD5M3LKC submitted 2026-08-07 cs.CL

Yuheng Zhang , Yuanchun Wang , Fanjin Zhang , Ruyu Zhao , Juanzi Li , Jie Tang , Jing Zhang This is my paper · ORCID
classification cs.CL
keywords surveyevaluationLLM-as-a-judgepeerreviewbenchmarkdatasethumanalignmentknowledgeaugmentationautomaticQwen3-32B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that off-the-shelf LLM-as-a-judge evaluation of survey papers is misaligned with what human reviewers actually judge, and that alignment can be measured and improved. It introduces SurveyReview, a dataset of 675 surveys with 1,630 peer-review reports annotated into four dimensions—readability, structure, comprehensiveness, criticalness—each with a numeric score and a supporting rationale. It also introduces SurveyAlign, a Qwen3-32B model fine-tuned with LoRA on this data, which on the test set cuts average MSE from 2.28 to 1.38 and MAE from 1.15 to 0.69 compared with GPT-5.2 zero-shot prompting. If the annotation conversion is faithful, the benchmark offers a reproducible target for future survey evaluators.

What carries the argument

The central object is the four-dimensional score–rationale representation derived from free-form peer reviews: annotators read each review and assign, for every nonzero dimension among Readability, Structure, Comprehensiveness, and Criticalness, a score from {−2, −1, +1, +2} with the most relevant sentences as supporting rationale. This structured ground truth powers two things: supervised fine-tuning of the evaluator, and the alignment metrics themselves—MSE/MAE for scores, a Reason Quality Score for rationales judged by GPT-5.2, and the composite Human-aligned Score. SurveyAlign realizes the mapping with four dimension-specific LoRA-fine-tuned Qwen3-32B sub-models, prepending a reference-expanded citation neighborhood (Comp-KA) for comprehensiveness and a temporally compressed summary (Crit-KA) plus DPO for criticalness, then fusing five sampled outputs by majority vote at inference.

What would settle it

Collect the original reviewers' own dimension-level ratings, or ask a sample of the original reviewers to rate their own reviews with the four-dimension rubric, and compare them with the annotator-derived ground truth. If agreement between the original reviewers and the annotations is no better than agreement between the automated evaluators and the annotations, then the benchmark measures annotator alignment rather than reviewer alignment, and the reported error reductions do not establish the paper's claim.

Watch

Extended reading notes

Core claim

The central claim is that authentic peer-review reports can be converted into structured, four-dimensional score–rationale annotations, and that supervised fine-tuning on those annotations produces an evaluator whose scores align with human reviewers substantially better than zero-shot prompting. SurveyAlign, built from Qwen3-32B with LoRA, uses dimension-specific sub-models, reference-expansion knowledge augmentation for comprehensiveness and criticalness, and majority-vote aggregation at inference. On the held-out test set it reports average MSE 1.38 and MAE 0.69 across the four dimensions, versus 2.28 and 1.15 for GPT-5.2 prompting, and the highest composite Human-aligned Score (0.74). The paper presents this as the first multi-dimensional reviewer-aligned benchmark and a strong baseline for the task.

Load-bearing premise

The load-bearing premise is that the scores and rationales annotators extract from free-form review comments faithfully represent what the original human reviewers meant on each of the four dimensions; the paper does not validate these annotations against the reviewers' own ratings.

Editorial extensions

If this is right

  • SurveyAlign attains the lowest MSE and MAE on every one of the four dimensions and the highest HAS (0.74) among all evaluated methods, including GPT-5.2, Gemini-3-pro, and Claude-Opus-4.5.
  • Task-specific supervision is necessary for reviewer-level consistency: every zero-shot LLM judge, even the strongest, lags behind on score alignment.
  • Knowledge augmentation helps comprehensiveness (MSE 1.36 to 1.26 with Comp-KA) but helps criticalness only when combined with DPO (MSE 1.96 to 1.52); Crit-KA alone does not improve the dimension.
  • RQS is judge-sensitive, yet SurveyAlign remains the top-ranked method under three different rationale judges and across all three data sources, so the system-level conclusion is stable.
  • The paper-level, source-stratified train/test split and fixed evaluation protocol allow future evaluators to be compared on the same MSE/MAE/RQS/HAS scales.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the free-form-to-score conversion is accepted, the same annotation pipeline could be applied to other genres—research papers, technical reports—yielding reviewer-aligned benchmarks beyond surveys.
  • The criticalness result suggests a general principle: giving an evaluator more external context does not help unless the training objective also rewards the target reasoning behavior.
  • Per-dimension MSE profiles give a diagnostic: an evaluator's error pattern identifies which aspect of reviewing it fails to model, enabling targeted fixes instead of tuning one aggregate score.
  • A natural next test is whether improving alignment on this benchmark transfers to downstream survey generation, for instance by using the evaluator's scores as a reward signal for iterative revision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper introduces SurveyReview, a benchmark and dataset for evaluating automatic survey-paper evaluators. It collects 675 survey papers with 1,630 peer-review reports from F1000Research, MOPRD, and OpenReview, and annotates each review into four dimensions (Readability, Criticalness, Comprehensiveness, Structure) with numeric scores from {-2,-1,+1,+2} and supporting rationales. The authors also propose SurveyAlign, a Qwen3-32B-based evaluator fine-tuned with LoRA and knowledge-augmented inputs, and report that it reduces average MSE from 2.28 to 1.38 and MAE from 1.15 to 0.69 compared with zero-shot GPT-5.2. The paper also defines a combined Human-aligned Score (HAS) integrating score error and rationale quality.

Significance. If the central claims hold, the dataset would be a useful resource: it is the first multi-dimensional benchmark derived from authentic peer-review reports, and it ships standardized splits, an evaluation protocol, and an open-source code/data release. The annotation reliability check (Krippendorff's alpha 0.74) and the robustness analyses (source-wise, multi-judge RQS, HAS weighting) are valuable additions that go beyond what many benchmark papers provide. However, the headline result — that SurveyAlign substantially improves reviewer alignment over zero-shot GPT-5.2 — is not supported by the evidence as presented, because no trivial baseline is included. The benchmark's claim to measure 'reviewer alignment' also rests on an unvalidated conversion from free-form review text to four-dimensional scores by third-party annotators.

major comments (1)
  1. [Appendix A] The annotation space initially includes 0 ('dimension not mentioned') and -3 ('unjudgeable'), which are then excluded from training and evaluation. This exclusion is not justified. If a review does not mention a dimension, that absence may be informative signal for a dimension-specific evaluator, and excluding it arbitrarily removes a part of the review signal. The authors should discuss this decision and, if possible, provide a secondary analysis that includes 0 as a valid label.
minor comments (1)
  1. [§5.2.2] The description of Crit-KA thresholding says 'α and β are fixed hyperparameters' but does not give their values in the main text or in Appendix F. Without concrete values the procedure is not reproducible; please report them.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild circularity in the RQS/HAS rationale-evaluation component; the central score-alignment claim is a clean held-out comparison.

  1. self definitional [Appendix G; Eq. (8); Section 5.1 (SFT targets)]
    "We compute the Reason Quality Score (RQS) using GPT-5.2 as a meta-evaluator to assess semantic consistency between predicted rationales and human-written reference rationales."

    RQS is defined as average semantic consistency between generated rationales and 'human-written reference rationales' (Appendix G), and HAS = 0.75·Align + 0.25·RQS (Eq. 8). Those reference rationales come from the same Section 4.3 annotation protocol whose (score, rationale) pairs are the SurveyAlign supervised fine-tuning targets (Section 5.1: 'the output includes a numeric score paired with a human-written rationale'). Thus the RQS component measures how well the model reproduces the annotation style it was trained to emit, so part of SurveyAlign's HAS advantage is self-consistency with the training target rather than independent evidence of reasoning fidelity.

full rationale

The central empirical claim—SurveyAlign reduces average MSE from 2.28 to 1.38 and MAE from 1.15 to 0.69 against GPT-5.2—is not circular: SurveyAlign is fine-tuned on the training split and evaluated on the paper-level test split; no test paper appears in training and no test label is fitted. The only self-referential element is the RQS/HAS pipeline: RQS scores generated rationales against reference rationales produced by the same annotation rubric used to build the SFT targets, making the HAS metric partially a measure of self-consistency with the training distribution. This is a mild metric-design circularity, not a forced derivation of the main result. The AMiner/OAG citations used for knowledge augmentation are external infrastructure, not load-bearing self-citations. The skewed-label distribution and the absence of a majority-class baseline is a genuine robustness concern about the strength of the improvement, but it is a statistical comparison issue rather than circularity. Similarly, the fact that 'ground-truth' scores are annotator interpretations of review comments rather than the original reviewers' own ratings is a construct-validity assumption, not a logical circle.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a small number of hand-chosen weighting parameters and on the assumption that annotator-converted scores are faithful proxies for reviewer judgments. The benchmark does not introduce new physical entities, but the metric definitions and label construction carry domain assumptions that are not independently verified.

free parameters (4)
  • HAS weights alpha1, alpha2 = 0.4, 0.6
    Chosen by the authors to combine MSE and MAE into the alignment score in Equation (7); no data-driven justification is given, though sensitivity to beta weights is explored in Appendix I.3.
  • HAS weights beta1, beta2 = 0.75, 0.25
    Chosen to weight alignment more heavily than RQS in the final Human-aligned Score; sensitivity is shown in Table 8, but the default values are used for the headline result.
  • Crit-KA thresholds alpha, beta = not specified
    Adaptive percentile thresholds for citation count and publication year in Appendix F.2; exact percentile values are not reported, so the construction is not fully reproducible.
  • Token budget B for Crit-KA = not specified
    Budget for temporal block summarization in Appendix F.2; no numerical value is given in the main text.
assumptions (3)
  • domain assumption Annotator-derived scores faithfully represent original reviewers' judgments on the four dimensions.
    The benchmark labels are produced by trained annotators converting free-form review comments into scores (Section 4.3); this conversion is not validated against explicit reviewer scores.
  • domain assumption The four dimensions (Readability, Criticalness, Comprehensiveness, Structure) are the relevant axes of survey quality and are independent across reviews.
    The paper selects these dimensions from reviewing practice but provides no factor analysis or evidence of coverage or completeness of the dimension set (Section 4.3, Appendix A.1).
  • domain assumption GPT-5.2 as a meta-judge yields valid Reason Quality Scores (RQS).
    RQS is computed by GPT-5.2 consistency scoring, validated on only 50 pairs with Pearson r=0.804 (Appendix G), a small sample that may not generalize.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators." pith.science (2026). https://pith.science/paper/MD5M3LKC

@misc{pith2026260807641,
  author       = {Pith},
  title        = {Pith review of: SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MD5M3LKC}},
  note         = {Machine review of arXiv:2608.07641}
}
read the original abstract

The rapid advancement of large language models has transformed survey writing from a months-long manual effort into an automated process. As generation scales, reliable evaluation becomes the bottleneck, and LLMs are increasingly used as survey evaluators. However, existing approaches largely rely on off-the-shelf LLM-as-a-judge methods without systematic alignment to human reviewers, and there remains a lack of systematic frameworks for quantifying alignment with human reviewers. To address this gap, we propose SurveyReview, a reviewer-aligned, multi-dimensional benchmark and dataset for survey evaluation. We collect and annotate 675 survey papers with 1,630 review reports. We structure authentic peer-review reports by converting free-form comments into four-dimensional scores (Readability, Criticalness, Comprehensiveness, Structure) paired with supporting rationales. We further release standardized train/test splits and an evaluation protocol to measure alignment between automatic evaluators and human reviewers. To validate the benchmark, we develop SurveyAlign, a strong baseline evaluator by fine-tuning Qwen3-32B with LoRA on our annotated data, augmented with external knowledge for knowledge-intensive dimensions. On the test set, SurveyAlign substantially improves reviewer alignment over prompt-based judging with GPT-5.2, reducing average MSE from 2.28 to 1.38 and MAE from 1.15 to 0.69 across all four dimensions. Our contributions are twofold: (1) we establish the first multi-dimensional, reviewer-aligned dataset with a reproducible evaluation framework for survey reviewing; (2) we develop a strong baseline evaluator that substantially improves alignment with human reviewers, providing a competitive reference for future research. Our code and data are available at https://surveyreview.github.io

Figures

Figures reproduced from arXiv: 2608.07641 by the authors.

Figure 1
Figure 1. SurveyReview Benchmark: Evaluating to what ex [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of SurveyReview. Left: dataset construction. Middle: the workflow of a single survey evaluator producing [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 12 canonical work pages

  1. [1]

    2008–2026. GROBID. https://github.com/kermitt2/grobid. swh:1:dir:dab86b296e3c3216e2241968f0d63b68e8209d3c

    work page 2008

  2. [2]

    Aradhye Agarwal, Ayan Sengupta, and Tanmoy Chakraborty. 2025. The Art of Scaling Test-Time Compute for Large Language Models. arXiv:2512.02008 [cs.CL] https://arxiv.org/abs/2512.02008

    arXiv 2025

  3. [3]

    Anthropic. 2025. Introducing Claude Opus 4.5. https://www.anthropic.com/ news/claude-opus-4-5

    work page 2025

  4. [4]

    Jing Chen, Zhiheng Yang, Yixian Shen, Jie Liu, Adam Belloum, Chrysa Papagainni, and Paola Grosso. 2025. SurveyGen-I: Consistent Scientific Survey Generation with Evolving Plans and Memory-Guided Writing. arXiv:2508.14317 [cs.CL] https://arxiv.org/abs/2508.14317

    work page Pith review arXiv 2025

  5. [5]

    DeepSeek-AI. 2025. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556 [cs.CL] https://arxiv.org/abs/2512.02556 SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea

    arXiv 2025

  6. [6]

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. arXiv:2506.11763 [cs.CL] https://arxiv.org/abs/2506.11763

    arXiv 2025

  7. [7]

    Kehua Feng, Keyan Ding, Hongzhi Tan, Kede Ma, Zhihua Wang, Shuangquan Guo, Yuzhou Cheng, Ge Sun, Guozhou Zheng, Qiang Zhang, and Huajun Chen. 2025. Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition. arXiv:2404.08008 [cs.LG] https://arxiv.org/abs/2404. 08008

    arXiv 2025

  8. [8]

    Google DeepMind. 2025. Gemini 3 Pro Model Card. https://deepmind.google/ models/model-cards/gemini-3-pro/

    work page 2025

Show all 38 references
  1. [9]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 [cs.CL] https://arxiv.org/abs/2106.09685

    2021 arXiv

  2. [10]

    Aleksi Huotala, Miikka Kuutila, Olli-Pekka Turtio, and Mika Mäntylä. 2025. AISysRev – LLM-based Tool for Title-abstract Screening. arXiv:2510.06708 [cs.SE] https://arxiv.org/abs/2510.06708

    2025 arXiv

  3. [11]

    Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, and Kai Jia. 2025. Re- portBench: Evaluating Deep Research Agents via Academic Survey Tasks. arXiv:2508.15804 [cs.CL] https://arxiv.org/abs/2508.15804

    2025 arXiv

  4. [12]

    Xun Liang, Jiawei Yang, Yezhaohui Wang, Chen Tang, Zifan Zheng, Shichao Song, Zehao Lin, Yebin Yang, Simin Niu, Hanyu Wang, Bo Tang, Feiyu Xiong, Keming Mao, and Zhiyu li. 2025. SurveyX: Academic Survey Automation via Large Language Models. arXiv:2502.14776 [cs.CL] https://arx...

    2025 arXiv

  5. [13]

    Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. InText Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013/

    2004

  6. [14]

    Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. 2023. MOPRD: A multidisciplinary open peer review dataset.Neural Computing and Applications35, 34 (Sept. 2023), 24191–24206. doi:10.1007/s00521-023-08891-5

    2023 doi

  7. [15]

    OpenAI. 2025. Introducing GPT-5.2. https://openai.com/index/introducing-gpt- 5-2/

    2025

  8. [16]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. InProceedings of the 40th Annual Meeting on Association for Computational Linguistics(Philadelphia, Penn- sylvania)(ACL ’02). Association for Comp...

    2002

  9. [17]

    Vik Paruchuri. 2025. Marker. https://github.com/VikParuchuri/marker. GitHub repository

    2025

  10. [18]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290 [cs.LG] https://arxiv.org/ abs/2305.18290

    2024 arXiv

  11. [19]

    Howcroft, Ondřej Plátek, and Adarsa Sivaprasad

    Patrícia Schmidtová, Saad Mahamood, Simone Balloccu, Ondřej Dušek, Albert Gatt, Dimitra Gkatzia, David M. Howcroft, Ondřej Plátek, and Adarsa Sivaprasad

  12. [20]

    Xiaofeng Shi, Qian Kou, Yuduo Li, Ning Tang, Jinxin Xie, Longbin Yu, Songjing Wang, and Hua Zhou. 2025. SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation. arXiv:2506.12689 [cs.AI] https://arxiv.org/abs/2506. 12689

    2025 arXiv

  13. [21]

    Weihang Su, Anzhe Xie, Qingyao Ai, Jianming Long, Xuanyi Chen, Jiaxin Mao, Ziyi Ye, and Yiqun Liu. 2026. SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation. arXiv:2508.15658 [cs.CL] https://arxiv.org/abs/ 2508.15658

    2026 arXiv

  14. [22]

    Hao Sun, Yunyi Shen, and Mihaela van der Schaar. 2025. OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models. arXiv:2505.21537 [cs.CY] https://arxiv.org/abs/2505.21537

    2025 arXiv

  15. [23]

    Zhaojun Sun, Xuzhou Zhu, Xuanhe Zhou, Xin Tong, Shuo Wang, Jie Fu, Guoliang Li, Zhiyuan Liu, and Fan Wu. 2025. SurveyBench: Can LLM(-Agents) Write Academic Surveys that Align with Reader Needs? arXiv:2510.03120 [cs.CL] https://arxiv.org/abs/2510.03120

    2025

  16. [24]

    Jie Tang. 2016. AMiner: Toward Understanding Big Scholar Data. InProceedings of the Ninth ACM International Conference on Web Search and Data Mining(San Francisco, California, USA)(WSDM ’16). Association for Computing Machinery, New York, NY, USA, 467. doi:10.1145/2835776.2835849

    2016

  17. [25]

    Mike Thelwall, Verena Weigert, Liz Allen, Zena Nyakoojo, and Eleanor-Rose Papas. 2019. Does the use of open, non-anonymous peer review in scholarly publishing introduce bias? Evidence from the F1000 post-publication open peer review publishing model. arXiv:1911.03379 [cs.DL] h...

    2019 arXiv

  18. [26]

    Haoyu Wang, Yujia Fu, Zhu Zhang, Shuo Wang, Zirui Ren, Xiaorong Wang, Zhili Li, Chaoqun He, Bo An, Zhiyuan Liu, and Maosong Sun. 2025. LLM×MapReduce- V2: Entropy-Driven Convolutional Test-Time Scaling for Generating Long-Form Articles from Extremely Long Resources. arXiv:2504....

    2025 arXiv

  19. [27]

    Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. 2024. AutoSurvey: Large Language Models Can Automatically Write Surveys. arXiv:2406.10252 [cs.IR] https://arxiv.org/abs/2406.10252

    2024 arXiv

  20. [28]

    Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2025. CycleResearcher: Improving Automated Research via Automated Review. arXiv:2411.00816 [cs.CL] https://arxiv.org/abs/2411.00816

    2025 arXiv

  21. [29]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  22. [30]

    Z.ai. 2025. GLM-4.7: Advancing the Coding Capability. https://z.ai/blog/glm-4.7

    2025

  23. [31]

    Fanjin Zhang, Xiao Liu, Jie Tang, Yuxiao Dong, Peiran Yao, Jie Zhang, Xiaotao Gu, Yan Wang, Bin Shao, Rui Li, et al. 2019. OAG: Toward linking large-scale heterogeneous entity graphs. InProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data m...

    2019

  24. [32]

    Fanjin Zhang, Shijie Shi, Yifan Zhu, Bo Chen, Yukuo Cen, Jifan Yu, Yelin Chen, Lulu Wang, Qingfei Zhao, Yuqing Cheng, et al. 2024. Oag-bench: a human-curated benchmark for academic graph mining. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mi...

    2024

  25. [33]

    Guo-Biao Zhang, Ding-Yuan Liu, Da-Yi Wu, Tian Lan, Heyan Huang, Zhijing Wu, and Xian-Ling Mao. 2026. DeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Survey. arXiv:2601.15307 [cs.AI] https: //arxiv.org/abs/2601.15307

    2026 arXiv

  26. [34]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. arXiv:1904.09675 [cs.CL] https://arxiv.org/abs/1904.09675

    2020 arXiv

  27. [35]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL] https://ar...

    2023 arXiv

  28. [36]

    Kun Zhu, Lizi Liao, Yuxuan Gu, Lei Huang, Xiaocheng Feng, and Bing Qin. 2025. Context-aware hierarchical taxonomy generation for scientific papers via llm- guided multi-aspect clustering. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing....

    2025

  29. [37]

    Jun Zhuang and Casey Kennington. 2024. Understanding Survey Paper Tax- onomy about Large Language Models via Graph Representation Learning. arXiv:2402.10409 [cs.CL] https://arxiv.org/abs/2402.10409 A Annotation Details Annotators label each (survey paper, review report, dimens...

    2024 arXiv

  30. [2024]

    arXiv:2408.09169 [cs.CL] https://arxiv.org/abs/2408.09169

    Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices. arXiv:2408.09169 [cs.CL] https://arxiv.org/abs/2408.09169

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.