Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses

Heejin Kook; Josh Seltzer; JunYoung Kim; Subin An; Yang Lu; Yugyeong Ji

REVIEW 4 major objections 4 minor 28 references

A two-stage LLM pipeline—gibberish filtering plus rubric scoring on effort, relevance, completeness—matches expert quality judgments well enough to automate response rejection.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 12:38 UTC pith:2IRJJMDP

load-bearing objection A genuinely new applied framework with a validation story too thin to support its strongest claims — worth refereeing, but not at face value. the 4 major comments →

arxiv 2510.06242 v1 pith:2IRJJMDP submitted 2025-10-03 cs.CL cs.AI

Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses

Subin An , Yugyeong Ji , Junyoung Kim , Heejin Kook , Yang Lu , Josh Seltzer This is my paper

classification cs.CL cs.AI

keywords open-ended survey responsesLLM-as-a-judgereference-free evaluationeffort relevance completenessgibberish detectionsurvey response qualityresponse rejectioncross-lingual evaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that existing LLM-as-a-judge metrics, built for LLM-generated text, misfire on human survey responses, which are shorter, more careless, more personal, and sometimes outright gibberish. It proposes a two-stage evaluation: first detect and discard nonsensical responses, then score each remaining response on three empirically derived dimensions—effort, relevance, and completeness—using a prompt that gives the model a rubric and asks for a numeric score plus a stated reason. Across small English and Korean test sets, the proposed scores correlate with an expert's ratings substantially better than length-based or embedding-based baselines, and a simple sum of the three dimensions predicts overall quality and accept/reject decisions with high agreement. If the claim holds, survey researchers could screen open-ended responses automatically and transparently, flag low-quality answers for follow-up, and avoid polluting analyses with careless inputs.

Core claim

The central discovery is that a deliberately simple, transparent prompt-based method—no reference answers, no learned regressor—can rank human open-ended survey responses in almost the same order as a domain expert, provided the scoring dimensions are chosen for human responses rather than model outputs. On the English test set, the method reaches Spearman correlations of 0.8198 for effort, 0.8614 for relevance, and 0.8245 for completeness with expert ratings; on the Korean test set, the corresponding figures are 0.7950, 0.6021, and 0.7457. The sum of the three normalized scores predicts expert overall-quality ratings with correlation 0.90 (English) and 0.60 (Korean), and response rejection

What carries the argument

The mechanism is a two-stage evaluation pipeline. Stage one is a language-specific gibberish filter—character-level Markov likelihood plus hand-tuned heuristics such as consonant/vowel repetition, valid-word ratio, and, for Korean, Jamo diversity and morphological analysis—that removes nonsensical inputs before costly LLM inference. Stage two is prompt-based dimension-wise scoring: for each of effort (0–7), relevance (0–4), and completeness (0–4), the LLM receives a rubric with anchor examples, evaluates topic and intent alignment for relevance, and returns an integer score and a natural-language reason. The authors' insight is that the rubrics make the evaluation interpretable while the thr

Load-bearing premise

The paper's numbers are correlations against a single marketing expert's ratings on small sets of responses, and for the Korean data those ratings were given to English translations of the original Korean responses—if that expert's preferences are idiosyncratic, or if translation distorts the effort, intent, or completeness being scored, then the reported agreement is with that expert on the translated text, not with the quality of the original responses.

What would settle it

Have several independent marketing experts rate the same 94 English and 100 Korean responses and recompute the LLM's Spearman correlation against the average expert rating and against each individual rater; if multi-rater agreement is substantially lower than the reported values, or if scoring the untranslated Korean responses gives markedly different correlations than scoring the translated versions, the central claim of strong expert alignment is weakened.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Survey platforms could automatically reject or flag low-quality open-ended responses in near-real time, before human review, using a simple sum of the three dimension scores.
Researchers could use the dimension-wise scores and their justifications to separate a response that is short but thoughtful from one that is merely brief, avoiding length-based quality penalties.
The same prompt-based method could generate targeted follow-up questions by identifying the weakest dimension of a partial response, turning static surveys into adaptive ones.
Quality filtering would reduce the labor and noise in downstream text analysis of open-ended survey data, since gibberish and off-topic responses are removed at the front end.
Response rejection with a score threshold offers a concrete operational policy for accept, hold, reject quality control that can be tuned to precision or recall needs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The headline alignment numbers rest on one domain expert's judgments, so a natural next test is multi-expert annotation: if agreement between the method and the average of several experts stays at the reported level, the dimensions are measuring something stable rather than one person's taste.
The Korean results being lower on relevance and overall quality—and the paper's own attribution of the gap to translation into English before annotation—suggest that evaluating responses in the original language, not a translated version, is likely to be the harder and more decisive test for cross-lingual generalisation.
Because the proprietary dataset is not publicly releasable, an independent replication on openly available survey response corpora would be needed before the exact correlation values are taken as portable across industries and question types.
The three dimensions are forward-looking in a practical sense: if effort, relevance, and completeness each drive expert judgments of quality, then giving participants immediate feedback on the weakest dimension could measurably improve the informativeness of follow-up survey rounds.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely new applied framework with a validation story too thin to support its strongest claims — worth refereeing, but not at face value.

read the letter

Colleague,

Read the survey-evaluation paper. The core idea is real and not in the cited literature: they build a two-stage pipeline specifically for human-written open-ended survey responses, with language-specific gibberish filtering (character bigram Markov for English, Jamo-level Markov plus KoNLPy/Okt for Korean) and then an LLM scoring three dimensions — effort, relevance, completeness — with detailed rubrics and per-dimension justifications. The prompts are fully in Appendix E, and the authors include a genuinely honest Limitations section. That transparency counts.

The empirical claims are the problem. The test sets are tiny (94 English, 100 Korean), and most of the headline improvements over baselines are not statistically significant by their own bootstrap CIs (Table 5). The ground truth is fragile: English test is rated by a single domain expert, and the Korean responses were translated into English before that expert rated them. Expert–crowd correlations are only 0.64–0.70, which tells you the gold standard itself is noisy. The authors' explanation for lower Korean results ('meaning distortion or loss of nuance during translation') concedes the point. Add the in-sample design: the rubrics were developed from the same real-world data used for validation, and the gibberish thresholds were empirically tuned. So part of the reported alignment is engineered in, not predicted out.

There's also a red flag in Table 4: identical F1, precision, recall, and accuracy (0.98) for both English and Korean gibberish detection — that invites a code audit.

None of these are fatal to the central intuition. The framework is plausible, interpretable, and clearly aimed at a real industrial problem. But the paper as written overclaims: 'consistently outperforming all baselines' is not supported when most CIs include zero. The authors also need to either release code and a non-proprietary subset of data, or explain how the thresholds and regression weights were selected.

Who is this for? Applied NLP folks and marketing researchers who want a cheap, explainable quality filter for open-ended responses. It's a useful tool, not a scientific breakthrough. I'd send it to a serious referee — the method is novel enough and the prompts are detailed enough to warrant close review — but the referee should demand a clearer separation of the fitted components from the validation, and ideally a larger, less idiosyncratic ground truth.

Yes, engage with it. But keep your expectations calibrated.

Referee Report

4 major / 4 minor

Summary. The paper addresses automated quality evaluation of open-ended human survey responses, a setting where existing metrics target LLM-generated text and typically require references. It proposes a two-stage framework: (i) language-specific gibberish filtering based on Markov models and linguistic heuristics, and (ii) prompt-based LLM scoring using three dimensions—effort, relevance, and completeness—with rubric-based prompts and natural-language justifications. The dimensions are motivated by preliminary analyses of real survey data. Using GPT-4o-mini, the authors report Spearman correlations with human ratings on English (N=94) and Korean (N=100) test sets: 0.82/0.86/0.82 for effort/relevance/completeness in English and 0.80/0.60/0.75 in Korean. They also report practical downstream results: overall-quality prediction (ρ=0.90 English, 0.60 Korean) and response-rejection AUC (0.97 English, 0.83 Korean), claiming consistent superiority over baselines.

Significance. If the results are reliable, the framework is a practical, transparent, reference-free evaluation tool for an underexplored problem: noisy, short, human-written survey responses. The paper's strengths include full disclosure of the evaluation prompts (Appendix E), ablation-style comparison against embedding and length baselines, a gibberish detector that outperforms a pretrained BERT detector and an LLM classifier, and explicit discussion of limitations. However, the significance is tempered by the proprietary, unreleased dataset, the small test sets, and the ground-truth annotation concerns detailed below. The authors' own Limitations section acknowledges LLM subjectivity, unverifiable explanations, the pragmatic rather than theoretical basis of the three dimensions, and the inability to release data—all of which make independent verification difficult.

major comments (4)

[§4.1, §B.2.1, Tables 2 and 6] The load-bearing claim is agreement with human quality judgments. English dimension labels in Table 2 are from a single domain expert; Korean dimension labels are crowd annotations, while Korean practical labels are from experts reading English translations (ChatGPT draft plus one bilingual reviewer). Table 6 shows expert–crowd Spearman correlations of 0.70, 0.66, and 0.64 for English effort/relevance/completeness and 0.53 for Korean overall quality, so the gold standard is substantially rater-dependent. The reported correlations therefore measure agreement with one expert on translated text for Korean, not agreement with a stable gold standard. Please add multi-expert ratings and inter-annotator agreement, state the annotator source directly in Table 2, and if Korean must be translated, validate on original Korean with bilingual expert judges. The authors' own explanation of Korean degr
[§3.2.2–3.2.3, Figs. 3–4] The dimensions and rubrics were derived from analyses of the same real-world dataset later used for validation. Fig. 3 is used to conclude that topic-plus-intent relevance is the right construct, and Fig. 4 motivates length-invariant completeness from expert-rated data. If those analyses used the 94/100 test responses, part of the observed human alignment is in-sample fitting, and the untuned baselines would be at a disadvantage. The paper should state explicitly which data split informed the preliminary studies, or freeze the prompts on a separate development set and report only the test evaluation.
[§C.2, Table 5] The central claim of consistent outperformance is not supported by the significance tests. In Table 5, the bootstrap 95% CI for Ours minus the second-best baseline excludes zero only for English relevance (both ρ and τ) and Korean completeness (τ); for effort and most Korean comparisons the CIs straddle zero. With N=94/100, the headline correlations have wide uncertainty. Please report CIs for the headline correlations and for the AUCs (0.97/0.83), and soften comparative claims for non-significant differences. AUC alone also does not establish operational rejection performance without a calibration or precision–recall analysis at the chosen threshold.
[§3.1, Appendix A] The two-stage framework is not evaluated end-to-end. Gibberish filtering is validated separately on a balanced 50/50 gibberish test set, while the dimension-scoring experiments in §5 appear to operate on already meaningful responses. No experiment quantifies how the filter affects dimension correlations, overall-quality prediction, or rejection AUC. If the framework's contribution is the combination, a pipeline-level ablation or end-to-end result is needed; otherwise the first-stage utility remains limited to the Appendix A metrics.

minor comments (4)

[§4.2, Appendix A] Typographical errors: 'Evalaution' in the §4.2 heading, 'reponses' in §3.2.3, 'instacnes' in Appendix A.3.3, and 'TThe' in the relevance prompt (Table 11).
[Table 1 and Table 2] Table 1's 'Dim. Annot.' / 'Prac. Annot.' distinction is hard to parse; spell out annotator types in the caption. Table 2 should state explicitly that English dimension labels are expert and Korean dimension labels are crowd, otherwise cross-language comparisons are misleading.
[§B.2.1, Appendix A] The ChatGPT translation prompt for Korean-to-English is not included; provide it for reproducibility. Also, the Markov-model thresholds are given as 'e.g., -4.0, -3.5' and the training-data URLs are not versioned; specify exact thresholds and dataset versions.
[Table 7] The quadratic-weighted kappa values (e.g., 0.36 for English effort, 0.45 for English relevance) are notably lower than the corresponding Spearman correlations. This should be discussed, as it suggests only moderate exact-scale agreement despite strong rank agreement.

Circularity Check

2 steps flagged

The relevance and completeness rubrics were designed from the same expert ratings later used as ground truth, so the headline correlations are partly in-sample rather than predictive.

specific steps

fitted input called prediction [§3.2.2 (Relevance, Preliminary Study), Figure 3; validation in §5.1 Table 2; annotation source in Appendix B.2.1]
"Figure 3 shows how we used an LLM to evaluate both the topic and intent alignment of responses and compares these evaluations to human judgments. The results indicate that responses reflecting the question’s intent, in addition to being topically relevant, were rated as having higher relevance overall. The approach that jointly considered both topic and intent achieved the highest agreement with human judgments."

The relevance rubric’s core design choice—joint topic+intent alignment—was adopted because it maximized agreement with human expert ratings. Appendix B.2.1 states that expert ratings exist for the test set ('In the case of the English test set, a single domain expert annotated all responses across all dimensions'), and Figure 3’s caption includes the actual Spearman correlations of 'Ours' vs. baselines. Table 2 then reports ρ=0.8614 for Relevance on that same English test set. The rubric was therefore selected in-sample against the same labels used as ground truth; the high correlation is an in-sample fit, not an out-of-sample prediction.
fitted input called prediction [§3.2.3 (Completeness, Preliminary Study), Figure 4; validation in §5.1 Table 2]
"Figure 4 shows the distribution of human-written response lengths and how human and LLM evaluations differ in treating length. We redefine completeness as the extent to which a response fulfills the core informational intent of the question."

The completeness rubric was redefined to 'core informational intent' after inspecting human expert-rated completeness scores by length (Figure 4) and finding that length-based/LLM completeness misaligned with those expert ratings. Per Appendix B.2.1, expert ratings are only described for the test set. The subsequent completeness correlations in Table 2 (English ρ=0.8245; Korean ρ=0.7457) are thus measured on the same data that motivated the rubric’s definition, making the validation partly in-sample rather than a fresh prediction.

full rationale

No self-citation chain is load-bearing here; the paper does not rely on the authors' prior theorems or imported uniqueness results. The main circularity is data-driven rubric construction: the relevance and completeness criteria were explicitly chosen because they matched human expert ratings shown in Figures 3 and 4, and Appendix B.2.1 indicates those expert ratings are the English test-set annotations (the Korean test set was translated and then expert-annotated). The headline claim in §5.1—'our prompt-based method shows the highest alignment with human ratings'—therefore rests substantially on validation against the same labels used to motivate the rubric. The LLM's raw scores are not fitted constants and the prompts still require the model to apply general language understanding, so the result is not fully forced; a different LLM or a different dataset could have failed. But the design was not made blind to the gold standard, so the reported correlations are optimistic in-sample numbers. The 'Regression' aggregation in §5.2 is also reported without any train/test split description, which may add further in-sample optimism, though the paper does not provide enough detail to prove that. Ground-truth validity (single expert, translated Korean) is a separate correctness risk, distinct from circularity. Overall: partial circularity in the central validation, score 6.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The framework's central claim rests on empirically tuned detection thresholds, hand-built rubrics, and assumptions about LLM judgment and expert ground truth. No new theoretical entities are introduced; the load is carried by fitted constants and domain assumptions whose supporting evidence is proprietary and small-scale.

free parameters (5)

Markov average-log-likelihood thresholds = -4.0 (English), -3.5 (Korean)
Text is classified as gibberish when average log-likelihood falls below these 'empirically determined' thresholds (Appendix A.1.2/A.2.2); tuned on conversational corpora of the same kind used for evaluation.
Language-specific filter thresholds = valid-word ratio 0.4; consecutive vowel/consonant length 10; syllable completeness 0.6; Jamo diversity 4
Hand-set heuristics in Appendix A, presented with 'e.g.' values indicating threshold search; they directly determine gibberish detection F1.
Rubric score scales = Effort 0–7; Relevance 0–4; Completeness 0–4; Overall 0–4
Chosen by hand (§3.2, Appendix E); scale width and anchor examples shape the LLM's score distribution and hence the correlations.
Ridge regression aggregation weights = not reported
§5.2/Table 3: 'Regression' aggregation is a fitted weighted sum; where weights are fitted and whether evaluation is held-out is not stated.
Gibberish test set composition = 50 gibberish / 50 non-gibberish per language
Appendix A.3.2: constructed test composition rather than the natural base rate, which inflates accuracy and F1 relative to deployment conditions.

axioms (5)

domain assumption Character-bigram (English) and Jamo-bigram (Korean) statistics from the cited conversational corpora capture the distribution of legitimate human survey responses.
Core of gibberish detection (Appendix A.1.2/A.2.2); if the corpora differ from survey text (abbreviations, slang, typos), legitimate responses are misclassified.
domain assumption A single domain expert's ratings are a valid, stable ground truth for response quality.
§4.1/B.2.1: English test set scored by one expert; Korean by one expert on translations. No inter-expert agreement is measured.
domain assumption Korean responses can be evaluated for effort/relevance/completeness after translation into English without losing the properties being rated.
B.2.1; the paper attributes the lower Korean correlations to translation distortion, which cuts against this assumption.
domain assumption gpt-4o-mini at temperature 0 with the provided rubrics produces stable, well-calibrated dimension scores.
§4.2; the whole framework rests on LLM judgment, and the Limitations section concedes scores reflect pretraining biases that cannot be fully eliminated.
domain assumption Effort, relevance, and completeness are the (exhaustive) quality-relevant dimensions for open-ended survey responses.
§3.2; claimed 'empirically identified' but the analysis is not shown, and the Limitations section concedes these dimensions are not yet theoretically grounded.

pith-pipeline@v1.3.0-alltime-deepseek · 19601 in / 19720 out tokens · 421969 ms · 2026-08-04T12:38:51.725331+00:00 · methodology

0 comments

read the original abstract

Open-ended survey responses provide valuable insights in marketing research, but low-quality responses not only burden researchers with manual filtering but also risk leading to misleading conclusions, underscoring the need for effective evaluation. Existing automatic evaluation methods target LLM-generated text and inadequately assess human-written responses with their distinct characteristics. To address such characteristics, we propose a two-stage evaluation framework specifically designed for human survey responses. First, gibberish filtering removes nonsensical responses. Then, three dimensions-effort, relevance, and completeness-are evaluated using LLM capabilities, grounded in empirical analysis of real-world survey data. Validation on English and Korean datasets shows that our framework not only outperforms existing metrics but also demonstrates high practical applicability for real-world applications such as response quality prediction and response rejection, showing strong correlations with expert assessment.

Figures

Figures reproduced from arXiv: 2510.06242 by Heejin Kook, Josh Seltzer, JunYoung Kim, Subin An, Yang Lu, Yugyeong Ji.

**Figure 2.** Figure 2: Overall pipeline of the user response evalu [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Figure 4: Average human expert-rated completeness scores (pink) and DeepEval Conversational Completeness scores (blue) across different response length bins. Grey bars indicate sample counts per bin. word count. As a result, verbose but uninformative responses are often overrated, while concise yet complete reponses are penalized [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

**Figure 5.** Figure 5: ROC curves for the English (left) and Korean [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

**Figure 6.** Figure 6: Score distribution of annotations by expert and crowd workers [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

**Figure 7.** Figure 7: A case study presenting the evaluation results of a single user response using our method in English [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

**Figure 8.** Figure 8: A case study presenting the evaluation results of a single user response using our method in Korean [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

28 extracted references · 1 canonical work pages

[1]

Sher Badshah, Ali Emami, and Hassan Sajjad. 2025. https://arxiv.org/abs/2504.07385 Tale: A tool-augmented framework for reference-free evaluation of large language models . CoRR

Pith/arXiv arXiv 2025
[2]

Steven Bird, Ewan Klein, and Edward Loper. 2009. http://www.oreilly.de/catalog/9780596516499/index.html Natural Language Processing with Python . O'Reilly

arXiv 2009
[3]

David Cheng - Han Chiang and Hung - yi Lee. 2023. https://doi.org/10.18653/v1/2023.acl-long.870 Can large language models be an alternative to human evaluations? In ACL, pages 15607--15631

doi:10.18653/v1/2023.acl-long.870 2023
[4]

James M. Crick. 2024. https://doi.org/10.1080/0965254X.2023.2176533 Analyzing survey data in marketing research: A guide for academics and postgraduate students . Journal of Strategic Marketing, 32(2):203--215

arXiv 2024
[5]

Jing Fan, Dennis Aumiller, and Michael Gertz. 2023. Evaluating factual consistency of texts with semantic role labeling. In SEM, pages 89--100

2023
[6]

Markus Freitag, David Grangier, and Isaac Caswell. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.5 BLEU might be guilty but references are not innocent . In EMNLP, pages 61--71

doi:10.18653/v1/2020.emnlp-main.5 2020
[7]

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. https://doi.org/10.48550/arXiv.2411.15594 A survey on llm-as-a-judge . CoRR

Pith/arXiv arXiv doi:10.48550/arxiv.2411.15594 2024
[8]

Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan. 2024. https://arxiv.org/abs/2402.12055 Are llm-based evaluators confusing nlg quality criteria? CoRR

Pith/arXiv arXiv 2024
[9]

Jeffrey Ip and Kritin Vongthongsri. 2025. deepeval . https://github.com/confident-ai/deepeval. Apache-2.0 License. Accessed: 2025-06-26

2025
[10]

Takumi Ito, Kees van Deemter, and Jun Suzuki. 2025. https://arxiv.org/abs/2501.12011 Reference-free evaluation metrics for text generation: A survey . CoRR

Pith/arXiv arXiv 2025
[11]

Madhur Jindal. 2021. https://huggingface.co/madhurjindal/autonlp-Gibberish-Detector-492513457 Gibberish detector: High-accuracy text classification model

2021
[12]

Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81--93

1938
[13]

Jon A Krosnick. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5(3):213--236

1991
[14]

Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81. Association for Computational Linguistics

2004
[15]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://arxiv.org/abs/2303.16634 G-eval: Nlg evaluation using gpt-4 with better human alignment . CoRR

Pith/arXiv arXiv 2023
[16]

Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024. https://arxiv.org/abs/2402.15754 Hd-eval: Aligning large language model evaluators through hierarchical criteria decomposition . CoRR

Pith/arXiv arXiv 2024
[17]

Oded Netzer, Ronen Feldman, Jacob Goldenberg, and Moshe Fresko. 2012. Mine your own business: Market-structure surveillance through text mining. Marketing Science, 31(3):521--543

2012
[18]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2022. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In ACL, pages 311--318

arXiv 2022
[19]

Park and Sungzoon Cho

Eunjeong L. Park and Sungzoon Cho. 2014. Konlpy: Korean natural language processing in python. In HCLT

2014
[20]

Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT : Sentence embeddings using S iamese BERT -networks. In EMNLP-IJCNLP, pages 3982--3992

2019
[21]

Jessie Rouder, Olivia Saucier, Rachel Kinder, and Matt Jans. 2021. https://doi.org/10.29115/SP-2021-0008 What to Do With All Those Open - Ended Responses ? Data Visualization Techniques for Survey Researchers . Survey Practice

work page doi:10.29115/sp-2021-0008 2021
[22]

Andrea Santilli, Adam Golinski, Michael Kirchhof, Federico Danieli, Arno Blaas, Miao Xiong, Luca Zappella, and Sinead Williamson. 2025. https://arxiv.org/abs/2504.13677 Revisiting uncertainty quantification evaluation in language models: Spurious interactions with response length bias results . CoRR

Pith/arXiv arXiv 2025
[23]

Charles Spearman. 1987. The proof and measurement of association between two things. The American journal of psychology, 100(3/4):441--471

1987
[24]

Meng-Chen Wu, Md Mosharaf Hossain, Tess Wood, Shayan Ali Akbar, Si-Chi Chin, and Erwin Cornejo. 2025. https://aclanthology.org/2025.findings-naacl.411/ SEE val: Advancing LLM text evaluation efficiency and accuracy through self-explanation prompting . In NAACL, pages 7357--7368

2025
[25]

Weinberger, and Yoav Artzi

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with BERT . In ICLR

2020
[26]

Xing, Hao Zhang, Joseph E

Lianmin Zheng, Wei - Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets\_and\_Benchmarks.html Judging llm-as-a-judge with mt-bench and chat...

2023
[27]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[28]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[1] [1]

Sher Badshah, Ali Emami, and Hassan Sajjad. 2025. https://arxiv.org/abs/2504.07385 Tale: A tool-augmented framework for reference-free evaluation of large language models . CoRR

Pith/arXiv arXiv 2025

[2] [2]

Steven Bird, Ewan Klein, and Edward Loper. 2009. http://www.oreilly.de/catalog/9780596516499/index.html Natural Language Processing with Python . O'Reilly

arXiv 2009

[3] [3]

David Cheng - Han Chiang and Hung - yi Lee. 2023. https://doi.org/10.18653/v1/2023.acl-long.870 Can large language models be an alternative to human evaluations? In ACL, pages 15607--15631

doi:10.18653/v1/2023.acl-long.870 2023

[4] [4]

James M. Crick. 2024. https://doi.org/10.1080/0965254X.2023.2176533 Analyzing survey data in marketing research: A guide for academics and postgraduate students . Journal of Strategic Marketing, 32(2):203--215

arXiv 2024

[5] [5]

Jing Fan, Dennis Aumiller, and Michael Gertz. 2023. Evaluating factual consistency of texts with semantic role labeling. In SEM, pages 89--100

2023

[6] [6]

Markus Freitag, David Grangier, and Isaac Caswell. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.5 BLEU might be guilty but references are not innocent . In EMNLP, pages 61--71

doi:10.18653/v1/2020.emnlp-main.5 2020

[7] [7]

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. https://doi.org/10.48550/arXiv.2411.15594 A survey on llm-as-a-judge . CoRR

Pith/arXiv arXiv doi:10.48550/arxiv.2411.15594 2024

[8] [8]

Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan. 2024. https://arxiv.org/abs/2402.12055 Are llm-based evaluators confusing nlg quality criteria? CoRR

Pith/arXiv arXiv 2024

[9] [9]

Jeffrey Ip and Kritin Vongthongsri. 2025. deepeval . https://github.com/confident-ai/deepeval. Apache-2.0 License. Accessed: 2025-06-26

2025

[10] [10]

Takumi Ito, Kees van Deemter, and Jun Suzuki. 2025. https://arxiv.org/abs/2501.12011 Reference-free evaluation metrics for text generation: A survey . CoRR

Pith/arXiv arXiv 2025

[11] [11]

Madhur Jindal. 2021. https://huggingface.co/madhurjindal/autonlp-Gibberish-Detector-492513457 Gibberish detector: High-accuracy text classification model

2021

[12] [12]

Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81--93

1938

[13] [13]

Jon A Krosnick. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5(3):213--236

1991

[14] [14]

Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81. Association for Computational Linguistics

2004

[15] [15]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://arxiv.org/abs/2303.16634 G-eval: Nlg evaluation using gpt-4 with better human alignment . CoRR

Pith/arXiv arXiv 2023

[16] [16]

Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024. https://arxiv.org/abs/2402.15754 Hd-eval: Aligning large language model evaluators through hierarchical criteria decomposition . CoRR

Pith/arXiv arXiv 2024

[17] [17]

Oded Netzer, Ronen Feldman, Jacob Goldenberg, and Moshe Fresko. 2012. Mine your own business: Market-structure surveillance through text mining. Marketing Science, 31(3):521--543

2012

[18] [18]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2022. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In ACL, pages 311--318

arXiv 2022

[19] [19]

Park and Sungzoon Cho

Eunjeong L. Park and Sungzoon Cho. 2014. Konlpy: Korean natural language processing in python. In HCLT

2014

[20] [20]

Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT : Sentence embeddings using S iamese BERT -networks. In EMNLP-IJCNLP, pages 3982--3992

2019

[21] [21]

Jessie Rouder, Olivia Saucier, Rachel Kinder, and Matt Jans. 2021. https://doi.org/10.29115/SP-2021-0008 What to Do With All Those Open - Ended Responses ? Data Visualization Techniques for Survey Researchers . Survey Practice

work page doi:10.29115/sp-2021-0008 2021

[22] [22]

Andrea Santilli, Adam Golinski, Michael Kirchhof, Federico Danieli, Arno Blaas, Miao Xiong, Luca Zappella, and Sinead Williamson. 2025. https://arxiv.org/abs/2504.13677 Revisiting uncertainty quantification evaluation in language models: Spurious interactions with response length bias results . CoRR

Pith/arXiv arXiv 2025

[23] [23]

Charles Spearman. 1987. The proof and measurement of association between two things. The American journal of psychology, 100(3/4):441--471

1987

[24] [24]

Meng-Chen Wu, Md Mosharaf Hossain, Tess Wood, Shayan Ali Akbar, Si-Chi Chin, and Erwin Cornejo. 2025. https://aclanthology.org/2025.findings-naacl.411/ SEE val: Advancing LLM text evaluation efficiency and accuracy through self-explanation prompting . In NAACL, pages 7357--7368

2025

[25] [25]

Weinberger, and Yoav Artzi

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with BERT . In ICLR

2020

[26] [26]

Xing, Hao Zhang, Joseph E

Lianmin Zheng, Wei - Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets\_and\_Benchmarks.html Judging llm-as-a-judge with mt-bench and chat...

2023

[27] [27]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[28] [28]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...