{"id":"f25da409-4e23-403f-b01f-7bb2a932bff1","arxiv_id":"2411.16077","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SAGEval uses a critiquing agent to revise LLM evaluator scores for reference-free surveys and forms, improving correlation with human ratings on a small dataset, though the improvement could be a score-shift artifact.","lead":"SAGEval is a new evaluation framework that adds a critiquing AI agent to a standard LLM evaluator, aiming to correct scores for open-ended, reference-free text like surveys and forms. Tests on 96 generated surveys with human ratings suggest the correction step brings machine scores closer to human judgments, but missing control baselines leave this claim unproven.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed improvement lacks statistical grounding and an attribution control; the uniform-shift baseline, as stated, would not alter rank correlations.","rationale":"The reader's overall rejection is justified, but the specific weakest assumption -- that a uniform downward shift of Evaluator scores could reproduce the gains -- is technically flawed for rank-based metrics like Spearman and Kendall. Any monotone transform of all scores preserves rank order exactly, so a uniform shift would leave these correlations unchanged. The deeper, load-bearing issue is that the paper never shows the reported gains are statistically reliable or attributable to the proposed critique architecture. With N=96, the differences for several aspects are small enough to be sampling noise, and the absence of significance testing is disqualifying for a claim of improvement across all criteria. The architecture confound is also real: SAGEval differs from G-Eval by more than just 'critique' -- it adds a second model call, stricter output distribution, and potentially modified scoring definitions. Without a single-pass strict-evaluator baseline or a direct-scoring SAGE Agent control, the observed alignment could come from prompt strictness rather than meta-evaluation. I therefore keep the reader's REJECT verdict, while correcting the proposed control: the test should be a strict-evaluator baseline and direct SAGE scoring, not a uniform score shift.","tokens_in":11702,"tokens_out":7426,"duration_ms":75209,"concrete_test":"Run a paired bootstrap over the 96 items (10,000 resamples), recomputing the SAGEval vs G-Eval Spearman difference for each aspect in Table 2; if any two-sided 95% confidence interval for the difference includes 0, the 'all criteria' claim fails. Then add two attribution controls: (a) a single-pass 'strict evaluator' with the same aspect definitions but explicit instructions to avoid leniency and use the full 1-5 scale; (b) the SAGE Agent prompted to score directly from text and definitions only, without seeing Evaluator Agent scores. If either control's Spearman correlations match SAGEval's within the bootstrap confidence intervals, the improvement cannot be attributed specifically to the critique/rectification mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SAGEval 'aligns better with human annotators across all scoring criteria' rests on point estimates in Table 2, with no confidence intervals, p-values, or bootstrap/permutation tests. At N=96, several reported gains are small (e.g., Coherence 0.32 vs 0.41, Relevancy 0.47 vs 0.48, Fairness 0.41 vs 0.44) and plausibly within sampling error, so 'across all criteria' is not established. This is compounded by the absence of any attribution control: SAGEval is a two-pass system that adds a second GPT-4 call and agent-proposed criteria modifications, but the paper never compares against a single-pass evaluator with an explicit strict-scoring instruction, nor against the SAGE Agent used as a direct scorer without seeing the Evaluator Agent's scores. Since 92% of rectifications are negative, the apparent improvement could stem from increased strictness rather than from the critique mechanism. Note that the reader's proposed 'uniform score shift' control is not the right test: Spearman and Kendall are rank-based, so any monotone downward shift leaves them unchanged. The appropriate control is a differently-prompted strict evaluator or direct SAGE scoring, either of which could reorder items in the same way and reproduce the gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SAGEval, a two-agent framework for reference-free evaluation of open-ended NLG output, specifically forms/surveys generated by LLMs. An Evaluator Agent (modeled on G-Eval) assigns scores on seven predefined aspects, and a critiquing SAGE Agent provides score rectifications, suggests modifications to aspect definitions, and proposes new aspects. The authors release a dataset of 96 GPT-3.5-Turbo-generated surveys/forms annotated by four linguists, and report Spearman and Kendall correlations against these annotations, claiming that SAGEval outperforms G-Eval, CheckEval, ChatGPT-4o, FreeEval, and MATEval across all criteria. The central claim is that the critiquing mechanism yields better human alignment; the paper also presents descriptive findings about the SAGE Agent's score distributions and aspect suggestions.","tokens_in":11955,"tokens_out":4307,"duration_ms":38411,"significance":"If the claims were statistically robust, SAGEval would be a useful step toward reliable reference-free evaluation of structured open-ended text, an area where existing benchmarks are scarce. The release of a 96-item dataset with human annotations for forms/surveys is a valuable resource for the community. However, the current evidence is not strong enough to establish the central claim: the evaluation relies on a small sample with point estimates only, and the improvement is not convincingly attributed to the critique mechanism rather than to a general strictness shift. The contribution is therefore potentially significant but not yet demonstrated.","major_comments":[{"comment":"The central claim that SAGEval \"aligns better with human annotators across all scoring criteria\" is not supported by inferential statistics. At N=96, the reported Spearman differences are small for several criteria (e.g., RELEV 0.48 vs. 0.47, FAIR 0.44 vs. 0.41, COH 0.41 vs. 0.32), and no confidence intervals, bootstrap intervals, permutation tests, or p-values are provided. Given that the approximate standard error of a Spearman correlation at this sample size is around 0.1, these differences are plausibly sampling noise. The authors should report interval estimates and significance tests for the difference between SAGEval and each baseline before claiming improvement \"across all scoring criteria.\"","section":"Section 6.2, Table 2"},{"comment":"The attribution of the observed improvement to the SAGE Agent's critique is not established because no control separates the effect of the critique mechanism from the effect of a general tendency toward lower scores. Table 1 and Figure 3 show that about 92% of rectifications are negative, shifting score mass from 4s/5s to 2s/3s. A plausible alternative explanation is that any instruction to be more critical, or even a fixed downward adjustment strategy, would yield higher rank correlation with human annotations on this dataset. The appropriate controls are (a) a single-pass Evaluator Agent prompted to be stricter and (b) the SAGE Agent scoring directly without first-pass scores; neither is reported. Note that a uniform score shift would not change rank correlations, so the reader's proposed control is not decisive, but the strict-prompt control would directly test whether the specific critique mechanism is responsible.","section":"Sections 6.1 and 6.2, Table 1, Figure 3"},{"comment":"The method description is insufficiently detailed to reproduce the reported results. Eq. (1) is typeset incorrectly and does not specify how the probabilities p(s_i) are obtained from the LLM, nor how the SAGE Agent's rectified scores are combined with or replace the Evaluator Agent's scores to produce the final SAGEval score used in Table 2. It is unclear whether the correlations are computed on the SAGE Agent's output alone or on some aggregation. Full prompt templates, the decoding configuration, and the exact score-rectification procedure should be provided.","section":"Section 4.4, Eq. (1), and Section 6.2"},{"comment":"The third contribution, that the SAGE Agent can propose new scoring aspects, is presented without validation. The paper reports that the SAGE Agent suggests aspects such as \"Creativity Score\" and \"Content Quality Score\" across many data points, but it does not evaluate whether these suggested aspects improve correlation with human judgments or add useful information beyond the predefined set. Without such analysis, this claim remains descriptive and does not support the stated contribution of demonstrating capabilities to propose useful new aspects.","section":"Section 6.3, Figure 4"}],"minor_comments":[{"comment":"The abstract contains typos: \"more more complex\" should be \"more complex,\" and \"doesn't exist or isn't amply available\" is informal.","section":"Abstract"},{"comment":"The sentence beginning \"SAGE Agent is prompted to also ensure if the pre-defined scoring criteria is\" is incomplete; the paragraph appears to be cut off.","section":"Section 6.3"},{"comment":"The column headers \"Neg Pos Total Definition\" are confusing. The \"Definition\" column header does not match its content (counts of instances), and \"Neg\" and \"Pos\" should be spelled out or defined in the caption.","section":"Table 1"},{"comment":"The text refers to \"LLMEval (which is based on G-EVal framework)\" but Table 2 labels the row \"G-Eval\"; use consistent terminology throughout.","section":"Section 6.2"},{"comment":"The sentence \"the annotated by 4 experienced linguists\" is missing a verb; it should read \"annotated by 4 experienced linguists.\"","section":"Section 5"},{"comment":"There is a typo in \"multi-agent farmeworks\" which should be \"frameworks.\"","section":"Section 2"},{"comment":"The caption \"Scores distribution by SAGE Agent compared scores assigned by Evaluator Agent\" is ungrammatical; consider \"Score distributions assigned by the SAGE Agent and the Evaluator Agent.\"","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a CL venue but needs substantial strengthening. The lack of significance testing and the missing attribution control are the main technical obstacles. The dataset, if released, could be a useful resource. One reference (Wadhwa et al., 2024) includes an author of this paper; this is not problematic per se, but the self-citation is not flagged. The paper also states in the abstract that the dataset will be released, but no link or repository is provided in the text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset is the contribution, the framework is G-Eval plus a second opinion, and the headline result is not proven. The 96 human-annotated forms/surveys are a genuinely new benchmark for reference-free evaluation of structured, list-like text, and I would use it. The SAGEval framework itself combines a G-Eval-style evaluator with a 'SAGE' agent that critiques and lowers the first pass's scores. Multi-agent critique is already in ChatEval and MATEval, so the novelty is the domain, not the mechanism.\n\nThe problem is the evidence. Table 2 reports Spearman and Kendall correlations at N=96 with no confidence intervals or significance tests, and several gains are tiny (Relevancy 0.47 vs 0.48, Fairness 0.41 vs 0.44, Coherence 0.32 vs 0.41). The claim that SAGEval is 'significantly better' is not supported. The bigger issue is attribution: since 92% of the SAGE agent's corrections lower the score, the improvement could simply be increased strictness. Note the reader's proposed uniform-score-shift control is not the right test, because rank correlations are invariant to monotone shifts. Appropriate controls would be a single-pass strict-scoring evaluator or the SAGE agent scoring directly without seeing the evaluator's numbers. Neither is present.\n\nThe manuscript also has writing problems: incomplete sentences in the introduction and conclusion, typos, and an inconsistency between the eight aspects defined in Section 4.1 and the seven in Table 2 (Sentiment disappears). The limitations section is honest about the narrow scope.\n\nWho this is for: researchers working on LLM-based evaluation of structured, reference-free outputs, and product teams that need cheap quality checks. The dataset alone is worth citing. The framework is a plausible baseline, but the central claim needs the missing controls and uncertainty quantification. Send it to review, but the authors should get a major-revision request.","headline":"SAGEval's new dataset is the real contribution; the claimed improvement from the critiquing agent is plausible but not established, and the paper needs a proper attribution control before the headline claim can be trusted.","tokens_in":12496,"tokens_out":3416,"would_cite":true,"duration_ms":30807,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAGEval claims that a role-based critiquing agent layered over a G-Eval-style scorer rectifies reference-free NLG scores and improves alignment with human annotations across all reported scoring criteria, with the largest gains on…","keywords":["SAGEval","reference-free NLG evaluation","LLM-as-a-judge","critiquing agent","meta-evaluation","open-ended text generation","survey generation","human alignment"],"falsifier":"Run a control where each Evaluator Agent score is decreased by a fixed amount, or by the same distribution of decreases that the SAGE Agent produces but decoupled from the specific text, and recompute the Spearman correlation with the human annotations; if this no-critique control matches or beats SAGEval, the critique content is not what is doing the work.","tokens_in":11544,"feed_emoji":"🤖","tokens_out":6265,"duration_ms":54339,"temperature":0.7,"pith_summary":"Open-ended, reference-free text such as generated surveys, quizzes, and forms is hard to evaluate automatically because there is no gold answer to compare against, and single-pass LLM scorekeepers tend to assign inflated ratings. This paper proposes SAGEval, a two-agent setup in which a G-Eval-style Evaluator Agent scores each generated form on eight pre-defined aspects and a separate SAGE Agent critiques those scores, proposes rectified scores, suggests revisions to criterion definitions, and recommends new criteria such as creativity and content quality. The central claim is that these agent-corrected scores align better with human linguist annotations than G-Eval, CheckEval, FreeEval, MATEval, or ChatGPT-4o across the criteria that were compared; the largest gains appear on Accuracy (Spearman 0.63 vs 0.49 for G-Eval), Audience Understanding, and Audience Engagement. If the claim holds, reference-free evaluation of structured open-ended text can get closer to human judgement without additional labeled data.","feed_headline":"A critiquing agent pushes LLM scores closer to human ratings","feed_subtitle":"On 96 generated surveys, pairing a critic with a scorer lifted accuracy correlation with human annotators from 0.49 to 0.63.","key_machinery":"The load-bearing mechanism is the SAGE Agent, a second LLM prompted as a meta-evaluator. After the G-Eval-inspired Evaluator Agent assigns a 1-to-5 score with chain-of-thought reasoning and in-context exemplars, the SAGE Agent independently examines the same text and can recommend score rectifications, propose changes to the wording of aspect definitions, and suggest additional aspects. The final rectified scores are normalized using the probabilities of output tokens, following G-Eval's approach, to dampen score skew and ties.","core_discovery":"The paper's discovery is that a second, role-based agent can serve as a meta-evaluator and correct the scores of a first-pass LLM evaluator in settings where no reference text exists. On a new benchmark of 96 GPT-3.5-Turbo generated surveys and forms annotated by four linguists, the SAGE Agent disagreed with the Evaluator Agent's scores in roughly 92 percent of rectifications, mostly moving scores downward from 4s and 5s to 2s and 3s. After those rectifications, SAGEval's scores showed the highest Spearman and Kendall-Tau correlations with human annotations among all compared methods, with Accuracy improving from 0.49 with G-Eval to 0.63, and Audience Engagement improving from 0.25 with G-Eval to 0.49. The SAGE Agent also proposed modified definitions for criteria such as Sentiment/Tone and suggested new criteria, most often Creativity Score and Content Quality Score, across all 96 examples. The authors interpret this as evidence that LLM evaluators can not only score but also critique and extend evaluation rubrics in the absence of ground-truth labels.","pith_inferences":["The paper's attribution of gains to the critique content is not fully controlled: since 92 percent of rectifications are downward, a uniform one-point lowering of the Evaluator Agent's scores might reproduce much of the correlation shift.","A natural extension is to apply the two-agent design to continuous reference-free text such as stories and dialogues, where rubric adaptation is even harder; the paper's own limitations section lists this as untested.","The 96-instance benchmark means the reported correlations carry wide uncertainty, so re-estimating them on a larger and multilingual set of generated forms would test whether the Accuracy gain of 0.14 is stable."],"forward_implications":["If SAGEval's claim is correct, single-agent LLM evaluation biases can be partially corrected by a second agent that critiques rather than votes.","Reference-free evaluation for structured open-ended outputs such as forms, surveys, and quizzes can reach human-alignment levels without labeled references.","The same setup can surface missing evaluation criteria, showing where initial rubrics under-specify quality.","Production cost stays modest relative to multi-agent debate: two LLM calls per instance are enough.","The released dataset and human annotations provide a reproducibility base for future reference-free evaluation work."],"supporting_citations":[{"why":"G-Eval supplies the Evaluator Agent's prompting design, probability-weighted score normalization, and the main single-pass baseline SAGEval is built on and compared against.","marker":"Liu et al., 2023b"},{"why":"GPT-Score motivates the scoring-as-desired approach and contributes the aspect definitions that SAGEval extends for reference-free evaluation.","marker":"Fu et al., 2023"},{"why":"X-eval provides the expanded multi-aspect evaluation idea that SAGEval adapts to forms and surveys.","marker":"Liu et al., 2023a"},{"why":"ChatEval establishes the multi-agent debate baseline that SAGEval distinguishes itself from by using a single critic rather than many agents.","marker":"Chan et al., 2023"},{"why":"MATEval supplies the multi-agent discussion baseline for open-ended text evaluation that SAGEval reports higher human alignment than.","marker":"Li et al., 2024"},{"why":"CheckEval provides a checklist-based evaluation baseline whose correlations SAGEval compares against.","marker":"Lee et al., 2024"},{"why":"FreeEval provides a modular evaluation baseline that SAGEval reports higher human alignment than.","marker":"Yu et al., 2024"},{"why":"Earlier reference-free text quality evaluation focused on continuous text, framing the structured-format gap that SAGEval targets.","marker":"Chen et al., 2023b"}],"fun_headline_variants":["Critic agent corrects LLM scores without references","SAGEval: critiquing agent lifts accuracy from 0.49 to 0.63","Agent critic fixes LLM scoring; no ground truth needed","Meta-evaluator agent improves LLM score correlation","A second agent catches LLM scoring slips, boosting accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on treating the SAGE Agent's critique content as the cause of improved alignment, but the paper never rules out the simpler explanation that just lowering the Evaluator Agent's inflated scores, which is what 92 percent of the critiques do, is what improves the correlation.","fun_headline_variants_meta":{"raw":{"variants":["Critic agent corrects LLM scores without references","SAGEval: critiquing agent lifts accuracy from 0.49 to 0.63","Agent critic fixes LLM scoring; no ground truth needed","Meta-evaluator agent improves LLM score correlation","A second agent catches LLM scoring slips, boosting accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1374,"prompt_tokens":974,"completion_tokens":400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":312}},"tokens_in":590,"tokens_out":400,"duration_ms":4240,"temperature":1.0,"reasoning_tokens":312,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:33:34.894612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a control where each Evaluator Agent score is decreased by a fixed amount, or by the same distribution of decreases that the SAGE Agent produces but decoupled from the specific text, and recompute the Spearman correlation with the human annotations; if this no-critique control matches or beats SAGEval, the critique content is not what is doing the work.","supporting_citations":[],"review_version":1}