{"id":"31e21ff9-c867-40c5-a49d-ed49aac52e3a","arxiv_id":"2411.15320","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PPLqa, the absolute difference between the perplexity of a question plus answer and the answer alone, ranks LLM responses about as well as GPTScore and G-EVAL on one long-form benchmark, with weak overall rank correlation.","lead":"The paper proposes a cheap, prompt-free metric for ranking chatbot answers by quality: the absolute difference between the perplexity of the question plus answer and the perplexity of the answer alone. The metric matches or beats two prompt-based baselines on one long-form benchmark, but its correlations with human and LLM rankings are weak, and no code or data is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (2)'s absolute-value PPL difference is not a monotone quality measure: a correct, question-entailed answer makes PPL(qa) much lower than PPL(a), yielding a large PPLqa, while a generic answer with stable perplexity scores low. The metric thus penalizes relevance and rewards length/style.","rationale":"The reader's verdict was CONDITIONAL, pending release of artifacts and softened claims. My stress-test identifies a more fundamental, mathematically accessible problem: the absolute value in Eq. (2) makes PPLqa non-monotone in the quantity the authors claim to measure. A correct, question-entailed answer can produce a large |PPL(qa) - PPL(a)| because the question sharply reduces perplexity, while a generic, less informative answer whose perplexity is stable with and without the question produces a small difference. This is not a subtle statistical issue or a missing baseline; it is a direct contradiction of the paper's central assertion that lower PPLqa implies better response quality. The length confound reinforces the concern: because PPL(qa) and PPL(a) average over different token sets, the difference is not a clean conditional-surprise measure, and Appendix Table 5 shows the components are length-dependent. In addition, the reported empirical correlations are weak (overall tau = 0.137 versus Claude; MCCs mostly below 0.3 versus humans), so the data do not currently disambiguate the metric from these artifacts. I therefore recommend REJECT rather than CONDITIONAL: the proposed metric's defining formula has a plausible, concrete failure mode that is central to the paper's contribution, and it would need to be resolved—not merely supplemented with code or softened wording—before the claim can be accepted. This is not an ad hominem or a demand for external consensus; it is an internal inconsistency between Eq. (2), the stated direction of the metric, and the stated intuition in Section 3.","tokens_in":12384,"tokens_out":7285,"duration_ms":72421,"concrete_test":"Run a controlled sanity-check set through Eq. (2) with the same evaluator LLM (Mistral-7B-Instruct-v0.2). Use paired long-form answers: (a) a correct, directly entailed answer; (b) a plausible but generic answer; (c) a longer, verbose version of (a); and (d) an unrelated but fluent answer. For each pair, compute PPLqa and regress PPLqa on answer word count. If the correct entailed answer does not receive the lowest PPLqa within its length-matched group, or if PPLqa correlates negatively with length, the 'lower is better' claim fails and the MT-Bench comparison is confounded.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim, stated in Section 3 after Eq. (2), is that 'a lower value of PPLqa implies better response quality.' This requires that D = |PPL(qa) - PPL(a)| monotonically decrease as answer quality improves. The absolute value breaks that monotonicity in a concrete way. Consider a correct, highly entailed answer: PPL(X(qa)) will be much lower than PPL(X(a)) because the question makes the answer tokens predictable; the absolute difference is then large, and the metric labels a good answer bad. Conversely, a generic or loosely related answer whose perplexity is similar with and without the question yields a small difference and is labeled good. The paper's own intuition—'If the response is unrelated to the question, then PPL(X(qa)) will be high'—only describes one of the two regimes that the absolute value conflates; the opposite regime, where the question sharply lowers conditional perplexity, is equally consistent with Eq. (2). No derivation, sign analysis, or controlled failure-case study in Section 3 addresses this. A second, compounding confound is that PPL(qa) averages over question-plus-answer tokens while PPL(a) averages over answer tokens alone; for short answers the difference inherits the question's perplexity, and Appendix Table 5 confirms that perplexity decreases with response length. The metric therefore rewards longer responses and punishes concise, highly conditioned ones. The reported empirical support is too weak to override these artifacts: overall Kendall's tau versus Claude is 0.137, and human-label MCC values in Tables 1-2 are mostly 0.14-0.28. The MT-Bench result (MCC 0.223) may reflect long-form and length confounds rather than quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PPLqa, an unsupervised quality metric for generative LLM responses, defined as the absolute difference between the perplexity of the concatenated question-answer (PPL(X(qa))) and the perplexity of the answer alone (PPL(X(a))). The authors claim that a lower PPLqa value implies better response quality, that the metric subsumes coherence, fluency, relevance, and consistency, and that it performs comparably to GPTScore and G-EVAL on the authors' four-domain dataset and better than those baselines on MT-Bench. Experiments compare PPLqa, GPTScore, and G-EVAL against human and Claude-based rankings using F1, accuracy, MCC, and Kendall's tau.","tokens_in":12667,"tokens_out":4843,"duration_ms":44290,"significance":"If validated, a prompt-free, unsupervised, language-independent quality metric would be practically valuable for model selection and would reduce reliance on ground-truth annotation or LLM-as-a-judge prompting. The paper's strengths include a simple functional form, a fixed evaluator model (Mistral 7B Instruct V0.2), reproducible experimental setup, and direct comparison with two popular baselines. The metric is falsifiable and the paper ships enough detail to recompute it. However, the central claim that 'a lower value of PPLqa implies better response quality' is not theoretically derived, and the empirical correlations are weak and sometimes negative, so the significance is only conditional on a substantial revision that addresses the definitional and evidential gaps.","major_comments":[{"comment":"The absolute value in PPLqa = |PPL(X(qa)) - PPL(X(a))| breaks the claimed monotonicity between the metric and answer quality. A correct, highly entailed answer will make PPL(X(qa)) much lower than PPL(X(a)) because the question makes answer tokens predictable, yielding a large absolute difference that the paper would label as low quality. Conversely, a generic or loosely related answer with similar perplexities in both conditions yields a small difference and is labeled high quality. The paper's own intuition in Section 3 ('If the response is unrelated to the question, then PPL(X(qa)) will be high') describes only one of the two regimes; the opposite regime is equally consistent with Eq. (2). Please provide a derivation or controlled failure-case analysis showing that the problematic regime does not dominate, or replace the absolute value with a signed difference whose direction is justified by the intended quality ordering.","section":"Section 3, Eq. (2)"},{"comment":"The claim that 'differencing normalizes across responses that are of different lengths' is not supported by the paper's own appendix. Appendix Table 5 shows that PPL(X(qa)) is negatively correlated with response length for all four listed models across most domains, and PPL(X(a)) is expected to behave similarly. Because the relationship between perplexity and length is nonlinear and model-dependent, simple subtraction of two length-dependent quantities does not cancel the length effect. The paper should report the correlation of PPLqa itself with response length and, if the correlation is substantial, either adjust the metric or explicitly limit the claim to responses of comparable length. As written, the metric likely rewards longer responses regardless of quality.","section":"Section 3, paragraph on length normalization"},{"comment":"The overall Kendall's tau of PPLqa against Claude rankings is 0.137 (p=0.01), which is lower than GPTScore's 0.197, and G-EVAL's value is negative (-0.074). A p-value of 0.01 only indicates that the correlation is distinguishable from zero; it does not indicate a strong or 'well' correlation as claimed in the abstract. The per-domain tau values are significant only for Macroeconomics (0.234, p=0.02) and are close to zero for AI (0.049), Astronomy (0.126, p=0.22), and Electronics (0.026, p=0.80). The authors should report confidence intervals or effect sizes, and temper the abstract's 'correlates well' claim. The claim that PPLqa 'performs as well as other related metrics' is contradicted by Table 4, where GPTScore achieves the highest overall correlation.","section":"Section 4, Table 4 and abstract"},{"comment":"Several reported MCC values are near zero or negative, which undermines the claim that PPLqa aligns with human or Claude rankings. For example, in Table 1, PPLqa (w/o prompt) has overall MCC 0.14, and MCC values of 0.01 in both AI and Electronics; in Table 2, PPLqa (w/prompt) has negative MCC in all four domains. In Table 3, PPLqa with prompts on MT-Bench yields MCC = -0.274. The paper should reconcile these weak results with the strong wording in the abstract, and should also clarify the orientation of the two F1 values in Tables 1-3 (label 0 vs. label 1) so the reader can interpret the binary classification results.","section":"Section 4, Experiment I and Tables 1-2"}],"minor_comments":[{"comment":"The phrase 'subsumes, but is not explicitly based on, coherence and fluency ... and relevance and consistency' is vague; please specify which terms in Eq. (2) are intended to capture each property, or state explicitly that these properties are emergent rather than designed.","section":"Section 3, after Eq. (2)"},{"comment":"The layout of Table 4 is confusing: each domain has two rows, the first giving tau and the second giving the p-value, but the column headers do not distinguish them. Use a sub-table or parenthetical notation such as '0.137 (p=0.01)' to make the numbers self-explanatory.","section":"Section 4, Table 4"},{"comment":"The exact prompts used for 'PPLqa (w/prompt)' are not shown; the prompt set from GPTScore is mentioned but an example of only the coherence prompt is given. Please provide the full prompt set in an appendix or a supplementary file.","section":"Section 4, Experiment II"},{"comment":"The reference to Friedland (2024) for the 'information-theoretic interpretation' is not sufficient for a self-contained argument; a brief derivation or a concise explanation of why the perplexity differential is information-theoretically meaningful would strengthen the paper.","section":"Section 5, Limitations"},{"comment":"The statement that the metric is a 'white-box' metric because 'one may investigate each neuron's activation in the softmax layer' is overstated; perplexity is a scalar aggregate over token probabilities and does not expose per-neuron activations. Please revise the wording.","section":"Section 6, Conclusions"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a useful problem and the empirical setup is straightforward, but the central metric definition contains a potential sign-error that undermines the claimed monotonic relationship between PPLqa and answer quality. The authors should be required to address the opposite-regime problem directly, either by changing the metric to a signed difference with a justified ordering or by providing controlled experiments that rule out the problematic regime. In its current form, the abstract's claims are not supported by the reported correlations. This is a borderline case; with a revised metric and more careful empirical claims it could become publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper defines PPLqa = |PPL(question+answer) - PPL(answer)| and claims lower values mean better LLM responses, with no ground truth and no prompt design. That's an appealingly cheap idea, and the MT-Bench comparison against GPTScore and G-EVAL is new and fairly run. But the central claim does not survive contact with the math, and the reported correlations are weak enough that the abstract's 'correlates well' is not supported.\n\nWhat is genuinely useful: the method is simple, language-agnostic, and needs no labels or prompt templates. Running it is almost free. The experiments use the same evaluator LLM (Mistral-7B) across methods, which is a fair control. The paper is transparent about its limitations and even says theoretical analysis is future work.\n\nThe soft spots are load-bearing. First, the absolute value destroys monotonicity. For a correct, highly entailed answer, conditioning on the question makes the answer tokens much more predictable, so PPL(qa) can be far below PPL(a); the absolute difference is then large and the metric flags a good answer as bad. An unrelated or generic answer can have similar PPL with and without the question, giving a small difference and a good score. The paper's intuition only covers the unrelated-answer regime; it never considers the entailed-answer regime. No derivation, sign analysis, or controlled failure cases are given. Second, the length confound: PPL(qa) averages over question+answer tokens while PPL(a) averages over answer tokens alone, so for short answers the difference inherits the question's perplexity. Appendix Table 5 confirms perplexity decreases with length, so the metric effectively rewards longer responses. Third, the empirical evidence is weak: overall Kendall tau against Claude is 0.137 (GPTScore gets 0.197), and MCCs on their own data are mostly 0.14-0.28. MT-Bench MCC of 0.223 is the best result, but that could be the length confound rather than quality. The paper also ships no code, no data, and no annotator reliability details, so the numbers cannot be checked.\n\nBottom line: this is a plausible starting point for a cheap baseline metric, but as defined it is not a valid quality measure. A signed difference or a properly length-normalized version might work, and the failure analysis is worth doing. The paper deserves a serious referee because the idea is important and the flaws are fixable, but I would not cite it as a reliable metric in its current form. I'd bring it to a reading group as a case study in how an appealing information-theoretic shortcut can go wrong.","headline":"The absolute-value perplexity difference conflates good and bad answers, and the paper's own correlations are too weak to support its 'correlates well' claim.","tokens_in":13261,"tokens_out":4805,"would_cite":false,"duration_ms":39082,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single perplexity gap between a question-answer pair and the answer alone ranks LLM response quality without ground truth.","keywords":["perplexity","LLM evaluation","unsupervised quality metric","model ranking","question answering","information theory","prompt-free evaluation"],"falsifier":"Take a set of question-answer triples where the answer is factually correct, subtly wrong, or off-topic, with the three answers matched for length and style, and check whether PPLqa strictly orders them from best to worst; any failure to separate the correct from the subtly wrong answer would refute the claim that lower PPLqa means better quality.","tokens_in":12180,"feed_emoji":"📊","tokens_out":6487,"duration_ms":53037,"temperature":0.7,"pith_summary":"PPLqa is a proposed answer-quality score for generative language models that needs no ground truth, no human annotations, and no evaluation prompt: it is simply the absolute difference between the perplexity of a question-answer pair and the perplexity of the answer alone, with a lower value taken to mean a better response. The paper argues that this one number captures both the writing quality of the response (through perplexity) and its relevance and consistency with the question (through the surprise of seeing the question and answer together). If the claim holds, anyone choosing among LLMs for a task can rank candidate models by feeding their responses through the metric, bypassing the costly creation of expert-written reference answers. The paper reports that across four subject domains and on the MT-Bench human-judgment set, PPLqa matches or beats prompt-based metrics such as GPTScore and G-EVAL, and correlates with both human rankings and rankings from an LLM judge. Because the score is computed from token probabilities alone, it is language-independent and could apply to any trained language model.","feed_headline":"Perplexity gap ranks LLM answers without ground truth","feed_subtitle":"One prompt-free score beats GPTScore and G-EVAL on MT-Bench and tracks human and LLM rankings.","key_machinery":"The load-bearing object is the perplexity differential of Equation (2), computed with a separate, performant evaluator LLM that assigns token probabilities. Perplexity is the exponentiated average negative log-probability of a token sequence, interpretable as the number of binary choices needed to encode the text. The paper's argument is that $|\\mathrm{PPL}(X(qa)) - \\mathrm{PPL}(X(a))|$ measures the surprise of a response given a question: a coherent, on-topic response adds little perplexity when the question is prepended, while an unrelated or hallucinated response makes the joint sequence much less predictable. The absolute-value differencing is the mechanism that removes length effects and isolates the context-dependent component of surprise, so the same fixed formula can rank responses without any task-specific prompt.","core_discovery":"The central discovery is that the quantity $\\mathrm{PPLqa} = |\\mathrm{PPL}(X(qa)) - \\mathrm{PPL}(X(a))|$, where $X(qa)$ is the concatenation of the prompt and response and $X(a)$ is the response alone, orders LLM responses by quality: a lower value of PPLqa indicates a better response. The reasoning is that perplexity of the response measures coherence and fluency, while the gap between the joint and marginal perplexities measures how surprising, and therefore how relevant and consistent, the response is given the question. Explicitly, an unrelated answer raises $\\mathrm{PPL}(X(qa))$, and differencing also normalizes the well-known tendency for perplexity to fall as text gets longer. On the authors' experiments, PPLqa performed in the same range as GPTScore and G-EVAL on binary and four-way comparisons across macroeconomics, astronomy, AI, and electronics questions, and it outperformed both on MT-Bench when used without prompts.","pith_inferences":["A likely untested consequence is that PPLqa inherits the biases of the evaluator LLM; swapping the scoring model could change rankings, so a multi-evaluator average would be a natural extension.","The differencing operation suggests a normalized variant, such as the ratio $\\mathrm{PPL}(X(qa))/\\mathrm{PPL}(X(a))$, which might give more stable comparisons across domains or languages where absolute perplexity scales differ.","A controlled test that holds response length and topic fixed while varying only factual correctness would separate genuine quality tracking from length or style effects; the paper does not report such a test.","The same formula could be applied to retrieval-augmented generation by replacing the question with the retrieved context plus question, providing a prompt-free way to score groundedness."],"forward_implications":["A user comparing LLMs for a task can rank them with only a list of questions and the models' answers, no expert-written references.","Long-form question answering is where PPLqa shows its strongest agreement with human judgments; short binary or multiple-choice responses are outside its intended scope.","No evaluation prompt needs to be designed; in the MT-Bench experiments, removing prompts improved PPLqa's accuracy while prompt-based competitors required them.","The metric is white-box and fully transparent in computation, allowing inspection of the softmax activations that produce the perplexity values.","Because no language-specific processing is used, the same formula applies to any language a trained evaluator model can score."],"supporting_citations":[{"why":"Introduces GPTScore, the prompt-based evaluator-LLM metric that PPLqa is compared against in all experiments.","marker":"Fu et al. (2023)"},{"why":"Introduces G-EVAL, the chain-of-thought form-filling metric that PPLqa is compared against.","marker":"Liu et al. (2023)"},{"why":"Provides the MT-Bench human-judgment dataset and the LLM-as-judge methodology used as ground truth in Experiment II.","marker":"Zheng et al. (2023)"},{"why":"Supplies the prior use of perplexity as an unsupervised quality estimator for generated text, which PPLqa extends.","marker":"Fomicheva et al. (2020)"},{"why":"Supports the connection between concatenating question and answer and detecting hallucination, the relevance mechanism in PPLqa.","marker":"Friel and Sanyal (2023)"}],"fun_headline_variants":["Perplexity gap scores LLM answers with no human labels","No annotations needed: perplexity difference evaluates LLMs","PPLqa: unsupervised metric that tracks human rankings","Perplexity gap beats prompt-based metrics on MT-Bench","Rank LLMs blindly using perplexity of prompt+answer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gap in perplexity between a question-and-answer pair and the answer alone tracks how good the answer is, rather than how long, stylized, or familiar to the scoring model it is.","fun_headline_variants_meta":{"raw":{"variants":["Perplexity gap scores LLM answers with no human labels","No annotations needed: perplexity difference evaluates LLMs","PPLqa: unsupervised metric that tracks human rankings","Perplexity gap beats prompt-based metrics on MT-Bench","Rank LLMs blindly using perplexity of prompt+answer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1395,"prompt_tokens":901,"completion_tokens":494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":413}},"tokens_in":517,"tokens_out":494,"duration_ms":5037,"temperature":1.0,"reasoning_tokens":413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:25:55.468495+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of question-answer triples where the answer is factually correct, subtly wrong, or off-topic, with the three answers matched for length and style, and check whether PPLqa strictly orders them from best to worst; any failure to separate the correct from the subtly wrong answer would refute the claim that lower PPLqa means better quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces G-EVAL, the chain-of-thought form-filling metric that PPLqa is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prior use of perplexity as an unsupervised quality estimator for generated text, which PPLqa extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the connection between concatenating question and answer and detecting hallucination, the relevance mechanism in PPLqa."}],"review_version":1}