{"id":"8d96d86c-41ce-4796-b565-7a0a93f58fd5","arxiv_id":"2508.13124","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Across 20 LLMs, contact center summaries systematically over-represent negative sentiment and under-represent named entities, topic variety, and chronological order, as quantified by the BlindSpot framework.","lead":"A new framework, BlindSpot, measures 15 types of operational bias in LLM-generated call summaries by comparing label distributions from the transcript and the summary. Applied to 2500 real contact center transcripts and 20 LLMs, it finds every model systematically distorts sentiment, topic, and other details, despite summaries appearing high quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The systemic-bias claim is not yet secure: the GPT-4o labeler could manufacture the observed label pattern via differential bias between turn-level and proposition-level inputs.","rationale":"The reader's verdict is CONDITIONAL with high confidence, and the reader's weakest assumption is the LLM labeler's accuracy. I agree, and I think this is the key load-bearing point. The paper's strongest empirical contribution is the cross-model, fine-grained label pattern; that pattern is produced by a single GPT-4o labeler applied to two different kinds of input units. The 93.7% human-agreement figure is not enough to rule out a differential bias: aggregate accuracy is dominated by easy dimensions, and the failure mode that matters is a unit-type-by-label interaction, not overall error rate. The failure mode is especially dangerous because the paper's headline labels are subjective (sentiment, rapport, directives), exactly where labeler errors concentrate. If the labeler is biased in a way that is correlated with summary versus transcript form, then JSD and Coverage are no longer measuring summarizer behavior; they measure the labeler. Because the paper itself concedes this possibility in its Limitations, the central claim should remain conditional pending per-dimension, per-unit validation or an independent human-label re-analysis. This does not invalidate the taxonomy or the framework as a tool, but it does gate the empirical audit's headline. I therefore keep the verdict at CONDITIONAL, matching the reader.","tokens_in":46138,"tokens_out":5693,"duration_ms":64011,"concrete_test":"Take a random subset of 300 transcripts x 5 diverse models (or the existing 1,000-pair validation set augmented to include both turns and propositions from the same conversations) and have two independent human annotators label all units for the 10 semantic dimensions with the paper's label sets. Recompute Fidelity Gap and Coverage using human labels on both the transcript and summary side. If the cross-model pattern (Negative over-represented; Building-Rapport and Directives under-represented) persists, the central claim is supported. If it weakens or reverses, the result is an artifact of the GPT-4o labeler.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that all models systematically over-represent Negative sentiment and Early segments and under-represent Building-Rapport and Directives—requires that the GPT-4o LLM Labeler (Section 2.3; Appendix B.2) measures the same latent categories with errors that are not systematically different on transcript turns versus summary propositions. That requirement is the least secure part of the design. Transcript distributions P_d are computed from labels on long conversational turns, while summary distributions Q_d are computed from labels on extracted propositions; the 'mapping' step only realigns positional/turn-dependent dimensions and does not remove the unit-type difference. A labeler can have 93.68% overall agreement with a human annotator and still apply labels with different systematic biases on the two input types. For example, if GPT-4o is more likely to label a short, context-free proposition as 'Negative,' or less likely to label 'Building Rapport' in a proposition that mentions the agent's action without the surrounding politeness cues, then the Fidelity Gap and Coverage metrics measure the labeler's differential sensitivity, not the summarizer's distortion. The published validation does not report per-dimension accuracy or confusion matrices broken down by input unit type, so this channel is unmeasured. The concern is aggravated by the fact that GPT-4o is also one of the summarizers under evaluation (Table 21), so its own distributional preferences can be entangled with the labeler's. The paper itself flags this class of risk in Section 6: 'potential propagation of existing model biases ... into the annotations themselves.' Until the labeler is independently validated on the actual transcript-summary pairs for the subjective dimensions (Sentiment, Politeness, Agent Action, Solution), the headline cross-model pattern is not fully distinguishable from labeler artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BlindSpot, a framework for quantifying what it calls operational bias in contact-center call summarization. BlindSpot defines a 15-dimension taxonomy (Content & Information Fidelity, Conversational Structure & Flow, Speaker & Role Representation, Linguistic & Stylistic, Affective & Pragmatic Interpretation), uses GPT-4o as a zero-shot labeler to obtain categorical label distributions over transcript turns and over summary propositions, and measures bias via the Jensen–Shannon Fidelity Gap and Coverage %. The authors evaluate 20 LLMs on 2,500 proprietary contact-center transcripts and report that biases are systemic across all models, with consistent over-representation of Negative sentiment and Early segments and under-representation of Building-Rapport and Directives. They also present a targeted-prompt mitigation experiment on nine models that improves Coverage by up to +4.87% and reduces JSD. The appendices provide detailed taxonomy label sets, labeler prompts, alternative divergence metrics, transcript-length analyses, and a human validation study of the LLM labeler (93.68% agreement).","tokens_in":46413,"tokens_out":2774,"duration_ms":32022,"significance":"If the central claim holds, the work is a useful and original contribution: it shifts summarization evaluation from holistic quality metrics to fine-grained, operationally meaningful bias dimensions, and it demonstrates that such bias measures are actionable through prompt intervention. The paper's strengths include a clearly specified taxonomy with concrete label sets, a reproducible prompt suite, an unusually broad model sweep (20 models across families and scales), robustness checks with alternative divergence metrics, and an explicit mitigation experiment with reported effect sizes. The empirical finding that high LLM-Judge scores can coexist with large Temporal Sequence bias is a valuable caution for practitioners. However, the systemic-bias claim is load-bearing and currently rests on a single-annotator validation that does not address the two channels that most threaten the measurement: differential labeler behavior on turns versus propositions, and the unit-granularity mismatch between the two distributions being compared. These concerns are fixable, but they require additional validation and analysis rather than presentation edits alone.","major_comments":[{"comment":"The central claim that all models systematically over-represent Negative and Early and under-represent Building-Rapport and Directives requires that the GPT-4o labeler measure the same latent categories with errors that are not systematically different on long conversational turns than on extracted propositions. The published validation reports only an aggregate 93.68% agreement with a single human annotator on 1,000 turn-proposition pairs; it gives no per-dimension accuracy, no confusion matrices, no inter-annotator agreement, and no breakdown by input unit type. A labeler can reach 93.68% overall agreement while applying, say, a lower threshold for 'Negative' on short context-free propositions than on multi-clause turns; under that scenario the Fidelity Gap and Coverage numbers would partly measure the labeler's differential sensitivity rather than the summarizer's distortion. Please report per-dimension agreement with the human annotations, and either validate the labeler on summary propositions separately or provide a per-unit-type error analysis showing that label bias does not vary with unit type.","section":"§2.3, Appendix B.2"},{"comment":"FidelityGap compares a distribution Pi,d computed over transcript turns with a distribution Qi,d computed over summary propositions, after only a turn-to-proposition mapping for turn-dependent dimensions. Because a turn can contain multiple claims and a proposition is, by construction, a single atomic claim, the two distributions are not defined over comparable units; even a verbatim-copy summarizer would generally produce a different label distribution over propositions than over turns. The paper does not establish that the metrics are invariant under proposition decomposition, so part of the measured 'bias' may be an artifact of unit granularity rather than a property of the summarizer. Please add a synthetic or controlled experiment (e.g., decompose transcripts turn-by-turn into propositions and show that JSD between turn-level and proposition-level distributions of the same source is near zero), or otherwise justify the comparability of the two distributions.","section":"§2.2, §2.3 (Equations defining Pi,d and Qi,d)"},{"comment":"GPT-4o is simultaneously the zero-shot labeler for all dimensions, the LLM Judge for holistic quality, and one of the 20 summarizers under evaluation. If the labeler shares the summarizer's distributional preferences, then the bias estimates for GPT-4o (and, to a lesser degree, for all models sharing the labeler's family) are partially confounded with the labeler's own biases. The paper's Section 6 acknowledges in general terms that LLM labeling can propagate model biases, but it does not test the impact on the empirical conclusions. Concretely, please report the main systemic-bias results with GPT-4o's summaries excluded, and/or re-label a subsample of GPT-4o summaries with an independently implemented labeler (e.g., a different model family) to show that the qualitative pattern of over- and under-represented labels is unchanged.","section":"Table 21, Table 2, §4"}],"minor_comments":[{"comment":"The claim that the two Gemini models are notable outliers (average JSD 0.119 vs. 0.077–0.087 for other models) is reported without any statistical test or confidence interval; with 15 dimensions and 20 models, a per-model or per-dimension significance test (or at least bootstrap intervals) would strengthen this assertion.","section":"Table 2 / §4"},{"comment":"The definition of a label being 'over-represented' or 'under-represented' is not fully specified; please state whether the comparison is between average summary-distribution mass and average transcript-distribution mass, and how the two annotated percentages in Figure 6 are computed.","section":"Figure 3 and Figure 6"},{"comment":"The phrase '1,000 turn-proposition pairs' is ambiguous: it should be clarified whether each pair is a transcript turn plus its associated summary proposition, and whether the human annotator validated all 15 dimensions on each pair or only a subset; a per-dimension table would resolve this.","section":"Appendix B.2"},{"comment":"The table header reads 'T otal V ariation Distance' in the appendix; this is a typo that should be corrected to 'Total Variation Distance'.","section":"Table 23"},{"comment":"The prompt text includes the literal template placeholder '{len(summary_propositions)}' inside the user prompt; if this is a real artifact of the prompt construction, it should be replaced with the actual count in the presented example, or the placeholders should be explicitly described as variables.","section":"Appendix E, Summary Labeling prompt"},{"comment":"The Pearson correlations (r = 0.76 for JSD, r = -0.88 for coverage) are computed over only 20 model-level points; please report the number of points, p-values, and ideally a per-transcript or per-dimension correlation to show that the relationship is not driven by the two Gemini outliers alone.","section":"§4, 'Influence of Compression on Bias'"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope and the effort is substantial, but the central 'systemic bias' claim is more fragile than the framing suggests. The most important next step is not more models or a larger corpus, but a labeler-validation study that breaks agreement down by dimension and by input unit type (turn vs. proposition), plus a granularity-invariance check for the JSD metric. If the authors can show that the labeler's errors are not systematically different across unit types and that the metrics are robust to proposition decomposition, the paper would be a solid contribution. I would also gently encourage the authors to consider whether the proprietary, non-releasable dataset weakens reproducibility enough to justify a small public release of annotated examples; the current dependency on a single human annotator compounds this concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is the first operational-bias benchmark for contact center summarization, and the 15-dimension taxonomy is a real contribution. It is also one of those papers where the measurement instrument (GPT-4o as a zero-shot labeler) is also one of the 20 models under test, so the headline claim that all models systematically over-represent Negative sentiment and Early segments should be read with caution.\n\nWhat the paper does well: the taxonomy is sensible and connects each dimension to a downstream use case — First Call Resolution, QA scorecards, churn risk. The pipeline is clearly described, and the appendix includes the full prompts and label sets, which is more than most papers offer. The audit is large: 2500 real transcripts, 20 models, plus robustness checks with Wasserstein/TVD/KL and a length-bucket split. The mitigation prompt is a nice demonstration that the framework gives actionable feedback, and the honest reporting of the compression-fidelity trade-off is a point in their favor.\n\nWhere it is soft: the labeler is validated at 93.7% on 1000 pairs with a single human annotator. That is fine for an overall sanity check, but the paper does not report per-dimension accuracy or inter-annotator agreement, and it does not validate the labeler separately on transcript turns versus summary propositions. If the labeler labels short decontextualized propositions differently than long turns — for example, by over-calling Negative on propositions — then the Fidelity Gap and Coverage are measuring labeler sensitivity, not summarizer bias. The fact that GPT-4o is also one of the summarizers makes this entangling worse. The compression confound is another soft spot: models that compress more have less room to preserve all labels, so part of the measured bias could simply be compression rather than a specific narrative strategy. The paper reports the correlations but does not control for compression.\n\nThe mitigation experiment partially rehabilitates the central claim. If the labeler were simply blind to Building-Rapport and Directives on propositions, telling the model to include them would not raise measured coverage. But this is not a formal fix for the differential-bias concern.\n\nBottom line: the taxonomy and benchmark are worth having, and the systemic pattern is plausible. The paper deserves a serious referee who will ask for per-dimension labeler validation on both input unit types, a second labeler or additional human annotations, and significance tests for the model-level differences. The proprietary data and missing code also block replication; at least the prompts and label sets are already in the appendix. I would send it to peer review with a conditional-accept mindset, and I would cite the taxonomy if I worked on dialogue summarization.","headline":"First operational-bias benchmark for contact-center summarization with a genuinely useful taxonomy, but the headline cross-model pattern rests partly on a labeler that is also one of the models under test.","tokens_in":47015,"tokens_out":2866,"would_cite":true,"duration_ms":30135,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"All 20 LLMs tested distort call summaries the same way — amplifying negative sentiment and early talk, dropping rapport and directives — and this paper quantifies the distortion with a 15-dimension taxonomy and two metrics.","keywords":["operational bias","LLM summarization","contact center transcripts","Jensen-Shannon divergence","coverage metric","bias taxonomy","LLM-as-labeler","abstractive summarization"],"falsifier":"Re-annotate a random sample of transcript turns and summary propositions with human raters, or with a different-family LLM as the labeler such as Claude, and check whether the reported per-dimension Fidelity Gaps and the \"negative-sentiment over-representation\" signature reproduce. If the labeler's own labels carry the same skew — for instance, tagging 'Negative' more readily on transcripts than on summaries, or vice versa — then the measured bias is an artifact of the annotation instrument, and the cross-model consistency would hold regardless of the summarizer. A second check: force all models to produce summaries at matched compression levels and see whether the Temporal Sequence JS divergence of roughly 0.37 persists, which would separate reordering bias from length-driven information loss.","tokens_in":45920,"feed_emoji":"🎧","tokens_out":8532,"duration_ms":74806,"temperature":0.7,"pith_summary":"Generative summaries of contact-center calls can be fluent and factually accurate and still misrepresent the conversation: they can amplify a customer's frustration, bury resolution steps, omit the entities needed for follow-up, or scramble the order of events. This paper argues that such \"operational bias\" is invisible to standard quality metrics and LLM-as-a-Judge scores, and it builds a framework, BlindSpot, to measure it. The framework labels every transcript turn and every summary proposition along 15 bias dimensions — sentiment, topic, position, politeness, urgency, and others — then compares the two label distributions with Jensen–Shannon divergence (the Fidelity Gap) and a Coverage percentage. Applied to 2,500 real transcripts summarized by 20 LLMs across five model families, nearly every model showed the same signature: over-representing Negative sentiment and Early segments while under-representing Building-Rapport and Directives. The authors also show that a system prompt engineered from those findings reduces bias across the tested models, suggesting the measurement is actionable rather than merely diagnostic.","feed_headline":"All 20 LLMs tested skew call summaries toward problems","feed_subtitle":"An audit framework finds models amplify negative sentiment and early talk while dropping rapport and directive content.","key_machinery":"The carrying mechanism is a pair of label distributions over a 15-dimension taxonomy of operational bias. An LLM labeler (GPT-4o) assigns categorical labels to each transcript turn for the semantic dimensions; structural dimensions (Speaker, Position, Turn Length) are computed directly from metadata and indices, and relational ones (Emotion Shift, Temporal Sequence) are derived from the primary labels. The generated summary is decomposed into atomic propositions, labeled with the same scheme, and mapped back to the source turns it condenses. Bias is then quantified dimension by dimension by the Fidelity Gap, the Jensen–Shannon divergence between the transcript distribution and the summary distribution, and by Coverage, the percentage of source labels that survive into the summary. This converts \"bias\" from a qualitative concern into two per-dimension numbers that expose which specific labels are over- or under-represented.","core_discovery":"BlindSpot's central claim is that biases in LLM call summarization are systemic: all 20 evaluated models, spanning several families and scales from 1-billion-parameter models to frontier systems, exhibit the same pattern of distortion, with only the two Gemini models standing out as substantially worse. The most consistent signature is a simplified, problem-focused narrative, with models over-representing negative sentiment and early parts of the call while under-representing rapport-building, directives, and other resolution-focused content. The hardest dimension is Temporal Sequence, with an average JS divergence of 0.370, meaning event order and cause-and-effect are frequently altered; the largest information loss is in Entity Type, where roughly half of named entities are omitted. Bias grows with compression (correlation 0.76 with JS divergence and −0.88 with coverage), while holistic LLM-Judge scores correlate only weakly with bias, so a summary can score 4.85 out of 5 and still badly reorder events. The framework's usefulness is demonstrated by a mitigation experiment: a targeted prompt built from the measured label-level skews lowered the average Fidelity Gap and raised Coverage for all tested models, with gains of up to +4.87 percentage points in coverage.","pith_inferences":["Because the same labeler (GPT-4o) that defines ground truth is itself one of the 20 summarizers under evaluation, the reported Fidelity Gaps may partly reflect the labeler's own labeling tendencies; a direct test would be to re-run the audit with a different-family labeler, such as Claude or Llama, and check whether the model rankings and label skews persist.","The distributional-comparison recipe — label source units, label summary units, compare with divergence plus coverage — transfers to other high-stakes summarization domains such as medical notes, legal documents, or incident reports, where a fluent summary that silently shifts sentiment or drops entities creates downstream risk.","The framework does not yet say which biases matter most for outcomes. A natural extension is to link Fidelity Gap and Coverage to measured downstream effects, for example whether an auditor or coach makes different decisions when true entity coverage is 52 percent rather than 80 percent.","The shared cross-model distortion fingerprint hints that the biases may be baked into training objectives that reward concise, problem-focused narratives; if so, the remedy would extend beyond prompting to evaluation-aware training or decoding that penalizes distribution shift.","pith_inferences_full_list_placeholder_removed"],"forward_implications":["Because bias is systemic rather than tied to model scale or family, choosing a larger or different-family model will not by itself fix the distortion; the same problem-focused narrative signature appears across all 20 models.","Standard holistic quality scores are insufficient safety checks: LLM-Judge scores correlate only weakly with bias (r ≈ −0.34 for JS divergence, 0.33 for coverage), so deployment decisions need bias-specific measurement.","Compression is a driver of bias: higher compression correlates strongly with a larger Fidelity Gap (r = 0.76) and lower Coverage (r = −0.88), so summary-length targets trade off directly against fidelity.","Operational metrics derived from summaries — such as First Call Resolution, agent QA scorecards, or churn-risk flags — inherit these biases, because omitted entities, reordered events, and sanitized politeness or disfluency all corrupt downstream judgments.","The measured label-level skews are actionable: a prompt that instructs the model to balance sentiment, cover all conversation segments, include rapport and directives, and preserve chronology reduces the Fidelity Gap and raises Coverage across the tested models.","pith_inferences_full_list_placeholder_removed"],"supporting_citations":[{"why":"grounds the task as abstractive dialogue summarization, the setting the bias framework targets","marker":"(Yuan and Yu, 2019)"},{"why":"establishes contact-center call summarization as a real large-scale LLM deployment, motivating the operational-bias question","marker":"(Sachdeva et al., 2023)"},{"why":"supplies the small-LLM telephone-call summarization context the evaluation builds on","marker":"(Thulke et al., 2024)"},{"why":"provides the Jensen–Shannon divergence used as the Fidelity Gap metric","marker":"(Menéndez et al., 1997)"},{"why":"the G-eval LLM-as-a-Judge approach whose weak bias correlation the paper tests against","marker":"(Liu et al., 2023)"},{"why":"prior bias-in-summarization measures, including Total Variation Distance, that the paper positions its operational-bias framework against","marker":"(Steen and Markert, 2024)"},{"why":"prior work on position bias and fairness gaps in summarization that the paper extends from social and structural bias to operational bias","marker":"(Olabisi and Agrawal, 2024)"},{"why":"documents positional bias in long-form summarization, the finding the Position dimension builds on","marker":"(Wan et al., 2024)"}],"fun_headline_variants":["All 20 LLMs skew call summaries toward negativity","Systematic bias: LLMs over-report problems in call summaries","Every LLM tested distorts call summaries the same way","Call summaries from 20 LLMs share a hidden negative bias","AI call summaries systematically favor problems over resolution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement rests on the GPT-4o labeler's annotations being an accurate ground-truth reading of both the transcript and the summary; if the labeler itself systematically mislabels — for example, over-tagging negative sentiment or politeness cues — the Fidelity Gap and Coverage numbers partly measure labeler bias rather than summarizer bias, and this is especially acute because GPT-4o is also one of the 20 summarizers under evaluation.","fun_headline_variants_meta":{"raw":{"variants":["All 20 LLMs skew call summaries toward negativity","Systematic bias: LLMs over-report problems in call summaries","Every LLM tested distorts call summaries the same way","Call summaries from 20 LLMs share a hidden negative bias","AI call summaries systematically favor problems over resolution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000747,"raw_usage":{"total_tokens":3368,"prompt_tokens":1026,"completion_tokens":2342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":2277}},"tokens_in":642,"tokens_out":2342,"duration_ms":15487,"temperature":1.0,"reasoning_tokens":2277,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:15:11.962398+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of transcript turns and summary propositions with human raters, or with a different-family LLM as the labeler such as Claude, and check whether the reported per-dimension Fidelity Gaps and the \"negative-sentiment over-representation\" signature reproduce. If the labeler's own labels carry the same skew — for instance, tagging 'Negative' more readily on transcripts than on summaries, or vice versa — then the measured bias is an artifact of the annotation instrument, and the cross-model consistency would hold regardless of the summarizer. A second check: force all models to produce summaries at matched compression levels and see whether the Temporal Sequence JS divergence of roughly 0.37 persists, which would separate reordering bias from length-driven information loss.","supporting_citations":[],"review_version":1}