{"id":"bb1703d1-8365-43cf-af34-743f7a8d1e07","arxiv_id":"2411.10080","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Sampling repeated LLM responses with adjusted temperature yields opinion distributions closer to human judgments than direct prompting, though the temperature effect is inconsistent across metrics.","lead":"This paper compares three ways to extract opinion distributions from GPT-3.5-turbo and finds that sampling repeated responses with adjusted temperature matches human annotations better than directly prompting the model. The study is a small empirical check on three subjective text classification datasets, suggesting that simple parameter adjustments can improve alignment with human disagreement.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temperature claim is not supported by Table 2: T=2 is worse on CE for all three datasets, and no significance testing is reported.","rationale":"The reader's verdict is CONDITIONAL, and my reading supports that conditionality. The most load-bearing issue is not the P_ann approximation, though that is a real limitation; it is that the paper's stated temperature finding is contradicted by parts of Table 2 and is never subjected to statistical testing. The paper's method-level claim, that sampling and log-probability estimation outperform direct prompting, is robust and consistent across all datasets and metrics, with large margins. However, the temperature-specific claim appears in the title, Section 5, and the conclusion, and without it the paper's novelty is substantially weaker. A concrete re-analysis of the per-item scores with paired tests and confidence intervals would settle whether the temperature ordering is real or noise. Given that the direct-versus-sampling result already justifies a conditional accept with requested revisions, the reader's verdict should remain CONDITIONAL; no verdict change is needed, but the requested revisions should explicitly include uncertainty quantification and a softened temperature conclusion.","tokens_in":6159,"tokens_out":7908,"duration_ms":79609,"concrete_test":"Recompute all metrics per item over the 101 examples for MC T=2, T=0.8, and T≈0, then report bootstrap 95% confidence intervals and paired Wilcoxon signed-rank tests for CE, EntCE, DistCE, L1, and ECE. If the T=2-versus-T=0.8 differences are not significant at α=0.05 or the confidence intervals cross zero, the Section 5 higher-temperature claim should be removed or replaced by a statement that temperature effects are metric-dependent and not statistically distinguishable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's temperature-specific conclusion in Section 5 ('sampling with higher temperatures reduced such overconfident predictions, resulting in distributions that are more closely calibrated with human opinions') is not supported by the paper's own results. In Table 2, for cross-entropy (lower is better), MC T=2 is worse than MC T=0.8 on every dataset: HS-Brexit 0.52 vs 0.34, ConvAbuse 1.25 vs 0.68, MD-Agree 1.35 vs 1.19; T=2 is also worse than T≈0 on HS-Brexit and MD-Agree. For ConvAbuse, T=2 is worse on EntCE, DistCE, and L1 as well. Only on MD-Agree do EntCE, DistCE, and L1 favor T=2, and the margins are 0.01–0.07. Table 3 ECE favors T=2, but ECE measures confidence calibration, not alignment with human opinions. With n=101 per dataset, no error bars, no paired significance tests, and no disclosed subset selection, the claimed temperature effect is not established. The broader claim that sampling/log-probability methods beat direct prompting is well supported (CE drops from 3.45–4.65 to at most 1.35), so the vulnerable part is specifically the temperature narrative in the title, discussion, and conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three ways of extracting opinion distributions from GPT-3.5-turbo on three binary subjective classification tasks from SemEval-2023: direct prompting for a distribution, Monte Carlo estimation by sampling repeated responses, and log-probability estimation by averaging normalized top-k token probabilities. Each method is evaluated under three temperature settings (T=2, T=0.8, T≈0) using cross-entropy, Jensen-Shannon divergence, L1 distance, distribution calibration error, entropy calibration error, and expected calibration error. The main empirical claim is that sampling and log-probability approaches with simple parameter adjustments align better with human opinion distributions than direct prompting; the discussion and conclusion additionally claim that higher temperatures reduce overconfidence and yield distributions more closely calibrated with human opinions.","tokens_in":6414,"tokens_out":3943,"duration_ms":41350,"significance":"The comparison of extraction methods is a useful practical contribution: the improvements over direct prompting are large and consistent across all three datasets (cross-entropy drops from roughly 3.45-4.65 to at most 1.35), so the central methodological recommendation is credible and likely to be useful to practitioners. The paper also draws attention to the importance of reporting temperature settings, which is a timely point. However, the temperature-specific narrative is not supported by the reported numbers: the key alignment metric (cross-entropy) favours lower temperatures, while the metrics that favour higher temperature differ by very small margins, and no uncertainty quantification or significance testing is provided. The paper therefore needs to be revised to either establish the temperature effect rigorously or substantially tone down that claim.","major_comments":[{"comment":"The claim that 'sampling with higher temperatures reduced such overconfident predictions, resulting in distributions that are more closely calibrated with human opinions' is contradicted by the paper's own cross-entropy results. Cross-entropy is lower-is-better, and in Table 2 MC T=2 is worse than MC T=0.8 on every dataset (HS-Brexit 0.52 vs 0.34, ConvAbuse 1.25 vs 0.68, MD-Agree 1.35 vs 1.19) and worse than MC T≈0 on HS-Brexit and MD-Agree. Only a subset of metrics (EntCE, DistCE, L1) favours T=2, and those gains are small (0.01-0.07) and not consistent across datasets. With no significance tests or error bars, the paper should not state the high-temperature conclusion as a finding; please report paired comparisons or confidence intervals and revise the conclusion accordingly.","section":"§5, Table 2"},{"comment":"The selection of the 'representative subset of 101 examples' is not described, despite the paper later noting that the sample size was 'carefully chosen to represent the overall distribution'. Because all results rest on n=101 per dataset and the temperature-related differences are small, the lack of a documented sampling procedure leaves open the possibility that the temperature effects are artefacts of the particular subset. Please specify exactly how the 101 examples were selected and provide bootstrap confidence intervals or permutation tests for the reported differences.","section":"§3, Methodology; §7, Limitations"},{"comment":"The ECE results are used to support the temperature narrative, but ECE measures confidence calibration, not alignment with human opinions, which is the paper's stated focus. Moreover, the ECE values in Table 3 are extremely high (0.92-0.99), indicating poor calibration in absolute terms, and no test is reported for the differences between temperature conditions. Saying that the results are 'consistent with [12]' is not sufficient; if ECE is to be part of the argument, it needs a proper analysis and an explicit caveat that it is not an alignment metric.","section":"§5, Table 3"}],"minor_comments":[{"comment":"There is a typo: 'allowing models to chose beyond' should be 'allowing models to choose beyond'.","section":"§1"},{"comment":"The column header 'J SD' has an unnecessary space, and the dataset name 'MD-Agree.' is inconsistently punctuated across tables and text.","section":"Table 2"},{"comment":"The notation 'H(.)' is introduced in running text but not defined as a function symbol; please define it explicitly when first used.","section":"§3.2"},{"comment":"The twelve panels in Figure 2 are too small to read the histograms; please enlarge the panels or split the figure by dataset so that entropy distributions are visually comparable.","section":"Figure 2"},{"comment":"The manuscript does not state whether code, the exact 101-example subset, or raw predictions will be released; for an empirical paper whose contribution is the comparison, this limits reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's core comparison of extraction methods is solid and likely useful, but the title and conclusion overstate a temperature effect that the reported results do not establish. A revised version that reframes the contribution as a method comparison, with temperature as a preliminary exploratory factor, and that provides uncertainty quantification for the temperature comparisons, would be substantially stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on opinion extraction or temperature reporting. The paper does one thing well: it tests three methods (direct prompting, Monte Carlo sampling, log-probability averaging) at three temperatures on three subjective classification benchmarks, and shows that sampling and log-probability clearly beat direct prompting. That finding is consistent across all metrics and datasets, with cross-entropy dropping from roughly 4 to below 1.4. It also correctly notes that prior work rarely reports temperature, which is a real methodological gap.\n\nSoft spots: the title and conclusion claim that higher temperature improves alignment with human opinions. Table 2 does not support that cleanly. On cross-entropy, T=2 is the worst of the three sampling temperatures on all three datasets; T=0.8 is best on ConvAbuse and MD-Agree, while T≈0 is best on HS-Brexit. Higher temperature only wins on some secondary metrics (EntCE, DistCE, L1) for MD-Agree. So the temperature narrative is an over-reading of mixed, metric-dependent results. There are no error bars or significance tests, and with n=101 per dataset, the differences could easily be noise. The subset selection is described as \"carefully chosen\" but not disclosed, which is a reproducibility issue. The paper acknowledges the Pann≈P assumption in the Limitations, which is good, but sparse annotators in ConvAbuse make that evaluation target shaky.\n\nWho it is for: people doing extraction of opinion distributions from LLMs and researchers thinking about evaluation methodology. It is a workshop-grade empirical note, not a definitive study. I would send it to review rather than desk reject, because the method comparison is useful and the temperature-reporting point is worth raising, but I would ask for major revisions: report standard errors, run paired significance tests, describe subset selection, and tone down the temperature claim to what the data actually show.","headline":"A useful method comparison with an overstated temperature story: sampling/log-prob beat direct prompting, but the paper's own table does not support higher temperature improving alignment.","tokens_in":6911,"tokens_out":3815,"would_cite":false,"duration_ms":32868,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sampling at higher temperature aligns LLMs with human opinions","keywords":["opinion alignment","temperature sampling","Monte Carlo estimation","log-probability estimation","human disagreement","subjective classification","LLM uncertainty","overconfidence"],"falsifier":"Run the same three methods on a fresh random sample of items from each SemEval-2023 dataset, with ten samples and temperatures near zero, 0.8, and 2.0; if direct prompting matches or beats Monte Carlo sampling on cross-entropy and L1 distance in a dataset, the claimed advantage is not general.","tokens_in":5953,"feed_emoji":"🌡️","tokens_out":7445,"duration_ms":72339,"temperature":0.7,"pith_summary":"The paper investigates whether simple changes to how an LLM is queried can make its outputs better reflect the distribution of human opinions in subjective classification. On three binary tasks from the SemEval 2023 'Learning with Disagreements' datasets, it compares directly prompting gpt-3.5-turbo for an opinion distribution with two alternatives: Monte Carlo sampling of repeated responses and repeated log-probability extraction. The central finding is that both sampling-based methods, particularly with higher temperature settings, produce distributions closer to aggregated human annotator votes than direct prompting does, mainly because higher temperatures reduce the model's overconfident zero-entropy predictions on items where humans mostly agree. The paper argues this matters because LLMs are increasingly used where people legitimately disagree, and temperature is a parameter that is often unreported yet strongly affects how pluralistic model outputs appear.","feed_headline":"Sampling at higher temperature aligns LLMs with human opinions","feed_subtitle":"Raising the LLM's sampling temperature beats direct prompting on three subjective abuse and offensiveness tasks.","key_machinery":"The load-bearing mechanism is the temperature parameter of the LLM's sampling distribution, which controls how much the model deviates from always choosing the maximum-likelihood token. At temperatures near zero the model becomes deterministic and overconfident; as temperature rises, sampled responses spread out across label categories. The paper estimates opinion distributions through Monte Carlo estimation, the frequency of valid responses over ten repeated samples, and log-probability estimation, averaging normalized exponentiated log-probabilities over samples. The comparison is measured with instance-level metrics: cross-entropy, Jensen-Shannon divergence, L1 distance, DistCE, EntCE, and expected calibration error.","core_discovery":"The paper's claim is that temperature-adjusted sampling is a viable, parameter-light way to align LLM distributions with human opinions on subjective tasks. The authors implement Monte Carlo estimation, sampling ten responses and counting valid category labels, and log-probability estimation, averaging normalized exponentiated log-probabilities of the top ten tokens across repeated samples, each at temperatures 2.0, 0.8, and near zero, and compare them against direct prompting. Across the three datasets, the sampling-based methods beat direct prompting on cross-entropy, Jensen-Shannon divergence, L1 distance, and the DistCE and EntCE calibration metrics; Monte Carlo estimation had the lowest scores overall, though the gap with log-probability estimation was small. At higher temperatures the model produced fewer zero-entropy distributions, and model entropy tracked human entropy more closely, while expected calibration error results were consistent with prior work showing that temperature can improve uncertainty calibration. The paper therefore proposes that temperature should be reported and considered in opinion-alignment studies, while cautioning that the evaluation target is the aggregated annotator distribution.","pith_inferences":["If temperature is a stand-in for the spread of opinions a model can express, the optimal setting may scale with the amount of annotator disagreement in the target data; a natural extension is to tune temperature per item or per dataset against annotator entropy.","Because the evaluation target is an aggregated annotator distribution, these results speak to matching crowd averages rather than representing any particular perspective; future work could weight annotators or model subpopulations separately.","The same mechanism suggests temperature-based sampling could improve calibration in other tasks where human label variation is treated as uncertainty, such as natural language inference or safety classification."],"forward_implications":["Higher-temperature Monte Carlo sampling can replace direct prompting as a simple baseline for extracting opinion distributions in subjective classification.","Temperature should be reported alongside results in any study that extracts opinion distributions from LLMs, since it materially changes apparent alignment.","Entropy-based metrics should not be used alone: the paper shows that two very different distributions can share the same entropy, so L1 or distributional distances are also needed.","The method works without extra human explanations or auxiliary models, making it cheap to apply to new subjective tasks.","The same temperature mechanism that improves alignment also improves expected calibration error, linking opinion pluralism with uncertainty calibration."],"supporting_citations":[{"why":"Supplies the Monte Carlo and log-probability estimation procedures that the paper adapts, and the earlier observation that GPT models produce overconfident opinion distributions.","marker":"[6]"},{"why":"Earlier study that extracted opinion distributions from token log-probabilities and found GPT outputs overly confident, the overconfidence pattern this paper targets.","marker":"[5]"},{"why":"Defines the direct prompting approach that serves as the baseline for comparison.","marker":"[7]"},{"why":"Provides the SemEval-2023 Task 11 datasets and the 101-example test subsets used in all experiments.","marker":"[8]"},{"why":"Defines the DistCE and EntCE metrics used to measure alignment with human distributions.","marker":"[10]"},{"why":"Prior result that temperature increases improve uncertainty calibration, which the paper's ECE results are said to be consistent with.","marker":"[12]"}],"fun_headline_variants":["Higher temperature sampling aligns LLMs with human opinions","Sampling at higher temps beats prompting for opinion alignment","Heat up LLM sampling to match human subjective views","Temperature-tuned sampling aligns LLMs to human judgments","Warm sampling, not prompts, aligns LLMs with human views"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the aggregated human-annotator vote distribution as the true distribution of human opinions, even though some items were annotated by only two or three people and the 101-item subset was asserted to be representative without a documented selection procedure.","fun_headline_variants_meta":{"raw":{"variants":["Higher temperature sampling aligns LLMs with human opinions","Sampling at higher temps beats prompting for opinion alignment","Heat up LLM sampling to match human subjective views","Temperature-tuned sampling aligns LLMs to human judgments","Warm sampling, not prompts, aligns LLMs with human views"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1137,"prompt_tokens":830,"completion_tokens":307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":229}},"tokens_in":446,"tokens_out":307,"duration_ms":3426,"temperature":1.0,"reasoning_tokens":229,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:58:42.639870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three methods on a fresh random sample of items from each SemEval-2023 dataset, with ten samples and temperatures near zero, 0.8, and 2.0; if direct prompting matches or beats Monte Carlo sampling on cross-entropy and L1 distance in a dataset, the claimed advantage is not general.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Monte Carlo and log-probability estimation procedures that the paper adapts, and the earlier observation that GPT models produce overconfident opinion distributions."},{"cited_title":"The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation","cited_arxiv_id":null,"evidence_quote":"Defines the direct prompting approach that serves as the baseline for comparison."},{"cited_title":"SemEval-2023 task 11: Learning with disagreements (LeWiDi)","cited_arxiv_id":null,"evidence_quote":"Provides the SemEval-2023 Task 11 datasets and the 101-example test subsets used in all experiments."},{"cited_title":"Stop measuring calibration when humans disagree","cited_arxiv_id":null,"evidence_quote":"Defines the DistCE and EntCE metrics used to measure alignment with human distributions."},{"cited_title":"SPUQ: Perturbation-based uncertainty quantification for large language models","cited_arxiv_id":null,"evidence_quote":"Prior result that temperature increases improve uncertainty calibration, which the paper's ECE results are said to be consistent with."}],"review_version":1}