REVIEW 15 cited by
The Effect of Sampling Temperature on Problem Solving in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In this research study, we empirically investigate the effect of sampling temperature on the performance of Large Language Models (LLMs) on various problem-solving tasks. We created a multiple-choice question-and-answer (MCQA) exam by randomly sampling problems from standard LLM benchmarks. Then, we used nine popular LLMs with five prompt-engineering techniques to solve the MCQA problems while increasing the sampling temperature from 0.0 to 1.6. Despite anecdotal reports to the contrary, our empirical results indicate that changes in temperature from 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks. In addition, these results appear to generalize across LLMs, prompt-engineering techniques, and problem domains. All code, data, and supplemental materials are available on GitHub at: https://github.com/matthewrenze/jhu-llm-temperature
Forward citations
Cited by 15 Pith papers
-
More Is Not More: What Matters for Diversity in LLM Opinions?
Diversity in LLM opinions comes mostly from the first persona sentence and from combining different interaction architectures, not from richer personas, temperature, or diversity instructions.
-
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Faithful tool use is unsaturated (best Clean Tool-Use Rate 86.33%), and models with similar scores fail differently—especially Llama Always-Call versus disciplined families on no-tool controls.
-
KnowTrace: Bootstrapping Iterative Retrieval-Augmented Generation with Structured Knowledge Tracing
KnowTrace builds a question-specific knowledge graph during iterative retrieval and uses backtracing to filter useful reasoning steps, improving multi-hop QA and self-bootstrapping.
-
AuPair: Golden Example Pairs for Code Repair
Providing a different submodularly selected (guess, fix) example on each repair call improves LLM code repair over best-of-N and self-repair.
-
Enhance-A-Video: Better Generated Video for Free
Enhance-A-Video computes the mean off-diagonal temporal attention weight and uses it, scaled by a per-prompt temperature, to boost the attention residual in DiT models during inference.
-
Optimizing Temperature for Language Models with Multi-Sample Inference
Selecting the temperature at the entropy turning point of a language model's generated text yields near-optimal multi-sample inference accuracy without labeled validation data.
-
DRAssist: Dispute Resolution Assistance using Large Language Models
DRAssist shows that zero-shot LLMs with chain-of-thought prompting can outperform simple baselines at suggesting dispute outcomes, though the evaluation ground truth is itself generated by LLMs.
-
Statistical Hypothesis Testing for Auditing Robustness in Language Models
A permutation-based hypothesis test on pairwise semantic similarities detects whether LLM outputs shift under arbitrary input or model perturbations.
-
VModA: An Effective Framework for Adaptive NSFW Image Moderation
VModA combines prompt engineering, region zooming, and LLM-based answer aggregation to improve zero-shot NSFW image moderation across multiple categories.
-
PerPO: Perceptual Preference Optimization via Discriminative Rewarding
PerPO trains multimodal LLMs by ranking their candidate answers with deterministic visual rewards (IoU, edit distance) and using the reward differences as margins in listwise preference optimization.
-
ReservoirChat: Interactive Documentation Enhanced with LLM and Knowledge Graph for ReservoirPy
ReservoirChat, a RAG and knowledge-graph assistant for ReservoirPy, improves domain-specific question answering and code debugging over its base model, but its custom benchmark may be contaminated by its own knowledge base.
-
Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
In a new three-game benchmark tracking planning, revision, and budget use across 12 LLMs, ChatGPT-o3-mini ranked highest, while overcorrecting models such as Qwen-Plus won few matches.
-
Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities
A systematic review organizing LLM-based text-to-SQL methods into pre-processing, in-context learning, fine-tuning, and post-processing paradigms, with a catalog of datasets, metrics, challenges, and future directions.
-
CoLA: Collaborative Low-Rank Adaptation
CoLA generalizes LoRA to multiple A and B matrices with a principal-component initialization and reports gains of roughly 2-4 accuracy points over PiSSA on low-sample fine-tuning benchmarks.
-
The Paradox of Stochasticity: Limited Creativity and Computational Decoupling in Temperature-Varied LLM Outputs of Structured Fictional Data
In three LLMs generating fictional names and birthdates, model choice dominates processing time and default name archetypes persist across temperature, while rare names appear mainly at mid-range temperatures.
Discussion (0). Continue with ORCID to comment.