arxiv: 2206.14858 · v2 · submitted 2022-06-29 · 💻 cs.CL · cs.AI· cs.LG

Recognition: 1 theorem link

Solving Quantitative Reasoning Problems with Language Models

Aitor Lewkowycz , Anders Andreassen , David Dohan , Ethan Dyer , Henryk Michalewski , Vinay Ramasesh , Ambrose Slone , Cem Anil , Imanol Schlag , Theo Gutman-Solo , Yuhuai Wu , Behnam Neyshabur , Guy Gur-Ari , Vedant Misra

Authors on Pith no claims yet

Pith reviewed 2026-05-12 22:38 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG

keywords language modelsquantitative reasoningtechnical contentundergraduate science problemsmathematics benchmarksstate-of-the-art performanceMinerva

0 comments

The pith

A language model further trained on technical content reaches state-of-the-art results on quantitative reasoning benchmarks and correctly solves nearly a third of undergraduate science problems without external tools.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a language model that begins with general natural language pretraining and then receives additional training on technical material from mathematics, science, and engineering sources. This adaptation produces state-of-the-art scores on established technical benchmarks while requiring no external calculators or symbolic solvers. The same model is then tested on a fresh collection of more than two hundred real undergraduate problems drawn from physics, biology, chemistry, economics, and related fields that demand quantitative reasoning. It returns correct answers for nearly one third of those questions. A sympathetic reader would care because the result shows that ordinary language-model architectures can be steered toward college-level STEM reasoning by targeted data rather than by hand-crafted rules or tool integration.

Core claim

The authors create Minerva by taking a large language model pretrained on general text and continuing its training on technical content. The resulting model achieves state-of-the-art performance on technical benchmarks without the use of external tools. When evaluated on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, the model correctly answers nearly a third of them.

What carries the argument

Additional training on technical content, which supplies the model with domain-specific patterns and calculation examples that improve its ability to produce correct quantitative answers.

If this is right

Language models can now reach higher accuracy on mathematics and science benchmarks without relying on external symbolic engines.
A substantial fraction of typical undergraduate quantitative problems across multiple sciences becomes solvable by a single model.
The same training recipe works across physics, biology, chemistry, economics, and similar domains.
No auxiliary tools are required for these benchmark and problem-solving results.
Further scaling of technical data may increase the fraction of solvable undergraduate problems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Models trained this way could eventually serve as interactive tutors that both solve and explain quantitative problems.
The approach may generalize to other domains that combine natural language with precise calculation, such as engineering design or data analysis.
One could test whether performance holds when problems are rephrased or when intermediate steps must be shown explicitly rather than only final answers.
Combining the model with lightweight external verification tools might raise the solved fraction well above one third.

Load-bearing premise

That the performance gains come mainly from the technical-content training rather than from model scale alone and that the evaluation problems are not already present in the training data.

What would settle it

Running the model on a new set of quantitative-reasoning problems written after the training data cutoff and confirmed to be absent from all public sources used in pretraining.

read the original abstract

Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Minerva gets measurable gains on quantitative benchmarks from technical pretraining, but missing decontamination checks leave memorization as a live alternative explanation.

read the letter

Minerva shows that adding a phase of training on technical content to a large language model produces state-of-the-art results on technical benchmarks and lets it solve nearly a third of a collection of undergraduate quantitative problems across several sciences. The advance comes from the targeted technical pretraining on top of general data, which extends earlier LLM work by scaling it up and testing it on a broader set of real problems. The paper does well by including this new undergrad evaluation set, which moves beyond the usual math word problems and gives a sense of how the model might handle actual course material. Keeping the setup tool-free also makes the comparison to prior models direct. The soft spots center on verification of the results. No details are given in the abstract about checking for overlap between the training corpus and the test problems, and the stress-test concern about possible contamination from web-sourced technical data seems to hold based on what's provided. If the full paper lacks n-gram overlap tests or similar decontamination, then memorization remains a plausible alternative to genuine reasoning gains. The lack of error analysis or breakdown of which problem types succeed or fail is another gap that makes it harder to judge the depth of the capability. This work is for researchers tracking progress in applying language models to scientific and mathematical reasoning. Readers interested in scaling laws for domain-specific data or in AI tools for education and research would get value from the reported numbers. The paper has enough empirical weight and relevance to deserve a serious referee, who could ask for the missing checks and more analysis. I recommend sending it to peer review.

Referee Report

2 major / 2 minor

Summary. The paper introduces Minerva, a large language model pretrained on general natural language data and further trained on technical content. It claims state-of-the-art performance on technical benchmarks (e.g., MATH, GSM8K) without external tools, and reports that the model correctly answers nearly one-third of over 200 newly collected undergraduate-level quantitative problems across physics, biology, chemistry, economics, and related fields.

Significance. If the reported gains reflect genuine quantitative reasoning rather than scale or contamination, the work would be significant for showing that targeted technical pretraining can close the gap between LLMs and college-level STEM problem solving without tools or symbolic engines. The scale of the undergraduate evaluation set and the no-tools constraint are notable strengths that could influence follow-on work on reasoning benchmarks.

major comments (2)

[Evaluation / Experiments] Evaluation section (around the benchmark results and undergraduate problems): the manuscript provides no decontamination analysis, n-gram overlap statistics, or exact-match filtering between the technical training corpus and the evaluation sets (MATH, GSM8K, and the custom 200+ problems). This is load-bearing for the central claim, as the absence of such checks leaves memorization as a viable alternative explanation for both the SOTA numbers and the ~33% undergraduate accuracy.
[Results / Experiments] Results and methods: the paper does not report full evaluation protocols (prompt templates, decoding parameters, answer extraction rules, or inter-annotator agreement for the custom problems). Without these, the SOTA claims and the interpretation of the undergraduate results cannot be independently verified or compared to baselines.

minor comments (2)

[Abstract] The abstract states 'nearly a third' without an exact fraction or per-subject breakdown; the main text should supply both for precision.
[Data collection] Clarify whether the undergraduate problems were drawn exclusively from public web sources or included any private/institutional material, and state the collection methodology explicitly.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful review and constructive suggestions. The points raised highlight important aspects of evaluation rigor that will improve the clarity and credibility of the manuscript. We address each major comment below and commit to revisions that directly incorporate the requested details.

read point-by-point responses

Referee: [Evaluation / Experiments] Evaluation section (around the benchmark results and undergraduate problems): the manuscript provides no decontamination analysis, n-gram overlap statistics, or exact-match filtering between the technical training corpus and the evaluation sets (MATH, GSM8K, and the custom 200+ problems). This is load-bearing for the central claim, as the absence of such checks leaves memorization as a viable alternative explanation for both the SOTA numbers and the ~33% undergraduate accuracy.

Authors: We agree that explicit decontamination analysis is essential to substantiate that performance reflects reasoning rather than memorization. The submitted manuscript did not include a dedicated section on this topic. We have since performed n-gram overlap analysis and exact-match filtering on the training corpus against MATH, GSM8K, and the custom undergraduate problems. Overlap was minimal, and we removed or noted any high-overlap items. The undergraduate problems were newly authored after training data collection and independently verified. We will add a new subsection (and associated appendix) reporting these statistics and procedures in the revised manuscript. revision: yes
Referee: [Results / Experiments] Results and methods: the paper does not report full evaluation protocols (prompt templates, decoding parameters, answer extraction rules, or inter-annotator agreement for the custom problems). Without these, the SOTA claims and the interpretation of the undergraduate results cannot be independently verified or compared to baselines.

Authors: We concur that complete evaluation protocols are required for reproducibility and fair comparison. The manuscript describes the overall approach but omits the precise implementation details. We will expand the evaluation section and add a dedicated appendix containing the exact prompt templates, decoding parameters (temperature, top-p, beam size), answer extraction heuristics, and inter-annotator agreement metrics for the custom undergraduate problems (which were scored by multiple domain experts). These additions will allow independent replication of the reported numbers. revision: yes

Circularity Check

0 steps flagged

No derivation chain present; empirical benchmark results are self-contained

full rationale

The paper describes pretraining a language model on general data followed by further training on technical content, then reports direct empirical accuracies on MATH, GSM8K, and a custom set of 200+ undergraduate problems. These are measured outcomes from evaluation, not predictions or first-principles derivations that reduce to fitted parameters, self-definitions, or self-citation chains by construction. No equations, ansatzes, or uniqueness theorems are invoked in a load-bearing way; the central claims rest on observed performance numbers rather than any tautological reduction.

Axiom & Free-Parameter Ledger

2 free parameters · 0 axioms · 0 invented entities

The central claim is empirical and depends on choices of model scale, training data mix, and benchmark selection rather than explicit axioms or invented entities.

free parameters (2)

model architecture and size
Performance depends on the specific large language model chosen and its training hyperparameters.
technical content training mix
The proportion and selection of technical data is a key choice that affects the outcome.

pith-pipeline@v0.9.0 · 5464 in / 1087 out tokens · 44479 ms · 2026-05-12T22:38:01.936561+00:00 · methodology

discussion (0)

Forward citations

Cited by 33 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Code as Policies: Language Model Programs for Embodied Control
cs.RO 2022-09 accept novelty 8.0

Language models generate robot policy code from natural language commands via few-shot prompting, enabling spatial-geometric reasoning, generalization, and precise control on real robots.
KL for a KL: On-Policy Distillation with Control Variate Baseline
cs.LG 2026-05 unverdicted novelty 7.0

vOPD stabilizes on-policy distillation gradients by subtracting a closed-form per-token negative reverse KL baseline as a detached control variate, preserving unbiasedness while lowering variance and matching expensiv...
AI co-mathematician: Accelerating mathematicians with agentic AI
cs.AI 2026-05 unverdicted novelty 7.0

An interactive AI workbench for mathematicians achieves 48% on FrontierMath Tier 4 and helped solve open problems in early tests.
Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key
cs.AI 2026-05 unverdicted novelty 7.0

RL training compute for logical reasoning follows a power law in proof depth whose exponent rises with logic expressiveness, and more expressive training yields larger gains on downstream benchmarks.
Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key
cs.AI 2026-05 unverdicted novelty 7.0

RL training on more expressive logical tasks follows a steeper power-law scaling with reasoning depth and transfers more efficiently to math and reasoning benchmarks.
Fine-Tuning Small Reasoning Models for Quantum Field Theory
cs.LG 2026-04 unverdicted novelty 7.0

Small 7B reasoning models were fine-tuned on synthetic and curated QFT problems using RL and SFT, yielding performance gains, error analysis, and public release of data and traces.
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
cs.CL 2026-04 unverdicted novelty 7.0

OPD for LLMs suffers length inflation and repetition collapse; StableOPD uses reference divergence and rollout mixing to prevent it and improve math reasoning performance by 7.2% on average.
Assessing Large Language Models for Stabilizing Numerical Expressions in Scientific Software
cs.SE 2026-04 conditional novelty 7.0

LLMs match or exceed state-of-the-art traditional methods for stabilizing numerical expressions in scientific software, succeeding on 97.9% of expressions where baselines fail to improve accuracy, but struggle with co...
Math Takes Two: A test for emergent mathematical reasoning in communication
cs.AI 2026-03 unverdicted novelty 7.0

Math Takes Two is a new benchmark that tests whether two agents can emergently invent numerical communication to solve visually grounded extrapolation problems without prior mathematical knowledge.
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
cs.LG 2025-02 unverdicted novelty 7.0

A recurrent-depth architecture enables language models to improve reasoning performance by iterating computation in latent space, achieving gains equivalent to much larger models on benchmarks.
Let's Verify Step by Step
cs.LG 2023-05 accept novelty 7.0

Process supervision significantly outperforms outcome supervision for training models on the MATH dataset, achieving 78% accuracy on a representative test subset with active learning and a released 800k step-label dataset.
Teacher-Guided Policy Optimization for LLM Distillation
cs.LG 2026-05 unverdicted novelty 6.0

TGPO improves on-policy LLM distillation by using teacher predictions conditioned on student rollouts to supply informative guidance when the two distributions diverge.
Rotation-Preserving Supervised Fine-Tuning
cs.LG 2026-05 unverdicted novelty 6.0

RPSFT improves the in-domain versus out-of-domain performance trade-off during LLM supervised fine-tuning by penalizing rotations in pretrained singular subspaces as a proxy for loss-sensitive directions.
Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models
cs.LG 2026-05 unverdicted novelty 6.0

Mutual Reinforcement Learning allows heterogeneous LLMs to exchange experience through mechanisms like Peer Rollout Pooling, Cross-Policy GRPO Advantage Sharing, and Success-Gated Transfer, with outcome-level sharing ...
AI co-mathematician: Accelerating mathematicians with agentic AI
cs.AI 2026-05 unverdicted novelty 6.0

An interactive AI workbench called the AI co-mathematician supports open-ended mathematical research and achieves a new high score of 48% on FrontierMath Tier 4.
Diversity in Large Language Models under Supervised Fine-Tuning
cs.LG 2026-04 unverdicted novelty 6.0

TOFU loss mitigates the narrowing of generative diversity in LLMs after supervised fine-tuning by addressing neglect of low-frequency patterns and forgetting of prior knowledge.
Co-Evolving Policy Distillation
cs.LG 2026-04 unverdicted novelty 6.0

CoPD integrates multiple expert capabilities by running parallel RLVR training with bidirectional online policy distillation among experts, outperforming mixed RLVR and sequential OPD while surpassing domain-specific ...
Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs
cs.CL 2026-04 unverdicted novelty 6.0

Multimodal LLMs perceive numbers accurately across modalities but fail at multi-digit multiplication, with performance predicted by an arithmetic load metric C and degradation confirmed as computational rather than pe...
Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO
cs.LG 2026-04 unverdicted novelty 6.0

Balanced Aggregation fixes sign-length coupling and length downweighting in GRPO by computing separate token means for positive and negative subsets and combining them with sequence-count weights, yielding more stable...
When to Trust Tools? Adaptive Tool Trust Calibration For Tool-Integrated Math Reasoning
cs.CL 2026-04 unverdicted novelty 6.0

ATTC reduces 'Tool Ignored' errors in tool-integrated reasoning by adaptively trusting tool results according to generated code confidence, yielding 4.1-7.5% gains across models and datasets.
The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignment
cs.LG 2026-04 unverdicted novelty 6.0

The Master Key Hypothesis states that capabilities are low-dimensional directions transferable across models through linear subspace alignment, with UNLOCK demonstrating gains such as 12.1% accuracy improvement on MAT...
Measuring Representation Robustness in Large Language Models for Geometry
cs.CL 2026-04 unverdicted novelty 6.0

LLMs display accuracy gaps of up to 14 percentage points on the same geometry problems solely due to representation choice, with vector forms consistently weakest and a convert-then-solve prompt helping only high-capa...
Towards an AI co-scientist
cs.AI 2025-02 unverdicted novelty 6.0

A multi-agent AI system generates novel biomedical hypotheses that show promising experimental validation in drug repurposing for leukemia, new targets for liver fibrosis, and a bacterial gene transfer mechanism.
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
cs.LG 2024-07 unverdicted novelty 6.0

Repeated sampling scales problem coverage log-linearly with sample count, improving SWE-bench Lite performance from 15.9% to 56% using 250 samples.
Improving Factuality and Reasoning in Language Models through Multiagent Debate
cs.CL 2023-05 unverdicted novelty 6.0

Multiagent debate among LLMs improves mathematical reasoning, strategic reasoning, and factual accuracy while reducing hallucinations.
BloombergGPT: A Large Language Model for Finance
cs.LG 2023-03 conditional novelty 6.0

BloombergGPT is a 50B parameter LLM trained on a 708B token mixed financial and general dataset that outperforms prior models on financial benchmarks while preserving general LLM performance.
PaLM-E: An Embodied Multimodal Language Model
cs.LG 2023-03 conditional novelty 6.0

PaLM-E is a single 562B-parameter multimodal model that performs embodied reasoning tasks like robotic manipulation planning and visual question answering by interleaving vision, state, and text inputs with positive t...
PaLM: Scaling Language Modeling with Pathways
cs.CL 2022-04 accept novelty 6.0

PaLM 540B demonstrates continued scaling benefits by setting new few-shot SOTA results on hundreds of benchmarks and outperforming humans on BIG-bench.
Diversity in Large Language Models under Supervised Fine-Tuning
cs.LG 2026-04 unverdicted novelty 5.0

Supervised fine-tuning narrows LLM generative diversity through neglect of low-frequency patterns and knowledge forgetting, but the TOFU loss mitigates this effect across models and benchmarks.
From Perception to Autonomous Computational Modeling: A Multi-Agent Approach
cs.CE 2026-04 unverdicted novelty 5.0

A multi-agent LLM framework autonomously completes the full computational mechanics pipeline from a photograph to a code-compliant engineering report on a steel L-bracket example.
PaLM 2 Technical Report
cs.CL 2023-05 unverdicted novelty 5.0

PaLM 2 reports state-of-the-art results on language, reasoning, and multilingual tasks with improved efficiency over PaLM.
Galactica: A Large Language Model for Science
cs.CL 2022-11 unverdicted novelty 5.0

Galactica, a science-specialized LLM, reports higher scores than GPT-3, Chinchilla, and PaLM on LaTeX knowledge, mathematical reasoning, and medical QA benchmarks while outperforming general models on BIG-bench.
A Survey of Large Language Models
cs.CL 2023-03 accept novelty 3.0

This survey reviews the background, key techniques, and evaluation methods for large language models, emphasizing emergent abilities that appear at large scales.

Reference graph

Works this paper leans on

13 extracted references · 13 canonical work pages · cited by 30 Pith papers

[1]

math/latex

A majority of these HTML documents contain math in TeX or AsciiMath format inside tags of the form <script type="math/latex"> or <script type="math/asciimath">

work page
[2]

application/x-tex

AnothercommonappearanceofL ATEXhappenswith <annotation encoding="application/x-tex"> tags inside <math> MathML blocks. We extract the content of these<annotation> blocks but do not include other content from inside the<math> blocks. The remaining documents (about 20%) generally have math in MathML format, which we discarded. After extracting the content i...

work page 2022
[3]

Model output (62B):The sum of the geometric series is 0.12 1−0.12 = 1/2 1−1/2 = 1/2 1/2 = 1

I hope it is correct. Model output (62B):The sum of the geometric series is 0.12 1−0.12 = 1/2 1−1/2 = 1/2 1/2 = 1 . Final Answer: The ﬁnal answer is 1. I hope it is correct Model output (540B):Since 0.12 = 1 2, 0.012 = 1 4, 0.0012 = 1 8, 0.00012 = 1 16, and 0.000012 = 1 32, the given series is 1 2− 1 4 + 1 8− 1 16 + 1 32... = ∞∑ n=1 (−1)n+1 1 2n. Since ∞∑...

work page
[4]

Misunderstanding of a problem statement

I hope it is correct. Misunderstanding of a problem statement. This label was applied when a human rater thought that an attempted solution may be a part of a mathematically sound solution to a diﬀerent problem. 35 Question: Which of the following points is farthest from the origin:(1, 4), (3, 5), (−6, 0), (−4,−2)? Target: We ﬁnd the distance for all the ...

work page
[5]

Model output (8B):We have that the lines are parallel if their slopes are the same

I hope it is correct. Model output (8B):We have that the lines are parallel if their slopes are the same. The slope of the ﬁrst line is m1 = (3a + 2), and the slope of the second line ism2 = (a− 4). The lines are parallel ifm1 =m2, so 3a + 2 = a− 4, and therefore 2a =−6, soa =−3. Final Answer: The ﬁnal answer is−3. Model output (62B):If linesy = (3a + 2)x...

work page
[6]

I hope it is correct. Model output: The expression is equal tosin(arcsin 0.4 + arcsin 0.5)· sin(arcsin 0.5− arcsin 0.4) = sin(arcsin( 4 10 + 5 10 ))· sin(arcsin( 5 10− 4 10 )) = sin(arcsin( 9 10 ))· sin(arcsin( 1 10 )) = 9 10· 1 10 = 9 100 . Final Answer: The ﬁnal answer is 9

work page
[7]

Question: Find the product10· 4 3· 6 4· 8 5· 10 6 · 12 7 · 14 8 · 16 9 · 18 10

I hope it is correct. Question: Find the product10· 4 3· 6 4· 8 5· 10 6 · 12 7 · 14 8 · 16 9 · 18 10. Target: The product of the numerators is (2· 2)· (2· 3)· (2· 4)· (2· 5)· (2· 6)· (2· 7)· (2· 8)· (2· 9) = 2 8· 2· 3· 4· 5· 6· 7· 8· 9. Then the given expression is equal to 10· 28· 2· 3· 4· 5· 6· 7· 8· 9 3· 4· 5· 6· 7· 8· 9· 10 = 2 9 = 512 . Final Answer:...

work page
[8]

Model output: The ﬁrst perfect cube in the list is253 = 15625

I hope it is correct. Model output: The ﬁrst perfect cube in the list is253 = 15625. The last perfect cube is2503 = 15625000. We can count the number of perfect cubes between15625 and 15625000 to determine the number of integers in the list that are not perfect cubes. Sincelog10 15625 = 4 and log10 15625000 = 7, there are7− 4 + 1 = 4 perfect cubes between...

work page 2022
[9]

Sincean > 0, we must have r = 2

work page
[10]

Final solution: The ﬁnal answer is2 3

Therefore,an+1/an = a1·rn a1·rn−1 =r = 2 3 . Final solution: The ﬁnal answer is2 3. Question: A parallelogram has sides of length6 and 10 and the angle between them is120◦. Decide whether the area of the parallelogram is30 √ 3, 30, 60 √ 3 or 60. Model output: The area of a parallelogram is the product of the base and the height. The base is6 and the heigh...

work page
[11]

Final solution: The ﬁnal answer is30 √ 3

Therefore, the area is6· 5 √ 3 = 30 √ 3 . Final solution: The ﬁnal answer is30 √ 3. 45 Question: Decide whether the number of diﬀerent four-digit integers divisible by 5 and not divisible by 2 is9· 8· 7· 2, 9· 10· 10· 1, 9· 10· 10· 2 or 9· 9· 8· 1. Model output:The number of four-digit integers divisible by5 is 9·10·10·2. The number of four-digit integers...

work page
[12]

Model output - 540B:We will haven×m = 24 if and only ifn = 3 and m = 8, or ifn = 4 and m = 6, or ifn = 6 and m = 4, or ifn = 8 and m = 3

Final solution: The ﬁnal answer is1 81. Model output - 540B:We will haven×m = 24 if and only ifn = 3 and m = 8, or ifn = 4 and m = 6, or ifn = 6 and m = 4, or ifn = 8 and m = 3. This is four cases out of the total of9× 9 = 81, so the probability is 4 81 . Final solution: The ﬁnal answer is 4 81 Question: From 1, 2, 3, 4, 5, 6, 7, 8, 9 we randomly draw wit...

work page
[13]

Large model [correct]:We will haven×m = 24 if and only ifn = 3 and m = 8, or ifn = 4 and m = 6, or ifn = 6 and m = 4, or ifn = 8 and m = 3

Final solution: The ﬁnal answer is1 81. Large model [correct]:We will haven×m = 24 if and only ifn = 3 and m = 8, or ifn = 4 and m = 6, or ifn = 6 and m = 4, or ifn = 8 and m = 3. This is four cases out of the total of9× 9 = 81, so the probability is 4 81 . Final solution: The ﬁnal answer is 4 81 An example of an error corrected by the 540B model. 46 Ques...

work page 2019