{"id":"de740594-d754-4f7b-bbc6-8dbd05a8e123","arxiv_id":"2506.21580","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs' performance on legal reasoning and on classical rationality tasks shows no statistically significant correlation across six models, suggesting domain expertise does not generalize to broader reasoning.","lead":"This paper tests six large language models on legal bar exam questions and on classic reasoning tasks, then checks whether skill in one predicts skill in the other. It finds no significant correlation and concludes that domain expertise does not transfer to general reasoning, a result with implications for how models are evaluated before deployment.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's p-values reveal n=7 correlations, where |r| must exceed 0.754 for significance, so the reported null result is a low-power artifact that cannot support the disconnection claim.","rationale":"The reader's rejection is correct, and the weakest assumption identified by the reader is the same one that is most load-bearing here: the correlation analysis has an unstated and severely underpowered unit of analysis. My stress-test sharpens this: the p-values in Table 4 are themselves diagnostic of n=7, because the exact p-values match the t-distribution with five degrees of freedom. This is an internal inconsistency with the paper's own description of 1,000 legal items and 240 rationality items across five rounds; all that evidence is collapsed into seven aggregate points per model. With n=7, the significance threshold is |r|>0.754, so the reported r values of −0.11 to 0.29 could never have been significant, and the claimed 'no correlation' is not a finding about the world but a near-guaranteed consequence of the design. The absence of data or code, noted by the reader, is secondary; even if the supplementary material were released, the analysis as described would still be invalid for the headline claim. An equivalence test or a multilevel model using all available items could in principle support the claim, but Table 4 as presented cannot. The paper does build a substantial benchmark and reports five evaluation rounds, which is a real asset, but the central inferential step discards that asset. Therefore the reader's REJECT verdict stands.","tokens_in":13768,"tokens_out":4844,"duration_ms":55110,"concrete_test":"Recompute the correlation analysis from the promised supplementary data after first verifying the unit of analysis. Take the seven per-domain MBE means and the four rationality-task means for each model and fit a mixed-effects model with random intercepts for model and prompt (or use all item-level responses), yielding a pooled law–rationality correlation with a bootstrap 95% CI. If the pooled CI is bounded away from zero, or even includes values above 0.2, Table 4's null result is a power artifact; if the CI is tightly centered near zero with an upper bound below 0.2, the disconnection claim would be supported. Also report the CI for each Table 4 row: with n=7, the row r=0.14 already has CI [−0.68, 0.82], which is decisive for interpretation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that narrow and broad cognitive abilities are disconnected rests entirely on the non-significant Pearson correlations in Table 4, and the reported p-values betray the problem: for every model, the p-value is exactly what one gets with n=7 observations. For example, with df=5, r=0.14 gives p≈0.76, r=0.29 gives p≈0.53, r=−0.11 gives p≈0.82, and r=0.22 gives p≈0.64, matching the table within rounding. The only observable structure with n=7 is the seven MBE domain averages paired against a single overall rationality score (or an analogous seven-point pairing), so each model's 'no significant relationship' is the product of a design in which a correlation must exceed |r|≈0.754 to reach p<0.05. Correlations of 0.1–0.3 are therefore automatically non-significant regardless of the true relationship, and the test has almost no power to detect a moderate transfer effect. A null result with n=7 is not evidence of absence: the 95% CI for r=0.14 is roughly [−0.68, 0.82], which includes very large positive or negative associations. The conclusion that 'narrow and broad cognitive abilities are disconnected' thus rests on accepting a confidence interval that is consistent with the opposite conclusion. The paper reports no equivalence test, no minimum detectable effect, and no multilevel analysis pooling the 1,000 legal items and 240 rationality items across six models, so the headline statistical claim is unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates six LLMs (ChatGPT-4, Gemini 1.0, ChatGPT-3.5, Claude 2, Llama 3, Mistral) on a simulated Multistate Bar Examination (1,000 questions across seven legal domains) and on four classical rationality tasks (240 items: Wason Selection Task, Conjunction Fallacy, Stereotype and Diagnostic Base Rate Neglect). It computes per-model Pearson correlations between average legal scores and average rationality scores, reports no statistically significant correlations, and interprets this as evidence that narrow domain expertise and broad reasoning abilities are disconnected in LLMs. It also reports qualitative findings on cognitive biases and test-retest inconsistency, and proposes a human-centered auditing framework.","tokens_in":14026,"tokens_out":6792,"duration_ms":63130,"significance":"If the transfer claim were rigorously established, the paper would make a useful contribution to LLM evaluation: it would caution against interpreting high scores on domain benchmarks as evidence of general reasoning, and it would support the need for multi-faceted auditing protocols. The work has strengths: it uses external, publicly motivated benchmarks rather than self-defined tasks, the task battery (MBE plus classical rationality instruments) is sensible, and the paper explicitly offers datasets and scripts as supplementary material. However, the current statistical analysis is not adequate to support the central claim, and the qualitative 'inconsistency' evidence is undocumented.","major_comments":[{"comment":"The reported correlation analysis is not reproducible because the sample size and unit of analysis are never stated. The p-values in Table 4 (e.g., r=0.14, p=0.76; r=0.29, p=0.53; r=−0.11, p=0.82) are exactly those expected with n=7 paired observations (df=5), and with n=7 the critical |r| for p<0.05 is approximately 0.754. The observed correlations of −0.11 to 0.29 therefore carry very wide confidence intervals that include substantial positive or negative associations, so the absence of significant p-values cannot support the claim that narrow and broad abilities are disconnected. The paper reports no equivalence test, no minimum detectable effect, and no multilevel or item-level analysis of the 1,000 legal items and 240 rationality items; as it stands, the central conclusion of Section 4.1 is unsupported by the evidence presented.","section":"Section 4.1, Table 4"},{"comment":"The claim that models 'frequently returned different answers' to identical prompts under deterministic conditions is made without any supporting data. The manuscript does not report the number of repeated questions, the number of models for which inconsistency was observed, any measure of agreement (e.g., percent repeat consistency), or the API parameters used. Because this test-retest instability is described as 'most concerning' and is repeated in the conclusion, it needs either a quantitative summary or removal from the paper.","section":"Section 4.3"},{"comment":"The operational definition of the paired observations used for the Pearson correlation is missing. The text says coefficients were calculated for each model, but Table 4 reports only two aggregate averages per model; it is unclear whether the correlation is computed over the seven legal domains, over the five evaluation rounds, or over some other grouping. Without this information, the statistical results cannot be checked, and the statement that the analysis is reproducible (Section 4) is not credible. The authors should specify the unit of analysis, state N, and ideally report item-level or multilevel estimates that pool information across the 1,240 available items.","section":"Methodology, Correlation Analysis"}],"minor_comments":[{"comment":"There is a factual conflict in model sizes: the methodology text says ChatGPT-4 has '175 billion parameters,' while Table 1 lists ChatGPT-4 as 'Estimated 1-1.76T,' and the GPT-4 technical report [52] does not disclose a parameter count. Please correct and cite a source for the reported size.","section":"Methodology and Table 1"},{"comment":"The header says 'Ranked by Model Size (Largest to Smallest),' but the rows do not follow the sizes given in Table 1; for example, ChatGPT-3.5 (≈175B) is placed after Gemini (≈70B) and before Claude (≈52–100B), and Claude is placed after ChatGPT-3.5. Either reorder the rows or change the header.","section":"Table 4"},{"comment":"The correlation coefficient is written as 'A' in the text (e.g., 'A = 0.14'), which appears to be a typo for 'r'; use a consistent symbol matching Table 4.","section":"Section 4.2"},{"comment":"Some symbols are missing in the text: 'All ?-values exceeded the 0.05 threshold' should read 'All p-values,' and 'con￿ated' should be 'conflated.'","section":"Section 4.1 and 4.3"},{"comment":"Reference [7] is incomplete ('Unknown Author, Unknown Journal, 2023'), and references [62] and [63] are duplicate citations of the same Tversky and Kahneman 1974 article. Please complete or remove [7] and merge the duplicates.","section":"References"},{"comment":"Figure 2 is referenced but the caption alone is not sufficient; please ensure the figure appears in the final version with labeled axes and data points for each model. Figure 3 is also referenced but not visible in the manuscript text.","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":"The statistical issue is serious enough that I would not accept the paper without a reanalysis. The p-values in Table 4 strongly suggest n=7; if the authors do not have item-level data, the transfer claim should be withdrawn. I also note that the paper's literature analysis of '2,024 publications' in Section 5 is presented without any search protocol or inclusion criteria, which makes it unverifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about arXiv:2506.21580 is that its central result is not credible. The paper claims that across six LLMs, performance on MBE legal questions does not correlate with performance on classic rationality tasks, and concludes that narrow and broad cognitive abilities are disconnected. But Table 4 reports Pearson r and p-values without sample size or unit of analysis, and the p-values are exactly what you get with n=7 (e.g., r=0.14, p=0.76; r=0.29, p=0.53). With n=7, a correlation has to exceed |r|≈0.75 to reach p<0.05. So the null result is a low-power artifact; the 95% CI on r=0.14 spans roughly [-0.68, 0.82]. The data are consistent with a large positive or negative relationship. You cannot conclude disconnection from this analysis.\n\nThat said, the paper has a plausible empirical angle: pairing MBE subdomains with four standard rationality tasks is a reasonable way to ask the transfer question, and the descriptive scores (e.g., ChatGPT-4 at 64.57 law vs 61.29 rationality) could be useful if the underlying data were public. The authors also cite the relevant literature, much of which already makes the point that high benchmark scores don't imply general reasoning.\n\nThe soft spots are substantial. No data, prompts, or code are actually provided, despite promises of supplementary material. The test-retest inconsistency claim in Section 4.3 has no counts or examples. The 'review of 2,024 publications' appears in the discussion with no methodology and no supplementary list. There is also an internal inconsistency: the text says ChatGPT-4 has 175B parameters, while Table 1 says 1-1.76T. These are not mere nitpicks; they compound the unreliability of the paper's empirical claims.\n\nI think the right call is to reject. The main statistical analysis is not just missing details—it is demonstrably underpowered, and the authors' conclusion overstates what the data could show. If the authors release the item-level data and a properly specified multilevel analysis, the paper could be reconsidered as a descriptive benchmark. As it stands, I would not send it to peer review.","headline":"The paper's headline claim of no transfer between legal reasoning and general rationality is an artifact of an underpowered n=7 correlation analysis, and the paper ships none of the data needed to check it.","tokens_in":14622,"tokens_out":4784,"would_cite":false,"duration_ms":42192,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that in six large language models, performance on a simulated bar exam is statistically uncorrelated with performance on classical reasoning tests, implying that narrow and broad cognitive abilities are disconnected in…","keywords":["LLM reasoning","domain generalization","legal reasoning","rationality","cognitive bias","correlation analysis","Multistate Bar Examination","Wason selection task"],"falsifier":"A concrete check would be to recompute the correlation using the underlying per-domain or per-item scores instead of one average law score and one average rationality score per model; if any model shows a statistically significant positive correlation across legal subdomains and rationality tasks, the disconnection claim collapses. A second check is to rerun identical prompts with temperature fixed: if the same model gives the same answer across repeated trials, the reported test-retest inconsistency is overturned.","tokens_in":13539,"feed_emoji":"⚖️","tokens_out":5928,"duration_ms":56263,"temperature":0.7,"pith_summary":"This paper tries to establish that the general reasoning ability of large language models is disconnected from their performance on specialized, knowledge-heavy tasks. Concretely, it measures six LLMs on 1,000 simulated Multistate Bar Examination questions and on 240 items from classical rationality tests, then correlates the two average scores for each model. In every model the correlation is weak and not statistically significant, and the authors conclude that domain expertise does not transfer to general reasoning. The stakes: if true, strong benchmark performance in law or medicine cannot be taken as evidence that a model reasons soundly.","feed_headline":"No correlation found between legal skill and general reasoning in LLMs","feed_subtitle":"Bar-exam scores do not predict rational reasoning across six language models, the study reports.","key_machinery":"The load-bearing object is the per-model Pearson correlation coefficient between a narrow-domain score and a broad-reasoning score. The narrow score comes from 1,000 simulated Multistate Bar Examination questions across seven legal subjects; the broad score comes from 240 items spanning the Wason Selection Task, the Conjunction Fallacy Task, and two base-rate-neglect tasks. This single statistic carries the entire argument: the conclusion that legal expertise and general reasoning are unrelated rests on these six correlation coefficients and their p-values.","core_discovery":"The paper's central claim is that narrow and broad cognitive abilities are disconnected in the intelligence structure of LLMs. For each of six models, the authors computed a Pearson correlation between average legal-reasoning accuracy (bar-exam questions across seven legal subjects) and average rationality accuracy (Wason Selection Task, Conjunction Fallacy Task, and two base-rate-neglect tasks). All six correlations were weak and statistically insignificant, with coefficients from -0.11 to 0.29 and p-values above 0.05. The authors read this as evidence that LLM performance is fragmented: expertise in one domain reflects pattern-based mimicry rather than an integrated reasoning capacity that generalizes across tasks.","pith_inferences":["Beyond the paper: correlating only six model-level averages makes a null result almost unavoidable; a stronger test would correlate many task-level scores within each model (legal subdomains versus individual rationality tasks) to see whether any shared reasoning factor appears.","Beyond the paper: if this disconnection holds, specialized leaderboards in law, medicine, or finance should not be treated as evidence of safe autonomous decision-making; deployment audits should add consistency and bias probes alongside accuracy.","Beyond the paper: the reported test-retest inconsistency can be quantified by repeatedly prompting the same model with temperature fixed and measuring answer entropy; the paper's qualitative finding predicts substantial entropy even on single-answer tasks."],"forward_implications":["High scores on specialized benchmarks such as the simulated MBE cannot be used as evidence of general reasoning competence.","Increasing model size improves average performance in both legal and rationality tasks, but does not create a meaningful correlation between the two, so scaling alone does not yield transferable reasoning.","Fine-tuning or optimizing LLMs for a narrow domain is unlikely to improve broader rational decision-making.","Reliability is a separate axis: models can answer the same prompt differently across repetitions, so accuracy and consistency must be evaluated separately.","Auditing frameworks for high-stakes LLM deployment should include reasoning-stability and cognitive-bias checks, not just benchmark accuracy."],"supporting_citations":[{"why":"Supplies the 1,000 simulated Multistate Bar Examination questions used as the narrow legal-reasoning benchmark.","marker":"[8]"},{"why":"Defines the Wason Selection Task used among the broad rationality measures.","marker":"[65]"},{"why":"Defines the Conjunction Fallacy Task used among the broad rationality measures.","marker":"[64]"},{"why":"Cited as the source of the Stereotype Base Rate Neglect Task in the rationality battery.","marker":"[47]"},{"why":"Cited as sources of the Diagnostic Base Rate Neglect Task in the rationality battery.","marker":"[15, 35]"},{"why":"Defines the Pearson correlation coefficient and underpins the significance testing used for the main null result.","marker":"[10]"}],"fun_headline_variants":["LLM legal expertise does not predict general reasoning","Domain skill and general reasoning uncorrelated in LLMs","LLM abilities fragment: legal skill not tied to reasoning","No link between legal expertise and LLM general reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion depends on the unstated assumption that the Pearson correlations in Table 4 were computed over enough paired observations, with independent items, to detect a real relationship; the paper reports only two aggregate averages per model and p-values, so the sample size and unit of analysis cannot be verified.","fun_headline_variants_meta":{"raw":{"variants":["LLM legal expertise does not predict general reasoning","Domain skill and general reasoning uncorrelated in LLMs","LLM abilities fragment: legal skill not tied to reasoning","No link between legal expertise and LLM general reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001105,"raw_usage":{"total_tokens":4534,"prompt_tokens":798,"completion_tokens":3736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":3672}},"tokens_in":414,"tokens_out":3736,"duration_ms":24134,"temperature":1.0,"reasoning_tokens":3672,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:24:42.685903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to recompute the correlation using the underlying per-domain or per-item scores instead of one average law score and one average rationality score per model; if any model shows a statistically significant positive correlation across legal subdomains and rationality tasks, the disconnection claim collapses. A second check is to rerun identical prompts with temperature fixed: if the same model gives the same answer across repeated trials, the reported test-retest inconsistency is overturned.","supporting_citations":[{"cited_title":"Barbri Simulated MBE","cited_arxiv_id":null,"evidence_quote":"Supplies the 1,000 simulated Multistate Bar Examination questions used as the narrow legal-reasoning benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Wason Selection Task used among the broad rationality measures."},{"cited_title":"Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment","cited_arxiv_id":null,"evidence_quote":"Defines the Conjunction Fallacy Task used among the broad rationality measures."},{"cited_title":"Dual process theory of reasoning: Contemporary issues and developmental applications","cited_arxiv_id":null,"evidence_quote":"Cited as the source of the Stereotype Base Rate Neglect Task in the rationality battery."},{"cited_title":"Pearson correla- tion coe￿cient","cited_arxiv_id":null,"evidence_quote":"Defines the Pearson correlation coefficient and underpins the significance testing used for the main null result."}],"review_version":1}