CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

Frank Binder; J\"org Frohberg

CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2112.11941 v3 pith:3KFMXVZH submitted 2021-12-22 cs.CL

Jörg Frohberg , Frank Binder This is my paper

classification cs.CL

keywords benchmarkmodelscounterfactualdatacrasslanguagelargenovel

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

We introduce the CRASS (counterfactual reasoning assessment) data set and benchmark utilizing questionized counterfactual conditionals as a novel and powerful tool to evaluate large language models. We present the data set design and benchmark that supports scoring against a crowd-validated human baseline. We test six state-of-the-art models against our benchmark. Our results show that it poses a valid challenge for these models and opens up considerable room for their improvement.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models
cs.CR 2025-09 reject novelty 3.0 of 10

DP-FedLoRA clips and adds Gaussian noise to per-client LoRA matrices in federated LLM fine-tuning, claiming unbiased updates and bounded variance, but the privacy calibration and experiments have significant gaps.

Pith tools