REVIEW 4 cited by
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present the ARC-DA dataset, a direct-answer ("open response", "freeform") version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset. While ARC has been influential in the community, its multiple-choice format is unrepresentative of real-world questions, and multiple choice formats can be particularly susceptible to artifacts. The ARC-DA dataset addresses these concerns by converting questions to direct-answer format using a combination of crowdsourcing and expert review. The resulting dataset contains 2985 questions with a total of 8436 valid answers (questions typically have more than one valid answer). ARC-DA is one of the first DA datasets of natural questions that often require reasoning, and where appropriate question decompositions are not evident from the questions themselves. We describe the conversion approach taken, appropriate evaluation metrics, and several strong models. Although high, the best scores (81% GENIE, 61.4% F1, 63.2% ROUGE-L) still leave considerable room for improvement. In addition, the dataset provides a natural setting for new research on explanation, as many questions require reasoning to construct answers. We hope the dataset spurs further advances in complex question-answering by the community. ARC-DA is available at https://allenai.org/data/arc-da
Forward citations
Cited by 4 Pith papers
-
SymStep: Symbolic Step Verification for Logical Reasoning
SymStep couples atomic LLM deductions to a deterministic constraint propagator with MRV hints, reaching ~97–100% on constraint-dense logic puzzles where CoT scores 0%.
-
Toward Preference-aligned Large Language Models via Residual-based Model Steering
Preference signals in LLM residual streams can be distilled into inference-time steering vectors that improve math and code benchmarks using only 100 preference pairs.
-
Beyond Manually Designed Pruning Policies with Second-Level Performance Prediction: A Pruning Framework for LLMs
A predictor-based agent generates LLM pruning policies in seconds and reports large perplexity reductions on Llama2-7B and Llama3-8B.
-
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
Checkpoint merging during constant-LR training can replace LR decay and yields improved LLM benchmark scores over Warmup-Stable-Decay.
Discussion (0). Continue with ORCID to comment.