REVIEW 26 cited by
LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Machine reading is a fundamental task for testing the capability of natural language understanding, which is closely related to human cognition in many aspects. With the rising of deep learning techniques, algorithmic models rival human performances on simple QA, and thus increasingly challenging machine reading datasets have been proposed. Though various challenges such as evidence integration and commonsense knowledge have been integrated, one of the fundamental capabilities in human reading, namely logical reasoning, is not fully investigated. We build a comprehensive dataset, named LogiQA, which is sourced from expert-written questions for testing human Logical reasoning. It consists of 8,678 QA instances, covering multiple types of deductive reasoning. Results show that state-of-the-art neural models perform by far worse than human ceiling. Our dataset can also serve as a benchmark for reinvestigating logical AI under the deep learning NLP setting. The dataset is freely available at https://github.com/lgw863/LogiQA-dataset
Forward citations
Cited by 26 Pith papers
-
Efficient Decentralized Multi-task Dataset Valuation via Model Merging
Task-arithmetic model merging approximates multi-task coalition utilities well enough to recover Dataset Shapley rankings privately and without retraining.
-
R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation
A 3,068-prompt benchmark with per-instance Q&A scoring shows that current text-to-image models, including reasoning-enhanced ones, handle reasoning-driven prompts poorly, with mathematical reasoning near zero.
-
Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards
Non-vacuous PAC-Bayes generalization bounds for billion-parameter RLVR models, obtained by a Gumbel-max reparameterization and aggressive TinyLoRA distillation/quantization, are claimed for four tasks.
-
PACE: A Proxy for Agentic Capability Evaluation
PACE builds proxy benchmarks from non-agentic instances via relevance and global selection plus regression to predict agentic scores with MAE under 4%, Spearman correlation above 0.80, and 85% ranking accuracy at unde...
-
What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs
VLM accuracy can be predicted from a scalar capability score derived from LLM text benchmarks plus multimodal data volume via a fitted transfer-absorption scaling law.
-
Reasoning or Fluency? Dissecting Probabilistic Confidence in Best-of-N Selection
Selection by log-likelihood, entropy, or self-certainty chooses correct reasoning traces almost as well after inter-step dependencies are removed, implying the scores track fluency and priors, not reasoning.
-
DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone
A masked diffusion language model built on a bidirectional Mamba backbone matches Transformer-based denoisers on quality while decoding with near-linear time scaling.
-
BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
A bilevel optimization method ranks pretraining data by training a small proxy model on weighted samples, yielding modest downstream-task gains without external pretrained models.
-
Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
DRER rewards CoT trajectories that increase the model's likelihood of the correct answer, plus a length penalty, and the new LogicTree benchmark reportedly lifts a 7B model's average accuracy from 0.13 to 0.60.
-
Language Models Improve When Pretraining Data Matches Target Tasks
Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.
-
GeLaCo: An Evolutionary Approach to Layer Compression
Evolutionary search over layer-merging configurations, scored by module-wise activation similarity, yields competitive LLM compression and the first size-quality Pareto fronts.
-
Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?
Long chain-of-thought and RL training on math problems improves general reasoning benchmarks, while short chain-of-thought math fine-tuning often degrades performance.
-
Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
Chameleon uses kernel ridge leverage scores on domain embeddings to set LLM training-mixture weights, matching DoGE-level pretraining quality at roughly one fifth the compute and improving finetuning perplexity.
-
Advantageous Parameter Expansion Training Makes Better Large Language Models
APEX expands high-activation parameters into low-activation ones during LLM training and reports gains, but the continued pre-training evidence is confounded by different learning rate schedules.
-
VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RL
LLM-derived rules, genetic expansion, and program-drawn images produce 660K visual logic puzzles; GRPO training on them lifts a VLM's accuracy on those puzzles and modestly on MathVista.
-
UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions
A new bilingual benchmark ties unanswerable questions to Wikidata facts and shows that LLMs often store the relevant knowledge yet fail to use it to recognize unanswerability.
-
GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining
GRAPE uses a minimax group-DRO scheme to reweight both source domains and target tasks during pretraining, improving multi-task reasoning and low-resource language modeling.
-
Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
A pretrained parametric memory module, taught to copy nearest-neighbor retrieval for next-token prediction, lets small frozen LMs match or beat much larger LMs at the same total parameter budget.
-
DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent
A synthetic benchmark of 9,000 multi-hop web-research questions with difficulty tiers and teacher-generated search trajectories, plus an open-source RL training framework that reportedly lets 3B-parameter agents beat ...
-
Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency
Setting the feature dimension of each linear-attention layer proportional to the estimated degrees of freedom of its input kernel improves distilled model accuracy without increasing total inference cost.
-
GTA: Grouped-head latenT Attention
GTA reuses attention scores across grouped heads and stores compressed latent values, roughly matching GQA-level quality in sub-1B models while reducing cache and compute.
-
LLM Data Selection and Utilization via Dynamic Bi-level Optimization
A bi-level-learned per-batch reweighting model improves average downstream accuracy of LLMs pre-trained on selected data by about one to two points, but without error bars or code the effect size is uncertain.
-
ReasonBridge: Efficient Reasoning Transfer from Closed to Open-Source Language Models
ReasonBridge distills 1,000 curated reasoning traces from a closed model into open-source models via hierarchical adapters and guided inference, claiming up to 23% benchmark gains.
-
Reason from Future: Reverse Thought Chain Enhances LLM Reasoning
A prompting method that alternates backward and forward reasoning improves small LLM accuracy on math and search tasks and reduces the number of visited search states.
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
-
Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models
A 4B-parameter model is claimed to explain its own reasoning through inverse attention analysis, but the paper offers no consistent evidence or artifacts.
Discussion (0). Sign in to comment.