SciFaultyQA: Benchmarking LLMs on Faulty Science Question Detection with a GAN-Inspired Approach to Synthetic Dataset Generation

Debarshi Kundu

REVIEW 4 major objections 6 minor 11 references

SciFaultyQA: Benchmarking LLMs on Faulty Science Question Detection with a GAN-Inspired Approach to Synthetic Dataset Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new benchmark finds that current LLMs answer nonsensical science questions instead of flagging them, with GPT-4o detecting only 16%.

desk verdict A promising but unvalidated benchmark: the GAN-style generation idea is worth a serious look, but the missing label-validation details undercut every headline detection rate. read the letter →

arxiv 2412.11988 v1 pith:FSJFSWCV submitted 2024-12-16 cs.CL cs.LG

Debarshi Kundu This is my paper · ORCID

classification cs.CLcs.LG

keywords faultyquestiondetectionLLMbenchmarkscienceansweringsyntheticdatasetgenerationGAN-inspiredmulti-agentsystemswebsearchintegrationqualityassessment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current large language models routinely fail to notice when a science question is logically or physically impossible, answering it anyway with an invalid number. To make this failure measurable, it builds SciFaultyQA, a 1,333-question benchmark of deliberately faulty questions, and reports that GPT-4o detects only 16% of them, Llama 3.1 12%, and Gemini Flash 6%. It also proposes a GAN-inspired pipeline in which one set of LLMs generates faulty questions and another tries to catch them, producing synthetic data at scale without hand-labeling every item. The reason to care is practical: if a model cannot tell a bad question from a good one, it will silently give wrong answers in tutoring, search, and scientific Q&A settings, wasting computation and misleading users.

What carries the argument

The load-bearing mechanism is a GAN-inspired generation loop. Several LLM 'generators' each take a valid science question from SciQ or SciQA and rewrite it into a faulty version, annotating the fault type and reason; an LLM 'discriminator' then examines the rewritten question without seeing the reason and either explains why it is faulty or answers it. The discriminator's responses are fed back to the generators to refine the next round, and the loop stops when the discriminator can no longer find faults. This loop is what makes the dataset scalable and is supposed to keep the faultiness at the edge of what current LLMs can detect, since questions one LLM can create are checked by another LLM rather than by the same model that wrote them.

What would settle it

Take a random sample of 100 SciFaultyQA questions and have independent science educators classify each as valid, invalid, or ambiguous using a written rubric. If the educators reject a large share of the 'faulty' labels, or if GPT-4o's detection rate changes materially after relabeling, the reported 6-16% baseline does not measure what the paper claims.

Watch

Extended reading notes

Core claim

The central discovery is a measurable inability: on the SciFaultyQA benchmark, the best tested model detects only 16% of intentionally faulty science questions, and the worst detects 6%. The paper shows that the failure is not fixed by reasoning strength alone; GPT-4o still answers '0.5' to the one-woman-three-men reproduction question in 8 of 10 trials. Tool access changes the picture: adding web search raises GPT-4o's detection rate from 16% to 65%, while a multi-model agent reaches 30%. The paper also reports temporal inconsistency in a single model's behavior, where a correct rejection of a faulty question makes a later rejection more likely but not reliable.

Load-bearing premise

The benchmark labels themselves are correct, because the paper's validation step (an AI agent with web search plus human evaluation) is described without reporting sample size, agreement, or a definition of 'faulty'; if many generated questions are actually valid, every detection rate loses its meaning.

Editorial extensions

If this is right

If the reported rates generalize, standard chatbots will continue to produce invalid answers to impossible science questions, so deployment in education or scientific assistance should include refusal or verification behavior.
Web-search integration is a concrete, immediately available mitigation: the paper's numbers suggest external information restores most of the missing fault detection without retraining.
Multi-agent verification gives only a partial improvement over a single model, implying that collaboration alone is not enough and architecture or training changes are also needed.
GAN-style adversarial generation offers a route to build new benchmarks automatically as models improve, addressing the 'running out of benchmarks' problem the paper raises.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension not tested in the paper: prompting a model to explicitly classify a question as valid, invalid, or unanswerable before solving would likely shift detection rates, and could be measured on the same 1,333 questions.
The adversarial loop could be ported to other domains where false premises are common, such as legal hypotheticals or medical self-diagnosis questions, where silently answering a nonsense input is similarly dangerous.
If the benchmark labels were re-audited by independent human raters with an explicit rubric, the headline detection rates could move substantially; the paper's own validation step is stated but not quantified.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, the core idea is genuinely worth attention: using two LLMs in a GAN-style loop—one generating faulty science questions, one judging them—to build a benchmark is a new framing for a real problem, and the reported failure mode (GPT-4o detects only 16% of faulty questions, Gemini Flash 6%) matches what many of us have seen anecdotally. Second, those numbers are not yet trustworthy. The paper's validation step is described in two sentences: check with an AI agent with web search, then human evaluation. No sample size, no annotator count, no agreement statistic, no rubric. The stress-test note is right: if even a fraction of the 1,333 items are not actually faulty, every detection rate in Tables 1 and 2 loses meaning.

What the paper does well: it identifies a real reliability gap, proposes a scalable generation method, and reports a plausible error-reduction result (web search raising GPT-4o from 16% to 65%). The idea of iterating generators against a discriminator until the discriminator can no longer find faults is clever and could be useful beyond this specific benchmark. Credit is also due for being transparent that this is work in progress—the diffusion-inspired part is labeled WIP, and open questions are honestly flagged.

Where it is soft: beyond the validation gap, there is no protocol detail on how detection was scored (exact match? rubric? human review of model responses?), no error bars or random baselines, and the improvement experiment picks the best model (GPT-4o) post hoc from the baseline table. The circularity concern is real but not automatically fatal: using an LLM discriminator to define faultiness could enrich the dataset for faults that particular LLM misses, which would overstate the failure rate, or it could produce trivially weird questions that understate it. Either way, without quantified human checks the labels are unverified.

Who this is for: anyone working on LLM robustness, evaluation methodology, or synthetic benchmark generation will find the idea worth discussing. As a benchmark claim, it is not yet solid enough to build on.

Recommendation: send it to serious peer review—the idea and the failure mode deserve referee time and the authors need pressure to release the dataset, code, and validation statistics. If they do, this could become a useful resource. If they don't, it stays a promising sketch. I would engage with it, but I would not cite the detection rates yet.

Referee Report

4 major / 6 minor

Summary. The paper introduces SciFaultyQA, a dataset of 1,333 intentionally faulty science questions, constructed by a GAN-inspired loop in which LLM generators create faulty variants of questions from SciQ/SciQA and an LLM discriminator evaluates them. The authors report that three LLMs detect faults in only 6-16% of items (Table 1), and that adding web search to GPT-4o raises the detection rate to 65% (Table 2). They also propose multi-agent verification, tool integration, and training-time exposure to faulty questions as error-reduction strategies.

Significance. If the SciFaultyQA labels are reliable, the reported detection rates constitute a concrete, falsifiable demonstration that current LLMs often fail to recognize nonsensical science questions, and the GAN-inspired generation method could be a scalable way to construct adversarial benchmarks. The paper also offers an interesting negative result with external tool access. However, the significance is entirely conditional on label validity and on a clearly specified evaluation protocol, neither of which is currently established. The strengths are the specificity of the headline numbers and the potentially reproducible dataset link; the weakness is that the experimental reporting is too incomplete to support those numbers as benchmark results.

major comments (4)

[Section 3, Tables 1 and 2] The detection rate is never defined. The paper does not state what prompt was used to elicit detection, whether the model was asked to classify the question as faulty or to answer it, how responses were parsed into correct/incorrect detections, how many trials per question were run, or the temperature/sampling settings. Without this protocol, the numbers 6%, 12%, 16%, and 65% are not reproducible. Moreover, because every item in SciFaultyQA is faulty by construction, a model that always classifies items as faulty would score 100% under a forced-choice protocol, while a model that always answers would score 0%; the absence of such baselines makes the low rates hard to interpret.
[Section 2, validation of SciFaultyQA labels] The paper's own question about false positives -- 'What if the generated results are not actually faulty but LLM says it is faulty?' -- is followed only by a two-step validation description ('Check with an AI agent which has access to web search 2. Human evaluation') with no sample size, number of annotators, inter-annotator agreement, rubric, or rejection rate. Since every detection rate in Tables 1 and 2 is computed against these labels, the central claim is uninterpretable until label reliability is quantified. The authors should report how many items were human-checked, how often the AI agent and humans agreed with the LLM discriminator, and how borderline or mislabeled items were resolved.
[Section 2, GAN-inspired generation, step 5] The iterative generation loop stops when the LLM discriminator 'can no longer find faults' in the generated questions. This creates a circularity risk: the same class of LLM that is later evaluated (GPT-4o is used both as a generator/discriminator in the pipeline and as an evaluated model in Table 1) is used to define what counts as 'hard' faulty. The dataset may be enriched for faults that GPT-4-class discriminators miss, exactly the models being tested. The paper should report the number of iterations, the distribution of fault types, the discriminator's agreement with human labels, and the overlap between the generator/discriminator model family and the evaluated models.
[Section 1.2, Key Contributions] The contribution list includes 'Diffusion-inspired dataset generation: WIP' and 'we listed few type of faults : TBD', yet the abstract and conclusion state that a novel method for generating synthetic datasets was developed. The diffusion-inspired method is not implemented or evaluated, and the taxonomy of fault types is left as future work. These are proposals, not contributions, and the manuscript should clearly separate completed results from intended extensions, otherwise the readers cannot tell which claims are meant to be load-bearing.

minor comments (6)

[Abstract] The claim that models answer nonsensically '8 out of 10 trials' is anecdotal; no trial count, temperature, or protocol is given for this observation.
[Section 1, Introduction] There are numerous typos and incomplete sentences, e.g., 'cmputation power waste', 'nonsesical questions', 'itslef wrong', 'standered has been the golden standered', and the clause 'This also shows that different LLMs are good in different fields. Therefore, if one LLM can' trails off without a main clause.
[Section 2, Methodology] The sentence 'In the 1 there are three LLM Gen...' presumably refers to Figure 1, but the figure is not referenced in the text and the caption does not explain the roles of LLM_Gen_a/b/c and LLM_Dis. Please add a proper cross-reference and a full caption.
[Section 3, Table 2] The column header says 'Accuracy (%)' while Table 1 says 'Detection Rate (%)'; the paper should use one consistent term, since 'accuracy' implies a known ground truth and a clear binary outcome, which is exactly what needs to be defined.
[References] Several references are incomplete or inconsistently formatted (e.g., 'Jones' lacks a full citation, 'Lyu et al., a' and 'b' are cited without years in the text, and some arXiv IDs are split across lines).
[Appendix] The GitHub repository link is a start, but the paper does not state whether the dataset, the generation prompts, or the evaluation scripts are available there, nor which license applies.

Circularity Check

1 steps flagged · score 4.0 of 10

Benchmark labels are produced and filtered by LLMs; the only external check is an unquantified human evaluation, so the detection rates are measured against an LLM-defined ground truth.

self definitional [Section 2, GAN-inspired dataset generation, steps 3-5 and validation paragraph; Tables 1-2]
"This iterative process continues until the LLM discriminator can no longer find faults in any of the generated faulty versions of the original question, or until a predefined maximum number of iterations is reached. ... After the faulty dataset generated by this method, What if the generated results are not actually faulty but LLM says it is faulty? We need to analyze False Positive cases. For this, we took two step process 1. Check with an AI agent which has access to web search 2. Human evaluation"

Faultiness is operationalized by LLM generators (step 3: 'Each generator also provides a reason why the generated question is faulty') and the LLM discriminator's inability to find faults is the stopping criterion. The final dataset's labels are therefore whatever LLMs assert to be faulty. Detection rates in Tables 1-2 count how often evaluated LLMs (GPT-4o, Gemini Flash, Llama 3.1) agree with that LLM-defined ground truth. The only external check is an 'AI agent which has access to web search' — itself an LLM-based agent — plus an unquantified 'Human evaluation'. With no sample size, annotator count, or agreement statistics reported, the human step cannot be shown to break the loop.

full rationale

The paper contains no formal derivation chain of equations, so the ordinary circularity patterns (fitted input called prediction, uniqueness imported from authors, ansatz smuggled via citation) do not apply. However, the central empirical claim is a benchmark result whose ground-truth labels are generated by LLMs and filtered by an LLM discriminator. The paper's own two-step validation is described in a single sentence and provides no sample size, no agreement statistics, and no rubric, so the claimed human evaluation cannot be independently assessed. Because the evaluated models overlap with the model families used as generators, the detection rates partly measure inter-LLM agreement rather than detection of objectively faulty science questions. This is a definitional/self-referential circularity in the benchmark construct. If the human validation had been quantified and shown to correct the LLM-produced labels, the circularity would be broken, but the manuscript supplies no such evidence. No load-bearing self-citations or imported uniqueness theorems are present. The concern is real but moderate, hence a score of 4 rather than higher.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numeric free parameters are fitted to data; the central numbers are raw detection-rate percentages. Design choices such as the maximum generator iterations, the choice of generators, and the selection of GPT-4o for the improvement study are arbitrary but not fitted values.

assumptions (3)

domain assumption Ground truth labels in SciFaultyQA are correct.
Section 2 states validation via an AI agent with web search and human evaluation, but gives no statistics or protocol details; the benchmark's correctness is assumed.
domain assumption The GAN-style loop converges to a harder question distribution.
Section 2, steps 4-5: iterative generation continues until the discriminator cannot find faults; there is no analysis that this process actually increases difficulty.
domain assumption LLM-generated fault labels align with human judgments of faultiness.
The generation method in Section 2 relies on LLM generators and an LLM discriminator to define what counts as faulty; without human eval numbers the alignment is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SciFaultyQA: Benchmarking LLMs on Faulty Science Question Detection with a GAN-Inspired Approach to Synthetic Dataset Generation." pith.science (2026). https://pith.science/paper/FSJFSWCV

@misc{pith2026241211988,
  author       = {Pith},
  title        = {Pith review of: SciFaultyQA: Benchmarking LLMs on Faulty Science Question Detection with a GAN-Inspired Approach to Synthetic Dataset Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FSJFSWCV}},
  note         = {Machine review of arXiv:2412.11988}
}

read the original abstract

Consider the problem: ``If one man and one woman can produce one child in one year, how many children will be produced by one woman and three men in 0.5 years?" Current large language models (LLMs) such as GPT-4o, GPT-o1-preview, and Gemini Flash frequently answer "0.5," which does not make sense. While these models sometimes acknowledge the unrealistic nature of the question, in many cases (8 out of 10 trials), they provide the nonsensical answer of "0.5 child." Additionally, temporal variation has been observed: if an LLM answers correctly once (by recognizing the faulty nature of the question), subsequent responses are more likely to also reflect this understanding. However, this is inconsistent. These types of questions have motivated us to develop a dataset of science questions, SciFaultyQA, where the questions themselves are intentionally faulty. We observed that LLMs often proceed to answer these flawed questions without recognizing their inherent issues, producing results that are logically or scientifically invalid. By analyzing such patterns, we developed a novel method for generating synthetic datasets to evaluate and benchmark the performance of various LLMs in identifying these flawed questions. We have also developed novel approaches to reduce the errors.

Figures

Figures reproduced from arXiv: 2412.11988 by the authors.

**Figure 1.** GAN inspired synthetic data generation flow = Gemini Pro, LLM generator3 = Llama 3.1, LLM generator4 = Mixtral), to generate faulty versions of the original questions. Each generator also provides a reason why the generated question is faulty and identifies the type of fault, such as logical fallacies, unrealistic scenarios, or violations of physical laws. 4. Feed the faulty questions generated in step 3 to another… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 3 canonical work pages

[1]

Auer, S., Barone, D. A. C., Bartz, C., Cortes, E. G., Jaradeh, M. Y., Karras, O., Koubarakis, M., Mouromtsev, D., Pliukhin, D., Radyush, D., Shilin, I., Stocker, M., and Tsalapati, E. The SciQA scientific question answering benchmark for scholarly knowledge. 13 0 (1): 0 7240. ISSN 2045-2322. doi:10.1038/s41598-023-33607-z. URL https://www.nature.com/artic...

doi:10.1038/s41598-023-33607-z
[2]

Think you have solved question answering? try ARC , the AI 2 reasoning challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try ARC , the AI 2 reasoning challenge. URL http://arxiv.org/abs/1803.05457

arXiv
[3]

AI now beats humans at basic tasks — new benchmarks are needed, says major report

Jones, N. AI now beats humans at basic tasks — new benchmarks are needed, says major report. 628 0 (8009): 0 700--701. doi:10.1038/d41586-024-01087-4. URL https://www.nature.com/articles/d41586-024-01087-4. Bandiera\_abtest: a Cg\_type: News Publisher: Nature Publishing Group Subject\_term: Machine learning, Computer science, Industry

work page doi:10.1038/d41586-024-01087-4
[4]

Learn to explain: Multimodal reasoning via thought chains for science question answering

Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022

2022
[5]

P., Pham, H., Hovy, E., Póczos, B., Salakhutdinov, R., and Morency, L.-P

Lyu, Y., Liang, P. P., Pham, H., Hovy, E., Póczos, B., Salakhutdinov, R., and Morency, L.-P. StylePTB : A compositional benchmark for fine-grained controllable text style transfer, a . URL http://arxiv.org/abs/2104.05196

arXiv
[6]

C., and Lee, H

Lyu, Y., Luo, T., Shi, J., Hollon, T. C., and Lee, H. Fine-grained text style transfer with diffusion-based language models, b . URL http://arxiv.org/abs/2305.19512

arXiv
[7]

C., Shoham, Y., Wald, R., and Clark, J

Maslej, N., Fattorini, L., Perrault, R., Parli, V., Reuel, A., Brynjolfsson, E., Etchemendy, J., Ligett, K., Lyons, T., Manyika, J., Niebles, J. C., Shoham, Y., Wald, R., and Clark, J. The ai index 2024 annual report. Technical report, AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA, April 2024. Available online

work page 2024
[8]

Learning to reason with LLMs

OpenAI . Learning to reason with LLMs . URL https://openai.com/index/learning-to-reason-with-llms/

work page

Show all 11 references

[9]

L., Stickland, A

Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA : A graduate-level google-proof q&a benchmark. URL http://arxiv.org/abs/2311.12022

arXiv
[10]

F., and Gardner, M

Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. URL http://arxiv.org/abs/1707.06209

arXiv
[11]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Auer, S., Barone, D. A. C., Bartz, C., Cortes, E. G., Jaradeh, M. Y., Karras, O., Koubarakis, M., Mouromtsev, D., Pliukhin, D., Radyush, D., Shilin, I., Stocker, M., and Tsalapati, E. The SciQA scientific question answering benchmark for scholarly knowledge. 13 0 (1): 0 7240. ISSN 2045-2322. doi:10.1038/s41598-023-33607-z. URL https://www.nature.com/artic...

doi:10.1038/s41598-023-33607-z

[2] [2]

Think you have solved question answering? try ARC , the AI 2 reasoning challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try ARC , the AI 2 reasoning challenge. URL http://arxiv.org/abs/1803.05457

arXiv

[3] [3]

AI now beats humans at basic tasks — new benchmarks are needed, says major report

Jones, N. AI now beats humans at basic tasks — new benchmarks are needed, says major report. 628 0 (8009): 0 700--701. doi:10.1038/d41586-024-01087-4. URL https://www.nature.com/articles/d41586-024-01087-4. Bandiera\_abtest: a Cg\_type: News Publisher: Nature Publishing Group Subject\_term: Machine learning, Computer science, Industry

work page doi:10.1038/d41586-024-01087-4

[4] [4]

Learn to explain: Multimodal reasoning via thought chains for science question answering

Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022

2022

[5] [5]

P., Pham, H., Hovy, E., Póczos, B., Salakhutdinov, R., and Morency, L.-P

Lyu, Y., Liang, P. P., Pham, H., Hovy, E., Póczos, B., Salakhutdinov, R., and Morency, L.-P. StylePTB : A compositional benchmark for fine-grained controllable text style transfer, a . URL http://arxiv.org/abs/2104.05196

arXiv

[6] [6]

C., and Lee, H

Lyu, Y., Luo, T., Shi, J., Hollon, T. C., and Lee, H. Fine-grained text style transfer with diffusion-based language models, b . URL http://arxiv.org/abs/2305.19512

arXiv

[7] [7]

C., Shoham, Y., Wald, R., and Clark, J

Maslej, N., Fattorini, L., Perrault, R., Parli, V., Reuel, A., Brynjolfsson, E., Etchemendy, J., Ligett, K., Lyons, T., Manyika, J., Niebles, J. C., Shoham, Y., Wald, R., and Clark, J. The ai index 2024 annual report. Technical report, AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA, April 2024. Available online

work page 2024

[8] [8]

Learning to reason with LLMs

OpenAI . Learning to reason with LLMs . URL https://openai.com/index/learning-to-reason-with-llms/

work page