{"id":"9c3c7478-c85f-4883-8228-0bb700eafb05","arxiv_id":"2412.11988","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"SciFaultyQA is a 1,333-question benchmark of intentionally faulty science questions on which GPT-4o detects only 16% of faults, improved to 65% with web search integration.","lead":"Large language models often answer intentionally impossible science questions as if they were valid, and the new SciFaultyQA benchmark measures how often models spot such faulty questions. A GAN-style loop generates the questions with one set of LLMs and evaluates them with another, with web search integration raising detection from 16% to 65% in the paper's tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated benchmark labels are the load-bearing risk: if many SciFaultyQA items are not actually faulty, every detection rate in Tables 1 and 2 is uninterpretable.","rationale":"The paper's central claim is a set of detection-rate numbers on a new benchmark. For those numbers to be meaningful, two things must hold: the dataset items must genuinely be faulty, and the detection protocol must be clearly specified and fairly applied. The reader identified the label-correctness issue as the weakest assumption; I agree. The paper's Section 2 validation description is too thin to establish label reliability, and the adversarial generation loop makes label noise especially plausible. The web-search improvement (Table 2) also depends on the same labels. Because the concern is empirically addressable with released data and a human-annotation study, the appropriate verdict remains CONDITIONAL — the paper's contribution is potentially useful, but it is not yet validated. I am not moving to REJECT because the concern is about missing evidence rather than demonstrated error. I also note the paper itself flags the false-positive question ('What if the generated results are not actually faulty?'), but does not report the outcome of its two-step check. My stress-test adds a concrete, feasible test that would settle the issue.","tokens_in":4699,"tokens_out":2950,"duration_ms":28276,"concrete_test":"Release SciFaultyQA with a fixed commit hash, then independently sample 200 items and have three annotators label each question as 'faulty', 'not faulty', or 'unclear' using a written rubric. Compute Fleiss' kappa and the proportion of items unanimously judged 'faulty'. Recompute GPT-4o's detection rate and the web-search rate restricted to the unanimous-faulty subset. If the restricted rates differ materially from 16% and 65%, or if inter-annotator agreement is below about 0.6, the benchmark labels are not reliable enough to support the paper's claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — GPT-4o detects 16% of faulty questions, Gemini Flash 6%, Llama 3.1 12%, and web search raises GPT-4o to 65% — depends entirely on the correctness of the SciFaultyQA labels. The paper's own validation procedure (Section 2) is described in two sentences: 'Check with an AI agent which has access to web search 2. Human evaluation.' No sample size, no number of annotators, no agreement statistics, no rubric, and no rejection rate are reported. This is especially concerning because the generation process is adversarial: the pipeline iterates until the LLM discriminator 'can no longer find faults' in the generated questions. The resulting dataset is therefore enriched for questions that a particular LLM judged non-faulty, many of which may in fact be valid or ambiguous. Without human-validated labels with quantified reliability, a model that correctly answers valid questions would be scored as failing to detect a fault, inflating the apparent failure rate. Conversely, if items are trivially faulty, the 16% may underestimate true ability. Either way, the headline numbers lose their meaning. The reader's weakest assumption — that the labels are correct — is exactly the right point, and the paper provides no evidence to support it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SciFaultyQA, a dataset of 1,333 intentionally faulty science questions, constructed by a GAN-inspired loop in which LLM generators create faulty variants of questions from SciQ/SciQA and an LLM discriminator evaluates them. The authors report that three LLMs detect faults in only 6-16% of items (Table 1), and that adding web search to GPT-4o raises the detection rate to 65% (Table 2). They also propose multi-agent verification, tool integration, and training-time exposure to faulty questions as error-reduction strategies.","tokens_in":5061,"tokens_out":3655,"duration_ms":33986,"significance":"If the SciFaultyQA labels are reliable, the reported detection rates constitute a concrete, falsifiable demonstration that current LLMs often fail to recognize nonsensical science questions, and the GAN-inspired generation method could be a scalable way to construct adversarial benchmarks. The paper also offers an interesting negative result with external tool access. However, the significance is entirely conditional on label validity and on a clearly specified evaluation protocol, neither of which is currently established. The strengths are the specificity of the headline numbers and the potentially reproducible dataset link; the weakness is that the experimental reporting is too incomplete to support those numbers as benchmark results.","major_comments":[{"comment":"The detection rate is never defined. The paper does not state what prompt was used to elicit detection, whether the model was asked to classify the question as faulty or to answer it, how responses were parsed into correct/incorrect detections, how many trials per question were run, or the temperature/sampling settings. Without this protocol, the numbers 6%, 12%, 16%, and 65% are not reproducible. Moreover, because every item in SciFaultyQA is faulty by construction, a model that always classifies items as faulty would score 100% under a forced-choice protocol, while a model that always answers would score 0%; the absence of such baselines makes the low rates hard to interpret.","section":"Section 3, Tables 1 and 2"},{"comment":"The paper's own question about false positives -- 'What if the generated results are not actually faulty but LLM says it is faulty?' -- is followed only by a two-step validation description ('Check with an AI agent which has access to web search 2. Human evaluation') with no sample size, number of annotators, inter-annotator agreement, rubric, or rejection rate. Since every detection rate in Tables 1 and 2 is computed against these labels, the central claim is uninterpretable until label reliability is quantified. The authors should report how many items were human-checked, how often the AI agent and humans agreed with the LLM discriminator, and how borderline or mislabeled items were resolved.","section":"Section 2, validation of SciFaultyQA labels"},{"comment":"The iterative generation loop stops when the LLM discriminator 'can no longer find faults' in the generated questions. This creates a circularity risk: the same class of LLM that is later evaluated (GPT-4o is used both as a generator/discriminator in the pipeline and as an evaluated model in Table 1) is used to define what counts as 'hard' faulty. The dataset may be enriched for faults that GPT-4-class discriminators miss, exactly the models being tested. The paper should report the number of iterations, the distribution of fault types, the discriminator's agreement with human labels, and the overlap between the generator/discriminator model family and the evaluated models.","section":"Section 2, GAN-inspired generation, step 5"},{"comment":"The contribution list includes 'Diffusion-inspired dataset generation: WIP' and 'we listed few type of faults : TBD', yet the abstract and conclusion state that a novel method for generating synthetic datasets was developed. The diffusion-inspired method is not implemented or evaluated, and the taxonomy of fault types is left as future work. These are proposals, not contributions, and the manuscript should clearly separate completed results from intended extensions, otherwise the readers cannot tell which claims are meant to be load-bearing.","section":"Section 1.2, Key Contributions"}],"minor_comments":[{"comment":"The claim that models answer nonsensically '8 out of 10 trials' is anecdotal; no trial count, temperature, or protocol is given for this observation.","section":"Abstract"},{"comment":"There are numerous typos and incomplete sentences, e.g., 'cmputation power waste', 'nonsesical questions', 'itslef wrong', 'standered has been the golden standered', and the clause 'This also shows that different LLMs are good in different fields. Therefore, if one LLM can' trails off without a main clause.","section":"Section 1, Introduction"},{"comment":"The sentence 'In the 1 there are three LLM Gen...' presumably refers to Figure 1, but the figure is not referenced in the text and the caption does not explain the roles of LLM_Gen_a/b/c and LLM_Dis. Please add a proper cross-reference and a full caption.","section":"Section 2, Methodology"},{"comment":"The column header says 'Accuracy (%)' while Table 1 says 'Detection Rate (%)'; the paper should use one consistent term, since 'accuracy' implies a known ground truth and a clear binary outcome, which is exactly what needs to be defined.","section":"Section 3, Table 2"},{"comment":"Several references are incomplete or inconsistently formatted (e.g., 'Jones' lacks a full citation, 'Lyu et al., a' and 'b' are cited without years in the text, and some arXiv IDs are split across lines).","section":"References"},{"comment":"The GitHub repository link is a start, but the paper does not state whether the dataset, the generation prompts, or the evaluation scripts are available there, nor which license applies.","section":"Appendix"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as an early draft: it contains WIP markers, TBD entries, dangling sentences, and a 'Proceedings of ICML 2025' header that seems inappropriate for an arXiv preprint. The core idea -- an adversarial LLM-based generation loop for faulty-question benchmarks -- is worth pursuing, and the headline numbers would be interesting if they survive rigorous validation. However, the lack of a defined evaluation protocol and quantified human validation is currently a load-bearing gap, not a cosmetic one. I recommend major revision rather than rejection because the gaps are potentially fixable within the scope of a revised paper, but the authors should be expected to add protocol details, baseline comparisons, and label-reliability statistics, not just polish the writing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is genuinely worth attention: using two LLMs in a GAN-style loop—one generating faulty science questions, one judging them—to build a benchmark is a new framing for a real problem, and the reported failure mode (GPT-4o detects only 16% of faulty questions, Gemini Flash 6%) matches what many of us have seen anecdotally. Second, those numbers are not yet trustworthy. The paper's validation step is described in two sentences: check with an AI agent with web search, then human evaluation. No sample size, no annotator count, no agreement statistic, no rubric. The stress-test note is right: if even a fraction of the 1,333 items are not actually faulty, every detection rate in Tables 1 and 2 loses meaning.\n\nWhat the paper does well: it identifies a real reliability gap, proposes a scalable generation method, and reports a plausible error-reduction result (web search raising GPT-4o from 16% to 65%). The idea of iterating generators against a discriminator until the discriminator can no longer find faults is clever and could be useful beyond this specific benchmark. Credit is also due for being transparent that this is work in progress—the diffusion-inspired part is labeled WIP, and open questions are honestly flagged.\n\nWhere it is soft: beyond the validation gap, there is no protocol detail on how detection was scored (exact match? rubric? human review of model responses?), no error bars or random baselines, and the improvement experiment picks the best model (GPT-4o) post hoc from the baseline table. The circularity concern is real but not automatically fatal: using an LLM discriminator to define faultiness could enrich the dataset for faults that particular LLM misses, which would overstate the failure rate, or it could produce trivially weird questions that understate it. Either way, without quantified human checks the labels are unverified.\n\nWho this is for: anyone working on LLM robustness, evaluation methodology, or synthetic benchmark generation will find the idea worth discussing. As a benchmark claim, it is not yet solid enough to build on.\n\nRecommendation: send it to serious peer review—the idea and the failure mode deserve referee time and the authors need pressure to release the dataset, code, and validation statistics. If they do, this could become a useful resource. If they don't, it stays a promising sketch. I would engage with it, but I would not cite the detection rates yet.","headline":"A promising but unvalidated benchmark: the GAN-style generation idea is worth a serious look, but the missing label-validation details undercut every headline detection rate.","tokens_in":5421,"tokens_out":1819,"would_cite":false,"duration_ms":17936,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark finds that current LLMs answer nonsensical science questions instead of flagging them, with GPT-4o detecting only 16%.","keywords":["faulty question detection","LLM benchmark","science question answering","synthetic dataset generation","GAN-inspired generation","multi-agent systems","web search integration","question quality assessment"],"falsifier":"Take a random sample of 100 SciFaultyQA questions and have independent science educators classify each as valid, invalid, or ambiguous using a written rubric. If the educators reject a large share of the 'faulty' labels, or if GPT-4o's detection rate changes materially after relabeling, the reported 6-16% baseline does not measure what the paper claims.","tokens_in":4495,"feed_emoji":"🧪","tokens_out":5189,"duration_ms":42999,"temperature":0.7,"pith_summary":"The paper argues that current large language models routinely fail to notice when a science question is logically or physically impossible, answering it anyway with an invalid number. To make this failure measurable, it builds SciFaultyQA, a 1,333-question benchmark of deliberately faulty questions, and reports that GPT-4o detects only 16% of them, Llama 3.1 12%, and Gemini Flash 6%. It also proposes a GAN-inspired pipeline in which one set of LLMs generates faulty questions and another tries to catch them, producing synthetic data at scale without hand-labeling every item. The reason to care is practical: if a model cannot tell a bad question from a good one, it will silently give wrong answers in tutoring, search, and scientific Q&A settings, wasting computation and misleading users.","feed_headline":"GPT-4o flags just 16% of nonsensical science questions","feed_subtitle":"A 1,333-question benchmark shows LLMs solve impossible science problems instead of rejecting them; web search lifts detection to 65%.","key_machinery":"The load-bearing mechanism is a GAN-inspired generation loop. Several LLM 'generators' each take a valid science question from SciQ or SciQA and rewrite it into a faulty version, annotating the fault type and reason; an LLM 'discriminator' then examines the rewritten question without seeing the reason and either explains why it is faulty or answers it. The discriminator's responses are fed back to the generators to refine the next round, and the loop stops when the discriminator can no longer find faults. This loop is what makes the dataset scalable and is supposed to keep the faultiness at the edge of what current LLMs can detect, since questions one LLM can create are checked by another LLM rather than by the same model that wrote them.","core_discovery":"The central discovery is a measurable inability: on the SciFaultyQA benchmark, the best tested model detects only 16% of intentionally faulty science questions, and the worst detects 6%. The paper shows that the failure is not fixed by reasoning strength alone; GPT-4o still answers '0.5' to the one-woman-three-men reproduction question in 8 of 10 trials. Tool access changes the picture: adding web search raises GPT-4o's detection rate from 16% to 65%, while a multi-model agent reaches 30%. The paper also reports temporal inconsistency in a single model's behavior, where a correct rejection of a faulty question makes a later rejection more likely but not reliable.","pith_inferences":["A natural extension not tested in the paper: prompting a model to explicitly classify a question as valid, invalid, or unanswerable before solving would likely shift detection rates, and could be measured on the same 1,333 questions.","The adversarial loop could be ported to other domains where false premises are common, such as legal hypotheticals or medical self-diagnosis questions, where silently answering a nonsense input is similarly dangerous.","If the benchmark labels were re-audited by independent human raters with an explicit rubric, the headline detection rates could move substantially; the paper's own validation step is stated but not quantified."],"forward_implications":["If the reported rates generalize, standard chatbots will continue to produce invalid answers to impossible science questions, so deployment in education or scientific assistance should include refusal or verification behavior.","Web-search integration is a concrete, immediately available mitigation: the paper's numbers suggest external information restores most of the missing fault detection without retraining.","Multi-agent verification gives only a partial improvement over a single model, implying that collaboration alone is not enough and architecture or training changes are also needed.","GAN-style adversarial generation offers a route to build new benchmarks automatically as models improve, addressing the 'running out of benchmarks' problem the paper raises."],"supporting_citations":[{"why":"Supplies the base science questions from SciQ that the GAN-style generators corrupt into faulty versions.","marker":"Welbl et al."},{"why":"Supplies the SciQA dataset, the second source of base questions for the faulty-question generation pipeline.","marker":"Auer et al."},{"why":"GPQA is the benchmark the paper cites to argue current models are nearing human-level science QA, motivating a new kind of fault-detection benchmark.","marker":"Rein et al."},{"why":"ScienceQA provides the multimodal question-answering context that motivates including image-plus-text faulty questions.","marker":"Lu et al., 2022"},{"why":"Cited for the GPT-o1 reasoning model that the paper tests and that motivates the need for harder benchmarks.","marker":"OpenAI"}],"fun_headline_variants":["LLMs answer impossible science questions instead of calling them out","SciFaultyQA: best LLM detects just 16% of faulty science questions","Web search lifts LLM fault detection from 16% to 65% on new benchmark","Nonsensical science problems: LLMs solve them, rarely spot the flaw","New benchmark exposes LLMs' blind spot for impossible science queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark labels themselves are correct, because the paper's validation step (an AI agent with web search plus human evaluation) is described without reporting sample size, agreement, or a definition of 'faulty'; if many generated questions are actually valid, every detection rate loses its meaning.","fun_headline_variants_meta":{"raw":{"variants":["LLMs answer impossible science questions instead of calling them out","SciFaultyQA: best LLM detects just 16% of faulty science questions","Web search lifts LLM fault detection from 16% to 65% on new benchmark","Nonsensical science problems: LLMs solve them, rarely spot the flaw","New benchmark exposes LLMs' blind spot for impossible science queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000702,"raw_usage":{"total_tokens":3160,"prompt_tokens":932,"completion_tokens":2228,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2127}},"tokens_in":548,"tokens_out":2228,"duration_ms":12767,"temperature":1.0,"reasoning_tokens":2127,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:22:41.259173+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 100 SciFaultyQA questions and have independent science educators classify each as valid, invalid, or ambiguous using a written rubric. If the educators reject a large share of the 'faulty' labels, or if GPT-4o's detection rate changes materially after relabeling, the reported 6-16% baseline does not measure what the paper claims.","supporting_citations":[],"review_version":1}