{"id":"b3ed664c-cf2d-4a8c-8a95-724e23985e62","arxiv_id":"2505.12058","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A lightweight multilingual QA smoke-test suite with a 52-item English core and an LLM-based generator is shown to reflect model size and language performance differences in seconds.","lead":"TQB++ is a tiny multilingual question-answer dataset plus a script that generates more such datasets on demand, meant to catch broken LLM pipelines in seconds rather than hours. It is a practical tool for teams that want a fast, unit-test-style check before running heavy benchmarks like MMLU.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No controlled test shows TQB++ actually flags prompt-template errors, tokenizer drift, or fine-tuning side-effects; the experiments only rank models, so the 'reliably flags' claim is unsupported.","rationale":"The paper's resource is plausible and useful: the 52-item English core is hand-curated, the code and datasets are openly released, and the experiments show expected directional gradients across model sizes and languages. I do not object to the benchmark's existence or to the resource contribution. The load-bearing issue is validation scope: the abstract's strongest claim is about detecting specific classes of regressions, yet the paper never introduces any such regression into an otherwise stable pipeline. All observed results are compatible with TQB++ merely reflecting static model and language capability, which is not the same as reliably flagging prompt-template errors, tokenizer drift, or fine-tuning side-effects. The reader's weakest assumption about synthetic label quality is real and relevant, but it is secondary: even if every generated label were perfect, the paper would still lack direct evidence of sensitivity to injected faults. I therefore keep the reader's CONDITIONAL verdict: the artifact likely works as a lightweight evaluation set, but the headline reliability claim should be conditioned on a fault-injection experiment. The proposed concrete test is deliberately small and cheap, matching the paper's own unit-testing ethos, and would resolve whether the strongest claim actually lands.","tokens_in":17397,"tokens_out":4362,"duration_ms":45734,"concrete_test":"Run a controlled fault-injection study with one fixed model and temperature 0 on core_en (52 items) and pack_en_10: (a) baseline with the standard prompt; (b) same prompt with an injected template corruption (e.g., duplicate system role or missing answer prefix); (c) same prompt evaluated after forcing a known tokenizer or Unicode normalization change (e.g., NFKD normalization or a modified subtoken split); (d) the same model after a small fine-tune on an unrelated domain intended to induce forgetting. Repeat each condition at least 5 times. If EM does not fall below the stated 0.95 pass threshold for conditions (b)-(d), the 'reliably flags' claim fails; if it does fall, compare the effect size and wall-clock latency against a full-scale suite to confirm the claimed 'seconds' advantage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a complete TQB++ run 'reliably flags prompt-template errors, tokenizer drift, and fine-tuning side-effects.' Every experiment in Section 7 is cross-sectional: it compares static model checkpoints of different sizes and families on fixed datasets, showing performance gradients. No experiment introduces a known fault into a single pipeline and checks whether TQB++ detects it. Prompt-template errors, tokenizer drift, and fine-tuning side-effects are described in Section 4 as use cases, but they are never operationalized as measurements. A smoke test's defining property is sensitivity to regressions, not merely the ability to rank models by capability. The observed score spread (e.g., 90.4 EM for gemma-3-12b vs. 50.0 for mistral-7b on core_en) is evidence of cross-sectional capability differences, not of reliability in flagging changes over time. This gap is compounded by Appendix A.1, which concedes that synthetic packs are only schema-validated, with no factuality or self-consistency post-filtering; on low-resource packs a low score could be caused by bad or ambiguous generated labels rather than by a real regression. The inconsistent naming of the generation model (gpt-3.5-turbo-0125 referred to as 'o3-mini') also makes the generation setup hard to audit. None of this is internally contradictory, but the strongest claim in the abstract is not supported by the evidence presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Tiny QA Benchmark++ (TQB++), an ultra-lightweight LLM smoke-test suite built around a 52-item English gold QA set (core_en), a LiteLLM-based synthetic QA generator, ten ready-made multilingual packs, Croissant metadata, and integrations for CI/CD, OpenAI Evals, LangChain, and observability dashboards. The experiments evaluate eight open-weight models on the English core, synthetic English packs of sizes 10–40, and French, Japanese, and Turkish 40-item packs, reporting Exact Match and Levenshtein Ratio scores. The results show plausible performance gradients by model family/size and by language, with English scores highest and Japanese/Turkish scores substantially lower. The abstract and Section 4 claim that a TQB++ run 'reliably flags prompt-template errors, tokenizer drift, and fine-tuning side-effects,' but the paper's experiments are cross-sectional comparisons of static checkpoints; no controlled fault-injection test of these failure modes is presented. The synthetic generation pipeline is only schema-validated, with no factuality or self-consistency post-filtering, which weakens the reliability of low scores on multilingual packs as regression signals.","tokens_in":17648,"tokens_out":4096,"duration_ms":40458,"significance":"If the headline claim were demonstrated, TQB++ would be a genuinely useful low-cost canary for continuous LLM evaluation, and the open release of the dataset, generator script, Croissant metadata, and SHA-256 provenance has real community value. The paper also correctly scopes itself as a smoke test rather than a ranking benchmark, citing tinyBenchmarks and acknowledging that small sets are unsuitable for fine-grained model differentiation. However, the current evidence only supports cross-sectional capability ranking, not the claimed reliability in detecting prompt-template errors, tokenizer drift, or fine-tuning side-effects. The lack of factuality verification for generated labels and the small per-pack sample sizes further limit the strength of the central claims. The resource itself is a reasonable engineering contribution, but the paper's central selling point is not yet validated by the experiments.","major_comments":[{"comment":"The abstract states that a complete TQB++ run 'reliably flags prompt-template errors, tokenizer drift, and fine-tuning side-effects,' but Section 7 contains no experiment that introduces a known prompt-template error, a tokenizer modification, or a fine-tuning checkpoint and checks whether TQB++ detects it. All reported evaluations compare static model checkpoints on fixed datasets, which supports capability ranking but not regression-detection reliability. For example, the Turkish tokenizer scenario in Section 4.4 is presented as a use case with a specific 18-percentage-point drop, but it is never operationalized as a measurement.","section":"Abstract; Section 7.1"},{"comment":"The generator's post-filtering is limited to schema validation: 'No further automated post-filtering, such as self-consistency checks or external factuality verification against a knowledge base, is implemented directly within this basic generation script. The quality relies on the LLM's ability to follow instructions and the schema validation.' Because generated packs are not factuality-checked, a low EM score on pack_ja_40 or pack_tr_40 could be caused by incorrect, ambiguous, or culturally inappropriate generated labels rather than by a real model regression. This directly undermines the smoke-test interpretation of low scores, which is a load-bearing part of the paper's utility claim.","section":"Appendix A.1"},{"comment":"The generation model is identified inconsistently: Section 4.8 says 'The current study focused on 'o3-mini' for generation,' while Appendix A.1 states that the model was OpenAI's gpt-3.5-turbo-0125 'referred to as o3-mini in Section 6.' The temperature setting also differs between Section 6 (temperature 0.0) and Appendix A.1 (temperature 1.0 for OpenAI reasoning models). This ambiguity makes the generation setup difficult to audit and impairs reproducibility of the synthetic packs.","section":"Appendix A.1; Section 4.8"},{"comment":"The Levenshtein Ratio acceptance threshold of 0.95 is selected as the F1-maximizing threshold against EM ground truth on the same model predictions that are then used with the threshold in reporting, and no held-out validation or confidence interval is reported. The reported F1 of 1.000 at the chosen threshold suggests overfitting to the calibration sample. The threshold analysis should be repeated with a held-out split or nested bootstrap, and the uncertainty in the optimal threshold should be reported.","section":"Section 7.4"},{"comment":"Several 'signals' rest on one-item differences on packs with n=10 or n=20. For example, Table 5 shows gemma-3-12b at EM 100.0 and gemma-3-4b at 90.0 on pack_en_10, a difference of a single answer. Section 7.5 describes intra-family deltas as detectable signals without any confidence intervals, error bars, or statistical test. The small-sample variance makes these directional claims unsupported; at minimum, bootstrap intervals for EM/LR scores should be reported before asserting sensitivity to intra-family differences.","section":"Section 7.5; Table 5"}],"minor_comments":[{"comment":"Section 2.1 states that no core items are labeled 'hard,' but Table 2 reports a 'Hard' column with scores (e.g., 49.0 for gemma-3-12b). The table should clarify whether these scores come from synthetic packs or a different subset, or the description of the core set should be corrected.","section":"Section 2.1; Table 2"},{"comment":"The generator size is given as 'approximately 40 lines of Python' in Section 3.1 but 'approx. 300 lines' (or '<200 when not packaged for PyPI') in Appendix A.1. These numbers should be reconciled.","section":"Section 3.1; Appendix A.1"},{"comment":"There are several typos and formatting errors: 'rsaw Levenshtein Ratio' in Section 6, 'reprodceability' in Appendix A.1, 'suplimentary' in the Appendix A heading, the equation label 'extcos' for cosine similarity, and the system prompt showing 'tagscategory' instead of 'tags.category'.","section":"Section 6; Appendix A.1"},{"comment":"The LR calibration figure is referenced as 'Figure ??', leaving a placeholder in the text. The figure should be properly labeled and cited.","section":"Section 7.4; Figure 5"},{"comment":"Some references are malformed, notably 'Osma endangering Trazona' in the Croissant citation and the undefined 'urldate' field used in several entries. The bibliography should be cleaned before publication.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This appears to be a workshop-style technical report rather than a full research contribution, and the evidence does not yet support the abstract's headline claim. The most productive revision path is to add controlled fault-injection experiments (prompt-template errors, tokenizer perturbations, and a fine-tuning checkpoint comparison) and to tighten the synthetic-data quality story. I would also flag for the editor that the reference list is heavily self-referential: roughly seven of the twenty-five entries are the author's own blog posts, preprints, or conference papers, and several are cited in support of generic claims. This is worth noting in assessing novelty and scholarly framing, though it is not by itself disqualifying."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: TQB++ is a real, lightweight evaluation artifact that practitioners will find handy, but the strongest sentence in the abstract — that it “reliably flags prompt-template errors, tokenizer drift, and fine-tuning side-effects” — is not supported by the evidence. Every experiment in Section 7 is cross-sectional: static checkpoints of different models on fixed datasets. No test introduces a template change, a tokenizer swap, or a fine-tuning side-effect and checks whether TQB++ catches it. The stress-test note is right.\n\nWhat is actually new: the 52-item English core is prior work, but the synthetic generator (LiteLLM script), the ten ready-made multilingual packs, Croissant metadata, and the CI/CD packaging are new artifacts. They are real, shippable, open-source, and seem easy to drop into a pipeline. That alone is a legitimate contribution for an LLMOps audience. The paper also earns credit for honesty: Appendix A.1 plainly states that synthetic labels are only schema-validated, with no factuality or self-consistency filtering, and the limitations section is appropriately modest.\n\nSoft spots, in proportion. The main gap is between the artifact and the claim. Prompt-template errors and tokenizer drift are described as use cases (Section 4) but never operationalized as measurements. If I were refereeing, I would ask for a simple controlled experiment: take one working pipeline, break the prompt template, swap the tokenizer, or apply a known fine-tuning degradation, and show TQB++ detects the change. That would make the paper's central claim credible. Also minor but annoying: the generation model is named inconsistently — gpt-3.5-turbo-0125 in Appendix A.1 is called “o3-mini” in the acknowledgments and Section 8 — and the LR threshold story is muddled: tables use 0.75, bootstrapping says 0.95 is optimal, and the paper never reconciles the two. No error bars on 10-item packs, but for a smoke-test that is a minor issue.\n\nWho is it for: engineers building LLM CI/CD gates, not researchers expecting new scientific insight. The resource is genuinely useful, and the paper's candor about its own limits is a good sign.\n\nRecommendation: send it to peer review, but expect a major revision. The artifact deserves referee time; the claims need to be aligned with the evidence, either by softening the language or adding the fault-injection experiment. I would not desk-reject this.","headline":"A genuinely useful, openly-shipped tool for LLM smoke testing, but the paper's headline claim that it 'reliably flags' regressions is not tested — the experiments are cross-sectional rankings, not fault-injection.","tokens_in":705,"tokens_out":939,"would_cite":true,"duration_ms":29663,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small QA set catches LLM regressions before big benchmarks do","keywords":["LLM evaluation","QA benchmarks","synthetic data","LLMOps","continuous integration","smoke testing","multilingual evaluation","Croissant metadata"],"falsifier":"Run a deliberately broken pipeline through TQB++—for example, a prompt template that truncates every answer to one character, or a tokenizer that drops non-Latin script—and compare accuracy on the 52-item core and a 40-item multilingual pack with accuracy on a human-checked version of the same questions. If the tiny suite stays above its pass threshold while the human-checked set drops by several points, the early-warning claim is refuted; separately, fact-check a generated pack and count how many labels are wrong or ambiguous.","tokens_in":17144,"feed_emoji":"🧪","tokens_out":10507,"duration_ms":93538,"temperature":0.7,"pith_summary":"TQB++ claims that a fixed 52-question English set, plus tiny on-demand multilingual packs, can act as the unit-test layer of LLM evaluation: a full run takes seconds and negligible cost, yet it can flag prompt-template errors, tokenizer drift, and fine-tuning side effects before a heavyweight suite such as MMLU or BIG-Bench is even set up. The reported experiments show top models scoring about 90% exact match on the core set, a 10-item synthetic pack reproducing the same coarse model ordering as a 40-item pack, and sharp language gradients (mean exact match 86.1 for English, 60.0 for French, 36.3 for Turkish, 29.1 for Japanese across the tested models). A short generator script mints schema-validated QA packs in any language, domain, or difficulty, stamps each item with a SHA-256 hash, and ships the packs with standard metadata, so teams can keep private, scenario-specific smoke tests. If the claims hold, continuous LLM quality assurance becomes a per-commit gate rather than an occasional research exercise.","feed_headline":"Small QA set catches LLM regressions before big benchmarks do","feed_subtitle":"A fixed English core plus generated multilingual packs gives CI pipelines a second-scale smoke test for prompts, tokenizers, and fine-tunes.","key_machinery":"The carrying object is a two-part benchmark-plus-generator pair. The first part is an immutable JSON core of 52 question-answer triples, each with a one-sentence context and category/difficulty tags; the second is a short generator script that sends two few-shot JSON exemplars to an LLM through a provider-agnostic client, validates that the returned list has the required keys, retries up to three times on malformed output, and hashes every item with SHA-256 for provenance. Evaluation uses normalized exact match for the deterministic core and a Levenshtein ratio for generated packs, with the acceptance threshold chosen by bootstrapping to maximize F1 against exact-match ground truth. That machinery makes failures salient and custom packs cheap, private, and reproducible.","core_discovery":"The central claim is that a small, deliberately easy QA set works as an early-warning canary for LLM systems. The paper argues that because competent models are expected to ace the 52-item core, any failure is conspicuous, and because the synthetic packs are cheap to regenerate, teams can guard against overfitting to a fixed test set. Its evidence is sensitivity: larger models in the same family generally outperform smaller siblings, scores drop as stated difficulty rises, and multilingual packs separate languages sharply, with the biggest drops on Japanese and Turkish. The author positions TQB++ not as a ranking instrument but as a pass/fail gate that runs in seconds and decides whether a model or pipeline is worth sending to expensive, comprehensive evaluation.","pith_inferences":["One untested extension, which the paper lists as future work, is to feed production telemetry into the generator so new smoke-test packs are minted from emerging failure themes; if it worked, the suite would update itself as the application drifts.","Because labels are only schema-validated, any team adopting the generator for a new language or domain should independently fact-check a sample of items before treating the pack as a hard CI gate; the paper concedes this is not implemented.","The same canonical-input, pass/fail logic should transfer to non-QA smoke checks such as instruction-following or tool-call formatting, though the paper does not demonstrate this."],"forward_implications":["CI/CD pipelines can gate every commit on TQB++: the 52-item core runs in about 0.5 seconds on a CPU and the paper suggests failing a build when exact-match accuracy drops below 0.95, roughly two errors.","Even a 10-item synthetic pack preserves the directional model-ordering signal visible in a 40-item pack, supporting randomized smoke tests that resist overfitting to a fixed set.","Multilingual packs turn localization regressions into visible signals; the paper's example is a Turkish tokenizer change that would be caught as an 18-percentage-point accuracy drop before user impact.","Fine-tuning runs can be checked at each checkpoint against relevant TQB++ packs to catch catastrophic forgetting early, before expensive full-scale evaluation is worth running.","A pass on TQB++ is not a green light for release; it is the cheap first stage of a two-stage pipeline that still ends in MMLU, BIG-Bench, or HELM."],"supporting_citations":[{"why":"Defines MMLU, the heavyweight suite TQB++ contrasts with and defers to after the lightweight gate passes.","marker":"Hendrycks et al., 2021"},{"why":"Defines BIG-Bench, another costly suite that motivates the need for a seconds-scale first-line check.","marker":"Srivastava et al., 2023"},{"why":"Supplies the result that about 100 curated examples can approximate larger benchmark rankings, the empirical warrant for small-set efficacy.","marker":"Polo et al., 2024"},{"why":"Caveat that very small benchmarks have high variance, which shapes TQB++'s stated scope as binary disaster detection rather than fine-grained ranking.","marker":"Hochlehnert et al., 2025"},{"why":"The original 52-item Tiny QA Benchmark core that TQB++ extends.","marker":"Koc, 2025d"},{"why":"Defines edit distance underlying the Levenshtein-ratio metric and its bootstrapped threshold.","marker":"Levenshtein, 1966"},{"why":"Defines Croissant JSON-LD, the metadata standard TQB++ adopts for dataset packaging and auto-discovery.","marker":"Akhtar et al., 2024"}],"fun_headline_variants":["Tiny QA benchmark catches LLM regressions in seconds","Second-scale smoke test for LLM pipelines, under 20kB","Small synthetic QA packs guard against model drift","Ultra-light LLM canary: runs in CI in seconds","A 52-item set to test prompts, tokenizers, and fine-tunes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every generated question has one correct, unambiguous answer: the generator only validates JSON schema and never checks facts, and the paper's appendix explicitly says label quality relies on the LLM following instructions.","fun_headline_variants_meta":{"raw":{"variants":["Tiny QA benchmark catches LLM regressions in seconds","Second-scale smoke test for LLM pipelines, under 20kB","Small synthetic QA packs guard against model drift","Ultra-light LLM canary: runs in CI in seconds","A 52-item set to test prompts, tokenizers, and fine-tunes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1482,"prompt_tokens":954,"completion_tokens":528,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":441}},"tokens_in":570,"tokens_out":528,"duration_ms":5016,"temperature":1.0,"reasoning_tokens":441,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:40:56.210049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a deliberately broken pipeline through TQB++—for example, a prompt template that truncates every answer to one character, or a tokenizer that drops non-Latin script—and compare accuracy on the 52-item core and a 40-item multilingual pack with accuracy on a human-checked version of the same questions. If the tiny suite stays above its pass threshold while the human-checked set drops by several points, the early-warning claim is refuted; separately, fact-check a generated pack and count how many labels are wrong or ambiguous.","supporting_citations":[],"review_version":1}