{"id":"629bcd69-ce61-4005-8007-f559dc8ea331","arxiv_id":"2501.01588","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning PHI-3 on TruthfulQA with a hand-tuned prompt yields a reported 90.8% accuracy, but the paper describes no held-out evaluation.","lead":"This paper fine-tunes Microsoft's PHI-3 language model on TruthfulQA multiple-choice questions and reports accuracy rising to 90.8% with lower perplexity. The results are not backed by a described held-out test split, so the improvement may be memorization rather than general skill.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out evaluation is described; the 90.8% accuracy may reflect training-set memorization, and the abstract/Table I baseline mismatch remains unexplained.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern: the absence of a described train/test split makes the reported 90.8% accuracy potentially a memorization artifact. The paper's Section IV-C, which documents positional overfitting, strengthens this concern because it shows the model can exploit prompt format rather than generalize. The abstract/Table I mismatch (62% vs. 78.3% baseline) further undermines trust in the reported metrics. My read does not change the reader's REJECT verdict; the paper needs a properly held-out evaluation and a corrected baseline before the claim can be assessed.","tokens_in":5653,"tokens_out":1916,"duration_ms":19460,"concrete_test":"Obtain the exact preprocessing script and the precise data split used (or re-run with a specified random split, e.g., 800 train / 200 test, seed 3407), then recompute Table I on the test subset only for both the PHI-3 baseline and the fine-tuned PHI-3.5, using the same best prompt. Report accuracy and perplexity on those held-out items, and also evaluate with shuffled option order to rule out positional bias. If held-out accuracy is materially below 90.8% or close to the baseline, the central claim is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that fine-tuning PHI-3.5 on TruthfulQA with optimized prompts improves MCQ accuracy from 62% to 90.8% and lowers perplexity from 4.68 to 2.27. For this claim to hold, these metrics must be computed on items not used during training. The paper never states a train/test split (Sections III and IV), and the training setup in Section IV-B (60 steps, effective batch size 8, 1,000-item dataset) implies the model saw a substantial portion of the dataset; with no held-out evaluation, the improvement could be memorization. This concern is reinforced by the paper's own admission of positional overfitting in Section IV-C, where the model learned to select the last option instead of reasoning. Additionally, the baseline accuracy is inconsistent: the abstract reports 62% while Table I reports 78.3% for the PHI-3 baseline, so the claimed gain is not even internally consistent. Without a described held-out split and a corrected baseline, the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of fine-tuning Microsoft PHI-3 (referred to in places as PHI-3.5) on the TruthfulQA dataset for multiple-choice question answering. The authors describe a pipeline of dataset preprocessing, prompt design, supervised fine-tuning with an 8-bit optimizer, and evaluation with perplexity, accuracy, F1, and recall. The central claim, stated in the abstract and Section V, is that fine-tuning improves accuracy from 62% to 90.8%, lowers perplexity from 4.68 to 2.27, and raises F1 from 66 to 90.6, with Table I reporting a PHI-3 baseline accuracy of 78.3%. The paper also acknowledges that an initial prompt format induced a positional bias where the model consistently selected the last option.","tokens_in":5967,"tokens_out":3137,"duration_ms":31663,"significance":"If the central claim were supported, the paper would provide a notable data point: a 1.3B-parameter model reaching 90.8% MCQ accuracy after only 60 training steps on a 1,000-item dataset would be remarkable for resource-constrained educational applications. The work also transparently reports training hyperparameters and points to code and data, which are strengths. However, the claimed results are not backed by a described evaluation protocol: no held-out split is stated, the abstract and Table I disagree on the baseline accuracy, and the paper itself admits a positional-overfitting failure mode. As a result, the main quantitative claims are currently unsupported.","major_comments":[{"comment":"The paper never states that evaluation was performed on items held out from fine-tuning. Section IV-B describes training for up to 60 steps with an effective batch size of 8 on a 1,000-item dataset, and Section V reports perplexity and accuracy improvements, but no train/test split, cross-validation, or evaluation-set construction is described anywhere in Sections III–V. Without a held-out split, the reported drop in perplexity from 4.68 to 2.27 and the accuracy gain from 62% (abstract) or 78.3% (Table I) to 90.8% are compatible with memorization of the training data, which is exactly the risk raised by the paper's own admission of positional overfitting in Section IV-C. This is load-bearing for the central claim and cannot be verified from the manuscript.","section":"Sections III, IV, V"},{"comment":"The reported baseline accuracy is internally inconsistent: the abstract states that accuracy rose from 62% to 90.8%, while Table I lists the PHI-3 baseline as 78.3%. Similarly, the abstract and Section V state that F1 increased from 66 to 90.6, but Table I lists baseline F1 as 0.75 and fine-tuned F1 as 0.90. These mismatches mean a reader cannot determine the actual improvement, and the central claim of a 62% to 90.8% jump is contradicted by the paper's own table.","section":"Abstract and Table I"},{"comment":"The paper states that with the initial prompt format the model 'consistently chose the last option' and 'learned to exploit the options' positions rather than engage in reasoning.' The revised prompt is said to fix this, but no evaluation on held-out data or ablation is provided to show that the final prompt does not still induce a positional or superficial bias. Given that the reported accuracy is the only evidence for the fine-tuning benefit, this acknowledged failure mode directly undermines the claim that the final results reflect understanding rather than format exploitation.","section":"Section IV-C"}],"minor_comments":[{"comment":"The text refers to 'Section 4.2' for the revised prompt details, but no numbered subsection 4.2 exists; the prompt revision is not actually described in detail anywhere.","section":"Section IV-C"},{"comment":"The paper says TruthfulQA contains '1,000 MCQs,' but the original TruthfulQA release contains 817 questions; the preprocessing that yields 1,000 items, if any, should be described and cited precisely.","section":"Section III-A.1"},{"comment":"The model is called both PHI-3 and PHI-3.5 in the abstract, Section V, and Table I; the authors should clarify which model version was actually fine-tuned and evaluated.","section":"Throughout"},{"comment":"Reference [5] lacks year, venue, and arXiv identifier, and the citation [12] in the Conclusion appears to be the TruthfulQA reference placed after 'future work' rather than where it is needed in the methodology section.","section":"References"},{"comment":"The introduction says 'as it is clear in I' without a clear figure reference, and Figure 1 has a caption but no visible image content; the figure should be either included or removed.","section":"Introduction and Figure 1"},{"comment":"The paper says training was run for 'a maximum of 60 steps' but does not report the actual number of steps used or the number of training examples per step, which makes it impossible to compute how many epochs the model saw.","section":"Section IV-B"},{"comment":"The code and dataset are referred to as 'in here' placeholders; the actual URLs are not printed in the manuscript, preventing reproducibility.","section":"Abstract and Section V"}],"recommendation":"reject","confidential_remarks":"This manuscript appears to be an early technical report. The central quantitative claim is unsupported by the reported evaluation because no held-out split is described and the baseline numbers are internally inconsistent. These are load-bearing issues that cannot be fixed with text edits alone; a proper held-out evaluation and corrected reporting would require rerunning the experiments. I would encourage the authors to resubmit after such a revision, possibly as a longer paper with a detailed evaluation protocol and ablations that address the acknowledged positional bias."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper reports a fine-tuning run of PHI-3 on TruthfulQA for MCQ answering, with a striking accuracy jump to 90.8% and a perplexity drop from 4.68 to 2.27. But the evaluation is not described as held-out, the baseline number changes from 62% in the abstract to 78.3% in Table I, and the paper itself admits the model learned to pick the last option. So the central claim is unsupported as written. That said, there is something honest and useful here: the author clearly documents a positional-overfitting failure, which is a real pitfall in MCQ fine-tuning and a more candid admission than most papers give.\n\nThe concrete training setup—60 steps, batch size 8, seed 3407, on a GTX 1650—is clearly stated, and the goal of adapting a 1.3B model for low-resource education is sensible. Unfortunately, the dataset and code links are placeholders ('here' with no URLs), so rerunning the experiment is not possible as submitted.\n\nThe load-bearing flaws are the ones the stress-test note flags. No train/test split is stated anywhere. With 1,000 examples and 60 training steps, the reported accuracy could easily be memorization. The abstract says baseline accuracy was 62%, Table I says 78.3%—that is a large unexplained discrepancy. The prompt section is vague ('best-performing prompt combined elements from both approaches'), so there is no reproducible prompt-engineering insight. And the introduction's claim of a 'novel approach to prompt design' is not backed by any specific innovation.\n\nThese are not minor blemishes; they undermine the headline result. The underlying experiment could be valid if rerun with a proper held-out split and corrected baselines, but this paper does not support the 90.8% claim. I would send it to peer review for the sole purpose of giving the author a clear path to fix the evaluation—the flaws are fundamental but fixable. I would not cite it, and I would not bring it to reading group except as a short example of how not to report fine-tuning results. The honest admission about positional bias is the only part I'd remember.","headline":"A candid but methodologically unsupported fine-tuning report; the 90.8% accuracy claim cannot be trusted without a held-out split, but the positional-bias admission is genuinely instructive.","tokens_in":6368,"tokens_out":2720,"would_cite":false,"duration_ms":25730,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Paper claims fine-tuning PHI-3 on TruthfulQA lifts multiple-choice accuracy to 90.8%.","keywords":["LLM","fine-tuning","multiple-choice questions","TruthfulQA","PHI-3","prompt design","perplexity"],"falsifier":"Hold out a random 20% of the TruthfulQA questions, fine-tune only on the remaining 80% with the paper's stated hyperparameters, and report accuracy on the held-out set; if held-out accuracy is well below 90.8% while training-set accuracy is near 90.8%, the reported gain is memorization rather than learning to answer MCQs.","tokens_in":5439,"feed_emoji":"🎯","tokens_out":6279,"duration_ms":57176,"temperature":0.7,"pith_summary":"The paper sets out to show that a compact language model, PHI-3 at 1.3 billion parameters, can be turned into an accurate multiple-choice question answerer by supervised fine-tuning on the TruthfulQA dataset and by redesigning the prompt. The author reports perplexity dropping from 4.68 to 2.27 and accuracy rising from 62% to 90.8% after only 60 training steps on a standardized 1,000-question set. The motivation is practical: if small models can reach this level, automated tutoring and assessment systems can run on modest hardware. The paper presents this as evidence that prompt design and fine-tuning matter as much as model scale for structured tasks like MCQ answering.","feed_headline":"Fine-tuned PHI-3 hits 90.8% on MCQ answers","feed_subtitle":"After 60 training steps, a 1.3B model beats its 62% baseline—if the gains hold on unseen questions.","key_machinery":"The machinery is the combination of dataset standardization and prompt engineering before supervised fine-tuning. TruthfulQA is a factual multiple-choice benchmark designed to expose hallucination, and the paper standardizes it by keeping the best correct answer and equalizing the option count. The prompt is the sensitive part: a simple completion prompt caused the model to exploit answer position, always picking the last option, while an Alpaca-style instruction prompt (a style of instruction-following prompt template) plus fine-tuning made the model answer by content. This prompt-plus-fine-tuning pipeline carries the claim, because the reported gains depend on replacing positional guessing with content-based selection.","core_discovery":"On the paper's own terms, the central discovery is that PHI-3, initially a general text generator, becomes a reliable MCQ answerer after supervised fine-tuning and prompt redesign. The best prompt combines the basic completion format with an Alpaca-style instruction format, and the dataset is preprocessed so each question has one best answer and a consistent number of options. The author reports that with this setup the fine-tuned PHI-3.5 reaches 90.8% accuracy, an F1 score of 90.6, and a perplexity of 2.27, outperforming the GPT-3 baseline in the comparison table despite using roughly 1% of the parameters. The paper attributes the gain to eliminating the positional bias that made the initial prompt always choose the last option.","pith_inferences":["The paper never states a train/test split, so an immediate extension is to report held-out accuracy separately from training-set accuracy to determine whether the 90.8% figure reflects generalization or memorization.","Varying the order of answer options during evaluation would measure how much of the gain is genuine content understanding rather than residual positional bias, which the paper itself identifies as a failure of its initial prompt.","Testing the same prompt design on other MCQ benchmarks such as MMLU or ARC would show whether the method transfers beyond TruthfulQA's specific format.","The abstract's 62% baseline and Table I's 78.3% baseline differ, so the size of the reported improvement depends on which baseline the reader uses."],"forward_implications":["If the result holds, a 1.3-billion-parameter model can outperform a reported GPT-3 baseline on TruthfulQA-style MCQs while being deployable on consumer hardware.","The design implies that prompt formatting is as important as model scale for MCQ tasks; a poorly structured prompt leads to positional overfitting rather than comprehension.","Resource-constrained educational platforms could use fine-tuned compact models for automated assessment and tutoring without large GPU clusters.","The perplexity drop from 4.68 to 2.27 suggests the fine-tuned model is substantially more confident in its answer choices, which would matter for real-time feedback systems."],"supporting_citations":[{"why":"Supplies the PHI-3 base model that the paper fine-tunes.","marker":"[1]"},{"why":"Supports the premise that multiple-choice questions are an efficient and robust format for evaluating LLMs.","marker":"[2]"},{"why":"Provides the MMLU benchmark context that motivates MCQ evaluation across academic subjects.","marker":"[9]"},{"why":"Supplies the TruthfulQA dataset used for training and evaluation, on which the central claim is measured.","marker":"[12]"},{"why":"Supports the idea that MCQs can probe the reasoning abilities of language models, the task PHI-3 is adapted for.","marker":"[5]"}],"fun_headline_variants":["Small PHI-3 fine-tune jumps MCQ accuracy to 90.8%","PHI-3 beats GPT-3 on MCQs with 1% params","Perplexity halved: PHI-3 fine-tune hits 90.8% on MCQs","Fine-tuned PHI-3 (1.3B) tops GPT-3 in MCQ tests","From 62% to 90.8%: PHI-3 fine-tune for MCQs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result assumes that the reported accuracy of 90.8% was earned on TruthfulQA questions that were not part of the 1,000-item training set, but the paper never states such a split.","fun_headline_variants_meta":{"raw":{"variants":["Small PHI-3 fine-tune jumps MCQ accuracy to 90.8%","PHI-3 beats GPT-3 on MCQs with 1% params","Perplexity halved: PHI-3 fine-tune hits 90.8% on MCQs","Fine-tuned PHI-3 (1.3B) tops GPT-3 in MCQ tests","From 62% to 90.8%: PHI-3 fine-tune for MCQs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000832,"raw_usage":{"total_tokens":3624,"prompt_tokens":927,"completion_tokens":2697,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":2575}},"tokens_in":543,"tokens_out":2697,"duration_ms":17232,"temperature":1.0,"reasoning_tokens":2575,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:24:14.550579+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a random 20% of the TruthfulQA questions, fine-tune only on the remaining 80% with the paper's stated hyperparameters, and report accuracy on the held-out set; if held-out accuracy is well below 90.8% while training-set accuracy is near 90.8%, the reported gain is memorization rather than learning to answer MCQs.","supporting_citations":[{"cited_title":"Can multiple-choice questions really be useful in detecting the abilities of llms?","cited_arxiv_id":null,"evidence_quote":"Supports the idea that MCQs can probe the reasoning abilities of language models, the task PHI-3 is adapted for."}],"review_version":1}