{"id":"842284b2-631b-4361-a48d-ecff0c92ca9b","arxiv_id":"2411.08563","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A fine-tuned GPT-3.5 Turbo model predicts the direction of held-out food-policy experiments with 79% accuracy, but only 55% on preregistered unpublished studies.","lead":"The authors fine-tuned a GPT-3.5 Turbo model on 74 food-policy experiments and found it predicted the direction of results in about 79% of held-out studies. The result is an early proof of concept for LLM-based decision support, though accuracy drops to 55% on truly unpublished studies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"79% direction accuracy may be inflated by study-level train/test leakage; the only clean generalization test gives 55%, so the headline claim is not yet supported.","rationale":"The paper is a proof-of-concept with a standard fine-tuning pipeline, and the authors deserve credit for testing on 12 preregistered unpublished studies, which is the right type of out-of-sample check. That check, however, produced 55% accuracy, barely above the 50% naive baseline and far below the 79% reported on the in-corpus test set. The most likely technical explanation for this gap is not simply distribution shift but study-level leakage in the train/test split: because multiple effect sizes can come from the same paper, and the prompt includes identifying details, the model can memorize paper-level outcomes. This makes the 41-prompt test set partially in-sample. A paper-stratified reanalysis would settle the question. If leakage is confirmed, the abstract's 80% claim should be withdrawn or heavily qualified; the honest conclusion would be that fine-tuned LLMs currently fail to generalize to unseen studies (55%). If leakage is absent, the paper still needs baselines, confidence intervals, and artifact release, as the reader noted. I therefore keep the CONDITIONAL verdict, adding this specific leakage test as the primary condition.","tokens_in":12744,"tokens_out":5344,"duration_ms":46097,"concrete_test":"Compute how many of the 41 test prompts share a source paper with the 144 training prompts. Then re-run the fine-tuning and evaluation with a paper-stratified split: assign all effect sizes from each of the 74 papers to a single partition (e.g., 80/10/10 papers) and report effect-direction accuracy on the held-out papers. If the overlap is nonzero and paper-stratified accuracy drops materially (e.g., below 65%), the 79% figure is inflated by leakage; if accuracy remains near 79%, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 splits 208 effect-size prompts from 74 papers into 144 training, 23 validation, and 41 test prompts, but does not state that all prompts from a single paper are kept in the same partition. With roughly 2.8 effect sizes per paper, a random prompt-level split will often place some outcomes from a given study in training and others in the test set. Because each prompt includes the article title, location, population, and sample size, the model can memorize study-specific outcome direction during fine-tuning and reproduce it for test prompts drawn from the same study. The paper's own out-of-corpus check (Section 2.4), 12 preregistered unpublished studies, yields 55% direction accuracy, essentially at chance. Therefore the reported 79% is not evidence that the model predicts outcomes of new empirical studies; it may only measure within-corpus memorization. The central claim in the abstract—'accurately predict the direction of outcomes in approximately 80% of empirical studies'—is load-bearing and unsupported until this overlap is ruled out.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PREDICT, a proof-of-concept in which a fine-tuned GPT-3.5 Turbo model is trained on prompts derived from 74 published food-policy and behavioral-intervention studies (208 effect sizes) and evaluated on held-out prompts. The authors report 79% accuracy in predicting effect direction on a held-out set of 41 prompts, with additional analyses of prompt design, training-set size, and feature removal. They also test the model on 12 preregistered, unpublished studies and report 55% direction accuracy. The paper argues that fine-tuned LLMs can serve as decision-support tools for predicting the direction of behavioral food-policy outcomes, while acknowledging limitations in generalization.","tokens_in":12888,"tokens_out":3290,"duration_ms":31616,"significance":"If the central claim were fully supported, the paper would make a useful methodological contribution: a low-cost, LLM-based screening tool for policy interventions, particularly in food policy, with implications for other behavioral-policy domains. The authors deserve credit for including a preregistered unseen-study validation set, for varying training size and prompt features, and for explicitly discussing external validity and limitations. However, the strength of the contribution currently rests on two load-bearing pillars that are not yet established: the held-out 79% accuracy is potentially inflated by prompt-level leakage, and the paper's only clean generalization test gives 55%, which is at chance. The manuscript also mislabels its headline quantitative error metric, and several claims about optimal training size rest on single-run observations without uncertainty quantification.","major_comments":[{"comment":"The train/validation/test split is described at the level of 208 prompts (144/23/41), but it is not stated whether all prompts originating from the same research paper are kept in the same partition. With 208 effect sizes drawn from 74 papers, a random prompt-level split will, with high probability, place some effect sizes from a given paper in training and others in the test set. Because each prompt contains the article title, location, population, and sample size, the model could memorize study-specific outcome directions during fine-tuning and reproduce them for test prompts from the same study. This would make the reported 79% direction accuracy an artifact of within-corpus memorization rather than a measure of prediction for new studies. The authors must report whether any paper appears in more than one partition, and ideally re-run the evaluation with a study-level split; if the overlap exists, the 79% figure cannot support the abstract's claim about predicting outcomes of empirical studies.","section":"Section 4.3"},{"comment":"The metric labeled 'average absolute error' in Table 1 is not an absolute error. Section 4.5 defines the error as |predicted| - |actual|, which is a signed difference of absolute values, and Table 1 reports negative values such as -0.058 and -0.009. A true mean absolute error cannot be negative. Because the paper uses negative values to mean 'underestimation', the reported quantity is a signed bias, not an absolute error. This mislabeling affects all quantitative accuracy claims for r and Cohen's d, including the comparisons between models MP3 and MP4 and the feature-removal analysis in Section 2.3. The authors should report mean absolute error and signed bias separately, with definitions matching the names used.","section":"Section 2.1, Table 1, and Section 4.5"},{"comment":"The unseen-study validation, which is the only test that rules out leakage, yields 55% direction accuracy on 12 preregistered studies. With 12 binary outcomes, chance-level performance is 50%, and the reported 55% is statistically indistinguishable from chance without a confidence interval or a hypothesis test. The paper acknowledges this result but then performs a post-hoc exclusion of two studies involving monetary incentives, reporting improved quantitative errors after seeing the outcomes. This is a selection-bias problem: the exclusion rule is not prespecified and appears to be motivated by the observed poor performance. The authors should report the preregistered analysis as primary, provide a confidence interval for the 55% estimate, and, if they retain the exclusion, justify it with a principled criterion defined before inspecting the results. As written, the abstract's 'approximately 80%' claim is inconsistent with the paper's own clean generalization evidence.","section":"Section 2.4"},{"comment":"The claim that performance is optimal at approximately 75 training prompts, with a non-monotonic pattern and 'catastrophic forgetting' beyond that point, is not supported by the presented evidence. The paper reports a single trajectory over training sizes N = 10, 75, 130, 144, and 167, with no confidence intervals, no repeated runs, and no statistical comparison. Given the small numbers of test prompts, the observed non-monotonicity could easily be sampling noise. The authors should either provide uncertainty estimates and repeated fine-tuning runs, or substantially soften the causal language about catastrophic forgetting and the existence of an optimal training size.","section":"Section 2.2 and Figure 2"}],"minor_comments":[{"comment":"The abstract states that 'approximately 75 prompts were required to achieve optimal results', but Section 2.2 reports that both 75 and 130 prompts are best-performing; Section 2.4 then refers to the model fine-tuned with 130 prompts as the best-performing model. This inconsistency should be resolved.","section":"Abstract and Section 2.2"},{"comment":"The text reports that the model 'correctly predicts the effect direction in approximately 6.6 out of 12 experiments'. If this is an average over 10 inference runs, it should be stated clearly; otherwise the number of correct predictions should be an integer. The phrase '6.6 out of 12' is confusing as written.","section":"Section 2.4"},{"comment":"The inference procedure should describe how non-numeric model outputs are handled. Models MP1 and MP2 have prediction probabilities below 100% for r and d; if some responses are unparseable, the accuracy and error calculations should clarify whether these are excluded or counted as errors.","section":"Section 4.5"},{"comment":"The terms 'average absolute error' and 'variance in absolute errors' are used inconsistently. Table 1 gives a single variance value associated with each signed error, but it is not clear whether the variance is computed over the 10 repeated LLM runs or over the 41 test prompts. The text in footnote 1 suggests both, which is ambiguous.","section":"Section 2.1 and Table 1"},{"comment":"The feature-removal analysis compares models after removing one feature at a time, but the paper does not report whether the same train/test split and the same repeated-run protocol are used for every removed-feature model. This should be stated to allow the comparisons in Figure 3 to be interpreted.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not appear to include code or a link to the dataset, which limits reproducibility of the fine-tuning pipeline. The dataset is said to derive from Lohmann et al. (2024b), and a data-availability statement would be helpful. The central issue for the editor is the prompt-level split: if the authors can show that no paper appears in both training and test sets and re-validate with a study-level split, the main claim may survive; otherwise the correction is substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper. It is a genuine first: fine-tuning GPT-3.5 to predict direction, r, and d for food-policy experiments, with a real external-validation attempt on 12 preregistered unpublished studies. But the headline 79% direction accuracy is very likely inflated by a split that ignores paper identity, and the only clean generalization test—those 12 unseen studies—gives 55%, barely above chance.\n\nWhat the paper does well: the authors are unusually honest about the unseen-study drop, they pre-registered the external set, and they report the per-prompt feature ablations clearly. The observation that 75 prompts beats 144 is interesting, though it could be an artifact of the small corpus.\n\nThe soft spots are serious. The split in Section 4.3 is at the prompt level: 208 effect sizes from 74 papers, split into 144/23/41 with no statement that all prompts from one paper stayed in the same partition. With ~2.8 effect sizes per paper and prompts containing title, location, and sample size, the model can memorize study-specific outcomes and reproduce them for test prompts from the same paper. Until the authors show a paper-level split, the 79% is not a trustworthy held-out estimate. The 55% on the unseen studies is the only clean number, and the authors themselves note it is close to the 50% naive baseline. Also, Table 1's 'average absolute error' is actually a signed difference of absolute values; the values are negative. That is a metric mislabel, minor but sloppy. There is no confidence interval or baseline for the 41-prompt test set, and the post-hoc exclusion of two monetary-incentive studies from the unseen set looks like cherry-picking, even if the authors are transparent about it. No code or data are released, so the whole evaluation is hard to check.\n\nWho is this for? Researchers working on LLM-based prediction of social-science outcomes, and anyone thinking about external validity in behavioral public policy. It is a useful proof-of-concept, but the central quantitative claim is not yet established.\n\nMy recommendation: send it to peer review, but with a clear demand for a paper-level split, explicit baselines, and the artifacts. If the 79% survives a paper-level split, the paper makes a real contribution. If it collapses toward 55%, the paper still has a useful methodological lesson: fine-tuned LLMs generalize poorly to truly unseen studies.","headline":"A genuinely new application with an honest external check, but the 79% headline is likely leakage-inflated; the clean generalization number is 55%.","tokens_in":13461,"tokens_out":2645,"would_cite":false,"duration_ms":23669,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned GPT-3.5 Turbo model predicts the direction of dietary-behavior intervention effects correctly in about 79% of held-out studies, suggesting LLMs can serve as low-cost screens for food-policy decisions.","keywords":["large language models","behavioral public policy","food policy","food waste","fine-tuning","effect size prediction","external validity","prompt engineering"],"falsifier":"Run the best prompt version (P4, 130 training prompts) on a preregistered set of 50 new food-policy experiments published after the model's training cutoff and drawn from multiple countries, populations, and intervention types. If direction accuracy is not clearly above the 50% naive baseline on that set, the central claim that the LLM can predict policy outcomes fails for real-world use.","tokens_in":12473,"feed_emoji":"🥗","tokens_out":8537,"duration_ms":72957,"temperature":0.7,"pith_summary":"The paper asks whether a large language model can forecast whether a food-policy intervention will move behavior in the intended direction. The authors fine-tune GPT-3.5 Turbo on 208 effect sizes extracted from 74 published experiments and report that the best prompts predict effect direction correctly on 79% of 41 held-out studies, along with approximate correlation coefficients and Cohen's d. They also find that short prompts with a direct completion request work best, that about 75 training prompts are optimal, and that accuracy collapses when too many prompts are added. A validation on 12 preregistered unpublished experiments drops direction accuracy to 55%, slightly above the 50% naive baseline. The contribution is a proof-of-concept that LLM-based screening could make evidence-based food policy cheaper and faster, if out-of-sample accuracy can be improved.","feed_headline":"Fine-tuned LLM predicts food-policy effects 79% of the time","feed_subtitle":"Trained on 74 food-behavior experiments, the tool shows prompt design and data mix decide how well it works.","key_machinery":"The central machinery is a fine-tuned GPT-3.5 Turbo model, prompted with structured descriptions of experiments. The prompt template fills in features—intervention type, location, target population, sample size, treatment and control group sizes, and the paper title—and asks the model to output the direction of the effect, the Pearson correlation r, and Cohen's d. Four prompt variants (P1–P4) differ in verbosity and the presence of a guided completion sentence; the concise variants P3 and P4 yield complete numerical responses, while the verbose variants often refuse or give poor predictions. The learned object is the function f:X→Y from experiment descriptions to effect parameters, trained on 144 prompts with 23 validation and 41 test prompts, plus a 12-study preregistered set for out-of-sample checks.","core_discovery":"The paper's central claim is that a fine-tuned LLM can learn a mapping from descriptions of food-policy experiments to their quantitative outcomes. On the best prompt templates (MP3 and MP4), the model predicts whether an intervention's effect is positive or negative with 79% accuracy on 41 held-out effect sizes from the same 74-paper corpus, and it estimates the correlation coefficient and Cohen's d with small average absolute errors. This performance is sensitive to prompt style: concise prompts with a direct request to predict outperform verbose step-by-step prompts, which fall to 36.7% and 23.1% direction accuracy. The paper also reports that roughly 75 training prompts are optimal, that accuracy declines when more prompts are added, and that on 12 genuinely unseen preregistered studies direction accuracy drops to 55%. The authors read these results as proof-of-concept that LLM-based decision support could help policymakers screen behavioral food-policy interventions.","pith_inferences":["A fair benchmark for this approach would compare it against human policy experts or conventional meta-regression on the same held-out and unseen studies; the paper only compares against a naive majority-class predictor, so the practical value added is not yet established.","Because the largest errors on unseen studies came from monetary-incentive interventions, which make up only 5% of the training data, deliberately oversampling rare intervention types is likely to improve out-of-sample accuracy more than adding further common examples.","The non-monotonic accuracy with training size (75 and 130 prompts) may reflect the API's automatic hyperparameter choices rather than an intrinsic property of fine-tuning; repeating the curve with controlled epochs and learning rates would show whether the pattern is real.","A natural next test is to run the same fine-tuning pipeline on road-safety, smoking, or energy-conservation interventions; the paper leaves this as speculation, but the method's architecture makes it directly testable."],"forward_implications":["A policy team could use the fine-tuned model as a screening step, ranking candidate interventions by predicted direction before committing to expensive field trials.","Prompt formulation is not a neutral detail: concise prompts with a direct completion request gave 100% response coverage and 79% accuracy, while verbose step-by-step prompts produced much lower direction accuracy, so prompt tuning materially determines whether the tool works.","Training on more examples is not always better: models fine-tuned on 75 and 130 prompts outperformed those trained on intermediate or larger sets, with accuracy declining beyond the optimum.","The 55% accuracy on unseen preregistered studies shows the model's predictive power is largely tied to the distribution of the training corpus, so representative training data is a precondition for practical use.","The same fine-tuning pipeline could, in principle, be transferred to other behaviorally informed policy areas such as road safety, smoking, alcohol abuse, and program take-up, though the paper does not test those domains."],"supporting_citations":[{"why":"It supplies the systematic review and dataset of 74 published food-policy experiments with 208 effect sizes that the model is fine-tuned on.","marker":"Lohmann et al., 2024b"},{"why":"It establishes few-shot learning in large language models, the behaviour the fine-tuning procedure relies on.","marker":"Brown et al., 2020"},{"why":"It provides the fine-tuning method used to adapt the pre-trained model to the prediction task.","marker":"Howard & Ruder, 2018"},{"why":"It provides the standard formulae used to convert reported effects into r and Cohen's d.","marker":"Ringquist (2013)"},{"why":"It motivates the paper's prompt-sensitivity experiments by documenting how prompt formatting changes LLM outputs.","marker":"Sclar et al., 2023"},{"why":"It supports the interpretation of performance loss from adding training prompts as catastrophic forgetting.","marker":"Luo et al., 2023"},{"why":"It sources the 12 preregistered unpublished experiments used to test generalization to genuinely unseen studies.","marker":"Pizzo et al., 2024; Lohmann et al., 2024a"}],"fun_headline_variants":["LLM anticipates food-policy study outcomes with 79% accuracy","Food-policy LLM predicts outcomes 79% on known, 55% on novel","LLM screens food policies: 79% right on trained, 55% on new","Prompt design decides if LLM predicts food-policy results well"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand only if the 41 held-out effect sizes drawn from the same 74-paper corpus represent the distribution of real-world policy contexts in which the tool would be used, and the drop to 55% accuracy on 12 unseen preregistered studies shows how much the conclusion depends on that assumption.","fun_headline_variants_meta":{"raw":{"variants":["LLM anticipates food-policy study outcomes with 79% accuracy","Food-policy LLM predicts outcomes 79% on known, 55% on novel","LLM screens food policies: 79% right on trained, 55% on new","Prompt design decides if LLM predicts food-policy results well"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000543,"raw_usage":{"total_tokens":2590,"prompt_tokens":928,"completion_tokens":1662,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":1587}},"tokens_in":544,"tokens_out":1662,"duration_ms":11321,"temperature":1.0,"reasoning_tokens":1587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:30:59.214738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the best prompt version (P4, 130 training prompts) on a preregistered set of 50 new food-policy experiments published after the model's training cutoff and drawn from multiple countries, populations, and intervention types. If direction accuracy is not clearly above the 50% naive baseline on that set, the central claim that the LLM can predict policy outcomes fails for real-world use.","supporting_citations":[{"cited_title":"What shapes sustainable food choices? a field ex- periment on the impact of a behaviorally informed intervention and a price variation on sustainable food choices","cited_arxiv_id":null,"evidence_quote":"It sources the 12 preregistered unpublished experiments used to test generalization to genuinely unseen studies."}],"review_version":1}