REVIEW 4 major objections 5 minor 18 references
Leveraging LLMs for Predictive Insights in Food Policy and Behavioral Interventions
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A fine-tuned GPT-3.5 Turbo model predicts the direction of dietary-behavior intervention effects correctly in about 79% of held-out studies, suggesting LLMs can serve as low-cost screens for food-policy decisions.
desk verdict A genuinely new application with an honest external check, but the 79% headline is likely leakage-inflated; the clean generalization number is 55%. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a fine-tuned GPT-3.5 Turbo model, prompted with structured descriptions of experiments. The prompt template fills in features—intervention type, location, target population, sample size, treatment and control group sizes, and the paper title—and asks the model to output the direction of the effect, the Pearson correlation r, and Cohen's d. Four prompt variants (P1–P4) differ in verbosity and the presence of a guided completion sentence; the concise variants P3 and P4 yield complete numerical responses, while the verbose variants often refuse or give poor predictions. The learned object is the function f:X→Y from experiment descriptions to effect parameters, trained on 144 prompts with 23 validation and 41 test prompts, plus a 12-study preregistered set for out-of-sample checks.
What would settle it
Run the best prompt version (P4, 130 training prompts) on a preregistered set of 50 new food-policy experiments published after the model's training cutoff and drawn from multiple countries, populations, and intervention types. If direction accuracy is not clearly above the 50% naive baseline on that set, the central claim that the LLM can predict policy outcomes fails for real-world use.
Extended reading notes
Core claim
The paper's central claim is that a fine-tuned LLM can learn a mapping from descriptions of food-policy experiments to their quantitative outcomes. On the best prompt templates (MP3 and MP4), the model predicts whether an intervention's effect is positive or negative with 79% accuracy on 41 held-out effect sizes from the same 74-paper corpus, and it estimates the correlation coefficient and Cohen's d with small average absolute errors. This performance is sensitive to prompt style: concise prompts with a direct request to predict outperform verbose step-by-step prompts, which fall to 36.7% and 23.1% direction accuracy. The paper also reports that roughly 75 training prompts are optimal, that accuracy declines when more prompts are added, and that on 12 genuinely unseen preregistered studies direction accuracy drops to 55%. The authors read these results as proof-of-concept that LLM-based decision support could help policymakers screen behavioral food-policy interventions.
Load-bearing premise
The results stand only if the 41 held-out effect sizes drawn from the same 74-paper corpus represent the distribution of real-world policy contexts in which the tool would be used, and the drop to 55% accuracy on 12 unseen preregistered studies shows how much the conclusion depends on that assumption.
Editorial extensions
If this is right
- A policy team could use the fine-tuned model as a screening step, ranking candidate interventions by predicted direction before committing to expensive field trials.
- Prompt formulation is not a neutral detail: concise prompts with a direct completion request gave 100% response coverage and 79% accuracy, while verbose step-by-step prompts produced much lower direction accuracy, so prompt tuning materially determines whether the tool works.
- Training on more examples is not always better: models fine-tuned on 75 and 130 prompts outperformed those trained on intermediate or larger sets, with accuracy declining beyond the optimum.
- The 55% accuracy on unseen preregistered studies shows the model's predictive power is largely tied to the distribution of the training corpus, so representative training data is a precondition for practical use.
- The same fine-tuning pipeline could, in principle, be transferred to other behaviorally informed policy areas such as road safety, smoking, alcohol abuse, and program take-up, though the paper does not test those domains.
Reading between the lines
- A fair benchmark for this approach would compare it against human policy experts or conventional meta-regression on the same held-out and unseen studies; the paper only compares against a naive majority-class predictor, so the practical value added is not yet established.
- Because the largest errors on unseen studies came from monetary-incentive interventions, which make up only 5% of the training data, deliberately oversampling rare intervention types is likely to improve out-of-sample accuracy more than adding further common examples.
- The non-monotonic accuracy with training size (75 and 130 prompts) may reflect the API's automatic hyperparameter choices rather than an intrinsic property of fine-tuning; repeating the curve with controlled epochs and learning rates would show whether the pattern is real.
- A natural next test is to run the same fine-tuning pipeline on road-safety, smoking, or energy-conservation interventions; the paper leaves this as speculation, but the method's architecture makes it directly testable.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PREDICT, a proof-of-concept in which a fine-tuned GPT-3.5 Turbo model is trained on prompts derived from 74 published food-policy and behavioral-intervention studies (208 effect sizes) and evaluated on held-out prompts. The authors report 79% accuracy in predicting effect direction on a held-out set of 41 prompts, with additional analyses of prompt design, training-set size, and feature removal. They also test the model on 12 preregistered, unpublished studies and report 55% direction accuracy. The paper argues that fine-tuned LLMs can serve as decision-support tools for predicting the direction of behavioral food-policy outcomes, while acknowledging limitations in generalization.
Significance. If the central claim were fully supported, the paper would make a useful methodological contribution: a low-cost, LLM-based screening tool for policy interventions, particularly in food policy, with implications for other behavioral-policy domains. The authors deserve credit for including a preregistered unseen-study validation set, for varying training size and prompt features, and for explicitly discussing external validity and limitations. However, the strength of the contribution currently rests on two load-bearing pillars that are not yet established: the held-out 79% accuracy is potentially inflated by prompt-level leakage, and the paper's only clean generalization test gives 55%, which is at chance. The manuscript also mislabels its headline quantitative error metric, and several claims about optimal training size rest on single-run observations without uncertainty quantification.
major comments (4)
- [Section 4.3] The train/validation/test split is described at the level of 208 prompts (144/23/41), but it is not stated whether all prompts originating from the same research paper are kept in the same partition. With 208 effect sizes drawn from 74 papers, a random prompt-level split will, with high probability, place some effect sizes from a given paper in training and others in the test set. Because each prompt contains the article title, location, population, and sample size, the model could memorize study-specific outcome directions during fine-tuning and reproduce them for test prompts from the same study. This would make the reported 79% direction accuracy an artifact of within-corpus memorization rather than a measure of prediction for new studies. The authors must report whether any paper appears in more than one partition, and ideally re-run the evaluation with a study-level split; if the overlap exists, the 79% figure cannot support the abstract's claim about predicting outcomes of empirical studies.
- [Section 2.1, Table 1, and Section 4.5] The metric labeled 'average absolute error' in Table 1 is not an absolute error. Section 4.5 defines the error as |predicted| - |actual|, which is a signed difference of absolute values, and Table 1 reports negative values such as -0.058 and -0.009. A true mean absolute error cannot be negative. Because the paper uses negative values to mean 'underestimation', the reported quantity is a signed bias, not an absolute error. This mislabeling affects all quantitative accuracy claims for r and Cohen's d, including the comparisons between models MP3 and MP4 and the feature-removal analysis in Section 2.3. The authors should report mean absolute error and signed bias separately, with definitions matching the names used.
- [Section 2.4] The unseen-study validation, which is the only test that rules out leakage, yields 55% direction accuracy on 12 preregistered studies. With 12 binary outcomes, chance-level performance is 50%, and the reported 55% is statistically indistinguishable from chance without a confidence interval or a hypothesis test. The paper acknowledges this result but then performs a post-hoc exclusion of two studies involving monetary incentives, reporting improved quantitative errors after seeing the outcomes. This is a selection-bias problem: the exclusion rule is not prespecified and appears to be motivated by the observed poor performance. The authors should report the preregistered analysis as primary, provide a confidence interval for the 55% estimate, and, if they retain the exclusion, justify it with a principled criterion defined before inspecting the results. As written, the abstract's 'approximately 80%' claim is inconsistent with the paper's own clean generalization evidence.
- [Section 2.2 and Figure 2] The claim that performance is optimal at approximately 75 training prompts, with a non-monotonic pattern and 'catastrophic forgetting' beyond that point, is not supported by the presented evidence. The paper reports a single trajectory over training sizes N = 10, 75, 130, 144, and 167, with no confidence intervals, no repeated runs, and no statistical comparison. Given the small numbers of test prompts, the observed non-monotonicity could easily be sampling noise. The authors should either provide uncertainty estimates and repeated fine-tuning runs, or substantially soften the causal language about catastrophic forgetting and the existence of an optimal training size.
minor comments (5)
- [Abstract and Section 2.2] The abstract states that 'approximately 75 prompts were required to achieve optimal results', but Section 2.2 reports that both 75 and 130 prompts are best-performing; Section 2.4 then refers to the model fine-tuned with 130 prompts as the best-performing model. This inconsistency should be resolved.
- [Section 2.4] The text reports that the model 'correctly predicts the effect direction in approximately 6.6 out of 12 experiments'. If this is an average over 10 inference runs, it should be stated clearly; otherwise the number of correct predictions should be an integer. The phrase '6.6 out of 12' is confusing as written.
- [Section 4.5] The inference procedure should describe how non-numeric model outputs are handled. Models MP1 and MP2 have prediction probabilities below 100% for r and d; if some responses are unparseable, the accuracy and error calculations should clarify whether these are excluded or counted as errors.
- [Section 2.1 and Table 1] The terms 'average absolute error' and 'variance in absolute errors' are used inconsistently. Table 1 gives a single variance value associated with each signed error, but it is not clear whether the variance is computed over the 10 repeated LLM runs or over the 41 test prompts. The text in footnote 1 suggests both, which is ambiguous.
- [Section 2.3] The feature-removal analysis compares models after removing one feature at a time, but the paper does not report whether the same train/test split and the same repeated-run protocol are used for every removed-feature model. This should be stated to allow the comparisons in Figure 3 to be interpreted.
Circularity Check
No significant circularity: the 79% figure is a genuine held-out supervised-learning result; self-citations are not load-bearing for the prediction claim.
full rationale
The paper's derivation chain is an empirical supervised-learning pipeline: 208 effect-size prompts are extracted from 74 papers, 144 are fine-tuned with known outcome labels, and 41 held-out prompts are scored for direction accuracy (79% for MP3/MP4). This is a standard out-of-sample evaluation, not an analytic identity: nothing in the fine-tuning loss or prompt template forces the test-set accuracy to equal the training labels by construction. The self-citations (Lohmann et al. 2024b for the systematic search and inclusion criteria; Pizzo et al. 2024 and Lohmann et al. 2024a for the 12 preregistered unpublished validation studies) are not used to derive the reported accuracy; the predictions are computed from model outputs on data not used in fine-tuning. The 12-study check is self-cited but preregistered and non-overlapping with the training corpus, and the paper honestly reports the resulting 55% accuracy as a limitation rather than using it to inflate the headline. A possible validity concern is that Section 4.3 does not explicitly state that the 144/23/41 split is study-disjoint, so prompt-level leakage could inflate the in-corpus 79%; however, this is a data-splitting correctness risk, not circularity, and the paper's separate out-of-corpus test partially mitigates it. With no load-bearing self-citation chain and no fitted parameter renamed as a prediction, the central claim retains independent empirical content. Score 2 reflects only the presence of minor, non-load-bearing self-citations.
Assumptions & free parameters
free parameters (3)
- Number of training prompts (N) =
75 and 130 (also tested 10, 144, 167)
- Prompt template version =
P4 (concise, with guided completion)
- Unseen-test exclusion rule =
2 monetary-incentive studies excluded
assumptions (5)
- standard math Ringquist (2013) conversion formulas between reported effect sizes and r/Cohen's d are correct and uniformly applied.
- domain assumption The manually extracted features (title, location, year, population, sample sizes) capture the information needed to predict intervention outcomes.
- domain assumption The 208 effect sizes from 74 papers are treated as independent units for random splitting into train/validation/test.
- ad hoc to paper The non-monotonic performance pattern across training counts (10, 75, 130, 144, 167) is due to catastrophic forgetting rather than sampling noise.
- domain assumption The 12 preregistered ongoing experiments are correctly matched to their prompts and their results accurately extracted.
Cite this review
Pith. "Pith review of Leveraging LLMs for Predictive Insights in Food Policy and Behavioral Interventions." pith.science (2026). https://pith.science/paper/ED277QMN
@misc{pith2026241108563,
author = {Pith},
title = {Pith review of: Leveraging LLMs for Predictive Insights in Food Policy and Behavioral Interventions},
year = {2026},
howpublished = {\url{https://pith.science/paper/ED277QMN}},
note = {Machine review of arXiv:2411.08563}
}
read the original abstract
Food consumption and production contribute significantly to global greenhouse gas emissions, making them crucial entry points for mitigating climate change and maintaining a liveable planet. Over the past two decades, food policy initiatives have explored interventions to reshape production and consumption patterns, focusing on reducing food waste and curbing ruminant meat consumption. While the evidence of "what works" improves, evaluating which policies are appropriate and effective in specific contexts remains difficult due to external validity challenges. This paper demonstrates that a fine-tuned large language model (LLM) can accurately predict the direction of outcomes in approximately 80\% of empirical studies measuring dietary-based impacts (e.g. food choices, sales, waste) resulting from behavioral interventions and policies. Approximately 75 prompts were required to achieve optimal results, with performance showing signs of catastrophic loss beyond this point. Our findings indicate that greater input detail enhances predictive accuracy, although the model still faces challenges with unseen studies, underscoring the importance of a representative training sample. As LLMs continue to improve and diversify, they hold promise for advancing data-driven, evidence-based policymaking.
Figures
Reference graph
Works this paper leans on
-
[3]
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,
1901
-
[6]
What do llms know about financial markets? a case study on reddit market sentiment analysis
Xiang Deng, Vasilisa Bashlovkina, Feng Han, Simon Baumgartner, and Michael Bendersky. What do llms know about financial markets? a case study on reddit market sentiment analysis. InCompanion Proceed- ings of the ACM Web Conference 2023, pp. 107–110,
work page 2023
-
[9]
Prefix-tuning: Optimizing continuous prompts for generation.arXiv preprint arXiv:2101.00190,
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation.arXiv preprint arXiv:2101.00190,
-
[10]
Choice architecture promotes sustainable choices in online food-delivery apps.PNAS Nexus, pp
Paul M Lohmann, Elisabeth Gsottbauer, James Farrington, Steve Human, and Lucia A Reisch. Choice architecture promotes sustainable choices in online food-delivery apps.PNAS Nexus, pp. pgae422, 2024a. Paul M Lohmann, Alice Pizzo, Jan Michael Bauer, Tarun M Khanna, and Lucia A Reisch. Demand-side interventions for sustainable food systems: A meta-analysis of...
-
[11]
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning.arXiv preprint arXiv:2308.08747,
-
[12]
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation.arXiv preprint arXiv:2305.16938,
-
[13]
Keiichi Namikoshi, Alex Filipowicz, David A Shamma, Rumen Iliev, Candice L Hogan, and Nikos Arechiga. Using llms to model the beliefs and preferences of targeted populations.arXiv preprint arXiv:2403.20252,
-
[14]
Language models as knowledge bases?arXiv preprint arXiv:1909.01066,
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases?arXiv preprint arXiv:1909.01066,
arXiv 1909
Show all 18 references
-
[17]
Fine tuning llm for enterprise: Practical guidelines and recommendations
Kushala VM, Harikrishna Warrier, Yogesh Gupta, et al. Fine tuning llm for enterprise: Practical guidelines and recommendations. arXiv preprint arXiv:2404.10779,
-
[18]
Bloomberggpt: A large language model for finance
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564,
-
[19]
Pixiu: A large language model, instruction data and evaluation benchmark for finance.arXiv preprint arXiv:2306.05443,
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. Pixiu: A large language model, instruction data and evaluation benchmark for finance.arXiv preprint arXiv:2306.05443,
-
[2013]
Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.arXiv preprint arXiv:2310.11324,
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.arXiv preprint arXiv:2310.11324,
-
[2018]
Brian Lester, Rami Al-Rfou, and Noah Constant
URL http://arxiv.org/abs/1801.06146. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691,
-
[2019]
What shapes sustainable food choices? a field ex- periment on the impact of a behaviorally informed intervention and a price variation on sustainable food choices
Alice Pizzo, Jan Michael Bauer, and Lucia A Reisch. What shapes sustainable food choices? a field ex- periment on the impact of a behaviorally informed intervention and a price variation on sustainable food choices. A Field Experiment on the Impact of a Behaviorally Informed I...
2024
-
[2020]
Evaluating the replicability of social science experiments in nature and science between 2010 and 2015.Nature human behaviour, 2(9): 637–644,
Colin F Camerer, Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A Nosek, Thomas Pfeiffer, et al. Evaluating the replicability of social science experiments in nature and science between 2010 and 2015.Nature h...
2010
-
[2021]
Template-based named entity recognition using bart
Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. Template-based named entity recognition using bart. arXiv preprint arXiv:2106.01760,
-
[2023]
Large language models for mathematical reasoning: Progresses and challenges.arXiv preprint arXiv:2402.00157,
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges.arXiv preprint arXiv:2402.00157,
-
[2024]
The impact of large language models on scientific discovery: a preliminary study using gpt-4.arXiv preprint arXiv:2311.07361,
Microsoft Research AI4Science and Microsoft Azure Quantum. The impact of large language models on scientific discovery: a preliminary study using gpt-4.arXiv preprint arXiv:2311.07361,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.