The Digital Sous Chef -- A Comparative Study on Fine-Tuning Language Models for Recipe Generation

Ganesh Bagler; Shubham Pundhir

REVIEW 5 major objections 5 minor 2 references

The Digital Sous Chef -- A Comparative Study on Fine-Tuning Language Models for Recipe Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning GPT-2 large with a fraction-aware tokenizer outperforms recurrent baselines on a 51,000-recipe benchmark.

desk verdict A standard GPT-2-vs-recurrent benchmark whose stated key contribution, fraction-aware tokenization, is never isolated; the abstract also overstates the gain by comparing against the wrong baseline. read the letter →

arxiv 2508.14718 v1 pith:FDGUJTSX submitted 2025-08-20 cs.CL

Shubham Pundhir , Ganesh Bagler This is my paper · ORCID

classification cs.CL

keywords recipegenerationGPT-2fine-tuningdomain-specifictokenizationfractiontokensBERTScorenaturallanguageDBbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that fine-tuning a large pre-trained transformer, GPT-2 large (774M parameters), with a domain-specific tokenizer produces better recipe text than traditional LSTM/RNN baselines. The authors build a reproducible benchmark on the 5-cuisine RecipeDB corpus, comparing two GPT-2 sizes against word-level recurrent models across seven automatic metrics. Their headline result is a BERTScore F1 of 0.92 versus 0.72 for the best recurrent baseline, with a 69.8% reduction in perplexity. The paper argues that preserving fraction tokens and structural boundary markers is a key part of this success, while also acknowledging that factual accuracy in generated recipes remains unsolved.

What carries the argument

The central mechanism is the custom tokenizer: GPT-2's byte-level BPE vocabulary is augmented with 23 fraction tokens and structural boundary tokens such as <INGR_START>, <NEXT_INGR>, and <INSTR_END>, and the model's embedding matrix is resized to accommodate them. This preserves exact fractional quantities and keeps recipe sections as single semantic units, which is what the paper claims improves numerical fidelity and structural coherence in generation.

What would settle it

Fine-tune GPT-2 large on the same 5-cuisine split using the stock GPT-2 tokenizer with identical data, sampling, and training settings; if BERTScore F1 and perplexity remain near 0.92 and 20.67, the custom tokenization is not the cause. A more direct test would measure exact reproduction rates of fraction tokens in generated text.

Watch

Extended reading notes

Core claim

On the 5-cuisine benchmark, a fine-tuned GPT-2 large model with an augmented tokenizer achieves the best reported recipe generation quality: BERTScore F1 0.92, BLEU-4 0.26, ROUGE-L F1 0.54, METEOR 0.47, diversity 0.73, and perplexity 20.67, beating GPT-2 small and both LSTM and RNN baselines on every metric. The authors attribute this to model scale plus a tokenization scheme that adds 23 common fraction tokens and custom structural markers, preventing numerical quantities and recipe-section boundaries from being fragmented by byte-level BPE. They frame this as a new state-of-the-art for the 5-cuisine recipe generation task.

Load-bearing premise

The paper assumes the custom fraction tokens and structural markers are what drive the quality gains, but it never tests that in isolation: every GPT-2 model uses them and every recurrent baseline does not, so tokenization is entangled with model scale and architecture.

Editorial extensions

If this is right

Domain-specific tokenization becomes a cheap, reusable lever for structured generation tasks where numbers and format matter.
A 774M-parameter GPT-2 model is sufficient to set a strong benchmark for recipe generation, suggesting that larger general-purpose LLMs are not required for this task.
The seven-metric evaluation suite gives later work a standard way to compare fluency, coherence, semantic relevance, and diversity in recipe generation.
The demonstrated weaknesses in factual accuracy point directly to retrieval-augmented generation and constrained decoding as the next necessary steps.
A fine-tuned text-generation engine of this kind can serve as the backbone for image-to-recipe and recipe-to-image multi-modal systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper never tests the tokenizer in isolation: since all GPT-2 runs use the custom tokens and all recurrent baselines use standard word-level tokens, the reported gains could come mostly from model scale and architecture rather than fraction-aware tokenization.
A direct, testable extension would be to measure exact fraction reproduction (e.g., what fraction of '1/2', '1/3', '2/3' tokens appear correctly in generated text) rather than relying on aggregate BERTScore.
If tokenization is the true driver, applying the same augmented tokenizer to the LSTM/RNN baselines should narrow the gap substantially; if it does not, the paper's stated contribution is not load-bearing.
The benchmark's usefulness would increase if future work reports variance across multiple random seeds, since the current table does not indicate whether the margins between GPT-2 small and large are stable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a fine-tuning comparison for recipe generation on the 5-cuisine corpus. The headline numbers are plausible: GPT-2 large beats GPT-2 small and the LSTM/RNN baselines on all seven metrics. What's actually new is the fraction-aware tokenizer with 23 fraction tokens and structural markers. That idea is sensible for a domain where "1/2 cup" and section boundaries matter. The paper is also honest in its limitations section: it flags factual inaccuracy, ingredient hallucination, repetition, and the need for human evaluation. Credit for that.

The soft spots are structural. The tokenizer is never ablated. All GPT-2 runs use the custom tokenizer, and all recurrent baselines use a word-level tokenizer. So you cannot attribute the gains to tokenization; it is confounded with architecture, scale, and pretraining. Section 5.2 says the custom tokens "proved highly effective," but no experiment isolates the tokenizer as the only variable. A simple ablation—GPT-2 with and without fraction tokens, or the custom tokenizer applied to the LSTM—would fix this.

There is also a factual error in the abstract: the ">20% relative improvement in BERTScore (0.92 vs 0.72) over the best recurrent baseline" is wrong. 0.72 is the RNN baseline; the LSTM is 0.87. The actual relative improvement over the best recurrent baseline is about 5.7%. That misstates the contribution's magnitude. Also no error bars, no seeds, no significance tests, so even the corrected gap may be noise. The paper says code is on GitHub, but I don't see a link; without code and data, reproducibility suffers.

The tokenization idea is worth taking seriously—it could transfer to other structured text with quantities. But as presented, the central claim is unsupported. The benchmark itself is routine: RecipeGPT and Ratatouille already fine-tuned GPT-2 for recipes. The paper does add a standardized evaluation on the 5-cuisine corpus with seven metrics, which would be more useful if the code and data were released. One oddity: the future work mentions "as noted by reviewers," which is an odd thing to have in an arXiv preprint, but it is not a substantive flaw.

Who is this for? People working on recipe generation or domain-specific tokenization. It could be a workshop or short-paper submission after major revision. It deserves peer review because the claim is testable and an ablation could make it acceptable. My recommendation: send it out, but the referee should demand an ablation and a corrected abstract. If the authors cannot show the tokenizer matters, the paper reduces to a routine benchmark.

Referee Report

5 major / 5 minor

Summary. The paper compares fine-tuned GPT-2 small and large models against word-level LSTM and RNN baselines for recipe generation on the 5-cuisine RecipeDB corpus. The authors state their key contribution as a domain-specific tokenization strategy that adds 23 fraction tokens and custom structural markers, and they report that GPT-2 large achieves the best automatic metric scores, including a BERTScore F1 of 0.92 and perplexity of 20.67. The paper also argues that this tokenization strategy is responsible for the improvements and claims a new state of the art for the benchmark.

Significance. If the central claim were supported, the paper would provide a useful benchmark and a practical tokenization recipe for structured, number-heavy generation tasks. The release of code, the use of seven automatic metrics, and the comparison across two GPT-2 scales and recurrent baselines are positive elements. However, the central claim that the custom tokenization is responsible for the gains is not supported by the experimental design: the tokenizer is varied together with architecture, scale, and pretraining, and no ablation isolates it. The paper also contains a quantitative misstatement in the abstract and a vocabulary-size inconsistency that casts doubt on the implementation details. The benchmark itself is useful, but the headline claims need substantial revision and additional experiments.

major comments (5)

[§3.2.2, §5.2, Table 4] The paper's key contribution is the targeted tokenization strategy, but no experiment isolates its effect. GPT-2 models use the custom tokenizer introduced in §3.2.2, while the LSTM/RNN baselines use a word-level tokenizer (§3.1). Thus the better scores of GPT-2 large in Table 4 could be due to architecture, scale, pretraining, or tokenization. Section 5.2 asserts that the custom tokens 'proved highly effective' with no control condition, e.g., GPT-2 large with and without the augmented fractions and structural markers. Without such an ablation, the central claim is not established.
[Abstract, Table 4, Table 1] The abstract claims a '>20% relative improvement in BERTScore (F1) (0.92 vs 0.72) over the best recurrent baseline.' In Table 4 the 0.92 vs 0.72 comparison is against the RNN baseline, not the best recurrent baseline; the LSTM baseline has BERTScore 0.87, so the relative improvement is (0.92−0.87)/0.87 ≈ 5.7%, not over 20%. The paper also states that GPT-2 large 'sets a new state-of-the-art' for the 5-cuisine benchmark, but Table 1 reports prior RecipeGPT and Ratatouille results without a direct, same-protocol comparison on this benchmark. The SOTA claim is therefore unsupported.
[§4.2, Table 4] Perplexity is reported for models with different tokenizers: GPT-2 uses a byte-level BPE augmented with custom tokens, while the recurrent baselines use a word-level vocabulary. Perplexity is not comparable across different tokenizations because the probability mass is distributed over different token units. Therefore the '69.8% perplexity reduction' claimed in the abstract is not a meaningful head-to-head comparison. A valid comparison would require either a shared tokenizer or a text-level perplexity estimate.
[Table 3, §3.2.2] Table 3 reports the vocabulary size as '∼50,257' (Base + Special). This is exactly the standard GPT-2 vocabulary size (50,257). If the paper really added 23 fraction tokens, custom structural markers, and a [PAD] token, the vocabulary size should be larger. Either the tokens were not actually added, the table is wrong, or the base vocabulary was replaced. This inconsistency directly affects the credibility of the tokenization contribution and needs to be resolved with the actual tokenizer configuration.
[§4.1, Table 4] All results in Table 4 are point estimates with no measures of variance. There is no report of multiple random seeds, no error bars, and no significance testing. Given that the differences between GPT-2 small and large are small (e.g., BERTScore 0.90 vs 0.92), and the sampling-based generation in §3.2.4 introduces stochasticity, the results may not be stable. The paper should report multiple seeds with standard deviations or confidence intervals, and use appropriate significance tests for the headline comparisons.

minor comments (5)

[Table 1] Table 1 lists BLEU-4 values of 0.085 and 0.080 with the column header 'BLEU-4 (%)'. If these are percentages, they are incompatible with Table 4, where BLEU-4 values are reported as 0.08–0.26. Clarify the unit convention consistently.
[§3.2.3] The text says 'the ’gpt2(774 Million)’ variant'; this should be 'GPT-2 large' or 'gpt2-large' for consistency with the rest of the paper. Also check the formatting in the same sentence.
[Table 3] 'V ocabulary' is a typo for 'Vocabulary'.
[§6] The sentence 'as noted by reviewers' in the future-work paragraph is inappropriate for a submitted manuscript; remove or rephrase.
[§4.3] The interpretation of BERTScore as 'capturing underlying cooking concepts' is overstated; BERTScore measures embedding similarity and should not be equated with conceptual correctness. Consider softening the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all headline numbers are held-out test-set measurements; the tokenization attribution is confounded but not circular.

full rationale

The paper is an empirical benchmark study rather than a derivation chain, so the circularity patterns do not apply. The headline results (BERTScore 0.92, PPL 20.67, BLEU-4 0.26, etc.) are computed on a held-out test set (Section 4.1, Table 4) using standard external metrics (BLEU-4, ROUGE-L, METEOR, BERTScore, perplexity) that compare generated text to ground-truth references or measure the model's own test-set likelihood. No equation defines these reported numbers in terms of the model's inputs or fitted parameters; the custom tokenizer (Section 3.2.2) is an experimental design choice, not a quantity fitted to the evaluation targets. The paper's main weakness is that the claimed benefit of the tokenization strategy is not isolated by an ablation: all GPT-2 models use the custom tokenizer while the LSTM/RNN baselines use word-level tokenizers, so the observed gains cannot be attributed to tokenization alone. This is a confound and an unsupported causal claim, but it is not circularity: the reported metrics do not reduce by construction to the tokenizer configuration. Similarly, the abstract's '>20% relative improvement over the best recurrent baseline' uses 0.92 vs 0.72, whereas Table 4 shows the best recurrent baseline (LSTM) achieves 0.87; this is a quantitative misstatement worth flagging as a correctness issue, but it is not a self-referential or definitional argument. The paper does include self-citations (RecipeDB dataset from Batra et al. 2020 and Ratatouille from Goel et al. 2022, both involving an overlapping author), but these are used as data source and background prior work, not as load-bearing evidence for the central claim. There is no uniqueness theorem imported from the authors, no fitted parameter renamed as a prediction, and no ansatz smuggled in only through self-citation. The paper's own limitations section (Section 5.3) honestly acknowledges factual inaccuracy and other failures, which further shows the results are not constructed to match the claims. Overall, the central results are self-contained empirical measurements; the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The empirical comparison depends on standard ML training choices and two hand-selected design elements: sampling parameters and the fraction-token vocabulary. The central claim does not rest on fitted physical constants, so circularity is low. The tokenization choices are asserted rather than validated, which is the main risk.

free parameters (3)

Sampling hyperparameters = T=0.7, top-p=0.95, top-k=50
Chosen by hand; directly shapes generated text and all downstream metrics. No sensitivity analysis is provided.
23 fraction tokens = 23 common fractions (not enumerated)
Hand-selected vocabulary additions, presented as the key contribution but never specified or ablated.
Training hyperparameters = LR=3e-5, batch=8, epochs=20
Standard choices for GPT-2 fine-tuning; their effect on the comparison is not tested.

assumptions (4)

domain assumption The 5-cuisine dataset split (80/10/10) is unbiased and representative
Section 3.2.1; no seed or stratified split details are given.
domain assumption Automatic metrics (BLEU, ROUGE, METEOR, BERTScore, diversity, perplexity) are valid proxies for recipe quality
Section 4.2; the paper itself notes that low n-gram overlap can be desirable in creative generation, which weakens this assumption.
ad hoc to paper Splitting fractions with standard BPE is harmful to recipe generation
Section 3.2.2; motivates the key contribution but no experiment demonstrates this harm or the benefit of the fix.
domain assumption Pre-trained GPT-2 weights transfer to culinary text
Section 3.2; relies on prior work but is not re-validated here.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Digital Sous Chef -- A Comparative Study on Fine-Tuning Language Models for Recipe Generation." pith.science (2026). https://pith.science/paper/FDGUJTSX

@misc{pith2026250814718,
  author       = {Pith},
  title        = {Pith review of: The Digital Sous Chef -- A Comparative Study on Fine-Tuning Language Models for Recipe Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FDGUJTSX}},
  note         = {Machine review of arXiv:2508.14718}
}

read the original abstract

We established a rigorous benchmark for text-based recipe generation, a fundamental task in natural language generation. We present a comprehensive comparative study contrasting a fine-tuned GPT-2 large (774M) model against the GPT-2 small (124M) model and traditional LSTM/RNN baselines on the 5-cuisine corpus from RecipeDB. Our key contribution is a targeted tokenization strategy that augments the vocabulary with 23 common fraction tokens and custom structural markers. This approach addresses a critical limitation of generic tokenizers by preserving essential recipe structures and precise numerical quantities, thereby enhancing domain specificity. Performance is evaluated using a comprehensive suite of seven automatic metrics spanning fluency (BLEU-4, METEOR), coherence (ROUGE-L), semantic relevance (BERTScore), and diversity. Our experiments show that the large transformer-based approach yields a >20% relative improvement in BERTScore (F1) (0.92 vs 0.72) over the best recurrent baseline, while reducing perplexity by 69.8%. We conclude with a discussion of remaining challenges, particularly regarding factual accuracy, and outline how this foundational study paves the way for integrating real-world constraints and multi-modal inputs in advanced recipe generation research.

Figures

Figures reproduced from arXiv: 2508.14718 by the authors.

**Figure 2.** High-level overview of the Transformer fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** A sample generation from the fine-tuned GPT-2 Large model, showing the input ingredient prompt, the generated step-by-step [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

[2019]

Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola

IEEE. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. A style-based generator for natural language generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 6423–6434, Online, 2020. Association for Computational Linguistics. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llio...

work page 2020
[6008]

Fine-tuning Language Models for Recipe Generation: A Comparative Analysis and Benchmark Study

Curran Associates, Inc., 2017. Anneketh Vij, Changhao Liu, Rahul Anil Nair, Theodore Eu- gene Ho, Edward Shi, and Ayan Bhowmick. Fine- tuning language models for recipe generation: A com- parative analysis and benchmark study. arXiv preprint arXiv:2502.02028, 2025. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi,...

work page Pith review arXiv 2017

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [2019]

Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola

IEEE. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. A style-based generator for natural language generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 6423–6434, Online, 2020. Association for Computational Linguistics. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llio...

work page 2020

[2] [6008]

Fine-tuning Language Models for Recipe Generation: A Comparative Analysis and Benchmark Study

Curran Associates, Inc., 2017. Anneketh Vij, Changhao Liu, Rahul Anil Nair, Theodore Eu- gene Ho, Edward Shi, and Ayan Bhowmick. Fine- tuning language models for recipe generation: A com- parative analysis and benchmark study. arXiv preprint arXiv:2502.02028, 2025. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi,...

work page Pith review arXiv 2017