Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Retrieval Augmented Recipe Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Retrieval-augmented recipes beat prior models on Recipe1M

desk verdict A plausible RAG-for-recipes paper with honest ablations but a likely training-time leakage issue and test-set-tuned SOTA numbers. read the letter →

arxiv 2411.08715 v2 pith:UZLBJQTV submitted 2024-11-13 cs.CV

Guoshan Liu , Hailong Yin , Bin Zhu , Jingjing Chen , Chong-Wah Ngo , Yu-Gang Jiang This is my paper · ORCID
classification cs.CV MSC 68T4568T50
keywords retrieval-augmentedgenerationrecipefoodcomputinglargemultimodalmodelshallucinationreductionself-consistencyvotingcross-modalretrievalRecipe1M
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented recipe generation proposes to fix the hallucination problem in large multimodal models when writing cooking instructions from food images. The paper builds a generator on frozen LLaVA with a trainable LoRA adapter, and at both training and inference time feeds it recipes retrieved from the Recipe1M training set that are semantically similar to the input image. A second component, self-consistency ensemble voting, generates several candidate recipes conditioned on different retrieved recipes and picks the one that agrees most with the others. On a fixed 5,000-image test subset of Recipe1M, the method reports state-of-the-art BLEU 30.11, SacreBLEU 6.42, and RougeL 38.93 for recipe generation, and F1 50.32 for ingredient recognition.

What carries the argument

The load-bearing mechanism has two parts. Stochastic Diversified Retrieval Augmentation (SDRA) builds a 'recipe demonstration' by randomly sampling multiple retrieved ingredient lists and one retrieved instruction list from the top-K retrieval results, giving the generator diverse but relevant context. Self-Consistency Ensemble Voting uses agreement among candidate recipes generated from different retrieved contexts as a confidence signal: the candidate with the highest average pairwise cosine similarity to the other candidates is chosen as the final output. The voting step is what converts the diversity created by SDRA into a single more reliable prediction.

What would settle it

Replace the retrieved recipes with random recipes from the Recipe1M training set while keeping the rest of the pipeline identical; if the gains over the fine-tuned LLaVA baseline shrink or reverse, the method's improvement depends on retrieval relevance rather than on the extra textual context or the voting mechanism alone.

Watch

Extended reading notes

Core claim

The paper's central claim is that retrieval augmentation can reduce hallucination in LMM-based recipe generation enough to beat all prior published systems on Recipe1M. The proposed architecture is an image-to-recipe retriever followed by a frozen LLaVA generator fine-tuned with LoRA. During training, Stochastic Diversified Retrieval Augmentation randomly samples three of the top-50 retrieved ingredient lists and one retrieved instruction list, concatenates them with fixed prompts telling the model that the search results are for reference only, and optimizes standard next-token prediction. During inference, Self-Consistency Ensemble Voting generates S candidate recipes by conditioning on the top-S retrieved recipes separately, computes pairwise cosine similarity among the candidates, and returns the candidate with the highest average agreement. The authors report that this combination outperforms two-stage methods such as InverseCooking, prior LMM-based methods such as FoodLMM, and a fine-tuned LLaVA baseline, and that ingredient recognition also improves without a dedicated recognition head.

Load-bearing premise

The central assumption is that the off-the-shelf retriever returns recipes that are semantically relevant to the input image; if retrieval is often off-target, the added context can inject wrong ingredients and make hallucinations worse, as the paper's own supplementary failure cases acknowledge.

Editorial extensions

If this is right

  • If the central claim is right, retrieval augmentation plus self-consistency voting is an effective hallucination remedy for LMMs in vertical generation tasks, not just for recipe generation.
  • The monotonic improvement with voting candidate count S up to 11 implies a simple compute-quality trade-off: using S around 5 already captures most of the gain, per the paper's own analysis.
  • Because the generator is frozen except for a LoRA adapter, the approach is a cheap way to specialise a large multimodal model to a new structured-output domain.
  • The method also lifts ingredient recognition metrics, suggesting that the retrieved ingredients act as a knowledge source that compensates for the model's limited food-domain vocabulary.
  • The fixed prompt that search results are 'only for referring' is a practical recipe for telling an LMM to use retrieval as context rather than as a literal answer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The voting mechanism is a form of unsupervised confidence estimation; if it generalizes, any multimodal task that can be conditioned on multiple retrieved or sampled contexts could use the same agreement-based selection without extra supervision.
  • The paper's own failure cases show that when retrieval is bad, it introduces wrong ingredients and steps; a natural test is to condition on retrieval quality per sample and measure how much the voting gain varies with it.
  • Automatic n-gram metrics may understate the true gain, since the paper notes cases where 'pasta' and 'spaghetti' are counted as mismatches; a human-evaluation study of recipe plausibility and hallucination rate would likely be more informative than the reported BLEU/SacreBLEU deltas.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a retrieval-augmented large multimodal model for generating recipes from food images. It uses an off-the-shelf cross-modal retriever (Salvador et al. [57]) over the Recipe1M training set as a datastore. During training, Stochastic Diversified Retrieval Augmentation (SDRA) randomly samples three retrieved ingredient sets and one retrieved instruction set to insert into the prompt; during inference, Self-Consistency Ensemble Voting generates S candidate instruction sets using the top-S retrieved recipes and selects the candidate with the highest average pairwise cosine similarity. Experiments on a fixed 5,000-sample random subset of Recipe1M report BLEU 30.11, SacreBLEU 6.42, and ROUGE-L 38.93 for instruction generation, and IOU 33.62 / F1 50.32 for ingredient recognition, which the paper claims are state-of-the-art. The paper also presents ablations for the retrieval pool size K, the voting number S, and the way retrieved information is concatenated, together with qualitative examples and supplementary analyses of retrieval successes and failures.

Significance. If the reported numbers were obtained under a clean protocol, the paper would make a useful contribution: it shows that retrieval augmentation with a frozen LLaVA backbone and LoRA tuning improves recipe generation metrics, and that multi-candidate self-consistency voting yields consistent improvements as S grows. The internal ablations in Tables 3 and 5 are a genuine strength, and the supplementary analysis of retrieval failures is unusually candid and helpful. However, the headline SOTA claim is currently not supported because the experimental protocol has two load-bearing weaknesses: the training-time retrieval may include the ground-truth recipe of the training image, and the final configuration (K=50, S=11) is selected on the same 5,000-sample test subset that is later reported as the result. The baseline comparisons in Table 1 are also not apples-to-apples. If these protocol issues are fixed, the method itself remains plausible and worth publishing.

major comments (4)
  1. [4.1.1, 4.1.3] The training-time retrieval setup does not rule out leakage of the ground-truth recipe. Section 4.1.1 states that the Recipe1M training set is used as the external datastore M, and Section 4.1.3 states that during training the top 50 recipes are retrieved for each image. Because the datastore contains the exact image-recipe pair used as the training query, and because the retriever [57] was trained to rank image-recipe matches, the ground-truth recipe of the training image is a plausible top-50 candidate. The paper never states that this recipe is excluded from the retrieved set. If it is not excluded, the model is trained with a near-answer copy in the prompt, which can inflate BLEU/SacreBLEU/ROUGE-L and reduce hallucination without any real retrieval benefit; at test time the ground truth is not in the datastore, so the reported SOTA gap would be an artifact of this asymmetry. Please report the fraction of training queries for which the ground-truth recipe appears in the top-50, and re-run training with the ground-truth recipe removed from the datastore for each query.
  2. [4.3.1, 4.3.2, Tables 3 and 5] The final configuration is selected on the same test subset used for the headline results. Section 4.1.1 fixes a 5,000-sample random subset of the test set for all experiments; Section 4.3.1 uses this subset to choose K=50 (Table 3), and Section 4.3.2 uses it to choose S=11 (Table 5), with the selection criterion being the same BLEU/SacreBLEU/ROUGE-L metrics that are later reported in Table 1. Thus the 'Ours' row is not an independent evaluation of a pre-specified model; it is a test-set-fitted value. A validation split should be used for model selection, and Table 1 should report results on an untouched test subset, or the paper should explicitly describe the numbers as test-set-selected and avoid claiming SOTA relative to baselines that did not receive this selection.
  3. [Table 1] The comparison with prior published systems is not apples-to-apples. The paper evaluates on a random 5,000-sample subset of the Recipe1M test set, while the baseline numbers in Table 1 are taken from prior papers that use the full test set or different splits and preprocessing (e.g., [11, 56, 65]). For example, under 'FIRE' BLEU is reported as '—' and SacreBLEU/ROUGE-L numbers may have been computed under different tokenization or ingredient-clustering settings. The only fully controlled baseline is LLaVA-FT, which the authors fine-tuned themselves. To support a SOTA claim, all compared methods should be re-evaluated on the same 5,000-sample subset with the same evaluation script, or the differences should be shown not to be an artifact of subset selection.
  4. [Section 4.2, Fig. 4] The paper's stated motivation is hallucination reduction, but no quantitative hallucination metric is reported. Section 4.2.3 and Figure 4 provide qualitative examples, yet BLEU/SacreBLEU/ROUGE-L measure surface overlap and can reward copying retrieved text rather than factual correctness. Please add an ingredient-level precision/recall or hallucination metric (e.g., fraction of predicted ingredients unsupported by the image or ground truth) and compare it directly against LLaVA-FT, with and without retrieval, so the hallucination-reduction claim can be verified.
minor comments (6)
  1. [Eq. (3)] Equation (3) has a duplicated conditioning term: pθ(yi|[X;R;y<i];θ) should be pθ(yi|[X;R;y<i]).
  2. [Section 3.3] The voting method in Equation (4) uses cosine similarity between generated recipes, but the paper never specifies how the recipe texts are vectorized (e.g., TF-IDF, sentence embeddings, or token overlap). Please state this explicitly, since it is essential for reproducibility.
  3. [Section 4.2.1] The text says 'relative improvement' of 2.25%, 0.18%, and 1.97% over FoodLMM, but the values are absolute differences (30.11 vs 27.86, etc.). Please use the correct term.
  4. [Section 4.3.1] The description of 'SDRA(fixed top 1)' is confusing: 'the top 2 retrieved ingredients sets and the top 3 ingredients along with top 1 instruction' is not clear about whether 'top 3 ingredients' means a third ingredient set or the top-3 ingredient items. Please rewrite.
  5. [Fig. 6] The caption says the confusion matrix is computed for 'top 11 retrieval-augmented prediction results', while the text refers to 'seven predictions'; please make the number of candidates consistent.
  6. [Section 5] The conclusion contains a typo: 'first retrieval augmented large multimodal mode' should be 'model'.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline SOTA metrics are partly selected on the test set; retrieval self-leak is an unaddressed confound but not needed for the score.

  1. fitted input called prediction [Section 4.1.1, Section 4.3.1 (Table 3), Section 4.3.2 (Table 5), Section 4.2.1 (Table 1)]
    "Generating long recipes word by word is time-consuming, so we randomly select 5,000 samples from test set and fix the seed for all experiments. ... The results displayed in Table 3 indicate that SDRA (top 50) performs the best ... We choose S=11 because the BLEU and SacreBLEU stopped improving at S=11."

    The same 5,000-sample test subset is used both to select the hyperparameters K=50 and S=11 and to produce the headline numbers. Table 3 chooses K by maximizing BLEU, SacreBLEU, and ROUGE-L on this subset; Table 5 chooses S by the same metrics on the same subset; Table 1 then reports the resulting 30.11 / 6.42 / 38.93 as state-of-the-art. The reported quantities are therefore the optimized objective of the hyperparameter search, not independent held-out predictions. The SOTA comparison is in part a test-set selection artifact.

full rationale

The architecture-level derivation is not circular: the generator is a frozen LLaVA with trained LoRA, the retriever [57] is external off-the-shelf, and the test-time voting selects among generated candidates without touching the ground truth. No load-bearing self-citation chain or imported uniqueness theorem is present; [70] is a baseline and [57] is external. The central circularity is in the evaluation protocol: K and S are chosen by maximizing the same metrics on the same 5,000-sample test set that is later reported as the SOTA result, so Table 1 partially reports fitted quantities. A separate validity concern, not counted as a derivation-level circular step, is that the Recipe1M training set is used as the retrieval datastore while each training image's own ground-truth recipe is a member of that datastore; the paper never states that this self-recipe is excluded from the top-50 training candidates. If it is retrieved, Equation 3 conditions on a prompt containing the target answer, which would confound the reported hallucination-reduction benefit. This should be ruled out in a revision, but unlike the K/S selection it depends on the retriever's ranking and is not circular by construction. Overall score 6 reflects the partial reduction of the headline numbers to test-set hyperparameter selection.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central performance claim rests on several unverified assumptions and on hyperparameters tuned on the test subset. The method introduces no new physical or conceptual entities.

free parameters (4)
  • K (retrieval pool size) = 50
    Ablated over 1, 10, 50, 100 on the test subset; 50 yields best BLEU in Table 3. This is a test-set fitted hyperparameter.
  • S (number of voting candidates) = 11
    Ablated from 1 to 11 in Table 5; chosen because BLEU and SacreBLEU stop improving at S=11 on the test subset.
  • Number of retrieved ingredient sets vs instruction sets = 3 ingredient sets, 1 instruction set
    Design choice in Section 3.2.1; Table 4 compares 1 set vs 2 sets but does not systematically vary the ratio, so the specific ratio is a free choice.
  • LoRA rank and alpha = not reported
    The paper does not specify LoRA rank, alpha, learning rate, or number of steps; these are unstated free parameters that affect the results.
assumptions (4)
  • domain assumption The pretrained cross-modal retriever [57] returns semantically relevant recipes for a given food image.
    The method relies on retrieval quality; Section 3.2.1 uses this retriever to obtain top-K ingredients and instructions. No retrieval accuracy numbers are reported.
  • domain assumption The Recipe1M training set can serve as a retrieval datastore without leaking test information.
    Section 4.1.1 states the training set is used as the external datastore M; the paper assumes train/test separation is clean.
  • domain assumption BLEU, SacreBLEU, RougeL, F1, and IOU adequately measure recipe quality and hallucination.
    Section 4.1.1 adopts these metrics from prior work; the paper claims hallucination reduction but does not use a hallucination-specific evaluation.
  • domain assumption The instruction 'The search results are only for referring, please focus on the image' prevents the model from copying erroneous retrieved content.
    This prompt is introduced in Section 3.2.1; the paper provides only qualitative evidence that the model balances image and retrieved context.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrieval Augmented Recipe Generation." pith.science (2026). https://pith.science/paper/UZLBJQTV

@misc{pith2026241108715,
  author       = {Pith},
  title        = {Pith review of: Retrieval Augmented Recipe Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UZLBJQTV}},
  note         = {Machine review of arXiv:2411.08715}
}
read the original abstract

Given the potential applications of generating recipes from food images, this area has garnered significant attention from researchers in recent years. Existing works for recipe generation primarily utilize a two-stage training method, first generating ingredients and then obtaining instructions from both the image and ingredients. Large Multi-modal Models (LMMs), which have achieved notable success across a variety of vision and language tasks, shed light to generating both ingredients and instructions directly from images. Nevertheless, LMMs still face the common issue of hallucinations during recipe generation, leading to suboptimal performance. To tackle this, we propose a retrieval augmented large multimodal model for recipe generation. We first introduce Stochastic Diversified Retrieval Augmentation (SDRA) to retrieve recipes semantically related to the image from an existing datastore as a supplement, integrating them into the prompt to add diverse and rich context to the input image. Additionally, Self-Consistency Ensemble Voting mechanism is proposed to determine the most confident prediction recipes as the final output. It calculates the consistency among generated recipe candidates, which use different retrieval recipes as context for generation. Extensive experiments validate the effectiveness of our proposed method, which demonstrates state-of-the-art (SOTA) performance in recipe generation tasks on the Recipe1M dataset.

Figures

Figures reproduced from arXiv: 2411.08715 by the authors.

Figure 1
Figure 1. (a) The structural differences between our retrieval-augmented framework and the “two-stage” [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Templates for Recipe Generation. 2.2. Vision-language Multimodal Models Due to the increasing demand for versatile deep learn￾ing models, various large pre-trained models like BERT [15], ViT [16], and GPT [67] have emerged. However, their single-modality limits generalization, leading to the development of multimodal models. Autoregressive lan￾guage models are now popular for vision-language tasks [1, 2, 4, 31, 38, … view at source ↗
Figure 3
Figure 3. Overview of our proposed model architecture. Our model consists of a retriever to search semantically similar recipes from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Qualitative results. The ingredients in generated recipes that overlap with ground truth (“GT”) are highlighted in yellow, while [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparison between generated recipes and GT recipes. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Confusion matrix of Self-consistency Ensemble Voting for 5,000 test samples. The horizontal axis represents the index sorted [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: One set for Recipe demonstration R. 5. Conclusion We have presented the first retrieval augmented large multimodal mode to mitigate the hallucination issue for recipe generation. We introduce the Stochastic Diversified Retrieval Augmentation to enable the model to bett…
Figure 1
Figure 1. Figure 1: Our training data organization. BOLD serves as a fixed prompt, the blue font highlights the three groups of ingredients and one group of instructions retrieved, and the rest of the content represents the ground truth. 4. Ablation of Self-consistency Ensemble Vot￾ing To…
Figure 2
Figure 2. Figure 2: Comparison between retrieval results and ground truth. BOLD represents the relevant and similar parts between the retrieved [PITH_FULL_IMAGE:figures/full_fig_p014_2.png]
Figure 3
Figure 3. Figure 3: Qualitative results of whether the best model uses retrieval information for inference. The yellow highlights indicate the parts [PITH_FULL_IMAGE:figures/full_fig_p015_3.png]
Figure 4
Figure 4. Figure 4: Comparison between retrieval results and ground truth. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Additional qualitative results. The ingredients in generated recipes that overlap with ground truth (“GT”) are highlighted in [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results of Ablation of Self-consistency Ensemble Voting. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Retrieved and predicted ingredients. BOLD represents the overlap of ingredients between both Retrieved ingredients and Predicted ingredients with the ground truth ingredients. The ingredients in red font are the ones incorrectly predicted due to errors in the retrieved…
Figure 8
Figure 8. Figure 8: Predicted and ground truth ingredients, instructions, and ingredients extracted from the predicted instructions. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Retrieval failure examples. Red text shows prediction errors influenced by the retrieved instruction, and yellow highlights indicate [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Two sets of retrieval results for “butternut squash salad with hazelnuts” and their corresponding predictions. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fine-tuning Language Models for Recipe Generation: A Comparative Analysis and Benchmark Study

    cs.CL 2025-02 conditional novelty 4.0 of 10

    Fine-tuning small language models for recipe generation produces mixed results: Phi-2 degrades on the authors' custom quality scores while SmolLM-360M and 1.7B perform similarly.

Reference graph

Works this paper leans on

85 extracted references · 63 canonical work pages · cited by 1 Pith paper

  1. [57]

    Revamping cross-modal recipe retrieval with hierarchical transformers and self-supervised learning

    Amaia Salvador, Erhan Gundogdu, Loris Bazzani, and Michael Donoser. Revamping cross-modal recipe retrieval with hierarchical transformers and self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15475–15484, 2021. 1, 3, 5

    work page 2021

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  4. [3]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446–461. Springer, 2014. 1

    2014

  5. [4]

    Visualgpt: Data-efficient adaptation of pretrained language models for image captioning

    Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed El- hoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030–18040, 2022. 3

    2022

  6. [5]

    Bench- marking large language models in retrieval-augmented gen- eration

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Bench- marking large language models in retrieval-augmented gen- eration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754–17762, 2024. 3

    2024

  7. [6]

    Deep-based ingredi- ent recognition for cooking recipe retrieval

    Jingjing Chen and Chong-Wah Ngo. Deep-based ingredi- ent recognition for cooking recipe retrieval. In Proceedings of the 24th ACM international conference on Multimedia , pages 32–41, 2016. 1, 2

    2016

  8. [7]

    A study of multi-task and region-wise deep learning for food ingredient recognition

    Jingjing Chen, Bin Zhu, Chong-Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. A study of multi-task and region-wise deep learning for food ingredient recognition. IEEE Trans- actions on Image Processing, 30:1514–1526, 2020. 1

    work page 2020

Show all 85 references
  1. [8]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310...

    arXiv

  2. [9]

    Cross- modal recipe retrieval with rich food attributes

    Jing-jing Chen, Chong-Wah Ngo, and Tat-Seng Chua. Cross- modal recipe retrieval with rich food attributes. In Proceed- ings of the 25th ACM international conference on Multime- dia, pages 1771–1779, 2017. 1

    2017

  3. [10]

    Image search with text feedback by visiolinguistic attention learn- ing

    Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolinguistic attention learn- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 3001–3011,

  4. [11]

    Fire: Food image to recipe generation

    Prateek Chhikara, Dhiraj Chaurasia, Yifan Jiang, Omkar Ma- sur, and Filip Ilievski. Fire: Food image to recipe generation. pages 8184–8194, 2024. 1, 2, 5, 6, 7

    2024

  5. [12]

    Food recognition: a new dataset, experiments, and re- sults

    Gianluigi Ciocca, Paolo Napoletano, and Raimondo Schet- tini. Food recognition: a new dataset, experiments, and re- sults. IEEE journal of biomedical and health informatics , 21(3):588–598, 2016. 1

    2016

  6. [13]

    Food logging: an information literacy perspective

    Andrew Martin Cox, Pamela McKinney, and Paula Goodale. Food logging: an information literacy perspective. Aslib Journal of Information Management , 69(2):184–200, 2017. 1

    2017

  7. [14]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. Ad- vances in Neural Information Processing Systems, 36, 2024. 1

    2024

  8. [15]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. 2018. 3

    2018

  9. [16]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. 2020. 3

    2020

  10. [17]

    Exploiting food choice biases for healthier recipe recommen- dation

    David Elsweiler, Christoph Trattner, and Morgan Harvey. Exploiting food choice biases for healthier recipe recommen- dation. In Proceedings of the 40th international acm sigir conference on research and development in information re- trieval, pages 575–584, 2017. 1

    2017

  11. [18]

    Chef transformer,

    Mehrdad Farahani, Kartik Godawat, Haswanth Aekula, Deepak Pandian, and Nicholas Broad. Chef transformer,

  12. [19]

    Intelligent food planning: personalized recipe recommendation

    Jill Freyne and Shlomo Berkovsky. Intelligent food planning: personalized recipe recommendation. In Proceedings of the 15th international conference on Intelligent user interfaces , pages 321–324, 2010. 1

    2010

  13. [20]

    Go figure: A meta evaluation of factuality in summarization

    Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. Go figure: A meta evaluation of factuality in summarization. 2020. 3

    2020

  14. [21]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jin- liu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023. 3

    2023 arXiv

  15. [22]

    Navigating weight prediction with diet diary

    Yinxuan Gui, Bin Zhu, Jingjing Chen, Chong Wah Ngo, and Yu-Gang Jiang. Navigating weight prediction with diet diary. In Proceedings of the 32nd ACM International Conference on Multimedia, page 127–136, 2024. 1

    2024

  16. [23]

    Cross-modal retrieval aug- mentation for multi-modal classification

    Shir Gur, Natalia Neverova, Chris Stauffer, Ser-Nam Lim, Douwe Kiela, and Austin Reiter. Cross-modal retrieval aug- mentation for multi-modal classification. 2021. 3

    2021

  17. [24]

    Retrieval augmented language model pre- training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre- training. pages 3929–3938, 2020. 2, 3

    2020

  18. [25]

    Lee, Ke Shu, Palakorn Achananuparp, Philips Kokoh Prasetyo, Yue Liu, Ee-Peng Lim, and Lav R Varshney

    Helena H. Lee, Ke Shu, Palakorn Achananuparp, Philips Kokoh Prasetyo, Yue Liu, Ee-Peng Lim, and Lav R Varshney. Recipegpt: Generative pre-training based cooking recipe generation and evaluation system. In Companion Proceedings of the Web Conference 2020, pages 181–184, 2020. 2

    2020

  19. [26]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. 2021. 1, 4, 5

    2021

  20. [27]

    Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge mem- ory

    Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge mem- ory. In Proceedings of the IEEE/CVF conference on com-...

  21. [28]

    Multi-scale multi-view deep feature aggregation for food recognition

    Shuqiang Jiang, Weiqing Min, Linhu Liu, and Zhengdong Luo. Multi-scale multi-view deep feature aggregation for food recognition. IEEE Transactions on Image Processing, 29:265–276, 2019. 1

    2019

  22. [29]

    Active retrieval augmented generation

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. Active retrieval augmented generation. 2023. 2, 3

    2023

  23. [30]

    Rode: Linear rectified mixture of diverse experts for food large multi-modal models

    Pengkun Jiao, Xinlan Wu, Bin Zhu, Jingjing Chen, Chong- Wah Ngo, and Yugang Jiang. Rode: Linear rectified mixture of diverse experts for food large multi-modal models. arXiv preprint arXiv:2407.12730, 2024. 1

    2024 arXiv

  24. [31]

    Lumen: Unleashing versatile vision-centric capabilities of large multimodal models.arXiv preprint arXiv:2403.07304, 2024

    Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Lumen: Unleashing versatile vision-centric capabilities of large multimodal models.arXiv preprint arXiv:2403.07304, 2024. 3

    2024 arXiv

  25. [32]

    Two-stage visual cues en- hancement network for referring image segmentation

    Yang Jiao, Zequn Jie, Weixin Luo, Jingjing Chen, Yu-Gang Jiang, Xiaolin Wei, and Lin Ma. Two-stage visual cues en- hancement network for referring image segmentation. In Proceedings of the 29th ACM international conference on multimedia, pages 1331–1340, 2021. 3

    2021

  26. [33]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692,

    arXiv

  27. [34]

    Pre-training via paraphrasing

    Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. Pre-training via paraphrasing. volume 33, pages 18470–18481, 2020. 3

    2020

  28. [36]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. volume 33, pages 9459–9474, 2020. 3

    2020

  29. [37]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  30. [38]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 3

    2023

  31. [39]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 3, 4

    2022

  32. [40]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. 2023. 3

    2023

  33. [41]

    Eyes can deceive: Benchmarking counterfactual reasoning abili- ties of multi-modal large language models

    Yian Li, Wentao Tian, Yang Jiao, and Jingjing Chen. Eyes can deceive: Benchmarking counterfactual reasoning abili- ties of multi-modal large language models. arXiv preprint arXiv:2404.12966, 2024. 3

    2024 arXiv

  34. [42]

    Food and ingredient joint learning for fine- grained recognition

    Chengxu Liu, Yuanzhi Liang, Yao Xue, Xueming Qian, and Jianlong Fu. Food and ingredient joint learning for fine- grained recognition. IEEE transactions on circuits and Sys- tems for Video Technology, 31(6):2480–2493, 2020. 1

    2020

  35. [43]

    From canteen food to daily meals: Generalizing food recognition to more practical scenarios

    Guoshan Liu, Yang Jiao, Jingjing Chen, Bin Zhu, and Yu- Gang Jiang. From canteen food to daily meals: Generalizing food recognition to more practical scenarios. IEEE Transac- tions on Multimedia, pages 1–10, 2024. 1

    2024

  36. [44]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. volume 36, 2024. 1, 2, 3, 4, 5, 6

    2024

  37. [45]

    Transformer with peak suppression and knowledge guidance for fine- grained image recognition

    Xinda Liu, Lili Wang, and Xiaoguang Han. Transformer with peak suppression and knowledge guidance for fine- grained image recognition. Neurocomputing, 492:137–149,

  38. [46]

    Generating personalized recipes from his- torical user preferences

    Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. Generating personalized recipes from his- torical user preferences. arXiv preprint arXiv:1909.00105 ,

    1909 arXiv

  39. [47]

    Wide-slice residual networks for food recognition

    Niki Martinel, Gian Luca Foresti, and Christian Micheloni. Wide-slice residual networks for food recognition. In 2018 IEEE Winter conference on applications of computer vision (WACV), pages 567–576. IEEE, 2018. 1

    2018

  40. [48]

    Visual classification via description from large language models

    Sachit Menon and Carl V ondrick. Visual classification via description from large language models. 2022. 1

    2022

  41. [49]

    A survey on food computing.ACM Computing Surveys (CSUR), 52(5):1–36, 2019

    Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain. A survey on food computing.ACM Computing Surveys (CSUR), 52(5):1–36, 2019. 1, 2

    2019

  42. [50]

    Large scale visual food recognition

    Weiqing Min, Zhiling Wang, Yuxin Liu, Mengjiang Luo, Liping Kang, Xiaoming Wei, Xiaolin Wei, and Shuqiang Jiang. Large scale visual food recognition. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2023. 2

    2023

  43. [51]

    Chef: cross-modal hierarchical embeddings for food domain retrieval

    Hai X Pham, Ricardo Guerrero, Vladimir Pavlovic, and Jia- tong Li. Chef: cross-modal hierarchical embeddings for food domain retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2423–2430, 2021. 1

    2021

  44. [52]

    In- context retrieval-augmented language models

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Am- non Shashua, Kevin Leyton-Brown, and Yoav Shoham. In- context retrieval-augmented language models. Transactions of the Association for Computational Linguistics , 11:1316– 1331, 2023. 2, 3

    2023

  45. [53]

    Smallcap: lightweight image captioning prompted with retrieval augmentation

    Rita Ramos, Bruno Martins, Desmond Elliott, and Yova Ke- mentchedjhieva. Smallcap: lightweight image captioning prompted with retrieval augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2840–2849, 2023. 3

    2023

  46. [54]

    Learning multi-subset of classes for fine-grained food recognition

    Javier R ´odenas, Bhalaji Nagarajan, Marc Bola˜nos, and Petia Radeva. Learning multi-subset of classes for fine-grained food recognition. In Proceedings of the 7th International Workshop on Multimedia Assisted Dietary Management , pages 17–26, 2022. 1

    2022

  47. [55]

    Foodai: Food image recognition via deep learning for smart food logging

    Doyen Sahoo, Wang Hao, Shu Ke, Wu Xiongwei, Hung Le, Palakorn Achananuparp, Ee-Peng Lim, and Steven CH Hoi. Foodai: Food image recognition via deep learning for smart food logging. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Min...

    2019

  48. [56]

    Inverse cooking: Recipe generation from food images

    Amaia Salvador, Michal Drozdzal, Xavier Gir ´o-i Nieto, and Adriana Romero. Inverse cooking: Recipe generation from food images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10453– 10462, 2019. 1, 2, 5, 6, 7

    2019

  49. [58]

    Learning cross-modal embeddings for cooking recipes and food im- ages

    Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. Learning cross-modal embeddings for cooking recipes and food im- ages. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3020–3028, 2017. 7

    2017

  50. [59]

    En- hancing recipe retrieval with foundation models: A data aug- mentation perspective

    Fangzhou Song, Bin Zhu, Yanbin Hao, and Shuo Wang. En- hancing recipe retrieval with foundation models: A data aug- mentation perspective. In European Conference on Com- puter Vision, pages 111–127, 2024. 1

    2024

  51. [60]

    Recipe recommendation using ingredient networks

    Chun-Yuen Teng, Yu-Ru Lin, and Lada A Adamic. Recipe recommendation using ingredient networks. In Proceedings of the 4th annual ACM web science conference , pages 298– 307, 2012. 1

    2012

  52. [61]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 3

    2023 arXiv

  53. [62]

    X-tra: Improving chest x-ray tasks with cross-modal retrieval augmentation

    Tom van Sonsbeek and Marcel Worring. X-tra: Improving chest x-ray tasks with cross-modal retrieval augmentation. In International Conference on Information Processing in Med- ical Imaging, pages 471–482. Springer, 2023. 3

    2023

  54. [63]

    Composing text and image for image retrieval-an empirical odyssey

    Nam V o, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6439–6448, 2019. 5

    2019

  55. [64]

    Structure-aware generation network for recipe gen- eration from images

    Hao Wang, Guosheng Lin, Steven CH Hoi, and Chunyan Miao. Structure-aware generation network for recipe gen- eration from images. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16, pages 359–374. Springer, 2020. 1, 2

    2020

  56. [65]

    Learning structural representations for recipe genera- tion and food retrieval

    Hao Wang, Guosheng Lin, Steven CH Hoi, and Chunyan Miao. Learning structural representations for recipe genera- tion and food retrieval. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(3):3363–3377, 2022. 1, 2, 5

    2022

  57. [66]

    Learning cross-modal embeddings with ad- versarial networks for cooking recipes and food images

    Hao Wang, Doyen Sahoo, Chenghao Liu, Ee-peng Lim, and Steven CH Hoi. Learning cross-modal embeddings with ad- versarial networks for cooking recipes and food images. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 11572–11581, 2019...

    2019

  58. [67]

    Xgpt: Cross-modal generative pre-training for image captioning

    Qiaolin Xia, Haoyang Huang, Nan Duan, Dongdong Zhang, Lei Ji, Zhifang Sui, Edward Cui, Taroon Bharti, and Ming Zhou. Xgpt: Cross-modal generative pre-training for image captioning. In Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2...

    2021

  59. [68]

    Food recognition using statistics of pairwise lo- cal features

    Shulin Yang, Mei Chen, Dean Pomerleau, and Rahul Suk- thankar. Food recognition using statistics of pairwise lo- cal features. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 2249–2256. IEEE, 2010. 1

    2010

  60. [69]

    Retrieval-augmented mul- timodal language modeling

    Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-augmented mul- timodal language modeling. 2022. 3, 5

    2022

  61. [70]

    Foodlmm: A versatile food assistant using large multi-modal model

    Yuehao Yin, Huiyan Qi, Bin Zhu, Jingjing Chen, Yu-Gang Jiang, and Chong-Wah Ngo. Foodlmm: A versatile food assistant using large multi-modal model. arXiv preprint arXiv:2312.14991, 2023. 1, 2, 5, 6

    2023 arXiv

  62. [71]

    Eagle: Towards efficient arbitrary re- ferring visual prompts comprehension for multimodal large language models

    Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, and Yu-Gang Jiang. Eagle: Towards efficient arbitrary re- ferring visual prompts comprehension for multimodal large language models. arXiv preprint arXiv:2409.16723, 2024. 3

    2024 arXiv

  63. [72]

    Eventhallusion: Diagnosing event hal- lucinations in video llms

    Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, and Yu-Gang Jiang. Eventhallusion: Diagnosing event hal- lucinations in video llms. arXiv preprint arXiv:2409.16597,

  64. [73]

    Sequential learning for ingredient recognition from images

    Mengyang Zhang, Guohui Tian, Ying Zhang, and Hong Liu. Sequential learning for ingredient recognition from images. IEEE Transactions on Circuits and Systems for Video Tech- nology, 2022. 1

    2022

  65. [74]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. volume 36, 2024. 3

    2024

  66. [75]

    Learn- ing from web recipe-image pairs for food recognition: Prob- lem, baselines and performance

    Bin Zhu, Chong-Wah Ngo, and Wing-Kwong Chan. Learn- ing from web recipe-image pairs for food recognition: Prob- lem, baselines and performance. IEEE Transactions on Mul- timedia, 24:1175–1185, 2021. 1

    2021

  67. [76]

    R2gan: Cross-modal recipe retrieval with generative adver- sarial network

    Bin Zhu, Chong-Wah Ngo, Jingjing Chen, and Yanbin Hao. R2gan: Cross-modal recipe retrieval with generative adver- sarial network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11477– 11486, 2019. 1

    2019

  68. [77]

    Cross- domain cross-modal food transfer

    Bin Zhu, Chong-Wah Ngo, and Jing-jing Chen. Cross- domain cross-modal food transfer. In Proceedings of the 28th ACM International Conference on Multimedia , pages 3762–3770, 2020. 1 Supplementary Material Retrieval Augmented Recipe Generation Guoshan Liu1,2*, Hailong Yin1,2∗, ...

    2020

  69. [78]

    Details of Training Data Organization As illustrated in Figure 1, we present an example of how the training data is organized in our proposed retrieval aug- mented framework

  70. [79]

    award winning soft chocolate chip cookies,

    Retrieved Results Our SDRA method utilizes a variety of retrieved recipes to enhance model performance. Figure 2, and Figure 4 re- spectively illustrate some of the training data alongside their corresponding retrieved data. It can be seen that there are significant overlaps b...

  71. [80]

    Ablation of Stochastic Diversified Retrieval Augmentation (SDRA) As described in Section 4.3.1, to investigate whether the ingredients and instructions added before Qinstructions in Recipe demonstration R need to come from the same top retrieval results, i.e., having the same ...

    2024 arXiv

  72. [81]

    S” refers to the number of generated recipes for ensemble voting. “Sum

    Ablation of Self-consistency Ensemble Vot- ing To demonstrate the impact of self-consistent ensemble voting, Table 2 examines the results of our model when us- ing cosine similarity, BLEU, SacreBLEU, and ROUGE L to calculate the mutual agreement among the generated recipe cand...

  73. [82]

    GT”, whereas other models—“LLaV A-FT

    More Qualitative Examples Figure 5 displays additional qualitative results. In these three cases, our predictions closely align with the ground truth “GT”, whereas other models—“LLaV A-FT”, “Inverse Cooking” [?], and “FoodLLM” [?] —exhibited various hal- lucinations. For insta...

  74. [83]

    As shown in Figure 7, in the first case, it can be observed that the model predic- tion partly overlaps with the ground truth

    Consistency between the Generated Ingredi- ents VS Retrieved Ingredients We compared the retrieved ingredients with the ingre- dients predicted by the model. As shown in Figure 7, in the first case, it can be observed that the model predic- tion partly overlaps with the ground...

  75. [84]

    In the first case, ingredients like ‘greens’ and ‘tomato’, which are visually prominent in the image, are successfully predicted

    Ingredients Extracted from Instructions and Directly Predicted Figure 8 shows a comparison between ingredients ex- tracted from generated instructions and those directly pre- dicted. In the first case, ingredients like ‘greens’ and ‘tomato’, which are visually prominent in the...

  76. [85]

    mango shrimp

    Retrieval Failure Case To explore retrieval failure cases, we analyzed some ex- amples of unsuccessful retrievals. Figure 9 provides an ex- ample of a retrieval failure. The prediction is influenced by retrieved information, like ”cut into bars,” and mixing nuts into the batte...

  77. [86]

    butternut squash salad with hazelnuts

    Broader Ingredient Categories To avoid situations where ingredients in the prediction and ground truth are very similar but are counted as com- pletely incorrect due to different terminology in the ingre- dients list—such as ’pasta’ and ’spaghetti’—we designed an alternative p...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.