{"id":"a8952602-94fa-4b5d-b43e-e6de419b14a3","arxiv_id":"2505.21068","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new multimodal benchmark for predicting implicit ingredients in procedural cooking videos, with a fine-tuned model that outperforms GPT-4o by 14.7-17% relative F1.","lead":"This paper introduces a new dataset, Implicit-VidSRL, that labels cooking instruction videos with semantic roles including implicit ingredients that must be inferred from earlier steps. The authors also fine-tune a vision-language model on silver-standard labels and report that it beats GPT-4o at predicting these implicit arguments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold test labels may inherit GPT-4o annotation style via GPT-4o-Mini pre-annotation; the fine-tuned model's 17% gain could partly be style-matching, not genuine understanding.","rationale":"The reader's weakest assumption focused on silver-standard label quality; I sharpen this to a specific circularity in label construction: the gold test labels themselves are initialized by GPT-4o-Mini and then corrected by a single annotator, so they may carry a systematic GPT-4o annotation style. The silver training labels come from GPT-4o with a nearly identical prompt, meaning the fine-tuned model is trained to reproduce that style. The evaluation then compares this style-matched model against GPT-4o zero-shot, whose output style is not explicitly trained to match the gold labels. If the gold labels are style-biased, the reported 17%/14.7% relative improvements could overstate genuine procedural understanding. This is a load-bearing concern because the central claim is exactly that fine-tuning improves implicit argument prediction; a style-matching artifact would undermine that interpretation. The paper has real strengths—public dataset, code, ablations, qualitative examples—so I do not reject it. The proposed test is concrete and feasible: re-annotating 50 videos independently is sufficient to assess both gold-label reliability and the stability of the model improvement. Until such a check is done, the conditional verdict stands.","tokens_in":24717,"tokens_out":5391,"duration_ms":62908,"concrete_test":"Take a random subset of, say, 50 test videos (or ~300 semantic frames). Have two independent annotators (with linguistic training) produce SRL annotations from scratch using only the instruction text, video, and the annotation guidelines, without seeing GPT-4o-Mini's pre-annotations. Compute IAA (e.g., Cohen's kappa on entity sets) between the two annotators, and between the annotators and the published gold labels. Then evaluate GPT-4o (V+T) and iSRL-Qwen2-VL (V+T) on this independently annotated subset, and compute the relative F1 improvement on what-implicit and where/with-implicit. If the relative improvement largely disappears (e.g., drops below 5%) or the IAA with published gold is low (kappa < 0.6), the original claim is substantially an artifact of annotation-style bias. If the improvement persists on the independent gold, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the benchmark's gold labels being a valid measure of implicit argument understanding. But the gold labels are not produced by independent human annotation from scratch: Stage 2 uses GPT-4o-Mini to auto-generate SRLs (with a CoT prompt and ICL examples), and Stage 3 has a single PhD student correct those outputs. No inter-annotator agreement is reported, and the Stage 1 implicit-entity annotations were split between two annotators with zero overlap (Appendix A.2), so reliability is unmeasured. The silver-standard training data is generated by GPT-4o with a very similar prompt (Figure 11) from Tasty text instructions. Thus, the fine-tuned iSRL-Qwen2-VL may learn GPT-4o's particular decisions about which ingredients to expand as implicit arguments, how to handle composite entities, and what to put in WHERE_OR_WITH. If the 'gold' test labels have been nudged toward the same style in Stage 3 (the annotator sees and corrects GPT-4o-Mini's output), the evaluation could reward style matching rather than robust procedure understanding. The reported 17% relative improvement over GPT-4o zero-shot could then be inflated because the test set shares a systematic bias with the training data, while GPT-4o zero-shot produces outputs in a slightly different style. The paper's own Limitations note reliance on initial SRL quality, but the issue is more specific: a circularity between training-label generation and test-label construction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Implicit-VidSRL, a dataset of 231 cooking videos from YouCook2 and Tasty annotated with semantic role labels ({verb, what, where/with}) that include implicit arguments inferred from textual and visual context. The authors define two tasks: a cloze-style implicit argument prediction task and a next-step prediction task. They evaluate several proprietary and open-source multimodal LLMs on these tasks and propose iSRL-Qwen2-VL, a Qwen2-VL model fine-tuned on silver-standard SRL data automatically generated by GPT-4o from Tasty instructions. The paper reports that iSRL-Qwen2-VL achieves 17% relative F1 improvement for what-implicit and 14.7% for where/with-implicit over GPT-4o with multimodal input, as well as modest gains in next-step prediction.","tokens_in":25051,"tokens_out":5248,"duration_ms":54659,"significance":"If the benchmark is valid, this is a useful contribution: it targets an underexplored problem (implicit arguments in procedural video), provides a public dataset and code, includes extensive experiments across multiple models and modalities, and shows that fine-tuning on automatically generated SRL data can improve implicit argument prediction. The paper is generally well written and the experimental setup is clearly described. The main weakness is that the validity of the gold-standard labels is not yet established, which is load-bearing for the central claim.","major_comments":[{"comment":"The gold-standard test labels are produced by GPT-4o-Mini pre-annotation followed by manual correction by a single annotator, while the silver-standard training data are generated by GPT-4o using a nearly identical chain-of-thought prompt (Figure 11 vs. Figure 8). This creates a concrete risk of circularity: the fine-tuned model may learn to reproduce GPT-4o's decisions about which entities to treat as implicit, how to decompose composite entities, and how to fill WHERE_OR_WITH, and the test set may have been nudged toward the same style through the GPT-4o-Mini pre-annotation that the annotator corrects. The reported gain of iSRL-Qwen2-VL over GPT-4o zero-shot in Table 2 could therefore be inflated by style-matching rather than genuine procedural understanding. The authors should provide evidence that the gold labels are not biased in this way, for example by releasing an independently human-annotated test subset (without model pre-annotation) and showing the improvement persists, or by analyzing the annotator's correction rate and the degree to which the fine-tuned model's predictions differ in style from GPT-4o's. The Limitations section acknowledges reliance on initial SRL quality, but the more specific circularity between training-label generation and test-label construction is not addressed.","section":"Section 3 (Data Annotation) and Section 5 (Silver-standard Dataset)"},{"comment":"The reliability of the core annotations is unmeasured. Stage 1 implicit-entity identification was performed by two annotators on disjoint halves of 700 samples with zero overlap, and no inter-annotator agreement is reported. Stage 3 relied on a single annotator to correct the GPT-4o-Mini SRLs. Given that the benchmark's defining contribution is implicit-argument annotation, the authors should report inter-annotator agreement (e.g., Cohen's kappa or a similar measure) on an overlapping subset for Stage 1, and ideally for Stage 3 as well. Without this, readers cannot assess whether the gold labels are stable and reproducible, which is a necessary condition for the benchmark to be reliable.","section":"Section A.2 (Annotator Recruitment and Payments)"},{"comment":"The headline improvements over GPT-4o are not accompanied by any variance or significance analysis. Fine-tuning with LoRA can be sensitive to random seeds and data ordering, and the evaluation metric involves a non-standard word-overlap IoU for non-exact matches. The authors should report mean and standard deviation over at least three training runs, or a significance test (e.g., bootstrap or paired test), to establish that the 17% and 14.7% relative improvements are not due to noise. This is particularly important because the differences are relative to a single zero-shot GPT-4o run.","section":"Section 7.1 (Table 2) and Section 6.2"}],"minor_comments":[{"comment":"The text in the prompt figures contains the typo \"wheather\" instead of \"whether\" (e.g., \"decide wheather you need to fill\"). Since these prompts are part of the reproducible annotation pipeline, the typos should be corrected.","section":"Prompt figures (Figure 9, 10, 11)"},{"comment":"The row \"whatemptyset/total count\" lacks a space; it should be \"what empty set / total count\".","section":"Table 1"},{"comment":"The GEPSAN row (Row 1) has empty values for #Params and Type. Either fill in the information or explain why it is omitted.","section":"Table 3"},{"comment":"The caption says the masked arguments are highlighted with red boxes, but the red boxes are not visible in the figure. Consider increasing contrast or adding explicit annotations.","section":"Figure 2"},{"comment":"The sliding-window evaluation for next-step prediction (matching against the next three gold steps) is described only briefly. Please specify how the window is handled near the end of a recipe and how ties are broken.","section":"Section 6.2"},{"comment":"The citation \"Assembly101 (Sener et al.)\" lacks a year. Also, the paper claims \"state-of-the-art performance\" in Section 7.2, but the comparison set is limited; consider softening the claim or adding more baselines.","section":"Section 2 (Related Work)"}],"recommendation":"major_revision","confidential_remarks":"The core concern is the potential circularity in the annotation pipeline: the gold test labels are produced with heavy assistance from GPT-4o-Mini, while the silver-standard training data are generated with GPT-4o using very similar prompts. If the authors can provide an independent human-annotated test subset or strong evidence that the fine-tuned model's gains are not due to style-matching, the paper would be a solid contribution. The lack of inter-annotator agreement is also a barrier to using the dataset as a reliable benchmark. The paper fits the journal's scope well, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a genuine dataset paper, and the headline result—fine-tuned iSRL-Qwen2-VL beating GPT-4o zero-shot on implicit argument prediction by 17% relative F1—is credible in direction, but the benchmark construction leaves a style-matching loophole that the paper doesn't close.\n\nWhat's new and good: Implicit-VidSRL extends implicit SRL to multimodal procedural videos, with 231 videos from YouCook2 and Tasty, 2545 semantic frames, and a task that requires tracking ingredients across long temporal context. That is a legitimate gap. The annotation scheme is simple but workable. The authors evaluate a range of open and proprietary models, use CoT prompting, and show that fine-tuning on silver-standard SRL data helps. They also include ablations on fine-tuning strategy and prompt design, and they ship code and data. Those are real assets.\n\nSoft spots: the gold test labels are produced in three stages. Stage 1 identifies implicit entities with two annotators who split the data with zero overlap. Stage 2 uses GPT-4o-Mini to auto-generate SRLs. Stage 3 has a single PhD student correct those outputs. No inter-annotator agreement is reported anywhere. The silver-standard training data is generated by GPT-4o with a prompt very similar to the Stage 2 prompt. So a fine-tuned model can learn GPT-4o's annotation style, and the test labels, having been pre-annotated by the same model family, may reward that style matching. The paper's Limitations note reliance on initial SRL quality, but the more specific issue—possible circularity between training-label generation and test-label construction—deserves a sharper treatment. That said, the human correction step in Stage 3 does break some of the circularity, and the gains are large enough that I don't think the result is fabricated. Still, without IAA or an independent gold annotation, the benchmark's reliability is unmeasured.\n\nWho this is for: people working on video-and-language procedural reasoning, SRL, and instruction understanding. It should go to peer review. I'd want the authors to add IAA, report variance, and analyze silver-label quality versus human labels before recommending acceptance, but the paper deserves referee time.","headline":"A solid dataset paper with a plausible headline result, but the gold-label construction leaves a style-matching circularity that the authors should address before publication.","tokens_in":25485,"tokens_out":2198,"would_cite":true,"duration_ms":29403,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Procedural instructions are full of unstated ingredients, and this paper shows that a small fine-tuned model can infer them better than GPT-4o.","keywords":["semantic role labeling","implicit arguments","procedural video understanding","multimodal large language models","entity tracking","cooking video datasets","next-step prediction","silver-standard annotations"],"falsifier":"A concrete check is to re-annotate the evaluation set from scratch with fresh human annotators who never see any GPT-4o output, then rerun iSRL-Qwen2-VL against GPT-4o on those labels; if the relative gains shrink or vanish, the silver-training hypothesis is the problem. A second check is to train the same model on silver labels generated by a different model family and see whether the improvements replicate.","tokens_in":24547,"feed_emoji":"🍳","tokens_out":9208,"duration_ms":82202,"temperature":0.7,"pith_summary":"Cookbook instructions omit things constantly: \"cook and cut it into cubes\" means cook the oiled pita bread you just made, not some unspecified object. This paper argues that this kind of implicit-argument inference is a real, testable capability that current multimodal models lack, and that it can be learned. To show this, the authors build Implicit-VidSRL, a semantic-role-labeling dataset of 231 cooking videos annotated as {verb, what, where/with} frames that mark which ingredients must be carried over from earlier steps or clips. They then fine-tune a 7B model, iSRL-Qwen2-VL, on silver-standard labels generated by GPT-4o, and report that it outperforms GPT-4o by 17% relative F1 on implicit what-arguments and 14.7% on implicit where/with-arguments. If this holds, it means a small open-weights model can track entities through procedural video as well as or better than a much larger closed model, a step toward assistants that understand instructions the way a sous-chef would.","feed_headline":"Fine-tuned 7B model beats GPT-4o on implicit arguments in videos","feed_subtitle":"Auto-generated labels let a small open model fill in unstated recipe ingredients.","key_machinery":"The load-bearing object is the semantic frame {verb, what, where/with}, a deliberately small role set for procedural steps: the verb is given, \"what\" names the objects the action affects, and \"where/with\" names the location or companion ingredients. The key move is that these arguments can be implicit — an entity inherited from an earlier step, possibly changed by cooking, like \"pork with spices\" or \"the dressing\" decomposed into its ingredient list. On top of this representation sit two evaluation tasks: a cloze task that masks the what and where/with arguments and asks the model to fill them given the verb and the preceding multimodal context, and a next-step prediction task that must generate the future instruction plus its semantic frames. Training signal comes from a silver-standard dataset: GPT-4o, prompted with chain-of-thought instructions and five in-context examples, converts Tasty instructions into frames with implicit arguments, producing roughly 2.5K training videos and 18K next-step samples, on which the 7B model is LoRA fine-tuned.","core_discovery":"The paper's central claim is that procedural instructions are systematically elliptical, and that a model which can fill in the missing argument — the \"implicit argument\" — has genuinely understood the procedure. The authors encode each instruction as one or more semantic frames of the form {verb, what, where/with}, where arguments marked implicit are ingredients named in earlier steps or visible in earlier clips but absent from the current sentence. On their new benchmark, Implicit-VidSRL, built from YouCook2 and Tasty videos, they find that strong multimodal models including GPT-4o and Qwen2-VL guess explicit ingredients decently but lose track of implicit ones, especially in later steps of long recipes. The proposed remedy, iSRL-Qwen2-VL, is a 7B Qwen2-VL model fine-tuned with LoRA on GPT-4o-generated silver SRL annotations; it closes most of the gap to the larger models and, with video-plus-text input, beats GPT-4o on the implicit-argument metrics. The paper also claims that emitting semantic frames as an intermediate representation improves next-step prediction, adding about 2% METEOR over the base model.","pith_inferences":["One testable extension is to swap the label generator: train on SRLs produced by a different LLM and verify the F1 gains survive; if they do not, part of the reported advantage is overfitting to GPT-4o's annotation style rather than to the implicit-argument facts.","The same silver-label recipe could transfer to other procedural domains with comparable ellipsis, such as assembly, medical, or craft instructions, where manual annotation would be far costlier.","The finding that both total entities and implicit arguments grow at later semantic-frame positions suggests that entity tracking, not verb semantics, is the limiting difficulty, so a natural next step is to vary the carry-over distance — how many steps back an ingredient must be retrieved — and measure where each model starts dropping entities.","Used as a probe, the cloze task could serve as a cheap diagnostic for long-context video understanding, since it isolates whether a model can track an entity across frames without requiring full sentence generation."],"forward_implications":["If the reported gains are correct, a 7B open-weights model fine-tuned on silver-standard labels can match or beat GPT-4o at implicit-argument prediction, so access to a large proprietary model is not the binding constraint for this capability.","Emitting {verb, what, where/with} frames while predicting the next step improves future-step prediction over the same model without frames, so SRL acts as an effective intermediate representation, not just an evaluation target.","Multimodal input (video plus text) helps only when combined with SRL training; untrained models stay biased toward surface text, so visual dynamics must be learned rather than assumed.","Longer recipes degrade implicit-argument scores for all tested models, meaning entity tracking across state changes is the limiting skill, and benchmarks should report performance by step position."],"supporting_citations":[{"why":"YouCook2 supplies 66 of the 231 annotated videos in Implicit-VidSRL.","marker":"(Zhou et al., 2018)"},{"why":"Tasty supplies the remaining videos and the training recipes from which the silver-standard SRL data is generated.","marker":"(Sener et al., 2022)"},{"why":"GPT-4o is both the strongest proprietary baseline and the generator of the silver-standard training labels.","marker":"(Hurst et al., 2024)"},{"why":"Qwen2-VL is the base vision-language model that iSRL-Qwen2-VL fine-tunes.","marker":"(Wang et al., 2024b)"},{"why":"LoRA is the parameter-efficient fine-tuning method used to train the proposed model.","marker":"(Hu et al., 2021)"},{"why":"Chain-of-thought prompting is the mechanism used to elicit the silver-standard SRL splits and implicit arguments from GPT-4o.","marker":"(Wei et al., 2022)"},{"why":"GEPSAN supplies the next-step-prediction protocol and the baseline the paper compares against.","marker":"(Abdelsalam et al., 2023a)"}],"fun_headline_variants":["7B model beats GPT-4o at inferring unstated recipe steps","Fine-tuned 7B model outperforms GPT-4o on implicit video args","Small model wins on implicit arguments in cooking videos","LoRA-tuned 7B model beats GPT-4o on implicit recipe roles","7B model fills in missing recipe steps better than GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated silver-standard labels are accurate enough to teach genuine entity tracking, and that recipe steps really do decompose into simple verb-what-where frames; if the labels carry GPT-4o's biases, or the frame decomposition is wrong for other procedures, the reported gains would not reflect real procedural understanding.","fun_headline_variants_meta":{"raw":{"variants":["7B model beats GPT-4o at inferring unstated recipe steps","Fine-tuned 7B model outperforms GPT-4o on implicit video args","Small model wins on implicit arguments in cooking videos","LoRA-tuned 7B model beats GPT-4o on implicit recipe roles","7B model fills in missing recipe steps better than GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001132,"raw_usage":{"total_tokens":4723,"prompt_tokens":985,"completion_tokens":3738,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":3640}},"tokens_in":601,"tokens_out":3738,"duration_ms":28099,"temperature":1.0,"reasoning_tokens":3640,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:35:47.554213+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to re-annotate the evaluation set from scratch with fresh human annotators who never see any GPT-4o output, then rerun iSRL-Qwen2-VL against GPT-4o on those labels; if the relative gains shrink or vanish, the silver-training hypothesis is the problem. A second check is to train the same model on silver labels generated by a different model family and see whether the improvements replicate.","supporting_citations":[],"review_version":1}