{"id":"83b4cc86-9d42-4b47-ab69-7278b2648510","arxiv_id":"2604.18236","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"COFFAIL is a dataset of successful and anomalous robot skill executions in coffee preparation, used to demonstrate imitation learning of robot policies.","lead":"The paper introduces COFFAIL, a dataset of robot executions for coffee preparation tasks that includes both successful and anomalous episodes collected on a physical robot. This provides training data for imitation learning that accounts for real-world failures rather than only successes.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's weakest assumption attributes to the paper a claim of 'meaningfully better policies' from mixed data; the actual strongest claim only states that the dataset contains both types and that an IL illustration is provided. Because the paper makes no comparative performance assertion, the representativeness issue is not load-bearing for the stated contribution.","tokens_in":1656,"tokens_out":224,"duration_ms":16001,"concrete_test":"Count the total episodes, distinct skills, and anomalous vs. successful split reported in the data-collection section, then re-run the exact imitation-learning pipeline described in the illustration subsection on the released data split to confirm the reported policy behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The manuscript's central claim is the release and description of the COFFAIL dataset (successful plus anomalous executions for coffee-preparation skills, some bimanual) together with a basic illustration of imitation learning on the data. No internal inconsistency, hidden assumption, or unsupported technical step appears in that limited-scope argument.","agreement_with_reader":"disagree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces the COFFAIL dataset of successful and anomalous robot skill executions for coffee preparation tasks, collected with a physical robot in a kitchen environment and including some bimanual manipulation episodes. It describes the data collection setup and collected data, and illustrates the dataset's use for learning a robot policy via imitation learning.","tokens_in":1675,"tokens_out":384,"duration_ms":42954,"significance":"A dataset explicitly containing both successful and anomalous executions fills a notable gap, as most robot manipulation datasets focus solely on successes; this could support research on failure-aware or robust policies. The inclusion of bimanual examples adds value for complex tasks. The basic imitation learning illustration shows one potential use case, though its impact depends on the quality and documentation of the anomalous data.","major_comments":[{"comment":"Abstract and data collection setup description: the central claim that the dataset includes anomalous skill execution episodes is load-bearing for the paper's contribution, yet no details are provided on how anomalies were identified, labeled, or verified (e.g., via human annotation, sensor thresholds, or post-hoc analysis), leaving reproducibility and representativeness unsupported.","section":"Abstract / data collection setup"},{"comment":"Imitation learning illustration: the paper states it illustrates use of the data to learn a robot policy using imitation learning, but supplies no quantitative results, error metrics, baselines, or comparison of policies trained with vs. without anomalous episodes, which undermines the demonstration of the dataset's utility.","section":"Imitation learning illustration"}],"minor_comments":[{"comment":"Consider adding a summary table of episode counts (successful vs. anomalous) per skill type to improve clarity of the collected data description.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their thoughtful comments on our manuscript describing the COFFAIL dataset. We address each major comment below and outline the revisions we will make.","responses":[{"response":"We agree that providing details on anomaly identification is essential for the dataset's utility and reproducibility. The current manuscript focuses on describing the setup and data but omits this aspect. In the revised manuscript, we will add a subsection under data collection explaining that anomalous episodes were identified through post-hoc review by the researchers, noting deviations such as spills, incorrect placements, or failed grasps based on video recordings and task outcomes.","revision_made":"yes","referee_comment":"Abstract and data collection setup description: the central claim that the dataset includes anomalous skill execution episodes is load-bearing for the paper's contribution, yet no details are provided on how anomalies were identified, labeled, or verified (e.g., via human annotation, sensor thresholds, or post-hoc analysis), leaving reproducibility and representativeness unsupported."},{"response":"The imitation learning example serves as a basic illustration of dataset usage for policy learning, consistent with the short paper format. We recognize that quantitative results would better showcase the dataset's value. We will revise to include simple quantitative metrics, such as the success rate of the learned policy on held-out test episodes. However, a full comparison of policies trained with and without anomalous data would necessitate additional training runs and analysis, which we view as extending beyond the illustrative purpose; we will instead add a discussion on how including anomalous data could aid in learning failure-aware policies.","revision_made":"partial","referee_comment":"Imitation learning illustration: the paper states it illustrates use of the data to learn a robot policy using imitation learning, but supplies no quantitative results, error metrics, baselines, or comparison of policies trained with vs. without anomalous episodes, which undermines the demonstration of the dataset's utility."}],"tokens_in":1225,"tokens_out":416,"duration_ms":24972,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"COFFAIL is a new dataset of robot-executed coffee preparation skills that includes both successful and anomalous runs, collected on physical hardware in a kitchen. A couple of the episodes use bimanual manipulation. The paper does a decent job describing the collection setup and showing one simple use case with imitation learning. Real-world robot data with failures is still relatively rare, so this fills a small gap for people working on robust manipulation policies. The soft spots are that the manuscript is very short and provides almost no quantitative information. There are no numbers on the total number of episodes, how the anomalous ones were identified or labeled, or any metrics showing that training on the mixed data improves performance over successful-only data. The imitation learning illustration is mentioned but not evaluated in any detail. This is the kind of paper that might interest a narrow group of researchers building datasets for robot learning or studying anomaly detection in manipulation. It does not claim to solve a big problem or introduce new methods. I would send it to peer review as a data paper, but the reviewers would likely ask for more details on the data characteristics and at least basic experiments comparing policies trained with and without the anomalous examples.","headline":"COFFAIL is a small new dataset of physical robot coffee-prep skills that includes anomalous executions, but the paper stays mostly descriptive with minimal validation.","tokens_in":2156,"tokens_out":309,"would_cite":false,"duration_ms":21056,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"The COFFAIL dataset supplies robot executions of coffee-preparation skills that include both successes and anomalies to train imitation-learning policies.","keywords":["robot learning","dataset","imitation learning","manipulation","anomalous executions","coffee preparation","bimanual manipulation","kitchen environment"],"falsifier":"Train one imitation-learning policy on only the successful COFFAIL episodes and another on the full set, then measure both policies on the same set of test tasks; if the mixed-data policy shows no higher success rate, the benefit of including anomalies is not demonstrated.","tokens_in":2524,"feed_emoji":"☕","tokens_out":596,"duration_ms":23532,"temperature":0.7,"pith_summary":"The paper introduces COFFAIL, a collection of robot skill episodes performed while making coffee. Most datasets record only successful runs, but this one also captures anomalous executions in which the robot encounters problems. The episodes were gathered on a physical robot in a kitchen, with a few tasks using two arms at once. The authors demonstrate how the combined data can be fed into imitation learning to produce a robot policy. Readers should care because everyday robot use will involve mistakes, and data that shows those mistakes may help policies recover or avoid them.","feed_headline":"Dataset adds robot failure cases to coffee skill training","feed_subtitle":"COFFAIL mixes successful and anomalous executions so imitation learning can produce policies that cope with errors.","key_machinery":"The COFFAIL dataset of mixed successful and anomalous robot skill executions for coffee preparation.","core_discovery":"The COFFAIL dataset comprises successful and anomalous skill execution episodes collected with a physical robot in a kitchen environment for coffee preparation tasks, including a couple performed with bimanual manipulation, and the data is shown to support robot policy learning through imitation learning.","pith_inferences":["The same approach of recording both successes and failures could be repeated for other household tasks such as cooking or cleaning.","The anomalous episodes might also be used to train separate modules that detect when a skill is going wrong.","Evaluating the learned policy on failures that were never seen during training would test whether the dataset generalizes beyond the collected anomalies."],"forward_implications":["Policies trained on the mixed data can be expected to handle errors that arise during coffee-preparation sequences.","The dataset supplies examples for both single-arm and two-arm manipulation within the same task domain.","Data gathered in a real kitchen setting can serve as a starting point for testing policies outside laboratory conditions."],"fun_headline_variants":["COFFAIL dataset of successful and anomalous robot coffee skills","Robot dataset includes successful and failed coffee skill episodes","COFFAIL captures success and anomaly in physical robot coffee tasks","Dataset of robot coffee prep with success and anomaly cases"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The recorded anomalous executions are representative of the failures a robot would meet in actual use, so that imitation learning on the mixed set yields better policies than success-only training.","fun_headline_variants_meta":{"raw":{"variants":["COFFAIL dataset of successful and anomalous robot coffee skills","Robot dataset includes successful and failed coffee skill episodes","COFFAIL captures success and anomaly in physical robot coffee tasks","Dataset of robot coffee prep with success and anomaly cases"]},"model":"grok-4.3","cost_usd":0.016774,"raw_usage":{"total_tokens":7086,"prompt_tokens":538,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":167737000,"prompt_tokens_details":{"text_tokens":538,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":6485,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":538,"tokens_out":63,"duration_ms":90745,"temperature":1.0,"reasoning_tokens":6485,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-10T04:10:25.452349+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Train one imitation-learning policy on only the successful COFFAIL episodes and another on the full set, then measure both policies on the same set of test tasks; if the mixed-data policy shows no higher success rate, the benefit of including anomalies is not demonstrated.","supporting_citations":[],"review_version":1}