{"id":"a6041d69-f1fa-4c5c-a57f-44d772a82f13","arxiv_id":"2506.06174","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An online mistake detector built from a ViT, Q-Former, and LLM achieves second place on HoloAssist with F1 55.0 and generates error explanations.","lead":"This technical report describes a video system that detects mistakes in egocentric video in real time and uses a large language model to write explanations of each error. The system placed second on the HoloAssist mistake detection benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central contribution—handling both procedural and execution errors—is never directly evaluated: Table 1 reports only aggregate binary correct/mistake metrics, so the error-type claim is an interpretation, not a measured result.","rationale":"I focused on the error-type claim rather than the reader's explanation-alignment concern because the paper's title and primary task are mistake detection; the LLM explanation is a secondary feature. The explanation concern is valid and is already captured by the reader's conditional verdict. However, the stronger unaddressed assumption is that the model actually handles the two error categories advertised. A per-type breakdown is the single test that would settle whether the central contribution is real. If the split shows no execution-error signal, the contribution reduces to a generic binary detector and the paper should be revised. The second-place leaderboard claim remains plausible but still needs code and a leaderboard snapshot; this does not change the conditional verdict.","tokens_in":4228,"tokens_out":9352,"duration_ms":100570,"concrete_test":"Use the HoloAssist validation set's per-mistake error descriptions to label each mistake as procedural (ordering/precondition) or execution (motor slip, tool misuse, spatial execution), then recompute the Section 3.1 F1/precision/recall separately for the two subsets. If execution-error recall/F1 is not clearly above a majority-'correct' baseline or is far below procedural-error performance, the claim of handling execution errors is unsupported, and the paper should be revised to claim only procedural-mistake detection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 1 frame the main contribution as an online detector that handles both procedural errors and execution errors (motor slips, tool misuse). The architecture in Section 2.1 is a Video Q-Former plus a linear classification layer; no component models error type, and no experiment measures performance by error type. Table 1 reports only an aggregate F1 and binary per-class precision/recall for correct vs. mistake. With Ours at 11.0 mistake precision and 21.0 mistake recall, the detector is very conservative: a model that rarely says 'mistake' can obtain the reported F1 while missing most execution errors. The conclusion's statement that the method 'detects execution and procedural errors' is therefore not supported by the evidence in the paper. HoloAssist includes detailed error descriptions (Section 3.2), so a per-type evaluation is feasible and should be required.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This technical report describes an online mistake detection system for egocentric procedural videos, consisting of a video encoder with a Video Q-Former and a linear classification head, together with an LLM-based explanation module that is gated on the detection score. The method is evaluated on the HoloAssist benchmark, reporting a second-place F1 score of 55.0 on the official test server, and the explanation module is compared against existing video LLMs using BLEU, ROUGE-L, and CIDEr. The paper claims that the approach detects both procedural errors (ordering, missing actions) and execution errors (motor slips, tool misuse), and that it generates explanations for detected mistakes.","tokens_in":4426,"tokens_out":4251,"duration_ms":38434,"significance":"The official server result is a credible empirical anchor: the method outperforms the RGB-only TimeSformer baseline and improves over GazeCompl by 3.8% F1 without using gaze, placing second on the HoloAssist leaderboard. If the explanations are genuinely grounded in the video content, the system would be a useful component for interactive assistants. However, the significance of the central claim—that the method handles execution errors in addition to procedural errors—is not established by the reported experiments, because Table 1 only reports aggregate binary metrics and the architecture does not explicitly model error types. The explanation evaluation also lacks essential details, making the reported gains difficult to interpret. Strengths include the use of the official held-out test set and the inclusion of qualitative results, but the manuscript is too sparse to fully substantiate its claims.","major_comments":[{"comment":"The claim that the method 'detects execution and procedural errors' is not directly evaluated. Table 1 reports only aggregate binary correct/mistake metrics; no breakdown by error type is provided, even though the HoloAssist dataset contains detailed error descriptions (cited in Section 3.2). Since the paper's stated contribution is to handle both procedural and execution errors, a per-type evaluation is necessary to support this claim. Without it, the statement is an interpretation of the aggregate result, not a measured finding.","section":"Abstract; Section 4"},{"comment":"The gating threshold τ is never specified, and the projection layer that maps Video Q-Former features into the LLM embedding space is described only as 'a linear transformation' with no explanation of whether it is trained, how it is trained, or what loss is used. The explanation quality depends critically on this projection being semantically aligned with the LLM's text space; if the projection is random or unaligned, the reported BLEU/ROUGE/CIDEr values could reflect surface similarity rather than genuine error understanding. Please report the threshold value, the training procedure for the projection (if any), and an analysis of the alignment, such as nearest-neighbor distances between projected visual features and text embeddings.","section":"Section 2.2, Eq. (1)"},{"comment":"The method's mistake-class precision is 11.0 and recall is 21.0, while the correct-class precision and recall are 96.0 and 91.0. This means the detector is very conservative and flags few mistakes; its overall F1 of 55.0 is dominated by the easy correct class. This makes the claim of 'handling' both procedural and execution errors especially problematic, because the detection of actual mistakes is weak. The paper should report the operating threshold used for detection, present a precision-recall curve across thresholds, and discuss the trade-off between correct-class accuracy and mistake recall.","section":"Section 3.1, Table 1"},{"comment":"The explanation evaluation lacks critical details needed for reproducibility and interpretation: the specific LLM used is not named, the prompt template is not given, the configurations of the comparison baselines (e.g., whether Video-LLaMA and Video-ChatGPT are zero-shot or fine-tuned) are not described, and the number of explanation samples evaluated is not reported. Since the gating mechanism may activate only for a small subset of detected mistakes, the reported scores could be based on very few examples. Please provide these details and the number of evaluated explanations.","section":"Section 3.2, Table 2"}],"minor_comments":[{"comment":"The dataset name is written inconsistently as 'Holo Assist' in some places (e.g., Table 1 and Section 3.1) and 'HoloAssist' in others; please use a single spelling.","section":"Throughout"},{"comment":"The column headers 'Prec' and 'Rec' under 'Correct' and 'Mistake' are visually confusing after typesetting; please add clear subheadings or separate columns for correct precision, correct recall, mistake precision, and mistake recall.","section":"Table 1"},{"comment":"The hyperparameters ts, tq, d1, and d2 are defined formally but their concrete values are never given. Since the method is claimed to be online, please report the segment length ts and the number of learnable queries tq used in the experiments.","section":"Section 2.1"},{"comment":"The caption states that 'double slashes indicate layers without backpropagation,' but it is unclear which layers have double slashes and whether the projection layer in the explanation path is frozen or trainable; please clarify in the caption or the main text.","section":"Figure 1"},{"comment":"The sentence about the dataset's 'frequent background segments and mix of procedural and execution errors highlight the robustness of our approach' is not supported by the results shown; either add an analysis that connects these dataset properties to the method's behavior or temper the claim.","section":"Section 3.1"},{"comment":"Reference [8] is cited as 'GazeCompl (CVPR24)' but the bibliography entry lists an arXiv preprint; please update the reference if the paper has been published.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads like a short challenge report rather than a full journal paper. The official leaderboard result is a strong point, but the main novelty claim (handling execution errors) is not experimentally substantiated, and the explanation module is under-specified. I recommend major revision to require the missing per-type evaluation, threshold value, and training details. If the authors cannot provide the per-type breakdown, the paper's scope should be narrowed to what the experiments actually support: an online binary mistake detector with an explanation component, without asserting error-type coverage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a challenge writeup that earns a second-place F1 of 55.0 on the official HoloAssist server using RGB only, and it shows a CIDEr improvement on the explanation task. The architecture is an adaptation of BLIP-2's Q-Former and VideoLLaMA, and the report is honest about that lineage. What's actually new is the specific combination: a Video Q-Former feeding a binary mistake classifier, with a gating mechanism that routes features to an LLM only when a mistake is predicted. That is a legitimate engineering contribution, and the benchmark result is independently checkable through the official server. The paper also compares fairly against TSformer and GazeCompl, and it doesn't oversell the architectural novelty.\n\nThe soft spots are real but not fatal. The abstract and conclusion claim the method handles both procedural and execution errors, yet no experiment measures error types. Table 1 reports only aggregate correct/mistake precision and recall. With mistake precision at 11 and recall at 21, the detector is very conservative and may be missing most execution errors. HoloAssist provides error descriptions, so a per-type breakdown is feasible; its absence undermines the main contribution claim. The explanation pipeline is also thinly described: a linear projection of Q-Former features into LLM embedding space with no training or alignment details, and the gating threshold tau is never specified. The BLEU/ROUGE/CIDEr scores could partly reflect surface similarity rather than genuine error understanding. Hyperparameters like segment length and query count are also missing. These omissions limit verification but are typical for a short technical report.\n\nThe central empirical claim—second place on the official leaderboard—is plausible and independent of the method's own training data, so circularity is not a serious concern. The bigger issue is that the paper's broader interpretive claims extend beyond what the data show.\n\nWho is this for? People working on egocentric procedural assistance or online mistake detection. It is a useful data point, not a methodological breakthrough. It deserves a serious referee: the benchmark result is worth checking, and a referee could push the authors to release code and add the missing per-type and threshold details. My recommendation is to treat this as a borderline accept for a workshop or short-paper venue, with revisions required; it is too thin for a main-conference paper without those details.","headline":"A workmanlike challenge report with a credible second-place F1 on HoloAssist, but the headline claim about handling execution errors is not actually measured.","tokens_in":4941,"tokens_out":2105,"would_cite":false,"duration_ms":21694,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An online egocentric video model detects both procedural and execution mistakes and generates LLM explanations, placing second on the HoloAssist benchmark.","keywords":["egocentric video","online mistake detection","procedural errors","execution errors","HoloAssist","vision-language models","error explanation","Q-Former"],"falsifier":"Replace the learned linear projection with a fixed random projection of the same output dimension and regenerate explanations; if BLEU, ROUGE, and CIDEr stay essentially unchanged, then the reported explanation scores do not show that the LLM is using the video-derived features. A second check is to feed features from correctly executed segments through the same gated path: if the LLM still produces confident mistake explanations, the explanation is driven by the prompt rather than by the detected error.","tokens_in":4046,"feed_emoji":"🎥","tokens_out":9179,"duration_ms":82631,"temperature":0.7,"pith_summary":"This report tries to establish that online egocentric mistake detection can cover both procedural errors, such as wrong ordering of actions, and execution errors, such as motor slips or tool misuse, rather than only sequencing mistakes. The proposed system processes a continuous egocentric video stream, detects a mistake as soon as the relevant visual features pass a threshold, and then uses a large language model to generate a text explanation of what went wrong. The authors report an F1 score of 55.0 on the HoloAssist mistake detection benchmark, which places the method second, and explanation-generation scores that match or exceed prior video-LLM baselines on BLEU, ROUGE-L, and CIDEr. If the claim holds, real-time assistants could tell an operator not just that something failed but why, and how to correct it.","feed_headline":"One video model catches wrong-order and wrong-execution mistakes","feed_subtitle":"It also writes LLM explanations for errors and ranks second on HoloAssist with F1 55.0.","key_machinery":"The load-bearing component is the Video Q-Former, a temporal extension of the BLIP-2 Q-Former built on a BERT encoder. It takes per-frame Vision Transformer features for a segment $[v_{t-t_s}, \\ldots, v_t]$ and, together with learnable queries $q \\in \\mathbb{R}^{t_q \\times d_2}$, compresses them into temporally aware features $f \\in \\mathbb{R}^{t_q \\times d_2}$. Those features feed a mistake classification layer that outputs logits $m \\in \\mathbb{R}^{t_q}$; a gating mechanism $g_m(\\sigma(m))$ lets the features pass to a linear projection layer only when the sigmoid of the predicted logit exceeds a threshold $\\tau$, and the projected features, together with a prompt, are passed to an LLM to generate the explanation. This design ties explanation generation directly to the same temporal video representation used for detection.","core_discovery":"The paper's central claim is that a single architecture can detect both procedural and execution errors from egocentric video and then generate useful text explanations for the detected mistakes. Visually, the method feeds frame segments from a Vision Transformer encoder into a Video Q-Former that produces temporally aware query features; those features go to a mistake classification layer, and when a sigmoid-gated score crosses a threshold, the same features are linearly projected into an LLM embedding space to produce an explanation. On the HoloAssist benchmark, the approach reaches F1 55.0 on the mistake detection task, outperforming the RGB-only TimeSformer baseline and improving on the gaze-based GazeCompl method without using eye gaze. On the explanation task, it achieves the highest CIDEr score among the compared methods (0.76) while matching or exceeding BLEU and ROUGE-L baselines.","pith_inferences":["The paper's explanation metrics do not yet show whether a user who reads an LLM explanation corrects the mistake faster or more reliably; a user study comparing flagged-only versus explained feedback would test that directly.","The reported mistake-class recall is low, so a practical extension would be to add gaze or hand cues to the gated projection to raise recall on short motor slips, a direction the paper itself mentions as future work.","A random-projection ablation would reveal how much of the explanation quality comes from the video features versus the LLM's prior text style, since the projection layer is never trained.","The Video Q-Former plus gated-LLM design should transfer to other procedural egocentric benchmarks, such as cooking or assembly tasks, if those datasets contain execution errors as well as ordering errors."],"forward_implications":["Real-time egocentric assistants could flag both wrong-order and wrong-execution errors, so users can correct mistakes as they happen rather than after the fact.","An explanation is generated only when a mistake logit crosses the threshold, so the system can offer immediate, targeted feedback without interrupting normal task execution.","Because the detection pipeline does not require eye gaze, it could run on ordinary RGB headset cameras and still beat gaze-based methods on the HoloAssist benchmark.","The approach closes part of the gap between pure action recognition and actual task assessment, since it judges how an action is performed, not only what action occurs."],"supporting_citations":[{"why":"It supplies the HoloAssist egocentric dataset, mistake annotations, ground-truth error descriptions, and the benchmark server used for evaluation.","marker":"[14]"},{"why":"It supplies the BLIP-2 Q-Former architecture that the paper extends into a Video Q-Former for temporal feature extraction.","marker":"[6]"},{"why":"It supplies the RGB-only TimeSformer baseline that the method's detection results are compared against.","marker":"[1]"},{"why":"It supplies the gaze-based GazeCompl method whose detection F1 the approach improves on without using eye gaze.","marker":"[8]"},{"why":"It serves as a baseline for explanation generation and as the source of the temporal Q-Former adaptation.","marker":"[16]"},{"why":"It serves as a video-LLM baseline for the explanation-generation comparison in Table 2.","marker":"[7]"},{"why":"It supplies prior online mistake detection work focused on procedural errors, defining the gap the paper addresses.","marker":"[4]"},{"why":"It supplies another prior online mistake detection method that the paper positions its broader error-type coverage against.","marker":"[11]"}],"fun_headline_variants":["Video model flags order and execution mistakes in real time","Egocentric AI catches wrong order and wrong execution errors","Second-place video model detects mistakes and writes feedback","One model handles procedural and execution errors in egocentric video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The explanation pipeline assumes that a linear projection of the Video Q-Former features into the LLM embedding space, with no described training or alignment, is enough for the LLM to generate correct mistake explanations.","fun_headline_variants_meta":{"raw":{"variants":["Video model flags order and execution mistakes in real time","Egocentric AI catches wrong order and wrong execution errors","Second-place video model detects mistakes and writes feedback","One model handles procedural and execution errors in egocentric video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3448,"prompt_tokens":827,"completion_tokens":2621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":2558}},"tokens_in":443,"tokens_out":2621,"duration_ms":17916,"temperature":1.0,"reasoning_tokens":2558,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:59:30.168387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the learned linear projection with a fixed random projection of the same output dimension and regenerate explanations; if BLEU, ROUGE, and CIDEr stay essentially unchanged, then the reported explanation scores do not show that the LLM is using the video-derived features. A second check is to feed features from correctly executed segments through the same gated path: if the LLM still produces confident mistake explanations, the explanation is driven by the prompt rather than by the detected error.","supporting_citations":[{"cited_title":"Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world","cited_arxiv_id":null,"evidence_quote":"It supplies the HoloAssist egocentric dataset, mistake annotations, ground-truth error descriptions, and the benchmark server used for evaluation."},{"cited_title":"Is space-time attention all you need for video understanding? InICML, page 4, 2021","cited_arxiv_id":null,"evidence_quote":"It supplies the RGB-only TimeSformer baseline that the method's detection results are compared against."},{"cited_title":"Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activities","cited_arxiv_id":"2406.08379","evidence_quote":"It supplies the gaze-based GazeCompl method whose detection F1 the approach improves on without using eye gaze."},{"cited_title":"Video-llama: An instruction-tuned audio-visual language model for video un- derstanding","cited_arxiv_id":null,"evidence_quote":"It serves as a baseline for explanation generation and as the source of the temporal Q-Former adaptation."},{"cited_title":"Prego: online mistake detection in procedural ego- centric videos","cited_arxiv_id":null,"evidence_quote":"It supplies prior online mistake detection work focused on procedural errors, defining the gap the paper addresses."},{"cited_title":"Differentiable task graph learning: Procedural ac- tivity representation and online mistake detection from ego- centric videos","cited_arxiv_id":null,"evidence_quote":"It supplies another prior online mistake detection method that the paper positions its broader error-type coverage against."}],"review_version":1}