{"id":"01e6cfc4-ccd0-49f8-a654-b7f55a76398d","arxiv_id":"2608.08736","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FitAQA introduces a unified-taxonomy benchmark for fitness action quality assessment and shows that current multimodal LLMs perform poorly on perception, judgement, and temporal grounding.","lead":"FitAQA is a new benchmark for judging how well AI models assess exercise form, built from 2,219 videos and 5,512 question-answer pairs across 30 bodyweight exercises with a unified error taxonomy. Current multimodal models score far below expert-level assessment, and the paper's claim that visual perception is the main bottleneck is weakened by a circular experimental setup.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The perception-oracle experiment may leak the judgement label: because the ground-truth perception answer determines the judgement answer by construction, the visual-perception bottleneck conclusion needs a decoupling control.","rationale":"The reader's weakest-assumption analysis identifies the same core issue: the controlled perception-oracle experiment may be circular because the perception answer and the judgement label are tied during construction. I agree that this is the most load-bearing threat to the paper's distinctive bottleneck claim, and that the GT-perception no-video row is the key piece of evidence raising the concern. However, I would phrase the mechanism more cautiously. The injected answer is not literally the judgement label unless the model already knows which option corresponds to correct execution; the option text plus domain knowledge can legitimately imply the label. The concern is therefore not that the result is certainly invalid, but that the design does not establish whether the model is using the semantic content of the perception answer as evidence or exploiting the deterministic option-to-label mapping. The no-video perception controls in Table 7 partially mitigate a pure textual-bias story, but they do not resolve this specific circularity. A decoupling test with free-text observations would settle it. The rest of the benchmark construction, annotation quality, and broad evaluation are solid, and the main 'current models struggle' claim is well supported; the conditional verdict remains appropriate pending the oracle control and the promised data/code release.","tokens_in":26442,"tokens_out":16414,"duration_ms":181867,"concrete_test":"Re-run the controlled oracle with the perception question and multiple-choice options replaced by the corresponding free-text, video-specific observation from the annotation (e.g., 'Both heels lift off the floor') while keeping the judgement question unchanged, and compare the no-video condition against the GT-perception no-video row of Table 2. If F1 stays near 94-96%, the model is reasoning from perceptual evidence and the bottleneck conclusion survives. If F1 drops toward the question-only or no-video baseline, the high GT-perception scores were an artifact of the option-to-label shortcut.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that visual perception is a key bottleneck rests on the controlled perception-oracle experiment (Table 2). The design assumes that injecting the ground-truth perception answer supplies only perceptual evidence and leaves the judgement to domain knowledge. This assumption is not secured by the construction. In the QA-generation pipeline, option A is defined as the correct-execution manifestation, and the judgement answer is derived by mapping option A to Yes and all non-A options to No (Appendix: Question and Answer Generation, Answer Inference). Thus the ground-truth perception answer is a deterministic function of the judgement label: it tells the model which option was observed, and the expert-revised option set is deliberately constructed so that each option maps unambiguously to Yes or No. The GT-perception no-video row of Table 2, reaching 95.8% F1 for GPT-5.5 without any video, shows that the judgement label is recoverable from the injected oracle alone. The paper asserts that the oracle 'only provides perception evidence', but it does not rule out the possibility that the model is exploiting the option-to-label mapping rather than engaging in the intended perception-plus-reasoning decomposition. The main results showing that current MLLMs struggle on FitAQA are less affected, but the more novel bottleneck conclusion is not supported by this experiment as currently designed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FitAQA, a benchmark for evaluating multimodal large language models (MLLMs) on fitness action quality assessment. It contains 2,219 videos and 5,512 QA instances across 30 bodyweight exercises, with a unified taxonomy of 38 form errors in six quality dimensions. The benchmark defines three tasks: perception, judgement, and temporal grounding. The authors evaluate a range of open-source and closed-source MLLMs and report that current models perform weakly, especially on perception and precise temporal localization. A controlled perception-oracle experiment is used to argue that visual perception is a key bottleneck because judgement improves substantially when ground-truth perceptual evidence is supplied.","tokens_in":26751,"tokens_out":7831,"duration_ms":77133,"significance":"If the results stand, FitAQA would be a valuable resource for the MLLM and sports-AI communities. The dataset construction is careful: expert-reviewed annotations, a blinded 10% cross-check with 98.4% exact-answer agreement for perception and 97.3% mIoU for temporal grounding, shuffled option order, and multiple simple baselines (always-correct, always-error, whole-video). The main finding that current MLLMs are weak at fitness AQA and temporal grounding is supported by the reported numbers. However, the more novel bottleneck claim is not supported: the perception-oracle experiment leaks the judgement label through the fixed option-to-answer mapping, and the paper's own no-video rows (Table 7) already show that strong priors can produce high judgement F1 without visual input. The benchmark contribution can be salvaged, but the controlled-experiment conclusion needs to be either fixed with a decoupling control or substantially reframed.","major_comments":[{"comment":"The central bottleneck conclusion rests on comparing baseline judgement with the GT-perception setting. However, during dataset construction the judgement answer is deterministically derived from the perception answer (option A maps to Yes, non-A options map to No; Appendix 'Answer Inference'). The ground-truth perception answer is therefore logically equivalent to the judgement label, and the 'GT-perception, no video' row in Table 2, which reaches 95.8% F1 for GPT-5.5, shows that the model can recover the judgement answer without any video. This means the comparison between baseline and GT-perception does not isolate visual perception; it at least partly measures whether the model exploits the option-to-label mapping. The stress-test concern is valid: the claim that 'visual perception is a key bottleneck' is not supported by this experiment as designed. A decoupling control is needed, for example a perception oracle whose evidence does not uniquely determine the judgement answer, or a human-subject check that the perception options are not semantically transparent about correctness. The paper should also explicitly discuss the no-video GT-perception row as a limitation.","section":"§Controlled Perception-Oracle Evaluation (Table 2) and Appendix 'Answer Inference'"},{"comment":"The no-video controls in Table 7 show that GPT-5.5 achieves a judgement F1 of 68.3% without any video, which is higher than its video-based baseline F1 of 57.8%. The paper attributes this to a prediction bias toward the error class. This prior also contaminates the oracle comparison: in the GT-perception no-video setting, the model can combine its prior with the leaked perception answer and reach 95.8% F1. Before interpreting Table 2 as evidence of a perception bottleneck, the paper must analyze how this strong no-video prior interacts with the oracle manipulation, rather than treating the no-video controls only as a sanity check for perception and grounding tasks.","section":"§Appendix 'Textual Bias Analysis' (Table 7)"},{"comment":"The monotonic relationship in Figure 5 between perception-oracle accuracy and judgement accuracy is presented as evidence of a positive dependence of judgement on perception quality. Because the oracle answers are generated from the same fixed option-to-label mapping, increasing oracle accuracy means the model is increasingly often told the correct judgement label; the monotonicity is therefore partly tautological. A more informative control would vary the correctness of perception evidence that is not label-equivalent, for example by providing a corrupted visual-state description rather than an option letter.","section":"§Controlled Perception-Oracle Evaluation (Figure 5)"}],"minor_comments":[{"comment":"The sentence 'supplying the perception question and its answer only provides perception evidence, leaving the model to reason with domain knowledge' is contradicted by the dataset-construction mapping and should be removed or replaced with a carefully qualified statement.","section":"§Controlled Perception-Oracle Evaluation"},{"comment":"In Table 1, the Qwen3.5-27B row contains the values '37.463.447.1' without separators; the formatting should be corrected to match the other rows.","section":"Table 1"},{"comment":"The main text states that 95 temporal-grounding instances require localizing multiple intervals, but it does not give the denominator or the distribution across the 388 instances; please add this breakdown.","section":"§Dataset Statistics"},{"comment":"The paper evaluates closed-source models at 2 fps and UCF101 at its native 1 fps, while Table 9 studies frame-rate effects only for Qwen3.5-27B; a sentence noting this limitation would help readers interpret the cross-model comparisons.","section":"§Experiments, Evaluated Models"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is solid and the main 'models are weak' result is believable, but the headline bottleneck claim is not supported by the current controlled experiment. I recommend inviting a revision that either adds a proper decoupling control or reframes the contribution around the benchmark itself and the weaker, but still useful, empirical observation that supplying correct perceptual evidence improves judgement. The paper already contains enough material (no-video controls, paired perception-judgement analysis) for an honest revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Readable, honest benchmark paper. The genuinely new piece is the unified 38-error taxonomy across 30 exercises and the decomposition into paired perception/judgement plus temporal grounding. That is a real step beyond action-specific AQA datasets and generic video QA. Construction is careful: expert review, blinded cross-check at 98.4% agreement, shuffled options, MILP rebalancing, and baselines that include always-correct, always-error, and whole-video. The main finding that current MLLMs are weak on fitness AQA is supported by the numbers: perception accuracy hovers near baselines and temporal grounding collapses at strict IoU thresholds.\n\nThe soft spot is the perception-oracle experiment, and it is load-bearing for the paper's more interesting conclusion. Judgement answers are derived from perception answers by a fixed mapping: option A maps to Yes, non-A to No. So when you inject the ground-truth perception answer, you are also injecting the judgement label. The GT-perception no-video row gets 95.8% F1 for GPT-5.5 without any video, which is exactly what you would see if the model is reading the option text and recovering the label rather than performing the intended perception-plus-reasoning task. The paper argues the options are neutral, but neutral only means they avoid the words correct/incorrect; the descriptions themselves still make clear which one is the proper execution. The perception-bottleneck conclusion therefore needs a decoupling control, e.g., varying the mapping between perception options and Yes/No, or providing perceptual evidence that does not semantically contain the standard.\n\nSmaller issues: the always-error baseline achieves 74.2% F1 on judgement, which should temper how the F1 numbers are read; the temporal-grounding subset is small (243 videos); and the data are promised but not yet released. None of these is fatal if the bottleneck claim is fixed.\n\nThe paper deserves serious peer review. The benchmark, if released, will be used. The authors just need to either repair the oracle experiment or soften the claim to \"judgement improves when given the answer to the perception question,\" which is true but less interesting.","headline":"A carefully built fitness-AQA benchmark whose headline bottleneck result is undercut by a circular perception-oracle experiment.","tokens_in":27208,"tokens_out":3301,"would_cite":false,"duration_ms":34716,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current multimodal language models cannot reliably assess the quality of bodyweight exercises, and the paper argues that visual perception — not domain reasoning — is the limiting factor.","keywords":["fitness action quality assessment","multimodal large language models","form error taxonomy","perception bottleneck","temporal grounding","video understanding benchmark","bodyweight exercises"],"falsifier":"Build judgement questions whose correct answer is not determined by the perception option — for example, where correctness depends on a threshold or combination that no single option encodes — and run the condition that gives the perception answer without the video. If judgement F1 still reaches the 94-96% range reported in Table 2, the perception-bottleneck conclusion is an artifact; the existing no-video row already approaches that range with no visual input at all.","tokens_in":26301,"feed_emoji":"🏋️","tokens_out":7789,"duration_ms":68259,"temperature":0.7,"pith_summary":"FitAQA is a benchmark that asks whether multimodal large language models can judge whether bodyweight exercises are performed correctly. The paper builds a unified taxonomy of 38 form errors across six quality dimensions, assembles 2,219 videos and 5,512 QA instances, and separates evaluation into perceiving visual evidence, judging correctness against domain knowledge, and localizing errors in time. The central finding is that current models perform near or below simple baselines on perception and temporal grounding, while judgement improves sharply when ground-truth perception is supplied, leading the authors to conclude that visual perception is the key bottleneck. The benchmark's paired questions are designed so a model's failure can be traced to one of these stages rather than to a single final score.","feed_headline":"MLLMs can't reliably judge exercise form","feed_subtitle":"Best model: 54% on visual cues; 94% on judgement when the perception answer is given.","key_machinery":"The central mechanism is the paired perception and judgement question. For each quality aspect, a perception question asks which neutral visual manifestation is present, with option A defined as the correct-execution manifestation, and the matched judgement question asks whether that aspect is correct, with Yes derived from option A. The paper varies what information the model receives before answering the judgement question — no extra input, the perception question alone, the model's own perception answer, or the ground-truth perception answer — and the jump in judgement performance under ground-truth perception is what identifies perception as the bottleneck. The unified form-error taxonomy, with 38 errors across alignment, symmetry, stability, coordination, tempo, and completeness, is the shared label space that makes this cross-exercise decomposition possible.","core_discovery":"The paper claims that current MLLMs still struggle to assess exercise quality comprehensively and to localize form errors precisely, with visual perception as the key bottleneck. On FitAQA, the best model reaches 54.3% question-macro accuracy on perception, 72.1% F1 on judgement, and 27.0% recall at an IoU threshold of 0.7 on temporal grounding; supplying ground-truth perception answers raises the best judgement F1 to 94.4%. The paper also reports that open-source models are at or below the uniform-random baseline in judgement F1, that models tend to overpredict correct execution, and that temporal grounding degrades sharply under stricter overlap thresholds. These results are presented as evidence that capabilities demonstrated on general video understanding benchmarks do not transfer to recognizing and localizing quality-relevant posture and motion evidence.","pith_inferences":["Editorial inference: the perception-oracle experiment may be confounded, because option A was defined as the correct-execution manifestation and maps directly to the judgement answer 'Yes'; the GT-perception-no-video row in Table 2, which reaches 95.8% F1 without any video, suggests the supplied perception answer already contains the judgement answer.","Editorial inference: a cleaner test of the bottleneck would decouple the perception question from the judgement label, for example by asking for a neutral kinematic quantity (such as knee angle in degrees) and providing the acceptable range only in the judgement stage.","Editorial inference: the per-dimension results suggest a diagnostic route for model improvement: symmetry and load-distribution errors are consistently hard, so targeted training data for those aspects may yield the largest gains.","Editorial inference: the observed over-localization tendency (median predicted duration above ground truth for five of six models) implies that strict IoU thresholds may be punishing systematic temporal over-coverage as much as true localization error, so a calibration-based metric could give a complementary view."],"forward_implications":["If the central finding holds, no evaluated MLLM is ready for unsupervised fitness coaching: each model falls at or below simple baselines on at least one of the three tasks.","Because perception is identified as the bottleneck, improving fine-grained visual evidence recognition — joint angles, motion trajectories, postural deviations — should matter more for fitness AQA than improving reasoning.","General video understanding benchmarks cannot be taken as evidence of quality-sensitive assessment; recognizing an action is different from judging how well it is performed.","Temporal grounding of form errors requires more than coarse action semantics; models locate the exercise but not the specific repetitions that deviate from correct form."],"supporting_citations":[{"why":"Supplies the QEVD-FIT-300k short clips and the QEVD long-video segments (COACH, Benchmark, Competition) that form the majority of FitAQA videos.","marker":"[Panchal et al. 2024]"},{"why":"Supplies EgoExo-Fitness clips used in the short-clip subset.","marker":"[Li et al. 2024b]"},{"why":"Supplies Kinetics-700 generic action-recognition videos that are filtered into fitness clips.","marker":"[Carreira et al. 2019]"},{"why":"Supplies UCF101 image-sequence samples used in the short-clip subset.","marker":"[Soomro, Zamir, and Shah 2012]"},{"why":"The GPT-5.4 model prompt used to generate initial perception and judgement question pairs from annotated descriptions.","marker":"[Singh et al. 2025]"},{"why":"Provides the prior observation that MLLMs tend to overpredict correct execution, which the paper's judgement results align with and build on.","marker":"[e Freitas et al. 2026]"}],"fun_headline_variants":["Perception, not reasoning, blocks MLLM fitness assessment","Fitness AQA: MLLMs fail at seeing form, not judging it","MLLMs can't spot form errors: visual perception bottleneck","New benchmark pinpoints MLLM weakness: exercise perception","Why MLLMs judge exercise wrong: perception first, then judgement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bottleneck conclusion assumes that giving the model the correct perception answer provides only visual evidence and not the judgement answer; but because the correct-execution option was always labeled A and maps directly to 'Yes', the supplied answer may already encode the judgement, so the improvement may come from that leak rather than from better perception.","fun_headline_variants_meta":{"raw":{"variants":["Perception, not reasoning, blocks MLLM fitness assessment","Fitness AQA: MLLMs fail at seeing form, not judging it","MLLMs can't spot form errors: visual perception bottleneck","New benchmark pinpoints MLLM weakness: exercise perception","Why MLLMs judge exercise wrong: perception first, then judgement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":1164,"prompt_tokens":944,"completion_tokens":220,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":130}},"tokens_in":560,"tokens_out":220,"duration_ms":3233,"temperature":1.0,"reasoning_tokens":130,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:25:43.121073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build judgement questions whose correct answer is not determined by the perception option — for example, where correctness depends on a threshold or combination that no single option encodes — and run the condition that gives the perception answer without the video. If judgement F1 still reaches the 94-96% range reported in Table 2, the perception-bottleneck conclusion is an artifact; the existing no-video row already approaches that range with no visual input at all.","supporting_citations":[],"review_version":1}