{"id":"9aff61e4-fd76-4dad-9a0b-7bf2e253a365","arxiv_id":"2509.07994","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 1,000-video benchmark of stroke patients performing box-and-block sub-actions, with raw frames and 2D skeletons, establishes baseline action classification accuracy for seven models.","lead":"The authors introduce StrokeVision-Bench, a dataset of 1,000 short videos of stroke patients performing Box and Block Test sub-actions, each paired with 2D skeleton keypoints. They benchmark seven action recognition models, reporting up to 87.68% accuracy, to seed automated and objective stroke rehabilitation assessment.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No label reliability or pose validation on stroke-impaired movements; the headline 87.68% R3D accuracy may reflect ambiguous annotations or pose artifacts rather than clinically meaningful BBT sub-actions.","rationale":"The reader's weakest assumption correctly identifies the absence of annotation and pose validation as the key load-bearing gap. I agree rather than partially or disagree because the central contribution is explicitly a benchmark, and a benchmark is only as meaningful as its ground truth. The paper honestly reports confusion matrices and standard deviations, and the class-imbalance discussion is transparent, so the concern is not about authorial intent or hidden results; it is about an unverified foundation. The alternative concerns, such as the missing code/data or the dataset-size mismatch, are real but secondary: if labels and skeletons are valid, those issues reduce reproducibility and trust but do not invalidate the measured accuracies; if labels or skeletons are invalid, even a fully released and reproducible benchmark would be worthless. The proposed clinician-kappa and pose-PCK checks are feasible with the existing data and would directly settle whether the transport-class ambiguity and skeleton errors are sufficient to undermine the central claim. Therefore, the reader's CONDITIONAL verdict is appropriate, with the condition being the release of annotation and pose-validation evidence; no verdict change is needed.","tokens_in":9216,"tokens_out":4665,"duration_ms":59126,"concrete_test":"Take a stratified random sample of 200 videos (50 per class), have two independent clinicians who are blind to each other and to model predictions annotate each video using the paper's stated class definitions, and compute Cohen's kappa overall and per class, especially for 'Transport with block' vs 'Transport without block'. Also, on 50 stroke-patient videos, compare Sapiens 2D keypoints (wrist, elbow, shoulder) against manually placed keypoints using PCK/OKS, and test whether skeleton-only clips allow human annotators to distinguish the two transport classes at better than chance. If kappa for the transport pair is below about 0.6 or pose PCK is poor, the benchmark's ground truth and skeleton modality are not validated and the reported accuracies should be treated as unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that StrokeVision-Bench is a usable benchmark of 1,000 clinically structured BBT sub-action videos with raw frames and 2D skeletons, on which R3D reaches 87.68%. This claim stands or falls on the validity of the ground-truth labels and the skeleton modality. Section 3.1 states only that 'Each video was manually annotated into four action classes'; no annotation protocol, inter-annotator agreement, or clinician validation is reported. Section 6, 'Limitations', omits these gaps entirely. The confusion matrices in Fig. 3 show that 'Transport with block' vs 'Transport without block' is the hardest pair for every model (e.g., Video Swin: 57.1%/67.9%; PoseConv3D: 65.6%/63.3%; even the best frame-based R3D confuses them at 6.9% each way). This is exactly the pair where label ambiguity is most plausible: in a 1-second clip, the distinction is object presence in the hand, and the 17-keypoint skeleton modality does not include the block. A model operating on 2D joints can only infer object presence from subtle kinematic correlates; without evidence that these correlates are consistent, the 84.29% MotionBERT result is not interpretable. Separately, Sapiens keypoints are not validated on hemiparetic upper-limb motion, so skeleton-based numbers could reflect pose-estimation artifacts. The dataset count also conflicts with Fig. 2b, which sums to 976 videos, not 1,000, suggesting the dataset inventory itself is not fully reliable. These are not peripheral omissions: if labels are noisy or pose estimates are inaccurate on this population, the reported benchmark accuracies do not measure clinically meaningful action classes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces StrokeVision-Bench, claimed to be the first dedicated dataset of 1,000 one-second videos of stroke patients performing the Box and Block Test, with each sample available as raw RGB frames and 2D skeleton keypoints extracted by Sapiens. The four action classes are Grasping, Non-task Movement, Transport with Object, and Transport without Object. The authors benchmark four video-based models (R3D, R2Plus1D, Video MViT, Video Swin Transformer) and three skeleton-based models (MotionBERT, PoseConv3D, MS-G3D), reporting that R3D achieves the highest accuracy of 87.68%.","tokens_in":9573,"tokens_out":2955,"duration_ms":36676,"significance":"If the dataset is released and the labels are trustworthy, StrokeVision-Bench addresses a real gap: existing stroke rehabilitation video datasets mostly cover daily activities or mix healthy and stroke-impaired subjects, while this work targets a standardized clinical test with a clinically structured sub-action taxonomy. The multimodal raw-frame + 2D-skeleton design is useful for privacy-preserving and joint-level analysis, and the benchmark numbers provide a reasonable first point of comparison for future methods. The paper makes falsifiable quantitative claims (e.g., R3D 87.68%) on a fixed split, so the empirical part is clear. However, the central artifact is a dataset, and the manuscript currently does not supply release access, annotation reliability evidence, clinical metadata, or pose-validation on the target population. These omissions are load-bearing because the benchmark's interpretability depends on the validity of the labels and the skeleton modality.","major_comments":[{"comment":"The abstract and Section 3.1 state that StrokeVision-Bench contains 1,000 videos, but Fig. 2b sums to 976: training counts 148 + 372 + 100 + 159 = 779 and validation counts 38 + 93 + 26 + 40 = 197, giving 779 + 197 = 976. This discrepancy undermines the central inventory claim. Please correct the number or the figure, and state the exact split sizes.","section":"Section 3.1 and Fig. 2b"},{"comment":"No dataset release mechanism is provided: there is no URL, download link, access protocol, or licensing information. For a benchmark paper, the dataset is the contribution, and without a release mechanism the reported results are not reproducible by the community. Also, the paper does not mention IRB/ethics approval or consent procedures for patient video data, which is essential for a clinical dataset.","section":"Section 3.1 and Section 6"},{"comment":"The annotation protocol is described only as 'Each video was manually annotated into four action classes.' There is no information about the number of annotators, their clinical background, inter-annotator agreement, or adjudication of disagreements. This is especially concerning because Fig. 3 shows that 'Transport with block' vs. 'Transport without block' is the most confused pair for every model (e.g., R3D confuses them 6.9% each way; Video MViT gets 57.1% on 'without block'). In 1-second clips, the distinction may rest on subtle object presence, and without label-reliability evidence the headline 87.68% cannot be interpreted as clinically meaningful discrimination. Additionally, no clinical metadata (number of patients, sessions, pre/post pairing, severity) is reported, so the 'tracking recovery' claim is not yet supported.","section":"Section 3.1 and Fig. 3"},{"comment":"The 2D skeleton modality is generated by Sapiens, but the paper does not validate pose accuracy on stroke-impaired upper-limb movements. Hemiparetic motion, compensatory trunk movements, and atypical postures can degrade keypoint estimation, yet no PCK/OKS-style evaluation or visual audit is reported. Moreover, the 17-keypoint skeleton does not include hand/object keypoints, so skeleton-based methods such as MotionBERT (84.29%) can only infer block presence through indirect kinematic cues. Without a pose-quality check on this population, the skeleton-based benchmark results are not yet interpretable as measuring the intended sub-actions. The limitations section should explicitly address this, but currently Section 6 omits it.","section":"Section 3.2 and Table 1"}],"minor_comments":[{"comment":"The claim 'first-ever dedicated dataset' and 'first stroke rehabilitation benchmarking dataset that focuses exclusively on stroke patients' should be supported by a more systematic comparison with prior datasets, especially StrokeRehab [12] and [23], to clarify what exactly is new (BBT sub-actions vs. daily activities).","section":"Section 1 (contributions)"},{"comment":"The sentence 'By including recordings before and after the sessions, our dataset captures changes in movement speed' is vague. Are these paired recordings of the same patient? If yes, the patient-level split should be described; if no, the claim about tracking recovery is not directly supported.","section":"Section 3.1"},{"comment":"Implementation details are minimal: no learning rate, batch size, number of epochs, input resolution, or frame sampling strategy is reported. This makes the baselines hard to reproduce even if the dataset were available.","section":"Section 4"},{"comment":"Reference [14] for MViTv2 is formatted as an arXiv preprint but lacks the arXiv ID; reference [15] cites the Swin Transformer paper and [16] the Video Swin Transformer paper, but in Section 5 the text cites [15] for Video Swin Transformer, which may confuse readers.","section":"References"},{"comment":"The confusion matrices show percentages but it is not stated whether rows are normalized per true class; the caption says 'accuracy' but the percentages appear to be row-normalized. Please clarify the normalization in the caption.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"This paper is short for a dataset contribution and currently lacks the most critical component: the dataset itself is not accessible. Even if the count discrepancy is fixed, I would not recommend acceptance unless the authors provide a release URL, annotation reliability metrics, and at least a basic pose-accuracy check on stroke-impaired subjects. The topic fits the workshop/journal scope, but the artifact needs to be verifiable. The 'first-ever' claim also deserves a careful literature check before final publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset idea is genuinely new and the benchmark numbers are honestly reported, but the paper does not yet support its own claims. The abstract says 1,000 videos; Figure 2b sums to 976. There is no data release URL, no annotation protocol, no inter-annotator agreement, no clinical metadata, and no validation of the Sapiens 2D pose on hemiparetic movement. Those gaps are real and load-bearing for a benchmark.\n\nWhat is good: relative to the cited literature, this is the first dataset that focuses exclusively on stroke patients performing Box and Block Test sub-actions, with both raw frames and 2D skeletons. StrokeRehab covers daily activities with mixed healthy/stroke subjects, and the other cited study uses healthy subjects only. The benchmarking is straightforward: externally pretrained models on a fixed train/validation split, confusion matrices for all seven models, no quantity fitted to the target outcome. So the accuracy figures mean what they appear to mean, assuming the labels are trustworthy. The authors also honestly note that the two transport classes are the hard pair; that is visible in every confusion matrix.\n\nWhere it gets soft: the label validity issue is severe for a benchmark. 'Transport with block' vs 'Transport without block' in a one-second clip is an object-presence distinction, and the skeleton modality has no block information. Without clinician validation or at least a second annotator, the 87.68% R3D number could be separating annotation heuristics rather than clinically meaningful sub-actions. The skeleton modality is never validated on this population either; Sapiens was not built for hemiparetic upper-limb motion. Section 6 lists limitations but omits these entirely, which is a gap in the paper's own self-assessment. The recovery-tracking claim also overreaches: the paper says recordings are pre/post session, but no patient-level metadata or paired session structure is provided, so the dataset as described cannot support a recovery-tracking study. The count discrepancy is likely a typo, but for a dataset paper it makes you worry about the inventory. The references also include several off-topic self-citations and one duplicate cite; minor, but sloppy.\n\nVerdict: worth a referee, and in fact a good test for the reviewers: require the authors to release the data, correct the count, and provide annotation reliability and pose-validation numbers. If those come back clean, this becomes a useful small benchmark. For now I'd skim it if rehab informatics is in your line, but not cite it.","headline":"Genuinely new stroke BBT dataset, but not yet a usable benchmark: count mismatch, no data release, no label or pose validation.","tokens_in":10064,"tokens_out":2802,"would_cite":false,"duration_ms":31074,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StrokeVision-Bench introduces a 1,000-video dataset of stroke patients performing the Box and Block Test, in raw frames and 2D skeletons, and reports that the best model, R3D, reaches 87.68% accuracy.","keywords":["stroke rehabilitation","Box and Block Test","action recognition","2D pose estimation","clinical video dataset","upper extremity assessment","skeleton-based action recognition","benchmarking"],"falsifier":"Have two independent clinicians label a held-out subset of the same 1,000 videos into the four sub-actions; if their agreement on 'transport with block' versus 'transport without block' is close to chance, the benchmark accuracies are inflated by label ambiguity. Separately, compare Sapiens 2D keypoints on stroke-impaired arms against manual joint annotations or motion capture; large keypoint error would make the skeleton-based results uninterpretable clinically.","tokens_in":9122,"feed_emoji":"🧠","tokens_out":6908,"duration_ms":74323,"temperature":0.7,"pith_summary":"StrokeVision-Bench tries to establish that a small, clinically structured video dataset of stroke patients can support objective, computer-vision-based tracking of upper-extremity recovery. It contributes 1,000 one-second videos, each labeled as one of four Box and Block Test sub-actions, with both raw frames and 17-joint 2D skeletons. The benchmark results show that frame-based 3D CNNs classify these sub-actions most accurately (R3D at 87.68%), that skeleton-based models such as MotionBERT come close (84.29%) while preserving privacy, and that the main remaining difficulty is distinguishing 'transport with block' from 'transport without block.' If the dataset and labels hold up, this gives clinicians an affordable, camera-only route toward progress monitoring in rehabilitation.","feed_headline":"A 1,000-video benchmark tests AI tracking of stroke recovery","feed_subtitle":"First stroke-only Box and Block Test video benchmark; best model hits 87.68% accuracy.","key_machinery":"The central object is the four-class sub-action taxonomy of the Box and Block Test: Grasping, Non-task Movement, Transport with Object, and Transport without Object. Each video is cut to one second and 30 frames, and is delivered in two modalities: raw RGB frames and Sapiens-derived 2D skeletons with 17 keypoints. The taxonomy turns a standard clinical test into a video classification problem, while the dual modalities let the benchmark compare frame-based and pose-based pipelines. The pre/post-session recording design is intended to capture movement-speed changes that indicate recovery progress, making the dataset a tool for longitudinal monitoring rather than only single-session recognitio","core_discovery":"The paper's central claim is that StrokeVision-Bench is the first dedicated dataset of stroke patients performing a clinically structured block-transfer task, and that standard action-recognition models can classify the four sub-actions well enough to be useful for automated rehabilitation assessment. Each of the 1,000 videos is labeled as Grasping, Non-task Movement, Transport with Object, or Transport without Object, and comes in two modalities: raw RGB frames and 2D skeletons extracted with the Sapiens pose estimator. Benchmarking seven models, the authors report that R3D achieves 87.68% accuracy, R2Plus1D 86.96%, and MotionBERT 84.29%, while vision transformers fall to the mid-to-high 70","pith_inferences":["The paper reports no inter-annotator agreement, so a natural next step is to measure how consistently human labelers separate the four sub-actions; if clinicians also struggle with the two transport classes, part of the benchmark confusion is label ambiguity rather than model failure.","No validation of the Sapiens skeletons on hemiparetic arms is reported; comparing those keypoints against manual annotation or motion capture on stroke-impaired movement would show whether skeleton-based accuracy reflects true motion or pose-estimation bias.","A direct extension the authors mention but do not build is regressing the standard Box and Block Test outcome (blocks transferred per minute) or shoulder-abdomen joint angles from the same videos, which would connect the benchmark to established clinical scores.","The pre/post-session recordings invite a longitudinal study of the same patients across weeks; the paper describes the dataset as enabling recovery tracking but does not evaluate whether model predictions change in ways that align with clinical improvement."],"forward_implications":["Clinics could use a single camera to automatically tag Box and Block Test sub-actions, since R3D reaches 87.68% accuracy without wearable sensors or specialized equipment.","For counting block transfers, frame-based CNNs are the best current option; for joint-level mobility assessment, skeleton-based MotionBERT is nearly as accurate and keeps raw video private.","Pretrained vision transformers are a weak fit for small clinical datasets, losing roughly ten points to CNNs and skeleton-based models.","The persistent confusion between 'transport with block' and 'transport without block' is the main technical barrier, and will need better modeling, more balanced data, or more refined labels before subtle recovery changes can be tracked reliably."],"supporting_citations":[{"why":"StrokeRehab, the prior stroke action-recognition dataset this work contrasts with, supplying the comparison showing existing datasets focus on daily activities rather than clinically structured block-transfer tasks.","marker":"[12]"},{"why":"Sapiens pose estimator, used to generate the 2D skeleton modality for every video in the benchmark.","marker":"[13]"},{"why":"Source of the R3D and R(2+1)D CNN architectures, providing the top-scoring frame-based baselines.","marker":"[33]"},{"why":"MotionBERT, the best-performing skeleton-based baseline at 84.29%, used to support the claim that pose-driven methods are a privacy-preserving alternative.","marker":"[40]"},{"why":"PoseConv3D skeleton action-recognition baseline, also cited for spatio-temporal CNN design.","marker":"[5]"},{"why":"MS-G3D skeleton action-recognition baseline, showing how a graph-based model without Kinetics pretraining performs on the benchmark.","marker":"[17]"},{"why":"Video Swin Transformer baseline, used to support the claim that vision transformers underperform on small clinical datasets.","marker":"[16]"}],"fun_headline_variants":["First stroke-patient video benchmark for Box and Block Test","New dataset tracks stroke recovery with video and pose","AI benchmark for stroke rehab hits 87.68% accuracy","Multimodal stroke recovery benchmark: video + 2D pose","First dataset for stroke block transfer test with AI baselines"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the manual labels separating the four sub-actions, especially 'transport with block' versus 'transport without block,' and the automatically extracted 2D skeletons are accurate on stroke-impaired movement, so the reported accuracies describe real clinical differences rather than label ambiguity or pose artifacts.","fun_headline_variants_meta":{"raw":{"variants":["First stroke-patient video benchmark for Box and Block Test","New dataset tracks stroke recovery with video and pose","AI benchmark for stroke rehab hits 87.68% accuracy","Multimodal stroke recovery benchmark: video + 2D pose","First dataset for stroke block transfer test with AI baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000329,"raw_usage":{"total_tokens":1692,"prompt_tokens":782,"completion_tokens":910,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":829}},"tokens_in":526,"tokens_out":910,"duration_ms":8183,"temperature":1.0,"reasoning_tokens":829,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:26:10.407698+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent clinicians label a held-out subset of the same 1,000 videos into the four sub-actions; if their agreement on 'transport with block' versus 'transport without block' is close to chance, the benchmark accuracies are inflated by label ambiguity. Separately, compare Sapiens 2D keypoints on stroke-impaired arms against manual joint annotations or motion capture; large keypoint error would make the skeleton-based results uninterpretable clinically.","supporting_citations":[{"cited_title":"Revisiting skeleton-based action recognition","cited_arxiv_id":null,"evidence_quote":"StrokeRehab, the prior stroke action-recognition dataset this work contrasts with, supplying the comparison showing existing datasets focus on daily activities rather than clinically structured block-transfer tasks."},{"cited_title":"Quantitative assessment of limb position sense following stroke","cited_arxiv_id":null,"evidence_quote":"Sapiens pose estimator, used to generate the 2D skeleton modality for every video in the benchmark."},{"cited_title":"A closer look at spatiotemporal convolutions for action recognition","cited_arxiv_id":null,"evidence_quote":"MotionBERT, the best-performing skeleton-based baseline at 84.29%, used to support the claim that pose-driven methods are a privacy-preserving alternative."},{"cited_title":"Non-task movement","cited_arxiv_id":null,"evidence_quote":"PoseConv3D skeleton action-recognition baseline, also cited for spatio-temporal CNN design."},{"cited_title":"Case report of modified box and blocks test with motion capture to measure prosthetic function","cited_arxiv_id":null,"evidence_quote":"MS-G3D skeleton action-recognition baseline, showing how a graph-based model without Kinetics pretraining performs on the benchmark."}],"review_version":1}