{"id":"80ecdbbe-3fa5-4dd9-9b2d-b25ef7fc7989","arxiv_id":"2411.09821","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Automated classification of infant general movement quality from uncontrolled home and hospital videos yields AUROC values around 0.7 to 0.8, but the small dataset and ambiguous validation protocol limit the strength of the claim.","lead":"This paper tests whether machine learning can classify infant general movements from home and hospital videos that were not recorded under controlled conditions. It finds that keypoint tracking plus simple classifiers reaches moderate accuracy, suggesting automated newborn screening may eventually be feasible.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline AUROC 0.8283 is absent from the main results: it traces to the best of three manual labellers in Appendix B (Table 4), not the aggregate or the fully automatic AggPose pipeline, so the central feasibility claim rests on a selected scenario.","rationale":"The reader's weakest assumption (clip-level label noise from single video-level labels) is real and explicitly acknowledged in Section 5, but it applies uniformly to all configurations and would not by itself explain the gap between the headline and the tables. The more load-bearing issue is that the paper's central number is not derivable from the main results: the strongest result in Table 3 is 0.77 for the semi-manual Label & Track early group and 0.72 for the fully automatic AggPose late group, while 0.8283 appears only as Labeller #1's rounded AUROC in Table 4. Since Table 3 deliberately averages the three labellers, the conclusion effectively discards the paper's own aggregation rule to cite the best individual. This is not a question of author intent; it is a straightforward check of the reported evidence. It also matters for the 'scalable newborn screening' framing: the configuration reaching about 0.83 requires a human to label keypoints and manually correct tracking outliers per video, whereas the automatic route performs worse. The proposed test (recompute aggregate and automatic maxima) would settle whether the headline number survives. If it does not, the verdict remains conditional, but the condition should include corrected, de-selected reporting of results rather than only finer annotations. The validation-protocol ambiguity is a separate concern that the public code can also resolve, but the missing provenance of 0.8283 is the most direct challenge to the central claim.","tokens_in":14790,"tokens_out":7366,"duration_ms":72226,"concrete_test":"Trace the provenance of 0.8283 in the public code: recompute the mean and standard deviation over the three labellers for every model, feature set, and age group in Table 4, and compare that aggregate best AUROC to 0.8283. Separately, run the identical 5-fold CV pipeline using only AggPose keypoints (no manual labelling or outlier correction) and report the best AUROC across models. If 0.8283 occurs only for one labeller and the aggregate/automatic maxima are 0.77/0.72, the conclusion's feasibility claim should be restated to name the manual, labeller-specific setting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The conclusion (Section 5) anchors the central claim on an AUROC 'of up to 0.8283', but Table 3 reports best aggregate AUROCs of 0.77 (Early GM, Label & Track, angles, RF) and 0.72 (Late GM, AggPose, angles, CNN). The 0.8283 value is the rounded 0.83 ± 0.13 shown only for Labeller #1 in Table 4 (Early GM, Label & Track, angles, RF). That is one of three manual labellers whose results are averaged as 'Mean' in Table 3, so the headline selects the most favorable manual keypoint set rather than the pipeline's aggregate performance. The same configuration also requires per-video manual keypoint labelling and two rounds of outlier correction (Section 3.2), so it does not support the stronger 'fully automatic' or 'scalable' reading of the claim; the fully automatic AggPose route peaks at 0.72. Because the maximum is taken over 3 labellers × 3 models × 3 feature sets × 2 age groups, the reported 'up to' value is expected to be inflated by selection, and no multiple-comparison control is described. This is load-bearing: the number that carries the central claim is not produced by the main evaluation protocol and is not reproducible from the aggregate tables.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a pipeline for automated General Movement Assessment (GMA) from infant videos recorded in uncontrolled settings. The pipeline extracts keypoints either by manual labelling in one frame plus TAPIR tracking (Label & Track) or fully automatically with AggPose, preprocesses the videos (resampling, cropping, outlier correction, fixed-length clipping), computes coordinate and angle features, and trains 1D-CNN, LSTM, and random forest classifiers. The evaluation uses a dataset of 76 infants split into early and late GM groups, with video-level binary labels, and reports AUROC, AUPRC, and accuracy under 5-fold stratified cross-validation. The central claim, stated in the conclusion, is that infant motor quality can be predicted from video recordings with an AUROC of up to 0.8283, supporting the feasibility of automated GMA in real-world settings.","tokens_in":15111,"tokens_out":3307,"duration_ms":34175,"significance":"If the central claim holds, this is a useful preliminary contribution toward scalable newborn screening: it demonstrates that a combination of existing pose estimation/tracking tools and simple time-series classifiers can operate on heterogeneous, clinically realistic video data. The paper has concrete strengths: the code is publicly available, the dataset is genuinely uncontrolled (varied devices, settings, ages), the comparison between manual tracking and fully automatic pose estimation is informative, and the limitations are acknowledged candidly, including the small dataset, label noise, and demographic homogeneity. However, the headline number is not produced by the main evaluation protocol, and the experimental reporting is not yet precise enough to support the strength of the conclusion. As a preliminary benchmark with a clearly qualified performance estimate, the work is valuable; in its current form, the central feasibility claim is overstated relative to what the tables show.","major_comments":[{"comment":"The hyperparameter selection protocol is described as: \"We performed a grid search over the hyperparameter space for each of the three classifier types in Section 3.4, choosing the values that performed best in the evaluation. The final models were trained using these hyperparameters with 5-fold stratified CV, and the results from the held-out test set, with 20% of the data, were collected.\" This is ambiguous and potentially leaky: if \"the evaluation\" refers to the same held-out test set, then the reported numbers are optimistically biased because hyperparameters were selected on the test data. If a separate validation split was used, that split is not described. Please specify the exact selection procedure, including whether any portion of the test set was used during model selection, and if so, report results from a truly held-out split or use nested cross-validation. This concern affects every number in Tables 3-6.","section":"§4.1"},{"comment":"The conclusion states \"We demonstrate that infant motor quality can be predicted from video recordings with an AUROC of up to 0.8283,\" but this value traces to a single configuration in Table 4: Early GM, Labeller #1, Angles, Random Forest (0.83 ± 0.13). It is not the aggregate result of the pipeline (Table 3 shows Early GM Label & Track Angles RF at 0.77, and fully automatic AggPose peaks at 0.72), and it is selected as the maximum over three manual labellers, three models, three feature sets, and two age groups with no multiple-comparison control. Moreover, this best configuration requires per-video manual keypoint labelling and two rounds of manual outlier correction, so it does not support the \"fully automatic\" or \"scalable\" reading of the claim. The central claim should be re-anchored to the aggregate or fully automatic results, or the \"up to\" number must be clearly labelled as the best manual-labeller scenario with appropriate caveats.","section":"§5 and Appendix B, Table 4"},{"comment":"The effective sample size is 76 infants, but the evaluation treats each fixed-length clip as a separate instance. If a single video produces multiple clips, those clips share the same video-level label and are highly correlated; the reported standard deviations are over 30 seeds only, not over infants or videos. The 20% held-out test set therefore very likely contains clips, not 20% of infants, even though the text claims \"no infant appears in both the training and testing datasets.\" Please clarify whether the split is performed at the infant level or the clip level, report the number of clips per infant, and consider computing performance with infant-level bootstrapping or with a single clip per infant to quantify the true variability. Without this, the reported confidence intervals and the generalizability claim are difficult to assess.","section":"§3.2 and §4.2"},{"comment":"The label-noise issue is load-bearing: every fixed-length clip receives the video's single binary label, even though the paper itself acknowledges in §5 that \"not all segments of a video are equally informative.\" This means the training targets are partially wrong, and the reported AUROC may reflect classification of uninformative segments as often as informative ones. The paper mentions this as a limitation but does not quantify the noise or test robustness, e.g., by training on a subset of clips with higher motion, by using multiple instance learning, or by annotating segment-level labels for a subsample. Since the central claim is about feasibility of automated screening, the effect of this label noise on the reported numbers should be at least qualitatively assessed.","section":"§3.2 and §5"}],"minor_comments":[{"comment":"Typos and misspellings: \"bottleck\" should be \"bottleneck\" (§4.1), \"sptial\" should be \"spatial\" (§5), \"Addionally\" should be \"Additionally\" (§5), \"T rack\" should be \"Track\" (§3.1), and \"F uture\" should be \"Future\" (§5).","section":"Throughout"},{"comment":"The caption says \"The mean performance across 30 seeds and 3 keypoint labellers is reported for the Label & Track method,\" but Table 3 reports only point estimates without standard deviations for Label & Track, whereas Appendix B Tables 4-6 include standard deviations. Please report standard deviations in Table 3 as well, or indicate that they are deferred to the appendix.","section":"Table 3 caption"},{"comment":"The description \"Extreme keypoints are then labelled\" is confusing because the previous paragraph already discusses \"extreme keypoints labelled\" in the labelling step. Please clarify the distinction between the keypoint labelling step and the later use of extreme keypoints for cropping.","section":"§3.2"},{"comment":"The list of keypoints in Figure 4 is hard to read because the \"All Keypoints\" and \"Extreme\" columns are not visually separated; the x markers make it unclear which keypoints are extreme. Consider reformatting this table for readability.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick heads-up before you quote this paper's headline: the 0.8283 AUROC in the conclusion is not the main result. It's the best single configuration from Appendix B—Labeller #1, angle features, random forest, Early GM group. The aggregate numbers in Table 3 are 0.77 (Label & Track, angles, RF) and 0.72 (AggPose, angles, CNN). So the 'up to' number is a selected maximum over labellers, models, feature sets, and age groups, with no multiple-comparison correction. That's the first thing to know.\n\nWhat's genuinely useful: the paper assembles a heterogeneous set of 76 home/hospital infant videos, builds a preprocessing pipeline (resampling, cropping, TAPIR tracking from one human-labeled frame, outlier correction), compares it against AggPose automatic labeling, and evaluates three simple classifiers. Code is public. It's a legitimate feasibility study for automated GMA in uncontrolled settings, and the authors are upfront about limitations: small dataset, demographic homogeneity, coarse video-level labels, and that not all segments are equally informative.\n\nThe soft spots are real but mostly about reporting. The headline number is not traceable to the main evaluation protocol; that's a load-bearing issue because the conclusion anchors on it. Second, the hyperparameter selection in Section 4.1 is underdescribed—'grid search... choosing the values that performed best in the evaluation' followed by evaluation on a held-out 20% test set is ambiguous. If test data influenced hyperparameter choice, the reported AUROCs are optimistic. Third, per-video labels are applied to every clip, so the effective sample size is 76 infants, not the number of clips. The authors acknowledge this, but it means the variance is higher than it appears.\n\nOn balance, I think the reader's conditional verdict is right, but I'd raise the soundness a bit: the flaws are in framing and reporting, not in the core empirical approach. The fully automatic AggPose pipeline tops out at 0.72, which is still a meaningful data point for the field.\n\nThis paper deserves a serious referee. It's an important problem, the dataset is real, and the issues are fixable. I'd ask for an aggregate headline (or a selection-adjusted estimate), a complete description of the hyperparameter protocol, and cluster-robust intervals. I wouldn't cite the 0.8283 number as is, but I'd engage with the work.","headline":"The headline AUROC is a selected maximum from one manual labeller, not the aggregate pipeline; the paper is a useful feasibility study with fixable reporting flaws.","tokens_in":15630,"tokens_out":3044,"would_cite":false,"duration_ms":29092,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully automatic video pipeline can predict newborn general-movement quality in uncontrolled home and hospital recordings, reaching an AUROC of up to 0.8283.","keywords":["general movement assessment","newborn screening","infant pose estimation","keypoint tracking","time-series classification","video-based screening","fidgety movements","writhing movements"],"falsifier":"Re-annotate each video at segment level, marking which 15–30 second windows actually contain informative writhing or fidgety movement, retrain the same classifiers on clip-level labels, and compare AUROC; if clip-level performance falls well below the reported 0.8283, the headline result is carried by video-level label sharing rather than by real movement signal.","tokens_in":14636,"feed_emoji":"🍼","tokens_out":8405,"duration_ms":74569,"temperature":0.7,"pith_summary":"This paper sets out to show that automated General Movement Assessment (GMA) can work on real-world infant videos recorded in uncontrolled hospital and home settings, not just in fixed lab setups. Its central claim is that infant motor quality can be predicted from such videos with an AUROC of up to 0.8283, with markedly better results for the younger writhing-movement group than for the older fidgety-movement group. That would matter because GMA is a reliable early predictor of neurodevelopmental conditions, yet few clinicians are trained to score it; a video-only pipeline could make newborn screening scalable. The authors test a two-branch keypoint pipeline followed by simple time-series classifiers on 76 heterogeneous recordings, each carrying one coarse movement-quality label per video.","feed_headline":"Video keypoints predict infant movement quality at AUROC 0.83","feed_subtitle":"An automatic pipeline reads pose keypoints from everyday hospital and home recordings, opening scalable newborn screening.","key_machinery":"The load-bearing machinery is a two-branch keypoint-extraction pipeline that converts raw video into pose time series. The Label & Track branch seeds 17 anatomical keypoints in one frame by hand and tracks them with the TAPIR point tracker, with automatic outlier detection prompting manual correction; the AggPose branch labels 21 keypoints in every frame independently using a vision transformer for infant pose estimation. From the tracked keypoints the pipeline computes x-y coordinate channels and joint angles at selected triplets (e.g., elbow angle, knee angle, head-to-shoulder angle), then feeds these time series to a 1D-CNN, an LSTM, or a Random Forest. The key mechanism is that angle features and temporal tracking encode movement quality without requiring frame-level movement annotations.","core_discovery":"The paper's central discovery, stated on its own terms, is that coarse video-level annotations and pose-derived time series are sufficient to predict general-movement quality: the best configuration reaches an AUROC of 0.8283, and the early-GM group (preterm infants in the writhing phase) is classified substantially better than the late-GM group (infants in the fidgety phase). Across classifiers and keypoint-extraction methods, angle-based features, alone or combined with coordinates, generally outperform raw coordinate channels, and the manual-labelling-plus-tracking branch performs comparably to, and in several settings better than, fully automatic per-frame pose estimation. The authors interpret this as evidence that automated GMA is feasible in uncontrolled, real-world settings when the preprocessing pipeline handles resampling, cropping, outlier correction, and stratified splitting.","pith_inferences":["Segment-level labels would likely move the reported numbers: if informative 15–30 second windows were identified, AUROC should either rise by removing noise or fall, revealing that the video-level label was carrying the signal; re-labelling a subset of clips would settle which.","The late-GM gap suggests fidgety-movement quality may be more temporal in nature, so sustained-motion features (e.g., velocity or wiggle statistics over seconds) might close the gap more than per-frame angles.","The high-risk cohort (about 65% and 27% abnormal in the two groups) means the AUPRC values are not transferable to general newborn screening; evaluating at roughly the 3% population prevalence is the decision-relevant check."],"forward_implications":["A screening tool could be assembled from equipment parents already have: any home video, one manual label in a single frame (or none with automatic labelling), and a light-weight classifier.","Angle-based representations of body configuration should be treated as a primary feature for movement-quality classification, since they consistently beat raw coordinates in the reported experiments.","Because the early-GM group performs much better than the late-GM group, age-specific models or age-specific features will likely be needed before fidgety-movement screening is reliable.","Simple, compute-efficient models (Random Forest, 1D-CNN, LSTM) are competitive on this task, so scalable deployment need not wait for large transformer models."],"supporting_citations":[{"why":"Defines general movements and the GMA scoring categories (normal vs poor repertoire, fidgety present/absent) that supply the paper's binary labels.","marker":"(Einspieler and Prechtl, 2005)"},{"why":"Provides the TAPIR point-tracking algorithm used in the Label & Track branch to propagate manually seeded keypoints through each video.","marker":"(Doersch et al., 2023)"},{"why":"Supplies AggPose, the fully automatic infant pose estimator used as the alternative keypoint-labelling baseline.","marker":"(Cao et al., 2022)"},{"why":"Motivates the joint-angle feature set computed from keypoint triplets for movement classification.","marker":"(Prakash et al., 2023)"},{"why":"Supports the claim that 15–30 seconds of relevant movement can suffice, justifying the fixed-length clip segmentation.","marker":"(Kapil et al., 2024)"},{"why":"Establishes the prior pose-estimation-plus-classifier approach that this pipeline extends to uncontrolled settings.","marker":"(Reich et al., 2021)"}],"fun_headline_variants":["Pose keypoints from home videos flag infant movement issues","Automated infant movement screening hits AUROC 0.83","Everyday videos train AI to spot early neuro issues","Angle features beat raw coordinates in infant GM scoring","Scalable newborn screening: AI reads pose from home videos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that every fixed-length clip cut from a video inherits the video's single movement-quality label, even though only part of a recording may show diagnostically informative movement, and the paper itself concedes in its limitations that not all segments are equally informative.","fun_headline_variants_meta":{"raw":{"variants":["Pose keypoints from home videos flag infant movement issues","Automated infant movement screening hits AUROC 0.83","Everyday videos train AI to spot early neuro issues","Angle features beat raw coordinates in infant GM scoring","Scalable newborn screening: AI reads pose from home videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000541,"raw_usage":{"total_tokens":2533,"prompt_tokens":822,"completion_tokens":1711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":1632}},"tokens_in":438,"tokens_out":1711,"duration_ms":13046,"temperature":1.0,"reasoning_tokens":1632,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:16:29.374392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate each video at segment level, marking which 15–30 second windows actually contain informative writhing or fidgety movement, retrain the same classifiers on clip-level labels, and compare AUROC; if clip-level performance falls well below the reported 0.8283, the headline result is carried by video-level label sharing rather than by real movement signal.","supporting_citations":[{"cited_title":"Prechtl's assessment of general movements: a diagnostic tool for the functional assessment of the young nervous system","cited_arxiv_id":null,"evidence_quote":"Defines general movements and the GMA scoring categories (normal vs poor repertoire, fidgety present/absent) that supply the paper's binary labels."},{"cited_title":"Video-based real-time assessment and diagnosis of autism spectrum disorder using deep neural networks","cited_arxiv_id":null,"evidence_quote":"Motivates the joint-angle feature set computed from keypoint triplets for movement classification."},{"cited_title":"o lte, Karin Nielsen-Saines, Florian B Pokorny, Robert Peharz, Luise Poustka, Florentin W \\","cited_arxiv_id":null,"evidence_quote":"Establishes the prior pose-estimation-plus-classifier approach that this pipeline extends to uncontrolled settings."}],"review_version":1}