{"id":"1a7aa7de-89f2-44d2-a1a0-aa66f3eae965","arxiv_id":"1908.10088","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An ensemble of residual networks with an attention mechanism achieves 0.875 overall F1 on multi-label 12-lead ECG abnormality classification.","lead":"This paper describes a computer system that reads 12-lead ECG recordings and flags nine types of heart abnormalities. The system combines residual neural networks with attention and achieved an overall F1 score of 0.875 on a competition test set.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported F1=0.875 is not tied to a specified inference-time windowing rule; random 30s truncation can discard diagnostic episodes, so the headline metric is under-defined.","rationale":"The central claim is an empirical number, F1=0.875, on a hidden test set. That number inherits every preprocessing decision, and the least supported link is the fixed 30s window. Section 3.1 chooses 30s for efficiency and overfitting control, not because it preserves diagnostic information. At training, all windows from a positive recording are labeled positive, so some windows contain no abnormal beat; at inference, the paper never states which window is used for a long test recording. This is an internal gap between the described procedure and the claim, not a disagreement with community consensus. The reader's CONDITIONAL verdict already captures the truncation concern; my analysis sharpens it by emphasizing the missing test-time protocol. A window-sensitivity check on public data, or on the official test set if available, would either validate the 0.875 value or show it is an artifact of the cropping rule. Since this is exactly the kind of condition that can be met by adding a stability analysis and code or predictions, the verdict remains CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":6193,"tokens_out":11263,"duration_ms":114447,"concrete_test":"On the public CPSC 2018 dataset (already used in pipeline 2), evaluate the trained ensemble on test recordings longer than 30s under three inference rules: fixed first 30s, one randomly selected 30s window, and max-pooled predictions over all non-overlapping 30s windows. Compare the per-class and macro F1 across the three rule outputs; if any class F1 changes by more than 0.01, or if the macro F1 shifts away from the reported 0.875, the reported result is not robust to the unspecified test-time window selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 fixes the target length to 30s and trains models on randomly selected padding/truncating windows, but neither Section 3.3 nor Section 4 specifies how test recordings longer than 30s are reduced to 30s at inference. For paroxysmal classes (PVC, PAC, ER), a random 30s window from a 60-90s recording may not contain the abnormal beat; because the recording-level label is applied to every training window, the model is trained on positive windows that may be visually normal, and at test it cannot detect events outside the chosen window. The reported single-point F1=0.875 therefore depends on an unspecified and possibly stochastic test-time cropping rule, and no per-window stability analysis, confidence interval, or released predictions are provided to show that the score is robust to the choice of window. Until the test-time protocol is specified and shown not to change the result, the central quantitative claim cannot be verified from the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automatic ECG abnormality detection pipeline for 12-lead recordings, combining preprocessing (baseline wander removal, wavelet denoising, length unification, data augmentation and class balancing), a deep residual convolutional network with attention, and an ensemble of models trained under four different pipelines. The method is evaluated on the hidden test set of the First China ECG Intelligent Competition using class-wise F1 and their macro-average, with a reported overall F1 of 0.875 across nine classes. The central claim is that this ensemble of attention-augmented residual networks yields strong multi-label classification performance on a realistic ECG dataset.","tokens_in":6406,"tokens_out":3350,"duration_ms":36379,"significance":"If the reported F1 of 0.875 is reproducible and robust, the result would be of practical value for automated ECG screening, especially because the problem is multi-label, the recordings vary in length up to 90 s, and some abnormalities are paroxysmal. The manuscript is commendable for providing explicit formulas for the F1 metric (Eqs. 2-9), for evaluating on an external hidden test set rather than a held-out portion of the training data, and for describing the training pipelines in enough detail to be partially replicable. However, the significance is substantially limited by the absence of baseline comparisons, the lack of any uncertainty quantification, and the underspecification of the test-time windowing rule, all of which prevent the reader from calibrating or verifying the headline number. The paper also claims that the ensemble is superior to individual pipelines, but it does not report the supporting per-pipeline results.","major_comments":[{"comment":"The inference-time windowing rule for test recordings longer than 30 s is never specified. Section 3.1 describes random padding/truncating as a training-time augmentation, and Section 3.3 explains how this creates training samples, but Section 4 reports F1 = 0.875 on the entire test set without stating whether a fixed 30-s window is cut from the beginning, a random window is sampled once, or multiple windows are aggregated. Because the recording-level labels of paroxysmal classes (PVC, PAC, ER) may appear only in a short segment, the reported macro-F1 depends on this underspecified choice. Please state the exact test-time protocol and demonstrate that the result is stable under different window positions, for example by reporting variation across several random windows or a deterministic aggregation rule.","section":"§3.1, §3.3, §4"},{"comment":"No uncertainty or significance estimates are reported for the macro-F1. With 8,500 test recordings and nine classes of very different prevalence (e.g., LAFB, ER, TWC at F1 ≈ 0.73-0.76), the single number 0.875 gives no indication of whether differences between models or between classes are meaningful. Add bootstrap confidence intervals for each class-level F1 and for the macro-F1, or state the official competition evaluation protocol if it defines a fixed scoring rule.","section":"Table 1 and §4"},{"comment":"No comparison with any baseline method is provided. The manuscript reports only the proposed method's F1, so the reader cannot determine whether the 0.875 score represents a genuine advance over, for example, a standard CNN, an LSTM-based model, or a simpler non-attention residual network. A comparison with at least one previously published ECG classifier on the same competition test set is needed to support the claimed promising performance.","section":"§4"},{"comment":"The conclusion states that the proposed ensemble 'achieved better results than each of the separated pipelines,' but no per-pipeline F1 scores or ensemble ablations are reported anywhere in the paper. This claim is therefore not supported by the presented evidence. Please provide the per-pipeline results or remove the superiority claim.","section":"§5"}],"minor_comments":[{"comment":"Equation (1) does not state the units of N, and the text gives conflicting cut-off frequencies: the stated lower bound is 0.5 Hz, but N = 500 gives f_co = 0.443 Hz, which is below that bound. Please clarify the intended window sizes and the corresponding cut-off frequencies.","section":"§3.1, Eq. (1)"},{"comment":"The text says the feature map length becomes 1/27 of the input length after the local feature learning stage, but the architecture has seven max-pooling layers; unless the pooling factors are not 2, this should be 1/128. Please justify or correct this factor.","section":"§3.2"},{"comment":"The description of pipeline 4 is incomplete: it says the input is normalized to zero mean and unit variance but does not specify whether normalization is applied per-lead, per-recording, or across the dataset, nor how this interacts with the denoising used in the other pipelines.","section":"§3.3 and Fig. 2"},{"comment":"The table reports F1 scores but not the number of test samples or the prevalence of each class. Adding per-class support counts would help interpret the low F1 values for LAFB, ER, and TWC.","section":"Table 1"},{"comment":"The paper would benefit from a summary of the official competition scoring rule and a clear statement of whether the reported 0.875 is the exact official score or a recomputed value, since the metric definition in Eqs. (2)-(9) may differ from the competition's own protocol.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially a competition methods report. The main technical issue is the underspecified test-time windowing, which is load-bearing for the headline F1. I do not see evidence of circularity, and the external test set is a genuine strength. I recommend revision rather than rejection because the central approach is plausible and the missing details are, in principle, fixable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper reports a strong result: F1 of 0.875 on a nine-class, multi-label 12-lead ECG classification task from an external hidden test set. That number is the whole ballgame, and it is both plausible and under-specified. The random padding/truncating augmentation is a simple, sensible idea to both regularize and balance training data, and the two-stage balance-then-retrain-with-imbalanced-data in pipeline 3 is a neat trick. The architecture itself is not new—residual blocks plus attention—but the specific combination with the random windowing strategy is a legitimate new application. The evaluation is against a held-out test set, so the headline score is not a re-statement of the training objective, and the F1 definitions are clear and internally consistent.\n\nThe soft spots are real, and one is load-bearing. Nowhere in Sections 3.3 or 4 does the paper say how test recordings longer than 30 seconds are reduced to 30 seconds at inference. For paroxysmal classes like PVC, PAC, and ER, a single 30-second window may not contain the abnormal beat; if the crop is random, a single test recording could receive different predictions on different runs. The training-time random window is described, but the inference-time protocol is not. That means the reported 0.875 depends on an unstated, possibly stochastic rule, and the reader cannot verify the central quantitative claim from the text alone. This is a reporting gap, not evidence the result is wrong, but it is a serious one.\n\nBeyond that, the paper has no confidence intervals, no comparison to any baseline method (not even a single-pipeline version), and no ablation supporting the Conclusion's claim that the ensemble beat each pipeline individually. No code or data are released. The citation pattern is fine—references are standard, the one self-citation is relevant. The paper takes a small step beyond prior work, but the incremental novelty is acknowledged by the authors' own references showing prior CNN, residual, and RNN ECG classifiers.\n\nOverall, this looks like a competent competition write-up with an incomplete technical description. It deserves a serious referee—the question of whether the reported result holds up is important for applied ECG deep learning—but it needs major revision: spell out the test-time windowing, add baselines and ablations, and ideally release code or predictions so the F1 can be checked. I would not cite it as a trustworthy baseline until that is done.\n\nSend it to peer review, but expect heavy revision.","headline":"Reported F1 of 0.875 is plausible but rests on an unspecified test-time windowing rule, so the headline number is not yet reproducible from the paper.","tokens_in":6914,"tokens_out":1732,"would_cite":false,"duration_ms":19701,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Residual networks with attention reach 0.875 F1 on ECG abnormalities.","keywords":["electrocardiogram","12-lead ECG","deep residual network","attention mechanism","ensemble model","multi-label classification","data augmentation","F1 score"],"falsifier":"One decisive check: rescore each hidden-test recording with a different randomly chosen 30-second window and compare per-class F1; if scores for paroxysmal classes such as PVC or PAC drop, the 0.875 result depends on which window the model saw, not on a stable signal of the abnormality.","tokens_in":6021,"feed_emoji":"🫀","tokens_out":8296,"duration_ms":77418,"temperature":0.7,"pith_summary":"This paper reports a computer pipeline for spotting nine types of abnormality in 12-lead ECG recordings. The pipeline denoises the signal, randomly pads or truncates every recording to 30 seconds, augments and re-balances the classes, and then trains an ensemble of deep residual convolutional networks whose final layer is an attention-weighted summary of local waveform features. On the hidden test set of the First China ECG Intelligent Competition, the ensemble achieves an overall F1 of 0.875, computed as the average of the nine per-class F1 scores. If that score holds, the approach is a plausible triage tool that could flag suspected cases for human review rather than replace a cardiologist.","feed_headline":"Residual networks with attention reach 0.875 F1 on ECG abnormalities","feed_subtitle":"The algorithm flags atrial fibrillation, heart blocks, and premature beats with F1 above 0.9 on a hidden test set.","key_machinery":"The carrying mechanism is a residual convolutional network with an attention layer, applied to 30-second slices of 12-lead ECG. The local-feature stage passes the raw signal through a 1D convolution followed by seven residual modules, each consisting of batch normalization, dropout, ReLU, and 1D convolution with an additive skip connection, with max-pooling after each module. The attention stage turns the resulting sequence of local feature vectors into a single global vector by learning a weight for each local vector and summing them; a fully-connected layer with per-class sigmoid outputs then makes independent predictions for each of the nine labels. Attention is what lets the model concentrate on the short episodes in a recording that actually carry a diagnosis, while the random padding/truncating scheme is what keeps the network from latching onto recording length or padding position.","core_discovery":"The central claim is that an ensemble of residual networks with attention can classify nine ECG abnormality classes with a macro-averaged F1 of 0.875 on an unseen test set. Per-class results are above 0.9 for atrial fibrillation, first-degree atrioventricular block, complete right bundle branch block, premature ventricular contraction, and premature atrial contraction, and near 0.75 for left anterior fascicular block, early repolarization pattern changes, and T-wave changes. The authors attribute the result to two design choices: random padding/truncating to a 30-second target length, which augments and balances the data while suppressing overfitting to where padding sits, and a model that first learns local features through seven residual convolutional modules and then weights those features with an attention distribution before classification. The ensemble averages probabilistic predictions from four training pipelines, one of which mixes in an external dataset and another of which retrains balanced models on imbalanced data.","pith_inferences":["Editorial inference: if the random 30-second window sometimes cuts off a transient abnormality, the reported F1 could overstate what the model does on a full recording; re-scoring with several windows per test recording would quantify this.","Editorial inference: the attention weights learned by the network are a ready-made saliency map over the ECG trace, but the paper does not check whether those weights align with where clinicians locate abnormalities.","Editorial inference: the two-stage balance-then-imbalance training used in one pipeline suggests a general strategy for imbalanced medical datasets, but its contribution is not isolated by an ablation.","Editorial inference: because each class uses an independent sigmoid, lowering the decision threshold would trade precision for recall and could recover F1 on LAFB, ER, and TWC."],"forward_implications":["If the 0.875 macro-F1 transfers to other patient cohorts, the pipeline can pre-screen 12-lead ECGs and route high-confidence abnormal recordings to a cardiologist before full manual review.","For the five classes with per-class F1 above 0.9, automated flags could be usable as decision support without per-case recalibration.","The three low-F1 classes (LAFB, ER, TWC) identify where the method needs more training data or additional features, since the paper attributes their weakness to relatively few samples.","Because the architecture is an ensemble of independently trained pipelines, adding pipelines with different preprocessing is a direct route to incremental accuracy gains.","The random padding/truncation augmentation means the model is intentionally insensitive to the temporal position of a 30-second window, which is a testable assumption about whether short windows are enough for ECG diagnosis."],"supporting_citations":[{"why":"Supplies the residual convolutional module design used in the local-feature stage.","marker":"[6]"},{"why":"Supplies the soft-threshold wavelet denoising used to remove powerline and muscle noise.","marker":"[9]"},{"why":"Provides a CNN-plus-LSTM ECG classification approach that the attention-residual model extends.","marker":"[8]"}],"fun_headline_variants":["Residual nets with attention hit 0.875 F1 for ECG abnormalities","ECG abnormality detection: ensemble of attention nets scores 0.875 F1","Attention boosts residual ensemble to 0.875 F1 on ECG tasks","Deep residual ensemble with attention flags ECG issues at 0.875 F1","Ensemble of attention residual nets detects ECG abnormalities at 0.875 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a randomly selected 30-second slice of a longer ECG recording contains every abnormality that a clinician would need to classify; the paper reports no measurement of how often truncation discards the relevant episode.","fun_headline_variants_meta":{"raw":{"variants":["Residual nets with attention hit 0.875 F1 for ECG abnormalities","ECG abnormality detection: ensemble of attention nets scores 0.875 F1","Attention boosts residual ensemble to 0.875 F1 on ECG tasks","Deep residual ensemble with attention flags ECG issues at 0.875 F1","Ensemble of attention residual nets detects ECG abnormalities at 0.875 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000682,"raw_usage":{"total_tokens":3119,"prompt_tokens":987,"completion_tokens":2132,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":2030}},"tokens_in":603,"tokens_out":2132,"duration_ms":12453,"temperature":1.0,"reasoning_tokens":2030,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:52:33.409526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One decisive check: rescore each hidden-test recording with a different randomly chosen 30-second window and compare per-class F1; if scores for paroxysmal classes such as PVC or PAC drop, the 0.875 result depends on which window the model saw, not on a stable signal of the abnormality.","supporting_citations":[{"cited_title":"In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp","cited_arxiv_id":null,"evidence_quote":"Supplies the residual convolutional module design used in the local-feature stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the soft-threshold wavelet denoising used to remove powerline and muscle noise."},{"cited_title":"L., Ng, E","cited_arxiv_id":null,"evidence_quote":"Provides a CNN-plus-LSTM ECG classification approach that the attention-residual model extends."}],"review_version":1}