{"id":"108211bd-8cdb-4f67-a67c-277f04a51d54","arxiv_id":"2506.06349","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On an unnamed ECG dataset, LightGBM with hand-crafted heart rate variability and RR interval features outperformed CNN classifiers trained on GAF, MTF, and recurrence plot images.","lead":"This paper compared machine learning and deep learning methods for classifying heartbeats from ECG signals, and reports that a LightGBM model with hand-crafted features reached 99% accuracy and F1 0.94, beating an image-based CNN that scored F1 0.85. The comparison is a practical benchmark, but the dataset and code are not released, so the numbers cannot be independently reproduced.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LightGBM-vs-CNN comparison rests on an undisclosed train/test split; if beats from the same patient appear in both partitions, the reported 0.94 vs 0.85 F1 gap may reflect memorized patient-specific morphology rather than generalizable feature superiority.","rationale":"I agree with the reader's weakest-assumption analysis and find no independent reason to change the conditional verdict. The manuscript's internal description does not rule out beat-level leakage, and the RR-interval/HRV features are exactly the kind of patient-identity cue that makes such leakage consequential. The concern is not that the numbers are implausible; LightGBM on HRV features can genuinely reach high F1 on within-patient splits. The concern is that the paper's central comparative claim is unverifiable without knowing whether the test set is patient-disjoint. This is a reproducibility/generalization issue, not an external-consensus dispute, and it is the single most load-bearing assumption. A patient-disjoint rerun is the direct check. If it passes, the conditional verdict can be upgraded; if it fails, the claim should be downgraded.","tokens_in":4977,"tokens_out":3248,"duration_ms":34360,"concrete_test":"Ask the authors to disclose the dataset and the exact train/test split, then independently rerun the two pipelines with the same hyperparameters and SMOTE-on-training-only, but with a patient-disjoint split: no beats from the same patient in both training and test. If LightGBM's macro-F1 advantage over the image CNN narrows to within statistical uncertainty or reverses, the central claim of hand-crafted feature superiority is not supported. Report per-class F1, 95% confidence intervals, and class counts for both split modes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II-A describes preprocessing and says only 'we had the dataset that contains separated beats and their labels'; Section II-B states SMOTE was applied 'only to the training data, while the test data remained unchanged,' but the paper never identifies the dataset or specifies whether the split is by patient, by record, or randomly by beat. If the split is random at the beat level, the same patient's beats contribute to both training and test sets. ECG beats of the same patient share baseline, electrode placement, and heart-rate dynamics; classifiers can memorize patient identity, and the RR-interval/HRV features make this memorization especially easy. The headline comparison (Table I: LightGBM accuracy 0.99, F1 0.94 vs image CNN F1 0.85) then measures within-patient discrimination, not generalization to new patients. The abstract's broader claim that hand-crafted features capture temporal and morphological variation better than image encodings depends on that unsupported comparison. No error bars, confidence intervals, per-class metrics, or code are reported, so the magnitude of the gap cannot be checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports an empirical comparison of two families of ECG heartbeat classifiers on a dataset that is not named. The first family uses hand-crafted features (heart-rate variability statistics, mean/variance, RR-interval logarithms) concatenated with 70-sample beat windows, fed to SVM, Random Forest, AdaBoost, LSTM, BiLSTM, 1D CNN, and LightGBM, with SMOTE applied to the training set. The second family encodes each beat as a 32-by-32 GAF, MTF, or recurrence-plot image and classifies the concatenated three-channel image with 2D CNNs (best: a custom Inception-style network). The authors report LightGBM as best (accuracy 0.99, macro F1 0.94) and conclude that hand-crafted features capture beat variation better than image encodings.","tokens_in":5217,"tokens_out":7326,"duration_ms":68802,"significance":"The question addressed—hand-crafted beat/HRV features versus imaging-based deep learning for single-beat ECG classification—is practically relevant, and the paper has some good practices: SMOTE is applied only to the training partition, hyperparameters are tuned by grid search with 3-fold cross-validation, and the headline metric is macro F1 rather than raw accuracy. If the evaluation were performed on a patient-disjoint split of a standard public dataset with uncertainty estimates, the LightGBM result would be a useful data point. As written, however, the manuscript does not identify the dataset or the split rule, and the comparison between approaches is not controlled; the central claim is therefore not verifiable from the paper. The reader's stress-test concern about beat-level leakage lands and is the main reason the manuscript needs major revision.","major_comments":[{"comment":"The dataset is never identified and the train/test split is never described. The text states only that 'we had the dataset that contains separated beats and their labels' and that SMOTE was applied 'only to the training data, while the test data remained unchanged.' It does not state whether the split is by patient, by record, or randomly by beat. Since beats from the same patient share morphology, electrode placement, and RR-interval dynamics, a random beat-level split allows the classifier to memorize patient identity. This is load-bearing because the headline comparison in Table I (LightGBM 0.99 accuracy/0.94 F1 vs image-based CNN 0.85 F1) would then reflect within-patient discrimination rather than generalization. Please name the dataset, specify the split rule, and report results on a patient-disjoint split (or both split regimes).","section":"§II-A, §II-B"},{"comment":"The two approaches are not evaluated under the same class-imbalance protocol. SMOTE is introduced in Section II-B for the hand-crafted-feature pipeline, with target counts of 100k V/S and 300k N beats, but Section II-C says only that the image pipeline uses 'the same dataset ... with the exception that no hand crafted feature was used.' If the image CNN was trained on the original imbalanced beats while LightGBM was trained on SMOTE-rebalanced data, the lower macro F1 of the image approach could be caused by the imbalance rather than by the image encoding. Please state explicitly whether SMOTE (or another rebalancing procedure) was applied to the image training set, and if not, re-run the comparison with matched protocols.","section":"§II-B, §II-C, Table I"},{"comment":"No uncertainty quantification is provided. The reported values are macro averages, but the paper gives no test-set size, per-class support, per-class precision/recall, confidence intervals, or multiple-seed repetitions. The differences among LightGBM (0.94), Random Forest (0.93), and BiLSTM (0.93) are one point in macro F1 and could easily be within noise. Add the number of test beats and patients per class and report confidence intervals or repeated-seed ranges so that the 'outperforming' and 'significantly lower' statements in the Abstract and Section IV are supported.","section":"Table I, §III"},{"comment":"The 1D CNN row reports accuracy 0.92 with macro precision 0.43, macro recall 0.39, and macro F1 0.39. Such a combination indicates that the model is accurate mostly because it assigns the majority class, while minority classes have near-chance or worse discriminating performance. Without per-class metrics or a confusion matrix, the reader cannot interpret this row, and Section IV uses these scores to discard models. Please provide the confusion matrix or a per-class breakdown for each model.","section":"Table I, 1D CNN row"}],"minor_comments":[{"comment":"The SMOTE counts are described inconsistently: Section II-B says 100k V/S and 300k N beats, while Section V says 'undersampling the signal 300000 times and then oversampling it 100000 times.' Please align the wording.","section":"§II-B, §V"},{"comment":"The formula is typeset incorrectly (missing brace/bracket, and the variable is given as xi rather than \\tilde{x}_i), and the constant T is not defined. Please fix the equation and define all symbols.","section":"§II-C, Eq. (1)"},{"comment":"The text says 'we immediately discarded the SVM, feed-forward neural network, and AdaBoost models,' but no feed-forward neural network appears in Table I or in the methods section. Please correct the list or clarify which model is meant.","section":"Section IV"},{"comment":"Reference [5] (Moreno-Camacho et al., supply-chain sustainability metrics) appears unrelated to ECG classification and is not cited in the text; please remove it or replace it with the intended citation.","section":"References"},{"comment":"Several typographical issues remain: 'the the unbalanced dataset' in Section II-B, '03e-4' for the learning rate in Section II-C, and the author name 'V o' in the byline. Please proofread.","section":"Throughout"},{"comment":"The manuscript does not include a code or data availability statement, which would help reproducibility; consider adding one if permissible.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"For the editor: The manuscript is an empirical engineering report with a clear question, but it lacks the dataset name, split description, and uncertainty metrics essential for any claim about generalization. The discrepancy between Sections II-B and V and the unrelated reference [5] suggest the draft is unfinished; however, the issues are fixable if the authors can supply the missing experimental details and re-run with a patient-disjoint split. I do not see a circularity problem, because no predictive derivation is claimed. The paper's contribution is modest and would fit a workshop or applied journal; if the dataset is proprietary and cannot be disclosed, the reported numbers will remain unverifiable, which may argue for rejection despite the fixable presentation issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a competent but unverifiable benchmark: LightGBM on hand-crafted HRV/RR features beats a CNN on GAF/MTF/recurrence-plot images for ECG beat classification (0.99 accuracy/0.94 F1 vs 0.85 F1). The thing to know before recommending it to anyone is that the dataset is never named and the train/test split is never described. The headline comparison therefore may be measuring within-patient memorization rather than generalization.\n\nCredit where due: the paper is honest about being an empirical comparison, not a new method. It tests a reasonable set of classifiers, uses standard encodings, describes the transform math correctly, says SMOTE was applied only to the training data, and reports macro-averaged metrics. The best part is that it identifies a practical question—whether hand-crafted features still earn their keep against image-based deep learning—and gives a coherent, if limited, answer.\n\nThe soft spots are substantial. First, no dataset identity. The preprocessing description (2-lead, R-peak files, beat annotations) strongly suggests MIT-BIH, but the paper never says. Second, no split details: random beat-level splitting would put beats from the same patient in both train and test, and with RR-interval features the model can memorize patient identity. That makes the 0.94 vs 0.85 F1 gap uninterpretable. The stress-test raises exactly this, and on reading the paper it holds. Third, no error bars, confidence intervals, per-class metrics, or code. There is also a red flag in Table I: the 1D CNN has accuracy 0.92 but F1 0.39, which means one or more classes are essentially never predicted correctly; the paper doesn't discuss it. Minor, and cosmetic: one citation (a supply-chain paper) does not fit the topic.\n\nThe abstract's broader claim—hand-crafted features capture temporal and morphological variation better than image encodings—is not supported by one undisclosed dataset with an unknown split. The paper is best read as a draft that needs its experimental foundation documented.\n\nWho is it for? Practitioners wanting a quick point of comparison, but only after the authors supply the missing details. It does not deserve a serious referee in its current form. I'd desk reject with a clear invitation to resubmit once the dataset, split, and code are provided and the 1D CNN anomaly is explained.","headline":"A plausible but unverifiable benchmark: without the dataset name and the train/test split, the LightGBM-vs-CNN F1 gap may be patient memorization, not generalization.","tokens_in":5710,"tokens_out":6404,"would_cite":false,"duration_ms":57401,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On an ECG heartbeat dataset, LightGBM trained on hand-crafted waveform and interval features reaches 0.94 macro F1, outperforming a CNN that classifies image-transformed beats at 0.85 macro F1.","keywords":["ECG","heartbeat classification","LightGBM","hand-crafted features","Gramian Angular Field","Markov Transition Field","Recurrence Plot","deep learning"],"falsifier":"Re-run the two best models with a patient-wise split—putting every beat of any given patient into either the training set or the test set, never both—and compare the macro F1 scores; if LightGBM's advantage over the image-based CNN shrinks or disappears, the claim that hand-crafted features generalize better is not supported by this dataset.","tokens_in":4786,"feed_emoji":"❤️","tokens_out":6764,"duration_ms":63418,"temperature":0.7,"pith_summary":"This paper asks which representation of a single ECG heartbeat serves automated classification best: a small set of hand-crafted statistics or an image built from the raw waveform. Using a dataset of 2-lead ECG beats with R-peak positions and beat annotations, the author trains classical models on 70-sample normalized beats plus HRV and RR-interval features, and deep CNNs on Gramian Angular Field, Markov Transition Field, and Recurrence Plot images. The best classical model, LightGBM, reaches 99% accuracy and 0.94 macro F1, beating the best image-based CNN at 0.85 macro F1. The paper's conclusion is that hand-crafted features capture temporal and morphological variation better than single-beat image encodings, and that SVM and AdaBoost are not competitive on this task.","feed_headline":"LightGBM beats CNNs on ECG heartbeat classification","feed_subtitle":"Hand-crafted waveform and interval features reach 0.94 F1, leaving image-based deep learning at 0.85 F1.","key_machinery":"The comparison runs on two competing input representations. Pipeline one concatenates each normalized 70-sample beat with heart-rate variability statistics and the logarithms of successive RR intervals, then feeds that vector to LightGBM, a gradient-boosted decision tree ensemble. Pipeline two encodes each beat as three 32x32 images—a Gramian Angular Field (a polar-coordinate matrix of temporal correlations), a Markov Transition Field (a matrix of transition probabilities between value quantiles), and a Recurrence Plot (a matrix marking times at which values repeat)—stacked as RGB channels and classified by an Inception-style CNN. The load-bearing mechanism is the information each representation preserves: the hand-crafted vector carries both beat morphology and inter-beat timing, while each image encodes only one beat's shape.","core_discovery":"The central claim is a direct performance comparison: for this ECG dataset, the LightGBM model fed hand-crafted features (mean, variance, median HRV, logarithms of consecutive RR distances, and the 70-sample beat) achieves 0.94 macro F1 and 99% accuracy, while the best deep-learning pipeline, an Inception-style CNN taking GAF, MTF, and RP images of the same beats as input, achieves 0.85 macro F1. The author interprets this gap as evidence that hand-crafted features encode the full variation of the beats and their timing, whereas the image representations contain information about a single beat only. Random Forest matches LightGBM closely at 0.93 F1, and LSTM variants trail slightly, which the paper reads as confirmation that ensemble trees on well-chosen features are sufficient for this task.","pith_inferences":["Because the train/test split is unspecified, the headline gap likely overstates performance on a new patient; a per-patient split is the immediate experiment that would settle it.","The comparison changes representation and model class at the same time, so it cannot separate feature engineering from architecture choice; a matched test feeding raw beats to both LightGBM and a CNN would isolate the cause.","The author's own proposal to classify a beat using four previous beats suggests that inter-beat context may matter more than either single-beat images or single-beat statistics."],"forward_implications":["On this dataset, hand-crafted features plus LightGBM give the best single-beat classification accuracy, with Random Forest close behind.","The 0.85 F1 of the image-based CNN is a concrete baseline for any future image-representation method on ECG beats.","SVM and AdaBoost perform too poorly to be competitive on these features.","The author's conclusion implies that converting beats to images for deep CNNs is not automatically the best route when interpretable features are available."],"supporting_citations":[{"why":"Supplies the preprocessing steps and the additional hand-crafted features (log RR distances) used by the winning model.","marker":"[8]"},{"why":"Introduces the Gramian Angular Field, Markov Transition Field, and Recurrence Plot image transforms used in the deep-learning pipeline.","marker":"[9]"},{"why":"Establishes the multimodal fusion idea of combining ECG image and feature representations.","marker":"[6]"},{"why":"Supports the claim that ML on hand-crafted features can outperform deep models.","marker":"[7]"},{"why":"Provides the 2-D ECG image representation approach for deep CNN classification.","marker":"[2]"},{"why":"Offers the patient-adapting classifier baseline using morphology and interval features.","marker":"[3]"}],"fun_headline_variants":["LightGBM hits 0.94 F1, beating CNN's 0.85 in ECG","Hand-crafted features beat image CNNs in ECG classification","ECG classification: LightGBM with crafted features tops CNNs","Images lose to features in ECG beat classification","Crafted ECG features outperform image-based deep learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported numbers assume the test set is a fresh, representative sample of beats that did not influence training, but the paper never states whether beats were split by patient or by record, so the 0.94 versus 0.85 F1 gap could partly reflect memorized patient-specific waveforms rather than genuine generalization.","fun_headline_variants_meta":{"raw":{"variants":["LightGBM hits 0.94 F1, beating CNN's 0.85 in ECG","Hand-crafted features beat image CNNs in ECG classification","ECG classification: LightGBM with crafted features tops CNNs","Images lose to features in ECG beat classification","Crafted ECG features outperform image-based deep learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2835,"prompt_tokens":972,"completion_tokens":1863,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1776}},"tokens_in":588,"tokens_out":1863,"duration_ms":12260,"temperature":1.0,"reasoning_tokens":1776,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:44:06.777058+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the two best models with a patient-wise split—putting every beat of any given patient into either the training set or the test set, never both—and compare the macro F1 scores; if LightGBM's advantage over the image-based CNN shrinks or disappears, the claim that hand-crafted features generalize better is not supported by this dataset.","supporting_citations":[{"cited_title":"ECG heartbeat classification using multimodal fusion","cited_arxiv_id":null,"evidence_quote":"Establishes the multimodal fusion idea of combining ECG image and feature representations."},{"cited_title":"An Automated ECG Beat Classification System Using Deep Neural Networks with an Unsurpassed Feature Extraction Technique","cited_arxiv_id":null,"evidence_quote":"Supports the claim that ML on hand-crafted features can outperform deep models."},{"cited_title":"Classification of arrhythmia by using deep learning with 2-D ECG signal image representation","cited_arxiv_id":null,"evidence_quote":"Provides the 2-D ECG image representation approach for deep CNN classification."},{"cited_title":"A Patient-Adapting Heartbeat Classifier Using ECG Morphology and Heartbeat Interval Features,","cited_arxiv_id":null,"evidence_quote":"Offers the patient-adapting classifier baseline using morphology and interval features."}],"review_version":1}