{"id":"d1358545-f0d1-45af-9478-2ea26e343ccf","arxiv_id":"2502.05315","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A controlled replication benchmark of nine AMR models on RadioML-2016A, plus experiments showing that moderate SNR training ranges and added recurrent layers can improve accuracy.","lead":"This paper replicates nine published deep learning models for automatic modulation recognition and compares their accuracy on the RadioML-2016A dataset under one consistent test setup. It also tests small training and architecture tweaks, reporting which models improve.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed SNR-window optimum in §III.A is selected post hoc and reported without variance or a pre-registered window, so the 0.63 vs. full-range figure may be seed noise rather than a genuine training-strategy gain.","rationale":"The reader's weakest assumption centers on replication fidelity: without code or comparison to original accuracies, the benchmark's fairness is unverified. My concern is adjacent but more specific: even if every replication were faithful, the headline SNR-window result would still be unsupported because it was selected post hoc and reported without variance or a pre-registered comparison. The reader does mention 'selects the SNR training range post hoc' in their rationale, so the overlap is partial. The appropriate disposition remains conditional: the survey and benchmark have potential value, but the enhancement recommendation in §III.A should not be relied upon until code is released, multiple seeds are reported, and the SNR-window comparison is pre-registered or validated on a held-out split. This does not change the reader's conditional verdict; it sharpens the primary reason for requiring revision.","tokens_in":10813,"tokens_out":7225,"duration_ms":71930,"concrete_test":"Run MCLDNN with 10 random seeds under two pre-registered conditions: training on the full [-20,18] SNR range and training on [-18,18], all else equal; report mean±std and a paired test on the same fixed test set. If the average advantage of [-18,18] is less than the pooled standard deviation or is not statistically significant, the post hoc optimal-window claim in §III.A is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most consequential claim—that training MCLDNN on SNR values from -18 to 18 dB achieves 0.63 average test accuracy versus the full -20 to 18 dB range—rests on a post hoc selection. Section III.A describes gradually adding low-SNR data from -2 to -20 dB, meaning multiple truncation windows were tried; only the best-performing window is reported. The test set is fixed and includes all SNR levels, including the -20 dB samples excluded from the winning window. No standard deviation, number of seeds, or significance test is given, and Table 1 shows the top models span roughly 0.53–0.60, so a ~0.03 gap is plausibly within run-to-run variation for 100-epoch Colab training with early stopping. Consequently, the 'optimal window' may be an artifact of selection over many windows amplified by training noise. The promised public code link is also absent, making it impossible to audit replication fidelity; if the MCLDNN implementation differs from the original, the entire benchmark and the SNR-window study are unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper replicates nine deep-learning automatic modulation recognition (AMR) models on the RadioML-2016A dataset, reports test accuracies under supposedly uniform training and evaluation conditions, compares accuracy against parameter count, and analyzes per-modulation and per-SNR performance. It then proposes two enhancements: training MCLDNN on a restricted SNR window (claimed optimum, -18 to 18 dB, yielding 0.63 average accuracy) and adding BiLSTM/GRU layers to existing architectures, reporting modified accuracies in Table 3. The final sections discuss future directions including synthetic data, transformers, federated learning, and ensembles.","tokens_in":11065,"tokens_out":3477,"duration_ms":36479,"significance":"If the benchmark is reproducible and the SNR-window and architecture-modification results are statistically sound, the paper would provide a useful reference point for AMR model comparison, especially because it uses a single dataset, a uniform accuracy metric, and parameter counts. The promise of publicly accessible replicated models is a concrete asset, though no link is provided. The SNR-window training result is an interesting and potentially actionable observation, and the architecture modifications are straightforward to test. However, the current lack of code, the absence of variance estimates, and the post hoc selection of the SNR window mean that the central quantitative claims are not yet established. The future-directions discussion is broad but does not add new technical content.","major_comments":[{"comment":"The central claim that training MCLDNN on the -18 to 18 dB SNR window gives the highest accuracy (0.63) is based on post hoc selection: the text says low-SNR data were gradually added from -2 to -20 dB, and only the best-performing window is reported. No standard deviation, number of seeds, or significance test is given, and the top models in Table 1 span roughly 0.53-0.60, so the 0.63 versus full-range gap may be within run-to-run variation for 100-epoch training with early stopping. Please report multi-seed means with errors and either pre-register the tested windows or confirm the optimum on a held-out set of SNR values.","section":"III.A, Figure 3"},{"comment":"The abstract and Section II state that the replicated models are publicly accessible, but the manuscript contains no repository link or code. Without the code and without a comparison of reproduced accuracies with the originally reported values in the cited papers, readers cannot verify that each replicated model matches its source implementation, preprocessing, and training budget. This is load-bearing for the benchmark claim; please provide a code link and a per-model fidelity table.","section":"II, Table 1, Abstract"},{"comment":"The architecture-modification results are reported as single accuracies per model. Differences such as 0.5423 to 0.5523 for CNN1, 0.5526 to 0.5753 for GRU, and the small decrease for MCNet are comparable to expected training noise, and the text itself attributes some differences to randomness. In addition, the insertion point, size, and number of added BiLSTM/GRU layers are not described, so the modifications are not reproducible. Please provide detailed modified-architecture descriptions and multi-seed statistics before claiming that particular models are enhanced.","section":"III.B, Table 3"},{"comment":"The claim of \"same hardware\" is not ensured by training on Google Colab, since GPU instances vary across sessions and over time. Also, early stopping means that models were trained for different numbers of epochs (39-100 in Table 1), so accuracy differences may reflect training budget rather than architecture alone. Please clarify how hardware variability was controlled, whether the early-stopping criterion and data split were identical for all runs, and how the resulting accuracy comparison is intended to be interpreted.","section":"II"}],"minor_comments":[{"comment":"The opening sentence of Section III says \"Building on discoveries found in Section I,\" but the relevant discoveries are in Section II; please correct the cross-reference.","section":"III"},{"comment":"The horizontal axis of Figure 3 labels training scenarios only as SNR ranges, and the three bars per scenario are explained only in the text; adding a legend and clear axis labels would make the figure self-contained.","section":"Figure 3"},{"comment":"Reference [28] contains \"None\" placeholders in the author list; this formatting error should be fixed.","section":"References"},{"comment":"The per-modulation table reports values such as 0.00 for CLDNN on GFSK and 0.03 for MCLDNN on GFSK; the accompanying text says \"close to 0.00,\" which is accurate, but it would be helpful to note explicitly that these are not missing entries.","section":"II, Table 2"},{"comment":"The introduction lists execution time as a challenge, but the benchmark reports only accuracy and parameter count; adding a sentence explaining that runtime is outside the scope would avoid misleading expectations.","section":"I"},{"comment":"The index term \"signal-to-noise\" appears incomplete; it should read \"signal-to-noise ratio.\"","section":"Index Terms"}],"recommendation":"major_revision","confidential_remarks":"The paper's main value is as a reproducibility-oriented benchmark, but the missing code link is a serious deficiency for a paper whose abstract promises public access to models. The post hoc SNR-window selection and the absence of variance estimates are the key technical weaknesses. I would not recommend rejection because the benchmark is a reasonable contribution and the claims can be fixed with additional experiments and reporting, but the revision needs to be substantial rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful but under-powered replication benchmark. The headline result — that training MCLDNN on an SNR window of -18 to 18 dB beats the full -20 to 18 dB range — is post hoc and not backed by any variance estimate.\n\nWhat the paper does well: it gathers nine published AMR models, runs them on the same dataset (RadioML-2016A) with a consistent metric, and reports per-SNR and per-modulation accuracy plus parameter counts. That is a genuinely useful service to a subfield where comparisons are often done on different splits and metrics. The observation that no single model dominates across modulation types is worth stating. The architecture modification (adding BiLSTM/GRU layers to all models) is routine, but at least it is applied systematically and reported in one place.\n\nThe soft spots, in order of weight. The SNR-window study in Section III.A is the weakest part. Training on several truncated SNR ranges and then reporting the best one, without error bars, multiple seeds, or a significance test, is textbook selection over the test set. The gap between 0.63 and the full-range result is plausibly training noise given the spread in Table 1. The paper should present all windows tried and some measure of run-to-run variability. Second, the abstract promises public access to the replicated models, but there is no code link or repository in the manuscript. That undermines the main value of the paper, which is reproducibility. Third, results in Tables 1 and 3 are single runs; the MCLDNN improvement from adding BiLSTM/GRU (0.598 to 0.658) is large enough to be interesting, but without variance it is hard to trust. Minor: some reference formatting issues (e.g., ref [28] has 'None' as authors).\n\nThe measured accuracy values themselves are new measurements, not copies from prior papers, so the empirical contribution is real. But I would not use the specific ranking numbers to make model-selection decisions yet. The paper is for practitioners in AMR who want a rough map of current models on a standard dataset. It deserves a serious referee, with the expectation of major revision: release the code, add seeds/error bars, and rewrite the SNR-window conclusion as a hypothesis rather than a result. I would accept it for review, not desk reject it.","headline":"A useful but under-powered AMR replication benchmark; the headline SNR-window result is post hoc and likely within training noise.","tokens_in":11553,"tokens_out":3088,"would_cite":false,"duration_ms":30290,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replicating nine AMR models on one benchmark shows MCLDNN leads and that trimming very noisy training samples raises its accuracy.","keywords":["automatic modulation recognition","deep learning","RadioML-2016A","signal-to-noise ratio","MCLDNN","modulation classification","replication benchmark","recurrent neural networks"],"falsifier":"Train MCLDNN on both the full -20 to 18 dB and the -18 to 18 dB SNR ranges with several random seeds and the same early-stopping budget; if the -18 to 18 dB advantage over the full range does not consistently reproduce, the paper's headline enhancement claim fails.","tokens_in":1590,"feed_emoji":"📡","tokens_out":1720,"duration_ms":63827,"temperature":0.7,"pith_summary":"This paper argues that the maze of published automatic modulation recognition (AMR) results can be made comparable by re-running nine high-performing models under one controlled protocol and reporting a single uniform test-accuracy metric. Using the RadioML-2016A dataset with the same 60/20/20 split, the same hardware, and the same early-stopping rule, it finds that MCLDNN achieves the highest average test accuracy at 0.5982, with IC-AMCNet close behind. The paper then shows that a deliberately chosen training SNR window improves MCLDNN's accuracy to 0.63, and that adding BiLSTM and GRU layers pushes it to 0.6580. A reader should care because this is a concrete, reproducible baseline against which future AMR models can be measured, plus two simple recipes that appear to raise accuracy without changing the dataset.","feed_headline":"A fair retest crowns MCLDNN for radio modulation recognition","feed_subtitle":"Training on a narrower SNR window pushes its accuracy to 0.63, above the full-range baseline.","key_machinery":"The load-bearing mechanism is a controlled replication protocol: the same RadioML-2016A dataset of synthetic IQ samples (11 modulation types, SNR from -20 to 18 dB), the same 60/20/20 train/validation/test split, the same test-accuracy definition averaged over all modulations and all SNR levels, and hyperparameters taken from each original paper. The paper's two interventions then probe that protocol: an SNR-windowed training schedule for MCLDNN that drops the most noise-dominated samples, and architectural augmentation that inserts BiLSTM and GRU layers into each replicated model. These interventions, not a single mathematical identity, carry the comparative claims.","core_discovery":"On a single controlled protocol, MCLDNN (a convolutional-LSTM hybrid) gives the highest average test accuracy at 0.5982 on RadioML-2016A, ahead of IC-AMCNet at 0.5870, while CGDNet trails at 0.4700. All nine models share the same qualitative behavior: test accuracy is below 0.3 for very low SNR (below -12 dB), rises gradually through the middle range, and plateaus around 0.9 even on clean signals, so none reaches perfect recognition. Restricting MCLDNN's training data to the SNR window from -18 to 18 dB rather than the full -20 to 18 dB range raises its average test accuracy to 0.63, and adding BiLSTM and GRU layers to the architecture raises it further to 0.6580. The paper also reports per-modulation accuracies showing that QPSK is easy (most models above 0.9) and GFSK is hard (best model, LSTM, reaches only 0.42).","pith_inferences":["Beyond the paper: if the SNR-window gain generalizes, every AMR study should report the SNR range used for training, because that choice alone can shift accuracy rankings.","Beyond the paper: adding BiLSTM and GRU layers increases parameter count, so a capacity-matched control experiment would separate gains from long-range memory from gains from simply having more parameters.","Beyond the paper: the paper does not compare its replicated accuracies with the numbers in the original papers, so the fairness claim is only as strong as the replication fidelity; publishing code and per-SNR raw numbers would let others verify it.","Beyond the paper: an immediate testable extension would be applying the -18 to 18 dB SNR window to the other eight models to see whether the improvement is specific to MCLDNN or a general property of noisy training data."],"forward_implications":["New AMR studies can benchmark against these nine accuracy numbers on RadioML-2016A under the same split and test-accuracy definition.","Training on a slightly narrowed SNR window (-18 to 18 dB) instead of the full -20 to 18 dB range is a zero-cost accuracy improvement for MCLDNN, raising average test accuracy to 0.63.","Adding BiLSTM and GRU layers to CLDNN, IC-AMCNet, and MCLDNN improves their test accuracy, with MCLDNN reaching 0.6580.","Recurrent and hybrid convolutional-recurrent architectures reach strong accuracy with far fewer parameters than larger CNNs, favoring them in resource-constrained settings.","Low-SNR performance remains the shared weakness: all models fall below 0.3 accuracy below -12 dB and plateau near 0.9 even on clean signals."],"supporting_citations":[{"why":"Supplies the RadioML-2016A dataset with 11 modulations and SNR range -20 to 18 dB used for every model.","marker":"[14]"},{"why":"Source of the CNN1 architecture and its suggested hyperparameters.","marker":"[16]"},{"why":"Source of the CNN2 architecture and its suggested hyperparameters.","marker":"[17]"},{"why":"Source of the CLDNN architecture, one of the models later improved by added recurrent layers.","marker":"[18]"},{"why":"Source of the IC-AMCNet architecture, the second-best original model.","marker":"[19]"},{"why":"Source of the MCNet architecture, the smallest model in the comparison.","marker":"[20]"},{"why":"Source of the LSTM architecture and the per-modulation GFSK baseline.","marker":"[21]"},{"why":"Source of the MCLDNN architecture, the best original model and the subject of the SNR-window training experiment.","marker":"[23]"},{"why":"Source of the CGDNet architecture, the lowest-accuracy original model.","marker":"[24]"}],"fun_headline_variants":["MCLDNN tops fair AMR benchmark on RadioML-2016A","Convolutional-LSTM hybrid wins radio modulation recognition","AMR benchmark: MCLDNN edges out IC-AMCNet, CGDNet lags","Training on narrower SNR range lifts MCLDNN accuracy to 0.63","Nine AMR models compared: no perfect recognition even at high SNR"],"cache_read_input_tokens":13824,"weakest_assumption_plain":"The rankings and enhancement results stand or fall on whether the nine replications truly match the architectures, preprocessing, and hyperparameters of the original papers; no independent check against originally reported accuracies is provided.","fun_headline_variants_meta":{"raw":{"variants":["MCLDNN tops fair AMR benchmark on RadioML-2016A","Convolutional-LSTM hybrid wins radio modulation recognition","AMR benchmark: MCLDNN edges out IC-AMCNet, CGDNet lags","Training on narrower SNR range lifts MCLDNN accuracy to 0.63","Nine AMR models compared: no perfect recognition even at high SNR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000717,"raw_usage":{"total_tokens":3219,"prompt_tokens":943,"completion_tokens":2276,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2174}},"tokens_in":559,"tokens_out":2276,"duration_ms":16001,"temperature":1.0,"reasoning_tokens":2174,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:47:33.684722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MCLDNN on both the full -20 to 18 dB and the -18 to 18 dB SNR ranges with several random seeds and the same early-stopping budget; if the -18 to 18 dB advantage over the full range does not consistently reproduce, the paper's headline enhancement claim fails.","supporting_citations":[{"cited_title":"Over-the-air deep learning based radio signal classification","cited_arxiv_id":null,"evidence_quote":"Supplies the RadioML-2016A dataset with 11 modulations and SNR range -20 to 18 dB used for every model."},{"cited_title":"Convolutional radio modulation recognition networks","cited_arxiv_id":null,"evidence_quote":"Source of the CNN1 architecture and its suggested hyperparameters."},{"cited_title":"Robust and fast automatic modulation classification with cnn under multipath fading channels","cited_arxiv_id":null,"evidence_quote":"Source of the CNN2 architecture and its suggested hyperparameters."},{"cited_title":"Deep architectures for modulation recognition","cited_arxiv_id":null,"evidence_quote":"Source of the CLDNN architecture, one of the models later improved by added recurrent layers."},{"cited_title":"Cnn -based automatic modulation classification for beyond 5g communications","cited_arxiv_id":null,"evidence_quote":"Source of the IC-AMCNet architecture, the second-best original model."},{"cited_title":"Mcnet: An efficient cnn architecture for robust automatic modulation classification","cited_arxiv_id":null,"evidence_quote":"Source of the MCNet architecture, the smallest model in the comparison."},{"cited_title":"Deep learning models for wireless signal classification with distributed low -cost spectrum sensors","cited_arxiv_id":null,"evidence_quote":"Source of the LSTM architecture and the per-modulation GFSK baseline."},{"cited_title":"A spatiotemporal multi- channel learning framework for automatic modulation recognition","cited_arxiv_id":null,"evidence_quote":"Source of the MCLDNN architecture, the best original model and the subject of the SNR-window training experiment."},{"cited_title":"Cgdnet: Efficient hybrid deep learning model for robust automatic modulation recognition","cited_arxiv_id":null,"evidence_quote":"Source of the CGDNet architecture, the lowest-accuracy original model."}],"review_version":1}