{"id":"7a8fdbdc-15c0-4bc1-9012-b6b9716d23c2","arxiv_id":"2502.03619","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Training a CNN on a dataset augmented across defender number, motion, and noise yields more accurate swarm-tactic classification, and optimizing defender trajectories against the trained network can raise its confidence (STP) and reduce the number of defenders needed.","lead":"This paper tests whether neural networks can classify swarm attack tactics when conditions change after training, by enriching training data across defender count, defender motion, and measurement noise. It also proposes an optimal-control framework that steers defender trajectories to elicit adversary responses the classifier is most confident about, while respecting operational constraints.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Optimization results report only STP (the NN's own confidence) and never show that optimized defender trajectories improve actual classification accuracy, leaving the deployment claim unsupported.","rationale":"The robustness contribution is partly convincing: combined-dataset CNNs do show higher test-set accuracy across defender number, motion, and noise, though error bars and significance tests are absent. The optimization contribution, however, rests entirely on STP, which is the NN's self-reported confidence rather than an external measure of correct classification. Since the same network proposes the labels and scores the optimization, there is a clear risk of confidence exploitation: fmincon can push defender trajectories toward regions of input space where the network's softmax is high for the simulated true tactic without improving (or even while degrading) discriminative accuracy. The paper's own emphasis on saliency gradients (Fig. 13) as a mechanism for improving STP makes this risk concrete. A held-out top-1 accuracy test would settle whether the optimization actually 'maximizes the probability of correct NN tactic classification' or merely inflates confidence. I partially agree with the reader's weakest assumption: needing exact PAk(PD) models is a real external-validity limitation, but the internal validation gap is more fundamental because it affects the claim even when the models are available. The verdict should remain CONDITIONAL: the robustness finding can stand, but the optimization claim needs this accuracy check before it can be accepted.","tokens_in":15557,"tokens_out":7672,"duration_ms":74062,"concrete_test":"Reproduce the Section IV.D optimization for 50 held-out random seeds using the same 'Comb DM+' NN and a fixed allowable area. For each seed, solve the STP optimal control problem for a given initial motion (e.g., Star), then compute the NN's top-1 classification accuracy on the adversary trajectories induced by the optimized defender motion and by the initial defender motion. Report the mean accuracy difference and 95% confidence interval. If the optimized motion does not significantly improve accuracy over initial motion, the STP gains do not establish the claimed maximization of correct classification probability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central deployment claim is that optimized defender trajectories maximize the probability of correct NN tactic classification (Abstract; Section II.D). The only success metric reported is STP, the sum of softmax probabilities the trained NN assigns to the true tactic, computed with the same simulation models and the same NN used as the optimization objective. Figures 9, 11, 12, and 14 show STP increases (e.g., oSTP=399/400), but no experiment reports whether optimized defender motion changes the NN's argmax label or improves top-1 accuracy on held-out engagements. STP is not a calibrated probability of correct classification: a high diagonal entry can coincide with a wrong argmax if another class receives higher probability, and the objective can be exploited through NN gradient structure (as the saliency maps in Fig. 13 suggest). Section IV.E asserts that 'Required STP ≥ 390' ensures 'minimum NN prediction accuracy of 90%' without providing a calibration curve. Thus the optimization framework's central claim is unvalidated even within the paper's own simulation setting, independent of the external concern about requiring perfect adversary response models.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two linked contributions for classifying swarm tactics from simulated trajectory data. First, it enriches the training data of a CNN time-series classifier by varying three 'variables of interest'—defender number, defender motion, and measurement noise—and shows that a CNN trained on the combined dataset outperforms CNNs trained on individual sub-datasets when tested on held-out sub-dataset test sets. Second, it casts defender trajectory design as an optimal control problem whose objective is to maximize the trained CNN's Sum of True Predictions (STP), i.e., the sum of softmax probabilities assigned to the true tactic under all possible adversary tactics. The authors use this framework to study how optimized defender motion affects STP and to estimate the minimum number of defenders needed to reach a required STP threshold.","tokens_in":15841,"tokens_out":3280,"duration_ms":33916,"significance":"If the claims are upheld, the paper would provide a practical recipe for making a pretrained swarm-tactic classifier robust to operational variability and a principled way to steer defender motion to improve classifier confidence. Strengths of the paper include the systematic treatment of three separate uncertainty axes, the use of held-out sub-dataset test sets for the robustness comparisons, and the release of a public code repository. The robustness result is internally consistent and, if replicated with error bars, would be a solid empirical contribution. The optimization contribution is conceptually interesting but, as detailed below, is currently validated only by the classifier's own confidence scores rather than by actual classification accuracy, which is a load-bearing gap for the central deployment claim.","major_comments":[{"comment":"The central claim that optimized defender trajectories 'maximize the probability of correct NN tactic classification' is not tested. All optimization results (Figures 9, 11, 12, 14) report STP, which is the trained NN's own softmax output, but no experiment reports whether optimized defender motion improves top-1 accuracy or changes the argmax label on held-out engagements. STP is not a calibrated estimate of classification accuracy: a high diagonal entry can coexist with a wrong argmax if another class receives even higher probability. The paper needs to report accuracy, confusion matrices, or calibration curves for optimized versus initial trajectories on held-out simulation data, not just STP values.","section":"Section IV.D and Abstract"},{"comment":"The minimum-defender analysis is based on a single engagement example and a single trained model, with no repeated seeds or error bars. The claim that 'Required STP ≥ 390' ensures 'minimum NN prediction accuracy of 90%' is asserted without any calibration experiment. Since the STP-to-accuracy mapping is not established, the derived minimum defender counts are not supported. The authors should repeat the optimization across multiple engagement initializations and, ideally, multiple training seeds, and show a calibration curve between STP and actual classification accuracy.","section":"Section IV.E, Figure 15"},{"comment":"The optimization objective requires the defender to compute PAk(PD), the exact adversary trajectory response for every possible tactic k, for any candidate defender trajectory PD. This presumes perfect knowledge of adversary dynamics and initial states over the whole observation window. In an actual deployment, the true tactic is unknown, so the defender cannot enumerate the needed responses without already possessing the model the classifier is meant to discover. This limitation should be stated explicitly, and the framework should be stress-tested under model mismatch—for example, optimize using nominal adversary dynamics and then evaluate STP and accuracy under perturbed dynamics or partially observed states.","section":"Section II.D and Section II.A"},{"comment":"The robustness comparisons rest on a single training run per architecture and dataset combination. Given the central claim that combined-dataset CNNs 'consistently' outperform single-condition CNNs, the absence of error bars or multiple random seeds weakens the statistical support. The paper should either provide repeated-training variability or temper the 'consistently' language to reflect the single-seed evidence.","section":"Section IV.A-C"}],"minor_comments":[{"comment":"The constraint notation is introduced informally; for example, 'PD(t) ∈ X' is not defined as a set, and the min/max velocity and acceleration constraints are written as separate inequalities without explicit variable definitions in the displayed problem. A table of symbols or a more complete mathematical statement would improve readability.","section":"Section II.D"},{"comment":"The truncation of datasets to the minimum time length (e.g., Combined ND has max time steps 45 while the original has 58) is mentioned but its effect on classification is not analyzed. The authors should state how much time is lost and whether this truncation affects the comparison across VOIs.","section":"Section III.A and Table II"},{"comment":"The statement that 'random generator seeds above 1,200 or 10,000 were used' is vague. Please specify exactly which seeds were used for each optimization experiment and how they were chosen, to support reproducibility.","section":"Section III.C"},{"comment":"The texts '5 NN' and '10 DN NN' are unclear; it should say 'five networks' and '10-defender NN' respectively. Also, the accuracy values printed on the plot are difficult to read and should be moved or formatted.","section":"Figure 4"},{"comment":"The legend distinguishes solid and dotted lines but the text refers to 'ensemble of 51 NN'; please clarify whether the dotted line is the ensemble of all noise-specific NNs or the combined-dataset NN, and make the legend consistent with the caption.","section":"Figure 8"},{"comment":"The saliency maps are described as supporting the claim that the 'Better NN' has improved gradients, but no quantitative measure of saliency is given. A simple metric (e.g., mean absolute gradient across inputs) would make the comparison more rigorous.","section":"Section IV.D, Figure 13"}],"recommendation":"major_revision","confidential_remarks":"The robustness half of the paper is reasonably solid and likely of interest to the swarm-classification community. The optimization half is the more novel part but needs significant additional validation before the deployment claim can be accepted. If the authors can supply held-out accuracy results for optimized trajectories and a calibration of STP to accuracy, I would be willing to reconsider. I did not find evidence of inappropriate citation or novelty issues, though the reliance on a single simulation model for both training and optimization should be made explicit in the final version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: the robustness half is credible and useful; the optimization half is not yet validated. The authors extend their TAES 2024 work by training CNNs on datasets that vary defender count, defender motion, and measurement noise. The combined-condition NNs consistently beat condition-specific NNs on held-out sub-dataset test sets. That is a clean, reproducible result, with code public. It gives a single classifier that works across engagements it wasn't trained on, which is real value.\n\nThe optimization framework is the newer piece: cast defender trajectory design as an optimal control problem that maximizes the sum of true predictions (STP) of the trained NN, subject to airspace and kinematic constraints. The formulation is sensible, and the plots show STP going up (e.g., from 345 to 367, and up to 399/400 with a larger NN). But that is the problem: the only success measure is STP, the NN's own softmax confidence for the true tactic. They never show that optimized defender motion actually changes the NN's argmax or improves top-1 accuracy on held-out engagements. STP is not calibrated accuracy. Section IV.E asserts \"Required STP ≥ 390\" guarantees \"minimum NN prediction accuracy of 90%\" with no calibration curve. That is a load-bearing claim for the minimum-defender analysis.\n\nThere is also a deeper conceptual issue: computing STP requires PAk(PD), the adversary response for every possible tactic, given the defender trajectory. The defender must already know the adversary behavior model the classifier is supposed to be discovering. In the simulation environment this is available; in the field it isn't. So the deployment story is weaker than the abstract implies.\n\nMinor caveats: the robustness figures have no error bars, and the minimum-defender curve comes from a single engagement seed. The NN architecture changes between experiments (input length, filters), though that is defensible as fair comparison with prior work.\n\nWho should read this? Anyone working on data augmentation for time-series classification or on using ML for autonomous swarm defense. The robustness finding deserves a serious referee. The optimization needs substantial additional validation before it should be promoted. I would send it to review, with the clear instruction that the STP-only evaluation and the missing calibration are major revision items.","headline":"Robustness half is solid and worth citing; the trajectory optimization half is a good idea that is undermined by evaluating only the NN's own confidence (STP) rather than actual classification accuracy.","tokens_in":16309,"tokens_out":2953,"would_cite":true,"duration_ms":28533,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CNNs trained on enriched swarm data classify adversary tactics under uncertainty, and optimized defender motion pushes correct-classification probability near its maximum.","keywords":["swarm tactics","neural network time series classification","data augmentation","robustness","optimal control","defender trajectory optimization","autonomous swarms","measurement noise"],"falsifier":"Evaluate the robust 'Combined' network on held-out engagements whose defender count, defender motion, and noise level all fall outside every training range (e.g., 16–20 defenders, a sixth motion type, and noise levels 51–60); if it does not beat the best single-condition network on that held-out region, the robustness claim is false.","tokens_in":15371,"feed_emoji":"🛡️","tokens_out":11694,"duration_ms":90010,"temperature":0.7,"pith_summary":"Building on the authors' earlier work, this paper claims that a neural network trained on engagements spanning the expected range of operational conditions — defender count from 1 to 15, five defender motion patterns, and measurement noise from 0 to 50 — classifies a swarming adversary's tactic more accurately under uncertainty than a network trained on any single condition. The paper further claims that defender trajectories are controllable inputs: solving an optimal control problem that maximizes the Sum of True Predictions (the network's probability assigned to the true tactic, summed over all tactics) yields defender motions that raise classification confidence while respecting speed, acceleration, collision, and operating-area constraints. If correct, this lets defenders identify an adversary swarm's tactics with fewer assets than the original 10-vs-10 setting required, and lets a planner read off the minimum number of defenders needed for a chosen confidence threshold. The evidence is built from simulated swarm-on-swarm engagements with four known adversary tactics (Greedy, Greedy+, Auction, Auction+).","feed_headline":"Combined training data cuts defenders needed to read swarm tactics","feed_subtitle":"Accuracy holds when defender counts, motions, and noise change; optimized motion raises it further.","key_machinery":"The central object is the Sum of True Predictions (STP), defined as the trace of the matrix $P_{\\text{stack}}$ whose entry $P_{kl}$ is the trained neural network's predicted probability of tactic $l$ when tactic $k$ is the true tactic generating the adversary response; thus $\\mathrm{STP}(P_D) = \\sum_{k=1}^{n} P_{kk}$, where $P_D$ is the matrix of all defender trajectories. STP is the objective of an optimal control problem: the defender dynamics are a double integrator, the decision variables are the defender trajectories $P_D$, each tactic's adversary response $P_{A_k}(P_D)$ feeds the network, and the constraints fix initial positions and bound velocity, acceleration, inter-defender separation, and the operating area. Maximizing STP by nonlinear programming steers the adversaries into motions that the network reads confidently. The supporting mechanism for robustness is dataset enrichment: sub-datasets spanning each Variable of Interest are generated independently and joined into a 'Combined' dataset with consistent scaling, and a CNN is trained on it.","core_discovery":"The authors establish two things. First, a 'combined' training dataset that varies defender number, defender motion, and measurement noise yields a CNN whose test accuracy stays high across the whole range of those variables, whereas networks trained on a single setting develop bias toward that setting and degrade on dissimilar conditions. Second, defender motion can be optimized for a trained network: treating the adversary's trajectory as a deterministic function of the defender trajectory for each tactic, and maximizing the trace of the network's probability-output matrix (the Sum of True Predictions) over candidate defender trajectories — subject to velocity, acceleration, collision-avoidance, and operating-area constraints — produces defender motions that raise the true-tactic probability. With a larger training set the optimized Sum of True Predictions reached 399 of a possible 400, and repeating the optimization for 1 to 10 defenders yields a 'Best STP @ ND' curve that identifies the minimum defender count needed for a required confidence level.","pith_inferences":["Inferred: in a real engagement the true tactic is unknown, so the optimizer cannot enumerate the adversary responses $P_{A_k}(P_D)$ from a simulator. A practical variant would optimize defender motion against the classifier's own saliency gradients, treating the network as the response model; that closed loop is not tested here.","Inferred: the same 'probe by controlled motion, then read intent' logic transfers to other multi-agent settings — e.g., a vehicle choosing a small maneuver to disambiguate another driver's intent — whenever a differentiable response model is available.","Inferred: the minimum-defender result is demonstrated for a single engagement seed; a robust planner would want the 'Best STP @ ND' curve averaged over many seeds and engagement geometries before committing assets.","Inferred: comparing the combined network against an ensemble of single-condition networks with equal total training budget would isolate whether the gain comes from data diversity or simply from more training examples."],"forward_implications":["A planner can use the 'Best STP @ ND' curve to choose the minimum number of defenders needed for a desired classification confidence, replacing the original study's fixed 10-defender baseline.","The combined-dataset training recipe extends to other deployment variations the paper lists as future work, such as formation shape, intraswarm dispersion, weapon and sensor range, and velocity or acceleration limits.","Because the optimal control formulation includes operating-area and collision constraints, the classifier becomes usable in restricted airspace, waterways, or other geometrically constrained environments.","Investing in more training instances compounds: the larger 'Combined DM+' network shows stronger input-to-output gradients, and the optimizer exploits those gradients to reach near-maximum STP from multiple initial defender motions.","A network trained on a single defender motion is biased toward similar motions (e.g., Star and Semi perform alike, Straight performs worst), so combined training is a prerequisite for the optimization framework to work across motion types."],"supporting_citations":[{"why":"Supplies the swarm-engagement simulator, the four adversary tactics, the CNN time-series classifier, and the baseline dataset that this paper enriches and optimizes.","marker":"[20]"},{"why":"Provides the nonlinear programming methods used to solve the defender-trajectory optimal control problem.","marker":"[23]"},{"why":"The MATLAB fmincon solver the paper uses for the numerical optimization.","marker":"[24]"},{"why":"The importNetworkFromTensorFlow tool that brings the trained TensorFlow CNN into the MATLAB optimization environment.","marker":"[25]"}],"fun_headline_variants":["Training on varied swarm data shrinks defender force needed","Optimized motion helps AI read swarm tactics with fewer defenders","Varied training data makes swarm-tactic AI robust to uncertainty","Defender paths tuned to boost swarm classification accuracy","Robust AI reads swarm tactics despite noise and motion changes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The optimization's value depends on the defender already having a complete and accurate model of how the adversary will respond for every tactic in the library, which is exactly the knowledge the classifier is meant to supply in the field.","fun_headline_variants_meta":{"raw":{"variants":["Training on varied swarm data shrinks defender force needed","Optimized motion helps AI read swarm tactics with fewer defenders","Varied training data makes swarm-tactic AI robust to uncertainty","Defender paths tuned to boost swarm classification accuracy","Robust AI reads swarm tactics despite noise and motion changes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000557,"raw_usage":{"total_tokens":2640,"prompt_tokens":927,"completion_tokens":1713,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":1634}},"tokens_in":543,"tokens_out":1713,"duration_ms":11233,"temperature":1.0,"reasoning_tokens":1634,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T04:19:02.054385+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the robust 'Combined' network on held-out engagements whose defender count, defender motion, and noise level all fall outside every training range (e.g., 16–20 defenders, a sixth motion type, and noise levels 51–60); if it does not beat the best single-condition network on that held-out region, the robustness claim is false.","supporting_citations":[{"cited_title":"Swarm Charac- teristics Classification Using Neural Networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the swarm-engagement simulator, the four adversary tactics, the CNN time-series classifier, and the baseline dataset that this paper enriches and optimizes."},{"cited_title":"Nocedal and S","cited_arxiv_id":null,"evidence_quote":"Provides the nonlinear programming methods used to solve the defender-trajectory optimal control problem."},{"cited_title":"Math- Works, Inc., 2024","cited_arxiv_id":null,"evidence_quote":"The MATLAB fmincon solver the paper uses for the numerical optimization."},{"cited_title":"MathWorks, Inc., 2024","cited_arxiv_id":null,"evidence_quote":"The importNetworkFromTensorFlow tool that brings the trained TensorFlow CNN into the MATLAB optimization environment."}],"review_version":1}