{"id":"70ccf08d-625d-49c8-a618-aa312f5afc8b","arxiv_id":"2507.07768","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TRIX dynamically mixes targeted and untargeted adversarial training per class to improve worst-class clean and robust accuracy, with gains on CIFAR-10, CIFAR-100, and STL-10.","lead":"TRIX is a new adversarial training method that gives weak image classes stronger untargeted attacks and strong classes weaker targeted attacks during training. It improves worst-class accuracy on CIFAR and STL benchmarks while keeping average accuracy, a practical step toward fairer robust models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (9) reverses the KL argument in the targeted branch relative to the objective defined in Eq. (4); because KL is asymmetric, the evaluated 'targeted' loss may not be the targeted TRADES objective that motivates the claim.","rationale":"The reader's weakest assumption concerned the reliability of the softmax-based class weights in Eq. (7) as a proxy for genuinely weak classes. That is a valid concern, but I find a more direct, checkable problem in the specification of the training objective itself: Eq. (9) reverses the order of the KL divergence in the targeted branch relative to Eq. (4). Since the whole contribution is the targeted-versus-untargeted policy, this is load-bearing: if the implementation matches Eq. (9), the empirical results do not support the stated mechanism; if the implementation matches Eq. (4), the paper's equations are internally inconsistent and need correction. The proposed audit and rerun is inexpensive relative to the paper's own training budget (about 12.5 GPU-hours for five CIFAR-10 ResNet-18 seeds) and directly tests whether the headline worst-class improvement survives under the objective the paper claims. I keep the reader's CONDITIONAL verdict but sharpen the condition: the authors should either show that Eq. (9)'s direction is a typo and the code implements Eq. (4), or re-report results under the corrected objective.","tokens_in":20464,"tokens_out":11122,"duration_ms":119997,"concrete_test":"Audit the released GitHub code's targeted-branch loss. In the TRIX training loop, determine whether the targeted term is computed as F.kl_div(F.log_softmax(f(x_adv_t)), F.softmax(f(x))) — which realizes Eq. (4)'s KL(fθ(x)||fθ(xadv_t)) — or as the reverse. If it is the reverse, rerun the Table 1 CIFAR-10/ResNet-18 protocol (5 seeds, AutoAttack, ε=8/255) with the targeted term replaced by Eq. (4)'s direction, holding all other hyperparameters fixed. If the worst-class robust accuracy shifts by more than about 1 point (TRIX beats DAFA by 2.45 points there), the paper's headline improvement is attributable to the reversed KL rather than the mixed targeted/untargeted policy.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central mechanism is that targeted adversaries are assigned to strong classes and untargeted adversaries to weak classes. Section 2 defines the targeted TRADES term as KL(fθ(x)||fθ(xadv_t)) in Eq. (4). Section 5, Eq. (9), implements the targeted branch as KL(fθ(xadv_ti)||fθ(xi)) — the reverse order. Because KL is not symmetric, these are different losses: the stated objective penalizes change from the clean prediction to the targeted-adversarial prediction, while the reversed form penalizes the opposite direction, which interacts differently with the target-optimized xadv_t and with model confidence. If the released code follows Eq. (9), the reported results test a different algorithm than the one the paper motivates; if the code follows Eq. (4), then Eq. (9) is wrong and the code is the only way to know which objective was actually used. This matters because the targeted/untargeted asymmetry is the paper's novel component. The ablation in Table 2 also suggests the full gain over DAFA is modest (TRIX 38.90 vs DAFA 36.50 robust worst-class with PGD, and TRIX-Basic only 28.23), so an implementation artifact in the targeted branch could explain or invert the reported advantage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TRIX, an adversarial training method built on TRADES that adaptively assigns targeted adversarial perturbations to strong classes and untargeted perturbations to weak classes, based on class weights derived from average softmax predictions. It combines this selection with per-class loss weighting and perturbation-radius scaling, and evaluates the method on CIFAR-10, CIFAR-100, STL-10, and Tiny-ImageNet across ResNet-18, PreActResNet-18, and WideResNet-28-10. The central empirical claim is that TRIX improves worst-class clean and robust accuracy and reduces inter-class robustness disparities while preserving average accuracy.","tokens_in":20784,"tokens_out":5415,"duration_ms":58264,"significance":"If the empirical claims hold, TRIX is a useful contribution to the adversarial-fairness literature: it demonstrates that a class-dependent mix of targeted and untargeted TRADES objectives can improve worst-class robustness, and the evaluation across datasets, architectures, and AutoAttack/PGD is broad. The method is simple enough for others to build on, and the ablations in Tables 2 and 6 give some evidence that the components matter. The paper also provides code and reports multiple seeds. However, the significance is currently limited by an ambiguity in whether Eq. (9) implements the stated objective, the absence of significance testing for the claimed improvements, and an under-specified class-weight rule that drives the method's core policy; these issues must be resolved before the central claims can be assessed.","major_comments":[{"comment":"The targeted branch in Eq. (9) is written as KL(fθ(x_adv_ti) || fθ(xi)), whereas the objective introduced in Eq. (4) is KL(fθ(xi) || fθ(x_adv_t)). Since KL divergence is not symmetric, these are different losses: Eq. (4) penalizes changes from the clean prediction to the targeted-adversarial prediction, while Eq. (9) penalizes the reverse direction. Because the targeted-versus-untargeted assignment is the paper's central novelty, the authors must state which direction was actually optimized in the experiments, correct Eq. (9) if it is a typographical error, and confirm that the released code implements the corrected objective.","section":"§5, Eq. (9); §2, Eq. (4)"},{"comment":"The abstract and Section 6 claim that TRIX 'significantly improves' worst-class accuracy, but no significance tests are reported and several key comparisons are within one standard deviation. For example, Table 1b reports CIFAR-100 robust worst-class accuracy of 2.89 ± 0.46 for TRIX versus 2.41 ± 0.45 for DAFA, and Table 1a reports CIFAR-10 average robust accuracy of 49.09 ± 0.24 for TRIX versus 49.74 ± 0.20 for TRADES. Appendix A.2 also states that for non-CIFAR-10 datasets baseline hyperparameters were selected as 'the best obtained,' which leaves the comparison protocol underspecified. The authors should report paired significance tests or confidence intervals over the five seeds and clarify how baseline hyperparameters were selected.","section":"§6, Tables 1–4; Appendix A.2"},{"comment":"The class-weight rule in Eq. (7) is under-specified. As written, wc = 1 + λ Σ_{j≠c}(Sc,j·Sj,j if Sc,c < Sj,j else −Sj,c·Sc,c) mixes row and column entries of the matrix S, can produce negative updates, and does not correspond to a 'feature similarity distance' as claimed; Eq. (5) defines S from average softmax probabilities, not from feature representations. Since Eq. (8) uses only w to choose targeted versus untargeted training, this ambiguity prevents readers from re-implementing the method and leaves unaddressed the risk that early-training miscalibration of w sends untargeted budget to strong classes. Please provide a precise definition, a worked example, and a sensitivity analysis of the policy with respect to w.","section":"§5, Eq. (7)"},{"comment":"The training protocol is internally inconsistent. Section 6 says loss weighting and perturbation scaling are applied after a warm-up phase of τ = 70 epochs, but Algorithm 1 labels epochs 1 through τ as the 'Main TRIX Training Phase' and epochs τ+1 through T as the 'Warm-up Phase (TRADES)'; the appendix text then states that the weights used at τ are computed from adversarial examples, while Eq. (5) defines them on clean samples. Please clarify which loss is used in each phase and on which predictions the class weights are computed.","section":"Algorithm 1 and §6"}],"minor_comments":[{"comment":"Figure 4 in the main text and Figure 5 in the appendix appear to be the same figure with the same caption; one should be removed or replaced with a distinct analysis.","section":"Figures 4 and 5"},{"comment":"There are several typographical errors: 'acheieves' in Section 6, and 'momentunm' and 'SDG' in Appendix A.2; additionally, Algorithm 1 uses LTRADES without defining the weighted form used with class weights.","section":"Throughout"},{"comment":"Eq. (9) states that fθ denotes the model's predicted logits, but KL divergence is defined between probability distributions; the paper should specify that softmax probabilities are used and keep notation consistent with Eq. (4).","section":"§5, Eq. (9)"},{"comment":"Reference [27] appears to cite a different DAFA paper with a different author list and venue from the DAFA method cited as [35] and actually used in the experiments; please correct the citation.","section":"References"},{"comment":"Theorem 1 is presented as a theorem but its proof assumes linear classifiers and equidistant class centers; it should be labeled as an informal motivation rather than a theorem about deep networks, since the paper's empirical claims do not depend on it.","section":"Appendix A.5"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting, but the KL direction in Eq. (9) versus Eq. (4) is a load-bearing ambiguity that the editor may want resolved by inspecting the released code. The contribution relative to DAFA is incremental in the loss-weighting and scaling components; the targeted/untargeted switch is the main novelty, so its implementation must be exact. The missing significance testing is also important given the modest margins in several tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TRIX is a solid incremental contribution to adversarial fairness. The genuinely new piece is the per-class assignment of targeted versus untargeted adversaries, decided by a softmax-similarity class weight and paired with uniform target sampling. That combination is not in DAFA or FAIR-TAT, and the evaluation is unusually thorough for this line: three datasets, three architectures, AutoAttack, five seeds, ablation study, and a code link. The worst-class robust accuracy gains are consistent across the board, and the Table 2 ablation indicates the mixed policy adds something beyond the DAFA-style weighting.\n\nSoft spots, in rough order. First, the KL direction mismatch. Equation (4) defines the targeted branch as KL(fθ(x) || fθ(xadv_t)), but Equation (9) implements KL(fθ(xadv_ti) || fθ(xi)) — the reverse. KL is asymmetric, so these are different losses. The paper never mentions this reversal, and the code is the only way to know which objective was actually used. This matters because the targeted/untargeted asymmetry is the novel mechanism; a flipped KL could change the effective training signal. It needs a fix, a justification, or a code-based clarification.\n\nSecond, the baseline tuning protocol. For datasets other than CIFAR-10, the baselines' hyperparameters were selected as the 'best obtained' while TRIX used its own settings. That favors the home team. Several of the headline gaps over DAFA are within one standard deviation, and no significance tests are reported. The pattern is plausible, but not airtight.\n\nThird, the theory appendix is informal. The One-vs-One/One-vs-All analogy is fine motivation, but labeling the linear-separability arithmetic as Theorem 1 oversells it.\n\nOn balance, the central claim—that this mixed-adversary scheme improves worst-class robust accuracy without hurting average accuracy—holds up qualitatively. The KL issue needs resolution before I would trust the exact numbers, but it does not look like a fundamental flaw. The paper is aimed at people working on adversarial training and robustness fairness; it is a useful data point for that community.\n\nMy recommendation: send it to peer review. A competent referee should ask for the code check, cleaner baseline tuning, and significance tests. I would read the revised version.","headline":"Consistent worst-class gains and a genuinely mixed adversarial policy, but the targeted KL divergence is reversed between Eq. (4) and Eq. (9), and baseline tuning is favorable.","tokens_in":21222,"tokens_out":3701,"would_cite":true,"duration_ms":36594,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TRIX claims that a class-aware mix of targeted and untargeted adversarial training, steered by softmax-based feature similarity, improves worst-case class accuracy and reduces inter-class robustness disparity without sacrificing overall…","keywords":["adversarial training","class-wise robustness","adversarial fairness","targeted adversarial attacks","untargeted adversarial attacks","worst-class accuracy","feature similarity","TRADES"],"falsifier":"A direct test: at the warm-up epoch, record TRIX's per-class weights and correlate them with end-of-training class-wise robust accuracies; if they do not select the classes that end up weakest, the mechanism is not doing the claimed work. A second decisive check is to replace the softmax proxy with ground-truth class difficulty; if worst-class gains vanish, the mixed objective alone is not the cause.","tokens_in":20301,"feed_emoji":"🎯","tokens_out":8640,"duration_ms":88405,"temperature":0.7,"pith_summary":"Adversarial training normally applies the same attack recipe to every class, but classes are not equally hard to attack: classes with overlapping or fragile features stay disproportionately vulnerable. TRIX (TRades-based mIXed adversarial training) claims this disparity should be handled by choosing the type of adversary per class, sending weaker targeted adversaries toward uniformly sampled wrong labels for strong classes and stronger untargeted adversaries for weak classes, with per-class loss weights and perturbation scaling derived from the model's clean-sample confidence. The paper reports that this mixed objective improves worst-case class accuracy on both clean and adversarial samples across CIFAR-10, CIFAR-100, STL-10, and Tiny-ImageNet, while keeping average accuracy close to the TRADES baseline. If those results hold, adversarial defenses can be fairer across classes without the usual sacrifice in overall robustness.","feed_headline":"Tailored attack types per class lift worst-case robustness","feed_subtitle":"Targeted for strong classes, untargeted for weak: worst-class accuracy rises without hurting average.","key_machinery":"The load-bearing mechanism is the mixed TRADES objective with a class-conditional policy. Adaptive class weights $w_c$ are computed from the class similarity matrix $S$, whose rows are the average softmax predictions $\\bar p_c$ on clean samples; Eq. (7) upweights classes with low self-confidence and high inter-class overlap. Each sample is then assigned either a targeted adversarial KL term toward a uniformly sampled wrong label or an untargeted adversarial KL term, depending on whether its class weight is below or above the batch average, and the same weights scale the cross-entropy loss and perturbation radius in Eq. (10). This couples the choice of adversary type and strength to a learned estimate of class difficulty.","core_discovery":"The central claim is that the type of adversary used during adversarial training should be a per-class decision, not a global constant. The authors build a class weight from the matrix of average predicted distributions on clean samples, upweighting classes with low self-confidence and high overlap with others, and use that weight to split training: weak classes receive untargeted adversarial perturbations with larger strength and higher loss weight, while strong classes receive targeted perturbations toward uniformly sampled wrong labels. In their experiments this policy raises worst-class robust accuracy under PGD and AutoAttack on several benchmarks and improves the reported fairness metric $\\varrho$ relative to TRADES and fairness-oriented baselines, while maintaining average accuracy. The explanation offered is asymmetric suppression of non-robust features: strong classes shed non-robust features early in training, whereas weak classes need broader untargeted pressure to separate from their confusing neighbors.","pith_inferences":["Recomputing class weights from adversarial predictions throughout training, rather than only at the warm-up epoch, could track shifts in class difficulty and is a natural variant the paper does not test.","Because targeted targets are drawn uniformly, a confusion-guided target selection for strong classes could push the same idea further; the paper's one-vs-one versus one-vs-all justification suggests targeted losses only need to refine a few close neighbors, so choosing those neighbors explicitly should concentrate capacity.","The softmax-similarity proxy is architecture- and data-dependent, so a portable version would need a calibration or normalization step before the method is applied to datasets with very different class priors or label noise.","A stress test for the fairness claim would evaluate TRIX under distribution shift, since weights are estimated on clean training samples and a shifted evaluation set could have a different set of genuinely weak classes."],"forward_implications":["On CIFAR-10, TRIX lifts worst-class robust accuracy under AutoAttack from 23.11% with TRADES to 32.65%, while average robust accuracy stays essentially unchanged at 49.09% versus 49.74%.","On CIFAR-100 and STL-10 the same pattern holds: worst robust accuracy rises to 2.89% and 10.97% respectively, and the fairness score $\\varrho_{\\text{robust}}$ reaches 0.92 and 0.44, indicating the worst-class gain is not bought by average accuracy.","The ablations show each component contributes: turning off loss and perturbation weighting or using uniform weights lowers worst robust accuracy, and full TRIX reaches 38.9% worst robust accuracy under PGD on CIFAR-10, above the closest fairness baseline DAFA at 36.5%.","The improvement transfers to other architectures, including PreActResNet-18 and WideResNet-28-10, and to Tiny-ImageNet, where TRIX raises the minimum 20% worst-class robust accuracy from 1.19% to 2.96% on ResNet-18."],"supporting_citations":[{"why":"Supplies the TRADES objective TRIX modifies, balancing clean cross-entropy loss with KL divergence between clean and adversarial predictions.","marker":"[24]"},{"why":"Provides the distance-based per-class loss weighting and perturbation margin adjustment scheme that TRIX builds on for its adaptive class weights.","marker":"[35]"},{"why":"Introduces targeted adversarial training guided by class-to-class confusion, the precedent for using targeted adversaries to shape decision boundaries.","marker":"[29]"},{"why":"Supports the empirical premise that strong classes suppress non-robust features early in training, justifying the split between targeted and untargeted adversaries.","marker":"[38]"},{"why":"Defines the worst-class robustness goal and the $\\varrho$ fairness metric used to measure whether gains in the worst class outpace average-accuracy changes.","marker":"[32]"},{"why":"Provides the AutoAttack evaluation procedure used to test robust worst-class accuracy under strong adversarial attacks.","marker":"[53]"}],"fun_headline_variants":["Per-class attack choice boosts worst-class robust accuracy","Weak classes get stronger untargeted attacks in TRIX","TRIX tailors attacks per class to close robustness gaps","Adaptive adversarial attack types per class improve fairness","Mixed adversarial training reduces inter-class robustness disparity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The policy assumes that a class's average softmax confidence on clean samples marks it as weak or strong, so early-training noise, mislabeled examples, or poor calibration could push stronger untargeted attacks to classes that do not actually need them.","fun_headline_variants_meta":{"raw":{"variants":["Per-class attack choice boosts worst-class robust accuracy","Weak classes get stronger untargeted attacks in TRIX","TRIX tailors attacks per class to close robustness gaps","Adaptive adversarial attack types per class improve fairness","Mixed adversarial training reduces inter-class robustness disparity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1498,"prompt_tokens":946,"completion_tokens":552,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":477}},"tokens_in":562,"tokens_out":552,"duration_ms":5228,"temperature":1.0,"reasoning_tokens":477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:32:34.995207+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: at the warm-up epoch, record TRIX's per-class weights and correlate them with end-of-training class-wise robust accuracies; if they do not select the classes that end up weakest, the mechanism is not doing the claimed work. A second decisive check is to replace the softmax proxy with ground-truth class difficulty; if worst-class gains vanish, the mixed objective alone is not the cause.","supporting_citations":[{"cited_title":"Theoretically principled trade-off between robustness and accuracy","cited_arxiv_id":null,"evidence_quote":"Supplies the TRADES objective TRIX modifies, balancing clean cross-entropy loss with KL divergence between clean and adversarial predictions."},{"cited_title":"Dafa: Distance- aware fair adversarial training","cited_arxiv_id":null,"evidence_quote":"Provides the distance-based per-class loss weighting and perturbation margin adjustment scheme that TRIX builds on for its adaptive class weights."},{"cited_title":"Fair-tat: Improving model fairness using targeted adversarial training","cited_arxiv_id":null,"evidence_quote":"Introduces targeted adversarial training guided by class-to-class confusion, the precedent for using targeted adversaries to shape decision boundaries."},{"cited_title":"Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data","cited_arxiv_id":"2410.08503","evidence_quote":"Supports the empirical premise that strong classes suppress non-robust features early in training, justifying the split between targeted and untargeted adversaries."},{"cited_title":"Wat: improve the worst-class robustness in adversarial training","cited_arxiv_id":null,"evidence_quote":"Defines the worst-class robustness goal and the $\\varrho$ fairness metric used to measure whether gains in the worst class outpace average-accuracy changes."},{"cited_title":"Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks","cited_arxiv_id":null,"evidence_quote":"Provides the AutoAttack evaluation procedure used to test robust worst-class accuracy under strong adversarial attacks."}],"review_version":1}