{"id":"d1e799cb-ef70-4687-ab57-d506c8616150","arxiv_id":"2411.11795","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A large-scale benchmark shows that JPEG AI resists most tested adversarial attacks better than other neural codecs, though its high-complexity mode is less robust than its base mode.","lead":"This paper benchmarks how well JPEG AI, the first neural-network-based image compression standard, survives adversarial attacks, comparing it with ten other neural codecs. It finds JPEG AI is relatively robust but still vulnerable, and that simple reversible preprocessing defenses can blunt most tested attacks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"JPEG AI robustness may be overstated because attacks are optimized against the core model only, while other codecs are attacked end-to-end; the comparative ranking needs an end-to-end attack check.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: attacks are constructed against the core JPEG AI model rather than the full codec configuration, yet robustness is measured on the full codec. I agree that this is the most fragile premise underlying the central claim. If JPEG AI's additional tools act as implicit defenses or gradient masks, the reported 'relatively high robustness' could be an artifact of the attack interface rather than a property of the deployed codec. The paper's own wording in Section 4.6 supports this reading, and the central claim in Section 5.2 rests on comparisons across codecs and versions that are not all attacked with the same degree of end-to-end fidelity. This does not make the claim false, but it makes it conditional on the core-only attack setup being representative. A targeted experiment attacking the full codec end-to-end would settle the question. Because the reader's verdict is already CONDITIONAL and explicitly flags this caveat, my stress-test does not change the verdict. I would keep CONDITIONAL rather than ACCEPT or REJECT: the concern is concrete and testable, but the existing evidence is sufficient for a conditional conclusion, and the proposed check could either confirm or overturn it. Secondary issues such as unreported attack hyperparameters and missing error bars reinforce the conditional status but are less load-bearing than the core-versus-full-codec asymmetry.","tokens_in":14559,"tokens_out":2838,"duration_ms":30996,"concrete_test":"Re-run the Section 5.2 comparison for JPEG AI 4.1/5.1/6.1 HOP/BOP with attacks optimized end-to-end through the full codec, including RVS, all filters, and Channel-Wise Gain, using straight-through gradient estimation for quantization, with the same attack budgets and hyperparameters as the core-only runs. Then check whether the orderings 'JPEG AI more robust than other NICs', 'BOP more robust than HOP', and '6.1 more robust than 5.1' persist under per-launch error bars. If the full-codec attack yields significantly larger ΔVMAF/ΔPSNR than the core-only attack, the reported ranking is not robust to attack-surrogate choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim in Section 5.2—that JPEG AI shows relatively high robustness, that HOP versions are less robust than BOP, and that robustness improves from 5.1 to 6.1—depends on an asymmetric attack setup. Section 4.6 states: 'a modified interface for attacking the core model was used. As a result, constructed attacks depended less on the choice of the codec configuration.' The attack's impact was then assessed 'not only on the core model's result but also on the entire codec with additional tools included from the codec's base configuration.' Thus the adversarial perturbation is optimized against the core encoder-decoder only, while the measured effect includes RVS, filters (Adaptive Re-Sampler, ICCI, LEF, Non-Linear Chroma Filter), and Channel-Wise Gain. For the other NIC models, there is no mention of excluding modules from the attack, so attacks are presumably full-model. This asymmetry can inflate JPEG AI's measured robustness: the optimizer never sees the additional tools, so any robustness that those tools confer is not challenged by the attack. It can also bias version and HOP/BOP comparisons if the tool configurations differ across versions. If a full-codec attack is substantially more effective, the paper's central ranking would not survive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a methodology for measuring the adversarial robustness of neural image codecs (NICs) using ΔPSNR, ΔMSE, ΔMS-SSIM, and ΔVMAF, and applies it in a large-scale comparison of 10 NIC models, including three versions of JPEG AI at two operation points each. Six white-box attacks, ten optimization losses, four datasets, and several preprocessing defenses are evaluated. The central empirical claims are that JPEG AI is relatively robust compared with other NICs, that high-operation-point (HOP) versions are less robust than base-operation-point (BOP) versions, and that robustness improved from version 5.1 to 6.1. The paper also reports attack transferability, artifact analyses, and bitrate-increase effects, and releases code and results online.","tokens_in":14953,"tokens_out":3028,"duration_ms":31588,"significance":"If the robustness ranking is correct, this is an important result: JPEG AI is the first neural compression standard, and knowing whether standard white-box attacks can break it, and whether higher-efficiency configurations are riskier, is directly relevant to deployment. The study is one of the largest of its kind, covering many codecs, attacks, losses, and defenses, and the use of standard full-reference quality metrics rather than fitted models makes the delta scores transparent and reproducible in principle. The transferability experiment across JPEG AI versions is a useful contribution, and the public release of code and evaluation data is a strength.","major_comments":[{"comment":"The attacks against JPEG AI are optimized using a modified interface that attacks only the core encoder-decoder, while the measured effect is computed on the full codec including RVS, filters, and Channel-Wise Gain, per the text: 'a modified interface for attacking the core model was used' and 'the attack's impact was assessed not only on the core model's result but also on the entire codec with additional tools included.' For the other NICs, the description indicates standard full-model attacks. This asymmetry is load-bearing for the central claim in Section 5.2 (that JPEG AI shows relatively high robustness, that HOP is less robust than BOP, and that 6.1 improves on 5.1). If the additional JPEG AI tools act as implicit defenses that the optimizer never sees, the measured robustness is not the robustness of the shipped codec. The authors should either run at least a subset of attacks end-to-end against the full JPEG AI configuration or explicitly justify why core-only attacks are the correct threat model, and they should report how the HOP/BOP and version comparisons change when the additional tools are included in the attack graph.","section":"4.6"},{"comment":"The implementation details state that each attack was run four times with varied learning rate, number of iterations, and perturbation bound, but none of these values or ranges are reported. Given that the paper's main comparisons are qualitative rankings without error bars or significance tests, the reader cannot assess whether the reported differences between codecs, between HOP and BOP, and between JPEG AI versions are larger than run-to-run variability or attack-strength variability. At minimum, the paper should report the exact attack settings and per-condition standard deviations or confidence intervals, and ideally a statistical test for the key pairwise claims in Section 5.2.","section":"4.6 and 5.2"}],"minor_comments":[{"comment":"The text says 'except for ΔNIQE' but NIQE is not introduced among the quality metrics in Section 4.5; either define it (and the corresponding delta) or remove the reference.","section":"5.4"},{"comment":"The caption of Figure 5 says 'Spearman Correlation Coefficients' while the text in the same section says 'pairwise Pearson correlation coefficients'; please make the statistic used consistent.","section":"5.4"},{"comment":"Reference [11] and reference [12] are the same paper, and the entry for [24] has the author order malformed ('et al Liu, Kang'); please correct these.","section":"References"},{"comment":"The sentence 'Fig. 2 shows how they influence the quality of uncompressed images' is unclear; since the figure shows Δ-metrics, it should probably say how the losses affect the adversarial images before compression or how each loss changes the delta metrics.","section":"5.1"}],"recommendation":"major_revision","confidential_remarks":"The core experimental design is sound in its scale and use of standard metrics, but the asymmetric attack setup for JPEG AI is a genuine methodological risk that could change the headline ranking. If the authors add end-to-end attacks on the full JPEG AI configuration, or convincingly argue that the additional tools are irrelevant to the attack surface, the paper would be a strong candidate for acceptance. The lack of attack hyperparameter reporting and significance testing needs to be fixed regardless."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis paper is the first large-scale robustness comparison for JPEG AI, and that alone makes it worth reading. The authors test three JPEG AI versions (4.1, 5.1, 6.1) at both HOP and BOP against six white-box attacks, ten losses, four quality metrics, on four datasets, plus transferability matrices and a defense screen. The scale is serious, and the presentation of the delta-metric methodology – extending Chen and Ma's delta-PSNR to delta-MSE, delta-MS-SSIM, and delta-VMAF – is clean.\n\nWhat worries me is in Section 4.6. For JPEG AI, the adversarial perturbation is optimized against the core encoder-decoder only, not against the full codec with its extra tools (RVS, filters, channel-wise gain). The attack's effect is then measured on the full codec. For the other NICs, no such separation is described, so they are presumably attacked end-to-end. That asymmetry can flatter JPEG AI: if the extra tools are not part of the attack objective, any robustness they contribute is never tested. The authors justify this as 'simplifying backward propagation,' but that is not a neutral choice. A proper white-box robustness evaluation should attack whatever system is being evaluated. Without an end-to-end attack on JPEG AI – or at least a control showing core-only and full-codec attacks give similar results – the central claim that JPEG AI is relatively robust, and the HOP/BOP and version comparisons, are not fully supported.\n\nOther soft spots are minor but real: attack hyperparameters (learning rate, iterations, epsilon) are not reported, so the 'averaged over four runs' claim is hard to reproduce; there are no error bars or significance tests on the rankings; and the artifact-correlation analysis uses a filtered subset of about 3000 images, which should be stated more prominently.\n\nThe paper does not overclaim. The conclusion says JPEG AI showed relatively high robustness, which is exactly the claim that the attack asymmetry puts at risk. I would not reject the paper for this; the benchmark and methodology are valuable, and the fix – attacking the full codec and reporting hyperparameters – is straightforward, if expensive. Given the importance of JPEG AI as a likely consumer standard, this deserves serious refereeing. I'd send it to review with a request for major revision.\n\nFor a reading group, I'd bring it – it is a good springboard for discussing how to define robustness for systems with non-differentiable or optional components.\n\nBest,\n[Your name]","headline":"Broad, useful robustness benchmark for JPEG AI, but the central ranking may be inflated by attacking the core model instead of the full codec.","tokens_in":15382,"tokens_out":3596,"would_cite":true,"duration_ms":35069,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper demonstrates that JPEG AI, the first end-to-end neural image compression standard, is relatively more robust to white-box adversarial attacks than other neural codecs, with base-operation-point versions more robust than…","keywords":["adversarial robustness","neural image compression","JPEG AI","white-box attacks","adversarial purification","transferability","image codecs","VMAF"],"falsifier":"Run the same six white-box attacks against the complete JPEG AI configuration with RVS, filters, and gain enabled, and compare the $\\Delta$VMAF rankings. If the high-operation-point versions no longer appear less robust than base-operation-point versions, or the version-to-version robustness differences change sign, the core-model ranking does not represent the shipping codec.","tokens_in":14387,"feed_emoji":"🛡️","tokens_out":4873,"duration_ms":41315,"temperature":0.7,"pith_summary":"This paper asks whether JPEG AI, the first international standard for end-to-end neural image compression, can survive adversarial attacks that alter pixels to wreck the decoded image or inflate file size. It builds a testbed of six white-box attacks, four full-reference quality metrics, and ten neural codecs, and finds that JPEG AI is comparatively robust: it loses less decoded-image quality under attack than most competitors. High-operation-point versions are the exception, being less robust, and newer versions are safer than older ones. The paper also shows that simple reversible preprocessing defenses largely neutralize the attacks, and that adversarial images grow compressed-file size even when the attack does not target bitrate. If these results hold, the first standard neural codec is not trivially breakable, but its high-efficiency settings deserve extra scrutiny before deployment.","feed_headline":"JPEG AI resists adversarial attacks better than rival neural codecs","feed_subtitle":"First neural compression standard holds up under white-box attacks; high-efficiency modes are its weak spot.","key_machinery":"The load-bearing machinery is the $\\Delta$score methodology: the difference in a full-reference quality metric (PSNR, MSE, MS-SSIM, VMAF) between the codec's reconstruction of a clean image and of its adversarial counterpart. This turns robustness into a single comparable number across codecs, attacks, and loss functions. The methodology is paired with a modified attack interface that perturbs the JPEG AI core model while evaluating effects on the full codec with its optional tools enabled, plus a transferability variant $\\hat{\\Delta}$score that measures how well an attack aimed at one codec degrades another codec's output.","core_discovery":"The central discovery is a robustness ranking: under a battery of six white-box attacks (FTDA, I-FGSM, MADC, PGD, SSAH, cAdv) plus Gaussian noise, JPEG AI shows relatively high robustness compared to other NIC models, while the diffusion-based CDC shows the lowest robustness. Within JPEG AI, the high-operation-point (HOP) versions are less robust than base-operation-point (BOP) versions, and robustness improves from version 5.1 to 6.1. Attacks constructed on the core model transfer across JPEG AI bitrates and versions, and adversarial inputs also increase the bitrate of compressed images, harming both quality and bandwidth. Simple reversible transformations\\u2014flip, random roll, and random ensemble\\u2014serve as effective purification defenses against these attacks.","pith_inferences":["Because adversarial attacks transfer between JPEG AI bitrates and versions, an attacker who can craft perturbations against any public version could plausibly aim them at a newer, unreleased version, making robustness certification a release-gate issue for the standard.","The finding that attacks inflate compressed bitrate even when not targeting BPP suggests a denial-of-service path: a small, invisible perturbation could multiply storage or bandwidth costs in large-scale deployments.","The defense results hint that random transforms are effective not because JPEG AI is inherently weak but because the attack's gradient is brittle; a testable extension would be to attack the defended pipeline end-to-end and see whether robustness collapses.","The reliance on full-reference quality metrics leaves open whether attacks that fool these metrics also degrade perceived quality; a human-study extension could connect the $\\Delta$metric scale to subjective annoyance."],"forward_implications":["If JPEG AI is deployed in consumer devices, the base operation point is a safer default than the high operation point under a white-box threat model.","Simple reversible preprocessing defenses\\u2014flip, random roll, and random ensemble\\u2014can nullify several of the tested attacks without retraining the codec.","Adversarial attacks increase compressed-file size even when they do not target bitrate, so a small perturbation can inflate storage and bandwidth costs.","Attack success transfers across JPEG AI bitrates and versions, so each new release of the codec should be re-evaluated for robustness before wide adoption.","JPEG AI's robustness improved from version 5.1 to 6.1, indicating that the standard's development process is already making the codec harder to attack."],"supporting_citations":[{"why":"Defines the JPEG AI standard and its operation points, the object under evaluation.","marker":"[5]"},{"why":"Supplies the $\\Delta$score methodology that this paper extends to multiple full-reference metrics.","marker":"[12]"},{"why":"Introduces the FTDA attack and the geometric self-ensemble defense used here.","marker":"[11]"},{"why":"Provides the MADC gradient-projection attack.","marker":"[35]"},{"why":"Provides the I-FGSM iterative attack.","marker":"[21]"},{"why":"Provides the PGD attack.","marker":"[27]"},{"why":"Provides the color and texture artifact metrics used to analyze attack effects.","marker":"[33]"},{"why":"Provides the BSQ-rate metric for codec compression-efficiency comparison.","marker":"[45]"}],"fun_headline_variants":["JPEG AI resists six attack types, outlasting rival neural codecs","First neural compression standard JPEG AI proves robust under adversarial fire","JPEG AI's weak spot: high-efficiency modes; simple flips defend it","Simple flips and rolls defend JPEG AI from adversarial attacks","JPEG AI more robust than diffusion-based codecs to attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The main assumption is that attacking the stripped-down core encoder\\u2013decoder of JPEG AI, rather than the full codec with its optional tools (RVS, filters, and gain) enabled, reveals the robustness that matters in real deployments.","fun_headline_variants_meta":{"raw":{"variants":["JPEG AI resists six attack types, outlasting rival neural codecs","First neural compression standard JPEG AI proves robust under adversarial fire","JPEG AI's weak spot: high-efficiency modes; simple flips defend it","Simple flips and rolls defend JPEG AI from adversarial attacks","JPEG AI more robust than diffusion-based codecs to attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001559,"raw_usage":{"total_tokens":6174,"prompt_tokens":836,"completion_tokens":5338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":5246}},"tokens_in":452,"tokens_out":5338,"duration_ms":33958,"temperature":1.0,"reasoning_tokens":5246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:07:00.433543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same six white-box attacks against the complete JPEG AI configuration with RVS, filters, and gain enabled, and compare the $\\Delta$VMAF rankings. If the high-operation-point versions no longer appear less robust than base-operation-point versions, or the version-to-version robustness differences change sign, the core-model ranking does not represent the shipping codec.","supporting_citations":[{"cited_title":"The jpeg ai standard: Providing efficient human and machine vi- sual data consumption","cited_arxiv_id":null,"evidence_quote":"Defines the JPEG AI standard and its operation points, the object under evaluation."},{"cited_title":"Toward robust neural image com- pression: Adversarial attack and model finetuning","cited_arxiv_id":null,"evidence_quote":"Supplies the $\\Delta$score methodology that this paper extends to multiple full-reference metrics."},{"cited_title":"Toward robust neural image com- pression: Adversarial attack and model finetuning","cited_arxiv_id":null,"evidence_quote":"Introduces the FTDA attack and the geometric self-ensemble defense used here."},{"cited_title":"Simoncelli","cited_arxiv_id":null,"evidence_quote":"Provides the MADC gradient-projection attack."},{"cited_title":"Goodfellow, and Samy Bengio","cited_arxiv_id":null,"evidence_quote":"Provides the I-FGSM iterative attack."},{"cited_title":"Towards deep learn- ing models resistant to adversarial attacks","cited_arxiv_id":null,"evidence_quote":"Provides the PGD attack."},{"cited_title":"Jpeg ai image compression visual artifacts: Detection methods and dataset, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the color and texture artifact metrics used to analyze attack effects."},{"cited_title":"Bsq-rate: a new approach for video-codec performance comparison and drawbacks of current solutions","cited_arxiv_id":null,"evidence_quote":"Provides the BSQ-rate metric for codec compression-efficiency comparison."}],"review_version":1}