{"id":"07a82d0b-092e-4082-814b-4936ff0bbf08","arxiv_id":"2608.01023","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Caliber maps a target accuracy-retention level to a Gaussian logit-noise variance using a margin-normalized logistic curve, and derives the per-input query multiplier needed to average the noise away.","lead":"This paper introduces Caliber, a defense that adds random noise to the logits behind a score-returning API and calibrates the noise level so a provider can set how much accuracy an attacker can extract. It reports that a single logistic curve, after normalizing noise by the model's median logit margin, predicts the accuracy drop across more than thirty CNN, LLM, and VLM model-dataset combinations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption is that one margin-normalized logistic curve transfers across architectures; the paper's own leave-one-architecture-out results show 11–13% misses for some families, so the single-law claim is not fully established.","rationale":"The reader's weakest assumption was the transferable calibration law; I agree that this is the load-bearing point. My read does not change the verdict: the paper is honest, the proofs of Proposition 1/2 are sound as stated, and the empirical evidence is broad, but the shared-curve transfer is not yet established for all families, and the one-shot wording is looser than the per-model sweep procedure. The proposed test would settle whether the transfer failure is a rare tail or a systematic gap. No ad hominem; the concern is about the strength of an empirical generalization.","tokens_in":16375,"tokens_out":13417,"duration_ms":123694,"concrete_test":"Run a leave-one-family-out transfer test that is stronger than the paper's LOO: exclude an entire architecture family from the shared-curve fit (e.g., train the CIFAR-100 shared curve on the ResNet/VGG/MobileNet/RepVGG models only, then calibrate ViT-B/16 or a Swin-T from its margin S and baseline A0 alone; on MMLU, fit the shared curve to the non-Qwen LLMs and calibrate Qwen2.5-7B). Use Eq. (10) to hit a 75% retention target and compare measured accuracy to target. If the relative error exceeds 10% (the observed tail for ResNet/Qwen), the shared-curve law is not universal and the one-shot cross-architecture claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central algorithmic claim (Section 3.3, Eq. 9) is that after normalizing noise variance by the squared median top-two margin, the noise–utility relationship is a single monotone logistic curve that can be fit once per task and reused for unseen architectures. This is an empirical regularity, not a consequence of Proposition 1: the proposition proves monotonicity only for agreement with the clean prediction, and Corollary 1 explicitly allows accuracy to be non-monotone because small noise can correct misclassified inputs. The paper's strongest evidence for the shared curve is the leave-one-architecture-out error in Appendix A: 3.7% vs 3.3% on CIFAR-10, 8.8% vs 8.0% on CIFAR-100, and 8.2% vs 7.2% on MMLU. These means conceal uneven transfer: holding out a ResNet on CIFAR-100 costs about 12%, and the Qwen models on MMLU cost about 11–13%. The abstract's 'single-digit held-out error' is therefore not true at the per-architecture level. If a deployed architecture lands in that tail, the closed-form inversion (Eq. 10) will return a noise level that misses the target by more than claimed, and the cross-architecture contribution degrades to per-model fitting, which still requires a calibration sweep. The per-model numbers (0.6–1.4%) are in-sample fits to the same sweep used for evaluation and do not establish extrapolation to target utilities outside the fitted range.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Caliber, an output-perturbation defense for score-returning APIs that adds i.i.d. Gaussian noise to the internal logits. It proves two theoretical properties: strict monotone degradation of agreement with the clean prediction (Proposition 1), with a computable accuracy corridor (Corollary 1), and a minimax lower bound on the number of repeated queries needed to recover a single input's clean logits (Proposition 2). Methodologically, Caliber normalizes the noise variance by the squared median top-two logit margin and fits a logistic curve to the normalized noise-utility relationship, either per model or shared per task, then inverts the curve in closed form to select the noise level that achieves a target utility. Experiments on 25 vision models, 8 LLMs, and 1 VLM report per-model calibration errors of 0.6–1.4%, shared-curve leave-one-architecture-out errors of 3.7–8.8%, and end-to-end distillation experiments showing that surrogate accuracy tracks the configured degradation.","tokens_in":16689,"tokens_out":3581,"duration_ms":33006,"significance":"If the central empirical claim holds — that after margin normalization the noise-utility relationship follows a single logistic law that transfers across architectures — Caliber would provide a practical, low-cost way to set output-perturbation strength without per-model tuning, with a clean theoretical backbone. The proofs of Proposition 1 and Corollary 1 are standard and correct; the accuracy corridor is a useful conceptual contribution; and Proposition 2 is a rigorous minimax statement for the per-input recovery problem. The paper is also unusually honest in its Section 5 limitations. The main risk is precisely the transferability assumption: the paper's own leave-one-architecture-out results show per-architecture errors of 11–13% for some families, so the cross-architecture single-law claim is not fully established at the per-architecture level.","major_comments":[{"comment":"The headline cross-architecture claim is not supported at the per-architecture level. The abstract and conclusion state that the shared curve achieves 'single-digit held-out error', but Appendix A reports leave-one-architecture-out per-architecture errors of roughly 12% on CIFAR-100 for a held-out ResNet and 11–13% on MMLU for the Qwen models. These are the architectures a provider might actually serve, and at those error levels the closed-form inversion (Eq. 10) will miss the target utility by more than the paper's 'single-digit' wording suggests. Since the central contribution is the cross-architecture single-pass calibration, the paper should report the per-architecture distribution of held-out errors (not just task means), temper the abstract accordingly, or add a mechanism (e.g., per-architecture confidence bounds or a small per-architecture correction) that addresses the tail.","section":"Section 3.3 / Appendix A (Eq. 9)"},{"comment":"The per-model calibration errors (0.6–1.4% mean absolute relative error) are in-sample interpolation figures: the logistic curve is fit to the same noise-utility sweep that is then inverted at target points within that sweep's range. They do not demonstrate extrapolation to target utilities outside the fitted range or to new architectures. Section 4.1 correctly notes that the finite-sample measurement floor is about one accuracy point, so sub-percent claims should be interpreted cautiously. The abstract's '0.6–1.4%' should be qualified as in-sample interpolation at the measurement floor, and the paper should separate interpolation error from extrapolation error if any targets outside the fitted range are tested.","section":"Section 4.2 / Table 1 / Table 4"},{"comment":"The conversion from the per-input recovery bound to a utility-level cost relies on the assumption that equal mean-squared-error residuals 'act like fresh Gaussian noise of the same intensity' when read through the accuracy curve. Proposition 2 bounds squared error, not the resulting soft-label or accuracy degradation; the equivalence is exact only for the sample mean, whose residual remains Gaussian. The paper labels this a 'mean-squared-error proxy' in Section 3.5, but the abstract and Section 4.3 present the query multiplier (e.g., 'recovering an essentially clean surrogate costs ~16x') as a consequence of the framework. This is an empirical assumption, not a theorem. Either provide direct evidence that equal-MSE residuals produce comparable utility across estimators, or state in the abstract and conclusion that the utility-level cost multiplier is an empirical observation, not a proven bound.","section":"Section 3.5 (used in Section 4.3)"}],"minor_comments":[{"comment":"The relative errors in Table 1 are signed (e.g., +2.8, -0.1). Consider presenting absolute errors or adding a column for the signed error so readers can see bias direction; the text gives the mean absolute error, but the table layout would be clearer with both.","section":"Section 4.2, Table 1"},{"comment":"The product formula in Eq. (4) is an approximation, while the later proof of Proposition 1 uses Eq. (5). The text states this, but it would help to explicitly note that Eq. (4) is not used in the proofs and serves only intuition, to avoid a reader mistaking it for a derived result.","section":"Section 3.1, Eq. (4)"},{"comment":"The phrase 'parameter-free' is confusing because the calibration curve has parameters (v50, p, c) and the scale S. The intended meaning is 'leaves the served weights untouched'; consider using 'weight-free' or 'no retraining' to avoid ambiguity.","section":"Section 2.2"},{"comment":"Figure 1 summarizes the four-step calibration procedure, but the steps are not referenced in the text at Figure 1. A sentence in Section 3.4 pointing the reader to the figure would improve readability.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core theoretical results are sound and the experimental breadth is good, but the central cross-architecture claim is currently stronger than the evidence supports. With revision—specifically, reporting per-architecture held-out errors, tempering the abstract, and clarifying the interpolation/proxy status of the per-model and utility-cost claims—the paper could be acceptable. I would not reject, because the limitations section already signals many of the issues; the task is to align the framing with the data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is the most usable output-perturbation defense paper I've read recently. It doesn't invent a new attack or a magical defense; it takes the known i.i.d. Gaussian logit-noise primitive, adds a margin-normalized calibration law that inverts in closed form, and validates it across a serious number of architectures. The theory is honest and the limitations section actually reads like an author who knows where the edges are.\n\nWhat's new: the margin-normalized logistic law (Eq. 9) and its closed-form inversion (Eq. 10). The idea of normalizing noise variance by the squared median top-two margin is simple and effective. The paper correctly credits Wang et al. [45] for the Gaussian primitive and order-preservation analysis. The theoretical parts are correct: Proposition 1's strict monotonicity under unique maximizer, Corollary 1's corridor, and Proposition 2's minimax lower bound on per-input recovery are all standard but clean. The paper is explicit that Proposition 2 does not bound full extraction, which is the right scope.\n\nWhere it's soft: the load-bearing claim is empirical, not derived. The shared cross-architecture curve is a regularity, and the paper's own leave-one-architecture-out numbers show uneven transfer: 11–13% error for some held-out families on MMLU and CIFAR-100. So the abstract's 'single-digit held-out error' is only true on average. For a provider deploying one architecture, that tail matters. The per-model 0.6–1.4% errors are in-sample fits — they interpolate the same sweep they invert, not extrapolate to unseen utilities. The one-shot wording is generous; you still need a calibration sweep per model or a shared curve that may not transfer. There are no artifacts, so independent reproduction is on the reader. These are fixable, but the paper should be clearer about what's interpolation vs extrapolation.\n\nThe citation pattern is honest: the paper builds on its own prior work [45] and says so, and the related-work coverage is appropriate. The reader's conditional verdict is fair; the stress-test concern about per-architecture tails is real. Who is this for? Anyone working on model extraction defenses, and API providers who actually want a knob to turn. The extraction experiments show the defense transfers to surrogates. I'd send this to serious reviewers. It deserves a careful hearing, with a revision request to add artifacts and to report per-architecture transfer numbers rather than just means.","headline":"A solid, honestly-scoped output-perturbation defense with a useful margin-normalized calibration law; the cross-architecture transfer claim is an empirical regularity with mean-masked tails, but the paper deserves a serious peer review.","tokens_in":17237,"tokens_out":2159,"would_cite":true,"duration_ms":19401,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Caliber maps any utility target to the right noise within about 1% per model.","keywords":["model extraction defense","output perturbation","knowledge distillation","logit noise calibration","margin normalization","logistic calibration curve","minimax query cost"],"falsifier":"Numerically integrate the label-preservation probability $\\pi(\\sigma)$ of Equation (5) for a fixed input with a unique clean maximizer across increasing $\\sigma$: any rise in the curve would falsify Proposition 1. Separately, hold out an architecture whose top-two margins are heavy-tailed or class-imbalanced, calibrate it from the shared per-task curve, and measure accuracy at the 75 percent retention target: a mean absolute relative error far above the reported single-digit range would falsify the transferable law.","tokens_in":1978,"feed_emoji":"🛡️","tokens_out":3085,"duration_ms":119784,"temperature":0.7,"pith_summary":"Caliber is a defense for score-returning APIs that adds i.i.d. Gaussian noise to a model's logits before the softmax, letting a provider cap how much a distillation-based attacker can learn from the returned scores. The paper's central claim is that this intervention becomes precisely controllable: once the noise variance is divided by the squared median top-two logit margin of the served model, the drop in normalized utility follows one logistic curve across CNNs, vision transformers, LLMs, and VLMs, and the curve can be inverted in closed form to hit any target utility. Per-model calibration lands on target with mean absolute relative errors of 0.6 to 1.4 percent, and a shared per-task curve transfers to unseen architectures with single-digit held-out error. The paper also proves strict monotone agreement degradation and a minimax lower bound on the repeated queries needed to average out the noise on a single input. The authors are explicit that the transferable law is an empirical regularity, not a theorem, and that accuracy control holds only up to a computable corridor plus their empirical fit.","feed_headline":"Caliber maps any utility target to the right noise within about 1%","feed_subtitle":"A shared logistic law across 30+ model-dataset pairs lets API providers throttle distillation at a preset cost.","key_machinery":"The central object is the margin-normalized logistic law: $\\tilde{A}(v) = c + (1-c)/(1+(v/v_{50})^p)$, where $v = \\sigma^2/S^2$, $S$ is the median top-two logit margin, $v_{50}$ is the half-degradation scale, $p$ is the steepness, and $c$ is a residual floor. It is fit by bounded least squares on a single noise-free sweep and inverted in closed form, $v^* = v_{50}((1-\\tilde{A}^*)/(\\tilde{A}^* - c))^{1/p}$, then scaled back to $\\sigma^2 = v^* S^2$. This machinery converts an abstract utility target into an actual noise level without retraining or per-point search, and its strict monotonicity makes the inversion unique and well-defined for every target above the residual floor.","core_discovery":"The discovery is a normalization that makes logit-noise attacks tunable. Within one architecture, adding Gaussian noise of variance $\\sigma^2$ to logits strictly decreases the probability that the perturbed prediction agrees with the clean prediction, from 1 down to chance level $1/K$, and the exact agreement probability is a closed integral over the top-class noise (Proposition 1). The operative scale is not $\\sigma$ itself but the normalized intensity $v = \\sigma^2/S^2$, where $S$ is the median top-two logit margin on a validation set. Plotted against $v$, the normalized utility (accuracy minus the chance level, rescaled by the clean baseline) collapses onto a monotone logistic curve with a flat head, a sharp transition, and a small residual floor, across more than thirty model-dataset combinations. Fitting that curve once, per model or shared per task, and inverting it in closed form gives the $\\sigma$ that reaches any target utility; the paper reports about 1 percent per-model error and single-digit shared-curve error. For a single input, averaging $M$ repeated fresh-noise queries can at best reduce the effective noise intensity from $v$ to $v/M$, and any estimator is bounded below by this minimax risk, so recovering clean logits costs at least $v/v_{\\mathrm{use}}$ queries (Proposition 2).","pith_inferences":["Beyond the paper, a provider could tighten the shared per-task curve by using a higher-order margin statistic or the full gap distribution, since the paper's own ablation shows the mean beat the median on most models and residual error tracks order statistics beyond the top-two gap.","Beyond the paper, the minimax bound covers only per-input recovery; the true extraction cost across many inputs remains unproven, so the measured roughly 16x query multiplier should be read as an empirical floor until a surrogate-level lower bound exists.","Beyond the paper, if the logistic law survives in free-form generation, a token-level noise schedule could throttle LLM APIs as well, but sequence-level utility would need its own calibration because errors compound across tokens."],"forward_implications":["A provider can set any accuracy retention target on a served model from a single noise-free validation pass, with no per-model tuning when using the shared per-task curve.","Distilled surrogates inherit the calibrated degradation: at the 75 percent retention setting the vision surrogates fell 8.6 to 10.8 points below their undefended ceiling, and two LLM students lost their entire distillation gain.","An attacker who averages repeated fresh-noise queries on one input cannot push the effective noise below $v/M$, so recovering usable logits costs at least $v/v_{\\mathrm{use}}$ queries per input, and deterministic per-input seeding removes the free repeats entirely.","Composing the perturbation with a query-volume monitor that caps $M$ makes the defense a cost-amplification mechanism rather than a denial mechanism, so flagged accounts can be throttled instead of banned.","The same internal noise injection works for hard-label APIs by flipping a controlled fraction of labels, and majority voting recovers them only at a similar multiplicative query cost."],"supporting_citations":[{"why":"Supplies the i.i.d. Gaussian logit-noise primitive and the order-preservation product approximation that Proposition 1 and the logistic law extend.","marker":"[45]"},{"why":"Defines the knowledge-distillation loss and the dark-knowledge signal that Caliber's noise is designed to corrupt.","marker":"[12]"},{"why":"Establishes black-box surrogate extraction from score-returning prediction APIs, the attack that Caliber throttles.","marker":"[41]"},{"why":"The reverse-sigmoid output-perturbation defense whose per-model tuning Caliber claims to remove.","marker":"[23]"},{"why":"Shows attackers can distill from queried data without ground-truth labels, the scenario used in the extraction evaluation.","marker":"[32]"},{"why":"Defines the residual-network family used as vision teachers and surrogates in the end-to-end extraction study.","marker":"[11]"},{"why":"Defines CIFAR-10 and CIFAR-100, the main vision calibration testbeds across seventeen architectures.","marker":"[21]"},{"why":"Defines ImageNet-1K, the large-scale vision calibration dataset for eight architectures.","marker":"[5]"}],"fun_headline_variants":["Caliber tunes logit noise to hit any utility target","A universal logistic law for tuning API extraction defense","Caliber: the 1%-accurate dial for model extraction defense","Set the noise, preset the cost: Caliber's closed-form law","Cross-architecture caliber: throttle logit theft with one curve"],"cache_read_input_tokens":19200,"weakest_assumption_plain":"The load-bearing premise is empirical: after dividing noise variance by the squared median top-two logit margin, every architecture is assumed to follow the same logistic utility curve, so if a new architecture's falloff diverges from that curve, the shared-curve calibration fails and only per-model fitting remains.","fun_headline_variants_meta":{"raw":{"variants":["Caliber tunes logit noise to hit any utility target","A universal logistic law for tuning API extraction defense","Caliber: the 1%-accurate dial for model extraction defense","Set the noise, preset the cost: Caliber's closed-form law","Cross-architecture caliber: throttle logit theft with one curve"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000278,"raw_usage":{"total_tokens":1716,"prompt_tokens":1073,"completion_tokens":643,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":558}},"tokens_in":689,"tokens_out":643,"duration_ms":6805,"temperature":1.0,"reasoning_tokens":558,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:14:21.047611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Numerically integrate the label-preservation probability $\\pi(\\sigma)$ of Equation (5) for a fixed input with a unique clean maximizer across increasing $\\sigma$: any rise in the curve would falsify Proposition 1. Separately, hold out an architecture whose top-two margins are heavy-tailed or class-imbalanced, calibrate it from the shared per-task curve, and measure accuracy at the 75 percent retention target: a mean absolute relative error far above the reported single-digit range would falsify the transferable law.","supporting_citations":[{"cited_title":"In: USENIX Security (2016)","cited_arxiv_id":null,"evidence_quote":"Establishes black-box surrogate extraction from score-returning prediction APIs, the attack that Caliber throttles."},{"cited_title":"In: IEEE S&P Workshops (2019)","cited_arxiv_id":null,"evidence_quote":"The reverse-sigmoid output-perturbation defense whose per-model tuning Caliber claims to remove."},{"cited_title":"In: CVPR (2019)","cited_arxiv_id":null,"evidence_quote":"Shows attackers can distill from queried data without ground-truth labels, the scenario used in the extraction evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines CIFAR-10 and CIFAR-100, the main vision calibration testbeds across seventeen architectures."}],"review_version":1}