REVIEW 3 major objections 6 minor 1 cited by
Saffron-1: Safety Inference Scaling
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Inference-time scaling can defend LLMs against jailbreaks, and replacing the process reward model with a multifurcation reward model makes the defense roughly three times more compute-efficient than Best-of-N sampling at roughly half the…
desk verdict The MRM vector-output mechanism is genuinely new and well-engineered, but the evaluation is circular because the same Llama Guard judge both supervises training and measures attack success, so the safety gains should be treated as preliminary until confirmed by an independent judge. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The multifurcation reward model (MRM) is the load-bearing object: a decoder-only Transformer that maps a sequence to a reward vector over the vocabulary instead of a scalar, so one call replaces many process-reward-model calls. It is trained with partial supervision, meaning only the next token that actually appears in a training prefix is supervised, with all process rewards precomputed and stored in the released Safety4M dataset. A conservative exploration constraint sets the predicted rewards of unseen tokens to $-\infty$ to keep tree search from exploring unsupported tokens, and a Trie-based key–value cache shares attention caches across sequences with common prefixes, with a proven reduction in time and space complexity.
What would settle it
Re-score the same SAFFRON-1 and Best-of-N outputs with human raters or a different safety classifier and compare attack success rates: if the reported gap of roughly 0.1–0.2 or the threefold compute advantage disappears, the claimed safety improvement is specific to optimizing the Llama Guard judge rather than a general reduction in harm.
Extended reading notes
Core claim
The paper's central claim is that the bottleneck in safety-oriented inference scaling is not exploration itself but the cost of reward evaluation: every process-reward-model call adds overhead that can outweigh the safety benefit of exploring more candidates. A multifurcation reward model $M_\theta:\mathcal{V}^+\to\mathbb{R}^{\mathcal{V}}$ computes, in one call, an approximate process reward $M_\theta(s)_a\approx R(sa)$ for every next token $a$, reducing the number of reward-model calls per search step from the number of candidate tokens to one. Trained by partial supervision on token-level rewards precomputed by Llama Guard 3 1B, restricted to seen tokens, and accelerated by Trie-based KV cache sharing, this model makes tree search efficient enough to outperform Best-of-N on both tested benchmarks and to shift the inference scaling curve by roughly a factor of three in compute.
Load-bearing premise
The argument rests on Llama Guard 3 1B as the judge of attack success; the same model provides the process rewards used to train the MRM and to score the Best-of-N baselines, so the reported safety gains are measured against this one judge.
Editorial extensions
If this is right
- Inference-time computation can act as a safety defense on its own: SAFFRON-1 substantially lowers attack success rates even though the policy model is not trained against Prefilling Attack.
- At a fixed attack-success target, safety scaling becomes roughly three times cheaper, needing about 60 TFLOP to reach an ASR near 0.4 where the strongest baseline needs about 190 TFLOP.
- At a fixed compute budget, tree-search safety defense beats sampling-based defense, roughly halving the attack success rate relative to Best-of-N in the reported comparisons.
- The trained MRM and the token-level reward dataset are reusable assets: any policy sharing the tokenizer can use the same MRM, and any tree search method that previously called a PRM can substitute the MRM directly.
Reading between the lines
- Because the judge and reward source are the same model, the reported gains quantify optimization of Llama Guard 3 1B; re-scoring with human raters or a different classifier would show how much of the safety improvement generalizes.
- The MRM is effectively a Q-function over tokens, so the same one-call vector scoring idea could reduce reward-evaluation costs in other open-ended scoring tasks, such as factuality or helpfulness ranking, where process rewards are expensive.
- Trie-based key–value sharing is independent of safety and could equally benefit reasoning-oriented tree search, although the paper does not test that transfer.
- The stated tokenizer dependence implies each new policy tokenizer needs its own MRM; a tokenizer-agnostic MRM would be a natural next step for wider deployment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SAFFRON, an inference-time safety-scaling method that replaces a standard process reward model (PRM) with a 'multifurcation reward model' (MRM) which outputs a reward vector over the next-token vocabulary, so that one MRM call scores all candidate continuations of a prefix. The MRM is trained by partial supervision to approximate the token-level rewards of Llama Guard 3 1B on the released Safety4M dataset, with a conservative constraint to suppress unseen tokens and a Trie-based KV cache to share computation across beam-search branches. The experiments on Harmful HEx-PHI and Ai2 Refusals under Prefilling Attack report that SAFFRON-1 achieves lower attack success rates (ASR) than Best-of-N, Rebase, and DeAL, and better scaling efficiency (Figure 2, Tables 2-3). The paper also provides two theoretical propositions (on unseen-token unreliability and on Trie cache complexity) and releases code, model, and data.
Significance. If validated, the work is significant: it opens a new direction of applying inference scaling to safety assurance, and the MRM idea (a single reward-vector call per prefix instead of multiple PRM calls) is a clever and practical way to reduce inference cost during tree search. The released SAFERON-1 model and Safety4M dataset are potentially valuable community resources. The paper also contains useful theoretical analysis, including a proof that unseen-token bias gradients are zero and a complexity bound for Trie-based KV cache sharing. However, the current empirical validation relies entirely on Llama Guard 3 1B both as the source of the MRM's training supervision and as the judge for ASR, which severely limits the evidential value of the safety claims until an independent evaluation is supplied.
major comments (3)
- [§4.2, §5.1, Table 2] The safety evaluation is circular. The MRM is trained to approximate Llama Guard 3 1B's token-level rewards (Eq. 5), the beam search in SAFFRON-1 selects continuations by maximizing that MRM's predictions (Eq. 3), and the ASR reported in Table 2 is computed by the same Llama Guard 3 1B model. The optimization objective and the evaluation metric are therefore the same function, so the reported ASR reductions partly measure how well SAFFRON-1 optimizes the judge's own score rather than safety independently. The paper needs an external judge (e.g., GPT-4- or human-annotated safety labels, or at least a different classifier such as Llama Guard 2) to support the central claim of 'robust and effective LLM safety.' The illustrative case studies in Section 5.6 do not substitute for a systematic cross-judge evaluation.
- [Tables 2, 3, Figure 2] All quantitative claims are presented as single point estimates without error bars, multiple seeds, or significance tests. Ai2 Refusals has only 200 prompts, so the difference between SAFFRON-1's ASR of 0.175 and Best-of-N's 0.285 could be within run-to-run variance; the same holds for the scaling curves in Figure 2 and the width sweep in Table 3. The authors should report means and standard errors over at least three random seeds, and ideally a paired significance test, to make the comparative claims convincing.
- [§3.1, §5.1, Appendix A.1] The computation of ASR and the accounting of TFLOP are under-specified. It is not stated how Llama Guard 3 1B's raw safety score is converted into the binary harmless/harmful decision underlying ASR, nor whether the 'excluded' prefilled tokens are excluded only from the judge input or also from the reward model scoring. The TFLOP measurements for the baselines and SAFFRON-1 are also not defined (e.g., whether the MRM's full output head, the Trie-based KV cache, and the policy forward passes are included). These details are necessary to reproduce the main scaling-efficiency results and to fairly compare methods.
minor comments (6)
- [Appendix A.1] The MRM training hyperparameters contain a blank: 'with learning rate .' — the numeric value appears to be missing and should be filled in.
- [Section 4.2] The name 'SAFFRON-1' is used both for the full tree-search method and for the trained MRM model itself, which can confuse readers; please rename one of the two, for example by calling the MRM 'SAFFRON-1 reward model' throughout.
- [Section 7] The limitation statement says 'safety inference scaling applies only to closed-source LLMs,' but the experiments use Llama 3 8B, which has open weights; please rephrase to 'models for which a suitable judge/PRM and tokenizer are available' or otherwise clarify the intended scope.
- [Section 4.4] In Proposition 2, the complexity reduction term uses the maximum over permutations, which is shown in the proof to be achieved by lexicographic order; adding a sentence in the main text that the MRM calls are processed in lexicographic order would make the claimed bound less puzzling.
- [Section 5.6] The text refers to tokens marked in red and yellow, but these colors are not visible in the plain-text version; please ensure the final PDF displays them or describe the marking differently.
- [Section 2 / Eq. (2)] The definition of the top-p set N_p^π(s) is non-standard and could be made clearer by explicitly stating that it corresponds to the nucleus (top-p) token set; a brief rephrasing or a reference to Holtzman et al. (2020) would help.
Circularity Check
SAFFRON-1's reported ASR gains are partly by construction: the MRM is fitted to Llama Guard 3 1B rewards, and the same model measures attack success, so no independent judge or human evaluation supports the central safety claim.
-
fitted input called prediction
[Sections 3.1, 4.2 (Eqs. 3 and 5), 5.1, Table 2]
"For the safety process reward model (PRM) R, we use Llama Guard 3 1B (Meta, 2024)... Prefilled tokens are excluded from safety evaluation. ... we pre-compute all process rewards R(s[0:j+1)) before training the MRM and store them as a token-level safety reward dataset. ... LMRM(s[0:j+1)) := (Mθ(s[0:j))sj − R(s[0:j+1)))2 ... B′ := argtop N_{sa: s∈B, a∈Nπp(s)} Mθ(s)a ... Following Section 3.1, we use Llama 3 8B as the policy model, Llama Guard 3 1B as the safety reward model, ASR as the safety metric, and Best-of-N (Sampling), Rebase (Beam Search), and DeAL (MCTS) as baselines."
The MRM Mθ is trained to approximate R (Llama Guard 3 1B) on Safety4M via Eq. (5). SAFFRON-1's beam selection chooses continuations with maximal Mθ(s)a via Eq. (3), i.e., maximal predicted R. The reported ASR in Table 2 is then measured with the same R used as the 'safety reward model' and 'ASR' safety metric (Sections 3.1 and 5.1). Thus the objective being optimized and the metric being reported are the same function up to MRM approximation error; lowering ASR under this judge is partly by construction. The paper offers no human evaluation or an independent second judge to show the selected outputs are safer, so the central safety claim is not externally anchored.
full rationale
The paper's efficiency and complexity contributions are largely self-contained: Proposition 2's Trie-based KV-cache bound is proved in the appendix without relying on the fitted judge, and Proposition 1 is a gradient calculation. The TFLOP accounting appears fair, including applying Trie caching to Best-of-N. However, the headline safety result — ASR 0.175 on Ai2 Refusals and 0.409 on Harmful HEx-PHI — rests on Llama Guard 3 1B being simultaneously the PRM whose token rewards supervise the MRM (Eq. 5), the model whose predicted rewards drive SAFFRON-1's token selection (Eq. 3), and the model that computes the reported attack success rate (Sections 3.1 and 5.1). This is a same-judge loop: the fitted reward model is an approximator of the evaluation metric, so the measured improvement is partly by construction and does not establish safety under a different judge or human raters. I therefore score 6: partial circularity driven by the identity between training target and evaluation metric, not by self-citation.
Assumptions & free parameters
free parameters (4)
- top-p threshold p =
0.8
- search width N =
varied 1 to 64; N=32 in main comparison
- TFLOP_Lim for ScalEff =
500 (Beam Search), 100 (MCTS)
- MRM learning rate =
not reported
assumptions (4)
- domain assumption Llama Guard 3 1B provides a valid measure of LLM output safety.
- domain assumption The MRM's reward vector accurately approximates the PRM's scalar rewards for all relevant next tokens.
- domain assumption Prefilling Attack is a representative and sufficiently challenging jailbreak for evaluating safety inference scaling.
- standard math A lexicographic sort maximizes total prefix overlap for the Trie caching bound.
Cite this review
Pith. "Pith review of Saffron-1: Safety Inference Scaling." pith.science (2026). https://pith.science/paper/IOCZIVFQ
@misc{pith2026250606444,
author = {Pith},
title = {Pith review of: Saffron-1: Safety Inference Scaling},
year = {2026},
howpublished = {\url{https://pith.science/paper/IOCZIVFQ}},
note = {Machine review of arXiv:2506.06444}
}
read the original abstract
Existing safety assurance research has primarily focused on training-phase alignment to instill safe behaviors into LLMs. However, recent studies have exposed these methods' susceptibility to diverse jailbreak attacks. Concurrently, inference scaling has significantly advanced LLM reasoning capabilities but remains unexplored in the context of safety assurance. Addressing this gap, our work pioneers inference scaling for robust and effective LLM safety against emerging threats. We reveal that conventional inference scaling techniques, despite their success in reasoning tasks, perform poorly in safety contexts, even falling short of basic approaches like Best-of-N Sampling. We attribute this inefficiency to a newly identified challenge, the exploration--efficiency dilemma, arising from the high computational overhead associated with frequent process reward model (PRM) evaluations. To overcome this dilemma, we propose SAFFRON, a novel inference scaling paradigm tailored explicitly for safety assurance. Central to our approach is the introduction of a multifurcation reward model (MRM) that significantly reduces the required number of reward model evaluations. To operationalize this paradigm, we further propose: (i) a partial supervision training objective for MRM, (ii) a conservative exploration constraint to prevent out-of-distribution explorations, and (iii) a Trie-based key--value caching strategy that facilitates cache sharing across sequences during tree search. Extensive experiments validate the effectiveness of our method. Additionally, we publicly release our trained multifurcation reward model (Saffron-1) and the accompanying token-level safety reward dataset (Safety4M) to accelerate future research in LLM safety. Our code, model, and data are publicly available at https://github.com/q-rz/saffron , and our project homepage is at https://q-rz.github.io/p/saffron .
Figures
Figures from the paper (1 more)
Forward citations
Cited by 1 Pith paper
-
A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.
Reference graph
Works this paper leans on
-
[4]
Le, Christopher Ré, and Azalia Mirhoseini
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V . Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787,
-
[5]
Chan, Jui-Hung Cheng, Mao Xun Huang, Chao-Ting Chen, and Hen-Hsen Huang
Brian J. Chan, Jui-Hung Cheng, Mao Xun Huang, Chao-Ting Chen, and Hen-Hsen Huang. Ef- ficient beam search for large language models using Trie-based decoding.arXiv preprint arXiv:2502.00085,
-
[6]
Group fairness via group consensus
Eunice Chan, Zhining Liu, Ruizhong Qiu, Yuheng Zhang, Ross Maciejewski, and Hanghang Tong. Group fairness via group consensus. InThe 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1788–1808,
work page 2024
-
[7]
WAPITI: A Watermark for Finetuned Open-Source LLMs
Lingjie Chen, Ruizhong Qiu, Siyu Yuan, Zhining Liu, Tianxin Wei, Hyunsik Yoo, Zhichen Zeng, Deqing Yang, and Hanghang Tong. W APITI: A watermark for finetuned open-source LLMs.arXiv, 2410.06467,
-
[8]
Rm-r1: Reward modeling as reasoning.arXiv preprint arXiv:2505.02387,
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-r1: Reward modeling as reasoning.arXiv preprint arXiv:2505.02387,
-
[9]
DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning
DeepSeek. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint, 2501.12948,
-
[10]
Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,
-
[12]
James Y . Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. DeAL: Decoding-time alignment for large language models.arXiv preprint arXiv:2402.06147,
Show all 30 references
-
[14]
Model-free graph data selection under distribution shift.arXiv preprint arXiv:2505.17293, 2025a
Ting-Wei Li, Ruizhong Qiu, and Hanghang Tong. Model-free graph data selection under distribution shift.arXiv preprint arXiv:2505.17293, 2025a. Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Che...
-
[15]
Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451, 2024a
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451, 2024a. Lihui Liu, Zihao Wang, Ruizhong Qiu, Yikun Ban, Eunice Chan, Yan...
-
[16]
Class-imbalanced graph learning without class rebalancing
Zhining Liu, Ruizhong Qiu, Zhichen Zeng, Hyunsik Yoo, David Zhou, Zhe Xu, Yada Zhu, Kommy Weldemariam, Jingrui He, and Hanghang Tong. Class-imbalanced graph learning without class rebalancing. InProceedings of the 41st International Conference on Machine Learning, 2024c. Zhini...
2014
-
[18]
Rule based rewards for language model safety
14 Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety. arXiv preprint arXiv:2411.01111,
-
[19]
GPT-4 technical report.arXiv preprint, 2303.08774,
OpenAI. GPT-4 technical report.arXiv preprint, 2303.08774,
-
[20]
OpenAI o1 system card.arXiv preprint, 2412.16720,
OpenAI. OpenAI o1 system card.arXiv preprint, 2412.16720,
-
[21]
Reconstructing graph diffusion history from a single snapshot
Ruizhong Qiu, Dingsu Wang, Lei Ying, H Vincent Poor, Yifang Zhang, and Hanghang Tong. Reconstructing graph diffusion history from a single snapshot. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1978–1988,
1978
-
[22]
do anything now
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. InProceed- ings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1671–1685,
2024
-
[23]
Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,
-
[24]
Bypassing the safety training of open-source LLMs with priming attacks.arXiv preprint arXiv:2312.12321,
Jason Vega, Isha Chaudhary, Changming Xu, and Gagandeep Singh. Bypassing the safety training of open-source LLMs with priming attacks.arXiv preprint arXiv:2312.12321,
-
[25]
Networked time series imputation via position-aware graph enhanced variational autoen- coders
Dingsu Wang, Yuchen Yan, Ruizhong Qiu, Yada Zhu, Kaiyu Guan, Andrew Margenot, and Hanghang Tong. Networked time series imputation via position-aware graph enhanced variational autoen- coders. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Minin...
-
[26]
Fair anomaly detection for imbalanced groups.arXiv, 2409.10951,
Ziwei Wu, Lecheng Zheng, Yuancheng Yu, Ruizhong Qiu, John Birge, and Jingrui He. Fair anomaly detection for imbalanced groups.arXiv, 2409.10951,
-
[27]
Ensuring user-side fairness in dynamic recommender systems
Hyunsik Yoo, Zhichen Zeng, Jian Kang, Ruizhong Qiu, David Zhou, Zhining Liu, Fei Wang, Charlie Xu, Eunice Chan, and Hanghang Tong. Ensuring user-side fairness in dynamic recommender systems. InProceedings of the ACM on Web Conference 2024, pages 3667–3678,
2024
-
[28]
Abdelza- her, Jiawei Han, and Hanghang Tong
Zhichen Zeng, Ruizhong Qiu, Wenxuan Bao, Tianxin Wei, Xiao Lin, Yuchen Yan, Tarek F. Abdelza- her, Jiawei Han, and Hanghang Tong. Pave your own path: Graph gradual domain adaptation on fused Gromov–Wasserstein geodesics.arXiv preprint arXiv:2505.12709,
-
[29]
OpenPRM: Building open-domain process-based reward models with preference trees
16 Kaiyan Zhang, Jiayuan Zhang, Haoxin Li, Xuekai Zhu, Ermo Hua, Xingtai Lv, Ning Ding, Biqing Qi, and Bowen Zhou. OpenPRM: Building open-domain process-based reward models with preference trees. InThe Thirteenth International Conference on Learning Representations, 2025a. URL...
-
[30]
Transformer copilot: Learning from the mistake log in LLM fine-tuning.arXiv preprint arXiv:2505.16270,
Jiaru Zou, Yikun Ban, Zihao Li, Yunzhe Qi, Ruizhong Qiu, Ling Yang, and Jingrui He. Transformer copilot: Learning from the mistake log in LLM fine-tuning.arXiv preprint arXiv:2505.16270,
-
[2015]
The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751,
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751,
1904 arXiv
-
[2019]
The Llama 3 herd of models.arXiv preprint, 2407.21783,
Meta. The Llama 3 herd of models.arXiv preprint, 2407.21783,
-
[2022]
Infalign: Inference-aware language model alignment.arXiv preprint arXiv:2412.19792,
Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inference-aware language model alignment.arXiv preprint arXiv:2412.19792,
-
[2023]
Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L.J. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling.arXiv preprint arXiv:2403.13787,
-
[2024]
Theoretical guarantees on the best-of-n alignment policy
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.01879,
-
[2025]
Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint, 2204.05862,
Anthropic. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint, 2204.05862,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.