{"id":"496baef6-33d4-4a78-a4a8-5f91a90e6778","arxiv_id":"2505.04654","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"The Relative Danger Coefficient is a weighted refusal-based metric for LLM safety, introduced and demonstrated on GPT, Gemini, and DeepSeek models.","lead":"This paper proposes a new score, the Relative Danger Coefficient, for comparing how often and how badly AI chatbots produce unsafe or harmful answers. The authors apply it to GPT, Gemini, and DeepSeek models across hundreds of prompts about explosives, hate speech, drugs, and moral dilemmas.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RDC rankings are not robust to the hand-set weights: Appendix B.2's claim that any ordered weights yield a meaningful comparison is false, so the central comparison is unestablished.","rationale":"The reader's REJECT verdict is directionally correct, but the strongest reason is stronger than 'weights may not reflect true harm'. The paper's own Appendix B.2 makes a mathematical invariance claim that is false: model rankings under Equation 1 can change when the weights are replaced by another tuple satisfying the paper's stated ordering. This is an internal inconsistency in the central argument, not merely a missing external validation. The single-annotator issue is real but secondary; it affects reliability of the G/U/P/D counts, while the weight-sensitivity issue affects the metric itself even if the counts are perfect. A computational sensitivity sweep would settle the concern: if all permissible parameterizations preserve the headline ranking, the concern fails and the metric's relative claims survive; if the ranking flips, the central comparison is arbitrary. I therefore agree with rejecting the paper as written, but for a more specific and more decisive reason than the reader gave. No ad hominem is intended; the issue is with the construction of RDC, not with the authors' conduct.","tokens_in":13525,"tokens_out":5508,"duration_ms":57415,"concrete_test":"Recompute RDC for the reported per-model response counts while sweeping all weight triples (Wu,Wp,Wd) on a grid satisfying 0<Wu<Wp<Wd with Wd fixed at 5.0 (e.g., Wu in {0.5,0.8,1.0,1.2,1.5,1.8}, Wp in {1.5,2.0,2.7,3.0,3.5,4.0}) and all penalty combinations within the appendix's stated integer scales (C=0/7/15, S=0/5/10, R=0/3/7/12, A=0/10/15/20). Record how often the paper's headline ordering—Gemini above GPT above DeepSeek—is preserved. If the ordering changes in more than a small fraction of permissible parameter sets, the metric does not provide a stable relative ranking.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that RDC meaningfully ranks LLM safety. The load-bearing weakness is internal: Equation 1's normalized term uses fixed exchange rates among U, P, and D (Wu=1.2, Wp=2.7, Wd=5.0), and the integer penalties C+S+R+A are added after the ceiling with no calibration. Appendix B.2 asserts that any weights preserving Wg≤0<Wu<Wp<Wd give a meaningful comparison and that the exact numeric values are not important. This is false for rankings. Example with N=20 and no penalties: Model A has 5 U and 15 G, giving RDC=6; Model B has 2 P and 18 G, giving RDC=5.4, so A ranks worse. With Wu=1.0, Wp=3.0, Wd=5.0, A gives 5 and B gives 6, so B ranks worse. Both weight sets satisfy the paper's stated ordering. The penalty terms can also flip rankings: they are unscaled absolute integers (C up to 15, S up to 10, R up to 12, A up to 20), so a model with a higher unsafe-response base score can be overtaken by a model with a lower base score but higher penalties. The paper gives no pilot data, no external validation against an established safety benchmark, and no inter-annotator agreement, so nothing ties the chosen parameterization to true harm rather than to the authors' prior expectations. If the ranking reverses under permissible reparameterizations, the paper's headline conclusions (Gemini best, DeepSeek highest risk) do not follow from the RDC as defined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a composite metric, the Relative Danger Coefficient (RDC), to quantify the ethical and safety risk of LLM responses. The authors define four response categories (Good, Uncertain, Partially Unsafe, Directly Unsafe), assign hand-chosen weights and four integer penalty terms, and apply the metric to GPT, Gemini, and DeepSeek models across manual and automated adversarial prompts. The headline conclusions are that Gemini performs best overall, DeepSeek shows the highest ethical risk, and no model is robust to adversarial reframing. The paper's main claimed contribution is the RDC metric and the resulting comparative ranking.","tokens_in":13842,"tokens_out":6832,"duration_ms":70175,"significance":"Comparative safety evaluation of LLMs is timely and important, and the paper contains a useful collection of adversarial prompt categories, including role-play, educational framing, and logical-puzzle variants. Some qualitative observations, such as DeepSeek endorsing stereotypes in a logic puzzle and Gemini contradicting itself in a trolley-type dilemma, are concrete and could inform future red-teaming work. However, the central claim that RDC provides a meaningful quantitative comparison is not supported. The weights and penalty values are uncalibrated, the response classifier is a single human annotator plus an undescribed automated checker, and no statistical analysis accompanies the comparisons. If the metric were calibrated and validated, it could become a helpful relative indicator, but in its current form the numerical rankings are not trustworthy.","major_comments":[{"comment":"The claim in Appendix B.2 that any weights satisfying Wg ≤ 0 < Wu < Wp < Wd yield a meaningful comparison is false, because rankings can reverse under permissible reparameterizations. For example, with N=7, no penalties, and weights (0, 1.2, 2.7, 5.0), a model with 5 Uncertain and 2 Good responses receives RDC=18, while a model with 2 Partially Unsafe and 5 Good responses receives RDC=16, so the first is worse. With the equally permissible weights (0, 1.0, 3.0, 5.0), the same two models receive RDC=15 and RDC=18, respectively, so the ranking flips. Since the paper's headline conclusions depend on the specific uncalibrated numeric weights, the comparative results do not follow from the RDC as defined.","section":"§3.1, Eq. (1); Appendix B.2"},{"comment":"The penalty terms C, S, R, and A are added after the normalized percentage term with no scaling or calibration, and their score assignments are presented as arbitrary integers (e.g., C=7 for minor inconsistency, S=10 for multiple severe outputs, A=20 for reliable adversarial exploits). Because these penalties can be comparable to or larger than the base weighted score, a model with fewer unsafe responses but higher subjective penalty ratings can be overtaken by a model with more unsafe responses and lower penalties. The statement in Appendix B.6 that a higher RDC indicates a greater proportion of unsafe output is therefore not accurate: the penalty terms are not proportions, and no pilot data or external validation is provided to tie the chosen integers to actual harm.","section":"Appendix B.3, Eq. (1)"},{"comment":"The RDC depends entirely on the classification of responses into G, U, P, and D, but the paper reports no inter-annotator agreement, no description of the automated checker, and indeed Section 6 explicitly acknowledges the use of a single annotator. Without a validated and reproducible annotation procedure, even a correctly specified formula cannot produce reliable comparisons, because classification noise directly changes the U, P, and D counts that feed the metric. The absence of any reliability analysis is a load-bearing gap for all quantitative conclusions.","section":"§3.1, §6; Appendix B.1"},{"comment":"The results section does not provide the data needed to support the comparative claims. Section 4.3 illustrates RDC values only for Gemini models, and the conclusion's statement that 'Gemini performed best overall' and 'DeepSeek' had the highest ethical risk is not accompanied by a full table of RDC values, per-model category counts, confidence intervals, or significance tests. The manual results in §4.1–4.2 are qualitative anecdotes, and the automatic results are described selectively. As a result, the paper's central comparative claims are not verifiable from the reported evidence, even if the metric itself were sound.","section":"§4.3, §5"}],"minor_comments":[{"comment":"The phrase 'brand-new DeepSeek-V3 (R1 with reasoning and without)' is informal and should be replaced with a precise model designation and version reference.","section":"Abstract; §1"},{"comment":"The reference 'Priyanka and , 2024' is missing the second author's name and should be corrected.","section":"§2"},{"comment":"The Limitations section contains a grammatical error: 'use of a our novel and Danger Coefficient (RDC) metric' should be 'use of our novel Danger Coefficient (RDC) metric.'","section":"§6"},{"comment":"The sentence 'It is possible to decrease RDC if a safe response is received by setting Wg = 0 to a negative value' is unclear; it should read 'by setting Wg to a negative value.'","section":"Appendix B.2"},{"comment":"Both figures lack detailed captions and axis labels, and Figure 2 is not accompanied by a comparison table for all models, making the visual results difficult to interpret.","section":"Figures 1 and 2"}],"recommendation":"reject","confidential_remarks":"The paper would benefit from calibration of the RDC against an established safety benchmark and from multi-annotator coding with reported agreement statistics; without these, the metric remains an unvalidated scoring rule. The manuscript also reads in places like a technical report rather than a full research paper, and the absence of a data or code availability statement further limits reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the RDC is the only genuinely new thing in this paper, and it doesn't withstand a robustness check. The authors claim (Appendix B.2) that any weights preserving Wg≤0<Wu<Wp<Wd give a meaningful comparison and exact values don't matter. That's false. With N=20 and no penalties, a model with 5U/15G scores 6 under (1.2, 2.7, 5.0) while one with 2P/18G scores 5.4 (or 6 after ceiling), but under (1,3,5) the first scores 5 and the second 6. Both weight sets satisfy their ordering, and the ranking flips. The integer penalties C,S,R,A are added after the ceiling and are large enough to dominate the base score, so they can flip rankings too. Since the weights and penalties were chosen 'based on pilot testing' with no pilot data, the paper's headline conclusions (Gemini best, DeepSeek worst) are not supported by the metric as specified.\n\nThat said, the paper isn't worthless. The prompt design is thoughtful—reframing bombs as educational, law-enforcement, historical anecdotes, and the logical-puzzle discrimination prompt is genuinely clever. The qualitative observations about model behavior, especially reasoning models becoming more vulnerable under sustained prompting, are plausible and line up with other red-teaming work. The limitations section is unusually candid about the single annotator, proprietary model churn, and the metric's reliance on subjective classification.\n\nThe soft spots are exactly where the reader put them. One annotator, no inter-annotator agreement, an automated checker that is never described, no release of the response counts per model/category, and no validation against an established safety benchmark. Without those, RDC is a formula, not a measurement. The manual qualitative findings could have been reported as observations and would have been fine; the problem is wrapping them in a quantitative metric whose claims outrun its evidence.\n\nMy verdict: this deserves a serious referee but not acceptance. A good reviewer would ask for sensitivity analysis over weights and penalties, inter-annotator reliability, a full data release, and either calibration to an existing benchmark or removal of the penalty terms. If the authors can do that, there might be a usable safety-evaluation contribution here. As is, it's a reasonable write-up of exploratory red-teaming with a questionable scoring layer on top.","headline":"RDC metric is not robust to its hand-set weights; the paper's central ranking claim fails, though the qualitative red-teaming is thoughtful.","tokens_in":14367,"tokens_out":3177,"would_cite":false,"duration_ms":30094,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces the Relative Danger Coefficient, a weighted 0–100 score for LLM harm, and reports that it ranks Gemini safest overall and DeepSeek at the highest ethical risk across hundreds of adversarial prompts.","keywords":["large language models","AI safety","Relative Danger Coefficient","ethical evaluation","red teaming","jailbreak resistance","model benchmarking","harm metric"],"falsifier":"Re-run the same 170 prompts with several independent annotators who label responses G/U/P/D, measure inter-annotator agreement, and recompute RDC under weight sets that preserve the ordering 0 < Wu < Wp < Wd (e.g., 0.1/0.5/1.0, 1/3/6). If the relative ranking of Gemini, GPT, and DeepSeek changes under these permissible weights or across annotators, the claim that RDC is a meaningful relative risk metric is not supported.","tokens_in":13301,"feed_emoji":"⚠️","tokens_out":5348,"duration_ms":52094,"temperature":0.7,"pith_summary":"This paper claims that LLM safety can be compared quantitatively with a single number, the Relative Danger Coefficient (RDC). It tests ten GPT, Gemini, and DeepSeek variants on hundreds of prompts, sorts every answer into four risk classes, and adds penalties for inconsistency, severity, repetition, and jailbreak success. On the paper's own results, Gemini ranks safest overall, DeepSeek shows the highest ethical risk, and no model resists adversarial reframing. A sympathetic reader would take the contribution as a structured way to turn messy refusal-versus-compliance behavior into a comparable risk score.","feed_headline":"New danger score ranks Gemini safest, DeepSeek most at risk","feed_subtitle":"A weighted 0-to-100 formula converts chatbot replies into comparable risk ratings across hundreds of adversarial prompts.","key_machinery":"The central object is the Relative Danger Coefficient, defined as $$\\text{RDC} = \\min\\!\\left(100,\\, \\max\\!\\left(0,\\, \\left\\lceil \\frac{W_gG + W_uU + W_pP + W_dD}{\\max(W_g,W_u,W_p,W_d)\\cdot N}\\cdot 100 \\right\\rceil + C + S + R + A\\right)\\right).$$ Each response is classified as Good, Uncertain, Partially Unsafe, or Directly Unsafe; the weights 0, 1.2, 2.7, and 5.0 encode increasing severity, and the penalties (Consistency, Severity, Repetition Vulnerability, Adversarial Exploitability) add integer points for behavioral vulnerabilities. The formula turns both response frequency and exploitability into a bounded 0–100 score, and the paper's comparability claim rests on applying the same weights and category definitions to every model.","core_discovery":"The paper's central claim is that a weighted formula over labeled responses—good, uncertain, partially unsafe, directly unsafe—plus four behavior penalties, yields a meaningful relative risk ranking of LLMs. It reports that Gemini models have the lowest RDC overall, DeepSeek models the highest, especially in moral dilemmas and discrimination-style logic puzzles, and GPT models sit in between with partial disclosures under persistent prompting. The paper also claims that newer model versions generally improve over older ones, and that explicit reasoning can both sharpen ethical disclaimers and open new disclosure routes.","pith_inferences":["A natural extension is to test whether RDC rankings generalize to held-out prompt categories; if rankings shift sharply on new prompts, the metric measures the prompt set more than the models.","The severity weights and penalty levels could be re-estimated from human harm ratings; if model rankings flip under different weight sets that preserve the ordering 0 < Wu < Wp < Wd, the 'relative' claim weakens.","RDC currently pools all unsafe content within a category; coupling category-level RDC with application-specific cost weights could produce deployment-specific risk scores.","Because the response labels rest on a single annotator, measuring inter-annotator agreement would clarify how much of the reported ranking is annotation noise rather than model behavior."],"forward_implications":["If RDC is accepted, safety claims about new LLMs can be stated as one number and tracked across model versions over time.","The finding that prompt reframing changes refusal outcomes implies that refusal rates alone understate risk; consistency and jailbreak resistance must be measured separately.","Deploying LLMs in high-stakes domains such as healthcare, security, and justice should include human oversight in high-RDC categories like weapons and substances.","Reasoning-enabled models may need extra auditing, since richer reasoning can produce more nuanced disclaimers and also more detailed partial instructions.","The paper's version-to-version comparisons suggest safety is not guaranteed to improve monotonically, so continuous benchmarking is needed."],"supporting_citations":[{"why":"Supplies the red-teaming benchmark whose prompt design and response categories inspire the automatic test protocol.","marker":"Tedeschi et al. (2024)"},{"why":"Provides the severity-weighted accounting principle used to justify weighting bad outputs increasingly by danger level.","marker":"Førsund (2009)"},{"why":"Supports the claim that LLM vulnerabilities emerge under repeated or adversarial prompts, motivating the C, S, R, and A penalties.","marker":"McIntosh et al. (2024)"},{"why":"Supplies the classic double-effect and trolley-problem framing used in the ethical dilemma test category.","marker":"Foot (1967)"},{"why":"Provides the killing-versus-letting-die formulation used in the moral dilemma prompts.","marker":"Thomson (1976)"},{"why":"Supplies the canonical trolley problem scenarios that anchor the sacrifice-style tests.","marker":"Thomson (1985)"},{"why":"Inspires the logic-puzzle embedding used to test whether models propagate discriminatory stereotypes inside a puzzle frame.","marker":"Smullyan (1978)"}],"fun_headline_variants":["RDC ranks Gemini safest, DeepSeek riskiest among AI models","Relative Danger Coefficient: new metric for LLM safety gaps","Gemini leads safety, DeepSeek lags in new LLM risk score","Study rates LLM ethics: Gemini top, DeepSeek bottom","New danger metric spots biggest ethical gaps in chatbots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings stand or fall on the hand-chosen severity weights and penalty values (based on pilot testing) together with the reliability of the single-annotator response classification; if those numbers or labels are off, the RDC ordering of models may not reflect real danger.","fun_headline_variants_meta":{"raw":{"variants":["RDC ranks Gemini safest, DeepSeek riskiest among AI models","Relative Danger Coefficient: new metric for LLM safety gaps","Gemini leads safety, DeepSeek lags in new LLM risk score","Study rates LLM ethics: Gemini top, DeepSeek bottom","New danger metric spots biggest ethical gaps in chatbots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1144,"prompt_tokens":789,"completion_tokens":355,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":266}},"tokens_in":405,"tokens_out":355,"duration_ms":3457,"temperature":1.0,"reasoning_tokens":266,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:40:05.592179+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 170 prompts with several independent annotators who label responses G/U/P/D, measure inter-annotator agreement, and recompute RDC under weight sets that preserve the ordering 0 < Wu < Wp < Wd (e.g., 0.1/0.5/1.0, 1/3/6). If the relative ranking of Gemini, GPT, and DeepSeek changes under these permissible weights or across annotators, the claim that RDC is a meaningful relative risk metric is not supported.","supporting_citations":[{"cited_title":"but the one is a child","cited_arxiv_id":null,"evidence_quote":"Supplies the canonical trolley problem scenarios that anchor the sacrifice-style tests."}],"review_version":1}