{"id":"e120fb21-c605-4503-9ff9-b56d1b0f495e","arxiv_id":"2505.13778","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CoIn verifies the count and semantic validity of invisible reasoning tokens in opaque LLM APIs using a Merkle tree over token embedding fingerprints plus learned relevance matching.","lead":"The paper proposes CoIn, a system that lets users audit hidden reasoning tokens billed by commercial AI APIs, combining a cryptographic hash tree of token embeddings with neural relevance checks. It is worth reading because it targets a real billing transparency problem, but it relies on the API provider voluntarily cooperating with the audit.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Merkle commitment is built by the audited provider and does not bind the reported count to the actual reasoning trace; the Section 3 naive inflation attack passes quantity verification, so the 94.7% DSR measures semantic detection, not count verification.","rationale":"I reviewed the protocol and experiments. The semantic-validity component is a reasonable heuristic and the empirical evaluations are detailed; I credit the authors for releasing code, measuring exposure rates, and acknowledging cooperative limitations in Section 7. The load-bearing problem is not the semantic classifier but the quantity verifier. The Merkle commitment is generated by COLA after the fact, so it cannot constrain the reported count. No independent witness (trusted execution environment, provider-signed log, or commitment computed before billing) binds the root to the actual computation. The padding rule in Algorithm 3 is symptomatic but not the root cause; even a perfect Merkle tree over m_f arbitrary leaves would pass. The abstract's 'trusted third-party auditor' framing is misleading because the auditor trusts the commitment source. The 94.7% figure, traced to Section 5.2, is the learning-based verifier's average DSR over adaptive content-injection attacks at IR=3.0, not a count-verification success rate. Because the central advertised capability—detecting token count inflation—is unsupported by the protocol, the REJECT verdict stands. If the paper were reframed as voluntary attestation or as semantic-fabrication detection only, a conditional acceptance could be considered, but as written the main claim does not hold.","tokens_in":14527,"tokens_out":5878,"duration_ms":59177,"concrete_test":"Run the formal naive attack from Section 3 end-to-end with the released code: take a real response with m reasoning tokens; have the provider report m_f = 2m and construct the Algorithm 3 tree with m_f leaves by duplicating the last leaf hash (or repeating arbitrary fingerprints), returning the new root; then run Algorithm 2 on 100 randomly sampled token indices using proofs from the inflated tree. If all 100 proofs verify, CoIn's quantity verification has zero detection power for count-only inflation, contradicting the abstract's claim that CoIn can 'effectively detect token count inflation' as a trusted auditor.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central quantity claim—that CoIn verifies the number of invisible reasoning tokens—rests on an unsound commitment. In Algorithms 2–3, COLA, the suspected adversary, constructs the Merkle tree and submits the root. A Merkle root is only a commitment to the leaf multiset the provider chooses; nothing ties that multiset to the actual hidden trace R. In particular, Algorithm 3 pads the tree by duplicating the last leaf hash (lines 24–28), so the auditor cannot even infer the true leaf count from the tree structure. More fundamentally, a provider running the Section 3 naive attack can simply build a fresh tree over m_f leaves (duplicates or arbitrary fingerprints) and return its root; every queried fingerprint will then verify against the inflated root by Algorithm 2. The proof establishes membership in the provider-chosen set, never equality with the real reasoning tokens. Section 5's 'Naive Inflation' experiment is not the formal naive attack: it injects random vocabulary tokens, which is a content forgery detectable by the semantic head. The reported 94.7% DSR is therefore semantic-validity classification accuracy on synthetic in-distribution attacks, not a detection rate for count-only inflation. Section 7's admission that CoIn requires active cooperation of COLA confirms the mechanism is a voluntary attestation, not an adversarial audit. The quantity component contributes no soundness argument against the adversary it claims to police.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CoIn proposes a two-component framework for auditing invisible reasoning tokens in commercial opaque LLM APIs (COLA). The first component builds a Merkle hash tree over token-embedding fingerprints and uses Merkle proofs to verify the reported number of hidden reasoning tokens; the second trains lightweight matching heads to score token-to-block and block-to-answer semantic relevance, aiming to detect fabricated or low-effort injected tokens. The paper formalizes naive and adaptive token-count inflation, describes the CoIn protocol with algorithms, and reports experiments on five DeepSeek-R1-derived datasets claiming a detection success rate (DSR) of up to 94.7%. The central claim is that CoIn, as a trusted third-party auditor, can verify both the quantity and semantic validity of invisible reasoning tokens without exposing proprietary content.","tokens_in":14764,"tokens_out":3108,"duration_ms":30243,"significance":"If the protocol worked as claimed, CoIn would address a real and timely problem: billing transparency for commercial reasoning APIs. The paper also makes a useful contribution by formalizing token-count inflation as a distinct security problem and by releasing a dataset and code for studying it. The reported semantic-matching evaluations are a reasonable proof-of-concept for detecting content-level anomalies in synthetic attack settings. However, the central quantity-verification claim is unsound as stated, and the experiments do not evaluate the protocol against the actual naive inflation attack defined in Section 3. The semantic component, while interesting, does not substitute for a sound count-verification mechanism. Given that the paper's headline contribution is the ability to count invisible tokens, the identified flaws are load-bearing and cannot be repaired by local revision.","major_comments":[{"comment":"The Merkle tree is constructed by COLA, the suspected adversary, over fingerprints that COLA itself computes, and the root is submitted by COLA to the auditor. A Merkle proof verifies only that a queried fingerprint is a member of the committed leaf set; it does not bind that set to the actual reasoning trace R. In particular, a provider running the Section 3 naive attack (reporting m_f > m without modifying the output) can build a fresh tree over m_f leaves—using, for example, the duplicated-last-leaf padding of Algorithm 3 lines 25-28—and return its root. Every queried token then verifies against that root, so the protocol provides no soundness against the count-only inflation it claims to prevent. The statement in §4.1 that the Merkle proof 'ensures both the structural integrity and the correctness of the reported token count' is therefore unsupported.","section":"§4.1, Algorithm 3"},{"comment":"The experimental 'Naive Inflation' is not the formal naive attack defined in Section 3. The Section 3 attack is a pure count misreport (m_f > m) with no change to the returned output. The experimental variant, however, injects random vocabulary tokens into the reasoning sequence, which is a content-forgery attack whose detection is attributable to the semantic-validity head. Consequently, the reported detection success rates, including the headline 94.7% figure, measure semantic classification on in-distribution injected content rather than the ability to detect a provider simply lying about the token count. The quantity-verification component is never tested against the attack it is designed to stop.","section":"§5.1, Table 1"},{"comment":"The evaluation is circular relative to the claimed adversarial setting. The inflated samples used in both training and testing are generated by the same Algorithm 1 procedure, with the same anchor sources, strategies, and weights that the matching heads are trained on. The threat model of Section 3, by contrast, describes COLA as an adversary that knows the protocol and can adapt. The reported DSR is therefore in-distribution classifier accuracy, not a detection rate against an adaptive adversary. A concrete test would be to train the detector only on benign data and then evaluate on attack strategies not present in the training distribution, or to have an independent party generate held-out attacks.","section":"§5.1, Algorithm 1 and §3 threat model"},{"comment":"The third stated limitation acknowledges that CoIn 'requires the active cooperation of COLA' and that COLA could deploy CoIn 'to attest to its own integrity.' This is in tension with the paper's framing of CoIn as a trusted third-party auditor that detects token-count inflation by a dishonest provider. As written, the protocol is a voluntary attestation scheme; it offers no cryptographic or computational binding between the committed tree and the actual model execution. Without such a binding mechanism—e.g., a trusted execution environment, provider-signed non-interactive proofs, or a secret embedded in the model that is infeasible to forge—the quantity-verification component provides no adversarial guarantee.","section":"§7, Limitations"}],"minor_comments":[{"comment":"The text says CoIn comprises 'two complimentary components'; 'complimentary' should be 'complementary.'","section":"§4"},{"comment":"The caption contains a typo: 'acorss' should be 'across.'","section":"Figure 9 caption"},{"comment":"The pseudocode line 'Randomly select a subset of reasoning tokens {r_i}^k_{i=1} from B_j (e.g., 10' is incomplete; it should state the fraction of tokens sampled, as in Section 4.3.","section":"Algorithm 4, line 16"},{"comment":"The conclusion that the LLM 'successfully preserves the core semantics' in the reconstruction experiment is stronger than the evidence supports: high BERTScore with low BLEU/ROUGE-L is consistent with paraphrase-level similarity, but the table reports no statistical significance or comparison against a retrieval-only baseline.","section":"Table 5 and §6"},{"comment":"The related-work section cites watermark-based and multi-agent auditing methods, but it would benefit from a brief discussion of why those techniques cannot be extended to count verification, since the paper's contribution is positioned as distinct from them.","section":"§2, References"}],"recommendation":"reject","confidential_remarks":"The reader's assessment aligns with my own reading. The core flaw is not a minor gap: the Merkle-based quantity verification does not bind the committed token set to the actual reasoning trace, and the experiments substitute a content-forgery attack for the formal count-only attack. This is a fundamental soundness problem for the paper's central claim. The authors may be able to redesign the protocol with a stronger attestation primitive, but that would be a substantially new contribution. The dataset and code release are commendable and could support future work, but they do not rescue the current claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper identifies a real problem: users are billed for invisible reasoning tokens in commercial LLM APIs, and there is no way to audit the count or the semantic quality of what they pay for. That framing is useful, and the two-component structure—a Merkle tree over token embeddings plus a learned semantic relevance head—is a sensible decomposition. The semantic validity part (token-to-block and block-to-answer matching) is a reasonable heuristic, and the experiments show it can detect synthetically injected irrelevant tokens. The authors also release code and data, which is more than most papers in this area do.\n\nThe problem is the quantity verification component. The protocol has the provider—the suspected adversary—build the Merkle tree and submit the root. A Merkle root only commits to the leaf multiset the provider chooses; nothing binds that multiset to the actual reasoning trace. The provider can simply build a tree over an inflated number of leaves (even duplicates) and return that root. Every Merkle proof will pass, but it only proves membership in the provider-chosen set, not equality with the real tokens. Algorithm 3's padding by duplicating the last leaf hash makes it even worse: the tree structure doesn't even reveal the number of leaves. The paper's own Section 7 concedes that CoIn requires the active cooperation of the provider—which turns it into a voluntary attestation scheme, not an adversarial audit. The abstract's claim of detecting token count inflation with 94.7% accuracy is therefore misleading: the experiments test semantic detection of injected random or retrieved tokens (a content-forgery attack), not the formal naive attack of count-only inflation. The classifier is trained on the same inflation algorithms it is evaluated on, so the number is largely in-distribution accuracy.\n\nThis is a load-bearing flaw, not a minor one. The central contribution as stated—verifying the quantity of invisible tokens against a malicious provider—does not hold. The semantic component is a useful heuristic and might survive as a voluntary attestation aid, but it does not rescue the count-verification claim.\n\nWho would benefit from this paper? Researchers working on LLM API transparency and billing audits, especially as a baseline for what not to do in designing commitments. It is not a working solution in its current form. I would still send it to peer review rather than desk-reject it: the problem is important, the framing is clear, and a good reviewer can push the authors to either reframe CoIn as a cooperative attestation mechanism or redesign the commitment with a trusted third party or TEE. The paper deserves a serious referee, even if the expected outcome is major revision or rejection.","headline":"The paper's central claim—that CoIn verifies the number of hidden reasoning tokens—does not survive the threat model: the Merkle tree is built by the audited provider, so the commitment is non-binding and the proof only checks membership in the provider's chosen set.","tokens_in":15361,"tokens_out":3549,"would_cite":false,"duration_ms":34999,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For proprietary LLM APIs that hide their reasoning traces, CoIn claims a third-party auditor can verify both the number of hidden tokens and whether they are semantically relevant, catching token count inflation with up to 94.7% success.","keywords":["token count inflation","opaque LLM APIs","reasoning tokens","Merkle tree","embedding fingerprints","semantic relevance verification","billing transparency","third-party auditing"],"falsifier":"Have a provider commit a cryptographic summary over a genuine reasoning trace, then bill for an inflated count by repeating the last token's summary to pad the tree to a power of two, and run CoIn's count check; if the padded summary still verifies, the quantity check cannot distinguish an honest count from an inflated one.","tokens_in":14281,"feed_emoji":"🧾","tokens_out":11397,"duration_ms":97539,"temperature":0.7,"pith_summary":"CoIn tackles a billing transparency gap in commercial LLM APIs that hide their reasoning traces: users are charged for invisible reasoning tokens, often the majority of the cost, with no way to check whether the count is genuine. The paper claims that a trusted third-party auditor can verify both the quantity and the semantic validity of these hidden tokens without seeing the tokens themselves. Quantity is verified by hashing token embedding fingerprints into a Merkle tree and spot-checking Merkle proofs; semantic validity is verified by matching heads that score whether a token is coherent with its block and whether a block supports the final answer. The reported detection success reaches 94.7% under adaptive inflation attacks, with less than 40% embedding exposure and less than 4% token visibility. If the claim holds, opaque reasoning APIs can be made auditable for billing while keeping their proprietary chains of thought concealed.","feed_headline":"Merkle tree and embedding checks catch hidden token overbilling","feed_subtitle":"CoIn lets a third party verify the count and semantic relevance of invisible reasoning tokens.","key_machinery":"The load-bearing object is the token fingerprint: for a token $r_i$ inside block $B_j$, CoIn forms $\\mathrm{Emb}_d(B_j) \\parallel \\mathrm{Emb}_d(r_i)$, hashes it as a Merkle leaf, and lets the Merkle root act as a commitment to the full hidden reasoning trace. Around this sits the matching head, a two-layer classifier that takes the feature vector $[a; b; a-b; a\\odot b; \\cos\\_\\mathrm{sim}]$ of two embeddings and outputs a relevance score in $[0,1]$. One matching head checks whether a token is coherent with its block, and another checks whether a block is relevant to the final answer; the Merkle proof supplies count integrity, and the two matching heads supply semantic relevance. A final verifier, either threshold averaging or a DeepSets classifier, aggregates the scores across sampled blocks into an accept or reject decision.","core_discovery":"On its own terms, CoIn's central claim is that hidden reasoning tokens in proprietary LLM APIs are auditable without disclosure. The provider hashes each token's fingerprint—the concatenation of a block embedding and a token embedding—into a Merkle tree and submits the root to the auditor; the auditor samples a subset of tokens, requests their fingerprints and Merkle paths, and recomputes the root to confirm that the claimed token count corresponds to leaves that exist in the committed tree. A separately trained matching head then checks each sampled token or block against its neighbors and against the final answer, so low-effort fabricated tokens inserted to inflate the bill can be flagged as semantically irrelevant. CoIn is probabilistic and requires the provider's active cooperation, and the experiments show that at high inflation rates the auditor detects most attacks while exposing only a small fraction of the reasoning trace.","pith_inferences":["Editorial inference: because the provider builds the cryptographic summary being verified, the quantity check is only as strong as the provider's cooperation; nothing in the protocol forces the summary to reflect the tokens the model actually generated.","Editorial inference: a determined attacker could pad a trace with on-topic filler—paraphrased restatements of the question or answer—that would likely pass both matching heads; the paper does not test this paraphrase-inflation case.","Editorial inference: the paper's reconstruction analysis shows semantically close reconstructions from exposed embeddings, so hiding the surface text does not hide the meaning; mining the reasoning content from leaked embeddings remains an open risk."],"forward_implications":["Users of reasoning-heavy APIs can demand a CoIn-style commitment and spot-check their bills without ever seeing the proprietary reasoning text.","Providers can keep reasoning sequences confidential while offering third-party attestation that billed tokens were real and relevant, changing the trust economics of opaque APIs.","The matching heads generalize to an out-of-domain benchmark in the paper, so the audit pipeline does not appear to require per-domain retraining.","Detection accuracy rises with inflation rate, making systematic overbilling at scale the easiest case to catch; even a 10% forgery rate still gives a 40.1% single-audit detection chance.","A learning-based verifier should be used when catching malicious samples is the priority, while a rule-based verifier yields fewer false alarms on benign samples."],"supporting_citations":[{"why":"Supplies the hash-tree commitment and Merkle proof mechanism that carries the token quantity verification.","marker":"Merkle [1987]"},{"why":"Provides the R1-style reasoning traces used to build the training and evaluation datasets for the matching heads.","marker":"DeepSeek-AI [2025]"},{"why":"Supplies the Sentence-BERT embedding model used to compute token fingerprints and matching-head features.","marker":"Reimers and Gurevych [2019]"},{"why":"Documents the o1 system card, a concrete instance of an opaque reasoning API that bills for hidden reasoning tokens.","marker":"Jaech et al. [2024]"},{"why":"Earlier API-auditing work CoIn contrasts with, focused on detecting model substitution rather than token count inflation.","marker":"Cai et al. [2025]"},{"why":"Supplies the DeepSets architecture used by the learning-based verifier that aggregates per-round match scores.","marker":"Zaheer et al. [2017]"}],"fun_headline_variants":["Auditing hidden reasoning tokens with Merkle trees and embeddings","CoIn exposes token inflation in opaque LLM APIs","Verifying invisible reasoning tokens with hash trees","Third-party auditor catches LLM token overbilling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire token-count check rests on the assumption that the provider's cryptographic summary of the hidden tokens is an honest reflection of what the model really generated, even though the provider is the party suspected of inflating that number.","fun_headline_variants_meta":{"raw":{"variants":["Auditing hidden reasoning tokens with Merkle trees and embeddings","CoIn exposes token inflation in opaque LLM APIs","Verifying invisible reasoning tokens with hash trees","Third-party auditor catches LLM token overbilling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3294,"prompt_tokens":953,"completion_tokens":2341,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2279}},"tokens_in":569,"tokens_out":2341,"duration_ms":14676,"temperature":1.0,"reasoning_tokens":2279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:10:58.890493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a provider commit a cryptographic summary over a genuine reasoning trace, then bill for an inflated count by repeating the last token's summary to pad the tree to a power of two, and run CoIn's count check; if the padded summary still verifies, the quantity check cannot distinguish an honest count from an inflated one.","supporting_citations":[{"cited_title":"A digital signature based on a conventional encryption function","cited_arxiv_id":null,"evidence_quote":"Supplies the hash-tree commitment and Merkle proof mechanism that carries the token quantity verification."}],"review_version":1}