{"id":"e653f62f-05a2-4205-bb61-057b9832216a","arxiv_id":"2412.11506","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Glimpse estimates full token distributions from top-K API probabilities, enabling white-box detectors like Fast-DetectGPT to reach about 0.95 AUROC on GPT-4, Claude-3, and Gemini-1.5 text.","lead":"This paper introduces Glimpse, a way to rebuild a language model's full token probabilities from just the top few probabilities an API returns, letting detection methods built for open models work on GPT-4 and other proprietary models. If it holds up, the strongest models can serve as accurate, low-cost detectors of their own generated text.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Estimated tail distributions are never validated on proprietary models, and the paper's own data show worse estimates yielding better detection, so the core 'full distribution' mechanism is unverified.","rationale":"The reader's conditional verdict centers on the same region: the shape of the probability tail is assumed recoverable from top-K values, and the paper shows an unexplained inverse relationship between estimation fidelity and detection accuracy. I agree that this is the soft spot. My stress-test sharpens it: the paper never validates the estimated distribution on the proprietary models where Glimpse is actually deployed, and its own ablation shows that a naive truncation retains most of the performance. That makes the central 'full distribution prediction' claim empirically unverified and potentially non-essential to the reported gains. I do not see a logical contradiction or fraud; the experiments are broad, the geometric/Zipfian variants are parameter-free, and the code is released. But the mechanism-of-action claim is not yet supported. Since the reader already assigned CONDITIONAL and my concern reinforces rather than redirects that judgment, the verdict should remain CONDITIONAL. The proposed test would settle whether the distribution-estimation component is load-bearing: if a no-tail baseline matches Glimpse, the paper should be reframed as a top-K white-box detection method; if Glimpse clearly beats the no-tail baseline, the current framing is justified.","tokens_in":25658,"tokens_out":5905,"duration_ms":61185,"concrete_test":"Take an open-source scoring model with full distribution access (e.g., Llama-3-8B or Qwen2.5-7B), build datasets as in the paper, and simulate the API by exposing only top-K logprobs per position. Compute three Fast-DetectGPT variants: (A) oracle using the true full distribution; (B) Glimpse-Geometric/Zipfian/MLP from top-K logprobs; (C) a no-tail baseline that renormalizes only the observed top-K probabilities to sum to 1, without any tail estimation. Compare AUROC across the five source models and Mix3/Mix6. If (C) performs as well as (B), the tail-estimation component is unnecessary; if (B) matches (A), the estimation is faithful enough. The same protocol should also be run on a proprietary model by comparing Glimpse against the same no-tail baseline via the API, to confirm the effect is not an artifact of open-source distribution shapes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Glimpse predicts full distributions from partial observations and that this is what lets white-box detectors use proprietary models. The load-bearing condition is that the estimated tail over ranks beyond top-K is faithful enough to the true distribution to make the curvature score meaningful. That condition is never checked on any proprietary model: Figure 2 and Figure 3 measure KL divergence only against Neo-2.7B, where full distributions are available; for GPT-3.5/GPT-4, the object being estimated is unobservable, so the method's success cannot be attributed to accurate distribution estimation. More seriously, the paper's own evidence cuts against the mechanism: Geometric has the largest KL divergence (Figure 2) yet the best detection AUROC (Figure 3, Table 1), and the Naive truncated distribution already achieves 0.9311 AUROC versus 0.9630 for Geometric (Section 3.4). If a knowingly wrong tail shape improves detection and a crude truncation retains most of the gain, then the reported results may be driven by the proprietary model's top-K probabilities and true token log-likelihoods rather than by the 'full distribution' reconstruction. The contribution as stated is therefore either unverified or not load-bearing for the reported gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Glimpse, a method that estimates a full token-level predictive distribution from the top-K probabilities returned by proprietary LLM APIs, using one of three tail-completion schemes (Geometric, Zipfian, or a trained MLP). Glimpse is then used to compute white-box zero-shot detection metrics (Entropy, Rank, LogRank, and Fast-DetectGPT) with proprietary scoring models such as GPT-3.5 and GPT-4, which normally expose only partial API information. The authors report that Fast-DetectGPT with Glimpse and GPT-3.5 reaches an average AUROC of about 0.95 across five source models, a 51% relative improvement over the open-source Fast-DetectGPT baseline, along with efficiency and robustness benefits. The paper includes extensive experiments across datasets, languages, source models, scoring models, and ablations, and it releases code and data.","tokens_in":25911,"tokens_out":7832,"duration_ms":70345,"significance":"If the contribution is taken at face value, Glimpse would be a practical and inexpensive way to bring the power of proprietary LLMs to white-box zero-shot detectors, and the experimental evidence for detection quality is strong and unusually broad: five source models, multiple datasets and languages, ablations over K, rank-list size, and prompts, plus robustness analyses under paraphrasing. The derivations in Section 2.3 are transparent and the authors ship code and data, which strengthens reproducibility. The reported improvements in AUROC over open-source surrogates are practically meaningful, and the observation that larger models can be universal detectors is interesting. However, the central interpretive claim — that Glimpse works because it accurately predicts the full distribution from partial observations — is not supported by the paper's own evidence, and this weakens the conceptual contribution even though the empirical results may stand.","major_comments":[{"comment":"The paper's own evidence is inconsistent with the claim that Glimpse improves detection by accurately estimating the full distribution. Figure 2 shows that the Geometric estimator has the largest KL divergence from the true Neo-2.7 distribution, yet Figure 3 and Table 1 show that it achieves the best detection AUROC among the three estimators. Moreover, the Naive truncated distribution in §3.4 reaches 0.9311 AUROC versus 0.9630 for Geometric, so the tail reconstruction contributes only about 0.03 AUROC. Even more concerning, Entropy (GPT-3.5) with Glimpse yields AUROC values as low as 0.2160 on ChatGPT Mix3 (Table 1), far below 0.5, which would be unlikely if the estimated distribution ßp_θ approximated the true p_θ. These observations suggest that the detection gains are driven mainly by the true token log-likelihood log p(x) and the top-K probabilities, not by the fidelity of the tail reconstruction. Please (i) validate the estimated distributions against full distributions on multiple open-source models (e.g., Llama-3-8B, Qwen2.5-7B, Phi-2) in terms of KL divergence and Brier score, and (ii) either provide a principled robustness argument showing why the curvature metric is insensitive to tail error, or reframe the contribution as a heuristic for extending white-box metrics to API logprobs.","section":"§3.2, §3.4, Table 1"},{"comment":"The MLP estimator is trained on probability distributions from a single open-source model, GPT-Neo-2.7B, and then applied to proprietary models without any analysis of domain shift. The only fidelity evaluation (Figure 2) uses the same model on which the MLP was trained, which does not test generalization to different model families or sizes. Since the intended use case is proprietary models with unknown tail shapes, the paper needs to show that the estimation approach transfers. Please train or evaluate the estimators on held-out open-source models from different families and report both distribution-level fidelity and the resulting detection accuracy, or clearly state that the estimator is tailored to GPT-like models and evaluate the sensitivity of the results to this choice.","section":"Appendix A.3, §3.2"},{"comment":"The claim that the full-distribution estimation is what 'enables' white-box methods is only modestly supported by the Naive ablation: the gain from Geometric over Naive is 0.032 AUROC, compared to a 0.048 AUROC gain over the open-source Fast-DetectGPT baseline. To substantiate the central narrative, the paper should decompose the contribution of each component: (a) the proprietary model's true top-K probabilities and log p(x), (b) the choice of tail estimator (including a uniform or random tail control), and (c) the scoring model itself. This would clarify whether the method's success is attributable to the distribution estimation or to the use of a stronger scoring model's partial information.","section":"§3.4, Table 1"}],"minor_comments":[{"comment":"For Rank and LogRank, the paper states that the rank of the current token is found by searching the closest p(k) to the token's probability. This is an unusual proxy for rank, and the text does not explain why it is a reasonable approximation or how errors in this rank estimate affect the metric. Please add a brief justification or an ablation.","section":"§2.4"},{"comment":"The description of the Completion API call says the model 'echoes' the input; please specify the exact API parameters (e.g., logprobs count, echo: true, max_tokens) and the version, as this is essential for reproducibility of the top-K probabilities.","section":"Appendix B.2"},{"comment":"Equation (16) defines an iterative adjustment for λ; please state the convergence criterion and typical number of iterations needed, since the appendix currently does not specify when the iteration terminates.","section":"Appendix A.1"},{"comment":"The 'Avg.' column is not explicitly defined in the caption or text. Based on the values, it appears to be the mean of the five source-model Mix3 AUROCs, but this should be stated for clarity.","section":"Table 1"},{"comment":"The low AUROC for Russian is attributed to 'potential under-training of the LLMs on this particular language.' This is a speculation; please provide supporting evidence or soften the wording.","section":"§3.5, Table 2"},{"comment":"The sentence 'we are the first to enable white-box methods in proprietary LLMs' should be reconciled with the discussion of DLAD (Zeng et al., 2024a), which also uses logits from black-box LLMs. Please clarify the novelty boundary.","section":"§5, Related Work"}],"recommendation":"major_revision","confidential_remarks":"The empirical study is thorough and the results are likely to be of interest, but the paper's framing overstates the role of the full-distribution estimation. The table of Entropy (GPT-3.5) with Glimpse showing AUROC below 0.5 is a red flag that the estimated distribution is systematically wrong, and the paper should address this head-on. I would encourage the editor to invite a revision that either substantially validates the estimation mechanism or reframes the contribution more honestly as a heuristic for using API logprobs with existing white-box scores. The code release and broad experiments are genuine strengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Glimpse is a practical trick with broad experiments: take the top-K logprobs a proprietary API already returns, fit a parametric tail (geometric, Zipfian, MLP), and use that completed distribution to compute white-box metrics like Fast-DetectGPT. It works—AUROC around 0.95 across five source models, with clear ablations and released code. That is genuinely useful, and the paper is honest about its own odd finding: the geometric estimator has the highest KL divergence against the true distribution yet the best detection accuracy, while a naive truncation that sets the tail to zero already gets 0.93 versus 0.96. That honesty cuts both ways. It means the method's success is not actually evidence for the 'full distribution estimation' story the abstract sells. The tail shape is not load-bearing; the top-K probabilities from the proprietary model are doing most of the work. And because the KL validation is only done against an open-source model where full distributions are visible, the paper never checks whether the estimated tails are faithful on GPT-4 or Claude—it can't, since those distributions are unobservable.\n\nThe firstness claim is also shaky. The paper cites DLAD (Zeng et al., 2024a), which reconstructs logits from top-K API outputs, but does not compare or discuss it. Given that, 'we are the first to enable white-box methods in proprietary LLMs' is too strong unless DLAD's setting is materially different, which is not argued.\n\nWhat's solid: the method is simple, clearly explained, and the experiments are extensive across models, languages, and domains. The code and data are released. The finding that larger models can be universal detectors with this technique is a useful counterpoint to earlier claims that smaller models are better. The ablation on prompt sensitivity for GPT-4 is a nice caution for practitioners.\n\nSoft spots beyond the mechanism: no error bars or significance tests—three runs, report median, no variance. The Zipfian and MLP hyperparameters are somewhat hand-wavy. These are minor relative to the main issue.\n\nWho is this for? Anyone working on zero-shot LLM-text detection who wants a working API-level detector. I'd cite it. If I were refereeing it fresh, I would send it to review: the empirical results are reproducible and the question of whether the mechanism is what the paper says is a good referee discussion. But the paper needs either a validation study on a model where ground-truth distributions are available (even an open model behaving like a proprietary one) or a repositioning of Glimpse as a top-K heuristic rather than full-distribution estimation. Also, the DLAD comparison is mandatory.","headline":"A practical and honest paper whose 'full distribution estimation' mechanism is not load-bearing; the top-K logprobs do the work, and the firstness claim is disputed by DLAD.","tokens_in":26432,"tokens_out":3373,"would_cite":true,"duration_ms":30702,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"White-box detection metrics can be run on proprietary LLMs by reconstructing each token's full probability distribution from the top-K probabilities an API returns.","keywords":["LLM-generated text detection","zero-shot detection","probability distribution estimation","white-box detection","Fast-DetectGPT","proprietary LLMs","top-K logprobs","conditional probability curvature"],"falsifier":"Run Fast-DetectGPT with an open-source scoring model whose full distribution is available, on a held-out corpus, first with the true per-position distributions and then with Glimpse's top-K reconstruction; then artificially perturb a band of mid-rank probabilities, adding a second hump of probability mass at ranks 50 to 300, and repeat. If the AUROC gap between true- and estimated-distribution scores jumps when the tail ceases to be monotone-decaying, the parametric tail-shape assumption is the load-bearing part; if not, the metric is robust to tail errors and the central claim holds more broadly.","tokens_in":2192,"feed_emoji":"🛡️","tokens_out":2275,"duration_ms":94794,"temperature":0.7,"pith_summary":"This paper claims that white-box text detectors, which normally need full access to a model's per-token probability distribution, can instead be run on proprietary models using only the top few probabilities the API returns. The method, Glimpse, fills in the missing tail of each token's distribution with a simple parametric estimate, then feeds the completed distribution to existing metrics like Fast-DetectGPT, Entropy, Rank, and Log-Rank. On five of the most recent language models, Fast-DetectGPT with GPT-3.5 as the scoring model reaches an average AUROC around 0.95, a 51 percent relative gain over the open-source surrogate baseline. If correct, this means the strongest, closed models can serve as accurate, cheap detectors of machine-written text, a capability previously limited to weaker open models or expensive sampling-based black-box methods.","feed_headline":"AI-text detection hits ~95% AUROC from top-5 API logprobs","feed_subtitle":"Estimating the hidden tail of token probabilities lets proprietary LLMs serve as their own detectors.","key_machinery":"The load-bearing object is the completed rank-ordered probability distribution $\\tilde{p}_\\theta$ per token position, built under a total-probability constraint ($\\sum_k p(k) = 1$) and a monotone-decay constraint. For Fast-DetectGPT the metric is the conditional probability curvature $d(x, p_\\theta) = (\\log p_\\theta(x) - \\tilde{\\mu})/\\tilde{\\sigma}$, where $\\tilde{\\mu}$ and $\\tilde{\\sigma}^2$ are the mean and variance of $\\log \\tilde{p}_\\theta$ under the estimated distribution, computed analytically as $\\tilde{\\mu}_j = \\sum_k p(k)\\log p(k)$ and $\\tilde{\\sigma}^2_j = \\sum_k p(k)\\log^2 p(k) - \\tilde{\\mu}_j^2$. Because these sums run over ranks rather than token identities, the missing vocabulary tail can be modeled by one of three estimators: geometric decay with a solved factor $\\lambda$ (Eq. 7), a Zipfian power law with a regularized loss-table search over $(\\alpha, \\beta)$ (Eq. 8), or an MLP trained on open-source distributions (Eq. 9). The same completed distribution extends Entropy, Rank, and LogRank to proprietary models.","core_discovery":"Glimpse establishes that a detection score can be computed from a deliberately approximate distribution. API completion endpoints reveal only the top-K token probabilities and the likelihood of the observed text at each position; Glimpse observes that the Fast-DetectGPT curvature metric, along with Entropy, Rank, and Log-Rank, depends only on probability values, not on which tokens carry them, so it models the rank-ordered distribution directly. Three estimators are proposed: a near-Geometric decay, a Zipfian power-law with regularized parameters, and a single-hidden-layer MLP trained on an open-source model's full distributions. The completed distribution is used only for the expected score and variance terms, while the observed text's own likelihood comes from the API exactly. Across ChatGPT, GPT-4, Claude-3 Sonnet, Claude-3 Opus, and Gemini-1.5 Pro, the approach reports AUROC values from 0.92 to 0.98, and the paper reads this as evidence that the latest LLMs can effectively detect their own outputs. The approach is limited to metrics built on predictive distributions rather than inner embeddings, and to models that expose a completion API returning top-K logprobs.","pith_inferences":["The paper's own numbers suggest that fidelity of tail estimation is not what drives accuracy: the geometric estimator has the largest KL divergence yet the best detection scores; a plausible reading, not tested in the paper, is that the curvature metric's power lives in the exact observed-token likelihood plus the top-K values, while the estimated mean and variance only need to be in the right bal","The rank-scaling trick, computing metrics from probabilities without token identities, is a general recipe that could extend to estimating sequence-level quantities like perplexity or predictive entropy for hallucination scoring on proprietary models, which the paper mentions only as a broader-impact remark.","Since different estimators win on different datasets, geometric on Writing and PubMed, Zipfian on XSum, MLP most balanced, a per-domain estimator selector or a small ensemble trained on a sentinel set could push the average AUROC above any single estimator without extra API calls.","Because the MLP is trained once on an open-source model and transfers to GPT-3.5 and GPT-4, a natural stress test is whether the transfer holds for future model families with sharper or flatter distributions than today's; the paper's decay-pattern analysis suggests top-1 probability and decay factor are the two quantities to monitor."],"forward_implications":["White-box zero-shot detection can be applied to the strongest proprietary models at the cost of a single API echo pass per text, about 4.1 times faster and roughly 10 times cheaper than DNA-GPT in the paper's settings.","Larger proprietary models outperform smaller ones as universal detectors across five source models, directly challenging the earlier finding that smaller models are better universal detectors.","The completed distribution supports several metrics beyond curvature, Entropy, Rank, and LogRank, so a family of white-box detectors inherits API-only access.","The approach transfers across six languages and four datasets, with near-perfect scores on Urdu, Indonesian, and Arabic, and it holds up better than open-source baselines under DIPPER paraphrasing.","The specific claim that advanced LLMs may be the best shield against themselves implies model providers can cheaply screen for machine-generated text using their own flagship models without retraining."],"supporting_citations":[{"why":"Supplies the Fast-DetectGPT metric (conditional probability curvature) that Glimpse extends, plus the open-source surrogate baseline (GPT-J/Neo-2.7) against which the 51-percent gain is measured.","marker":"Bao et al. (2023)"},{"why":"Introduces the DetectGPT probability-curvature idea and the evaluation setup (XSum, Writing, PubMed) that the paper follows.","marker":"Mitchell et al. (2023)"},{"why":"DNA-GPT provides the black-box baseline for accuracy, speed, and cost comparisons, including the 4.1-times speed advantage Glimpse reports.","marker":"Yang et al. (2023a)"},{"why":"Provides the Zipfian law that underlies the power-law estimator's assumed rank-frequency decay.","marker":"Zipf (1946; 2013)"},{"why":"Supplies the typical Zipfian parameter values (alpha = 1, beta = 2.7) used to regularize the loss-table search.","marker":"Piantadosi (2014)"},{"why":"The M4 dataset provides the six-language evaluation data (Chinese, Russian, Urdu, Indonesian, Arabic, Bulgarian).","marker":"Wang et al. (2024)"},{"why":"The claim that smaller LLMs are better universal detectors, which the paper's larger-model results directly rebut.","marker":"Mireshghallah et al. (2023)"},{"why":"The DIPPER paraphraser is used to test robustness under lexical and order-diversity attacks.","marker":"Krishna et al. (2024)"},{"why":"GPT-Neo-2.7B supplies the open-source distributions the MLP estimator is trained on and serves as a primary surrogate baseline.","marker":"Black et al. (2021)"}],"fun_headline_variants":["Glimpse lets white-box detectors use proprietary LLMs via logprob estimation","White-box detection for proprietary LLMs: estimate logprobs, catch AI text","Glimpse: Use top-K logprobs to run Fast-DetectGPT on GPT-3.5","Proprietary LLMs can detect their own output via Glimpse's logprob estimation","Glimpse makes white-box detection work on proprietary LLMs"],"cache_read_input_tokens":28672,"weakest_assumption_plain":"The method assumes that the probability mass beyond the top-K tokens follows one of three fixed decay shapes, so that the tail can be reconstructed from the top-K values; if a model's true distribution has an unusual tail, for example a second concentration of probability at mid ranks, the estimated mean and variance in the curvature score would be wrong, and the paper's own KL-divergence measurements show the geometric estimator is the least faithful yet the most accurate, so the assumed link between faithful estimation and good detection is not fully understood.","fun_headline_variants_meta":{"raw":{"variants":["Glimpse lets white-box detectors use proprietary LLMs via logprob estimation","White-box detection for proprietary LLMs: estimate logprobs, catch AI text","Glimpse: Use top-K logprobs to run Fast-DetectGPT on GPT-3.5","Proprietary LLMs can detect their own output via Glimpse's logprob estimation","Glimpse makes white-box detection work on proprietary LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001271,"raw_usage":{"total_tokens":5237,"prompt_tokens":1018,"completion_tokens":4219,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":4111}},"tokens_in":634,"tokens_out":4219,"duration_ms":23622,"temperature":1.0,"reasoning_tokens":4111,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:51:25.626182+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Fast-DetectGPT with an open-source scoring model whose full distribution is available, on a held-out corpus, first with the true per-position distributions and then with Glimpse's top-K reconstruction; then artificially perturb a band of mid-rank probabilities, adding a second hump of probability mass at ranks 50 to 300, and repeat. If the AUROC gap between true- and estimated-distribution scores jumps when the tail ceases to be monotone-decaying, the parametric tail-shape assumption is the load-bearing part; if not, the metric is robust to tail errors and the central claim holds more broadly.","supporting_citations":[],"review_version":1}