{"id":"00469b38-cdb7-48be-af2e-89f2a243bd1b","arxiv_id":"2504.16574","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"PIS prunes tokens with BERT attention and TF-IDF, chooses per-sentence compression ratios with a DDQN, and drops similar sentences via a Russian roulette rule, reporting modest QA and summarization gains on GPT-4o-mini.","lead":"PIS is a prompt compression method that deletes tokens using BERT attention scores plus a small reinforcement learning network, then removes near-duplicate sentences, and reports gains over LLMLingua-2 on GPT-4o-mini benchmarks. It matters because cheaper long-context prompting would help LLM deployment, but the paper's theoretical bridge from attention to importance sampling relies on an invalid equation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.3's central identification of attention scores with the next-token probability distribution is dimensionally inconsistent and cannot ground importance sampling.","rationale":"The reader's strongest_claim correctly flags the false equality in Section 3.3, and I agree that this is the central theoretical assertion. However, the reader's weakest_assumption focuses on the BERT-to-LLM transfer in Section 4.1. That transfer issue is real but secondary: even if BERT's attention rankings matched GPT-4o-mini's, the paper has not established that the target model's attention scores are importance-sampling weights. The fundamental failure is earlier, in Section 3.3, where attention weights over prompt positions are equated with a next-token probability distribution over the vocabulary. This is an internal formal inconsistency, not merely a disagreement with common practice. Because the paper's claimed novelty is the theoretical link between attention and importance sampling, this defect undercuts the central contribution. The empirical SOTA claims may still be true as a heuristic outcome, but the manuscript does not provide the promised theoretical support. The reader's REJECT verdict remains appropriate, so I recommend no change to that verdict.","tokens_in":15098,"tokens_out":4986,"duration_ms":53278,"concrete_test":"Analytical re-derivation: fix a concrete prompt of n tokens and a tokenizer with vocabulary V. Compose the attention normalization ∑_{i=1}^n α_t,i = 1 and the next-token normalization ∑_{x∈V} pθ(x|x_<t) = 1, then test whether the equality α_t,i = pθ(x_i|x_<t) can hold for every prompt token and every step t, including cases with duplicate tokens. If no non-degenerate assignment satisfies both normalizations, the derivation in Section 3.3 is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's theoretical link is asserted in Section 3.3 as α_t,i = pθ(x_i|x_<t). This is not a harmless approximation: it is a category error. In a Transformer, α_t,i is a softmax over the n prompt positions, so ∑_{i=1}^n α_t,i = 1. By the paper's own definition, pθ(·|x_<t) is the model's next-token distribution over the vocabulary V, so ∑_{x∈V} pθ(x|x_<t) = 1. Summing the asserted equality over the prompt positions gives 1 = ∑_{i=1}^n pθ(x_i|x_<t), which holds only if the finite set of prompt tokens accounts for the entire probability mass of the next-token distribution. With |V| typically around 50,000 and n in the hundreds, this fails in general, and it fails even more clearly when prompt tokens repeat. The subsequent proportionality α_t,i ∝ w*(xi)·exp(<Q_t,K_i>/√d) inherits the same confusion and is not a derivation from importance-sampling theory. Additionally, the implemented token-level saliency uses BERT-base attention rather than the target LLM's α_t,i, so even a corrected version of the equation would not describe the actual pruning signal. The theoretical grounding of PIS therefore does not follow from the text as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Prompt Importance Sampling (PIS), a dual-level prompt compression method: token-level pruning based on attention scores extracted from a small encoder-only model, an RL-trained policy for selecting per-sentence compression ratios, and sentence-level deletion via a Russian-roulette similarity mechanism. The authors claim a measure-theoretic link between attention scores and importance weights, state-of-the-art compression quality (a 15% improvement at equal compression ratios and a 38% latency reduction), and occasional improvements in downstream reasoning. The evaluation compares against LLMLingua, LLMLingua-2, and Selective Context on MeetingBank, GSM8K, BBH, and LongBench-GovReport, with additional LLM comparisons in Appendix B.","tokens_in":15541,"tokens_out":5252,"duration_ms":50497,"significance":"If the central claims held, the paper would offer an interesting connection between importance sampling and attention, together with practical efficiency gains. The manuscript has clear strengths: it provides open-source code, reports ablations (Tables 5 and 6), includes a latency study (Table 4), and evaluates across several target models in Appendix B. However, the theoretical bridge is built on an incorrect equality between attention weights and conditional token probabilities; the practical saliency signal is obtained from BERT, not from the target LLM, with no transfer validation; and the in-domain headline results on MeetingBank are confounded by the overlap between the RL reward metrics and the evaluation metrics. These issues jointly undermine the paper's main theoretical and empirical claims as written.","major_comments":[{"comment":"The central identification alpha_{t,i} = p_theta(x_i | x_<t) is technically false. The attention weight alpha_{t,i} is a softmax over the n prompt positions, so sum_{i=1}^n alpha_{t,i} = 1, whereas p_theta(. | x_<t) is a distribution over the vocabulary V, so sum_{x in V} p_theta(x | x_<t) = 1. Summing the asserted equality over prompt positions implies that all next-token probability mass is concentrated on the n prompt tokens, which fails for standard vocabularies and for repeated tokens. The subsequent proportionality alpha_{t,i} proportional to w*(x_i) * exp(<Q_t,K_i>/sqrt(d)) inherits this error, so the theoretical grounding of attention-based pruning in importance sampling does not follow. Moreover, the implemented token saliency in Sec. 4.1 uses BERT-base attention rather than the target LLM's alpha_{t,i}, so even a corrected equation would not describe the actual pruning signal used by the method.","section":"Sec. 3.3"},{"comment":"The assumption that BERT-base attention rankings transfer to GPT-4o-mini is asserted without evidence: the paper claims that token representations share a key property such that relative relationships are consistent, but no experiment compares BERT attention rankings with the target LLM's attention rankings, nor measures agreement of the resulting compressed prompts. Since all token-level pruning decisions in PIS are based on BERT scores, the abstract's claim of using \"LLM-native attention scores\" is unsupported. This transfer assumption is load-bearing and requires direct validation, for example by reporting rank correlation of attention scores or downstream agreement between BERT-based and target-LLM-based compression.","section":"Sec. 4.1"},{"comment":"The in-domain results in Table 1 are circular with respect to training: the DDQN is trained on MeetingBank's training split (Sec. 5.1) with a reward containing ROUGE-1 and BLEU terms (Sec. 4.2), and Table 1 evaluates the same metrics on MeetingBank. The reported improvement therefore partly reflects fitting the reward function rather than an independent compression advantage. The paper should report results with reward metrics excluded from training or evaluated on a held-out domain, and it should provide standard deviations or confidence intervals; the statement that results are averaged over five rounds does not substitute for reporting the variance.","section":"Sec. 5.3.1 and Sec. 5.1"},{"comment":"The passage from sequence-level importance weights to token-level weights is not derived. The paper defines the importance weight w(omega) = p*(omega)/p_theta(omega) on full sequences but then uses token-level weights w*(x_i) = p*(x_i)/p_theta(x_i) without showing that the sequence weight factorizes into a product of such token terms. For an autoregressive model p_theta(omega) = product_t p_theta(x_t | x_<t), but p_theta(x_i | x_<t) is a conditional next-token probability, not a marginal probability suitable for a per-token weight. Without an explicit factorization argument, token-level pruning cannot be justified by importance-sampling theory.","section":"Sec. 3.2 and Sec. 3.3"}],"minor_comments":[{"comment":"The assumption first states that there is \"one or more optimal answers\" but then posits \"a unique optimal answer a*\", which is internally inconsistent; please clarify the intended assumption.","section":"Sec. 3.1"},{"comment":"The symbol tau is overloaded: it is a quality threshold in the reward components R_content and R_fluency, while the experiment tables use 1/tau to denote compression ratio; please disambiguate these uses.","section":"Sec. 4.2"},{"comment":"The deletion probability P_delete = k/6 is defined for k = 1,...,6, but the text says k is the number of consecutive similar sentences, which can exceed 6; specify whether k is capped at 6.","section":"Sec. 4.3"},{"comment":"The latency comparison does not state the hardware, batch size, input length distribution, or whether the reported time includes compression overhead; this limits reproducibility.","section":"Table 4"},{"comment":"The weighted importance score uses TF-IDF but does not define the document unit or corpus over which IDF is computed; please specify these details.","section":"Sec. 4.1"},{"comment":"The case-study discussion contains evaluative claims such as \"superior compression\" and \"functionally superior\" without quantitative evidence; please either add supporting measurements or temper the language.","section":"Appendix D"}],"recommendation":"reject","confidential_remarks":"The editors may wish to note that the in-domain SOTA claim on MeetingBank is especially fragile because the DDQN is trained on MeetingBank with rewards that include the same ROUGE and BLEU metrics used for evaluation. In addition, the citation list contains several of the authors' own preprints, and some entries appear tangential; a tightened bibliography would strengthen the paper if a revised version is submitted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's stated theoretical foundation, the link between attention and importance sampling, is wrong. Section 3.3 literally equates attention scores α_t,i with pθ(x_i|x_<t). Attention is a softmax over prompt positions, so Σ_i α_t,i = 1; the next-token distribution is a softmax over the vocabulary. The asserted equality only holds if prompt tokens carry the entire probability mass, which is not the case. So the derivation is a category error and the 'measure-theoretic grounding' does not hold up.\n\nWhat is genuinely new is the pipeline: BERT-base attention for token saliency, a 9-layer DDQN to pick per-sentence compression ratios, and a Russian roulette similarity-based sentence deletion. I don't know of another paper that combines those three. The ablations suggest both levels matter, the latency table shows meaningful gains over LLMLingua-2, and the out-of-domain results on GSM8K/BBH and LongBench are better than the baselines at similar compression ratios. If those numbers are right, this is a useful heuristic.\n\nThe soft spots beyond the theory: the saliency signal comes from BERT, not the target LLM, and the paper asserts without evidence that well-trained encoders produce consistent relative attention rankings. That assumption needs a direct experiment. Second, the RL reward uses ROUGE-1 and BLEU, and the in-domain evaluation uses the same metrics, so the headline MeetingBank SOTA is partly reward-fitting. The out-of-domain results are cleaner, but there are no error bars anywhere, so we only have point estimates. They also claim open-source code but I don't see a link or a repo in the text.\n\nOverall: the theory should be discarded, but the empirical combination is worth engaging with. A reviewer would need the authors to validate the BERT transfer, add error bars, and temper the claims. That's a realistic revision path.\n\nI'd send this to peer review — the community would be better served by a corrected or de-theorized version of this paper than by a desk reject.","headline":"The theoretical grounding in Section 3 is mathematically wrong, but the empirical pipeline is genuinely new and worth a serious look.","tokens_in":15950,"tokens_out":2378,"would_cite":false,"duration_ms":22306,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that attention scores can be read as importance-sampling weights, and builds a two-level prompt compressor on that idea that beats earlier baselines with roughly a third less latency.","keywords":["prompt compression","importance sampling","attention mechanisms","reinforcement learning","DDQN","Russian roulette sampling","token pruning","LLM inference efficiency"],"falsifier":"Take the prompts used in the paper and compute BERT's attention-score ranking of tokens against the target model's own attention-score ranking: if the overlap of the top-ranked tokens is at chance level, the cross-model transfer premise fails and the token-level pruning is not grounded in the target model's importance signal.","tokens_in":14932,"feed_emoji":"✂️","tokens_out":10583,"duration_ms":85077,"temperature":0.7,"pith_summary":"Prompt compression, the paper argues, should be treated as importance sampling: generation is a sample from a distribution over token sequences, and the goal of compression is to reweight that sample toward the tokens that matter for a good answer. The authors claim that transformer attention scores can be interpreted as conditional token probabilities, making them a principled proxy for the ideal importance weights. They build PIS, a two-level compressor: token-level pruning uses attention-derived saliency with a TF-IDF correction and a reinforcement-learned per-sentence compression ratio, and sentence-level deletion uses a Russian-roulette rule that progressively removes near-duplicate sentences. On QA and summarization benchmarks the method reports better scores than LLMLingua, LLMLingua-2, and Selective Context at comparable compression ratios, a 38% latency reduction, and occasional accuracy gains over the uncompressed prompt.","feed_headline":"Attention-based prompt compression beats baselines, cuts latency 38%","feed_subtitle":"A two-level sampler - token pruning plus sentence roulette - preserves quality while shrinking prompts for LLMs.","key_machinery":"The central object is the softmax attention score $\\alpha_{t,i}=\\exp(\\langle Q_t,K_i\\rangle/\\sqrt{d})/\\sum_j\\exp(\\langle Q_t,K_j\\rangle/\\sqrt{d})$, which the paper reinterprets as the conditional probability $p_\\theta(x_i|x_{<t})$ and therefore as a per-token importance weight. The framework couples that identity with two sampling mechanisms: token-level importance sampling, which deletes tokens with high attention variance (TF-IDF provides a corrective term so frequent, distinctive tokens survive), and sentence-level Russian-roulette deletion, which removes a new sentence with probability $k/6$ when it has high cosine similarity to already-kept sentences. A lightweight 9-layer Double Deep Q-Network chooses each sentence's compression ratio, taking BERT embeddings of the current, previous, and next sentences as its state and receiving a reward that blends ROUGE-1, BLEU, and the achieved compression rate.","core_discovery":"The paper's core claim is that an LLM's generation can be modeled as a sampling procedure over a probability space of token sequences, and that prompt compression is the act of importance-sampling that space toward the tokens that would produce an optimal answer. The load-bearing identity is the assertion that the attention score $\\alpha_{t,i}$ equals the model's conditional probability of token $i$ given the preceding tokens, $\\alpha_{t,i} = p_\\theta(x_i|x_{<t})$, which the authors combine with the definition of the optimal importance weight $w^*(x_i)=p^*(x_i)/p_\\theta(x_i)$ to obtain $\\alpha_{t,i} \\propto w^*(x_i)\\exp(\\langle Q_t,K_i\\rangle/\\sqrt{d})$. Attention scores therefore track the ideal importance-sampling weights, so deleting low-saliency tokens and near-duplicate sentences should reduce sampling variance and push generation closer to the optimal answer. PIS is presented as the concrete implementation of this view, with measured gains in accuracy and latency across in-domain and out-of-domain benchmarks.","pith_inferences":["A training-free variant that ranks tokens by attention-score variance across heads and layers could test whether the RL layer is essential; the paper's own latency numbers suggest such a variant would be even cheaper.","The cross-model transfer premise could be tested directly by comparing BERT's token-saliency rankings with the target model's own attention rankings; the reported gains give reason to expect meaningful overlap.","The Russian-roulette deletion rule is a generic near-duplicate filter that could be applied to retrieval-augmented generation or long-context memory, where repeated chunks inflate cost without adding information.","The occasional accuracy gain over the uncompressed prompt implies some prompt content actively hurts generation; profiling which tokens the compressor removes most often could yield an inventory of noise tokens for prompt design."],"forward_implications":["At equal compression ratios, PIS reports higher QA exact match and summarization scores than Selective Context, LLMLingua, and LLMLingua-2 on MeetingBank, e.g. 89.05 EM vs 87.19 for LLMLingua-2.","Compressed prompts can outperform the uncompressed prompt on downstream tasks, with the paper reporting around a 5% accuracy improvement from the optimized context structure.","Inference latency drops by roughly 30–40%, with the largest gains at long inputs: at 1,500 tokens and 5× compression, PIS takes 2.65 s versus 3.85 s for LLMLingua-2.","Ablation shows both compression levels are necessary: removing either token-level or sentence-level sampling drops performance below LLMLingua-2 across all reported metrics.","The same pipeline transfers to other target models (GPT-3.5, GPT-4, Mistral-7B, Llama-2-7B), with consistent gains over baselines in the appendix."],"supporting_citations":[{"why":"Supplies the attention formula that the paper reinterprets as a token probability.","marker":"(Vaswani, 2017)"},{"why":"Provides BERT-base-uncased, the encoder used for attention-score extraction and for the RL state embeddings.","marker":"(Devlin et al., 2018)"},{"why":"Introduces LLMLingua, the generative compression baseline PIS compares against in every evaluation.","marker":"(Jiang et al., 2023a)"},{"why":"Introduces LLMLingua-2, the strongest token-pruning baseline; its evaluation protocol is adopted and its results are the main comparison.","marker":"(Pan et al., 2024a)"},{"why":"Introduces Selective Context, the heuristic pruning baseline used in all evaluations.","marker":"(Li et al., 2023)"}],"fun_headline_variants":["LLM prompt compressor uses attention sampling, cuts latency 38%","PIS: prompt compression via importance sampling, faster reasoning","Prompt importance sampling trims tokens, speeds LLMs by 38%","Attention-guided sampling shrinks prompts, keeps quality, cuts delay","New prompt compressor: token pruning and sentence roulette"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that token-saliency rankings from a small encoder-only model (BERT) transfer to a larger, different target LLM, so that pruning based on BERT attention scores reflects what the target model itself would treat as important; the paper asserts that relative token relationships are consistent across well-trained models but does not test this against the target model's own attention.","fun_headline_variants_meta":{"raw":{"variants":["LLM prompt compressor uses attention sampling, cuts latency 38%","PIS: prompt compression via importance sampling, faster reasoning","Prompt importance sampling trims tokens, speeds LLMs by 38%","Attention-guided sampling shrinks prompts, keeps quality, cuts delay","New prompt compressor: token pruning and sentence roulette"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1345,"prompt_tokens":977,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":282}},"tokens_in":593,"tokens_out":368,"duration_ms":4548,"temperature":1.0,"reasoning_tokens":282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:00:20.667479+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the prompts used in the paper and compute BERT's attention-score ranking of tokens against the target model's own attention-score ranking: if the overlap of the top-ranked tokens is at chance level, the cross-model transfer premise fails and the token-level pruning is not grounded in the target model's importance signal.","supporting_citations":[],"review_version":1}