{"id":"2d0db92a-e386-42ad-9a28-f7254aa7d4ec","arxiv_id":"2608.09176","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Allocating visual tokens by the cost of getting a question wrong reduces cost-weighted error of VLMs under a fixed total token budget, with controlled evidence that the gain comes from error cost, not content sensitivity.","lead":"This paper proposes allocating visual token budgets in vision-language models according to the downstream cost of errors, not just image content. On a controlled document benchmark it cuts high-stakes errors more than twofold at the same total compute, and the principle transfers to several benchmarks and models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Within-task attribution rests on a single hand-designed content baseline; a stronger image-only allocator that predicts tier from image statistics could reproduce the gain, undermining the central claim.","rationale":"I deliberately choose a different concern from the reader's weakest assumption. The reader focuses on the consequence weights and tier assignments being arbitrary or miscalibrated. While that is a real scope limitation, it does not threaten the internal validity of the controlled within-task experiment: under the stated weights, the allocation does minimize the stated cost-weighted objective. The more load-bearing threat is the possibility that the empirical separation between 'consequence-sensitive' and 'content-driven' is an artifact of a weak content baseline. The paper's headline claim is precisely that content alone cannot do what consequence does. If a stronger image-only allocator can reproduce the gain, the paper's central contribution is not the allocation principle but merely a particular instantiation of a content allocator. The paper's own cross-task result shows content can win when there is a sensitivity gap; the within-task design is supposed to eliminate this by matching image content, but the matching is asserted rather than demonstrated (no image-pairing or distributional balance check). The proposed test — a learned image-only tier predictor — directly settles whether the within-task separation holds. Until such a test is run, the central attribution remains conditional, so the reader's CONDITIONAL verdict stands.","tokens_in":16807,"tokens_out":15513,"duration_ms":226571,"concrete_test":"Train a logistic regression on vision-encoder features of the 600 within-task images to predict high/low consequence tier from the image alone (cross-validated). Allocate budgets by predicted tier: top half of images get 512 tokens, bottom half get 96, preserving the total budget. Measure high-tier error and CWE with the paper's scorer. If this image-only allocator achieves high-tier error below ~0.2 or significant McNemar vs uniform, the paper's claim that content cannot reveal consequence is falsified; if near-uniform performance results, the attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central attribution claim ('content-driven allocator performs no better than uniform') is established with exactly one content baseline: a fixed feature-diversity rule that assigns budgets proportional to (1 - mean pairwise cosine similarity), clamped and redistributed. The abstract asserts that 'content alone cannot reveal which questions are costly to get wrong' because high/low tiers are 'drawn from the same document images,' but the experimental setup (Section 4) draws 600 questions from 1184 scanned DocVQA items without any explicit image pairing or content-matching check between tiers. If high- and low-consequence questions come from different images, image statistics (text density, layout, presence of tables/numbers) can differ systematically between tiers, and a learned content-based allocator could recover the tier assignment from the image. The paper shows only that its particular diversity measure is uninformative (Table 2: content allocator assigns 305/303 tokens and achieves high-tier error 0.300), which does not rule out stronger baselines such as a logistic regression on vision-encoder features or a difficulty/uncertainty-based allocator. If such a baseline reduces high-tier error comparably to the consequence-sensitive allocation, the observed gain is not uniquely attributable to consequence information. The paper's Section 3.5 acknowledges cross-task settings where content wins, but does not test the within-task cell with a strong content baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes consequence-sensitive visual token compression: instead of compressing every image to the same average token budget, the method first estimates tier-specific error-budget curves offline and then allocates a fixed total token budget across consequence tiers so as to minimize cost-weighted error. The central controlled experiment is a within-task DocVQA setup in which high- and low-consequence questions are split by a frozen keyword rule; on that benchmark, the consequence-aware allocation (512,96) reduces high-tier error from 0.300 to 0.133 under an equal total budget, while a hand-designed content-diversity allocator matches uniform allocation. The paper also reports an allocation frontier across cost ratios, cross-dataset replications on ChartQA and InfographicVQA, invariance across two budget-realization mechanisms, two VLMs, and six token selectors, and a three-tier mixed workload with a 38% cost-weighted-error reduction and about 21% latency reduction.","tokens_in":17010,"tokens_out":11690,"duration_ms":118774,"significance":"If the attribution result holds, the paper opens a genuinely new axis for VLM efficiency: allocating visual computation by downstream error cost rather than by input content or difficulty. The experimental hygiene is strong in several respects: the equal-total-budget protocol is asserted per image and per strategy, the primary comparison is pre-specified, McNemar tests are exact, CWE differences are assessed by paired bootstrap, and the headline numbers survive strict exact-match and ANLS scoring. The authors also deserve credit for explicitly rejecting the convex water-filling derivation because their measured curves are non-convex, and for being candid about the cases where content-driven allocation wins. The main risk is that the within-task attribution claim, which is the paper's central evidence that consequence information adds value beyond content, rests on an unverified content-balance assumption and a single content baseline; in addition, part of the deployment-oriented evaluation appears to be in-sample. Both concerns are addressable in revision.","major_comments":[{"comment":"The within-task design does not currently establish that high- and low-consequence questions share identical visual content. The confirmatory 600 samples are drawn from 1184 DocVQA items and split by a question-text keyword rule; the paper does not state that high/low questions are paired to the same images, and it reports no image-level balance check (e.g., native token count, text density, layout, presence of tables) across tiers. Because keywords like amount, price, date, and number are likely correlated with document type (invoices vs. forms), an image-only allocator could plausibly recover tier assignment from visual statistics. The paper tests only one hand-designed feature-diversity baseline, which happens to assign nearly equal budgets (305/303), so the conclusion that 'content alone cannot reveal which questions are costly to get wrong' is stronger than the evidence supports. I request three additions: (a) image-level balance diagnostics or, preferably, a construction that pairs high- and low-consequence questions on the same underlying images; (b) an image-only tier-prediction sanity check (e.g., logistic regression on vision-encoder features, or a nearest-neighbor density test) showing chance-level discrimination; and (c) one or two stronger content-based allocation baselines, such as uncertainty-based or difficulty-based budget assignment, evaluated on the same within-task protocol.","section":"Section 4 / Table 2"},{"comment":"The 'deployed optimum' (576,32) at r=5 appears to be selected from the full confirmatory 600 and then evaluated on the same 600: the reported discordant pairs 65:5 and the ΔCWE CI [0.032,0.122] are not described as held-out results. As written, these numbers re-describe the in-sample grid optimum rather than providing out-of-sample evidence for calibrate-then-allocate. The split-half calibration is the right idea, but at r=5 the predicted allocation is only rank 3/13 (and rank 6/13 at r=3), and the paper does not state whether the deployed policy's error numbers come from the held-out half or from the full set. Please report the deployed (576,32) policy on a genuinely disjoint held-out set, or explicitly restrict the 'deployed' claims to the split-half evaluation and report the held-out CWE for the selected allocation.","section":"Section 5.2 / Table 3"}],"minor_comments":[{"comment":"The abstract and introduction say high- and low-consequence questions are 'drawn from the same document images,' but the experimental section says they come from the same DocVQA pool and does not pair questions to identical images; please revise the wording to 'matched image distribution' or explicitly describe the pairing if it exists.","section":"Abstract / Section 1 / Section 4"},{"comment":"The table header does not clearly separate the E_hi/E_lo columns from the CWE-at-ratio columns; adding a two-level header or explicit labels for all six numeric columns would prevent misreading.","section":"Table 3"},{"comment":"The interior optimum at r=3 is unstable under split-half calibration (rank 6/13), yet the deployed policy at r=5 jumps to the boundary (576,32); please discuss how sensitive the boundary choice is to the grid endpoints and whether a finer grid near the boundary would change the deployment recommendation.","section":"Section 5.2"},{"comment":"The sentence 'corresponding to only a two-sample high-tier difference' is unclear; please restate the small-sample optimism analysis in terms of the actual discordant pair counts.","section":"Technical Supplement, Section B"},{"comment":"The 'frontier-LLM' judge is described only as an independent frontier LLM; please name the model and version so the reader can interpret the 72.5% agreement and the bias taxonomy.","section":"Technical Supplement, Section G"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-executed study with unusually careful budget accounting and statistical reporting. The central risk is not the execution but the interpretation of the within-task attribution: without evidence that the keyword split is content-balanced across tiers, the claim that content-driven allocators are blind is not fully supported. The second risk is that the deployment-oriented evaluation of (576,32) appears in-sample. Both are fixable with additional analysis, so I recommend major revision rather than rejection. The authors' prior work on consequence-aware reasoning compute allocation (Wen, He, He 2026) is cited; the editor may want to confirm that the visual-token contribution is sufficiently distinct from that earlier framework."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real result wrapped in a slightly overbroad attribution claim. The controlled within-task experiment—pre-specified (512,96) allocation, disjoint pilot and confirmatory sets, exact McNemar tests, equal realized budgets—is the right way to test whether cost-aware allocation helps, and the 2.25x high-tier error reduction with the anti-direction control is convincing. The paper also does several things unusually well: it explicitly rejects the naive water-filling derivation because the measured curves are non-convex, uses exhaustive grid search, reports split-half calibration (honestly flagging the r=3 interior optimum as rank 6/13), and states plainly that ChartQA's CWE gain is not significant. The label validation against gold-answer content and an LLM judge, plus the reproducibility artifact, are genuine assets.\n\nThe main soft spot is the attribution claim that \"content alone cannot reveal which questions are costly to get wrong.\" That conclusion is supported by exactly one content baseline—a feature-diversity rule that assigns near-uniform budgets (305/303) and matches uniform. That rules out that particular rule, not the class of content-driven allocators. The paper says high- and low-consequence questions come from the same document images, but the experimental section doesn't show explicit per-image pairing between tiers; if the two tiers actually draw from different images, a learned allocator on image statistics (text density, layout, number of digits) could recover tier and reproduce part of the gain. A logistic regression on vision-encoder features or an uncertainty-based baseline would settle this. The authors acknowledge the two-axis map and the cross-task cell where content wins, so they know the boundary; they just didn't test the stronger within-task cell.\n\nOther soft spots are minor: the consequence weights (c=5 vs c=1) are arbitrary but the paper sweeps ratios and validates labels; the mixed-workload tiered budget looks hand-picked without held-out calibration; ChartQA's CWE CI includes zero at r=5. None of these break the main controlled result.\n\nWho this is for: anyone working on efficient VLM inference or adaptive compute allocation. The calibrate-then-allocate recipe and the invariance study make it directly useful. It deserves a serious referee; I'd send it out with a request to add a stronger content baseline and to state explicitly whether the within-task tiers share the same physical images. That is an addressable revision, not a fatal flaw.","headline":"A genuinely careful controlled experiment shows cost-aware visual token allocation helps under fixed budgets, but the claim that content alone cannot drive the gain rests on one weak baseline and an unverified same-image assumption.","tokens_in":17603,"tokens_out":2847,"would_cite":true,"duration_ms":30845,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Allocating visual tokens by downstream error cost, not image content, cuts high-stakes errors 2.25x under the same total budget.","keywords":["visual token compression","consequence-sensitive allocation","cost-weighted error","vision-language models","token budget allocation","error-budget curves","DocVQA","adaptive inference"],"falsifier":"Compute the real monetary or operational cost of each wrong answer on a deployed workload and compare the keyword-rule tiers against it; if the tier labels misclassify enough high-cost questions (the paper already leaks transactional questions like 'budget', 'tel', and 'zip' into the low tier), cost-weighted error will be no better than uniform or worse. Alternatively, find a within-task dataset where high-consequence questions are measurably more token-sensitive than low-consequence ones, with a marginal error-reduction gap well above the reported standard error; such a gap would collapse the attribution of the gain to consequence alone.","tokens_in":16542,"feed_emoji":"⚖️","tokens_out":5436,"duration_ms":48117,"temperature":0.7,"pith_summary":"This paper argues that visual token compression for vision–language models should be guided by the cost of being wrong, not just by image content. It introduces consequence-sensitive token allocation: estimate how each cost tier's error rate responds to token budget offline, then split a fixed total budget across tiers at inference. In a controlled within-task benchmark, where high- and low-consequence questions share identical document images, this reduces high-stakes error from 0.300 to 0.133 under the same total token budget, while content-driven allocation performs no better than uniform. The paper also maps the allocation frontier: uniform allocation is optimal when errors are equally costly, and transferring tokens to high-consequence questions pays off as cost asymmetry grows.","feed_headline":"Cost-aware token budgets cut high-stakes VLM errors 2.25x","feed_subtitle":"Same total token budget, but tokens move to the questions where a wrong answer costs more.","key_machinery":"The central object is a tiered cost-weighted error minimization: given tiers with weights $c_t$ and measured error-budget curves $e_t(b)$, minimize $\\sum_t n_t c_t e_t(b_t)$ subject to a fixed total budget $B$, solved by exhaustive search over a discrete budget grid and checked by a swap condition (no single grid-step transfer to either tier may lower the objective). The classical water-filling rule $c_t|e'_t(b_t)|=\\lambda$ is used only descriptively because the measured curves are non-convex and budgets are discrete. Budgets are realized either by exact visual token deletion with preserved spatial coordinates or by resolution reallocation that resizes the image so the encoder emits the desired token count.","core_discovery":"On the paper's own terms, for VLM inference under a fixed vision-token budget, minimizing cost-weighted error instead of average error yields large gains when error costs differ and content cannot reveal them. In the controlled within-task setting (DocVQA, high- and low-consequence questions drawn from the same document images), consequence-sensitive allocation cuts high-tier error from 0.300 to 0.133 (54:4 discordant pairs, p≈3×$10^{-12}$ by McNemar), whereas a content-based allocator is indistinguishable from uniform (4:4, p=1.0). The two tiers have statistically indistinguishable token sensitivity (marginal error reductions 0.00173 vs 0.00171, SE≈2×$10^{-4}$), so the gain is attributed to cost asymmetry, not content. Unweighted accuracy rises from 0.337 to 0.488 because the method optimizes cost-weighted error by design. The principle replicates across three dense benchmarks, two realization mechanisms (token deletion and resolution reallocation), two architectures, and multiple token selectors; on a realistic mixed workload it reduces cost-weighted error by 38% while achieving about 21% lower latency than full-resolution inference.","pith_inferences":["The same calibrate-then-allocate scheme could govern other adjustable compute axes, such as reasoning steps, number of image crops, or ensemble size, whenever downstream error costs are asymmetric; the paper does not test these transfers.","The 'both gaps' regime (a content-observable sensitivity gap plus a cost gap) is untested; a natural extension is a hybrid allocator that first routes tokens by content sensitivity and then applies consequence weights, with no evidence yet on whether the gains compose.","Because a lightweight text classifier matches the LLM judge's consequence labels better than the fixed keyword rule (83.5% vs 72.5% agreement), consequence tiers could in practice be learned from application-specific cost data rather than hand-written, mitigating the paper's key limitation.","Practitioners could use the reported break-even ratios as a deployment rule: estimate the true cost ratio of the workload and only transfer tokens to the high-cost tier when that ratio exceeds the benchmark's break-even point, otherwise keep uniform allocation."],"forward_implications":["When error costs are asymmetric, uniform token allocation is suboptimal; shifting tokens to high-consequence questions lowers cost-weighted error at the same total compute.","Content-based allocators (attention, redundancy, difficulty) cannot substitute for consequence information when high- and low-cost questions share the same image content; the two signals are orthogonal.","The allocation gain is a property of budget allocation, not of any specific token selector, model, or reduction mechanism, so consequence-sensitive allocation composes with existing pruners such as FastV, SparseVLM, and VisionZip.","The measured break-even cost ratio tells when transfer pays: roughly 2.8:1 for DocVQA, 3.6:1 for ChartQA, and 0.7:1 for InfographicVQA, so denser visual tasks justify token transfer at smaller cost gaps.","On a realistic mixed workload, task identity alone (without per-question keyword rules) yields a 38% reduction in cost-weighted error and about 21% lower latency than full-resolution inference."],"supporting_citations":[{"why":"Supplies DocVQA, the dataset used for the within-task benchmark and the primary attribution experiment.","marker":"(Mathew, Karatzas, and Jawahar 2021)"},{"why":"Provides Qwen2.5-VL, the primary VLM whose native dynamic-resolution tiling the resolution mechanism exploits.","marker":"(Qwen Team 2025)"},{"why":"Supplies LLaVA-OneVision, the second architecture used to test whether the allocation gain is model-specific.","marker":"(Li et al. 2024)"},{"why":"FastV, a token-selection baseline used in the selector-invariance study.","marker":"(Chen et al. 2024)"},{"why":"SparseVLM, another token-selection baseline used to test orthogonality of the allocation principle.","marker":"(Zhang et al. 2025)"},{"why":"VisionZip, a third token-selection baseline included in the selector-invariance study.","marker":"(Yang et al. 2025)"},{"why":"Provides the cost-sensitive learning foundation that motivates the cost-weighted error objective.","marker":"(Elkan 2001)"},{"why":"Supplies ChartQA, a generalization benchmark on which the frozen tier rule is applied.","marker":"(Masry et al. 2022)"},{"why":"Supplies InfographicVQA, a generalization benchmark on which the frozen tier rule is applied.","marker":"(Mathew et al. 2022)"}],"fun_headline_variants":["Cost-aware token cuts: 2.25x fewer high-stakes VLM errors","Send tokens to costly questions: 2.25x fewer mistakes that cost","Consequence-sensitive token budgets cut costly VLM errors 2.25x","Cost-weighted token budgets: 55% fewer costly VLM errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole gain rests on the assumption that the cost weights and tier assignments used for allocation match the true downstream costs of being wrong, and that these costs are known before inference; the paper's frozen keyword rule with weights (5 vs 1) is only indirectly validated.","fun_headline_variants_meta":{"raw":{"variants":["Cost-aware token cuts: 2.25x fewer high-stakes VLM errors","Send tokens to costly questions: 2.25x fewer mistakes that cost","Consequence-sensitive token budgets cut costly VLM errors 2.25x","Cost-weighted token budgets: 55% fewer costly VLM errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0014,"raw_usage":{"total_tokens":5736,"prompt_tokens":1098,"completion_tokens":4638,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":714,"completion_tokens_details":{"reasoning_tokens":4555}},"tokens_in":714,"tokens_out":4638,"duration_ms":36771,"temperature":1.0,"reasoning_tokens":4555,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:00:28.395822+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the real monetary or operational cost of each wrong answer on a deployed workload and compare the keyword-rule tiers against it; if the tier labels misclassify enough high-cost questions (the paper already leaks transactional questions like 'budget', 'tel', and 'zip' into the low tier), cost-weighted error will be no better than uniform or worse. Alternatively, find a within-task dataset where high-consequence questions are measurably more token-sensitive than low-consequence ones, with a marginal error-reduction gap well above the reported standard error; such a gap would collapse the attribution of the gain to consequence alone.","supporting_citations":[],"review_version":1}