{"id":"ff7039fb-8e9e-445a-b74e-d7b484f23ce9","arxiv_id":"2504.18406","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 27-dataset benchmark shows current vision-language models average about 50% on high-resolution image tasks and display uneven regional attention in synthetic grid tests.","lead":"The authors built HRScene, a benchmark for testing AI vision-language models on very large images, from pathology slides to satellite photos. Across 28 models, the best scores hover around 50 to 62 percent, and synthetic tests reveal the models ignore or mishandle large regions of high-resolution images.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The synthetic lost-in-the-middle and Regional Divergence findings are not yet separated from vision-encoder positional/token-order artifacts; only the linear patch-order alternative is tested (Supp. 8), leaving 2D spatial priors as a live confound.","rationale":"The real-world benchmark contribution is solid: the aggregate ~50% accuracy is well above random (~22%) and below the reported human average (~65%), and the per-dataset tables in the supplement corroborate the aggregate. Training-data contamination, if present, would tend to inflate VLM scores and therefore make the headline gap conservative, so I do not treat it as the load-bearing concern. The load-bearing weakness sits in the synthetic diagnostics. The paper claims a novel Manhattan-distance lost-in-the-middle effect and Regional Divergence as evidence of ineffective HRI region utilization, but the experiments do not isolate the HRI-attention mechanism from the vision encoder's 2D positional and token-order biases. The supplementary linear patch-distance analysis excludes one alternative but not the more plausible 2D spatial-prior alternatives. A token-order reversal experiment on a single open-source model would decide the question. This does not overturn the paper; it sharpens the condition under which the diagnostic claims can be accepted. Hence the reader's CONDITIONAL verdict is appropriate, with the additional control explicitly required. I agree with the reader's weakest assumption.","tokens_in":27707,"tokens_out":16870,"duration_ms":178971,"concrete_test":"For one open-source VLM (e.g., InternVL2-8B), re-run the ComplexGrid 10x10 evaluation with the visual token sequence reversed: keep the pixel content identical, hold the ViT patch positional embeddings fixed, but feed the patch embeddings to the LLM in reverse raster order so the first visual token carries the bottom-right patch and the last carries the top-left patch, then recompute the Manhattan-distance accuracy curve. If the U-shape inverts or shifts to follow the new token order, the reported lost-in-the-middle is a token-order/positional-embedding artifact of the encoder; if it remains anchored to the physical grid layout, the regional-utilization interpretation survives.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"HRScene's central synthetic contribution—Regional Divergence and the Manhattan lost-in-the-middle phenomenon—depends on interpreting per-position accuracy in WhiteBackground/ComplexGrid as evidence about how VLMs utilize regions in high-resolution images. That interpretation is underdetermined. The needles are low-resolution VQAv2 images tiled into a larger canvas, and every evaluated VLM re-encodes the full canvas through a vision transformer with learned 2D positional embeddings, resolution-constrained resizing, tiling, and padding. Per-position performance differences can therefore be generated by the encoder's positional-embedding geometry, raster-order attention biases, border/padding effects, or aspect-ratio-dependent tiling, rather than by a content-dependent failure to attend to middle regions. The supplementary ablation (Section 8) tests only the linear patch-order version of lost-in-the-middle and finds no pattern; it does not control for Euclidean distance to center, row/column main effects, distance-to-border, tiling boundaries, or the effect of reversing the visual token sequence. Because 'lost-in-the-middle-Manhattan' is explicitly claimed as novel and distinct from the text NIAH result, this confound is load-bearing: if the U-shape is a token-order or positional-embedding artifact, the diagnostic conclusion about HRI-specific region utilization is unsupported, even though the ~50% real-world accuracy result is left intact. The low-resolution VQAv2 needle in a white or similar-image grid is also at best an indirect proxy for detailed high-resolution perception, which further weakens the mechanistic interpretation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces HRScene, a benchmark for high-resolution image (HRI) understanding that combines 25 real-world datasets (7,068 samples, resolutions from 1K to roughly 35,000×26,000 pixels) with two synthetic NIAH-style diagnostic datasets. The authors evaluate 28 VLMs (6 proprietary, 22 open-weights) and report that current models reach only about 50% average accuracy on real-world tasks, with the best model (Qwen2-VL 72B) scoring 61.85% versus a non-expert human baseline of 64.72%. On the synthetic diagnostics, the paper observes Regional Divergence (accuracy varies across grid positions) and a 'lost-in-the-middle-Manhattan' U-shape in which accuracy declines with Manhattan distance from the top-left corner of the grid.","tokens_in":28007,"tokens_out":4186,"duration_ms":41544,"significance":"The real-world part of HRScene is a substantial and useful contribution: it covers an unusually wide range of scene types and resolutions, includes 2,008 re-annotated or scratch-annotated samples, and provides a consistent evaluation of 28 models. The headline finding that all current VLMs, including Gemini 2.0 Flash and GPT-4o, perform around 50% on these tasks is credible and likely to be widely cited. The diagnostic claims, however, are only as strong as the assumption that per-position performance differences in the synthetic grids reflect content-dependent region utilization; the paper currently does not rule out vision-encoder positional or tiling artifacts. If the authors can close that gap with additional ablations, the diagnostic contribution would be a valuable tool for the field.","major_comments":[{"comment":"The lost-in-the-middle-Manhattan claim is not adequately separated from vision-encoder spatial priors. The supplementary ablation only rules out linear patch-order distance; it does not control for 2D positional-embedding geometry, row/column main effects, distance to image border, tiling boundaries, or the effect of reversing the visual token sequence. Because the needle is a low-resolution VQAv2 image re-encoded inside a synthetic grid by each model's vision transformer, per-position accuracy differences could arise from the encoder's positional geometry rather than from content-dependent attention to image regions. Since the paper explicitly claims this phenomenon is novel and distinct from text NIAH, this confound is load-bearing for the diagnostic contribution. Please add ablations such as random permutation of grid cells, reversed or shuffled visual token order, or regression on row, column, and distance-to-border to establish that the U-shape reflects region utilization rather than encoder artifacts.","section":"§4.3 and Supplementary §8"},{"comment":"Regional Divergence is defined as the difference between the best region's accuracy and the mean accuracy across regions. With roughly 500 samples split across up to 100 grid regions on the 10×10 setting, per-region estimates are based on about five samples each, so the reported divergence values (e.g., Gemini-2.0-Flash at 39.85%) may be dominated by sampling noise. The paper should report confidence intervals, bootstrap estimates, or significance tests to establish that the divergence is not an artifact of small per-region sample sizes.","section":"§4.3, Table 4"},{"comment":"The benchmark reuses datasets that are standard VLM training corpora (VQAv2, InfographicVQA, DocStruct4M, NovaChart) without a contamination analysis. Since HRScene's purpose is to measure current capability gaps, the paper should test for overlap with common training data or explicitly discuss how contamination would affect the headline numbers. This concern is especially relevant for the WhiteBackground diagnostic, which is constructed directly from VQAv2 and inherits any question-answer memorization present in the evaluated models.","section":"§3.2, §3.4, Table 1"}],"minor_comments":[{"comment":"The human performance for Medical is 23.81%, only 1.5 points above random; the annotators are graduate students rather than domain experts, so the 'human' row should be labeled as non-expert human performance or accompanied by a caveat for expert-dependent categories.","section":"Table 3"},{"comment":"There are recurring typos in model names and metrics: 'exact math' should be 'exact match', 'Calude' should be 'Claude', and 'molMo' appears for 'Molmo'.","section":"§4.1 and throughout"},{"comment":"The sentence 'no significant patter can be observed' contains a typo ('patter' for 'pattern'), and the figure caption should state the number of models and samples used for the patch-order ablation, which is currently missing.","section":"Supplementary §8"},{"comment":"The x-axis label '1k 2k 3k 4k 8k 5×10' is unclear; '5×10' appears to denote 5×10^6 pixels but this is not defined in the caption.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical contribution on the real-world side, but the synthetic diagnostic section overclaims given the confounds that remain untested. The revision path is clear: add the ablations suggested in Major Comment 1, quantify sampling noise in the Regional Divergence metric, and either perform or transparently discuss contamination analysis. If the authors can supply those analyses, the paper would be a solid benchmark contribution. I would not reject, because the real-world findings are valuable and the diagnostic claims are fixable in scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HRScene is a well-executed consolidation of 25 existing high-resolution image benchmarks into one taxonomy, with fresh re-annotation, human baselines, and a broad evaluation of 28 VLMs. The headline that average accuracy sits around 50% is supported by the tables and will be a useful reference. The comparison between Qwen2-VL and GPT-4o on HRVQA vs Galaxy is a nice example of how the benchmark can expose resolution handling differences. The benchmark design—splits, leaderboard plan, easy-to-use format—is thoughtful.\n\nThe soft spots are mostly in the diagnostic half. Contamination is a real risk because many source datasets (VQAv2, InfographicVQA, DocStruct4M, NovaChart) are common training corpora. The paper doesn't test for overlap, which weakens the absolute accuracy claims but doesn't break relative model comparisons. Also, human baseline performance is at chance on some expert tasks like LungHist (14% vs 14.29% random), which suggests those annotations are either ambiguous or too hard; that deserves a closer look.\n\nThe bigger problem is the Manhattan lost-in-the-middle finding. The synthetic diagnostics place a low-resolution VQAv2 needle into a large grid canvas, and the per-position accuracy shows a U-shape with Manhattan distance. But every evaluated VLM re-encodes the entire canvas through a vision transformer with learned 2D positional embeddings, resizing, tiling, and padding. The U-shape could easily come from those encoder artifacts rather than from a content-dependent failure to utilize middle regions. The supplementary ablation only tests linear patch-order and finds no pattern; it doesn't control for Euclidean distance to center, row/column main effects, border distance, or tiling boundaries. That's not enough to support the claim that this is a distinct 'lost-in-the-middle-Manhattan' phenomenon specific to HRI. I'd want to see either additional controls or a substantially more cautious interpretation.\n\nThe real-world benchmark results stand on their own. For that part, I'd recommend this paper for peer review. The diagnostic sections should be revised—either by running the extra controls or by reframing the findings as observations about current VLM behavior on grid-based canvases, without the mechanistic claim. This is a fixable weakness, not a fatal one.\n\nI'd cite the benchmark for its real-world results and likely bring it to our reading group.","headline":"A useful real-world high-resolution VLM benchmark, but the diagnostic 'Manhattan lost-in-the-middle' finding is underdetermined until 2D positional-embedding artifacts are ruled out.","tokens_in":28593,"tokens_out":3598,"would_cite":true,"duration_ms":35578,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HRScene, a new benchmark spanning 25 real-world and two synthetic high-resolution image tasks, finds that current vision-language models average about 50% accuracy and show a U-shaped 'lost-in-the-middle' pattern in Manhattan distance…","keywords":["high-resolution image understanding","vision-language models","benchmark","needle-in-a-haystack","lost-in-the-middle","regional divergence","VLM evaluation","image-grid diagnostics"],"falsifier":"A control experiment that embeds the same needle images into a continuous real high-resolution scene instead of a white or similar-image grid, then measures accuracy against the ground-truth pixel position of the target, would settle whether the Manhattan-distance U-shape is a general property of high-resolution perception or an artifact of grid composition; the paper's supplementary ablation rules out only the linear patch-order alternative.","tokens_in":27542,"feed_emoji":"🖼️","tokens_out":7814,"duration_ms":67623,"temperature":0.7,"pith_summary":"This paper introduces HRScene, a benchmark built to measure whether vision-language models actually understand high-resolution images rather than merely accepting them as input. It assembles 25 real-world tasks drawn from microscopy, radiology, street views, remote sensing, documents, art, and telescope imagery, and adds two synthetic needle-in-a-haystack tests that place a target image at different positions in a grid. Across the 28 models tested, average accuracy on the real-world tasks is about 50%, well below the human annotators' score on the same validation items. On the synthetic tests, accuracy falls as the grid grows and as the target moves toward the middle, producing a U-shaped curve in Manhattan distance from the top-left corner. The authors' central claim is that current models process high-resolution inputs unevenly, with systematic regional blind spots that a good benchmark can expose and quantify.","feed_headline":"VLMs hit only ~50% on new high-resolution benchmark","feed_subtitle":"Across 25 real-world scenes and synthetic grids, models show regional blind spots and a U-shaped loss of accuracy.","key_machinery":"The load-bearing mechanism is the pair of synthetic diagnostic constructions built on the needle-in-a-haystack idea. WhiteBackground places a small natural image with its question as the needle in an $N \\times N$ grid of identical white cells and quantifies Regional Divergence as the difference between the best-performing region and the mean over regions. ComplexGrid places the needle among visually similar distractor images and asks the model to output the needle's row and column, with accuracy plotted against Manhattan distance $\\lvert x-1\\rvert + \\lvert y-1\\rvert$ from the upper-left cell at row 1, column 1. The real-world arm supplies the accuracy baseline and the scale range: 25 curated tasks, partly re-annotated by graduate annotators, spanning eight categories with images from 1,024 × 1,024 up to 35,503 × 26,627 pixels.","core_discovery":"The central discovery is that current vision-language models do not yet understand high-resolution images in a spatially uniform way. On HRScene's real-world multiple-choice tasks, the average accuracy of all 28 evaluated models is 49.68%, and the best model reaches only about 62%. The two diagnostic datasets locate the failure: in WhiteBackground, a single natural-image needle is pasted onto white $N \\times N$ grids of increasing size, and accuracy drops while the gap between the best and average region grows; in ComplexGrid, the needle sits among visually similar distractors and models must report its row and column, and accuracy follows a U-shape in Manhattan distance to the top-left corner rather than in linear patch order. The paper argues that this is a distinct spatial version of lost-in-the-middle for high-resolution images, not the linear-context phenomenon, and that the position-dependent drop is robust across model families and sizes.","pith_inferences":["The paper does not test whether the two diagnostic patterns come from the vision encoder's positional bias rather than from the language model's spatial reasoning; a control that moves a real object within one continuous high-resolution scene would separate 'regional utilization failure' from 'collage layout artifact.'","If the U-shape reflects distance from the first visual patch, reordering how an image is tiled (for example, center-out or serpentine scanning) could be a cheap, training-free fix for middle-region accuracy; the paper explores no reordering.","Exact-match scoring on multiple-choice options may hide partial knowledge; a likelihood-based or open-ended scoring variant could change model rankings and give a finer signal for small improvements.","Because the synthetic diagnostics reuse a single source of needle images, porting the same construction to radiology, remote sensing, and document domains would show whether regional divergence is content-dependent or a general property of current VLMs."],"forward_implications":["If HRScene is taken as the measurement, no current VLM is close to reliable high-resolution understanding: average accuracy near 50% versus roughly 65% for the human annotators.","Native-resolution input handling is a visible lever: the best-performing model is the only one above 60% overall and is especially strong on very large images, indicating that resolution-adaptive architectures matter as much as parameter count.","Regional divergence means that where the answer sits in a high-resolution image materially changes a model's measured ability, so position-uncontrolled benchmarks can mislead.","The U-shaped Manhattan-distance curve gives a concrete target: reducing the middle-region drop should improve accuracy on grid-like and multi-subimage inputs.","Model-size scaling improves accuracy only logarithmically, so simply making models bigger is unlikely to close the high-resolution gap."],"supporting_citations":[{"why":"Defines the lost-in-the-middle phenomenon for long text that the two diagnostic datasets adapt to image grids.","marker":"[48]"},{"why":"Supplies the image-question pairs used as the needle in both synthetic diagnostics.","marker":"[23]"},{"why":"Provides several real-world high-resolution tasks and the main existing benchmark comparison point.","marker":"[79]"},{"why":"The prior multimodal needle-in-a-haystack work that this paper extends from mixed text-image input to a single grid image.","marker":"[67]"},{"why":"Source of multi-image long-context tasks reused in the benchmark's multi-subimage category.","marker":"[58]"},{"why":"The vision-language model family whose native-resolution design becomes the best performer and motivates the resolution analysis.","marker":"[63]"},{"why":"The image-similarity model used to select visually similar distractor images for ComplexGrid.","marker":"[41]"},{"why":"Proprietary model family evaluated in the benchmark; also motivates the high-resolution context discussion.","marker":"[60]"}],"fun_headline_variants":["VLMs hit ~50% on HRScene, expose high-res blind spots","New high-res benchmark: VLMs average 50%, miss grid needles","HRScene shows VLMs struggle with high-res spatial regions","VLMs lose the needle in high-res grids, says HRScene","High-res VLM benchmark reveals regional divergence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The diagnostic tests assume that a small natural image pasted onto a white or similar-image grid faithfully represents how a model uses regions of a genuinely high-resolution real scene, and that the resulting position-dependent accuracy reflects HRI utilization rather than an artifact of the collage layout.","fun_headline_variants_meta":{"raw":{"variants":["VLMs hit ~50% on HRScene, expose high-res blind spots","New high-res benchmark: VLMs average 50%, miss grid needles","HRScene shows VLMs struggle with high-res spatial regions","VLMs lose the needle in high-res grids, says HRScene","High-res VLM benchmark reveals regional divergence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000253,"raw_usage":{"total_tokens":1599,"prompt_tokens":1015,"completion_tokens":584,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":497}},"tokens_in":631,"tokens_out":584,"duration_ms":6030,"temperature":1.0,"reasoning_tokens":497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:16:46.342693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A control experiment that embeds the same needle images into a continuous real high-resolution scene instead of a white or similar-image grid, then measures accuracy against the ground-truth pixel position of the target, would settle whether the Manhattan-distance U-shape is a general property of high-resolution perception or an artifact of grid composition; the paper's supplementary ablation rules out only the linear patch-order alternative.","supporting_citations":[{"cited_title":"Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang","cited_arxiv_id":null,"evidence_quote":"Defines the lost-in-the-middle phenomenon for long text that the two diagnostic datasets adapt to image grids."},{"cited_title":"Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing","cited_arxiv_id":null,"evidence_quote":"Supplies the image-question pairs used as the needle in both synthetic diagnostics."},{"cited_title":"Needle in a multimodal haystack","cited_arxiv_id":null,"evidence_quote":"The prior multimodal needle-in-a-haystack work that this paper extends from mixed text-image input to a single grid image."},{"cited_title":"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022","cited_arxiv_id":null,"evidence_quote":"The image-similarity model used to select visually similar distractor images for ComplexGrid."}],"review_version":1}