{"id":"d8918af3-b8b5-43ac-808b-c5c9c0e69ccd","arxiv_id":"2412.13614","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Introduces PL-VEL, a pixel-mask-based visual entity linking task, and MaskOVEN-Wiki, a 5.2M-annotation dataset built via reverse annotation, plus a semantic tokenization method that yields a 5-point accuracy gain.","lead":"This paper proposes a new visual entity linking task that identifies objects via pixel masks instead of text queries, and releases a 5.2-million-annotation dataset to support it. If the dataset's claimed quality holds, it could make object identification more practical for interactive and fine-grained visual systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 94.8% annotation-success estimate is entity-balanced (one sample per entity), not example-weighted; the 5.2M-annotation dataset has ~560 examples per entity on average, so the headline rate may not measure the released dataset's per-annotation quality.","rationale":"The paper's contribution is a new dataset and a new task; the 25.2% end-model accuracy is a demonstration that the dataset is usable, but the dataset's value depends on annotation quality. I agree with the reader that the 2,000-sample evaluation is thin, but the sharper problem is the mismatch between the sampling unit and the inference target. The manual evaluation is entity-balanced, while the dataset claim and downstream training are example-weighted. This is not an internal inconsistency; it is an unverified extrapolation. A rare-entity sample gives each entity one vote, whereas the 5.2M annotations give an entity with 1,000 images 1,000 votes. If segmentation success is higher for frequent, visually stereotyped entities (e.g., common plants and animals) and lower for long-tail or scene-level entities, the entity-balanced 94.8% could overstate per-annotation quality. Conversely, if high-frequency entities are easier and dominate the dataset, the example-weighted rate could be even higher than 94.8%; the current report simply does not distinguish these cases. The fix is straightforward and does not require re-running training: resample the existing dataset proportionally and re-review. I also credit the paper for making the dataset URL public and for comparing against strong text-based baselines; the concern is about the load-bearing quality estimate, not the task formulation or the architecture. The verdict should remain conditional pending the example-weighted quality audit.","tokens_in":16064,"tokens_out":8733,"duration_ms":77409,"concrete_test":"Draw a fresh sample of 2,000 annotations uniformly from the final 5.2M dataset, with sample sizes per split proportional to annotation counts (Entity Split should dominate), and re-run the manual review using the same published rubric or, better, a pre-registered rubric scored by two independent annotators. Report example-weighted accuracy, per-split accuracy, accuracy by entity-frequency deciles, and a 95% confidence interval. If example-weighted accuracy is within about 1 point of 94.8%, the current claim stands; if it falls below 90%, the dataset-quality claim and any conclusions about training on MaskOVEN-Wiki need to be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central artifact is MaskOVEN-Wiki, and the central quality claim is that reverse annotation achieves 94.8% success on the 5.2M annotations. The supporting evidence in Table 1 and the Appendix (Table 6) is a 2,000-item manual evaluation in which each entity contributes at most one sample: 1,400 Entity Split, 400 Query Split, 200 Wiki Split, with Entity Num equal to Case Num. That is an entity-balanced estimate. The released dataset is not entity-balanced: Table 2 lists 4,464,748 Entity-Split training examples over 7,943 seen entities, roughly 560 examples per entity, and the full 5.2M total is concentrated in the Entity Split. Example-weighted annotation accuracy can differ from entity-balanced accuracy whenever annotation success correlates with entity frequency or with the entity types that dominate the example count. The paper reports no per-split accuracy, no accuracy by entity-frequency stratum, and no confidence interval, so it does not establish that the 94.8% rate applies to the majority of the 5.2M annotations. The additional filtering of location, building, and sports entities makes the remaining distribution even more important to characterize, because the entity-balanced evaluation set may not reflect the actual frequencies in the published dataset.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Pixel-Level Visual Entity Linking (PL-VEL), where an object in an image is referred to by a pixel mask rather than a text query, and the goal is to link that visual mention to an entity in a large knowledge base. To support this task, the authors construct MaskOVEN-Wiki by reversing the direction of the existing OVEN-Wiki dataset: given an entity and a text query, they prompt open-vocabulary segmentation models (Grounded-SAM and SEEM) with knowledge-augmented textual references, then apply heuristic filtering and correction rules. They report a 94.8% annotation-success rate based on a 2,000-item manual evaluation and a final dataset with 5,245,421 annotations. They also propose a visual semantic tokenization scheme that adds FastSAM region features (sorted by area) to the patch features of an Osprey-based MLLM, and they report that fine-tuning on MaskOVEN-Wiki raises test accuracy from 1.3% (zero-shot) to 25.2%, with semantic tokenization contributing about 5 points.","tokens_in":16284,"tokens_out":8912,"duration_ms":76975,"significance":"If the quality estimate is reliable, this is a useful contribution: the paper proposes a new task, releases a large-scale pixel-level entity linking dataset, and provides a baseline that shows mask-based references can approach text-based VEL performance. The reverse annotation rationale is coherent and avoids the prohibitive 6M-entity search space of direct recognition. The paper is also transparent about the main filtering categories and makes code and data available. The central weakness is that the headline 94.8% annotation-quality figure is not yet established for the actual example-weighted dataset, and the downstream model comparisons inherit this uncertainty. The paper does not appear to contain circular derivations; training uses external labels from OVEN-Wiki and a separate evaluation split.","major_comments":[{"comment":"The 94.8% annotation-success estimate is entity-balanced, not example-weighted. Table 6 shows that in the manual evaluation every entity contributes exactly one sample (Case Num equals Entity Num in each split), while Table 2 lists 4,464,748 Entity-Split training examples over 7,943 seen entities, i.e., roughly 560 examples per entity on average. If annotation success correlates with entity frequency or with the entity types that dominate the example count, the entity-balanced rate need not equal the per-annotation quality of the released 5.2M masks. Please report accuracy separately for each split, by entity-frequency stratum, and with confidence intervals; ideally, re-run the manual evaluation with example-weighted sampling.","section":"Annotation Quality / Table 6"},{"comment":"The paper does not disclose the rubric used in the manual evaluation. It is not stated what counts as a successful annotation: e.g., whether the mask must reach an IoU threshold with the target object, whether partial or occluded objects are accepted, or how cases where the entity is absent from the image are coded. No inter-annotator agreement measure is provided, and the single 'overall after filtering' number in Table 1 has no confidence interval. Without this operationalization, the headline quality claim cannot be independently checked.","section":"Annotation Quality / Definition of success"},{"comment":"The heuristic filtering rules induce a selection bias that is acknowledged qualitatively but not quantified. The Data Filtering section states that location, building, and sports entities have the highest proportion of unannotated entities, and the Appendix reports that the rules exclude 124,896 annotations in the Entity Split, 7,920 in the Query Split, and 176 in the Human Set. This changes the category composition of the final dataset relative to OVEN-Wiki. Moreover, the manual evaluation distribution in Figure 4 differs substantially from the final dataset distribution in Figure 5(a) for location (18.8% vs. 6.2%) and building (12.9% vs. 4.9%), so the evaluation set is not representative of the published data. Please report the category distribution before and after filtering and give per-category annotation accuracy, especially for the categories most affected by the filters.","section":"Data Filtering / Figure 5(b)"},{"comment":"The main text says visual semantic tokenization improves overall performance by 3.4% on the validation set and 5.2% on the test set, but the cited tables show a validation improvement of 3.7 points without pre-training (+FT+Seg vs. +FT) and 5.0 points with pre-training (+PT+FT+Seg vs. +PT+FT), and a test improvement of 5.2 points. The reported 3.4% does not match any number in the tables. This discrepancy should be corrected and the exact comparison used for the headline '5-point improvement' claim should be stated explicitly.","section":"Experiments / Table 3 and Table 4"}],"minor_comments":[{"comment":"The sentence 'Osprey-7B-Seg improves the overall performance by 3.4% on the validation set' is inconsistent with the numbers in Table 3 and Table 4; please reconcile the figure or the tables.","section":"Main Results"},{"comment":"Figure 5(b) contains a file path and timestamp ('2024/6/11 12:14 Awesome-pyecharts file:///Users/...') and appears to be a screenshot; it should be redrawn as a clean vector figure with labeled axes.","section":"Figure 5(b)"},{"comment":"The Figure 3 caption says the illustration image was generated by AI and cites (Chang et al. 2024), but that reference is a paper on GPU kernel fusion (FLUX), not an image-generation model; the citation is incorrect.","section":"Figure 3 caption"},{"comment":"The symbols in Table 1, such as '#' in '# entity label' and '# text query', are not defined, and the row 'overall after filtering' reports a single aggregate without explaining how the aggregate is computed across the settings and splits.","section":"Table 1"},{"comment":"Table 2 is difficult to read: the 'Human Set' label appears in the header but the human-set statistics are not clearly separated from the other columns; please align the column structure or add a note describing which splits are included.","section":"Table 2"},{"comment":"Equation (3) is referenced in the text but the displayed equation for the region feature set appears without a visible equation number; please check all equation numbering.","section":"Equation (3)"}],"recommendation":"major_revision","confidential_remarks":"The central artifact is the dataset, so the example-weighted quality estimate is the load-bearing claim. If the authors can provide an example-weighted manual evaluation or per-frequency-stratum accuracy, a clear rubric, and a quantification of the filtering bias, the paper could be acceptable. I do not see evidence of circularity or a fundamental flaw in the reverse annotation idea; the issues are about evidence quality rather than feasibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the headline: this is a genuine contribution. PL-VEL (pixel masks instead of text queries for visual entity linking) is a sensible gap, and MaskOVEN-Wiki is the first large-scale dataset for it. Reverse annotation—going from entity labels to masks rather than masks to entities—is the right call: it shrinks the search space from millions of entities to a few image regions, and the knowledge-augmented prompts (intension + extension) are a reasonable way to help open-vocabulary segmenters. The numbers support usefulness: Osprey-7B goes from 1.3% zero-shot to 20% after fine-tuning on the dataset, and their semantic tokenization adds another 5 points. Those are real gains, and the paper is candid about the absolute accuracy being only ~25%.\n\nThe soft spot is the quality estimate. The 94.8% comes from a 2,000-sample manual inspection with no disclosed rubric, no confidence intervals, and—as you noted—each entity appears at most once in the sample. That gives an entity-balanced success rate, not an example-weighted per-annotation rate. The released Entity Split has roughly 560 examples per entity on average, so the two rates can differ if annotation success correlates with entity frequency or type. The filtering rules also deliberately drop location, building, and sports entities, and Figure 4 shows the eval set is 'similar but more balanced'—which means it isn't actually representative of the dataset's final distribution. This is fixable: report example-weighted accuracy, break it down by split and entity-frequency stratum, and give the annotation rubric plus confidence intervals.\n\nOne more minor thing: the model improvements (semantic tokenization, pre-training) lack statistical testing, but the deltas are consistent across validation and test, so it is not a dealbreaker.\n\nIf the quality measurement is tightened, this paper deserves publication. The task is new, the dataset is a resource people will use, and the method is a solid extension of Osprey. The repo exists (I did not try to reproduce anything), so the code/data commitment looks real.\n\nMy recommendation: send it to peer review, not a desk reject. Reviewers should push for an example-weighted quality estimate and a more transparent manual evaluation.","headline":"Real new task and a substantial dataset, but the 94.8% quality headline is an entity-balanced estimate that needs to be re-checked example-weighted before it is trusted.","tokens_in":16861,"tokens_out":3153,"would_cite":true,"duration_ms":29022,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that pixel-level visual entity linking becomes tractable by reversing the annotation direction: start from known entity labels, segment their masks, and train on the resulting 5.2M-annotation dataset to reach 25.2%…","keywords":["visual entity linking","pixel-level visual entity linking","reverse annotation","MaskOVEN-Wiki dataset","visual semantic tokenization","region-interacted attention","open-vocabulary segmentation","multimodal large language models"],"falsifier":"Take a fresh random sample of about 1,000 annotations from the full MaskOVEN-Wiki, stratified by entity category, and have independent annotators check mask-region alignment under a pre-registered rubric; or, alternatively, have humans draw masks for a few hundred of the same (image, entity) pairs and compare how a model fine-tuned on MaskOVEN-Wiki performs with those human masks versus the pipeline masks. If the category-stratified accuracy falls well below 94.8%, or accuracy drops sharply with human-drawn masks, the central claim that reverse annotation produces a high-quality large-scale data resource would be weakened.","tokens_in":15867,"feed_emoji":"🖼️","tokens_out":8734,"duration_ms":69694,"temperature":0.7,"pith_summary":"The paper introduces Pixel-Level Visual Entity Linking (PL-VEL): given an image and a pixel mask that points at one object, the model must name that object's entity among millions of Wikipedia entries. The central claim is that building training data for this task is feasible through reverse annotation: instead of asking a model to identify a segmented region, start from entity labels already paired with images in the OVEN-Wiki benchmark and use open-vocabulary segmentation models to draw each entity's mask. Knowledge-augmented text references (Wikidata hypernyms plus GPT-3.5-derived referring expressions), a model ensemble, and heuristic filtering raise annotation accuracy to 94.8% on a 2,000-sample manual check, yielding the 5.2M-annotation MaskOVEN-Wiki dataset. A model fine-tuned on this dataset—with a new visual semantic tokenization that turns segment-everything regions into tokens—improves linking accuracy from 1.3% (zero-shot) to 25.2%, about 5 points of which come from the tokenization. The upshot is that pixel masks offer a more practical reference than elaborate text queries for fine-grained visual understanding.","feed_headline":"Mask-to-entity linking jumps from 1.3% to 25.2%","feed_subtitle":"Reverse annotation turns known entity labels into 5.2M pixel-mask training pairs.","key_machinery":"The load-bearing mechanism is the reverse annotation framework. Given a triple (entity, image, text query) from OVEN-Wiki, it constructs an enriched text reference: intensional expansion adds Wikidata super-categories (instance-of and subclass-of) to the entity label, and extensional expansion uses GPT-3.5 to extract the query's spatial or relational referring expressions. This reference is fed to two open-vocabulary segmentation models—the pipeline Grounded-SAM (Grounding-DINO plus SAM) and the end-to-end SEEM—whose mask proposals are then corrected by heuristic rules: drop non-visual entity types, treat location and building entities as whole-image masks, use IOU agreement to catch pipeline error propagation, and apply morphological erosion/dilation followed by connected-component analysis to fix foreground-background confusion in dense scenes. On the modeling side, the second load-bearing piece is visual semantic tokenization: FastSAM runs 'segment everything' on the image; Osprey's mask-aware visual extractor encodes each resulting region into a semantic and a positional embedding; the region tokens are sorted by descending area (simulating coarse-to-fine human attention) and concatenated with the patch features of the vision encoder before entering the LLM. The model generates a 4-token ALD code for the target entity, following the generative entity-recognition framework of GER-ALD, with Vicuna as the language backbone fine-tuned by LoRA.","core_discovery":"On the paper's own terms, the discovery is an asymmetry: direct region-to-entity annotation is hard (even GPT-4V reaches only about 25.5% accuracy on the OVEN-Wiki labeling task), while the reverse direction—entity-to-region segmentation—is tractable (about 81% with a plain Grounded-SAM pipeline, raised to 94.8% with knowledge augmentation, ensembling, and filtering). Exploiting that asymmetry yields a dataset of 5,245,421 masks aligned with 20,077 entities across 5,214,965 images, drawn from the existing OVEN-Wiki entities, queries, and Wikipedia-image splits. The paper further claims that training on this data works: Osprey-7B goes from 1.3% zero-shot accuracy to 20.0% after fine-tuning, and adding their visual semantic tokenization—which replaces fixed-size patches with semantic region tokens produced by FastSAM—reaches 25.2% overall accuracy on the test set. The authors present PL-VEL as a task definition, a dataset, and a baseline, arguing that pixel-mask references supplement text-based visual entity linking and enable finer-grained visual understanding in multimodal LLMs.","pith_inferences":["The entity-to-region asymmetry suggests a general recipe for open-vocabulary annotation: when the label space is millions of classes, generating the visual evidence conditioned on a known label is far easier than classifying an unlabeled region; the same trick could apply to building data for other fine-grained recognition tasks.","The reported 94.8% annotation success rests on a 2,000-sample check that is balanced by entity rather than by image, and the filtering rules deliberately drop location, building, and sports entities; an independent, category-stratified re-estimate would be needed to confirm the figure holds across the full 5.2M dataset.","Because the final accuracy is only about 25%, the masks themselves may carry a systematic bias (easy entities, filtered hard categories), meaning gains measured on MaskOVEN-Wiki might not transfer to arbitrary user-provided masks; a small evaluation with human-drawn masks on the same images would test transfer.","Visual semantic tokenization is a portable idea: segment-everything regions could replace fixed patches in any region-grounded multimodal LLM task—region captioning, referring expression comprehension, VQA—so its 5-point gain here is a plausible lower bound for other fine-grained tasks."],"forward_implications":["PL-VEL becomes a new benchmark: the MaskOVEN-Wiki dataset supplies train/validation/test splits in entity, query, and wiki variants plus a human-evaluated set, all with pixel-mask references.","Pixel masks are a practical stand-in for text queries: mask-based linking reaches accuracy comparable to strong text-based models (25.2% vs. 25.5% for GPT-4V) and outperforms box references (1.6%), suggesting masks carry richer spatial information.","Fine-tuning on MaskOVEN-Wiki is what turns a zero-shot model into a functional linker—1.3% to 20.0%—so the dataset, not the model architecture alone, drives the gain.","The reverse-annotation recipe generalizes beyond this dataset: knowledge augmentation plus model ensembling raised annotation accuracy from 81% to 94.8%, a pipeline that can be applied to other region-labeling efforts.","Retrieval augmentation is an open direction: a retrieval-free text VEL model performs within 0.5 points of the proposed model, implying that adding retrieval-based entity search to PL-VEL could close much of the remaining gap."],"supporting_citations":[{"why":"Supplies the OVEN-Wiki entity-image-text-query triples and the 6M-entity Wikipedia knowledge base that MaskOVEN-Wiki is built upon.","marker":"(Hu et al. 2023)"},{"why":"Grounded-SAM is the primary segmentation annotator; its 81.4% accuracy on plain references is the baseline the reverse pipeline improves.","marker":"(Ren et al. 2024)"},{"why":"SEEM provides the end-to-end complementary segmentation used for ensembling and for correcting Grounded-SAM's pipeline error propagation.","marker":"(Zou et al. 2023)"},{"why":"Osprey is the pixel-level MLLM backbone and supplies the mask-aware visual extractor reused for both the model and the semantic tokenization.","marker":"(Yuan et al. 2023)"},{"why":"GER-ALD supplies the ALD-code generative entity-recognition framework that the model uses to decode target entities.","marker":"(Caron et al. 2024b)"},{"why":"Provides the direct-annotation baseline (GPT-4V at 25.5%) and the text-based AUTOVER results that frame the comparison against PL-VEL.","marker":"(Xiao et al. 2024)"},{"why":"FastSAM performs 'segment everything' and acts as the visual semantic tokenizer that produces the region tokens for region-interacted attention.","marker":"(Zhao et al. 2023)"},{"why":"SAM defines the mask-generation capability used inside Grounded-SAM and motivates pixel-mask references as the task's input format.","marker":"(Kirillov et al. 2023)"}],"fun_headline_variants":["Reverse annotation gives 5.2M mask-entity pairs, lifting accuracy to 25.2%","Reverse labeling hits 94.8% success, 5.2M masks, 25.2% accuracy","Turning entities to masks: 94.8% auto-labeling, 5.2M pairs, 25.2% score","Pixel-level VEL via reverse annotation: 5.2M masks, 25.2% accuracy","Reverse annotation: 94.8% success, then 5.2M pairs train Osprey to 25.2%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 94.8% annotation-success rate measured on 2,000 hand-checked samples represents the quality of the full 5.2M-annotation dataset; the heuristic filters that discard location, building, and sports entities are what keep the sample looking clean, and the validity of every downstream accuracy number depends on that small sample being representative.","fun_headline_variants_meta":{"raw":{"variants":["Reverse annotation gives 5.2M mask-entity pairs, lifting accuracy to 25.2%","Reverse labeling hits 94.8% success, 5.2M masks, 25.2% accuracy","Turning entities to masks: 94.8% auto-labeling, 5.2M pairs, 25.2% score","Pixel-level VEL via reverse annotation: 5.2M masks, 25.2% accuracy","Reverse annotation: 94.8% success, then 5.2M pairs train Osprey to 25.2%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000763,"raw_usage":{"total_tokens":3420,"prompt_tokens":1014,"completion_tokens":2406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2261}},"tokens_in":630,"tokens_out":2406,"duration_ms":14117,"temperature":1.0,"reasoning_tokens":2261,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:56:59.974834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh random sample of about 1,000 annotations from the full MaskOVEN-Wiki, stratified by entity category, and have independent annotators check mask-region alignment under a pre-registered rubric; or, alternatively, have humans draw masks for a few hundred of the same (image, entity) pairs and compare how a model fine-tuned on MaskOVEN-Wiki performs with those human masks versus the pipeline masks. If the category-stratified accuracy falls well below 94.8%, or accuracy drops sharply with human-drawn masks, the central claim that reverse annotation produces a high-quality large-scale data resource would be weakened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SEEM provides the end-to-end complementary segmentation used for ensembling and for correcting Grounded-SAM's pipeline error propagation."},{"cited_title":"Grounding Language Models for Visual Entity Recognition","cited_arxiv_id":"2402.18695","evidence_quote":"Provides the direct-annotation baseline (GPT-4V at 25.5%) and the text-based AUTOVER results that frame the comparison against PL-VEL."}],"review_version":1}