{"id":"7fa06407-220d-4440-b4bb-bd8019aa833a","arxiv_id":"2507.18300","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LMM-Det makes a 7B LMM detect objects on COCO at 47.5 AP, above prior LMM-based detectors (38.5 AP) but below specialist detectors (57.3 AP), using pseudo-label distillation and per-category inference.","lead":"A large multimodal model is trained to output object bounding boxes directly, reaching 47.5 AP on COCO without a detection-specific head, but with help from a specialist detector's pseudo-labels during training. The recipe uses a detection-pretrained vision backbone, per-category prompting at inference, and boosted recall to beat prior LMM detectors while still lagging specialist detectors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-category prompting turns multi-class COCO detection into 80 binary localization queries; the reported 47.5 AP is not earned under the standard detection protocol and is not directly comparable to specialist detectors.","rationale":"The strongest claim in the paper is that an LMM can do object detection without specialist modules. The evidence for this is ultimately a COCO AP number obtained by asking the model 80 separate questions, one per category. This protocol is a deliberate design choice (Section 3.3), not a bug, but it materially changes the task: the model is told which class to look for, so it never has to classify a box among competing categories. That makes the comparison against single-pass specialist detectors in Table 3, and the 'comparable performance' statement in Figure 1, misleading. The reader identified the fixed-known-category aspect as the weakest assumption; I agree and would sharpen it into a comparability concern. A single-pass evaluation would settle whether LMM-Det actually 'possesses detection capability' in the standard multi-class sense or only acts as 80 category-conditioned localizers. The secondary issues about Salience-DETR pseudo-labels and the OWLv2 backbone are real but less decisive: they concern how much of the capability is borrowed, while the per-category protocol concerns what capability is being measured. The paper is otherwise internally consistent and the experiments are limited by missing error bars, but not fatally flawed. The CONDITIONAL verdict from the reader is appropriate; the authors should add a single-pass result and temper the comparability claims.","tokens_in":17287,"tokens_out":8968,"duration_ms":88152,"concrete_test":"Run the released LMM-Det on COCO val with a single all-categories instruction (e.g., 'Detect all objects in the image and output each bounding box with its class'), using identical decoding and post-processing, and compute COCO AP. If single-pass AP is substantially below 47.5 (say, below 42), the per-category protocol accounts for a large share of the reported performance and the claim should be scoped accordingly. As a control, evaluate a specialist detector under the same 80-query protocol to quantify the protocol's benefit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 ('Inference Optimization') replaces single-pass multi-class detection with one query per category: 'Detect all the objects in the image that belong to the category set {cat}.' Each forward pass receives the target class as part of the prompt, so the model never performs the class-selection step that is essential to object detection; it only decides whether instances of that specific class are present and where they are. The reported 47.5 COCO AP (Table 3) is thus achieved under a protocol that decomposes one 80-class detection problem into 80 binary localization problems. Specialist detectors in Table 3 are evaluated in the standard single-pass setting, so the 'comparable to specialist detectors' framing in Figure 1 and the claim that 'a large multimodal model possesses detection capability' (Abstract) are not supported on equal terms. The model also depends on Salience-DETR pseudo-labels (Section 3.2) and the OWLv2 detection-pretrained encoder (Section 3.1, Table 5), which further qualify the 'no extra detection modules' claim, but the per-category protocol is the more load-bearing issue because it changes what is being measured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LMM-Det, a method that adapts a large multimodal model (OWLv2-ViT encoder, linear projector, Vicuna-7B) to perform COCO-style object detection without a dedicated detection head or region proposer. The authors identify low recall as the main bottleneck and address it with two techniques: data distribution adjustment (merging ground-truth labels with Salience-DETR pseudo-labels via NMS) and inference optimization (querying the model separately for each category). After Stage III fine-tuning, the method reports 47.5 AP on COCO val, and an optional Stage IV preserves some captioning and VQA ability. The paper claims that a large multimodal model can possess detection capability without extra detection modules.","tokens_in":17448,"tokens_out":5188,"duration_ms":56213,"significance":"If the central claim is accepted, the paper is a useful demonstration that an LMM can be turned into a competitive detector using only data and prompting changes, and the recall-oriented analysis is a valuable addition to the literature on LMMs and detection. The strengths include a clear three-stage training recipe, systematic ablations (Table 5), and released datasets, models, and code. However, the headline claim is materially qualified by the evaluation protocol and by the use of detection-pretrained components and specialist pseudo-labels, so the paper's significance depends on how these qualifications are resolved.","major_comments":[{"comment":"The inference protocol decomposes one 80-class detection problem into 80 per-category localization problems: each image is queried with 'Detect all the objects in the image that belong to the category set {cat}', and the model never performs class selection. The specialist detectors in Table 3 are evaluated in a single pass over all classes. The reported 47.5 AP is therefore not earned under the same task protocol and is not directly comparable to the specialist numbers in that table. The Abstract and Figure 1 should be reframed as category-conditioned localization, or the authors should provide a single-pass multi-class evaluation to support the 'vanilla object detection' claim.","section":"Section 3.3 / Section 4.1 / Table 3"},{"comment":"The claim of operating 'without any extra detection modules' is undercut by the choice of OWLv2-ViT as the visual encoder. OWLv2 is an open-vocabulary object detector, and its ViT backbone carries detection-oriented pretraining; Table 5 shows that swapping CLIP-ViT for OWLv2-ViT alone raises AP by 3.4 points (38.7 to 42.1). The phrase 'no extra detection modules' should be restricted to the absence of task-specific heads, proposers, or decoders, and the detection-pretrained encoder should be acknowledged as a form of specialist knowledge transfer.","section":"Section 3.1 / Table 5"},{"comment":"The data distribution adjustment trains the model to imitate pseudo-labels generated by Salience-DETR, a specialist detector, including the confidence scores assigned by that detector. The +2.1 AP attributed to DDA in Table 5 is therefore partly inherited from the specialist model that the paper compares against in Table 3. To support the claim that the LMM itself acquires detection capability, the paper should present a control with pseudo-labels from a different source or otherwise quantify how much of the gain is due to additional label coverage versus imitation of the specialist.","section":"Section 3.2 / Table 5"}],"minor_comments":[{"comment":"Appendix A states that 'we set a threshold for calculating AP and AR' but does not give the threshold value or explain whether this replaces the standard COCO AP sorting procedure. Please specify the exact evaluation protocol, since the reported AP values depend on it.","section":"Appendix A"},{"comment":"The evaluation protocol for LLaVA* is inconsistent: Section 2.1 and Appendix A say exploratory models output all bounding boxes simultaneously, while Figure 1 and its caption describe LLaVA* as being queried per class. Please clarify which protocol produced the 38.7 AP reported for LLaVA* in Table 3.","section":"Section 2.1 / Figure 1 / Table 3"},{"comment":"The term 'zero-shot' is used for LMM-Det after Stage II training on Object365 and for other models evaluated with their released checkpoints; this is misleading. Please relabel the setting as 'zero-shot on COCO' and explicitly note the different training-data exposures of the compared models.","section":"Table 2"},{"comment":"The claim that LMM-Det 'maintains' captioning and VQA abilities is overly strong: after Stage IV, CIDEr drops from 108.9 to 99.0 and VQAv2 accuracy from 78.5 to 74.1. Please qualify the wording to reflect a measurable degradation.","section":"Section 4.4 / Table 4"},{"comment":"There are several typographical issues, including 'LLM-Det' in Section 3.2, inconsistent 'LLaV A' spacing, and 'OVLv2' in Table 5's header. A proofreading pass is needed.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The per-category inference protocol is the most consequential issue: it changes what the reported 47.5 AP measures, and the current abstract overstates the claim. If the authors add a single-pass evaluation or clearly reposition the method as category-conditioned detection, the paper could be acceptable after revision. The use of OWLv2-ViT and Salience-DETR pseudo-labels should also be foregrounded in the abstract, as they materially affect the 'no extra detection modules' narrative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nLMM-Det is a training and inference recipe that pushes a 7B LMM to 47.5 AP on COCO val without a detection head at inference. That is a real result — the best for LMM-only detectors by a wide margin. The paper is worth reading for its recall analysis and for showing that pseudo-labels from a specialist detector plus a detection-pretrained encoder can close a large chunk of the gap to specialist detectors.\n\nWhat is actually new: the combination of Object365 pre-training, Salience-DETR pseudo-labels, per-category instruction data, and per-category prompting. Each piece is known, but the assembled recipe works and the ablations are informative. The zero-shot result (24.5 AP) also beats prior LMMs. They ship code, models, and data, so the work is reproducible.\n\nThe soft spots are proportional. The most serious is the evaluation protocol. At inference, the model is queried 80 times, once per COCO class, with the class name given in the prompt. That turns 80-class detection into a set of binary localization tasks; the model never has to decide which classes are present. The paper is transparent about this, but it means the comparison to Faster R-CNN, RT-DETR, and Salience-DETR in Table 3 is not on equal terms. Those detectors do multi-class detection in a single pass. Calling LMM-Det 'comparable to specialist detectors' oversells it. The model also leans on Salience-DETR pseudo-labels in training and uses OWLv2, a detection-pretrained encoder, which the ablation shows is worth 3.4 AP. The 'no extra detection modules' claim holds only for the final architecture, not the training process.\n\nThe backbone mismatch with prior LMMs (OWLv2 vs CLIP) weakens the SOTA comparison, though the ablation at least quantifies the gain. No error bars, which is minor. The 4-second per-image latency is disclosed and acknowledged as a limitation.\n\nOverall, the central result is plausible and useful, but the framing overreaches. It is a solid empirical paper with a fixable evaluation gap, not a dishonest one. I would send it to peer review with a request for a standard single-pass baseline, a backbone-matched comparison to prior LMMs, and a more careful statement of what the per-category protocol does and does not demonstrate.","headline":"A solid empirical recipe for LMM-based detection that sets a new SOTA, but the per-category inference protocol changes the task and the 'no extra detection modules' claim is nested inside specialist-pretrained components.","tokens_in":18095,"tokens_out":3340,"would_cite":true,"duration_ms":34514,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A large multimodal model, with only a vision encoder, a projector, and a language model, can perform object detection at 47.5 AP on COCO without any specialist detection module.","keywords":["object detection","large multimodal models","recall rate","per-category prompting","pseudo-labeling","instruction tuning","COCO","zero-shot detection"],"falsifier":"Run LMM-Det on COCO val with the identical training recipe but replace the per-category prompts with a single all-objects prompt; if AR@100 stays near 63.6 rather than falling toward the roughly 27.5 the authors attribute to a single-pass model, the recall mechanism claimed for inference optimization would be refuted. A second check is to evaluate without retraining on Object365's 365 categories using the same per-category prompts; if AR@100 collapses, the fixed-vocabulary assumption is confirmed as load-bearing.","tokens_in":17007,"feed_emoji":"🎯","tokens_out":6706,"duration_ms":64334,"temperature":0.7,"pith_summary":"This paper seeks to show that a large multimodal model, composed only of a vision encoder, a projector, and a large language model, can do vanilla object detection without any specialist detection module bolted on. The authors identify the main bottleneck as a low recall rate: the model produces far fewer usable boxes than specialist detectors. They attack that bottleneck with two moves: adjusting the training data distribution by merging pseudo-labels from a pretrained specialist detector with ground truth via non-maximum suppression, and an inference strategy that queries the model once per object category instead of asking for all boxes at once. On COCO val, the resulting LMM-Det reaches 47.5 AP, 66.5 AP50, and 63.6 AR@100 after fine-tuning, while preserving captioning and VQA ability in the four-stage variant. The practical stake is that a single generalist model could replace the detector-plus-LMM stacks currently used for grounded dialogue, robotics, and scene understanding.","feed_headline":"No extra detector: vision-language model hits 47.5 AP on COCO","feed_subtitle":"Per-category prompts and pseudo-label merging fix the low recall that held multimodal models back.","key_machinery":"The mechanism is the combination of a high-resolution open-vocabulary visual encoder (OWLv2-ViT, 1008×1008 input, with visual tokens fed uncompressed), a linear projector, and a 7B language model; data distribution adjustment, in which a pretrained specialist detector generates pseudo-labels that are merged with COCO and Object365 ground truth by NMS, and the model is trained to emit each box's coordinates followed by a confidence score token; and inference optimization, where detection is decomposed into one autoregressive pass per category, each pass producing boxes for that single class, repeated over the 80 COCO categories. The reorganized multi-turn instruction data, with positive and negative category questions in a 1:1 ratio and per-turn and per-box randomization, keeps training consistent with this per-category inference. The load-bearing object is the per-category prompt: it raises recall by letting each decoding step focus on one class, which the authors show yields more and better proposals than a single all-objects query.","core_discovery":"The paper's central claim is that a large multimodal model possesses detection capability without any extra detection modules. Concretely, LMM-Det, trained from an OWLv2-ViT visual encoder, a linear projector, and a 7B language model, outputs object coordinates and class-specific confidence scores as plain text and reaches 47.5 AP, 66.5 AP50, and 63.6 AR@100 on COCO val after fine-tuning, compared with 38.7 AP for a re-trained LLaVA baseline and 55-57 AP for strong specialist detectors. The authors argue that the gap to specialists is caused mainly by insufficient recall, and they show that two targeted interventions, data distribution adjustment via pseudo-label merging and per-category inference prompting, raise AR@100 from 50.5 to 63.6 and AP from 38.7 to 47.5. They also show a variant trained on the standard 665K LLaVA instruction set keeps most of the detection gain (47.1 AP) while retaining image captioning and VQA performance.","pith_inferences":["The fixed-category assumption is the main boundary: at inference the model is asked one question per category, so on open-vocabulary benchmarks such as LVIS or on a very large label set the per-category strategy would multiply cost or require a different aggregation; the reported 47.5 AP should not be expected to transfer unchanged.","Because pseudo-labels come from a single specialist detector, the ceiling of the data distribution adjustment is partly set by that detector's own recall; a weaker pseudo-labeler would likely produce a smaller gain, which could be tested by swapping pseudo-label sources.","One testable extension is to cluster or batch categories per pass, for example asking for 'person, dog, handbag' in a single prompt, to cut the 80-fold inference cost while retaining most of the recall benefit, since the paper's argument only requires per-category focus, not literally one category.","The per-category design may interact with model scale and resolution; the authors use a 7B language model and 1008×1008 input, so it is an open question whether larger models or higher resolutions would narrow or widen the gap to specialist detectors."],"forward_implications":["Specialist detection modules, such as region proposal networks, Grounding DINO, or other bolt-on detectors, are not required for an LMM to do vanilla object detection; LMM-Det's gains come from data and inference changes, not architecture additions.","Raising recall, rather than classification or localization quality, is the main lever for closing the LMM-specialist detection gap on fixed-vocabulary benchmarks.","A single model can serve detection, referring expression comprehension, captioning, and VQA: the Stage-IV variant keeps 47.1 AP on COCO while retaining captioning and VQA, and RefCOCO val improves from 81.4 to 85.7 when detection and REC are trained together.","Inference cost scales with category count: LMM-Det needs about 4.0 seconds per image with greedy decoding for 80 category passes on COCO, trading latency for accuracy compared with real-time specialist detectors.","Zero-shot detection on COCO improves to 24.5 AP after detection pretraining on Object365, suggesting the capability is learnable without any specialist component."],"supporting_citations":[{"why":"Supplies the base multimodal architecture, the linear-projector choice, the 595K/665K instruction data, and the LLaVA* baseline that the method is measured against.","marker":"[23]"},{"why":"OWLv2-ViT is the visual encoder whose high 1008×1008 resolution and uncompressed visual tokens are credited with a 3.4 AP gain in the ablation.","marker":"[28]"},{"why":"Salience-DETR generates the pseudo-labels merged into the training data during data distribution adjustment and is also the strongest specialist comparison.","marker":"[13]"},{"why":"Object365 is the large-scale detection pretraining data used in Stage II; adding it at high resolution lifts AP from 17.7 to 38.7 in the exploration.","marker":"[35]"},{"why":"COCO is the benchmark and fine-tuning dataset defining all reported AP and AR numbers.","marker":"[22]"},{"why":"Shikra is the prior direct-coordinate LMM baseline and the source of the finding that direct token prediction beats extra vocabulary tokens.","marker":"[7]"},{"why":"InternVL-2.5 is the main comparison for single-step simultaneous all-objects prediction, which the inference optimization is designed to beat.","marker":"[8]"},{"why":"Grounding DINO is the specialist detector bolted onto VisionLLM v2; its performance drop when integrated motivates the no-extra-modules design.","marker":"[25]"}],"fun_headline_variants":["LMM-Det: vision-language model hits 47.5 AP on COCO","Object detection without detectors: LMM achieves 47.5 AP","Multimodal model spots objects: 47.5 AP on COCO","LMM-Det: no detection modules, just 47.5 AP","Vision-language model detects objects at 47.5 AP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that detection is evaluated against a fixed, known category vocabulary: inference runs one prompt per class, so if the label set is unknown, open-ended, or very large, the per-category prompting strategy cannot be applied as-is and the reported recall gains would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["LMM-Det: vision-language model hits 47.5 AP on COCO","Object detection without detectors: LMM achieves 47.5 AP","Multimodal model spots objects: 47.5 AP on COCO","LMM-Det: no detection modules, just 47.5 AP","Vision-language model detects objects at 47.5 AP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3688,"prompt_tokens":1005,"completion_tokens":2683,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":2586}},"tokens_in":621,"tokens_out":2683,"duration_ms":17680,"temperature":1.0,"reasoning_tokens":2586,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:15:12.328424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LMM-Det on COCO val with the identical training recipe but replace the per-category prompts with a single all-objects prompt; if AR@100 stays near 63.6 rather than falling toward the roughly 27.5 the authors attribute to a single-pass model, the recall mechanism claimed for inference optimization would be refuted. A second check is to evaluate without retraining on Object365's 365 categories using the same per-category prompts; if AR@100 collapses, the fixed-vocabulary assumption is confirmed as load-bearing.","supporting_citations":[{"cited_title":"Microsoft coco: Common objects in context","cited_arxiv_id":null,"evidence_quote":"COCO is the benchmark and fine-tuning dataset defining all reported AP and AR numbers."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the base multimodal architecture, the linear-projector choice, the 595K/665K instruction data, and the LLaVA* baseline that the method is measured against."},{"cited_title":"Scal- ing open-vocabulary object detection","cited_arxiv_id":null,"evidence_quote":"OWLv2-ViT is the visual encoder whose high 1008×1008 resolution and uncompressed visual tokens are credited with a 3.4 AP gain in the ablation."},{"cited_title":"Salience detr: Enhancing detection trans- former with hierarchical salience filtering refinement","cited_arxiv_id":null,"evidence_quote":"Salience-DETR generates the pseudo-labels merged into the training data during data distribution adjustment and is also the strongest specialist comparison."},{"cited_title":"Objects365: A large-scale, high-quality dataset for object detection","cited_arxiv_id":null,"evidence_quote":"Object365 is the large-scale detection pretraining data used in Stage II; adding it at high resolution lifts AP from 17.7 to 38.7 in the exploration."}],"review_version":2}