Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Open-Det: An Efficient Learning Framework for Open-Ended Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Open-Det achieves open-ended object detection with 1.5% of GenerateU's training data, 20.8% of its epochs, and fewer GPUs, while raising APr on zero-shot LVIS by 1.0 point.

desk verdict Solid systems paper with a plausible efficiency story, but the headline comparison mixes two GenerateU baselines and needs a controlled re-run before I'd trust the +1.0% APr claim. read the letter →

arxiv 2505.20639 v1 pith:3OGRSPX3 submitted 2025-05-27 cs.CV

Guiping Cao , Tao Wang , Wenjian Huang , Xiangyuan Lan , Jianguo Zhang , Dongmei Jiang This is my paper · ORCID
classification cs.CV
keywords open-endedobjectdetectionzero-shotvision-languagealignmentknowledgedistillationefficienttraininggenerativelanguagemodelnamegenerationLVIS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-Det is a framework for open-ended object detection—locating objects and generating their category names in free-form text without any fixed vocabulary at inference. The paper's claim is that this task does not need the web-scale grounding datasets and heavy GPU budgets used by the existing GenerateU approach. Trained on the 77,398 Visual Genome images (1.5% of GenerateU's 5.077M images) for 31 epochs (20.8% of GenerateU's 149) on 4 V100 GPUs, Open-Det reports an APr of 21.0 on zero-shot LVIS MiniVal, 1.0 point higher than GenerateU trained with GRIT5M. The framework reaches this efficiency by rebuilding the detector and name generator, distilling vision-language knowledge into 'VL-prompts', and replacing contradictory alignment supervision with masked and joint losses. If the efficiency comparisons survive matched training budgets, cheap and practical open-ended detection becomes a realistic target for modest compute settings.

What carries the argument

The central mechanism is the VL-prompt: a decoder query transformed into an image-like representation aligned to text-embedding space, produced by VLD-M through deformable cross-attention over backbone and encoder features, MLP projection, and linear fusion, then supervised against the VLM's text embeddings by cosine similarity. The VL-prompt, rather than the raw vision query, is what the LLM sees, which is what lets Open-Det bridge the cross-modal gap and train the name generator without large paired data. Around it sit the bidirectional alignment score $S_{\text{align}} = \cos(Q_d M_{VL}, T_e) + \cos(Q_d, T_e M_{LV})$, the decoupled one-to-many/one-to-one decoder, and the two loss corrections: Masked Alignment Loss (binarized text-text mask set by threshold 0.99) and Joint Loss combining binary, IoU, and alignment scores.

What would settle it

Run a controlled experiment that trains the original GenerateU code on the same 4 V100 GPUs, batch size 8, optimizer configuration, and 31 epochs used for Open-Det, then compare on LVIS MiniVal; if GenerateU reaches an equal or higher APr under that budget, the claimed efficiency gain is not caused by the proposed architecture.

Watch

Extended reading notes

Core claim

On its own terms, Open-Det claims that accelerating open-ended detection is a matter of architecture and supervision design rather than data scale. Its Object Detector uses a decoupled decoder: the first four layers train with one-to-many matching to speed box convergence, the last two use one-to-one matching to suppress duplicates, and a threshold on encoder-token scores selects queries so the model detects a variable number of objects. Instead of feeding raw vision queries to the T5-based name generator, a Vision-to-Language Distillation Module (VLD-M) enriches each query with backbone and encoder context through deformable cross-attention, projects it into the text-embedding space, and distills a frozen CLIP's knowledge into these 'VL-prompts' with a cosine-similarity loss. A Bidirectional Vision-Language Alignment module (BVLA-M) computes alignment scores in both V-to-L and L-to-V directions, a Masked Alignment Loss zeroes out contradictory same-category negative pairs, and a Joint Loss couples binary classification with IoU and alignment scores. The reported outcome is a model that, from 0.077M images and 31 epochs, outperforms GenerateU trained on 5.077M images for 149 epochs, and scales further with larger backbones (APr 31.2 with Swin-Large versus 22.3).

Load-bearing premise

The central claim depends on the epoch conversion in Appendix B.1 faithfully equating GenerateU's and Open-Det's training budgets despite different batch sizes, GPU types, and optimizer schedules; if those budgets are not equivalent, the reported efficiency advantage could be an artifact of schedule and hardware rather than the proposed architecture.

Editorial extensions

If this is right

  • Open-ended detection can be trained from a single small labeled dataset (Visual Genome, 77k images) rather than multi-million-image grounding corpora, reducing data acquisition cost by roughly 65 times.
  • Rare-class performance, historically the weak spot in zero-shot detection, improves most: +3.6% APr over GenerateU at equal data, +1.0% APr over GenerateU with GRIT5M, and +8.9% APr with a larger backbone.
  • The same framework transfers zero-shot to COCO2017 and Objects365 with +2.2 and +3.3 AP over GenerateU, so the data-efficiency gain is not confined to the LVIS benchmark.
  • Convergence speed is compressed to about one fifth of the epochs, making iterative experimentation on open-ended detection practical on 4 V100 GPUs instead of 16 A100s.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the design suggests the real bottleneck in open-ended detection is the modality gap rather than data volume; if so, applying VL-prompt distillation to other generative detectors could yield similar efficiency gains.
  • Editorial inference: the reported efficiency advantage depends on the epoch conversion in Appendix B.1; a controlled re-run of GenerateU under Open-Det's batch size and optimizer would cleanly separate architecture effects from schedule and hardware effects.
  • Editorial inference: freezing the language model head and training a LoRa head during early epochs, plus text denoising, may generalize as a recipe for fine-tuning any heavy generative head on noisy multimodal inputs.
  • Editorial inference: threshold-based query selection makes the detector's object count dynamic, a property that could be extended to other DETR-like detectors beyond the open-ended setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Open-Det, a framework for open-ended object detection that combines an object detector with a generative language model. The main contributions are a decoupled one-to-many/one-to-one decoder and threshold-based query selection, a bidirectional vision-language alignment module, a prompts distiller that transfers VLM knowledge into VL-prompts, a LoRA-based object name generator with text denoising, and two loss functions (Masked Alignment Loss and Joint Loss). The authors claim that Open-Det matches or exceeds GenerateU while using only 1.5% of the training data, 20.8% of the training epochs, and fewer GPU resources, and they support the claim with experiments on LVIS MiniVal, COCO, and Objects365, together with extensive ablations.

Significance. If the efficiency claims are validated, Open-Det would be a meaningful step toward practical open-ended detection: it addresses a real training-cost bottleneck in GenerateU and introduces several mechanistically motivated components (distilled VL-prompts, masked alignment, and a joint classification loss). The paper is commendable for releasing source code, providing component-wise ablations, and reporting score-distribution analyses. The main risk is the headline comparison: the claimed simultaneous advantages in data, epochs, and accuracy are assembled from two different GenerateU baselines, and no controlled re-run of the reference model in the train-LLM setting is provided. The contribution is therefore plausible but not yet established at the level claimed.

major comments (3)
  1. [Abstract and Table 1] The headline efficiency claim combines two different GenerateU baselines. The '1.5% of training data (0.077M vs 5.077M)' and '+1.0% APr' numbers compare Open-Det on VG with GenerateU trained on VG+GRIT5M (APr 20.0), while the '20.8% of epochs (31 vs 149)' number compares with GenerateU trained on VG only (APr 17.4). No single row of Table 1 supports all three ratios simultaneously, and the +1.0% APr gain does not correspond to the 31-vs-149 epoch comparison. The paper should either report a single matched baseline for all three quantities or clearly separate the two comparisons in the abstract and conclusions.
  2. [Appendix B.1 and Table 7] The conversion from iterations to epochs (180,000 x 64 / 77,398 = 149; 300,000 x 8 / 77,398 = 31) is arithmetically correct, but it does not equate the training protocols: GenerateU uses batch size 64 on 16 A100 GPUs, while Open-Det uses batch size 8 on 4 V100 GPUs, and the optimizer schedules and evaluation harness are not matched. Since the paper demonstrates in Table 7 that GenerateU can be re-run on 4 V100 GPUs with the public official code (in the frozen-LLM setting), a controlled train-LLM re-run of the reference row is feasible and should be supplied. Without such a re-run, the claimed data/epoch/GPU advantages cannot be attributed to the proposed architecture rather than to protocol differences.
  3. [Section 3.4 and Appendix A.5, Eq. (7)] The total-loss decomposition defines the distillation loss as LVLD = 1 - cosine(Qd, Te), using the decoder queries Qd, whereas Section 3.4 and Figure 3 state that the cosine-similarity supervision is applied to the distilled VL-prompts Pvl. This is a material inconsistency for reproducibility: if the implementation follows Section 3.4, Eq. (7) is wrong; if Eq. (7) is correct, the description of the Prompts Distiller is wrong. Please reconcile the notation and specify exactly which tensors enter the cosine loss.
minor comments (5)
  1. [Section 3.6, Eq. (3)] The symbol M is used both for the number of text embeddings and for the binary mask matrix, which makes the dimension of the product VLalign x M easy to misread; please use distinct symbols (e.g., M for the mask and M_text or K for the number of embeddings).
  2. [Section 3.2, Eq. (1)] The set notation for Qid and the statement that decoder queries Qd are selected from E according to Qid is not fully precise; please specify how the selected indices are converted into query embeddings and how variable-length query sets are handled by the transformer decoder.
  3. [Section 4.1 and Table 1] For the GenerateU trained on VG+GRIT5M row, the training epoch count is omitted, so the reader cannot verify the '20.8% of epochs' comparison against that baseline; please report the epoch count or explicitly state that it is not available.
  4. [Appendix B.1, Table 6] There is a typo in the table: 'GenereteU' should be 'GenerateU'; also, 'LoRa' and 'LoRA' are used inconsistently throughout the paper.
  5. [Appendix A.5, Eq. (9)] The symbol N is reused for the number of queries in Eq. (4) and for the number of tokens in the output sequence in Eq. (9); please use a different symbol (e.g., T) for the token count to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: Open-Det's central efficiency/accuracy claims are empirical comparisons against an external baseline, with no equation-level reduction to inputs and no load-bearing self-citation chain.

full rationale

Open-Det is an empirical systems paper. The claimed derivation chain—architecture components (ODR, VLD-M, BVLA-M, ONG) plus losses (MAL, Joint Loss)—is validated by ablations on the VG-trained model against the independent GenerateU baseline published by Lin et al. (2024). No core result is defined in terms of the quantity it claims to predict: APr/AP are computed with a fixed pre-trained text encoder following the established GenerateU protocol, and the efficiency ratios are arithmetic conversions of reported iterations to epochs (Table 6), not fitted parameters renamed as predictions. The self-citations (Strip-MLP, MLP-DINO, Cao et al. 2023/2024) appear only in related work and as suggestions for stronger backbones in Limitations; they are not load-bearing. The abstract's combined '1.5% data, 20.8% epochs, +1.0% APr' compares data size and APr gain against the VG+GRIT5M GenerateU row while the epoch ratio refers to the VG-only GenerateU row, and the reference GenerateU runs were not re-run under identical hardware/batch (only GenerateU† in Table 7 was re-run, under a frozen-LLM setting). These are experimental-control and reporting concerns, not circularity: no equation reduces to its own input, and no prediction is forced by construction. Appendix Eq. 7 also writes LVLD with Qd rather than Pvl, an internal inconsistency that does not constitute circular reasoning.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central empirical claim does not derive from first principles. It rests on the assumptions above plus the hand-set thresholds lambda, tau, and alpha. The exact VLM used for distillation is not named, and the controlled comparison with a train-LLM GenerateU baseline is missing. No new physical entity is introduced; VL-prompts are internal learned features.

free parameters (4)
  • query selection threshold lambda = 0.05 (default)
    Eq. (1) selects encoder tokens as decoder queries when sigmoid(Linear(e_t)) > lambda. The threshold controls the variable number of detected objects and is chosen without sensitivity analysis.
  • mask binarization threshold tau = 0.99 (default)
    Eq. (3) binarizes text-embedding similarity Te*Te^T at tau to decide which pairs share a category. This directly determines which negative gradients are masked and is not ablated.
  • Joint Loss scale alpha = 0.25 (default)
    Eq. (4) uses alpha to modulate the consistency term between binary score, alignment score, and IoU. No ablation over alpha is reported.
  • LoRA head rank and scaling = not specified
    Sec. 3.5 and Appendix A.2 report only that trainable parameters drop to 0.526M. The rank and scaling of the LoRA head are not given, leaving an important hidden configuration.
assumptions (5)
  • domain assumption Cosine similarity between VL-prompts and text embeddings from a frozen VLM is a valid training signal for transferring region-level semantics from image-level VLM knowledge.
    Sec. 3.4 uses a cosine similarity loss to distill VLM knowledge into Pvl. This presumes CLIP-style text embeddings are a good semantic space for object names and that image-level alignment transfers to region-level features.
  • ad hoc to paper The mini-batch text-embedding similarity matrix Te*Te^T, binarized at tau=0.99, correctly identifies when two text embeddings refer to the same object category.
    Eq. (3) uses this mask to suppress contradictory negative losses. If synonyms or paraphrases have similarity below 0.99, the mask fails to suppress them and the contradiction the loss claims to remove persists.
  • domain assumption Published GenerateU numbers are directly comparable to Open-Det after converting iterations to epochs, despite differences in batch size, hardware, and optimizer schedule.
    Appendix B.1 and Table 6 provide epoch conversion but do not control for other training differences. The headline efficiency comparison rests on this assumption.
  • domain assumption The evaluation metric, which matches generated object names to ground-truth categories via a fixed text encoder, is an accurate measure of open-ended detection performance.
    Sec. 4 defines metrics following GenerateU. AP computed this way may differ from vocabulary-based LVIS evaluation used for OVD baselines in the same table.
  • domain assumption T5 can reconstruct object names from text embeddings corrupted with Gaussian noise of standard deviation equal to the embedding standard deviation.
    Sec. 3.5 introduces Text Denoising without derivation. It assumes the denoising objective does not distort the LLM's ability to generate names from VL-prompts at inference.
invented entities (1)
  • VL-prompts (Pvl)
    purpose: Intermediate query features that fuse detector queries with background context and are distilled toward CLIP text embeddings; used as LLM input instead of raw vision queries.
    VL-prompts have no external falsifiable handle. Their support comes only from internal ablations and similarity scores reported in the paper, not from any independent measurement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Open-Det: An Efficient Learning Framework for Open-Ended Detection." pith.science (2026). https://pith.science/paper/3OGRSPX3

@misc{pith2026250520639,
  author       = {Pith},
  title        = {Pith review of: Open-Det: An Efficient Learning Framework for Open-Ended Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3OGRSPX3}},
  note         = {Machine review of arXiv:2505.20639}
}
read the original abstract

Open-Ended object Detection (OED) is a novel and challenging task that detects objects and generates their category names in a free-form manner, without requiring additional vocabularies during inference. However, the existing OED models, such as GenerateU, require large-scale datasets for training, suffer from slow convergence, and exhibit limited performance. To address these issues, we present a novel and efficient Open-Det framework, consisting of four collaborative parts. Specifically, Open-Det accelerates model training in both the bounding box and object name generation process by reconstructing the Object Detector and the Object Name Generator. To bridge the semantic gap between Vision and Language modalities, we propose a Vision-Language Aligner with V-to-L and L-to-V alignment mechanisms, incorporating with the Prompts Distiller to transfer knowledge from the VLM into VL-prompts, enabling accurate object name generation for the LLM. In addition, we design a Masked Alignment Loss to eliminate contradictory supervision and introduce a Joint Loss to enhance classification, resulting in more efficient training. Compared to GenerateU, Open-Det, using only 1.5% of the training data (0.077M vs. 5.077M), 20.8% of the training epochs (31 vs. 149), and fewer GPU resources (4 V100 vs. 16 A100), achieves even higher performance (+1.0% in APr). The source codes are available at: https://github.com/Med-Process/Open-Det.

Figures

Figures reproduced from arXiv: 2505.20639 by the authors.

Figure 1
Figure 1. Performance curves of GenerateU and Open-Det, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Main architecture of the Open-Det framework. It consists of 4 collaborative components: (1) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overall architecture of the proposed VLD-M. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Visualization results for Ground Truth, GenerateU, and Open-Det on the LVIS MiniVal dataset. Open-Det [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: The simplified pipeline of the Open-Det framework. The Vision-Language Model (VLM) model and input texts [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Illustration of contradictory loss generation in the query-text alignment process. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: 3D surface plots of (a) positive weights, (b) negative weights, and (c) Joint Loss. To simplify the visualization in [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Comparison of VL alignment scores between GenerateU and Open-Det using violin and box plots. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Visualization of query-text alignment score and text similarity score in zero-shot domain transfer on LVIS [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Visualization of detection results for Ground Truth, GenerateU, and Open-Det models on the LVIS MiniVal [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Visualization of detection results for Ground Truth, GenerateU, and Open-Det on the LVIS MiniVal dataset. The [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [6]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    Loshchilov, I. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    arXiv

  3. [7]

    Kosmos-2: Grounding multimodal large language models to the world

    Peng, Z., Wang, W., Dong, L., Hao, Y ., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824,

    arXiv

  4. [10]

    De- formable detr: Deformable transformers for end-to-end object detection

    Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. De- formable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159,

    arXiv 2010

  5. [11]

    For each object query, the corresponding word in the text encoder is treated as a positive sample, while all other words in the same mini-batch are treated as negative samples

    text encoder. For each object query, the corresponding word in the text encoder is treated as a positive sample, while all other words in the same mini-batch are treated as negative samples. This arrangement establishes a contrastive learning framework, where the model maximizes the similarity between the query and its positive text embedding while simult...

    work page 2019

  6. [12]

    soft loss

    and BCE loss, the proposed Joint loss adjusts the binary loss weights using consistent weights by associating the IoU score, alignment score, and binary score (), adaptively generating a “soft loss”. For positive samples, the weight increases as the difference between the binary score and the IA Score grows; for negative samples, the weight increases only...

    work page 2020

  7. [13]

    As presented in Table 6, Open-Det achieves higher performance with only 31 training epochs, which is 20.8% of the epochs required by GenerateU ( 149 training epochs)

    is trained using 16 A100 GPUs with a batch size of 64 for 180,000 iterations, while Open-Det is trained using 4 V100 GPUs with a batch size of 8 for 300,000 iterations. As presented in Table 6, Open-Det achieves higher performance with only 31 training epochs, which is 20.8% of the epochs required by GenerateU ( 149 training epochs). This highlights that ...

    work page 2024

  8. [2017]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499,

    arXiv

Show all 13 references
  1. [2019]

    Llama 2: Open foundation and fine- tuned chat models

    11 Open-Det: An Efficient Learning Framework for Open-Ended Detection Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint ar...

    arXiv

  2. [2021]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  3. [2022]

    D., Hu, D

    Zareian, A., Rosa, K. D., Hu, D. H., and Chang, S.-F. Open- vocabulary object detection using captions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14393–14402, 2021a. Zareian, A., Rosa, K. D., Hu, D. H., and Chang, S.-F. Open- v...

    arXiv

  4. [2023]

    Llms meet vlms: Boost open vocabulary object detection with fine- grained descriptors

    Jin, S., Jiang, X., Huang, J., Lu, L., and Lu, S. Llms meet vlms: Boost open vocabulary object detection with fine- grained descriptors. arXiv preprint arXiv:2402.04630,

    arXiv

  5. [2024]

    Lin, T.-Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Doll´ar, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–

    2014

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.