Pith. sign in

REVIEW 3 major objections 5 minor 63 references

Locality-Aware Zero-Shot Human-Object Interaction Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LAIN adapts a frozen CLIP with locality and interaction adapters to detect unseen human-object interactions, reporting state-of-the-art zero-shot HOI detection on HICO-DET and V-COCO.

desk verdict A well-executed adapter-based SOTA for zero-shot HOI detection, with the main caveat being unverified cross-paper baselines and missing hyperparameters. read the letter →

arxiv 2505.19503 v1 pith:KYGQJYA3 submitted 2025-05-26 cs.CV

Sanghyun Kim , Deunsol Jung , Minsu Cho This is my paper
classification cs.CV
keywords zero-shothuman-objectinteractiondetectionCLIPadaptationlocalityawarenessreasoningHICO-DETV-COCOvision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that CLIP's image representations, while strong for whole-image recognition, miss the fine-grained local detail needed to tell interactions apart when adapted to human-object interaction (HOI) detection. To fix this, the authors build LAIN, which inserts two lightweight adapters—a Locality Adapter that aggregates neighboring image patches and spatial layout information, and an Interaction Adapter that reasons about how human and object regions relate—into a frozen CLIP. On HICO-DET and V-COCO, LAIN reports higher mean average precision than prior zero-shot HOI methods across all zero-shot settings, and also improves fully-supervised rare-class performance. If correct, this shows that a small, parameter-efficient adaptation can unlock CLIP's generalization for detecting interactions with objects and verbs never seen in training.

What carries the argument

The central mechanism is the pair of adapters, LA and IA, inserted before each layer of a frozen CLIP vision transformer. LA updates every image patch token with aggregated neighborhood context and a spatial-layout prior; IA updates each human-object token with interaction-pattern features produced by cross-attention between human and object region features, in a module the paper calls the Interaction Pattern Reasoning Module. The claim is that these two complementary injections let CLIP keep its global knowledge while gaining the local and relational detail needed to score unseen HOI categories.

What would settle it

Run LAIN and the strongest prior methods (for example CMMP and CLIP4HOI) in a single codebase with the same DETR detector, the same text prompt, and the same evaluation script on the HICO-DET UV and RF-UC settings; if LAIN no longer exceeds them on unseen-class mAP, the claimed state-of-the-art margin is an artifact of comparison conditions.

Watch

Extended reading notes

Core claim

The paper's central claim is that injecting locality awareness and interaction awareness into CLIP's patch and human-object tokens lets CLIP transfer to zero-shot HOI detection without hurting its generalization—unlike prior CLIP-based HOI adapters, which in some settings scored below raw CLIP on unseen classes. The authors show that the Locality Adapter (multi-scale convolutions over patch tokens with a spatial layout embedding derived from detected boxes and object text embeddings) plus the Interaction Adapter (ROI-aligned human and object features refined by cross-attention interaction-pattern reasoning) raise unseen-class mAP well above both CLIP alone and existing CLIP-adapted zero-shot methods. The paper also reports that gains are largest on small human and object boxes, where local detail matters most.

Load-bearing premise

The reported margins assume that prior methods' published results were produced under the same detector, text prompt template, and evaluation code as LAIN, since the paper does not run those baselines in its own pipeline.

Editorial extensions

If this is right

  • Zero-shot HOI detection can exceed raw CLIP's own zero-shot performance, closing the domain gap that hurt earlier CLIP adapters.
  • A few million trainable parameters (3.0M) suffice to adapt CLIP for HOI, far fewer than methods that train heavy decoders.
  • Small-instance HOI detection benefits most from locality-aware adaptation, suggesting local detail is the bottleneck for those cases.
  • The same adapters improve fully-supervised HOI detection, especially on rare classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The locality-and-interaction adapter pattern may transfer to other region-level CLIP tasks, such as open-vocabulary detection or segmentation, which also suffer from CLIP's global bias.
  • Because the paper compares against published numbers, a unified re-implementation of baselines with the same detector and prompt template might change the reported margins; a fair head-to-head is needed to confirm the magnitude of the improvement.
  • The spatial-layout embedding that encodes object text embeddings suggests the method partly works by aligning patch features to object semantics, a component that could be isolated and tested in future ablations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces LAIN, a zero-shot human-object interaction (HOI) detection framework built on a frozen CLIP visual encoder. LAIN inserts two lightweight adapters between CLIP layers: a Locality Adapter that aggregates neighboring patch information and spatial layout priors to capture fine-grained object details, and an Interaction Adapter that refines human-object tokens through cross-attention-based interaction pattern reasoning. The method is evaluated on HICO-DET and V-COCO under five zero-shot settings (UC, RF-UC, NF-UC, UV, UO) and fully supervised settings, reporting state-of-the-art results across all zero-shot settings. The paper also provides ablations showing that both adapters contribute, an analysis of adapter placement, a small-box analysis, and parameter/FLOP comparisons.

Significance. The central idea of injecting locality and interaction awareness into CLIP representations via parameter-efficient adapters is plausible and timely, addressing a known limitation of CLIP for region-level tasks. The extensive evaluation across multiple zero-shot settings and benchmarks, together with ablations and efficiency analysis, is a strength. However, the empirical claims are currently compromised by missing implementation details and uncontrolled comparisons to prior work; the reported margins are not independently verifiable. If the authors supply the missing hyperparameters, code, and a clearer baseline protocol, the contribution would be a solid step for zero-shot HOI detection. As written, the significance is conditional on these reproducibility and comparison issues being resolved.

major comments (3)
  1. [Section 4.1, Equations (7), (13), (17)] The experimental section does not report critical hyperparameters: the inference suppression weight λ in Eq. (17), the adapter dimension D_a, the number of IPRM queries N_p, the convolutional kernel sizes K, the number of adapter layers L, the learning rate, batch size, number of epochs, and the number of random seeds. Without these values, the reported mAP numbers cannot be reproduced or compared against, and the claimed state-of-the-art result is not independently verifiable. This is a load-bearing issue because the method's contribution is empirical and the exact architecture choices (e.g., D_a, N_p, kernel sizes) are part of the method's design.
  2. [Section 4.2, Table 1] The claim that LAIN "outperforms all previous methods by a significant margin under all zero-shot settings" rests on Table 1 numbers taken from prior publications without a unified evaluation harness. The paper does not specify the DETR detector version, detection confidence thresholds, HO-pair construction strategy (top-k vs all valid pairs), text prompt template, or inference suppression settings (Eq. 17) used for the baseline rows. In particular, the CLIP baseline row, which is used in the introduction and Section 4.2 to argue that existing methods degrade CLIP's zero-shot performance, is not described at all. If the baselines used a different detector or prompt template, the reported margins (e.g., +2.04 UC unseen over CLIP4HOI, +2.38 over CMMP) could be inflated by protocol differences rather than by locality and interaction awareness. The authors should either re-run baselines under their own pipeline or clearly state the protocol used for each row and provide error bars over multiple runs.
  3. [Table 5 and Section 4.3] The box-size analysis is presented as supporting evidence for the locality-awareness claim, but it is not a controlled comparison. The footnote states that ADA-CM's pretrained weights were obtained under "UC-RF" rather than UV, while the main ablation study is conducted under the UV setting; it is unclear under which zero-shot setting LAIN was trained for this table. If LAIN was trained under RF-UC (as the table caption suggests) while the ablation study uses UV, the comparison mixes training protocols and detector configurations. The authors need to clarify the settings and, ideally, compare all methods under the same setting with the same detected boxes so that the box-size gaps are attributable to the proposed modules rather than to protocol differences.
minor comments (5)
  1. [Author affiliations] The first author's email address "sanghuyn.kim" appears to be a typo for "sanghyun.kim".
  2. [Abstract] The phrase "we devise, LAIN," contains an unnecessary comma; consider rewording.
  3. [Section 3.3, Equations (7) and (13)] In Eq. (7), γ_LA is defined as a learnable parameter in R^{D_clip}, but in Eq. (13) γ_IA is not explicitly typed. Please clarify whether γ_IA is a scalar or a vector in R^{D_clip}.
  4. [Table 5 footnote] The footnote uses "UC-RF" interchangeably with "RF-UC"; these terms should be unified to avoid confusion.
  5. [Figure 1 and Table 1] The paper does not explain how the CLIP (RF-UC) and CLIP (UV) numbers in Figure 1(c) and Table 1 are computed, including which detector and text template are used; adding this description would improve the clarity of the motivation and the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LAIN's zero-shot results are empirical outcomes on held-out categories, not reconstructed from fitted inputs or self-citations.

full rationale

The derivation in arXiv:2505.19503 is self-contained against external benchmarks and does not reduce to its own inputs. The zero-shot evaluation holds out unseen HOI categories during training (Section 4.1), and the reported mAP values in Tables 1 and 2 are empirical outcomes of the trained adapters plus frozen CLIP and DETR, not quantities fitted to the test labels. No equation in Section 3 defines a predicted quantity in terms of the same evaluation target: Eq. (15) computes cosine similarity between HO tokens and text embeddings of HOI categories, Eq. (17) multiplies by detector confidence, and neither contains a parameter fitted to unseen-category performance. The only same-author citation is reference [22] (Kim, Jung, Cho, CVPR 2023), used in the Introduction and Related Work to describe conventional fully-supervised HOI methods and transformer-based one-stage detectors; it is not load-bearing for LAIN's architecture or its zero-shot claims. The comparison to published baseline numbers is a methodological fairness concern rather than a circularity concern, because LAIN's contribution is not definitionally equivalent to any prior method's output. Therefore no circular step is identified and the score is 0.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The method introduces two new adapter modules (LA and IA) but these are learned network components rather than newly postulated physical or conceptual entities with independent falsifiable evidence. The central claim rests on the quality of CLIP and DETR, the text prompt template, and a handful of unstated hyperparameters. No new mathematically fitted constants are derived from first principles; the results are empirical.

free parameters (7)
  • lambda (inference suppression weight) = not reported
    Appears in Eq. 17 and controls how much DETR box confidence is multiplied into HOI scores. Its value likely affects the final mAP but is not stated.
  • number of IPRM queries (Np) = not reported
    Defines the number of learnable queries in the Interaction Pattern Reasoning Module (Eq. 9). Sets the capacity of the cross-attention, but its value is not given.
  • adapter dimension (Da) = not reported
    Dimensionality after the first FFN in the adapters (Section 3.3). Affects the number of parameters and capacity, but the value is not given.
  • convolution kernel sizes K = not reported
    The set of kernel sizes k_n used in the Locality Adapter (Eq. 5). Determines the neighborhood size for locality aggregation, but is not reported.
  • adapter position (number of layers) = 1-12 (likely)
    Table 6 shows best results when adapters are inserted into all 12 layers. The paper does not report a search over this choice.
  • learnable temperature tau = learned
    In Eq. 15, tau rescales the logits and is learned during training. Its final value is not reported.
  • scaling factors gamma_LA, gamma_IA = learned
    Learnable vectors that balance the adapter outputs with the original features (Eq. 7, Eq. 13). Learned from training data, values not reported.
assumptions (4)
  • domain assumption CLIP's visual representations lack fine-grained local information needed for distinguishing interactions.
    Invoked in the Introduction and Section 2.2, citing [5,53,61,63]. The entire method is motivated by this premise; if CLIP already captured local detail, the adapters would be unnecessary.
  • domain assumption The pre-trained DETR detector provides accurate object boxes, classes, and features.
    Section 3.2 constructs HO tokens from DETR outputs. If the detector is unreliable, the region features and spatial layout embeddings would be noisy, and the claimed gains on small objects would not hold.
  • domain assumption Text embeddings produced by the template 'A photo of a person [verb-ing] a [object]' are semantically discriminative for both seen and unseen HOI categories.
    Section 3.5 uses this template for all categories. The zero-shot recognition relies on these embeddings for unseen verbs and objects.
  • domain assumption Inserting adapters into the frozen CLIP ViT layers preserves the model's pre-trained knowledge.
    The paper keeps CLIP weights frozen and only trains the adapters. This assumes the adapter outputs can be integrated without destroying CLIP's representations, which is supported by results but not formally proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Locality-Aware Zero-Shot Human-Object Interaction Detection." pith.science (2026). https://pith.science/paper/KYGQJYA3

@misc{pith2026250519503,
  author       = {Pith},
  title        = {Pith review of: Locality-Aware Zero-Shot Human-Object Interaction Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KYGQJYA3}},
  note         = {Machine review of arXiv:2505.19503}
}
read the original abstract

Recent methods for zero-shot Human-Object Interaction (HOI) detection typically leverage the generalization ability of large Vision-Language Model (VLM), i.e., CLIP, on unseen categories, showing impressive results on various zero-shot settings. However, existing methods struggle to adapt CLIP representations for human-object pairs, as CLIP tends to overlook fine-grained information necessary for distinguishing interactions. To address this issue, we devise, LAIN, a novel zero-shot HOI detection framework enhancing the locality and interaction awareness of CLIP representations. The locality awareness, which involves capturing fine-grained details and the spatial structure of individual objects, is achieved by aggregating the information and spatial priors of adjacent neighborhood patches. The interaction awareness, which involves identifying whether and how a human is interacting with an object, is achieved by capturing the interaction pattern between the human and the object. By infusing locality and interaction awareness into CLIP representation, LAIN captures detailed information about the human-object pairs. Our extensive experiments on existing benchmarks show that LAIN outperforms previous methods on various zero-shot settings, demonstrating the importance of locality and interaction awareness for effective zero-shot HOI detection.

Figures

Figures reproduced from arXiv: 2505.19503 by the authors.

Figure 1
Figure 1. (a)-(b): Since CLIP primarily encodes global informa [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall architecture of LAIN. All valid human-object pairs are constructed and embedded into HO tokens based on detection [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Qualitative results on HICO-DET under UV settings. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 54 canonical work pages

  1. [1]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 2, 3, 4

    work page 2020

  2. [2]

    Learning to detect human-object interactions

    Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In 2018 ieee winter conference on applications of computer vision (wacv), pages 381–389. IEEE, 2018. 2, 5, 6

    work page 2018

  3. [3]

    Reformulating hoi detection as adaptive set prediction

    Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi detection as adaptive set prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9004– 9013, 2021. 6

    work page 2021

  4. [4]

    Vision transformer adapter for dense predictions

    Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534, 2022. 2

    arXiv 2022

  5. [5]

    Maskclip: Masked self- distillation advances contrastive language-image pretraining

    Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskclip: Masked self- distillation advances contrastive language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10995–11005, 2023. 2, 4

    work page 2023

  6. [6]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 8

    arXiv 2010

  7. [7]

    Zero-shot out-of-distribution detection based on the pre-trained model clip

    Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pre-trained model clip. In Proceedings of the AAAI confer- ence on artificial intelligence, pages 6568–6576, 2022. 2

    work page 2022

  8. [8]

    Dirv: Dense interaction region voting for end-to-end human-object interaction detection

    Hao-Shu Fang, Yichen Xie, Dian Shao, and Cewu Lu. Dirv: Dense interaction region voting for end-to-end human-object interaction detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1291–1299, 2021. 2

    work page 2021

Show all 63 references
  1. [9]

    ican: Instance- centric attention network for human-object interaction detec- tion

    Chen Gao, Yuliang Zou, and Jia-Bin Huang. ican: Instance- centric attention network for human-object interaction detec- tion. arXiv preprint arXiv:1808.10437, 2018. 2

    2018 arXiv

  2. [10]

    Drg: Dual relation graph for human-object interaction detection

    Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang. Drg: Dual relation graph for human-object interaction detection. In European Conference on Computer Vision , pages 696–

  3. [11]

    Beyond instance-level im- age retrieval: Leveraging captions to learn a global visual representation for semantic retrieval

    Albert Gordo and Diane Larlus. Beyond instance-level im- age retrieval: Leveraging captions to learn a global visual representation for semantic retrieval. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 6589–6598, 2017. 1

    2017

  4. [12]

    Visual semantic role la- beling

    Saurabh Gupta and Jitendra Malik. Visual semantic role la- beling. arXiv preprint arXiv:1505.04474, 2015. 2, 5, 6

    2015 arXiv

  5. [13]

    No- frills human-object interaction detection: Factorization, lay- out encodings, and training techniques

    Tanmay Gupta, Alexander Schwing, and Derek Hoiem. No- frills human-object interaction detection: Factorization, lay- out encodings, and training techniques. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 9677–9685, 2019. 2

    2019

  6. [14]

    Neighborhood attention transformer

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6185–6194, 2023. 2

    2023

  7. [15]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 4

    2017

  8. [16]

    Image captioning: Transforming objects into words

    Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image captioning: Transforming objects into words. Advances in Neural Information Processing Systems , 32,

  9. [17]

    Vi- sual compositional learning for human-object interaction de- tection

    Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Vi- sual compositional learning for human-object interaction de- tection. In European Conference on Computer Vision, pages 584–600. Springer, 2020. 2

    2020

  10. [18]

    Affordance transfer learning for human-object interaction detection

    Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Affordance transfer learning for human-object interaction detection. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 495–504, 2021. 6

    2021

  11. [19]

    Detecting human-object interaction via fab- ricated compositional learning

    Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Detecting human-object interaction via fab- ricated compositional learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14646–14655, 2021. 2, 6

    2021

  12. [20]

    Uniondet: Union-level detector towards real-time human-object interaction detection

    Bumsoo Kim, Taeho Choi, Jaewoo Kang, and Hyunwoo J Kim. Uniondet: Union-level detector towards real-time human-object interaction detection. In European Conference on Computer Vision, pages 498–514. Springer, 2020. 2

    2020

  13. [21]

    Hotr: End-to-end human-object in- teraction detection with transformers

    Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end human-object in- teraction detection with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 74–83, 2021. 5, 6

    2021

  14. [22]

    Relational context learning for human-object interaction detection

    Sanghyun Kim, Deunsol Jung, and Minsu Cho. Relational context learning for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR) , pages 2925–2934,

  15. [23]

    Efficient adaptive human-object interac- tion detection with concept-guided memory

    Ting Lei, Fabian Caba, Qingchao Chen, Hailin Jin, Yuxin Peng, and Yang Liu. Efficient adaptive human-object interac- tion detection with concept-guided memory. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 6480–6490, 2023. 1, 3, 5, 6, 7

    2023

  16. [24]

    Explor- ing conditional multi-modal prompts for zero-shot hoi de- tection

    Ting Lei, Shaofeng Yin, Yuxin Peng, and Yang Liu. Explor- ing conditional multi-modal prompts for zero-shot hoi de- tection. In European Conference on Computer Vision, pages 1–19. Springer, 2025. 3, 6, 7, 8

    2025

  17. [25]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv e-prints , pages arXiv–1607,

  18. [26]

    Neural- logic human-object interaction detection

    Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural- logic human-object interaction detection. Advances in Neu- ral Information Processing Systems, 36, 2024. 1, 2, 6

    2024

  19. [27]

    Transfer- able interactiveness knowledge for human-object interaction detection

    Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. Transfer- able interactiveness knowledge for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3585– 3...

    2019

  20. [28]

    Hoi analysis: Integrating and decomposing human-object interaction

    Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-object interaction. Advances in Neural Information Processing Systems, 33:5011–5022, 2020. 2

    2020

  21. [29]

    Ppdm: Parallel point detection and matching for real-time human-object interaction detection

    Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Ji- ashi Feng. Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 482–490, 2020. 2

    2020

  22. [30]

    Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection

    Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 20123–20132, 2022. 3, 6

    2022

  23. [31]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740–755. Springer, 2014. 5

    2014

  24. [32]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. In Pro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 5

    2017

  25. [33]

    Consnet: Learning consistency graph for zero-shot human-object in- teraction detection

    Ye Liu, Junsong Yuan, and Chang Wen Chen. Consnet: Learning consistency graph for zero-shot human-object in- teraction detection. In Proceedings of the 28th ACM Interna- tional Conference on Multimedia, pages 4235–4243, 2020. 2

    2020

  26. [34]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 2, 7

    2021

  27. [35]

    Visual knowledge graph for human action rea- soning in videos

    Yue Ma, Yali Wang, Yue Wu, Ziyu Lyu, Siran Chen, Xiu Li, and Yu Qiao. Visual knowledge graph for human action rea- soning in videos. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia , pages 4132–4141, 2022. 2

    2022

  28. [36]

    Clip4hoi: Towards adapting clip for practi- cal zero-shot hoi detection

    Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: Towards adapting clip for practi- cal zero-shot hoi detection. Advances in Neural Information Processing Systems, 36, 2024. 1, 3, 4, 5, 6, 8

    2024

  29. [37]

    Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models

    Shan Ning, Longtian Qiu, Yongfei Liu, and Xuming He. Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23507–23517, 2023. 1, 3, 5, 6, 8

    2023

  30. [38]

    Learning human-object interactions by graph parsing neural networks

    Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions by graph parsing neural networks. In Proceedings of the Eu- ropean conference on computer vision (ECCV) , pages 401– 417, 2018. 2

    2018

  31. [39]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  32. [40]

    Stand-alone self- attention in vision models

    Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self- attention in vision models. Advances in neural information processing systems, 32, 2019. 7

    2019

  33. [41]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information process- ing systems, 28, 2015. 2

    2015

  34. [42]

    Qpic: Query-based pairwise human-object interaction detec- tion with image-wide contextual information

    Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human-object interaction detec- tion with image-wide contextual information. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10410–10419, 2021. 2, 6

    2021

  35. [43]

    Iwin: Human-object inter- action detection via transformer with irregular windows

    Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. Iwin: Human-object inter- action detection via transformer with irregular windows. In European Conference on Computer Vision , pages 87–103. Springer, 2022. 6

    2022

  36. [44]

    Vsgnet: Spatial attention network for detecting human ob- ject interactions using graph convolutions

    Oytun Ulutan, ASM Iftekhar, and Bangalore S Manjunath. Vsgnet: Spatial attention network for detecting human ob- ject interactions using graph convolutions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13617–13626, 2020. 2

    2020

  37. [45]

    Exploiting clip for zero-shot hoi detection requires knowledge distillation at multiple lev- els

    Bo Wan and Tinne Tuytelaars. Exploiting clip for zero-shot hoi detection requires knowledge distillation at multiple lev- els. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1805–1815, 2024. 3

    2024

  38. [46]

    Bilateral adaptation for human-object interac- tion detection with occlusion-robustness

    Guangzhi Wang, Yangyang Guo, Ziwei Xu, and Mohan Kankanhalli. Bilateral adaptation for human-object interac- tion detection with occlusion-robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27970–27980, 2024. 3, 6

    2024

  39. [47]

    Contextual heterogeneous graph network for human-object interaction detection

    Hai Wang, Wei-shi Zheng, and Ling Yingbiao. Contextual heterogeneous graph network for human-object interaction detection. In European Conference on Computer Vision , pages 248–264. Springer, 2020. 2

    2020

  40. [48]

    Clipn for zero-shot ood detection: Teaching clip to say no

    Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1802–1812, 2023. 2

    2023

  41. [49]

    Learning transferable human-object interaction detector with natural language su- pervision

    Suchen Wang, Yueqi Duan, Henghui Ding, Yap-Peng Tan, Kim-Hui Yap, and Junsong Yuan. Learning transferable human-object interaction detector with natural language su- pervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 939–948,

  42. [50]

    Contextual similarity distillation for asymmetric im- age retrieval

    Hui Wu, Min Wang, Wengang Zhou, Houqiang Li, and Qi Tian. Contextual similarity distillation for asymmetric im- age retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9489–9498, 2022. 1

    2022

  43. [51]

    Difnet: Boosting visual information flow for image captioning

    Mingrui Wu, Xuying Zhang, Xiaoshuai Sun, Yiyi Zhou, Chao Chen, Jiaxin Gu, Xing Sun, and Rongrong Ji. Difnet: Boosting visual information flow for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 18020–18029,

  44. [52]

    End-to-end zero-shot hoi detec- tion via vision and language knowledge distillation

    Mingrui Wu, Jiaxin Gu, Yunhang Shen, Mingbao Lin, Chao Chen, and Xiaoshuai Sun. End-to-end zero-shot hoi detec- tion via vision and language knowledge distillation. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 2839–2846, 2023. 1, 3, 6

    2023

  45. [53]

    Clipself: Vision trans- former distills itself for open-vocabulary dense prediction

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision trans- former distills itself for open-vocabulary dense prediction. arXiv preprint arXiv:2310.01403, 2023. 2, 4

    2023 arXiv

  46. [54]

    Exploring visual relationship for image captioning

    Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European conference on computer vision (ECCV), pages 684–699, 2018. 1

    2018

  47. [55]

    Image-to-image retrieval by learning similarity be- tween scene graphs

    Sangwoong Yoon, Woo Young Kang, Sungwook Jeon, SeongEun Lee, Changjin Han, Jonghun Park, and Eun-Sol Kim. Image-to-image retrieval by learning similarity be- tween scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10718–10726, 2021. 1

    2021

  48. [56]

    Rlip: Rela- tional language-image pre-training for human-object interac- tion detection

    Hangjie Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, and Mingqian Tang. Rlip: Rela- tional language-image pre-training for human-object interac- tion detection. Advances in Neural Information Processing Systems, 35:37416–37431, 2022. 6

    2022

  49. [57]

    Mining the benefits of two-stage and one-stage hoi detection

    Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li. Mining the benefits of two-stage and one-stage hoi detection. Advances in Neural Information Processing Systems, 34:17209–17220, 2021. 6

    2021

  50. [58]

    Spatially conditioned graphs for detecting human-object in- teractions

    Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object in- teractions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13319–13327, 2021. 2, 5

    2021

  51. [59]

    Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer

    Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20104–20112, 2022. 4, 5, 6

    2022

  52. [60]

    Exploring structure-aware trans- former over interaction proposals for human-object interac- tion detection

    Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang-Wen Chen. Exploring structure-aware trans- former over interaction proposals for human-object interac- tion detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1...

    2022

  53. [61]

    Regionclip: Region- based language-image pretraining

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chun- yuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2022

  54. [62]

    Human-object interaction detection via disentangled transformer

    Desen Zhou, Zhichao Liu, Jian Wang, Leshan Wang, Tao Hu, Errui Ding, and Jingdong Wang. Human-object interaction detection via disentangled transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19568–19577, 2022. 2

    2022

  55. [63]

    Zegclip: Towards adapting clip for zero-shot se- mantic segmentation

    Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot se- mantic segmentation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 11175–11185, 2023. 1, 2, 4

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.