Pith. sign in

REVIEW 2 major objections 4 minor 55 references

Improving Personalized Search with Regularized Low-Rank Parameter Updates

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that personalized image retrieval can be achieved by a rank-one update to the value transform of CLIP's final text attention layer, with no textual inversion and no large-scale pretraining.

desk verdict A simple and well-ablated rank-one LoRA update to CLIP's final text value transform beats textual inversion for personalized retrieval; the core result is solid, but ConCon-Chi's SOTA claim needs seed-level variance. read the letter →

arxiv 2506.10182 v1 pith:ZF2VBZ2Q submitted 2025-06-11 cs.CV

Fiona Ryan , Josef Sivic , Fabian Caba Heilbron , Judy Hoffman , James M. Rehg , Bryan Russell This is my paper · ORCID
classification cs.CV
keywords personalizedretrievalvision-languagemodelslow-rankadaptationLoRAtextualinversionCLIPparameter-efficientfine-tuningmulti-conceptqueries
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

POLAR claims that personalized image retrieval—recognizing a new concept such as 'my dog Fido' from a handful of photos—can be solved by a small regularized update to the internal weights of CLIP's text encoder, rather than by learning pseudo-word tokens through textual inversion. Specifically, it learns a rank-one update to the value transform of the final attention layer, hooked to a fixed vocabulary token, and this single update is enough to compose the personal concept with general language knowledge. If this is right, personalizing a retrieval model becomes a sub-second optimization of only $2d$ parameters per concept, with no large-scale pretraining and no measurable loss of the model's general knowledge. The paper supports this with state-of-the-art results on DeepFashion2 and ConCon-Chi and a new VLM-caption metric showing general-knowledge retention.

What carries the argument

The load-bearing object is the rank-one adapted value transform $V'_{L,c}=V_L+B_{L,c}A_{L,c}$, where $V_L$ is the value projection of the final transformer attention layer in CLIP's text encoder and $B_{L,c}A_{L,c}$ is a LoRA-style low-rank update with rank one. LoRA reparameterizes a weight matrix $W$ as $W+BA$; here $B_{L,c}$ has shape $d\times 1$ and $A_{L,c}$ has shape $1\times d$, so the whole concept lives in $2d$ parameters. The argument is carried by the regularization pair: the squared-L2 penalty on $B_{L,c}$ and the unit-norm constraint on $A_{L,c}$ keep the perturbation from overriding CLIP's general knowledge, while applying it only at the last layer injects the personal concept late in encoding, where it can adjust the final representation without rewriting earlier context. The fixed placeholder token 'sks' is what lets the update hook into natural-language queries without a learned token vocabulary.

What would settle it

Take two concepts whose images are visually near-identical to the frozen CLIP image encoder, for example two black skirts that differ only in pattern, and train POLAR on one of them. If a query for that concept cannot retrieve its own images over the twin's because the image features are essentially the same, then the central claim fails for such concepts, exactly as the paper's stated limitation predicts.

Watch

Extended reading notes

Core claim

The central claim is that for a frozen CLIP dual encoder, the adapted text encoder defined by $V'_{L,c}=V_L+B_{L,c}A_{L,c}$—a rank-one low-rank update to the value transform of the final attention layer—is a better representation of a personal concept than a learned pseudo-token. The update is trained with an MSE alignment loss plus a squared-L2 penalty on $B_{L,c}$, with $A_{L,c}$ normalized, which keeps the update small enough to preserve general knowledge. On DeepFashion2 this reaches context mRR 40.72 with ViT-L/14 versus 27.62 for the strongest previous method; on ConCon-Chi it reaches context mRR 46.33 versus 43.93, and concept-only mAP 68.71 versus 61.94, while VLM caption recall@10 stays at 52.62 against original CLIP's 52.69. The paper also shows that updates for multiple concepts can be merged by addition, and that the approach works with a single training image.

Load-bearing premise

The load-bearing premise is that the frozen CLIP image encoder already encodes enough fine-grained visual detail to distinguish the personal concept from visually similar concepts, since only the text encoder is updated and the paper's Limitations section says this sets an inherent bound on performance.

Editorial extensions

If this is right

  • Personalized retrieval can be done without pretrained textual inversion networks or pseudo-token vocabularies: each concept is stored as a rank-one update of size $2d$, and training converges in under a second on a single V100 GPU.
  • The same frozen CLIP backbone retains its general retrieval ability: VLM-caption recall@10 stays at 52.62 versus 52.69 for original CLIP, and Flick30k general retrieval stays at parity with the unmodified model.
  • Multiple personal concepts can be combined in one query by simply adding their updates, which is equivalent to a rank-two update and outperforms averaging and orthogonal-adaptation merging.
  • The method achieves state-of-the-art results on both standard personalized-retrieval benchmarks, improving context retrieval by 4 to 22 percent over prior approaches and also succeeding when only a single training image per concept is available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, because the update is confined to the text encoder, the same rank-one mechanism could be applied on top of any frozen dual encoder, and the per-concept cost of $2d$ parameters makes it a natural candidate for on-device personalization; the paper does not evaluate either extension.
  • Beyond the paper, the unit-norm constraint on $A_{L,c}$ suggests a diagnostic the authors do not run: inspecting which tokens in a query most strongly activate the learned update could reveal whether the concept is stored as a visual pattern or as a textual association.
  • Beyond the paper, the paper's success with additive concept merging implies the per-concept updates may form a nearly linear representation space; a testable consequence is that interpolating between two concepts' updates should produce retrieval behavior of a hybrid concept, which would go beyond the additive combination the paper reports.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces POLAR, a method for personalized vision-language retrieval that learns a rank-one low-rank (LoRA) update to the value transform of the final layer of CLIP's text encoder, using a fixed placeholder token V* and an MSE training loss regularized by an L2 penalty on the B factor and a unit-norm constraint on the A factor. The authors report state-of-the-art results on the DeepFashion2 and ConCon-Chi benchmarks, ablations over rank, layer, parameter subset, regularization, and multi-concept merging, and introduce a VLM-caption recall metric to measure preservation of general knowledge.

Significance. If the reported results hold, POLAR is a significant contribution: it demonstrates that a simple, parameter-efficient update to the text encoder can replace textual inversion for per-concept personalization, requiring no pretraining, under one second of optimization, and retaining general knowledge. The ablations are thorough and the DeepFashion2 results include standard errors over multiple runs. The TEST-UNSEEN check in the supplementary is a good practice. However, the ConCon-Chi state-of-the-art claim currently lacks statistical support, and the retrieval setup leaves an important ambiguity about whether training images appear in the retrieval database.

major comments (2)
  1. [Table 2 and Section 6] The ConCon-Chi state-of-the-art claim rests on single-run numbers. The full TEST split includes the 3 validation concepts used to select the regularization weight λ (Section 4.4), and the TEST-UNSEEN result in Table 9 is also a single run. Because per-concept training uses random LoRA initialization and random template sampling and costs under one second per concept, seed variance is cheap to measure. Without standard errors, the ~2.4 mRR margin over SEARLE on both splits is not demonstrated to be statistically significant. Please report means and standard deviations over at least 5 seeds for POLAR and the main baselines on both the TEST and TEST-UNSEEN splits.
  2. [Section 4.1 and Tables 1, 2] The paper does not state whether the training images used for personalization are excluded from the retrieval database. The perfect concept-only mRR of 100.00 in Table 2 raises the concern that the model may be retrieving the training images it was optimized on, which would measure memorization rather than generalization to new instances. If training images are excluded from the retrieval set, please state this explicitly and describe the split; if they are included, the concept-only results should be re-reported on a retrieval set that excludes training images, and the context-query results should be examined for the same effect.
minor comments (4)
  1. [Table 3 and Section 13] The VLM caption recall@10 metric is a new evaluation, and the supplementary discloses that a manual check found 88/100 captions accurate. The main text presents only the r@10 threshold; reporting the metric at multiple thresholds (as in Table 17) or discussing caption noise in the main text would make the general-knowledge claim more robust.
  2. [Section 3.2, Eq. (2)] The exact shape of the value transform update is not fully specified: for multi-head attention, is the LoRA update applied per head, to the concatenated value matrix, or to the full attention module? Clarifying the dimensions in Eq. (4) would improve reproducibility.
  3. [Abstract and Section 1] The claimed improvement of '4%–22%' over prior art is not defined as absolute or relative. Please specify the metric and the baseline for this range.
  4. [Section 4.4] The sentence 'we ablate the merging strategy for multi-concept queries in Sec. 4.4' is circular; the merging ablation is actually presented in Table 8 within the same subsection. Reword to reference Table 8.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: POLAR's rank-one update is trained with an independent MSE loss and evaluated on held-out queries; the few self-citations are contextual, and the main caveat is statistical robustness, not circular reasoning.

full rationale

POLAR is presented as an empirical method, and no derivation-from-principles claim is made. Equation (4) defines V'_L,c = V_L + B_L,c A_L,c; Eq. (5) uses an MSE loss to align text embeddings of template queries to frozen CLIP image embeddings of the concept's training images; Eq. (6) is the explicit L2 regularizer on B_L,c with ||A_L,c||_2=1. None of the evaluation quantities appears in these equations: context-query mRR/mAP and concept-only mRR/mAP are computed on the retrieval splits (Sec. 4.2), and the VLM caption recall@10 metric is defined from LLaVA captions of retrieval images, independent of the personalization objective. The regularization's effect of keeping the output close to original CLIP is explicit in Eq. (6), but the paper validates the retention/generalization trade-off empirically (Tabs. 7 and 20) rather than treating it as a derived prediction. The lambda hyperparameter is chosen on the 3-concept validation split (Sec. 4.4); the full TEST split includes those concepts, but the TEST-UNSEEN split (Tab. 9) excludes them and shows the same gains (46.17 vs. 43.88 mRR), so the main result is not forced by construction. The self-citations are non-load-bearing: [38] is explicitly excluded from the main comparisons because it uses a different joint-training setting, and [24] is cited only for the known V* sensitivity in the Limitations. There is no uniqueness theorem imported from the authors' prior work, no fitted parameter renamed as a prediction, and no known result relabeled as a new concept. The remaining weaknesses, single-run ConCon-Chi numbers and the acknowledged frozen-image-encoder bound (Section 16), are statistical robustness and performance limits, not circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or architectural entity; the V* placeholder token and the LoRA update are both drawn from prior work. The free parameters are all hyperparameters tuned on validation splits, and the axioms are standard LoRA and CLIP assumptions plus the frozen-image-encoder premise the authors themselves flag as a limitation.

free parameters (5)
  • Regularization weight λ = λ=0.35 (ConCon-Chi), λ=0.1 (DeepFashion2)
    Tuned on the ConCon-Chi validation split (3 concepts) and chosen per dataset; controls the balance between personalization and general-knowledge retention. For each ablation, the supplementary reports a separate λ sweep.
  • LoRA rank r = 1
    Chosen to represent a single concept with minimal parameters (2d per concept); ablations show ranks 2-16 give no consistent gain.
  • Target layer L = 12 (final transformer layer)
    Selected via validation-split ablations (Table 5) as the best layer for the rank-1 update.
  • Target parameter subset = Value transform V in final attention layer
    Ablation (Table 6) shows value transform outperforms Q, K, O, MLP, and final projection.
  • Placeholder token V* = 'sks'
    A fixed CLIP-vocabulary token used in queries; the authors note sensitivity to this choice in Section 16.
assumptions (4)
  • standard math The low-rank approximation W+BA captures the needed personalized update without corrupting the pretrained mapping (Eq. 2, LoRA).
    Standard LoRA assumption from Hu et al. (2021), relied on here to constrain the update to a rank-1 subspace.
  • domain assumption MSE loss between normalized text and image embeddings aligns the personal concept in CLIP space (Eq. 5).
    This assumes cosine-similarity space is linear enough that pushing text and image embeddings together for a few training pairs transfers to new queries.
  • domain assumption The frozen image encoder's output for the training images is a sufficient representation of the visual concept.
    The method never updates ψI; if image features lack fine-grained detail, retrieval fails (explicitly conceded in Section 16).
  • domain assumption VLM-generated captions (LLaVA) are a valid proxy for measuring general-knowledge retention.
    The new VLM caption recall@10 metric assumes captions are accurate enough to benchmark forgetting; the authors manually checked 100 captions and found 88 accurate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Personalized Search with Regularized Low-Rank Parameter Updates." pith.science (2026). https://pith.science/paper/ZF2VBZ2Q

@misc{pith2026250610182,
  author       = {Pith},
  title        = {Pith review of: Improving Personalized Search with Regularized Low-Rank Parameter Updates},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZF2VBZ2Q}},
  note         = {Machine review of arXiv:2506.10182}
}
read the original abstract

Personalized vision-language retrieval seeks to recognize new concepts (e.g. "my dog Fido") from only a few examples. This task is challenging because it requires not only learning a new concept from a few images, but also integrating the personal and general knowledge together to recognize the concept in different contexts. In this paper, we show how to effectively adapt the internal representation of a vision-language dual encoder model for personalized vision-language retrieval. We find that regularized low-rank adaption of a small set of parameters in the language encoder's final layer serves as a highly effective alternative to textual inversion for recognizing the personal concept while preserving general knowledge. Additionally, we explore strategies for combining parameters of multiple learned personal concepts, finding that parameter addition is effective. To evaluate how well general knowledge is preserved in a finetuned representation, we introduce a metric that measures image retrieval accuracy based on captions generated by a vision language model (VLM). Our approach achieves state-of-the-art accuracy on two benchmarks for personalized image retrieval with natural language queries - DeepFashion2 and ConCon-Chi - outperforming the prior art by 4%-22% on personal retrievals.

Figures

Figures reproduced from arXiv: 2506.10182 by the authors.

Figure 1
Figure 1. Left: Prior works use a pretrained textual inversion net￾work (F) to compute a pseudo-token to represent a new concept, which may be further optimized during personalization. Right: We present POLAR, which represents new concepts as a small low￾rank parameter update within the text encoder. Instead of inserting a learned pseudo-token to the text query, we use a fixed vocabulary token V*. We show that our method is e… view at source ↗
Figure 2
Figure 2. Overview of POLAR. For a personal concept, POLAR learns a rank-1 update to the value transform in the final layer of the text encoder. To maintain general knowledge during personal￾ization, we impose a regularization loss on the update. for the input image. Importantly, composed image retrieval differs from personalized retrieval in that it does not require instance-level recognition of the same concept (e.g., retri… view at source ↗
Figure 3
Figure 3. We compare the top 3 retrievals of our method vs. SEARLE for personal queries in the ConCon-Chi and DeepFashion2 datasets, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Our parameter updates enable personalized retrieval without overriding the model’s general knowledge. On the left we show the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Our method sometimes struggles to differentiate between concepts of the same class with similar visual attributes such as color [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 38 canonical work pages

  1. [1]

    Myvlm: Personalizing vlms for user-specific queries

    Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aber- man, and Daniel Cohen-Or. Myvlm: Personalizing vlms for user-specific queries. In European Conference on Computer Vision, pages 73–91. Springer, 2025. 2, 5

    work page 2025

  2. [2]

    Conditioned and composed image retrieval combining and partially fine-tuning clip-based features

    Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Al- berto Del Bimbo. Conditioned and composed image retrieval combining and partially fine-tuning clip-based features. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 4959–4968, 2022. 2

    work page 2022

  3. [3]

    Effective conditioned and composed im- age retrieval combining clip-based features

    Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Al- berto Del Bimbo. Effective conditioned and composed im- age retrieval combining clip-based features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21466–21474, 2022. 2

    work page 2022

  4. [4]

    Zero-shot composed image retrieval with textual inversion

    Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Al- berto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 15338–15347,

    work page

  5. [5]

    Lasp: Text-to- text optimization for language-aware soft prompting of vi- sion & language models

    Adrian Bulat and Georgios Tzimiropoulos. Lasp: Text-to- text optimization for language-aware soft prompting of vi- sion & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23232–23241, 2023. 3

    work page 2023

  6. [6]

    Plot: Prompt learning with optimal transport for vision-language models

    Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253, 2022. 3

    arXiv 2022

  7. [7]

    this is my unicorn, fluffy

    Niv Cohen, Rinon Gal, Eli A Meirom, Gal Chechik, and Yuval Atzmon. “this is my unicorn, fluffy”: Personalizing frozen vision-language representations. In European confer- ence on computer vision, pages 558–577. Springer, 2022. 1, 2, 3, 4, 5

    work page 2022

  8. [8]

    Variational prompt tuning improves generalization of vision-language models

    Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor Guilherme Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Variational prompt tuning improves generalization of vision-language models. 2022. 3

    work page 2022

Show all 55 references
  1. [9]

    Drea- martist: Towards controllable one-shot text-to-image gen- eration via positive-negative prompt-tuning

    Ziyi Dong, Pengxu Wei, and Liang Lin. Drea- martist: Towards controllable one-shot text-to-image gen- eration via positive-negative prompt-tuning. arXiv preprint arXiv:2211.11337, 2022. 2

    2022 arXiv

  2. [10]

    An image is worth one word: Personalizing text-to- image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash- nik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to- image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022. 2

    2022 arXiv

  3. [11]

    Efros, and Jacob Steinhardt

    Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting clip’s image representation via text-based de- composition. In ICLR, 2023. 7

    2023

  4. [12]

    Clip-adapter: Better vision-language models with feature adapters

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2): 581–595, 2024. 3

    2024

  5. [13]

    Deepfashion2: A versatile benchmark for de- tection, pose estimation, segmentation and re-identification of clothing images

    Yuying Ge, Ruimao Zhang, Xiaogang Wang, Xiaoou Tang, and Ping Luo. Deepfashion2: A versatile benchmark for de- tection, pose estimation, segmentation and re-identification of clothing images. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognitio...

    2019

  6. [14]

    Personal- ized residuals for concept-driven text-to-image generation

    Cusuh Ham, Matthew Fisher, James Hays, Nicholas Kolkin, Yuchen Liu, Richard Zhang, and Tobias Hinz. Personal- ized residuals for concept-driven text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8186–8195, 2024...

    2024

  7. [15]

    Svdiff: Compact param- eter space for diffusion fine-tuning

    Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact param- eter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7323–7334, 2023. 2

    2023

  8. [16]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 2, 3

    2021 arXiv

  9. [17]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023. 3

    2023

  10. [18]

    Personalised clip or: how to find your vacation videos

    Bruno Korbar and Andrew Zisserman. Personalised clip or: how to find your vacation videos. 2022. 2, 5

    2022

  11. [19]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931–1941, 2023. 2, 3, 4

    1931

  12. [20]

    Read-only prompt optimization for vision-language few-shot learning

    Dongjun Lee, Seokwon Song, Jihee Suh, Joonmyeong Choi, Sanghyeok Lee, and Hyunwoo J Kim. Read-only prompt optimization for vision-language few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1401–1411, 2023. 3

    2023

  13. [21]

    The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021. 3

    2021 arXiv

  14. [22]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 5, 3

    2024

  15. [23]

    Prompt distribution learning

    Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5206–5215, 2022. 3

    2022

  16. [24]

    Customizing 9 motion in text-to-video diffusion models

    Joanna Materzynska, Josef Sivic, Eli Shechtman, Antonio Torralba, Richard Zhang, and Bryan Russell. Customizing 9 motion in text-to-video diffusion models. arXiv preprint arXiv:2312.04966, 2023. 4

    2023 arXiv

  17. [25]

    Yo’llava: Your personalized lan- guage and vision assistant.arXiv preprint arXiv:2406.09400,

    Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. Yo’llava: Your personalized lan- guage and vision assistant.arXiv preprint arXiv:2406.09400,

    arXiv

  18. [26]

    Orthogonal adaptation for modular customization of diffusion models

    Ryan Po, Guandao Yang, Kfir Aberman, and Gordon Wet- zstein. Orthogonal adaptation for modular customization of diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 7964–7973, 2024. 2, 8

    2024

  19. [27]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  20. [28]

    Concon-chi: Concept-context chimera benchmark for personalized vision-language tasks

    Andrea Rosasco, Stefano Berti, Giulia Pasquale, Damiano Malafronte, Shogo Sato, Hiroyuki Segawa, Tetsugo Inada, and Lorenzo Natale. Concon-chi: Concept-context chimera benchmark for personalized vision-language tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  21. [29]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  22. [30]

    Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  23. [31]

    Pic2word: Mapping pictures to words for zero-shot composed image retrieval

    Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19305– ...

    2023

  24. [32]

    Collie: Continual learning of language grounding from language-image em- beddings

    Gabriel Skantze and Bram Willemsen. Collie: Continual learning of language grounding from language-image em- beddings. Journal of Artificial Intelligence Research , 74: 1201–1223, 2022. 5, 2

    2022

  25. [33]

    Key-locked rank one editing for text-to-image personaliza- tion

    Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personaliza- tion. In ACM SIGGRAPH 2023 conference proceedings , pages 1–11, 2023. 2

    2023

  26. [34]

    Multimodal few-shot learning with frozen language models

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Es- lami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200–212, 2021. 3

    2021

  27. [35]

    Composing text and image for image retrieval-an empirical odyssey

    Nam V o, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6439–6448, 2019. 2

    2019

  28. [36]

    Dual modality prompt tuning for vision-language pre-trained model

    Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guo- qiang Liang, Peng Wang, and Yanning Zhang. Dual modality prompt tuning for vision-language pre-trained model. IEEE Transactions on Multimedia, 2023. 3

    2023

  29. [37]

    Visual- language prompt tuning with knowledge-guided context op- timization

    Hantao Yao, Rui Zhang, and Changsheng Xu. Visual- language prompt tuning with knowledge-guided context op- timization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6757–6767,

  30. [38]

    Meta-personalizing vision- language models to find named instances in video

    Chun-Hsiao Yeh, Bryan Russell, Josef Sivic, Fabian Caba Heilbron, and Simon Jenni. Meta-personalizing vision- language models to find named instances in video. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19123–19132, 2023. 2, 3, 4, 1

    2023

  31. [39]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. Transactions of the association for computational lin- guistics, 2:67–78, 2014. 4

    2014

  32. [40]

    Inserting anybody in diffusion models via celeb ba- sis

    Ge Yuan, Xiaodong Cun, Yong Zhang, Maomao Li, Chenyang Qi, Xintao Wang, Ying Shan, and Huicheng Zheng. Inserting anybody in diffusion models via celeb ba- sis. arXiv preprint arXiv:2306.00926, 2023. 2

    2023 arXiv

  33. [41]

    Tip- adapter: Training-free adaption of clip for few-shot classi- fication

    Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kun- chang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip- adapter: Training-free adaption of clip for few-shot classi- fication. In European conference on computer vision, pages 493–510. Springer, 2022. 3

  34. [42]

    Conditional prompt learning for vision-language mod- els

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language mod- els. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 16816–16825,

  35. [43]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,

  36. [44]

    Prompt-aligned gradient for prompt tuning

    Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Han- wang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15659–15669, 2023. 3 10 Improving Personalized Search with Regularized Low-Rank Paramete...

    2023

  37. [45]

    However unlike zero-shot methods like SEARLE, we use these 3 validation concepts to select the λ regularization hyperparameter

    Results on ConCon-Chi TEST-UNSEEN split In order to compare to the baselines reported in the orig- inal ConCon-Chi paper [28], we report results on the full TEST split, which contains 3 validation concepts and 17 unseen concepts. However unlike zero-shot methods like SEARLE, w...

  38. [46]

    10 with 5 randomly selected train images for each concept per run

    Standard Error on DeepFashion2 We report the mean and standard error over 5 runs with dif- ferent random seeds on the DeepFashion2 test set in Tab. 10 with 5 randomly selected train images for each concept per run

  39. [47]

    11, LoRA layers (Tab

    Ablation Validation Split Results & Hyper- parameters We provide the ConCon-Chi validation split results and the value for the regularization weight hyperparameterλ for the ablations reported in the main paper: LoRA rank (Tab. 11, LoRA layers (Tab. 12), and LoRA parameters (Ta...

  40. [48]

    [38] Yeh et al

    Comparison to Y ehet al. [38] Yeh et al . [38] propose a textual inversion approach for PerVL that meta-learns a per-class basis on large scale data, over which the V ∗ tokens for new concepts are learned as a linear combination. Both the V ∗ token and basis are up- dated at p...

  41. [49]

    We report results for our method in this setting in Tab

    Single Training Image Experiments on ConCon-Chi The original ConCon-Chi paper [28] also reports results where only a single training image is used per concept. We report results for our method in this setting in Tab. 15. We use the same hyperparameters as our main ConCon-Chi e...

    1940

  42. [50]

    Personalization Time Analysis POLAR is fast to personalize and does not require pretrain- ing. For all experiments in Section 4 (main text), we op- timize for 500 iterations to ensure all variants converge; however for our main method setting (rank=1, layers=12, params=V ,λ=0....

  43. [51]

    puppet” as they are animal-like objects created from household mate- rials, but this is unlikely to align with CLIP’s concept of “puppet

    Additional Implementation Details DeepFashion2. We train our ViT-B/32 model for 50 iter- ations, and our ViT-L/14 model for 200 iterations. We use the Adam optimizer with learning rate 0.001. We use the token “sks” as V ∗. ConCon-Chi. We train our ViT-L/14 model for 500 itera-...

  44. [52]

    Caption this image in 1-2 sentences

    Evaluation of General Knowledge VLM Captions. To generate the captions for calculating our VLM caption recall@10 metric, we prompt LLaV A- 1.5-7B [22] with the image and the prompt “Caption this image in 1-2 sentences.” To assess noise in the captions, we manually checked 100 ...

  45. [53]

    This strategy is similar to weight decay, which also encourages learning small weights, but differs in two key aspects

    Comparison to Weight Decay We regularize our personalized parameter updates via the ||AL,c||2 = 1 constraint and imposing a squared-L2 penalty on BL,c. This strategy is similar to weight decay, which also encourages learning small weights, but differs in two key aspects. First...

  46. [54]

    Generalization of Ablations to DeepFash- ion2 While we report our main ablations on the ConCon- Chi dataset, we observe similar trends on DeepFashion2. Tab. 21 shows ablating the parameters on which the LoRA is learned on DeepFashion2 for a single run of 5 training im- ages. W...

  47. [55]

    sks” in our main experiments. We observe that selecting a V ∗ for which CLIP likely has a strong existing representation (e.g., “dog

    Limitations Like existing approaches in the space of personalized gen- eration that use a fixed V ∗ token in place of new concepts, we experience sensitivity to the choice of V ∗. Similar to prior work [14, 19, 24] we find unique single tokens to be the most effective, and we ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.