arxiv: 2602.03454 · v2 · submitted 2026-02-03 · 💻 cs.CV

Recognition: 2 theorem links

· Lean Theorem

Contextualized Visual Personalization in Vision-Language Models

Yeongtak Oh , Sangwon Yu , Junsung Park , Han Cheol Moon , Jisoo Mok , Sungroh Yoon

Authors on Pith no claims yet

Pith reviewed 2026-05-16 08:18 UTC · model grok-4.3

classification 💻 cs.CV

keywords contextualized visual personalizationvision-language modelspersonalized image captioningreinforcement learningvisual contextVLMsdiagnostic evaluation

0 comments

The pith

CoViP trains vision-language models to link new images to a user's past visual experiences for personalized responses.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper formalizes contextualized visual personalization as the ability of VLMs to recognize and retrieve a user's accumulated visual-textual experiences when processing new images. It centers the solution on personalized image captioning as a core task, then applies reinforcement-learning post-training and caption-augmented generation to strengthen visual-context usage. Diagnostic evaluations are introduced to block textual shortcut solutions and confirm genuine visual grounding. Experiments reveal that both open-source and proprietary VLMs currently fall short on this capability, while the proposed approach delivers gains in captioning and carries over to other personalization tasks.

Core claim

By treating personalized image captioning as the foundational task and applying reinforcement-learning-based post-training together with caption-augmented generation, CoViP enables VLMs to associate new visual inputs with a user's accumulated visual-textual context, as confirmed by diagnostic tests that explicitly exclude reliance on textual patterns alone.

What carries the argument

CoViP, a unified framework that uses reinforcement-learning post-training and caption-augmented generation to ground personalized image captioning in visual context.

If this is right

Personalized image captioning accuracy rises substantially over prior VLM baselines.
Performance improves across multiple downstream personalization tasks without task-specific retraining.
Existing open-source and proprietary VLMs show clear limitations when forced to rely on visual rather than textual cues.
Diagnostic evaluations can be reused to verify visual grounding in future personalization methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same post-training recipe could be applied to video or multi-image memory tasks where users expect models to recall earlier visual episodes.
If the diagnostic tests prove reliable, they could become standard checks for any claim that a VLM is using visual memory rather than prompt leakage.
Scaling the approach might reduce the need for explicit user-provided context in long conversations by letting the model retrieve its own stored visual history.

Load-bearing premise

The reinforcement-learning post-training and caption-augmented generation cause the model to draw on actual visual features rather than learned textual patterns.

What would settle it

A test set in which visual features are altered while textual descriptions stay identical; if personalization accuracy drops sharply, the model was using visuals; if it stays high, it was using textual shortcuts.

Figures

Figures reproduced from arXiv: 2602.03454 by Han Cheol Moon, Jisoo Mok, Junsung Park, Sangwon Yu, Sungroh Yoon, Yeongtak Oh.

**Figure 1.** Figure 1: Qualitative example of the use-case for contextual visual personalization in VLMs. Note that our CoViP effectively responds to the question while integrating the mentioned personal details from the given multimodal contexts. quiring additional task-specific processing. Accordingly, we leverage personalized image captioning as a proxy task to effectively model and learn this shared process. Building on this… view at source ↗

**Figure 2.** Figure 2: Illustration of the proposed personalized image captioning benchmark construction. Furthermore, we observe that hθ is inherently aligned with the objective of personalized image captioning. As captioning is a fundamental generation task that avoids extraneous reasoning (e.g., thinking) steps, the resulting caption s directly reflects the model’s user-specific contextual understanding. Accordingly, we ad… view at source ↗

**Figure 3.** Figure 3: Visualization of diagnostic personalization tasks. the user’s contextual history. Given that the context contains multiple interactions involving the same individual, the model must retrieve all relevant entries and perform temporal reasoning to determine the correct answer. This task, therefore, requires grounding visual input in user-specific history rather than relying on partial matches or surfacele… view at source ↗

**Figure 4.** Figure 4: Results of the human preference evaluation. Here, Win denotes the win rates of CoViP compared to the baseline. human preferences. We consider three strong baselines, and as shown in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

**Figure 6.** Figure 6: analyzes the relationship between recognition and retrieval. As shown in the figure, the average F1 score exhibits a moderate increase across models, whereas MCQA accuracy improves by a substantially larger margin at comparable F1 levels. This indicates that baseline models already achieve reasonable recognition capability, but their low performance under our benchmark probing stems from retrieval as th… view at source ↗

read the original abstract

Despite recent progress in vision-language models (VLMs), existing approaches often fail to generate personalized responses based on the user's specific experiences, as they lack the ability to associate visual inputs with a user's accumulated visual-textual context. We newly formalize this challenge as contextualized visual personalization, which requires the visual recognition and textual retrieval of personalized visual experiences by VLMs when interpreting new images. To address this issue, we propose CoViP, a unified framework that treats personalized image captioning as a core task for contextualized visual personalization and improves this capability through reinforcement-learning-based post-training and caption-augmented generation. We further introduce diagnostic evaluations that explicitly rule out textual shortcut solutions and verify whether VLMs truly leverage visual context. Extensive experiments demonstrate that existing open-source and proprietary VLMs exhibit substantial limitations, while CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks. These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

CoViP formalizes contextualized visual personalization for VLMs and combines RL post-training with caption augmentation to target it, but the diagnostics leave open whether gains truly come from visual context rather than textual patterns.

read the letter

The paper's core move is to name contextualized visual personalization as a distinct requirement for VLMs: the model must link a new image to a user's accumulated visual-textual history rather than treating each input in isolation. CoViP treats personalized captioning as the central training task, applies reinforcement learning post-training, and adds caption augmentation during generation, then layers on diagnostic checks meant to block pure textual shortcuts. This is a straightforward extension of existing RL fine-tuning ideas to a user-specific setting, and the motivation lands because real applications like personal photo tools do need this kind of memory. The diagnostics are a positive step in principle; trying to verify that the model depends on the image rather than learned text correlations shows some care about evaluation pitfalls. What remains soft is the strength of that verification. The abstract asserts the diagnostics rule out textual shortcuts, yet without concrete details on negative-example construction, text-only controls, or quantitative thresholds, it is possible the reported downstream gains still trace to improved language modeling from the augmentation step. No numbers or error bars appear in the summary, so effect sizes and robustness are hard to judge from the available text. The work sits in the practical VLM adaptation space. Readers who already run RL post-training on vision-language models will recognize the pieces and may find the specific framing and diagnostics useful as a starting point, even if they want tighter controls on the visual-dependency claim. It deserves peer review because the problem is clearly stated and the method is reproducible in outline; referees can check the actual diagnostic implementations and numbers rather than desk-rejecting on motivation alone.

Referee Report

1 major / 1 minor

Summary. The paper formalizes contextualized visual personalization for VLMs, proposes the CoViP framework that uses personalized image captioning as the core task improved via RL-based post-training and caption-augmented generation, introduces diagnostic evaluations claimed to rule out textual shortcuts, and reports that existing VLMs are limited while CoViP yields gains on captioning and downstream personalization tasks.

Significance. If the diagnostic evaluations genuinely isolate visual context use and the reported gains are substantial, reproducible, and not reducible to improved language modeling alone, the work would meaningfully advance personalized VLM capabilities beyond generic responses.

major comments (1)

[Diagnostic Evaluations] Diagnostic Evaluations section: the claim that the introduced diagnostics 'explicitly rule out textual shortcut solutions' is load-bearing for the central claim that CoViP produces genuine visual-context use rather than learned textual patterns from caption augmentation; however, the manuscript provides no concrete construction details for negative examples, no text-only ablation results on the same images, and no quantitative threshold demonstrating visual dependency, leaving open the possibility that downstream gains are explained by language modeling improvements alone.

minor comments (1)

[Abstract] Abstract and results sections: no quantitative metrics, error bars, or baseline comparisons are visible in the provided abstract, which undermines the ability to assess the magnitude of the claimed 'substantial limitations' in existing VLMs and 'holistic gains' with CoViP.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback on our diagnostic evaluations. We address the single major comment below and will revise the manuscript to incorporate additional details and results.

read point-by-point responses

Referee: [Diagnostic Evaluations] Diagnostic Evaluations section: the claim that the introduced diagnostics 'explicitly rule out textual shortcut solutions' is load-bearing for the central claim that CoViP produces genuine visual-context use rather than learned textual patterns from caption augmentation; however, the manuscript provides no concrete construction details for negative examples, no text-only ablation results on the same images, and no quantitative threshold demonstrating visual dependency, leaving open the possibility that downstream gains are explained by language modeling improvements alone.

Authors: We agree that the current presentation of the diagnostics would benefit from greater specificity to fully support the claim of ruling out textual shortcuts. In the revised manuscript we will expand the Diagnostic Evaluations section with: (1) explicit construction details and examples for the negative cases designed to block textual pattern exploitation; (2) text-only ablation results run on the identical image sets, quantifying the performance drop when visual input is removed; and (3) quantitative thresholds or dependency metrics (e.g., accuracy delta and visual-attention scores) that demonstrate reliance on visual context rather than language-modeling gains alone. These additions will directly address the concern and strengthen evidence that CoViP improvements arise from genuine visual-context utilization. revision: yes

Circularity Check

0 steps flagged

No circularity: standard RL post-training and new diagnostics applied to a formalized task

full rationale

The paper formalizes contextualized visual personalization as a new challenge, then applies reinforcement-learning post-training and caption-augmented generation to the core task of personalized image captioning. Diagnostic evaluations are introduced to check for visual context use versus textual shortcuts. No equations, parameters, or claims reduce by construction to fitted inputs or self-definitions; results are shown via experiments on downstream tasks using external benchmarks. No load-bearing self-citations, uniqueness theorems, or ansatz smuggling appear in the derivation chain. The framework is self-contained against standard VLMs and RL techniques.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No explicit free parameters, axioms, or invented entities are described in the abstract; the work relies on standard vision-language model architectures and reinforcement learning methods from prior literature.

pith-pipeline@v0.9.0 · 5485 in / 1008 out tokens · 24578 ms · 2026-05-16T08:18:23.596792+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We propose CoViP, a unified framework that treats personalized image captioning as a core task... through reinforcement-learning-based post-training and caption-augmented generation... diagnostic evaluations that explicitly rule out textual shortcut solutions
IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We design diagnostic evaluation tasks... Last-Seen Detection (LSD), Last-Action Recall (LAR), Instruction-Triggered Recall (ITR)

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Omni-Persona: Systematic Benchmarking and Improving Omnimodal Personalization
cs.CV 2026-05 unverdicted novelty 7.0

Omni-Persona benchmark with 18 tasks shows open-source models have audio-visual grounding gaps, RLVR narrows them but leads to conservative outputs, and scale or recall alone fail as diagnostics.

Reference graph

Works this paper leans on

44 extracted references · 44 canonical work pages · cited by 1 Pith paper · 1 internal anchor

[1]

Object Hallucination in Image Captioning

Springer, 2024. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pp. 8748–8763. PmLR, 2021. Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, ...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[2]

I still remember the day I saw John at Lake Francesborough on 2025-09-09

Last-Seen Detection (LSD)In the LSD task, each dialogue contains an explicit reference to when and where the user encountered the individual, e.g.,“I still remember the day I saw John at Lake Francesborough on 2025-09-09. ” Given a new query image, the user asks:“Where did I last see the person in this image?” To answer correctly, the model must identify ...

work page 2025
[3]

Oh wait, I need to go to the post office to return this package

Last-Action Recall (LAR)LAR extends LSD by requiring recall of a finer-grained personal action rather than a location. For each context, we append an additional user utterance to thelast-seen dialogueof the query individual, describing a specific action, e.g.,“Oh wait, I need to go to the post office to return this package. ” The action is randomly sample...

work page
[4]

If this person ever shows up again, remind me by saying the keyword SKS

Instruction-Triggered Recall (ITR)ITR evaluates a more proactive form of personalization. In this task, the last-seen dialogue includes an instruction of the form:“If this person ever shows up again, remind me by saying the keyword SKS. ” At inference time, the user asks a generic question such as:“Where did I last see the person in this image?”without ex...

work page 2024
[5]

Conversely, optimizing only rcaps also yields consistently weaker results, suggesting retrieval signals without fine-grained visual supervision are inadequate

Necessity of joint supervision.Training with only rvis (without rcaps) degrades performance, in some cases falling below the Qwen3-VL-8B baseline, indicating that visual supervision alone is insufficient for personalized image captioning. Conversely, optimizing only rcaps also yields consistently weaker results, suggesting retrieval signals without fine-g...

work page
[6]

F1-based vision VR vs. binary consistency VR.Replacing RePIC’s object-consistency VR (OCT), which provides binary correctness feedback, with our set-based F1 VR rvis consistently improves performance on positive accuracy. This indicates that set-level supervision provides a denser and more robust learning signal for multi-concept perception

work page
[7]

Effect of increasing the number of positive concepts.Varying the number of positive concepts included during 23 Contextualized Visual Personalization in Vision-Language Models (a) Category Accuracy (%)(b) Per TaskAccuracy (%) MMIU Figure S.10.Results on MMIU (11K MCQs across 52 tasks), which evaluates multi-image relational understanding spanning temporal...

work page 2025
[8]

Any concept image is occluded or not fully visible in the query image

work page
[9]

Any concept does not appear in the query image at all

work page
[10]

yes" only when every concept appears in the query image. Carefully examine the images and output the final result only as

Any object or person in the query image is significantly different from the corresponding concept image. [Answering rule] Output "yes" only when every concept appears in the query image. Carefully examine the images and output the final result only as"yes"or"no". 28 Contextualized Visual Personalization in Vision-Language Models Table S.11.Showcase of a u...

work page
[11]

This looks like Pino again, perhaps older than in the park photo from Busan Station

Recall and reuse detailsfrom the previous dialogues (object names, appearances, places, times, and relationships). – Treat the previous dialogues as long-term memory. – If an object in the new image appears similar to one mentioned in the past, refer to it using the same name and contextual background. 2.Ground your description in the new image’s visual c...

work page
[12]

Keep your tone natural and human-like, as if describing something familiar to the same user

work page
[13]

Do not restate previous dialogues verbatim; instead, synthesize and extend them with new image- grounded observations

work page
[14]

6.Use only relevant memories

Write in paragraph form, not in a dialogue format. 6.Use only relevant memories. – If an object or scene from the previous dialogues doesnot appear in the new image, ignore it completely. – Include contextual information only for objects that actually appear. – Avoid unrelated names, locations, or events. 29 Contextualized Visual Personalization in Vision...

work page
[15]

• Do not invent or alter the name

The main object’s name ({name}) must be used consistently throughout the dialogue. • Do not invent or alter the name

work page
[16]

last summer at the riverside,

The user should describe a personal experience related to{name}. • The experience must include at least oneobjective contextual element, such as a specificplace, time,event, orsituation(e.g., “last summer at the riverside,” “during my first year in college,” “in my grandmother’s backyard”)

work page
[17]

The model should respond naturally and empathetically — acknowledging, asking gentle questions, or adding brief reflections

work page
[18]

Keep the tone human-like, calm, and realistic — not overly emotional or robotic

work page
[19]

The conversation should have6 turns total(User→Model→User→Model→User→Model)

work page
[20]

Focus on thepersonal connectionandshared observa- tionof the object

Avoid encyclopedic or factual world knowledge. Focus on thepersonal connectionandshared observa- tionof the object. [Output Format] Dialogue: User: ... Model: ... User: ... Model: ... User: ... Model: ... 30 Contextualized Visual Personalization in Vision-Language Models Table S.13.Visualization of a prompt used to generate MCQA pairs from the dialogue. M...

work page
[21]

Each question must target an objective detail present in the conversation (e.g., name, place, time, habit/action)

work page
[22]

Avoid emotions, opinions, or meta-dialogue

work page
[23]

Each question must have exactly 3 options: A, B, C

work page
[24]

Exactly one option is correct among A, B, C

work page
[25]

Make the wrong options (A/B/C except the correct one) plausible but clearly incorrect

work page
[26]

DoNOTrequire external/world knowledge; answers must come from the conversation content

work page
[27]

qa": [ {

Output must be valid JSON only: no additional text and no trailing commas. [JSON Output Schema] { "qa": [ { "id": "Q1", "question": "<string>", "options": { "A": "<string>", "B": "<string>", "C": "<string>" }, "correct_answer": "A" | "B" | "C" }, { "id": "Q2", "question": "<string>", "options": { ... }, "correct_answer": "A" | "B" | "C" }, { "id": "Q3", "...

work page
[28]

Read the description carefully

work page
[29]

• Ifnoneof A/B/C can be confirmed from the description, choose D

For each question, choose thesingle bestoption: • If one of A/B/C is explicitly or clearly supported by the description, choose that option. • Ifnoneof A/B/C can be confirmed from the description, choose D

work page
[30]

You must ignore any information that is not in the description

work page
[31]

• Then, on a separate line, output the final choice in the exact format: [Required output format] Answer:\boxed{X} whereXis one ofA, B, C, or D

For each question: • You may briefly explain your reasoning in natural language. • Then, on a separate line, output the final choice in the exact format: [Required output format] Answer:\boxed{X} whereXis one ofA, B, C, or D. Inside\boxed{}there must beexactly one letter, with no extra text. [Given] •[Description]{Generated caption} •[Question]{Pre-define...

work page
[32]

• Do not invent, alter, or omit the name

The person’s name ({name}) must be used consistently throughout the dialogue. • Do not invent, alter, or omit the name

work page
[33]

• The experience must include at least one concreteevent or situation(e.g., bumping into them, having a short conversation, noticing what they were doing)

The user must describe apersonal experiencerelated to{name}. • The experience must include at least one concreteevent or situation(e.g., bumping into them, having a short conversation, noticing what they were doing). • It should include at least onesensory or situational detailthat makes the memory feel realistic

work page
[34]

I saw them on{seen date}at{seen place}

The user must explicitly mentionboththe date and the place: • Date:{seen date} • Place:{seen place} • Preferably within a single user turn (e.g., “I saw them on{seen date}at{seen place}...”)

work page
[35]

• Do not introduce new factual information beyond what the user provides

The model should respond naturally and empathetically, acknowledging the user’s experience or asking gentle follow-up questions. • Do not introduce new factual information beyond what the user provides

work page
[36]

Avoid encyclopedic or factual descriptions

Keep the tone calm, realistic, and human-like. Avoid encyclopedic or factual descriptions

work page
[37]

[Output Format] Dialogue: User:

The conversation must haveexactly 6 turnsin total: User→Model→User→Model→User→Model. [Output Format] Dialogue: User: ... Model: ... User: ... Model: ... User: ... Model: ... 34 Contextualized Visual Personalization in Vision-Language Models Table S.17.Prompt visualization used for diagnostic downstream tasks. Personalized Image Understanding Prompt: You a...

work page
[38]

This looks like Pino again, now indoors instead of the park near Busan Station

Recall and reuse detailsfrom the previous dialogues (object names, appearances, places, times, and relationships). • Treat the previous dialogues as your long-term memory. • If an object in the new image appears similar to one mentioned in the past, refer to it with the same name and contextual background. 2.Ground your understanding in the new image’s vi...

work page
[39]

Keep your tone natural and human-like — as if you are interpreting something familiar to the same user

work page
[40]

Instead, synthesize memory with the current image content

Do not restate previous dialogues verbatim. Instead, synthesize memory with the current image content

work page
[41]

Oh wait, I think I left my wallet at the Guess store, so I’m going back to check

Write inparagraph form, not in a dialogue format. 6.Use only relevant memories. • If an object or context from past dialogues doesnotappear in the new image, ignore it completely. • Add contextual information only when it helps understanding of what is visible. • Avoid mentioning unrelated names, locations, or experiences. 35 Contextualized Visual Persona...

work page
[42]

A question asked to the model

work page
[43]

A ground-truth reference answer (GT)

work page
[44]

Evaluation Criteria: • The generated response isCorrectif it semantically includes the core information conveyed by the ground-truth reference

A generated response from the model Your task is to decide whether the generated response isCorrectorWrong. Evaluation Criteria: • The generated response isCorrectif it semantically includes the core information conveyed by the ground-truth reference. • The wording does NOT need to match exactly. Paraphrases, rephrasings, or additional details are allowed...

work page