{"id":"0d8977a3-5444-4f6f-a785-941dee725706","arxiv_id":"2506.05198","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning VLMs on image-only or text-only personas yields a significant, asymmetric cross-modal memorization gap that persists with model scale, unlearning, and multi-hop reasoning.","lead":"This paper measures how vision-language models memorize facts learned in one modality and recall them in the other, using synthetic personas with paired images and text descriptions. It finds that knowledge transfers between image and text but with a persistent, asymmetric gap, which matters for privacy and personalization in multimodal AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The asymmetry claim relies on uncalibrated image and text probes; if one modality is intrinsically easier, the gap is task difficulty, not transfer.","rationale":"The reader's weakest assumption identifies the same uncalibrated-probe concern, and it is indeed the most load-bearing issue: the paper's headline asymmetry claim depends entirely on interpreting the source-target accuracy gap as a transfer property. Without a human baseline or difficulty-matched control, the observed gap could be an artifact of probe sensitivity. The paper's other limitations—small test set (100 personas, one held-out image/description), surrogate unlearning via retraining with different names, and membership-inference AUC without confidence intervals—are real but secondary; they weaken precision and generalizability but do not threaten the core interpretation as directly. The proposed test is feasible: human raters or an off-the-shelf VLM can provide per-modality baseline accuracy on the same probes, allowing a matched-difficulty analysis. If the gap persists after calibration, the conditional acceptance would be well justified; if not, the central claim would need to be revised from an asymmetric transfer effect to a task-difficulty effect. Since the reader already conditioned acceptance on precisely this check, the verdict should remain unchanged.","tokens_in":12330,"tokens_out":3618,"duration_ms":43844,"concrete_test":"Collect human (or strong pre-trained VLM) accuracy on the same 100 held-out images and 100 held-out descriptions, asking respondents to identify the correct persona name among the same 100-name candidate set. If the two modalities yield different baseline accuracies, calibrate the transfer comparison by either (a) selecting a difficulty-matched subset of descriptions/images with equal human accuracy, or (b) computing a per-probe difficulty index and re-fitting the transfer curves. Then recompute the image-to-text versus text-to-image gap. If the gap shrinks below statistical significance (e.g., a 100-name multinomial with appropriate confidence intervals), the asymmetry claim is not supported; if it remains, the claim survives calibration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—that cross-modal transfer is asymmetric, with image-to-text more effective than text-to-image—rests on comparing test accuracy on held-out images versus held-out descriptions without establishing that the two probes are equally sensitive to the underlying fact. Training on descriptions yields source accuracy near 1.0 and image target accuracy around 0.3 (Figure 3a), while training on images gives description target accuracy around 0.4 (Figure 3b). The paper interprets the 0.1 gap as an asymmetry in transfer, but it could equally reflect that recognizing a name from a visual appearance is intrinsically more difficult than from a verbose text description, for any model. No human baseline, no matched-difficulty subset, and no mixed-modality training control is reported. Because the asymmetry is explicitly invoked in the abstract, conclusion, and takeaways 1 and 3, this confound is load-bearing: if the probes are not equated, the claimed direction of the gap is not identified. The issue enters at Section 3.3 (Figures 2–4) and propagates to the unlearning and multi-hop interpretations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a synthetic persona dataset of 100 fictional individuals, each with 100 training image variants, 100 training description variants, and one held-out test image and test description. The authors fine-tune Gemma-3 vision-language models with LoRA on either the image or the description alone, then evaluate name-recall accuracy on held-out inputs in both modalities. They report that factual knowledge transfers across modalities but with a source-target accuracy gap, that the gap is asymmetric (image-to-text transfer is more effective than text-to-image), and that the gap persists across model sizes, a re-labeling-based unlearning procedure, and a multi-hop favorite-number task. They also propose an in-distribution image-caption augmentation that improves transferability, and they report a membership-inference experiment showing that a VLM can unintentionally memorize a task-irrelevant favorite number overlaid on training images.","tokens_in":12494,"tokens_out":7487,"duration_ms":88897,"significance":"If the empirical claims hold, this is a useful and timely contribution to the study of memorization and privacy in multimodal models. The controlled canary-style dataset, the held-out evaluation protocol, and the systematic variation of training modality, model size, and training diversity are genuine strengths. The paper makes falsifiable predictions (e.g., the direction of the cross-modal gap, the ineffectiveness of OOD augmentation, the presence of membership-inference signal for visual PII) that other groups can test. The main weakness is that the central asymmetry claim is not yet identified because the two modality probes are not calibrated against each other.","major_comments":[{"comment":"The paper's central claim that cross-modal transfer is asymmetric (image-to-text more effective than text-to-image) rests on comparing the target accuracy of a description-trained model on held-out images (around 0.3 in Figure 3a) with the target accuracy of an image-trained model on held-out descriptions (around 0.4 in Figure 3b). These two probes are not calibrated against each other. Recognizing a name from a single photograph may be intrinsically harder than recognizing it from a verbose, deliberately disambiguating description, regardless of training modality. Without a human baseline, a matched-difficulty subset, or a mixed-modality training control that equates per-probe difficulty, the observed gap does not identify an asymmetry in transfer; it may simply reflect task difficulty. Because the asymmetry is invoked in the abstract, in the conclusion, and in Takeaways 1 and 3, this is a load-bearing confound. I ask the authors to add a probe-difficulty calibration (e.g., human accuracy on the same test items, or an internal control where a model trained on both modalities is measured on both probes) and to qualify the asymmetry claim until such calibration is provided.","section":"§3.3, Figures 2–4"},{"comment":"All experimental curves and tables are reported as single-run point estimates. With 100 personas, the standard error of a reported 0.30 accuracy is about 4.6 percentage points, so a 10-point cross-modal gap is roughly two standard errors. Claims that the transferability rate is 'nearly unchanged' across model sizes (Takeaway 3) and that unlearning effects are modality-dependent (Takeaway 5) depend on the shapes and slopes of these curves. I request that the authors run at least three seeds per configuration, report means with error bars or confidence intervals, and, where they assert that a slope or rate is unchanged, provide a quantitative fit with confidence intervals rather than visual inspection.","section":"§3.3–§4, Figures 2–8 and Table 1"},{"comment":"The ID augmentation is described as adding 'held-out synthetic persona images and descriptions' to the training set. This wording is ambiguous and potentially problematic: the dataset design reserves one distinct test image and one test description per persona, and if those reserved test items are used as augmentation, the mitigation result in Figure 5 is invalid by construction. Please clarify whether the augmented pairs come from the unused training-pool variants or from the actual test items, state how many pairs are added, and specify the auxiliary captioning objective used for these pairs. If the augmented data are from the training pool, the 'held-out' terminology should be changed.","section":"§3.4, Figure 5"},{"comment":"The unlearning procedure is not machine unlearning in the usual sense; it is continued training on the same inputs with completely re-assigned names. The conclusion that unlearning in a cross-modal setting does not transfer may be an artifact of this overwriting procedure rather than a property of forgetting: for example, the image model may require more epochs to overwrite a visually learned association, producing the observed asymmetry in relative accuracy drop. I recommend either adopting a more standard unlearning method (e.g., gradient ascent on the forget set, with the safeguards mentioned in the text) or explicitly re-framing the section as 'continued training with re-labeled data' and adjusting the strength of Takeaway 5 accordingly.","section":"§3.5, Figure 6"}],"minor_comments":[{"comment":"There is a typo in the opening paragraph: 'or or their textual descriptions' should read 'or their textual descriptions'.","section":"§3.1"},{"comment":"Please report the chance accuracy for the name-recall task (1/100) as a reference point; the target accuracies around 0.3–0.4 are clearly above chance, but the reader cannot assess effect sizes without this baseline.","section":"§3.2"},{"comment":"The term 'transferability rate' is not defined quantitatively; the text infers a slope from a few points without any fitted line. Please either define the rate and report its estimated value with uncertainty, or use a different expression such as 'the accuracy curves remain roughly parallel.'","section":"§3.3, Figure 4"},{"comment":"The row and column structure of Table 1 is difficult to parse because the Training Data columns (Base Data, Multi-Hop Data) are not visually separated from the Test Type columns. Reformating with subheaders or separate panels for Desc⇒Name and Image⇒Name base data would greatly improve readability.","section":"Table 1"},{"comment":"There is a typo in the title of Figure 7 ('Unintential Memorization'), and the AUC of about 0.7, while above chance, is a modest signal; please state explicitly that this is a single-run estimate and report a confidence interval.","section":"§4, Figure 8"}],"recommendation":"major_revision","confidential_remarks":"This is a well-motivated and clearly written empirical study, and I believe the core observation—that cross-modal transfer exists but is incomplete in VLMs—is likely to be of interest to the journal's readership. The main risk is that the paper's flagship asymmetry claim is not yet identified because the two modality probes are not calibrated. If the authors add a calibration control and report variance across seeds, the contribution would be solid. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is worth a serious referee, but its central asymmetry claim is not yet nailed down. The authors build a controlled synthetic persona dataset (100 identities, 100 image and text variants each) and show that fine-tuned VLMs recall a person's name in the training modality far better than in the held-out modality. That gap is real and interesting. The dataset and the basic result are the paper's real contribution.\n\nWhat the paper does well: it extends memorization studies from unimodal to cross-modal settings, tests across three model sizes, and documents that the gap persists with scale. It also shows that in-distribution image-caption augmentation helps transfer while OOD data does not, which is a useful, actionable finding. The multi-hop curse and the membership-inference result for incidental visual PII are nice additions, even if preliminary.\n\nThe soft spots are significant. The stress-test note is on target: the asymmetry claim (image-to-text better than text-to-image) compares accuracy on held-out images against accuracy on held-out descriptions without any calibration that the two probes are equally sensitive. The descriptions are long, attribute-rich paragraphs generated by Gemini; the images are synthetic close-up photos. Recognizing a name from a text that explicitly lists hair color, glasses, and clothing is plausibly easier than from a photo, regardless of transfer. So the observed directional gap could be task difficulty, not a property of cross-modal memorization. The authors never report a human baseline, a matched-difficulty subset, or a mixed-modality training control. That is a load-bearing omission because the asymmetry is repeated in the abstract, conclusion, and takeaways.\n\nSecond, there are no error bars, no multiple seeds, and the test set is just 100 held-out items per modality. Claims that the transferability rate is unchanged with scale (Takeaway 3) and that unlearning is modality-dependent (Takeaway 5) rest on single-run curves. The unlearning experiments also use a surrogate procedure—retraining with swapped names—rather than a real unlearning method; the authors acknowledge this, but the results should be labeled more clearly as a proxy. The membership-inference AUC (~0.7) is modest and the injected numbers are synthetic, so the privacy implications are suggestive, not demonstrated.\n\nWhere does this leave us? The core observation of a cross-modal gap holds up; the direction of the asymmetry and several secondary claims do not, yet. The right fix is addressable: calibrate the two probes (e.g., human accuracy on the same test items, or logistic-regression baselines on image and text features), run multiple seeds, and report error bars. The authors should also release data and code.\n\nI'd bring this to a reading group to discuss experimental design and confounding. I would not cite the asymmetry result in its current form, but the benchmark itself is citable. A serious editor should send this to peer review with a request for major revision.","headline":"A solid empirical study with a new synthetic benchmark, but the headline asymmetry claim is confounded by uncalibrated probe difficulty and single-run evidence.","tokens_in":13038,"tokens_out":2286,"would_cite":true,"duration_ms":30496,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision-language models transfer facts across modalities, but the transfer is uneven and asymmetric.","keywords":["cross-modal memorization","vision-language models","factual knowledge transfer","membership inference","machine unlearning","multi-hop reasoning","synthetic persona dataset","privacy leakage"],"falsifier":"Run the same training and evaluation protocol with a matched-difficulty control: have human annotators (or a strong probe model) achieve equal accuracy on name-from-image and name-from-description for the same persona set, and if the VLM still shows an asymmetric gap then the asymmetry is real. Conversely, if the probes are equally difficult and a VLM trained on a single image variant achieves equal source and target accuracy at any epoch, the central claim of an irreducible cross-modal gap would be disproven.","tokens_in":12121,"feed_emoji":"🧠","tokens_out":3979,"duration_ms":46319,"temperature":0.7,"pith_summary":"The paper studies how vision-language models (VLMs) memorize factual knowledge when trained on only one modality — either a photo of a person or a textual description — and then are tested on the other. Using a synthetic dataset of 100 personas with paired images and descriptions, it finds that facts learned in one modality do transfer to the other, but accuracy in the target modality lags behind accuracy in the source modality. The gap is asymmetric: training on images transfers to text more effectively than training on text transfers to images, and this asymmetry persists in larger models, in machine unlearning attempts, and in multi-hop reasoning. The paper also shows that VLMs unintentionally memorize irrelevant visual private information, such as a 'favorite number' overlaid on an image, detectable through membership inference. These findings matter for building trustworthy multimodal assistants and for privacy protection when models are trained on personal photos and messages.","feed_headline":"Image-trained VLMs recall facts in text better than text-trained ones","feed_subtitle":"Synthetic persona study shows the asymmetric transfer gap persists in bigger models and survives unlearning attempts.","key_machinery":"The central object is a synthetic persona dataset, a set of 'canaries' with full experimental control: 100 unique personas, each with a fixed name, a favorite number (acting as surrogate PII), a set of visual attributes, 100 training images and 100 training textual descriptions plus one held-out image and description for testing. The evaluation protocol trains a VLM on one modality via LoRA fine-tuning, then measures name-recall accuracy on held-out inputs of both modalities, comparing source accuracy with target accuracy to quantify the cross-modal transfer gap. For the privacy experiment, the machinery is a membership-inference attack: for each test image, the model's loss is computed with the ground-truth favorite number overlaid versus a random one, and the AUC of the resulting ROC curve measures unintentional memorization.","core_discovery":"The paper's central claim is that cross-modal factual memorization in VLMs is real but incomplete and systematically imbalanced. In controlled fine-tuning experiments, a model trained on descriptions of a person can often name them from a held-out description, but struggles more when shown a photo; a model trained on photos shows better transfer to text. This asymmetry — image-to-text transfer stronger than text-to-image — holds across model sizes within the Gemma-3 family, meaning scaling does not fix the transfer gap. The paper further shows that cross-modal unlearning is incomplete: removing a fact in the modality it was learned leaves the other modality largely intact. In multi-hop settings, models trained on description-to-name and name-to-favorite-number fail to answer description-to-favorite-number, a cross-modal variant of the 'two-hop curse.' Finally, by injecting a three-digit 'favorite number' into training images, the paper demonstrates unintentional memorization of task-irrelevant visual PII, with membership-inference AUC reaching about 70% after 100 epochs even though direct extraction succeeds almost never.","pith_inferences":["The asymmetric transfer (image→text better than text→image) might reflect an asymmetry in information content: a photo is a single concrete rendering, whereas a textual description is an abstract, lossy summary; a controlled experiment equalizing information content would separate this from a genuinely modality-specific learning bias.","The membership-inference signal at 70% AUC, despite near-zero direct extraction, suggests that current alignment or safety training does not prevent latent memorization of visual PII; testable extensions include probing whether the signal survives different unlearning methods or stronger data augmentation.","The finding that OOD augmentation fails to improve transfer implies that generic web-scale multimodal pretraining may not automatically make downstream fine-tuning more cross-modally robust; future work could test whether mixing on-distribution synthetic pairs at scale closes the gap.","The multi-hop failure being more severe for description-trained models hints that image-based training may produce more disentangled or visual-anchored representations; a representational similarity analysis between the two training regimes could directly test this hypothesis."],"forward_implications":["Applications like personal assistants that learn from a user's photos or text messages will be less reliable when asked to recall facts in the modality they were not trained on, and this shortfall cannot be cured simply by scaling the model.","Privacy-removal procedures (machine unlearning) must be applied to every modality separately; unlearning a fact from text does not remove it from the model's visual memory of the same person.","Augmenting training data with in-distribution synthetic image-text pairs improves cross-modal transfer, whereas out-of-distribution data (e.g., generic COCO captioning) does not, suggesting that data diversity must match the target distribution to help.","VLMs can memorize task-irrelevant visual PII, so data filtering and redaction pipelines should scrub incidental sensitive content from images even when that content is unrelated to the training objective.","The cross-modal multi-hop curse implies that compositional knowledge (A→B, B→C) does not automatically yield A→C across modalities, which limits reasoning in multimodal knowledge-intensive tasks."],"supporting_citations":[{"why":"Establishes the canary-injection methodology for measuring unintended memorization, which the synthetic persona dataset adapts to the multimodal setting.","marker":"[Carlini et al., 2019]"},{"why":"Shows that memorization scales with model size and data repetition, motivating the model-size comparison and the multi-epoch training design.","marker":"[Carlini et al., 2022]"},{"why":"Demonstrates that paraphrased training variants help knowledge learning, directly motivating the image/description variants experiments.","marker":"[Allen-Zhu and Li, 2023]"},{"why":"Defines the two-hop curse in LLMs, which the paper extends to the cross-modal multi-hop learning experiments.","marker":"[Balesni et al., 2025]"},{"why":"Supplies the text-to-image generation model (Imagen 3) used to create the synthetic persona images.","marker":"[Baldridge et al., 2024]"},{"why":"Provides the Gemini model used to generate natural-language descriptions of the synthetic personas.","marker":"[Team et al., 2023]"},{"why":"The Gemma-3 technical report describes the base VLMs (4B, 12B, 27B) that are fine-tuned in all experiments.","marker":"[Team et al., 2025]"},{"why":"The COCO dataset is used as the source of out-of-distribution image-caption pairs in the augmentation mitigation experiments.","marker":"[Lin et al., 2014]"}],"fun_headline_variants":["Image-to-text transfer outperforms text-to-image in VLMs","VLM transfer gap persists despite scaling and unlearning","Cross-modal unlearning fails: facts persist in other modality","VLMs leak personal info from images with low extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the accuracy gap reflects a property of cross-modal memorization assumes that the image and text test sets are equally sensitive probes of the same underlying fact; if recognizing a person from a photo is intrinsically easier or harder than from a description, the measured asymmetry could be an artifact of task difficulty rather than a genuine transfer asymmetry.","fun_headline_variants_meta":{"raw":{"variants":["Image-to-text transfer outperforms text-to-image in VLMs","VLM transfer gap persists despite scaling and unlearning","Cross-modal unlearning fails: facts persist in other modality","VLMs leak personal info from images with low extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000833,"raw_usage":{"total_tokens":3651,"prompt_tokens":975,"completion_tokens":2676,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":2610}},"tokens_in":591,"tokens_out":2676,"duration_ms":23470,"temperature":1.0,"reasoning_tokens":2610,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:22:50.731866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same training and evaluation protocol with a matched-difficulty control: have human annotators (or a strong probe model) achieve equal accuracy on name-from-image and name-from-description for the same persona set, and if the VLM still shows an asymmetric gap then the asymmetry is real. Conversely, if the probes are equally difficult and a VLM trained on a single image variant achieves equal source and target accuracy at any epoch, the central claim of an irreducible cross-modal gap would be disproven.","supporting_citations":[],"review_version":1}