{"id":"7139641a-3a48-4567-be05-f3b1bd8bff8e","arxiv_id":"2505.23121","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ContextQFormer adds a queue of [CLS] image and text features to a Q-Former style module and reports a 4.2% available-rate gain over LoRA-only on the new TMDialog benchmark.","lead":"This paper introduces ContextQFormer, a memory-queue module for multi-turn multimodal chat models, plus a new long-context multimodal dialogue dataset called TMDialog. On its own 329-sample benchmark, ContextQFormer reports a 68.17% available rate versus 64.01% for a LoRA-only ablation, a gain of about four points.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The available-rate comparison is scored by GPT-4 from text descriptions rather than the images themselves, so the claimed 2%-4% improvement may reflect description coverage rather than genuine visual grounding; the key LoRA-only gap also has no reported uncertainty.","rationale":"The reader identified the image-description evaluation as the weakest assumption, and I agree. The paper's headline claim is an empirical improvement in available rate, but the only supporting measurement is a GPT-4 judge that never sees the images. Since hallucination is defined relative to the image, this is a direct validity threat to the dependent variable. The concern is not about disagreement with consensus or about the authors' intentions; it is about whether the measured quantity corresponds to the claimed quantity. The proposed test would settle the issue by re-scoring a subset with actual image access and checking whether the relative ranking and gap size survive. I am not moving the verdict because the paper is already CONDITIONAL and the concern reinforces that condition rather than proving the result false. The external baselines being untrained on TMDialog-IT is also a limitation, but the LoRA-only ablation is the controlled comparison, and its weakness is the same evaluation-validity problem. Therefore the reader's conditional verdict stands unchanged.","tokens_in":13055,"tokens_out":3931,"duration_ms":44855,"concrete_test":"Re-score a random subset of at least 100 TMDialog-Eva dialogues using a vision-capable judge (human annotators or GPT-4V) that actually sees each image, using the same rubric and scoring instruction. Compare ContextQFormer against LoRA-only on this subset: if the image-based available-rate gap is not comparable to the 4.16-point text-based gap, or if agreement between text-based and image-based scores is low, the reported improvement is an artifact of text-only evaluation. Independently, compute bootstrap confidence intervals for the full 329-sample difference to check whether the 4.16-point gap is statistically distinguishable from zero.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the 'available rate' metric accurately measures whether responses are rational and free of hallucination. Section 5.2 states that GPT-4 has no image input and that 'the evaluation is based on the image description and the dialogue history input.' But hallucination is defined as content 'not present in the image or information that cannot be inferred from it.' The judge cannot check the image; it can only check the description. If a description omits or alters visual details, a response that correctly uses the actual image can be scored as hallucinated, while a vague or generic response that matches the description can be scored as rational. This is not a symmetric noise source: ContextQFormer is designed to activate and use more historical and visual information, so it may produce more specific visual claims that a text-only judge cannot verify. The controlled LoRA-only comparison (68.17% vs. 64.01%, a 4.16-point gap) could therefore be inflated or even reversed under image-grounded evaluation. In addition, no error bars, significance tests, or per-category confidence intervals are reported. With 329 benchmark dialogues, a binary available-rate difference of 4.16 points has a standard error of roughly 3.7 points if scored per dialogue, making the headline improvement potentially within sampling noise. The central claim is load-bearing on the completeness of the text descriptions and on the statistical stability of a small self-built evaluation set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ContextQFormer, a module that maintains a queue of [CLS] features from past visual and textual turns and fuses them with the current instruction via learnable queries and cross-attention, aiming to improve long-context multi-turn multimodal dialogue. The authors also construct TMDialog, a GPT-4-generated dataset for pre-training, instruction tuning, and evaluation, with an evaluation benchmark of 329 handwritten multi-turn dialogues. They train LLaMA-7B with LoRA plus ContextQFormer and compare against LoRA-only, mPLUG-owl, and visualGLM, reporting an \"available rate\" (rational and non-hallucinated responses) of 68.17% for ContextQFormer versus 64.01% for LoRA-only, 66.40% for mPLUG-owl, and 62.46% for visualGLM. The central claim is that the memory-queue module improves response quality in long-context settings beyond identical training without the module.","tokens_in":13346,"tokens_out":5707,"duration_ms":57032,"significance":"If the result holds, ContextQFormer offers a lightweight, parameter-efficient way to improve multi-turn multimodal LLMs without full fine-tuning, and TMDialog could be a useful resource for long-context dialogue research. The controlled LoRA-only comparison is the right experimental design and gives the paper independent grounding for its central claim. The paper also contributes a new benchmark and a clear evaluation metric, and the authors openly discuss limitations such as model-collapse risks from synthetic data. However, the current evidence is weakened by the evaluation protocol (a text-only GPT-4 judge that cannot see images), the small benchmark size with no significance testing, and an internal inconsistency in the dataset statistics; these issues make the headline 2%-4% improvement less convincing than it appears.","major_comments":[{"comment":"The evaluation protocol is load-bearing for the central claim: hallucination is defined as content \"not present in the image or information that cannot be inferred from it,\" yet the GPT-4 judge receives only a text description of each image and the dialogue history. As the paper states, \"The evaluation is based on the image description and the dialogue history input.\" A judge that cannot see the image cannot verify whether a response is grounded; it can only check consistency with the provided description. This is not a symmetric error: ContextQFormer is designed to activate more historical and visual information and may produce more specific visual claims, which a text-only judge might mark as unverifiable or hallucinated, while a generic response that matches the description could be scored as rational. I request an image-grounded evaluation (e.g., GPT-4V or human annotators viewing the images) or at least a human spot-check on a random sample, and a per-category analysis of judge errors, to establish that the LoRA-only gap is not an artifact of description coverage.","section":"§5.2"},{"comment":"The main results are reported from a single evaluation run on 329 benchmark dialogues with no confidence intervals, significance tests, or error analysis. For a binary per-dialogue available-rate outcome, the difference between 68.17% and 64.01% (4.16 percentage points) has an approximate standard error of 3-4 percentage points (sqrt(p(1-p)/n) for n≈329 per group, two-group difference), so the observed gap is about one standard error. The paper states \"our method exhibits a 4.2% improvement in the available rate compared to Lora-only\" and treats this as decisive, but the evidence is not statistically persuasive. Please provide bootstrap confidence intervals across the 329 dialogues, report the number of response-level units evaluated, and if possible run the evaluation multiple times or across judge temperature settings to quantify stability.","section":"Table 5"},{"comment":"There is a direct internal inconsistency in the dataset characterization. Table 2 reports that TMDialog-IT has an average of 4.83 turns and an average length of 51.20, while Table 4 lists TMDialog-IT as having 9.1 average turns and 46.2 average length. The paper's abstract and Section 3.3 claim that TMDialog contains \"longer conversations\" than other datasets, a claim that depends on these statistics. The discrepancy must be resolved: either the numbers in Table 4 are for a different subset (e.g., only the generated portion vs. the full set), or one table is erroneous. Without clarification, the dataset's reported properties are unreliable and the comparison to ImageChat and VisualDialogue is unverifiable.","section":"Tables 2 and 4"},{"comment":"The comparison against mPLUG-owl and visualGLM is not controlled: these external baselines were not trained or instruction-tuned on TMDialog-IT, whereas ContextQFormer and LoRA-only were trained on TMDialog-IT and the same TMDialog-PT. The abstract's claim of a \"2%-4% improvement over baselines\" conflates the controlled LoRA-only comparison with these uncontrolled ones. The only controlled comparison is LoRA-only, which is the appropriate ablation. Please either fine-tune the external baselines on TMDialog-IT (if feasible with the same budget), or clearly state in the abstract and Section 5.3 that mPLUG-owl and visualGLM are evaluated zero-shot and that the state-of-the-art claim is limited to the controlled LoRA-only comparison.","section":"§5.3"}],"minor_comments":[{"comment":"The abstract states an \"improvement of 2%-4% in available rate,\" but Table 5 shows improvements of 1.77 percentage points over mPLUG-owl, 4.16 points over LoRA-only, and 5.71 points over visualGLM. Please use precise percentage-point numbers or a range that matches the data.","section":"Abstract"},{"comment":"In the introduction, the text says \"As demonstrated in Figure 2, previous multi-modal LLMs may miss out on essential context,\" but Figure 2 in the paper is titled \"The process of data generation.\" The correct reference appears to be Figure 1 or a new figure showing the baseline failure modes.","section":"§1"},{"comment":"The section heading \"Limation\" should be \"Limitations.\" Additionally, in §4.2, \"prat prompts\" should be \"prompt templates\" or similar, and \"training object\" should be \"training objective.\"","section":"§6"},{"comment":"The phrase \"two examples of handwriting\" is unclear; it likely means \"handwritten examples\" or \"manually written examples.\" Please rephrase.","section":"§3.1"},{"comment":"The text says visual features are extracted with \"the pre-trained ViT\" and textual features with \"Roberta.\" Use the standard capitalization \"RoBERTa\" and specify the exact checkpoint (e.g., RoBERTa-base) and the source of the ViT (e.g., CLIP ViT-L/14) for reproducibility.","section":"§4.2"},{"comment":"The evaluation prompt is described only schematically as \"<History><Description><Instruction>\". Including the full prompt (or an appendix) would strengthen reproducibility and allow readers to assess the judge's instructions for rationality, information, hallucination, and safety.","section":"§5.2"},{"comment":"The table lists \"Continuous Question\" as a category with 199 samples, but the text in §3.2 says the benchmark includes \"two types called Long Conversation and Continuous Question\" in addition to the three instruction-tuning types. Please ensure the type names and counts are consistent across the text and table.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong controlled-ablation setup (LoRA-only), which gives it some independent grounding, but the current evidence is not yet convincing because of the text-only judge and the lack of statistical power. The dataset statistics inconsistency, if not fixed, undermines the dataset contribution. I recommend major revision rather than rejection, as the evaluation gap could be addressed with image-grounded human or model-based evaluation, significance testing, and corrected statistics. The paper's scope and claims are appropriate for a conference or journal submission, but the authors should also consider whether the 2%-4% claim in the abstract is overstated given the actual numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this before reading: ContextQFormer is a queue-based memory module that stores [CLS] tokens from text and images and re-activates them via Q-Former cross-attention. The only clean comparison is against LoRA-only trained on the same data, and that gap is 4.2 available-rate points. That gap is suggestive, but the evaluation is not strong enough to support the paper's claims.\n\nWhat's genuinely new: the TMDialog dataset, with its long-memory and multi-image categories, is a useful resource for the subfield, and the specific combination of a [CLS] token queue with Q-Former for multimodal dialogue is not in the cited prior work. The controlled LoRA-only ablation is the right way to isolate the module's effect, and it points in a consistent direction. The paper also honestly acknowledges that training data is GPT-4-generated and that training on synthetic data risks degradation.\n\nThe soft spots are real and located in the evaluation. Section 5.2 says GPT-4 scores hallucination using a text description of each image, not the image itself. If a description omits something the model can see, a correct answer can be scored as hallucinated, and a generic answer that matches the description can be scored as rational. This is a genuine threat, though it would need to be biased against ContextQFormer specifically to overturn the LoRA gap. More damning: with 329 dialogues and a binary outcome, a 4.2-point gap in available rate has a standard error of roughly 3.7 percentage points if treated as independent proportions. No significance test, confidence interval, or paired analysis is reported. The external baselines were not fine-tuned on TMDialog-IT, so the \"state-of-the-art\" phrasing is not supported. Finally, code and data are not yet public, and the architecture is underspecified (queue capacity, number of queries, pruning).\n\nThe Limitation section covers training data and short-dialogue overhead but not the text-description judge, which is the more serious threat to the headline claim.\n\nOverall, the core idea is sound and the controlled ablation is a good step. The paper deserves a serious referee if the authors commit to releasing the dataset and code, adding image-grounded human evaluation on a subset, and reporting uncertainty. As submitted, the headline gain is plausible but unproven. I would not cite it in its current form, but I'd bring it to a reading group to discuss the evaluation trap.","headline":"The module is a plausible engineering contribution, but the headline 2-4% gain rests on a GPT-4 judge that never sees the images and on a 329-dialogue comparison with no error bars.","tokens_in":13888,"tokens_out":2714,"would_cite":false,"duration_ms":26719,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ContextQFormer stores past images and turns as compact tokens in a memory queue, and this lifts long multi-turn multimodal dialogue quality by 2-4% over baselines.","keywords":["multi-modal large language models","multi-turn dialogue","long context understanding","memory mechanism","context modeling module","dialogue dataset","hallucination evaluation","available rate"],"falsifier":"Run the 329 TMDialog-Eva samples with human raters (or an image-capable judge) who see the actual images, and recompute available rates; if ContextQFormer's gap over LoRA-only disappears or reverses, the claimed improvement is an artifact of description-based scoring.","tokens_in":12874,"feed_emoji":"💬","tokens_out":4680,"duration_ms":41991,"temperature":0.7,"pith_summary":"The paper argues that open-source multimodal language models lose relevant context in long multi-turn conversations because historical noise accumulates, and that a dedicated context-modeling module can fix this cheaply. It introduces ContextQFormer, which stores a compact [CLS] representation of each past image and text turn in a memory queue and lets the current question cross-attend to that queue. On a new long-context benchmark, TMDialog-Eva, the module improves the available rate (responses that are both rational and hallucination-free) by about 4 percentage points over the same LoRA-tuned model without the module, and by 1.8-5.7 points over two open-source baselines. The paper also contributes TMDialog, a GPT-4-generated multi-turn multimodal dataset with longer and more image-relevant conversations than existing ones. If the result holds, adding such a memory module is a low-cost way to make multimodal assistants hold longer, more coherent conversations.","feed_headline":"Memory queue lifts multimodal dialogue quality by 2-4%","feed_subtitle":"A compact context module lets long multi-turn chats stay coherent and hallucination-free, beating three baselines.","key_machinery":"The central object is the ContextQFormer memory queue: a first-in-first-out buffer of [CLS] token embeddings, one per image (from ViT) and one per text turn (from RoBERTa), which is fused with the current instruction through Q-Former-style cross-attention. The queue gives the model a fixed-size memory of the whole dialogue without extending the self-attention context window, and it is the component whose removal produces the LoRA-only baseline.","core_discovery":"ContextQFormer is a Q-Former-style module with an attached memory queue. For every image and every text turn in the dialogue, a [CLS] token is extracted with ViT or RoBERTa and pushed into the queue; when a new instruction arrives, learnable queries first self-attend over the instruction and then cross-attend over the queued historical representations, so a late question can reactivate an image or fact from many turns earlier. On the 329-sample TMDialog-Eva benchmark, ContextQFormer reaches a 68.17% available rate versus 64.01% for LoRA-only, 66.40% for mPLUG-owl, and 62.46% for visualGLM, with the largest gains in interaction, long-memory, and multi-image categories. The authors claim the module can be inserted into larger models without full fine-tuning, and they observe that the GPT-4-generated training data may cap performance below GPT-4 itself.","pith_inferences":["If description-based scoring is replaced with a judge that sees actual images, the reported 2-4% advantage could shrink or grow; this is testable with an image-capable judge.","The memory-queue idea could extend to other modalities such as audio or video, or to streaming settings where the context window must stay bounded.","The gain on long-memory questions suggests a natural extension: evict stale queue entries or weight queue items by recency and relevance instead of using a plain FIFO queue.","Because TMDialog is generated by GPT-4, models trained on it may implicitly match GPT-4's text-description style, which could inflate scores from a GPT-4 judge."],"forward_implications":["Adding ContextQFormer to a LoRA fine-tune of a frozen 7B LLM improves long-context response quality without changing the training data or loss.","The biggest gains appear in interaction, long-memory, and multi-image categories, while short continuous-question dialogues show little difference across methods.","The available rate metric, defined as the percentage of responses that are both rational and non-hallucinated, offers a single scalar for comparing multimodal dialogue models.","TMDialog, with average dialogue length around 46 tokens and most conversations image-relevant, gives the community a long-context resource for pretraining, instruction tuning, and evaluation.","Because the module is separate from the frozen LLM, it can be appended to larger models without full fine-tuning."],"supporting_citations":[{"why":"Supplies the Q-Former architecture that ContextQFormer adapts, with learnable queries cross-attending to visual features.","marker":"(Li et al., 2023)"},{"why":"Defines LoRA, the parameter-efficient tuning method used by both ContextQFormer and the LoRA-only baseline.","marker":"(Hu et al., 2021)"},{"why":"Provides the memory-mechanism precedent that ContextQFormer extends to multimodal dialogue.","marker":"(Wu et al., 2022)"},{"why":"RoBERTa supplies the [CLS] representation used for each text turn in the memory queue.","marker":"(Liu et al., 2019)"},{"why":"The Vision Transformer supplies image features, with its [CLS] token stored in the queue.","marker":"(Dosovitskiy et al., 2020)"},{"why":"mPLUG-owl is one of the three comparison baselines on TMDialog-Eva.","marker":"(Ye et al., 2023)"},{"why":"visualGLM is the second open-source baseline, noted for lacking multi-image support.","marker":"(Du et al., 2022)"},{"why":"GPT-4 generates the TMDialog training data and scores the evaluation responses from image descriptions.","marker":"(OpenAI, 2023)"}],"fun_headline_variants":["Memory queue boosts long multimodal chats by 2-4%","Context module sharpens multi-turn vision-language dialogue","Long-context memory improves multimodal dialogue quality","ContextQFormer: Memory queue powers coherent multi-turn chats"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that a GPT-4 judge can score rationality and hallucination correctly from a text description of each image rather than from the image itself, and that those descriptions capture everything a model could see.","fun_headline_variants_meta":{"raw":{"variants":["Memory queue boosts long multimodal chats by 2-4%","Context module sharpens multi-turn vision-language dialogue","Long-context memory improves multimodal dialogue quality","ContextQFormer: Memory queue powers coherent multi-turn chats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000394,"raw_usage":{"total_tokens":2044,"prompt_tokens":895,"completion_tokens":1149,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":1086}},"tokens_in":511,"tokens_out":1149,"duration_ms":8652,"temperature":1.0,"reasoning_tokens":1086,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:52:47.882292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 329 TMDialog-Eva samples with human raters (or an image-capable judge) who see the actual images, and recompute available rates; if ContextQFormer's gap over LoRA-only disappears or reverses, the claimed improvement is an artifact of description-based scoring.","supporting_citations":[{"cited_title":"To further expand the capabilities of large language models, multi-modal models are developed to in- corporate various types of input beyond text","cited_arxiv_id":null,"evidence_quote":"GPT-4 generates the TMDialog training data and scores the evaluation responses from image descriptions."}],"review_version":1}