ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations

Haitao Leng; Qingjie Liu; Shaoguo Liu; Tingting Gao; Yiming Lei; Yunhong Wang; Zeming Liu; Zhizheng Yang

REVIEW 4 major objections 7 minor 16 references

ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ContextQFormer stores past images and turns as compact tokens in a memory queue, and this lifts long multi-turn multimodal dialogue quality by 2-4% over baselines.

desk verdict The module is a plausible engineering contribution, but the headline 2-4% gain rests on a GPT-4 judge that never sees the images and on a 329-dialogue comparison with no error bars. read the letter →

arxiv 2505.23121 v2 pith:TY3LLDH2 submitted 2025-05-29 cs.CL cs.AI

Yiming Lei , Zhizheng Yang , Zeming Liu , Haitao Leng , Shaoguo Liu , Tingting Gao , Qingjie Liu , Yunhong Wang This is my paper · ORCID

classification cs.CLcs.AI

keywords multi-modallargelanguagemodelsmulti-turndialoguelongcontextunderstandingmemorymechanismmodelingmoduledatasethallucinationevaluationavailablerate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that open-source multimodal language models lose relevant context in long multi-turn conversations because historical noise accumulates, and that a dedicated context-modeling module can fix this cheaply. It introduces ContextQFormer, which stores a compact [CLS] representation of each past image and text turn in a memory queue and lets the current question cross-attend to that queue. On a new long-context benchmark, TMDialog-Eva, the module improves the available rate (responses that are both rational and hallucination-free) by about 4 percentage points over the same LoRA-tuned model without the module, and by 1.8-5.7 points over two open-source baselines. The paper also contributes TMDialog, a GPT-4-generated multi-turn multimodal dataset with longer and more image-relevant conversations than existing ones. If the result holds, adding such a memory module is a low-cost way to make multimodal assistants hold longer, more coherent conversations.

What carries the argument

The central object is the ContextQFormer memory queue: a first-in-first-out buffer of [CLS] token embeddings, one per image (from ViT) and one per text turn (from RoBERTa), which is fused with the current instruction through Q-Former-style cross-attention. The queue gives the model a fixed-size memory of the whole dialogue without extending the self-attention context window, and it is the component whose removal produces the LoRA-only baseline.

What would settle it

Run the 329 TMDialog-Eva samples with human raters (or an image-capable judge) who see the actual images, and recompute available rates; if ContextQFormer's gap over LoRA-only disappears or reverses, the claimed improvement is an artifact of description-based scoring.

Watch

Extended reading notes

Core claim

ContextQFormer is a Q-Former-style module with an attached memory queue. For every image and every text turn in the dialogue, a [CLS] token is extracted with ViT or RoBERTa and pushed into the queue; when a new instruction arrives, learnable queries first self-attend over the instruction and then cross-attend over the queued historical representations, so a late question can reactivate an image or fact from many turns earlier. On the 329-sample TMDialog-Eva benchmark, ContextQFormer reaches a 68.17% available rate versus 64.01% for LoRA-only, 66.40% for mPLUG-owl, and 62.46% for visualGLM, with the largest gains in interaction, long-memory, and multi-image categories. The authors claim the module can be inserted into larger models without full fine-tuning, and they observe that the GPT-4-generated training data may cap performance below GPT-4 itself.

Load-bearing premise

The evaluation assumes that a GPT-4 judge can score rationality and hallucination correctly from a text description of each image rather than from the image itself, and that those descriptions capture everything a model could see.

Editorial extensions

If this is right

Adding ContextQFormer to a LoRA fine-tune of a frozen 7B LLM improves long-context response quality without changing the training data or loss.
The biggest gains appear in interaction, long-memory, and multi-image categories, while short continuous-question dialogues show little difference across methods.
The available rate metric, defined as the percentage of responses that are both rational and non-hallucinated, offers a single scalar for comparing multimodal dialogue models.
TMDialog, with average dialogue length around 46 tokens and most conversations image-relevant, gives the community a long-context resource for pretraining, instruction tuning, and evaluation.
Because the module is separate from the frozen LLM, it can be appended to larger models without full fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If description-based scoring is replaced with a judge that sees actual images, the reported 2-4% advantage could shrink or grow; this is testable with an image-capable judge.
The memory-queue idea could extend to other modalities such as audio or video, or to streaming settings where the context window must stay bounded.
The gain on long-memory questions suggests a natural extension: evict stale queue entries or weight queue items by recency and relevance instead of using a plain FIFO queue.
Because TMDialog is generated by GPT-4, models trained on it may implicitly match GPT-4's text-description style, which could inflate scores from a GPT-4 judge.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this before reading: ContextQFormer is a queue-based memory module that stores [CLS] tokens from text and images and re-activates them via Q-Former cross-attention. The only clean comparison is against LoRA-only trained on the same data, and that gap is 4.2 available-rate points. That gap is suggestive, but the evaluation is not strong enough to support the paper's claims.

What's genuinely new: the TMDialog dataset, with its long-memory and multi-image categories, is a useful resource for the subfield, and the specific combination of a [CLS] token queue with Q-Former for multimodal dialogue is not in the cited prior work. The controlled LoRA-only ablation is the right way to isolate the module's effect, and it points in a consistent direction. The paper also honestly acknowledges that training data is GPT-4-generated and that training on synthetic data risks degradation.

The soft spots are real and located in the evaluation. Section 5.2 says GPT-4 scores hallucination using a text description of each image, not the image itself. If a description omits something the model can see, a correct answer can be scored as hallucinated, and a generic answer that matches the description can be scored as rational. This is a genuine threat, though it would need to be biased against ContextQFormer specifically to overturn the LoRA gap. More damning: with 329 dialogues and a binary outcome, a 4.2-point gap in available rate has a standard error of roughly 3.7 percentage points if treated as independent proportions. No significance test, confidence interval, or paired analysis is reported. The external baselines were not fine-tuned on TMDialog-IT, so the "state-of-the-art" phrasing is not supported. Finally, code and data are not yet public, and the architecture is underspecified (queue capacity, number of queries, pruning).

The Limitation section covers training data and short-dialogue overhead but not the text-description judge, which is the more serious threat to the headline claim.

Overall, the core idea is sound and the controlled ablation is a good step. The paper deserves a serious referee if the authors commit to releasing the dataset and code, adding image-grounded human evaluation on a subset, and reporting uncertainty. As submitted, the headline gain is plausible but unproven. I would not cite it in its current form, but I'd bring it to a reading group to discuss the evaluation trap.

Referee Report

4 major / 7 minor

Summary. The paper proposes ContextQFormer, a module that maintains a queue of [CLS] features from past visual and textual turns and fuses them with the current instruction via learnable queries and cross-attention, aiming to improve long-context multi-turn multimodal dialogue. The authors also construct TMDialog, a GPT-4-generated dataset for pre-training, instruction tuning, and evaluation, with an evaluation benchmark of 329 handwritten multi-turn dialogues. They train LLaMA-7B with LoRA plus ContextQFormer and compare against LoRA-only, mPLUG-owl, and visualGLM, reporting an "available rate" (rational and non-hallucinated responses) of 68.17% for ContextQFormer versus 64.01% for LoRA-only, 66.40% for mPLUG-owl, and 62.46% for visualGLM. The central claim is that the memory-queue module improves response quality in long-context settings beyond identical training without the module.

Significance. If the result holds, ContextQFormer offers a lightweight, parameter-efficient way to improve multi-turn multimodal LLMs without full fine-tuning, and TMDialog could be a useful resource for long-context dialogue research. The controlled LoRA-only comparison is the right experimental design and gives the paper independent grounding for its central claim. The paper also contributes a new benchmark and a clear evaluation metric, and the authors openly discuss limitations such as model-collapse risks from synthetic data. However, the current evidence is weakened by the evaluation protocol (a text-only GPT-4 judge that cannot see images), the small benchmark size with no significance testing, and an internal inconsistency in the dataset statistics; these issues make the headline 2%-4% improvement less convincing than it appears.

major comments (4)

[§5.2] The evaluation protocol is load-bearing for the central claim: hallucination is defined as content "not present in the image or information that cannot be inferred from it," yet the GPT-4 judge receives only a text description of each image and the dialogue history. As the paper states, "The evaluation is based on the image description and the dialogue history input." A judge that cannot see the image cannot verify whether a response is grounded; it can only check consistency with the provided description. This is not a symmetric error: ContextQFormer is designed to activate more historical and visual information and may produce more specific visual claims, which a text-only judge might mark as unverifiable or hallucinated, while a generic response that matches the description could be scored as rational. I request an image-grounded evaluation (e.g., GPT-4V or human annotators viewing the images) or at least a human spot-check on a random sample, and a per-category analysis of judge errors, to establish that the LoRA-only gap is not an artifact of description coverage.
[Table 5] The main results are reported from a single evaluation run on 329 benchmark dialogues with no confidence intervals, significance tests, or error analysis. For a binary per-dialogue available-rate outcome, the difference between 68.17% and 64.01% (4.16 percentage points) has an approximate standard error of 3-4 percentage points (sqrt(p(1-p)/n) for n≈329 per group, two-group difference), so the observed gap is about one standard error. The paper states "our method exhibits a 4.2% improvement in the available rate compared to Lora-only" and treats this as decisive, but the evidence is not statistically persuasive. Please provide bootstrap confidence intervals across the 329 dialogues, report the number of response-level units evaluated, and if possible run the evaluation multiple times or across judge temperature settings to quantify stability.
[Tables 2 and 4] There is a direct internal inconsistency in the dataset characterization. Table 2 reports that TMDialog-IT has an average of 4.83 turns and an average length of 51.20, while Table 4 lists TMDialog-IT as having 9.1 average turns and 46.2 average length. The paper's abstract and Section 3.3 claim that TMDialog contains "longer conversations" than other datasets, a claim that depends on these statistics. The discrepancy must be resolved: either the numbers in Table 4 are for a different subset (e.g., only the generated portion vs. the full set), or one table is erroneous. Without clarification, the dataset's reported properties are unreliable and the comparison to ImageChat and VisualDialogue is unverifiable.
[§5.3] The comparison against mPLUG-owl and visualGLM is not controlled: these external baselines were not trained or instruction-tuned on TMDialog-IT, whereas ContextQFormer and LoRA-only were trained on TMDialog-IT and the same TMDialog-PT. The abstract's claim of a "2%-4% improvement over baselines" conflates the controlled LoRA-only comparison with these uncontrolled ones. The only controlled comparison is LoRA-only, which is the appropriate ablation. Please either fine-tune the external baselines on TMDialog-IT (if feasible with the same budget), or clearly state in the abstract and Section 5.3 that mPLUG-owl and visualGLM are evaluated zero-shot and that the state-of-the-art claim is limited to the controlled LoRA-only comparison.

minor comments (7)

[Abstract] The abstract states an "improvement of 2%-4% in available rate," but Table 5 shows improvements of 1.77 percentage points over mPLUG-owl, 4.16 points over LoRA-only, and 5.71 points over visualGLM. Please use precise percentage-point numbers or a range that matches the data.
[§1] In the introduction, the text says "As demonstrated in Figure 2, previous multi-modal LLMs may miss out on essential context," but Figure 2 in the paper is titled "The process of data generation." The correct reference appears to be Figure 1 or a new figure showing the baseline failure modes.
[§6] The section heading "Limation" should be "Limitations." Additionally, in §4.2, "prat prompts" should be "prompt templates" or similar, and "training object" should be "training objective."
[§3.1] The phrase "two examples of handwriting" is unclear; it likely means "handwritten examples" or "manually written examples." Please rephrase.
[§4.2] The text says visual features are extracted with "the pre-trained ViT" and textual features with "Roberta." Use the standard capitalization "RoBERTa" and specify the exact checkpoint (e.g., RoBERTa-base) and the source of the ViT (e.g., CLIP ViT-L/14) for reproducibility.
[§5.2] The evaluation prompt is described only schematically as "<History><Description><Instruction>". Including the full prompt (or an appendix) would strengthen reproducibility and allow readers to assess the judge's instructions for rationality, information, hallucination, and safety.
[Table 3] The table lists "Continuous Question" as a category with 199 samples, but the text in §3.2 says the benchmark includes "two types called Long Conversation and Continuous Question" in addition to the three instruction-tuning types. Please ensure the type names and counts are consistent across the text and table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main ContextQFormer vs. LoRA-only comparison is a controlled ablation, and the GPT-4-based evaluation is a proxy-validity concern rather than a reduction by construction.

full rationale

The paper's central empirical claim is that ContextQFormer raises the available rate by 2%–4% over baselines. The comparison that carries this claim is ContextQFormer against LoRA-only, which Section 5.3 states is obtained by removing the context-modeling module while maintaining the same training data and configuration; hence the gap is not fitted from the evaluation set and is not equivalent to the module's input by construction. The evaluation benchmark (TMDialog-Eva) is a separately handwritten set of 329 samples, not the GPT-4-generated training data. GPT-4 does act as both a data generator for TMDialog-IT and as the judge, and Section 5.2 explicitly states that scoring is based on image descriptions rather than images; this is a real threat to construct validity because hallucination is defined relative to the image but checked against a text proxy. However, it is not circularity: the available-rate score is not defined as 'GPT-4 agreement' and the same judge/description protocol is applied to all methods, so the ContextQFormer advantage is not forced by the definition. No load-bearing self-citations, uniqueness theorems, or ansatz-by-citation steps appear; the Q-Former architecture is cited to external BLIP-2 work. The paper's own Limation section discloses the GPT-4-generated-data risk. The lack of error bars is a statistical reporting concern, not a circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No single fitted constant is central to the claim; the method uses ordinary trained weights and unreported hyperparameters. The main assumptions are about the sufficiency of [CLS] summarization, the fidelity of image descriptions, and the reliability of GPT-4 as judge.

free parameters (1)

ContextQFormer queue capacity and number of learnable queries = not reported
These architectural choices determine how much historical context is retained and how it is attended to; without them the module cannot be reproduced.

assumptions (4)

domain assumption The [CLS] token output of ViT and RoBERTa is a sufficient representation of an image or utterance for historical context.
Section 4.2 enqueues only [CLS] features as memory; if this pooling discards needed visual or textual detail, the memory block cannot support long-context answers.
domain assumption The image descriptions in TMDialog-Eva faithfully represent the images they describe.
Section 3.2 says comprehensive descriptions will be provided for evaluation; GPT-4 judges responses from these descriptions, so any gap between description and image breaks hallucination measurement.
domain assumption GPT-4 scoring of rationality, information, hallucination, and safety from text descriptions is a valid measure of response quality.
Section 5.2 uses GPT-4 as the sole judge on a self-built benchmark; no human agreement or calibration is reported.
domain assumption The LoRA-only ablation isolates the effect of ContextQFormer.
Section 5.3 states LoRA-only removes the context module while keeping the same data and configuration; this assumes no interaction effects between the context module and LoRA training.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations." pith.science (2026). https://pith.science/paper/TY3LLDH2

@misc{pith2026250523121,
  author       = {Pith},
  title        = {Pith review of: ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TY3LLDH2}},
  note         = {Machine review of arXiv:2505.23121}
}

read the original abstract

Multi-modal large language models have demonstrated remarkable zero-shot abilities and powerful image-understanding capabilities. However, the existing open-source multi-modal models suffer from the weak capability of multi-turn interaction, especially for long contexts. To address the issue, we first introduce a context modeling module, termed ContextQFormer, which utilizes a memory block to enhance the presentation of contextual information. Furthermore, to facilitate further research, we carefully build a new multi-turn multi-modal dialogue dataset (TMDialog) for pre-training, instruction-tuning, and evaluation, which will be open-sourced lately. Compared with other multi-modal dialogue datasets, TMDialog contains longer conversations, which supports the research of multi-turn multi-modal dialogue. In addition, ContextQFormer is compared with three baselines on TMDialog and experimental results illustrate that ContextQFormer achieves an improvement of 2%-4% in available rate over baselines.

Figures

Figures reproduced from arXiv: 2505.23121 by the authors.

**Figure 1.** A comparison of open-sourced methods and the proposed ContextQFormer. VisualGLM cannot recognize multiple images, and mPLUGowl shows serious hallucinations. long context. In ContextQFormer, a memory block is introduced to store core multi-modal information and the context embedding is fused with the memory block to obtain a better context embedding. Specifically, images and text are represented with the representa… view at source ↗

**Figure 2.** The process of data generation. Demonstration examples, image descriptions, and instructions are concatenated as the input to GPT-4. Then, GPT-3.5-turbo is utilized to recheck the output of GPT-4. multi-turn conversation datasets including MiniGPT4 (Zhu et al., 2023), LLAVA (Liu et al., 2023a), VQA-v2 (Goyal et al., 2017) and some unpublished data. The second part consists of a natural language multi-turn conver… view at source ↗

**Figure 3.** The example of different datasets. image-text pairs. In the second stage, multi-modal multi-turn dialogues are employed to enhance the model’s instruction-following ability, thereby improving the reliability and responsiveness of responses. 4.1. Pretrain The architecture of pre-training is shown on the left side of the figure 4. Large language models, such as LLaMA (Touvron et al., 2023), are trained on extensive a… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Overview of the ContextQFormer. In addition to the LoRA module, we have also designed a dedicated module called ContextQFormer for context modeling, as illustrated on the right side of the figure 4. During fine-tuning, the prat prompts we use are as follows: Human:<Im…

**Figure 5.** Figure 5: The result of different categories. The experimental results are shown in the table 5. The rationality, information, hallucination, and safety values displayed in the table represent the average results. When compared to mPLUGowl and visualGLM, our model demonstrates…

**Figure 6.** Figure 6: The response of ContextQFormer. uous questioning, interaction, long memory, multiimages, and long conversations, respectively. ContextQFormer has achieved amazing results in continuous questioning, interaction, and long memory. Even in the case of multiple images, thi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 8 canonical work pages

[1]

To further expand the capabilities of large language models, multi-modal models are developed to in- corporate various types of input beyond text

Introduction Large language models (LLMs) lie at the heart of natural language processing and artificial intel- ligence (OpenAI, 2023; Xu et al., 2023; Tu et al., 2023; Singhal et al., 2023) thanks to their astonish- ing capabilities in natural language generation. To further expand the capabilities of large language models, multi-modal models are develop...

work page 2023
[2]

Related Work 2.1. Multi-Modal Large Language Models Whilelargelanguagemodels(LLMs)haveachieved significant success in natural language process- ing, other modalities such as vision and audio in combination with LLMs are still in the exploratory stage. Existing fusion methods for multi-modal LLMs can generally be categorized into three dif- ferent paradigm...

work page 2023
[3]

The first part comprises the data used for training, including multi-modal pretraining and instruction tuning

Dataset The data required during the experiment can be categorized into two parts. The first part comprises the data used for training, including multi-modal pretraining and instruction tuning. The second part consists of the proposed benchmark, which effec- tivelyevaluatesthecapabilitiesofmulti-modallarge language models in multi-turn dialogue tasks. Thu...

work page 2011
[4]

Statistics show that dialogues in TMDialog are much longer, with more relevant questions and answers, than other datasets

To reduce hallucinations, GPT-3.5 is utilized to recheck the output. Statistics show that dialogues in TMDialog are much longer, with more relevant questions and answers, than other datasets. To facilitate model comparison, we conduct bench-marking experiments on the dataset. Specif- ically, experiments are conducted on mPLUG- owl (Ye et al., 2023), visua...

work page 2023
[5]

Experimental Setup During the pre-training and fine-tuning phases, we execute a total of 40,000 and 10,000 training itera- tions, respectively

Experiments 5.1. Experimental Setup During the pre-training and fine-tuning phases, we execute a total of 40,000 and 10,000 training itera- tions, respectively. For hardware, we employ four clusters of 8 A100(80GB) GPUs and one cluster of 8 V100(32GB) GPUs, with batch sizes per GPU of 256 and 32 for the pre-training and instruction- tuning,respectively. T...

work page 2023
[6]

Method To achieve an effective multi-modal large language model, we propose a two-stage training approach. The purpose of the first stage is to align the image encoder and the text decoder to acquire vision-language knowledge by utilizing amounts of TMDialog-PT TMDialog-IT Give this picture a suitable name. Okay, a picture name that blends animal elements...

work page 2023
[7]

We intro- duce a method that effectively utilizes the GPT-4 API for multi-modal multi-turn dialogue data gen- erating

Conclusion In this paper, we propose a two-stage learning framework to enhance the model’s visual under- standing and instruction-following ability. We intro- duce a method that effectively utilizes the GPT-4 API for multi-modal multi-turn dialogue data gen- erating. We carefully build a train and evalua- tion dataset for multi-modal multi-turn dialogue t...

work page
[8]

While we applied prompt en- gineering techniques and performed data filtering andmodification,itispossiblethatsomelow-quality dialogues remain in the resulting dataset

Limation Due to the high cost of manual labeling, a consider- ableportionofthetrainingdataisgenerateddirectly using the GPT-4 API. While we applied prompt en- gineering techniques and performed data filtering andmodification,itispossiblethatsomelow-quality dialogues remain in the resulting dataset. More- over, researchers (Shumailov et al., 2023) have dem...

work page 2023

Show all 16 references

[10]

Bibliographical References Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Mal- colm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Mariann...

2022 arXiv
[12]

Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, and Qingwei Lin

Addressing some limitations of trans- formers with feedback memory.arXiv preprint arXiv:2002.09402. Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, and Qingwei Lin. 2022. Mmdialog: A large- scale multi-turn dialogue dataset towards multi-...

2002 arXiv
[15]

Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston

The curse of recursion: Training on gen- erated data makes models forget.arXiv preprint arxiv:2305.17493. Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston. 2018. Image chat: En- gaging grounded conversations.arXiv preprint arXiv:1811.00945. Karan Singhal, Shekoofe...

2018 arXiv
[16]

arXiv preprint arXiv:2303.11381

Mm-react: Prompting chatgpt for mul- timodal reasoning and action. arXiv preprint arXiv:2303.11381. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, An- wen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. mplug-owl: Modularization empowers large lan-...

2023 arXiv
[2011]

Jordi Pont-Tuset, Jasper Uijlings, Soravit Chang- pinyo, Radu Soricut, and Vittorio Ferrari

Im2text: Describing images using 1 mil- lion captioned photographs.Advances in neural information processing systems, 24. Jordi Pont-Tuset, Jasper Uijlings, Soravit Chang- pinyo, Radu Soricut, and Vittorio Ferrari. 2020. Connecting vision and language with localized narratives...

2020 arXiv
[2019]

Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra

Transformer-xl: Attentive language mod- els beyond a fixed-length context.arXiv preprint arXiv:1901.02860. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE conferen...

1901 arXiv
[2020]

are capable of processing long inputs, they may still struggle to capture crucial contextual in- formation in exceptionally lengthy texts. As demon- strated in Figure 2, previous multi-modal LLMs may miss out on essential context from preceding text because of the accumulation...

2023 arXiv
[2023]

Xiaoming Shi, Zeming Liu, Li Du, Yuxuan Wang, Hongru Wang, Yuhang Guo, Tong Ruan, Jie Xu, Xiaofan Zhang, and Shaoting Zhang

Hugginggpt: Solving ai tasks with chat- gpt and its friends in huggingface.arXiv preprint arXiv:2303.17580. Xiaoming Shi, Zeming Liu, Li Du, Yuxuan Wang, Hongru Wang, Yuhang Guo, Tong Ruan, Jie Xu, Xiaofan Zhang, and Shaoting Zhang. 2024. Med- ical dialogue system: A survey of...

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

To further expand the capabilities of large language models, multi-modal models are developed to in- corporate various types of input beyond text

Introduction Large language models (LLMs) lie at the heart of natural language processing and artificial intel- ligence (OpenAI, 2023; Xu et al., 2023; Tu et al., 2023; Singhal et al., 2023) thanks to their astonish- ing capabilities in natural language generation. To further expand the capabilities of large language models, multi-modal models are develop...

work page 2023

[2] [2]

Related Work 2.1. Multi-Modal Large Language Models Whilelargelanguagemodels(LLMs)haveachieved significant success in natural language process- ing, other modalities such as vision and audio in combination with LLMs are still in the exploratory stage. Existing fusion methods for multi-modal LLMs can generally be categorized into three dif- ferent paradigm...

work page 2023

[3] [3]

The first part comprises the data used for training, including multi-modal pretraining and instruction tuning

Dataset The data required during the experiment can be categorized into two parts. The first part comprises the data used for training, including multi-modal pretraining and instruction tuning. The second part consists of the proposed benchmark, which effec- tivelyevaluatesthecapabilitiesofmulti-modallarge language models in multi-turn dialogue tasks. Thu...

work page 2011

[4] [4]

Statistics show that dialogues in TMDialog are much longer, with more relevant questions and answers, than other datasets

To reduce hallucinations, GPT-3.5 is utilized to recheck the output. Statistics show that dialogues in TMDialog are much longer, with more relevant questions and answers, than other datasets. To facilitate model comparison, we conduct bench-marking experiments on the dataset. Specif- ically, experiments are conducted on mPLUG- owl (Ye et al., 2023), visua...

work page 2023

[5] [5]

Experimental Setup During the pre-training and fine-tuning phases, we execute a total of 40,000 and 10,000 training itera- tions, respectively

Experiments 5.1. Experimental Setup During the pre-training and fine-tuning phases, we execute a total of 40,000 and 10,000 training itera- tions, respectively. For hardware, we employ four clusters of 8 A100(80GB) GPUs and one cluster of 8 V100(32GB) GPUs, with batch sizes per GPU of 256 and 32 for the pre-training and instruction- tuning,respectively. T...

work page 2023

[6] [6]

Method To achieve an effective multi-modal large language model, we propose a two-stage training approach. The purpose of the first stage is to align the image encoder and the text decoder to acquire vision-language knowledge by utilizing amounts of TMDialog-PT TMDialog-IT Give this picture a suitable name. Okay, a picture name that blends animal elements...

work page 2023

[7] [7]

We intro- duce a method that effectively utilizes the GPT-4 API for multi-modal multi-turn dialogue data gen- erating

Conclusion In this paper, we propose a two-stage learning framework to enhance the model’s visual under- standing and instruction-following ability. We intro- duce a method that effectively utilizes the GPT-4 API for multi-modal multi-turn dialogue data gen- erating. We carefully build a train and evalua- tion dataset for multi-modal multi-turn dialogue t...

work page

[8] [8]

While we applied prompt en- gineering techniques and performed data filtering andmodification,itispossiblethatsomelow-quality dialogues remain in the resulting dataset

Limation Due to the high cost of manual labeling, a consider- ableportionofthetrainingdataisgenerateddirectly using the GPT-4 API. While we applied prompt en- gineering techniques and performed data filtering andmodification,itispossiblethatsomelow-quality dialogues remain in the resulting dataset. More- over, researchers (Shumailov et al., 2023) have dem...

work page 2023