{"id":"f5de1d45-3336-4fd0-969c-6911871dad7e","arxiv_id":"2412.00832","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"EventGPT adapts a LLaVA-style MLLM to event camera streams via three-stage training (image-language, event-language, instruction tuning) and outperforms RGB-based MLLMs on its own benchmark.","lead":"EventGPT is a multimodal large language model that reads event camera streams and answers questions, writes captions, and reasons about scenes using a three-stage training pipeline. The paper introduces two large event-text datasets and claims to outperform existing vision-language models on event-based understanding.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Superiority claim rests on an in-distribution, auto-scored benchmark with untrained baselines and test-set hyperparameter selection; Table 1 (§5.2) does not establish that EventGPT understands event streams better than prior MLLMs.","rationale":"Reader's weakest_assumption is the same benchmark-validity concern, so I agree. I do not find an internal inconsistency in the architecture or training pipeline that would separately undermine the claim; the weak spot is entirely in the evaluation. The fix is straightforward and does not require changing the method: a controlled comparison with an event-trained baseline on a held-out or external benchmark, plus human scoring and variance reporting. Because the reader already conditioned acceptance on these issues, my stress-test does not change the verdict; it sharpens why the condition is necessary.","tokens_in":12879,"tokens_out":8217,"duration_ms":77249,"concrete_test":"Fine-tune Qwen2-VL-7B (the strongest generalist baseline in Table 1) on the same Event-Chat training split used for EventGPT, with identical event-frame inputs and instruction data; choose Nw on a validation split, then evaluate both models on a held-out Event-Chat test split and on a small external event-language set with blind human scoring on DC/CR/VQA. If the fine-tuned baseline matches or beats EventGPT, or if EventGPT's lead collapses on the external set, then the superiority claim reflects training-distribution overlap and test-set selection rather than event understanding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 is the only quantitative support for the abstract claim that EventGPT 'surpasses previous state-of-the-art MLLMs.' The evaluation is performed on test splits of N-ImageNet-Chat and Event-Chat (§5.1), the exact datasets used in Stages 2 and 3 of training. The comparison MLLMs are given event frames but no event-domain training, so the comparison conflates 'trained on the test distribution' with 'better event understanding.' The temporal-window hyperparameter Nw is selected by maximizing Event-Chat test performance (Table 3), so the test split has already influenced the final model. The 1–5 quality scores come from an unspecified automatic judge with no reported human validation, no error bars, and no external benchmark. If the test split merely reflects the training distribution and the judge favors the trained model's response style, the reported gains do not establish superior event-stream understanding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EventGPT, an MLLM for event stream understanding, with an event encoder, a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Training proceeds in three stages: visual-language alignment on LLaVA-Pre-train, event-language alignment on a synthetic N-ImageNet-Chat dataset (about 1M samples), and instruction fine-tuning on an Event-Chat dataset (about 120k samples). The authors also release two event-text datasets and a benchmark with GPT-based scoring on Detail Captioning, Complex Reasoning, and VQA. The central claim is that EventGPT surpasses previous state-of-the-art MLLMs in generation quality, descriptive accuracy, and reasoning capability for event streams.","tokens_in":13061,"tokens_out":3136,"duration_ms":29277,"significance":"If the evaluation were properly controlled, this would be a useful contribution: EventGPT appears to be one of the first attempts to adapt an LLM to event streams, and the release of large event-text datasets could benefit the neuromorphic and multimodal communities. The three-stage training paradigm is a reasonable, if incremental, recipe for cross-modal alignment. However, the current evidence does not establish the paper's headline claim of superiority over existing MLLMs, because the benchmark is in-distribution for EventGPT and out-of-distribution for all baselines, the automatic judge is unspecified, and key hyperparameters are selected on the test set. These are load-bearing issues, not presentation details, and they affect Table 1, which is the only quantitative support for the abstract claim.","major_comments":[{"comment":"The quantitative comparison is not a valid test of the claim that EventGPT 'surpasses previous state-of-the-art MLLMs.' The test splits of N-ImageNet-Chat and Event-Chat are derived from the same datasets used in Stages 2 and 3 of training, while the baseline MLLMs (LLaVA, Qwen2-VL, Intern2VL, DeepSeek-VL) are evaluated on event frames without any event-domain training. Thus Table 1 conflates 'trained on the test distribution' with 'better event understanding.' To support the claim, the authors need either event-adapted baselines (e.g., fine-tuned on the same Event-Chat training data), an external held-out event benchmark, or at least a matched protocol where all models see the same event representation and receive equivalent fine-tuning.","section":"§5.1, §5.2, Table 1"},{"comment":"The scoring procedure is underspecified and potentially circular. The paper says responses are 'quantitatively scored on a 1-5 scale' and that the benchmark is 'inspired by Qwen2-72B-Instruct,' but it does not state which model judges the outputs, whether the judge is the same model that generated the training data, or whether any human validation or inter-annotator agreement was performed. Since EventGPT is fine-tuned on GPT-generated text, an automatic judge that prefers that response style could inflate EventGPT's scores. The authors should report the exact judge, prompt, sample size, human correlation, and variance or confidence intervals.","section":"§5.1, Evaluation Metrics; Table 1"},{"comment":"The temporal window number Nw is selected by maximizing performance on Event-Chat, and Table 1 then reports EventGPT results using the selected Nw=5. This means the test set has influenced model selection, so the reported numbers in Table 1 are not unbiased estimates of generalization. The selection should be done on a validation split, or the paper should report both validation-selected and test-selected results with an explicit statement that the test set was used only once.","section":"§5.3, Table 3"},{"comment":"The evaluation is described as zero-shot, but EventGPT is fine-tuned on Event-Chat, which contains the same tasks (captioning, VQA, reasoning) and the same data distribution as the test set. Calling this zero-shot is misleading. The authors should either rephrase the evaluation as in-distribution, or provide a genuinely zero-shot setting (e.g., held-out event domains or tasks) and compare against baselines under the same conditions.","section":"§5.1, 'zero-shot question-answering setting'"}],"minor_comments":[{"comment":"The text states that average pooling is applied along the temporal and spatial dimensions, but Eq. (6) concatenates 'max-pooled representations' Zmax_T and Zmax_S. Please harmonize the notation and text so the reader knows which pooling operation is actually used.","section":"§4.2, Eqs. (4)–(6)"},{"comment":"The dataset sizes are stated inconsistently: Section 3.1 says N-ImageNet-Chat has over 1,000,000 instances, N-ImageNet-Instruction has 69,000 entries, and Event-Chat has 59,000 entries, while the abstract and Fig. 3 refer to '120,000 instruction samples' for Event-Chat. Please clarify how the 69k and 59k numbers combine into 120k, and define which split is used for training versus testing.","section":"§3.1, §3.2, Fig. 3"},{"comment":"The ablation gains are small (roughly 1–3% relative improvement) and are reported without error bars, significance tests, or multiple runs. Given the 1–5 scoring scale and high baseline scores (e.g., 4.15 VQA), the paper should show that these differences are not within judge noise.","section":"§5.3, Table 2"},{"comment":"The definition of Z uses separate temporal bins Et, but the relationship between the time window T, the temporal window number Nw, and the event tensor representation is not defined precisely. Please state how Nw is used to construct the tensor in Eq. (3).","section":"§4.2, Eq. (3)"},{"comment":"The claim 'the first MLLM for event stream understanding' should be softened to 'to the best of our knowledge' in the abstract and introduction, and it would be helpful to cite or discuss the closest existing works (e.g., EventCLIP, EventBind, ExACT) more explicitly to support the novelty statement.","section":"§1, Abstract"}],"recommendation":"major_revision","confidential_remarks":"The core architecture and dataset construction are potentially publishable, but the evaluation needs substantial reworking before the claimed superiority can be accepted. I would not reject outright because the weaknesses are fixable: add fair event-adapted baselines or an external benchmark, specify and validate the automatic judge, move hyperparameter selection off the test set, and report statistical significance. The current Table 1 does not support the abstract's strong claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: EventGPT is likely the first chat-oriented MLLM built for event streams, and the two datasets it introduces are a genuine resource. The three-stage alignment recipe is a reasonable adaptation of LLaVA-style training to a new modality. The qualitative examples show the model can use temporal cues (cup breaking, tunnel exit) in ways the RGB baselines do not.\n\nThat said, the quantitative case for “surpasses previous SOTA” does not hold as presented. The evaluation is on test splits of the same datasets used in training (N-ImageNet-Chat and Event-Chat), so the model is in-distribution by construction. The baselines—LLaVA, Qwen2-VL, Intern2VL, DeepSeek-VL—are fed event frames with no event-domain training at all, so the comparison conflates “trained on the test distribution” with “better event understanding.” The 1–5 scores come from an automatic GPT-based judge with no reported human validation and no error bars. Add to that the selection of Nw=5 on the Event-Chat test set (Table 3), and the reported gains in Table 1 look more like an artifact of the setup than a robust finding.\n\nThere are also smaller internal slips: the aggregation text describes average pooling but Equation 6 uses max-pooled features, and the text cites EventBind [48] but that reference is actually E-CLIP. These are easy fixes but suggest the manuscript was not carefully proofread.\n\nThe ablations are honest but weak—2–3% gains with no variance—so the component-wise claims are plausible but not proven. The datasets and the three-stage paradigm are the real contributions; the evaluation section is the load-bearing weakness.\n\nBottom line: this paper deserves a serious referee because it opens a new line of work and provides resources the community will use. But the “SOTA surpass” claim needs fair baselines (trained on event frames or at least given event-adapted inputs), an external or held-out benchmark, human-evaluated or at least validated scoring, and error bars. The test-set hyperparameter choice should be justified or moved to a validation split. If those get addressed in revision, the paper would be solid. As it stands, treat the quantitative headline as unverified.","headline":"First event-stream chat MLLM with two useful new datasets, but the quantitative superiority claim rests on an in-distribution, self-scored benchmark with untrained baselines and test-set hyperparameter tuning.","tokens_in":13596,"tokens_out":2387,"would_cite":false,"duration_ms":20854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EventGPT is presented as the first multimodal large language model built for event-camera streams, with a three-stage training scheme that aligns sparse asynchronous pixel-change signals to language and outperforms image-based multimodal…","keywords":["event camera","event stream understanding","multimodal large language model","spatio-temporal aggregation","cross-modal alignment","instruction tuning","event-text dataset","visual question answering"],"falsifier":"Run the same three metrics with human raters on a held-out real-event dataset whose question-answer pairs are written by event-camera experts after the fact, keeping EventGPT frozen, and compare against an image-based MLLM that has been fine-tuned on the same event-text training data; if EventGPT does not beat that model, the reported superiority is an artifact of the benchmark.","tokens_in":1692,"feed_emoji":"⚡","tokens_out":1784,"duration_ms":66223,"temperature":0.7,"pith_summary":"This paper argues that a multimodal large language model can be made to understand event-camera streams—not just static images—by progressively aligning the event representation with language in three training stages. It introduces EventGPT, built from an event encoder, a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Because event streams are sparse brightness-change signals rather than dense images, the authors first warm up the projector on ordinary image-text data, then align event features to language with a synthetic million-sample dataset, and finally fine-tune the whole model on real-world instruction data. On their own benchmark, EventGPT outperforms existing image-based multimodal LLMs on detailed captioning, complex reasoning, and visual question answering for event scenes. The paper also releases the two event-text datasets it builds.","feed_headline":"First event-stream MLLM beats image models on event scenes","feed_subtitle":"Three-stage training aligns sparse event-camera signals with language, enabling captioning, reasoning, and QA in low light and fast motion.","key_machinery":"The central mechanism is a two-stage cross-modal projector driven by a three-stage training schedule. The spatio-temporal aggregator divides the event stream into $T$ temporal bins, encodes each bin, applies pooling along the temporal and spatial dimensions, and concatenates the pooled features into $\\hat{Z} \\in \\mathbb{R}^{(T+S)\\times D}$, capturing motion cues that a single static event frame lacks. The event-language adapter then maps these fused features into the LLM's embedding space. Its role is to make sparse event data expressible in the language space already anchored by the LLM, so the model can describe scenes, answer questions, and reason rather than merely classify.","core_discovery":"The central claim is that event-stream understanding can be brought to LLMs by treating the event stream as a sequence of temporal windows, encoding each window, pooling spatial and temporal dimensions separately, and fusing the result before projecting it into language space. The authors call this model EventGPT and present it as the first multimodal large language model designed specifically for event streams. The load-bearing device is the three-stage schedule: freeze most modules while training only the projector on image-text pairs, then freeze the encoder, projector, and LLM while training the spatio-temporal aggregator and event-language adapter on synthetic event-text data, then unfreeze everything for instruction tuning on real-world data. This progression, the paper argues, narrows the domain gap between asynchronous event data and natural language and yields superior generation quality, descriptive accuracy, and reasoning compared with state-of-the-art image-oriented multimodal LLMs evaluated on the same event-frame inputs.","pith_inferences":["The same three-stage recipe could transfer to other asynchronous sensors, such as LiDAR, radar, or neuromorphic audio, because the first stage establishes language grounding and the second stage adapts a specialized encoder.","A credible next test would be human-scored evaluation on real outdoor event data with out-of-distribution scenes; the paper's current numbers do not yet establish transfer to unseen distributions because the test questions come from the same sources as the training data.","Because EventGPT inherits the LLM's world knowledge, its descriptions may be plausible but not always grounded; an object-presence probe on synthetic event scenes with known ground truth could quantify the rate of ungrounded claims."],"forward_implications":["EventGPT extends multimodal LLM capabilities to event cameras, enabling scene summarization, reasoning, and visual question answering in low-light and high-dynamic-range scenarios.","The text generated by EventGPT can be piped into open-set detection and segmentation heads, yielding object-level reasoning from event data.","The two released datasets—one million synthetic samples and 120,000 instruction samples—provide a training and evaluation foundation for future event-language models.","The paper reports that a temporal window count of five gives the best results, with performance degrading when windows are too few or too many.","The three-stage alignment paradigm is presented as a reusable recipe for bridging large cross-modal gaps beyond the event-domain setting."],"supporting_citations":[{"why":"Supplies the visual instruction tuning recipe and the image-text pre-training data used in stage one.","marker":"[21]"},{"why":"Provides the synthetic event dataset from which the N-ImageNet-Chat texts are generated.","marker":"[15]"},{"why":"Supplies real-world driving event data used to build the Event-Chat instruction dataset.","marker":"[11]"},{"why":"Supplies additional real-world event data used in the Event-Chat dataset.","marker":"[30]"},{"why":"Provides the evaluation strategy and metric definitions used to score EventGPT.","marker":"[24]"},{"why":"One of the state-of-the-art image-oriented MLLMs used as a comparison baseline.","marker":"[36]"},{"why":"One of the state-of-the-art image-oriented MLLMs used as a comparison baseline.","marker":"[5]"},{"why":"One of the state-of-the-art image-oriented MLLMs used as a comparison baseline.","marker":"[23]"},{"why":"Enables the open-set object detection downstream application demonstrated with EventGPT.","marker":"[22]"},{"why":"Enables the instance segmentation downstream application demonstrated with EventGPT.","marker":"[31]"}],"fun_headline_variants":["EventGPT: First MLLM that truly understands event streams","EventGPT: Event cameras meet language models, beats RGB models","First MLLM for event data wins on low-light scenes","EventGPT: LLM for event streams, three-stage training wins big","EventGPT: Understands event streams without RGB, beats SOTA"],"cache_read_input_tokens":15744,"weakest_assumption_plain":"The reported gains are measured on test splits drawn from the same data distributions used in training, with comparison models that never received event-specific training; if those gains do not survive human-scored or out-of-distribution evaluation, the claimed superiority would not hold.","fun_headline_variants_meta":{"raw":{"variants":["EventGPT: First MLLM that truly understands event streams","EventGPT: Event cameras meet language models, beats RGB models","First MLLM for event data wins on low-light scenes","EventGPT: LLM for event streams, three-stage training wins big","EventGPT: Understands event streams without RGB, beats SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000802,"raw_usage":{"total_tokens":3560,"prompt_tokens":1012,"completion_tokens":2548,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":2459}},"tokens_in":628,"tokens_out":2548,"duration_ms":16893,"temperature":1.0,"reasoning_tokens":2459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:56:38.358207+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three metrics with human raters on a held-out real-event dataset whose question-answer pairs are written by event-camera experts after the fact, keeping EventGPT frozen, and compare against an image-based MLLM that has been fine-tuned on the same event-text training data; if EventGPT does not beat that model, the reported superiority is an artifact of the benchmark.","supporting_citations":[{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the visual instruction tuning recipe and the image-text pre-training data used in stage one."},{"cited_title":"N-imagenet: Towards robust, fine-grained object recognition with event cameras","cited_arxiv_id":null,"evidence_quote":"Provides the synthetic event dataset from which the N-ImageNet-Chat texts are generated."},{"cited_title":"Dsec: A stereo event camera dataset for driving scenarios","cited_arxiv_id":null,"evidence_quote":"Supplies real-world driving event data used to build the Event-Chat instruction dataset."},{"cited_title":"Events-to-video: Bringing modern computer vision to event cameras","cited_arxiv_id":null,"evidence_quote":"Supplies additional real-world event data used in the Event-Chat dataset."},{"cited_title":"Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks","cited_arxiv_id":null,"evidence_quote":"One of the state-of-the-art image-oriented MLLMs used as a comparison baseline."}],"review_version":1}