{"id":"7a659c06-e0bb-4fcb-a717-e8da934bcbb9","arxiv_id":"2507.20077","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Suppressing the EOS token during sequence training makes pretrained VLMs generate longer, more detailed image captions, with a trade-off of more hallucinations.","lead":"The paper shows that vision-language models produce short captions partly because they are biased to predict the end-of-sequence token, and that suppressing this token during training makes them write longer, more detailed captions without extra supervision. The method is simple and works on three existing models, though it also increases hallucinated details.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control tests whether the effect is specific to EOS: the same sequence-training recipe with a non-EOS token penalty is never run, so the central causal mechanism is unverified.","rationale":"I agree with the reader's CONDITIONAL verdict and with the identification of the latent-capability premise as a real risk, but I see a more fundamental gap: the paper never isolates EOS as the causal variable. The only ablation is against inference-time EOS blocking, which differs in both training and inference. If a non-EOS penalty produces the same gains, the method's name and mechanism are wrong even though the procedure might still work. The proposed control of penalizing a matched non-EOS token would settle this. The latent-capability question is then tested by whether gains persist with a blank or input-ablated image; I would add that as a second-order check, but the token-specificity ablation is the one that should be required before acceptance. The limitations on hallucination are acknowledged and quantified in Table 3, so they do not by themselves overturn the central claim; they support the conditional framing. I therefore keep the verdict CONDITIONAL, with the added explicit condition of a token-specificity control.","tokens_in":13779,"tokens_out":5854,"duration_ms":62794,"concrete_test":"Run the identical sequence-training pipeline on BLIP-2 T5, replacing the EOS penalty in Eq. 2 with an equal penalty on a matched-frequency non-EOS token (e.g., the comma/period token or 'the'), keeping all hyperparameters and the length-60 stopping rule fixed. Report CAPTURE, coherence, and hallucination on FineCapEval, DCI, and DOCCI. If this control matches the EOS-debiased results, the claimed EOS-specific mechanism is not supported; if it fails, the causal claim survives this check.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that penalizing the EOS token during sequence training, rather than sequence training in general, recovers latent detailed-captioning ability. The only comparison in Table 2 is against inference-time EOS blocking ('triv.'), which shares no training component with the method and therefore does not test whether the training signal must be applied to EOS. Section 5.3's mechanistic explanation, that the penalty suppresses an entire subspace of early-terminating sequences, is never tested by a control that penalizes a different token. Without such a control, the observed gains could be due to sequence training itself, to the contrastive decoding exploration described in Section 4.1, or to any token-level negative signal, leaving the 'Devil is in the EOS' causal claim unverified. A secondary concern is the underspecified checkpoints: Section 4.1 says training continues until generated captions reach length 60, yet Table 2 reports final lengths of 21-39, and Figure 2 and Table 5 evaluate separately selected checkpoints without stating how the reported models were chosen. The latent-capability premise identified by the reader is real, but the token-specificity control is the more direct test of the paper's own mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EOS debiasing, an unsupervised sequence-training method for detailed image captioning. For a VLM already finetuned on short captions (BLIP-2 OPT, BLIP-2 T5, PaliGemma on COCO), the authors continue training only the cross-modal bridge with a REINFORCE-style gradient that reduces the log-probability of the EOS token in generated sequences. They evaluate on FineCapEval, DCI, and DOCCI using CIDEr, CAPTURE, and GPT-4-based coherence, plus retrieval on Urban-1k and hallucination metrics CHAIRi and ALOHa. They report substantial length and CAPTURE gains over base models and over an inference-time EOS-blocking baseline, with increased hallucination as expected, and conclude that EOS debiasing recovers latent detailed-captioning ability without reward models or supervised detailed-caption data.","tokens_in":13975,"tokens_out":4143,"duration_ms":45309,"significance":"If the central claim holds, the result is significant: it offers an almost parameter-free, reward-free way to extract existing detailed-captioning ability from pretrained VLMs, closing much of the gap to a heavily instruction-tuned 3B model. The paper's strengths are its simplicity, the breadth of the evaluation (three models, three detailed-caption benchmarks, retrieval, and hallucination metrics), the qualitative evidence that added detail is integrated throughout captions rather than appended, and the explicit acknowledgment of limitations and hallucination trade-offs. The main weakness is that the paper's core causal claim, that the EOS token specifically is the devil, is not tested by any control that applies the same sequence-training recipe to a non-EOS token; without such a control, the mechanism remains unverified.","major_comments":[{"comment":"The central claim that EOS debiasing, rather than sequence training in general, is responsible for the gains is not supported by any control experiment. The only training comparison is against the base model and an inference-time EOS-blocking baseline ('triv.'), which shares no training component with the method and therefore does not test whether the negative signal must be applied to the EOS token. A control that applies the identical sequence-training recipe while penalizing a frequent non-EOS token (e.g., 'the' or a uniformly random token) would directly test the proposed mechanism. Without it, the observed improvements could equally be attributed to sequence training itself, to the contrastive decoding exploration described in §4.1, or to any token-level negative signal. The mechanistic account in §5.3, that the penalty suppresses 'an entire subspace of sequences that are likely to terminate early,' is a post-hoc hypothesis that this control would either support or refute.","section":"§3.2, §4.4, §5.3"},{"comment":"The checkpoint-selection and stopping procedure is underspecified and risks selection on the outcome variable. §4.1 states that training continues until generated captions reach a sequence length of 60, yet Table 2 reports final caption lengths of 21–39, not 60. Figure 2 and Table 5 evaluate four later checkpoints (steps 6000, 7500, 8400, and 9150), but the text does not state how these checkpoints were chosen or which checkpoint corresponds to the final model in Tables 2 and 3. Since the training objective directly drives up caption length, and length is itself one of the reported outcomes, stopping or selecting checkpoints based on length can inflate apparent gains. The authors should report the selection rule, results for all checkpoints, and error bars or multiple seeds for the main tables.","section":"§4.1, Table 2, Figure 2, Table 5"},{"comment":"The BLIP-2 OPT result is in tension with the proposed mechanism and needs explicit reconciliation. The paper reports that for BLIP-2 OPT the token being penalized (ID 50118) is the newline token '\\n', not the actual OPT EOS token '</s>' (ID 2). Yet EOS debiasing still improves CAPTURE and retrieval for this model. If the method works by penalizing a newline token, then the description 'EOS debiasing' is inaccurate for this model; if it works despite penalizing a non-termination token, then the claim that the effect is specific to the true EOS token is weakened. The discussion attributes the coherence drop to this mismatch, but the success of the method under the mismatch requires more analysis, for example an ablation that penalizes the true OPT EOS token instead.","section":"§5.2"}],"minor_comments":[{"comment":"The displayed title contains an extra space in 'T raining'; please correct the typographical error.","section":"Title and §4.1"},{"comment":"The GPT-4o coherence evaluation is described with a single prompt and temperature 1.0, but no information is given about the number of captions scored per model-dataset cell, the variance of scores across repeated API calls, or the agreement with human judgments. Please provide these details or cite an established protocol.","section":"§4.3"},{"comment":"The baseline label 'triv.' is introduced in §4.3 but is not defined in the table caption. A one-sentence definition in the caption would make the table self-contained.","section":"Table 2"},{"comment":"The related work cites Yue et al. (2024) on over-generation from weakened EOS probability, but the paper does not connect this to the observed wave-like EOS-probability dynamics in Figure 1b. A brief discussion of how the training dynamics differ from the over-generation failure mode would strengthen the analysis.","section":"§2.3 and §5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the empirical effect is interesting and potentially important. The main reason for major revision, rather than acceptance, is the missing token-specificity control: the paper's title and central claim are about EOS, but the experiments never rule out that sequence training with a different token-level penalty would work just as well. The checkpoint-selection issue is also important to fix. If the authors can add the control experiment and report a clear checkpoint-selection rule with variance estimates, I would be willing to reconsider."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper shows that sequence-training a VLM to minimize the probability of the EOS token reliably produces longer, more detailed captions across three base models and three benchmarks. That is a real and useful empirical result. The main weakness is that the authors never run the control that would justify the title: sequence training with a non-EOS token penalty. Without that, the 'Devil is in the EOS' causal story is supported only by the absence of any other explanation, which is not a control.\n\nWhat is genuinely good: the method is cheap and unsupervised, and the evaluation goes beyond length. Retrieval improves, unigram recall improves, and they explicitly track the hallucination trade-off. The qualitative examples are persuasive—the added detail is integrated, not appended. They also catch a real implementation bug in BLIP-2 OPT where the VLM's EOS is a newline token rather than the actual OPT EOS, and they discuss how that affects coherence. That kind of honesty is creditable.\n\nThe soft spots are real but modest. The numbers come from single runs with no error bars or significance tests. The stopping rule is defined by the outcome variable: they train until generated captions reach length 60, then report final lengths of 21–39, which is a bit confusing. The checkpoint selection for Tables 2 and 5 is underspecified. More importantly, the training objective is a negative signal on one token, and the paper does not show that the signal must be on EOS. The inference-time EOS-blocking baseline is useful, but it does not test the training mechanism. I think the correct interpretation is that the method works, not that we know why.\n\nAll that said, the paper deserves a serious referee. The empirical trend is consistent and the method is simple enough that it should be reproducible. I would ask for the token-specificity control, error bars, and a cleaner description of the checkpoint selection before accepting, but I would not reject on the current evidence.","headline":"EOS debiasing works in practice, but the paper does not actually test whether EOS is the critical token.","tokens_in":14577,"tokens_out":2986,"would_cite":true,"duration_ms":30720,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that penalizing the end-of-sequence token during sequence training is enough to make pretrained vision-language models generate longer, more detailed captions, and that this recovers latent capability rather than adding…","keywords":["EOS debiasing","sequence training","detailed image captioning","vision-language models","hallucination","caption length","unsupervised adaptation","end-of-sequence token"],"falsifier":"One concrete check: take a VLM whose language backbone was not pretrained on long-form text (or whose EOS token in the VLM vocabulary is not the true sentence terminator, as in BLIP-2 OPT), apply EOS debiasing, and see whether recall and coherence rise with length; if the extra tokens are not image-grounded, the recovery-of-latent-capability explanation is wrong.","tokens_in":13550,"feed_emoji":"🖼️","tokens_out":7912,"duration_ms":69973,"temperature":0.7,"pith_summary":"The paper claims that a large part of the detail deficit in vision-language captioning is not missing knowledge but a training artifact: cross-entropy training on short captions biases the model toward predicting the end-of-sequence (EOS) token too early. It proposes an unsupervised remedy: during a short sequence-training phase, minimize the probability of the EOS token only, with no reward model and no detailed captions as supervision. Across three pretrained VLMs and three detailed-captioning benchmarks, this single negative signal lengthens captions, raises recall of image-relevant details, and improves retrieval, while coherence stays largely stable. The authors interpret the result as recovering capabilities already latent in the vision and language backbones, rather than instilling new ones. If right, the method offers a training-data-free first step for any task whose available examples are short and generic.","feed_headline":"Penalize the stop token, unlock detailed image captions","feed_subtitle":"No new captions or reward models: a single negative gradient on the end-of-sequence token recovers latent detail.","key_machinery":"The machinery is EOS debiasing as a sequence-training objective: from each model-generated caption ending in EOS, the only training signal is $\\nabla L(\\theta) = \\nabla_\\theta \\log p_\\theta(t_n)$, a REINFORCE-style gradient that lowers the probability of the EOS token. This suppresses an entire subspace of early-terminating sequences and reshapes the token-level probability landscape; the self-attention mechanism then redistributes probability mass toward sequences whose structure differs from early termination, so detail is integrated naturally rather than appended at the end.","core_discovery":"The central discovery is that the EOS token is the handle that controls caption length: because every training caption ends in EOS, teacher-forced cross-entropy training overshoots the probability of EOS, and generation terminates prematurely. Sequence training with the gradient $\\nabla L(\\theta) = \\nabla_\\theta \\log p_\\theta(t_n)$ for the EOS token only, applied after standard finetuning on COCO, gradually suppresses that bias. The resulting models do not simply postpone termination; the extra detail is distributed throughout the caption and remains grounded in the image, as shown by CAPTURE gains, retrieval improvements, and training-progression plots where recall rises with length. The approach outperforms the inference-time alternative of simply blocking EOS generation, which yields incoherent concatenations of short captions.","pith_inferences":["The paper recommends brief EOS debiasing before supervised finetuning; a testable extension is measuring how much less detailed caption data is needed after the debiasing stage.","The BLIP-2 OPT result, where the debiased 'EOS' is actually a newline token, implies practitioners should verify the identity of the termination token first; penalizing a non-termination token may be what causes the coherence drop.","A direct falsification of the latent-capability story would be to apply EOS debiasing to a model whose language backbone was never exposed to long-form text and show that longer output degrades into repetition.","The interaction between EOS debiasing and decoding is underexplored: the paper trains with contrastive decoding but generates with beam search plus repetition penalty, so the optimal inference recipe after debiasing remains an open question."],"forward_implications":["Any pretrained VLM can be pointed at detailed captioning without detailed data or a reward model; the paper demonstrates this on BLIP-2 with OPT, BLIP-2 with FlanT5, and PaliGemma by finetuning only the cross-modal bridge.","EOS debiasing beats the trivial inference-time fix of blocking EOS generation on CAPTURE and coherence for nearly every model and dataset, because gradual sequence training preserves distributional coherence.","The added length is informative, not padding: recall of reference objects rises with length during training, and text-to-image retrieval on Urban-1k improves sharply.","Hallucination grows with length, but stays in the same range as a much larger instruction-tuned model trained explicitly for detailed captioning.","As a direct corollary, the method transfers to any conditional task whose available training text is short and generic, provided the base model already has latent knowledge of the target output."],"supporting_citations":[{"why":"Establishes that the EOS token is a learned length counter; removing it allows extrapolation to longer sequences, motivating the debiasing target.","marker":"Newman et al. (2020)"},{"why":"Shows EOS oversmoothing biases models toward short sequences and premature termination, the failure mode the paper addresses.","marker":"Kulikov et al. (2021)"},{"why":"Introduces sequence training with REINFORCE-style gradients, the training paradigm used for EOS debiasing.","marker":"Ranzato et al. (2016)"},{"why":"Adapts sequence training to image captioning by optimizing whole generated sequences, the framework the paper builds on.","marker":"Rennie et al. (2017)"},{"why":"Documents that weakened EOS probabilities in instruction-tuned VLMs cause over-generation, defining the balance the paper seeks.","marker":"Yue et al. (2024)"},{"why":"SMILE, the closest related method, penalizes missing ground-truth tokens but not extra ones; the paper positions EOS debiasing against it.","marker":"Yue et al. (2023)"},{"why":"Provides FineCapEval and a CLIP-reward baseline; a benchmark and a comparison point for reward-based detailed captioning.","marker":"Cho et al. (2022)"},{"why":"Supplies the DCI benchmark and the LongCLIP encoder used to measure retrieval improvement.","marker":"Urbanek et al. (2024)"},{"why":"Supplies the DOCCI benchmark with long, detailed captions used for evaluation.","marker":"Onoe et al. (2024)"},{"why":"Provides the CAPTURE metric used to measure recall and completeness of visual elements in generated captions.","marker":"Dong et al. (2024)"}],"fun_headline_variants":["Penalize EOS to reveal hidden caption detail","Debias the stop token, captions grow longer","Less EOS bias, more description in captions","One gradient tweak for richer image captions","Stop token bias: the simple fix for caption detail"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the base VLM already knows how to describe images in detail, so suppressing the probability of the EOS token will surface image-grounded detail instead of incoherent repetition; the paper also assumes, more quietly, that the token it penalizes is the model's true termination token.","fun_headline_variants_meta":{"raw":{"variants":["Penalize EOS to reveal hidden caption detail","Debias the stop token, captions grow longer","Less EOS bias, more description in captions","One gradient tweak for richer image captions","Stop token bias: the simple fix for caption detail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1771,"prompt_tokens":879,"completion_tokens":892,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":817}},"tokens_in":495,"tokens_out":892,"duration_ms":9196,"temperature":1.0,"reasoning_tokens":817,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:49:41.127964+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete check: take a VLM whose language backbone was not pretrained on long-form text (or whose EOS token in the VLM vocabulary is not the true sentence terminator, as in BLIP-2 OPT), apply EOS debiasing, and see whether recall and coherence rise with length; if the extra tokens are not image-grounded, the recovery-of-latent-capability explanation is wrong.","supporting_citations":[{"cited_title":"Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling","cited_arxiv_id":"2112.08914","evidence_quote":"Shows EOS oversmoothing biases models toward short sequences and premature termination, the failure mode the paper addresses."},{"cited_title":"Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation","cited_arxiv_id":"2306.13460","evidence_quote":"SMILE, the closest related method, penalizes missing ground-truth tokens but not extra ones; the paper positions EOS debiasing against it."},{"cited_title":"A Picture is Worth More than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions","cited_arxiv_id":null,"evidence_quote":"Supplies the DCI benchmark and the LongCLIP encoder used to measure retrieval improvement."},{"cited_title":"DOCCI: Descriptions of Connected and Contrasting Images","cited_arxiv_id":null,"evidence_quote":"Supplies the DOCCI benchmark with long, detailed captions used for evaluation."}],"review_version":2}