{"id":"a5e62b9d-b5ea-4952-97a5-fd7ca8db2e43","arxiv_id":"2412.11556","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Prepending each early layer's decoded sentence embedding to the next layer's input improves prompt-based sentence embeddings from decoder-only LLMs without fine-tuning.","lead":"The paper introduces Token Prepending, a training-free tweak that inserts a summary token into early layers of a decoder-only language model so earlier words can see information from later words. It reports consistent gains on sentence similarity and classification benchmarks with almost no extra inference cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main tables conflate token prepending with early exit: PromptEOL+TP exits at layer 27 while baseline PromptEOL uses the last layer, so the reported gains may be due to early exit alone.","rationale":"The reader's verdict is CONDITIONAL, and the rationale explicitly flags the missing early-exit-only baseline for PromptEOL and the conflation of TP with early exit. My stress-test identifies this as the single most load-bearing concern: it directly undermines the abstract's claim that the TP operation itself, independent of early exit, grants the improvement. The concern is concrete and testable via a small ablation, and the paper already contains the relevant comparison for two other prompts (Figure 4d), suggesting the authors have the means to run it for PromptEOL. I did not choose the reader's stated weakest assumption (semantic meaningfulness of layer-boundary injections) because the empirical evidence, if properly controlled, would largely address that concern; the missing control is the more urgent issue. Since the reader already asked for additional experiments of exactly this kind, my read does not change the verdict; it sharpens the condition. I mark agreement as 'partial' because the reader's weakest_assumption field was about a different assumption, although the rationale did identify the same confound.","tokens_in":19989,"tokens_out":5758,"duration_ms":46375,"concrete_test":"Run the missing ablation on LLaMA2-7B with PromptEOL: (1) vanilla PromptEOL with last-layer exit (baseline); (2) vanilla PromptEOL with early exit at layer 27 (exit-only); (3) PromptEOL+TP with exit at layer 27 (as reported); (4) PromptEOL+TP with exit at the last layer (TP-only, no early exit). Compare the average Spearman correlation across the seven STS datasets. If (2) is close to (3), or (4) is close to (1), the headline claim overstates TP's contribution; if (3) clearly exceeds both (2) and (4), and (4) exceeds (1), then TP itself is validated independently of early exit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract; Section 5.3) is that Token Prepending itself improves prompt-based sentence embeddings. However, the primary evidence in Table 1 compares PromptEOL (vanilla, last-layer output) with PromptEOL+TP, which uses both TP (layers 1–8) and early exit at layer 27 (Section 5.1). LLaMA2-7B has 32 layers, so the two conditions differ in two variables: the TP operation and the exit layer. Because prior work and the paper's own Section 4.3 note that later layers are less semantically informative, early exit alone could plausibly explain a substantial part of the 7.16-point gain. The paper does provide Figure 4(d) for Pretended CoT and Knowledge Enhancement showing TP improves over no-TP at matched exit layers, but no such matched-exit-layer ablation is given for PromptEOL, the method with the largest reported gain. Without a 2×2 decomposition (exit layer × TP) for PromptEOL, the abstract's attribution of the gain to TP is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Token Prepending (TP), a training-free, inference-time intervention for extracting sentence embeddings from decoder-only LLMs. TP inserts a placeholder token <PST> at the beginning of a prompt, then for the first k layers replaces that placeholder's embedding with the previous layer's hidden state of the sentence-embedding token (<SET>), allowing earlier tokens to attend to full-sentence information despite causal attention. After layer k, standard forward propagation resumes, and the final sentence embedding is taken from an intermediate layer M (early exit) rather than the final layer. Experiments on seven STS benchmarks, SentEval transfer tasks, and MTEB subsets show consistent gains over prompt-based baselines (e.g., +7.16 average for PromptEOL on STS, 20/21 improvements on transfer tasks) with inference time reported as about 1.04× the baseline.","tokens_in":20232,"tokens_out":7068,"duration_ms":59813,"significance":"If the causal claim is validated, TP is a simple, plug-and-play, training-free technique that could broadly improve prompt-based sentence embeddings across LLMs. The paper's strengths include a wide empirical scope (STS, SentEval, MTEB subsets), a released code repository, and thorough ablations of the placeholder position, initialization, layer scope, and early-exit layer. However, the central attribution of the improvements to TP itself is currently undermined by the confound with early exit in the headline results; the matched-exit-layer evidence provided in the paper covers only two of the four prompt baselines. The method is plausible and the experimental foundation is solid, but the main claim requires additional controlled comparisons.","major_comments":[{"comment":"The headline comparison in Table 1 is confounded: the PromptEOL baseline is described in Section 3 as using the final layer's hidden state for the sentence-embedding token, whereas PromptEOL+TP uses exit layer M=27 (Section 5.1). Because the paper itself argues (Section 4.3 and Figure 4(d)) that intermediate layers provide better STS performance than the final layer, the reported +7.16 average gain cannot be attributed to TP alone without a matched-exit-layer baseline. The paper provides such matched-layer evidence in Figure 4(d) for Pretended CoT and Knowledge Enhancement, but not for PromptEOL, which is the method with the largest reported gain. Please add a 2x2 comparison for PromptEOL and MetaEOL (exit layer in {last, 27} × TP in {off, on}) and state explicitly which exit layers were used for every baseline in Table 1.","section":"Section 5.1 / Table 1"},{"comment":"The transfer-learning results in Table 5 are presented without specifying the exit layers used for the baselines. Section 5.8 states that ending TP at layers 14–21 enhances transfer performance, but it does not say whether the PromptEOL, Pretended CoT, and Knowledge baselines use the final layer or the same early-exit layer as their TP counterparts. If the baselines use the final layer, the reported improvements (e.g., +0.48 for PromptEOL+TP, +1.04 for Pretended CoT+TP on MRPC) may again be partly due to early exit. Please provide the exact layer configurations for all rows of Table 5 and, ideally, matched-layer baselines.","section":"Section 5.8 / Table 5"}],"minor_comments":[{"comment":"The formal specification of intermediate token prepending is ambiguous: the displayed definition of f(h^{l-1}) contains a sequence with a duplicated last element and does not explicitly state that the hidden state at the <PST> position i* is replaced by the hidden state of the <SET> token while all other positions are copied unchanged. Please rewrite the equations with a clear substitution rule for position i* and make the indexing consistent with Figure 2.","section":"Section 4.2.2"},{"comment":"The phrase \"ensuring the same output layer\" in the Time column caption is inconsistent with the score columns, which appear to use layer 27 for TP and the final layer for the PromptEOL baseline; please clarify exactly which output-layer setting was used for the timing measurement.","section":"Table 1 caption"},{"comment":"There are several typos and encoding artifacts: \"autogressive\" in Section 6, \"utlizes\" in Section 5.2, \"langauge\" in Section 1, \"V oorhees\" in the reference list, and stray \"/uni000...\" strings in Section 5.5 and the caption of Figure 3 that appear to be PDF-extraction artifacts. Please proofread the source files.","section":"Throughout"},{"comment":"The text says \"we pretend the sentence embedding token <SET> to replace <PST>\" where \"prepend\" or \"use\" is clearly intended; please correct this wording.","section":"Section 4.2.2"},{"comment":"The Limitations section acknowledges that the best hyperparameters (end layer k and exit layer M) vary by model, dataset, and prompt. Since the abstract calls the method \"training-free,\" it would be helpful to state explicitly that this tuning is performed on a validation split, and to report the sensitivity of PromptEOL's final score to k and M in the same way Figure 4 does for Pretended CoT and Knowledge Enhancement.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is valid: the paper's largest reported gain is for PromptEOL, and no matched-exit-layer ablation is provided for that prompt. The fix is straightforward (a 2x2 decomposition), and the rest of the evidence suggests the method has merit, so I recommend major revision rather than rejection. No concerns about citation patterns or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the TP operation is a genuinely new, simple trick for improving training-free sentence embeddings, and the aggregate evidence is real. But the main table compares PromptEOL+TP (early exit at layer 27) against vanilla PromptEOL (last layer), so the headline 7.16-point gain isn't cleanly attributable to TP alone. The stress-test note is right about the confound, though not fatal: Figure 4(d) shows TP beating no-TP at matched exit layers for Pretended CoT and Knowledge Enhancement, so the technique does something; we just don't have that ablation for the method that benefits most.\n\nWhat's actually new: injecting the previous layer's sentence-token hidden state into the next layer's input at a prepended position, for the first ~8 layers, is not in the cited prompt-engineering or repetition literature. It's a clean, training-free intervention with a plausible mechanism (giving earlier tokens access to full-sentence information under causal attention). The evaluation is broad: STS 2012-2016, STS-B, SICK-R, SentEval transfer tasks, and an MTEB subset in the appendix, across LLaMA2-7B/13B, LLaMA3-8B, Qwen2-7B, Gemma2-9B. Improvements are consistent in aggregate (26/28, 20/21, 40/44), and the code is public.\n\nSoft spots, in proportion. The confound in Table 1 is the main one; it's fixable with a 2x2 decomposition (exit layer x TP) for PromptEOL. Also missing: variance or significance info, so we don't know if the small gains (e.g., 0.40 on Knowledge) are noise. The method has four hyperparameters tuned on STS-B dev; the authors admit this in the Limitations section, and it does temper the 'plug-and-play' claim. The lack of any theoretical story for why layer-boundary hidden-state injection works is a gap, but not a fatal one for an empirical paper. The appendix's masked-<PST> and resumption ablations are thoughtful and actually strengthen the paper.\n\nWho this is for: anyone working on LLM-based sentence embeddings or probing intermediate representations. It deserves a serious referee. My recommendation: send it out, but ask for the PromptEOL early-exit-only ablation and matched-exit comparisons, plus some measure of variance across runs.","headline":"A simple, training-free trick that mostly works, but the main table muddies the credit by bundling early exit with token prepending.","tokens_in":20770,"tokens_out":4157,"would_cite":true,"duration_ms":32926,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free trick makes LLMs better sentence encoders.","keywords":["sentence embeddings","large language models","token prepending","causal attention","backward dependency","training-free","semantic textual similarity","early exit"],"falsifier":"Run the STS-B comparison with TP, but instead of injecting the previous layer's sentence-token hidden state, inject the hidden state of a randomly chosen token from a different sentence, or from a shuffled version of the same sentence. If the Spearman gain over the vanilla prompt remains statistically unchanged, the effect is not due to carrying whole-sentence meaning backward through the placeholder, and the paper's mechanism account fails.","tokens_in":19779,"feed_emoji":"🧩","tokens_out":9078,"duration_ms":77428,"temperature":0.7,"pith_summary":"Autoregressive LLMs use causal attention, so early tokens in a sentence cannot see the later tokens that complete its meaning. The paper tries to fix this at inference time, with no training, by prepending a single placeholder token before the sentence and, between the first few transformer layers, overwriting that placeholder with the sentence-embedding token's hidden state from the previous layer. Because every later token can attend to this prepended position, the model effectively gains backward dependency while keeping its causal structure. The paper shows that this Token Prepending (TP) technique improves existing prompt-based sentence embedding methods on semantic textual similarity benchmarks and downstream classification tasks across several LLMs, with inference cost close to unchanged. If the gain is real, it gives a plug-and-play way to make decoder-only LLMs better sentence encoders without fine-tuning.","feed_headline":"Insert one token, get LLM sentence embeddings improving across tasks","feed_subtitle":"By piping each layer's decoded sentence back to the start, earlier tokens finally see the whole sentence with no retraining.","key_machinery":"The load-bearing object is the layer-boundary substitution of the <PST> placeholder by the previous layer's sentence-embedding-token hidden state. At the first layer, <PST> is just a randomly initialized token that keeps sequence length stable; from layer 2 up to layer k, its embedding is overwritten with the SET's hidden state before the next transformer layer runs. Since causal attention lets every token to the right of <PST> attend to it, this single position acts as a channel through which complete sentence information flows back to all early tokens. The early-exit choice is a second mechanism: because later layers are tuned for token generation, an intermediate layer's hidden state is selected on the validation set and used as the final embedding.","core_discovery":"The paper's central claim is that a decoder-only LLM can be turned into a stronger sentence encoder by letting the whole sentence be visible to every token before the final embedding is read. Concretely, it inserts a placeholder token <PST> before the input sentence in the prompt, then for the first k layers (around 7-8 in LLaMA2-7B) it replaces that placeholder's input embedding at each layer boundary with the hidden state of the sentence-embedding token (the prompt's final token, which has already attended to the full sentence). Early tokens thus attend to a token that carries entire-sentence information, in effect creating backward dependency without changing the attention mask or any weights. After layer k the model runs normally, and an intermediate-layer hidden state is used as the sentence embedding. On seven STS benchmarks the technique improves PromptEOL by 7.16 points on average on LLaMA2-7B, improves other prompt baselines and other decoder-only backbones, and improves 40 of 44 tasks in the paper's extended evaluation, at under 1.04x inference cost.","pith_inferences":["An untested extension: the same layer-wise input injection might serve as a training-free way to give decoder-only models bidirectional context for other tasks that need it, such as coreference or relation extraction, without changing their weights.","Because the optimal TP horizon lands in early layers (around 7-8 for 7B models), the stopping layer could be probing where the model transitions from contextual to more abstract semantic representations; one could test whether this horizon shifts with model depth, amount of pretraining, or task type.","TP could be composed with other inference-time steering techniques, such as repetition or auxiliary prompts, and whether the gains are additive, overlapping, or saturating is not tested in the paper.","In practice, TP requires validation-set selection of the stopping layer and the exit layer, and the paper's own limitations section notes these choices vary by model, dataset, and prompt; an automatic rule would make the technique turnkey."],"forward_implications":["Any prompt-based method that decodes a sentence embedding from the last token can be upgraded with TP; the largest measured gain is +7.16 average Spearman on STS for PromptEOL with LLaMA2-7B.","TP transfers across decoder-only backbones: it improves LLaMA2-7B and LLaMA2-13B, LLaMA3-8B, Qwen2-7B, and Gemma2-9B, with the largest backbone gain on Qwen2-7B.","The resulting embeddings benefit downstream tasks: the paper reports gains in 20 of 21 transfer-classification comparisons and 40 of 44 extended task comparisons, including clustering, reranking, and pair classification.","TP narrows the gap between different prompts, so simple prompts like PromptEOL gain the most and prompt engineering matters less for final quality.","The added inference cost is small, since only one token is inserted and KV cache is reused; measured time is about 1.04 times that of the plain PromptEOL baseline."],"supporting_citations":[{"why":"Introduces PromptEOL, the last-token prompt template TP plugs into and the baseline with the biggest measured improvement.","marker":"Jiang et al. (2023)"},{"why":"Repetition-based Echo embeddings, the prior approach for giving early tokens access to later context; TP is positioned as a cheaper alternative.","marker":"Springer et al. (2024)"},{"why":"MetaEOL meta-task prompts, one of the prompt baselines TP is combined with.","marker":"Lei et al. (2024)"},{"why":"Pretended CoT and Knowledge Enhancement prompts, the baselines used for cross-backbone and transfer experiments.","marker":"Zhang et al. (2024)"},{"why":"BeLLM's backward-dependency enhancement motivates TP's goal and supplies a bidirectional-attention comparison.","marker":"Li and Li (2024)"},{"why":"Evidence that removing the causal mask hurts LLM embeddings, used in the appendix's bidirectional-attention ablation.","marker":"BehnamGhader et al. (2024)"},{"why":"Shows intermediate layers carry more semantic information than the final layer, supporting the early-exit strategy.","marker":"Liu et al. (2024b)"}],"fun_headline_variants":["Training-free token prepending lifts LLM sentence embeddings","Prepend one token to let earlier LLM tokens see the full sentence","Simply prepend a token: LLM embeddings improve 7 points without training","Causal attention workaround: prepend sentence info token to LLM","No training needed: prepend token to get better LLM sentence embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gain rests on the assumption that overwriting the placeholder token with the previous layer's sentence-token hidden state is a semantically meaningful injection for a model that was never trained on such layer-boundary edits, and not just a benign or disruptive perturbation.","fun_headline_variants_meta":{"raw":{"variants":["Training-free token prepending lifts LLM sentence embeddings","Prepend one token to let earlier LLM tokens see the full sentence","Simply prepend a token: LLM embeddings improve 7 points without training","Causal attention workaround: prepend sentence info token to LLM","No training needed: prepend token to get better LLM sentence embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001231,"raw_usage":{"total_tokens":5077,"prompt_tokens":984,"completion_tokens":4093,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":4014}},"tokens_in":600,"tokens_out":4093,"duration_ms":24823,"temperature":1.0,"reasoning_tokens":4014,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:48:36.772242+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the STS-B comparison with TP, but instead of injecting the previous layer's sentence-token hidden state, inject the hidden state of a randomly chosen token from a different sentence, or from a shuffled version of the same sentence. If the Spearman gain over the vanilla prompt remains statistically unchanged, the effect is not due to carrying whole-sentence meaning backward through the placeholder, and the paper's mechanism account fails.","supporting_citations":[],"review_version":1}