{"id":"60e9d97f-87ac-465d-befa-c2065042223e","arxiv_id":"2411.12156","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"HNCSE reports 2-point average STS gains over SimCSE using positive mixing and hard-negative mixing, but the method is under-specified and unverified.","lead":"An extension of SimCSE that mixes the hardest in-batch negative into the positive and generates extra hard negatives via embedding interpolation reports gains of about two points on semantic similarity benchmarks. The paper's central equation is incomplete and no code or data is released, so the gains cannot currently be verified or reproduced.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main update rule is under-specified and Eq. (2)'s d1<d2 branch has negative coefficients; the central claim is not reproducible from the text.","rationale":"The reader's weakest assumption is that cosine-nearest in-batch negatives are semantically the hardest negatives and that linear interpolation produces valid positives/negatives. That is a real concern, but I see a more immediate, load-bearing blocker: the method as written cannot be executed. The d1<d2 branch of Eq. (2) produces negative coefficients, and no weight definitions, normalization, or hyperparameter values are provided for Eq. (7)-(8). Even if the semantic assumption were granted, no reader could implement the model from the text. The paper's limitation section (Section 5) discloses only issues about data noise and lack of contextual information, not the missing specification or unexecutable equations; the appendix claims theoretical analysis but does not derive the main update rule. I therefore agree with the REJECT verdict, though the idea might become acceptable if the authors release a corrected, fully specified implementation with code and seed-level variance. Since my read reinforces rather than changes the reader's verdict, I mark the verdict as UNCHANGED.","tokens_in":17467,"tokens_out":5009,"duration_ms":54568,"concrete_test":"Evaluate Eq. (2) on a concrete pair with d1=0.6 and d2=0.8: the d1<d2 branch yields s+2=(-1/3)s+1+(-1/4)s-1. Check whether this vector is a valid positive training target, i.e., whether it has nonnegative coefficients and unit norm after the operation. If it is not valid, the central PM construction is undefined and the reported STS gains cannot be attributed to it; request a corrected equation and code from the authors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that HNCSE-PM and HNCSE-HNM improve on SimCSE, requires that the method be implementable from the paper. It is not. In Eq. (2), the third branch applies when d1<d2; then both (d1-d2)/d1 and (d1-d2)/d2 are negative, so s+2 is a negatively weighted combination of the positive and hardest-negative embeddings. This is not a convex mixup and, if embeddings are l2-normalized, the output is not normalized; no renormalization is specified. In the first branch, w1 and w2 are never defined, so the mixup weights are unconstrained. Likewise, Eq. (7) and (8) leave k, m, alpha, and f(sim(Gi,Gj)) unspecified; Eq. (7)'s notation even switches from top-k to G={G1,...,Gm}. The appendix does not resolve this: A.2 uses lambda in [0,1] for a different mixup, but does not connect to main-text Eq. (2), and A.3/A.4 are duplicated. No code or data are released, and no variance or seed information is given; the reported gains over SimCSE (e.g., 78.38 vs 76.16 average on STS-Base) could be within run-to-run noise. Because the load-bearing update rule cannot be instantiated from the text, the reader's rejection is warranted; the empirical claim might survive only if a corrected, code-backed version is supplied.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HNCSE, an extension of unsupervised SimCSE for sentence embeddings, with two components: Positive Mixing (PM), which interpolates the positive embedding with the hardest in-batch negative, and Hard Negative Mixing (HNM), which generates synthetic hard negatives by linearly combining top-k in-batch negatives. The authors report Spearman correlations on seven STS benchmarks and accuracies on seven transfer tasks, claiming consistent improvements over SimCSE. They also include an appendix intended as a theoretical analysis of HNM.","tokens_in":17941,"tokens_out":5589,"duration_ms":48107,"significance":"Hard negatives are known to improve contrastive learning in vision, so adapting this idea to sentence embeddings is a plausible and potentially useful direction; the experimental study covers standard benchmarks. However, the central contribution is not implementable from the text: the positive-mixing rule is mathematically undefined or invalid, the HNM hyperparameters and selection function are unspecified, and the baselines are inconsistently chosen. The appendix does not provide a derivation from stated assumptions to the claimed improvements. Unless these gaps are closed and code, seeds, and variance estimates are supplied, the reported gains cannot be verified or attributed to the proposed mechanism.","major_comments":[{"comment":"The positive-mixing update rule is not well-defined. The weights w1 and w2 are introduced but never given a value or constraint. In the third branch (d1 < d2), both coefficients (d1-d2)/d1 and (d1-d2)/d2 are negative, producing a non-convex, potentially non-normalized combination of s+_1 and s-_1; no renormalization is specified. Since s+_2 is used as the training target, this branch is mathematically invalid as written and prevents faithful reproduction.","section":"Section 3.1, Eq (2)"},{"comment":"The hard-negative generation procedure is under-specified. The set G is first called 'top k vectors' but defined as G={G1,...,Gm}; the roles of k and m are unclear. The mixing coefficient α is given only as α=f(sim(Gi,Gj)) with f unspecified, and no values for α, k, or m are reported in the experiments. Appendix A.2 defines a different mixup with λ∈[0,1] and never connects it to Eq (7).","section":"Section 3.2.1, Eqs (7)-(8)"},{"comment":"The comparisons are not apples-to-apples. Implementation details state that HNCSE uses a BERT model ('Utilizing a BERT model that's been finely optimized'), and Table 1 lists 'SIMCSE' without architecture, while Table 2 uses 'SimCSE-RoBERTa' as the baseline. If HNCSE is BERT-based, comparing against SimCSE-RoBERTa in Table 2 is favorable to HNCSE and does not isolate the proposed components. The text also overclaims: Section 4.3.1 says HNCSE surpasses SimCSE across all STS benchmarks, but Table 1 shows HNCSE-HNM-Base underperforms SIMCSE(reproduce) on STS12 (69.76 vs 70.82).","section":"Section 4.2 and Tables 1-2"},{"comment":"No code, data, seeds, or variance estimates are provided, so the reported improvements over SIMCSE(reproduce) (e.g., 78.38 vs 76.16 averaged over seven STS tasks) cannot be distinguished from run-to-run noise. The so-called theoretical analysis in Appendix A is largely a restatement of the contrastive loss and generic claims; A.3 and A.4 are identical, and parameters β, λ, γ in Eqs (16), (17), (24) are never used in experiments. This does not provide the announced 'theoretical analysis of HNM'.","section":"Section 4.3, Table 3, and Appendix A"}],"minor_comments":[{"comment":"The row 'SIMCSE(reproduce)' has STS-B 77.24, while the original SimCSE paper reports 76.85 for STS-B; the authors should clarify whether the reproduced baseline matches the original hyperparameters and report the variance across runs.","section":"Table 1, Base section"},{"comment":"The sentence 'In the study by Zhang et al. (2022), we benchmarked...' is grammatically awkward; the citation appears misplaced and should be rewritten.","section":"Section 4.3.1"},{"comment":"Figure 3 contains garbled placeholder text (e.g., '???1', '?????????') and should be redrawn with proper labels.","section":"Figure 3"},{"comment":"Equation (7) uses the same symbol uo for the unnormalized and normalized vector; the notation should distinguish them.","section":"Eq (7)"},{"comment":"The description of HNCSE-PMsingle ('only optimize the case where d1 is greater than d2') conflicts with the three-branch structure of Eq (2); clarify which branch is used.","section":"Section 4.3.4"}],"recommendation":"reject","confidential_remarks":"This manuscript appears to be an early preprint with substantial writing and reproducibility gaps. The core method cannot be evaluated as written. I recommend rejection, but note that a corrected version with a fully specified algorithm, code release, and proper baselines could be a legitimate incremental contribution. I am not questioning the authors' integrity; the issues are internal to the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on HNCSE. The one thing worth remembering: Positive Mixing—using the hardest in-batch negative to nudge the positive embedding toward the query—is a real twist on SimCSE that I haven't seen elsewhere. The paper also adapts Kalantidis-style hard negative mixing to sentence embeddings, and the reported STS and transfer gains are consistent across both variants and both model sizes. The ablation study supports both components, and the comparison with LLaMA2 prompt methods is a reasonable addition. So the empirical story, if true, would be a cheap and useful upgrade.\n\nBut the story cannot be checked from the manuscript. Equation (2) is the load-bearing update, and as written it is not a valid mixup: w1 and w2 are never defined, and in the d1<d2 branch both coefficients are negative. That branch alone makes the method unimplementable as stated. Section 3.2 is just as loose: k, m, alpha, and the function f in Eq. (8) are all unspecified, and the notation in Eq. (7) even changes from top-k to a set G of size m. No code or data are released, no variance or seeds are given, and the appendix duplicates two sections and offers restatements of the contrastive loss rather than derivations. A single unverified run with a 2-point gain over SimCSE is not enough to carry that weight.\n\nI want to be fair: the idea is plausible, and the authors cite Kalantidis and Zhang et al. 2022 appropriately. This is not a citation-pattern problem. It is a writing and completeness problem. A corrected version with a well-defined PM branch, concrete HNM settings, and released code would deserve a serious look. The current version does not.\n\nMy take: desk reject, with an invite to resubmit a corrected and reproducible version. If you want to use this in a reading group as an example of how a promising idea can be sunk by under-specification, it could be useful. Otherwise, skip it.","headline":"Promising new Positive Mixing idea, but Eq. (2) is broken and the method is not reproducible from the text; desk reject.","tokens_in":18434,"tokens_out":4014,"would_cite":false,"duration_ms":37167,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hard-negative mixing into positives and negatives gives sentence embeddings that beat SimCSE","keywords":["unsupervised sentence representation learning","contrastive learning","hard negative samples","SimCSE","positive mixing","semantic textual similarity","transfer learning","sentence embeddings"],"falsifier":"Retrain HNCSE with the same hyperparameters but with the 'hardest negative' replaced by a random in-batch sentence; if the average STS Spearman score stays near 78, the claimed effect is not due to hardness. Alternatively, compare against a variant that defines hardness by a paraphrase model instead of cosine distance: if the paraphrase-based hard negatives do not improve on the cosine-based ones, the geometric assumption is the active ingredient; if they do, the paper's mechanism is a proxy rather than the cause.","tokens_in":17251,"feed_emoji":"📈","tokens_out":6282,"duration_ms":58524,"temperature":0.7,"pith_summary":"This paper tries to establish that unsupervised sentence representations improve when contrastive training deliberately exploits hard negatives—samples that sit close to the query in embedding space and are therefore easy to confuse with the positive. The authors extend SimCSE with two mechanisms. Positive Mixing interpolates the positive target with the hardest in-batch negative when the positive is unexpectedly farther from the query than that negative, pulling the positive closer. Hard Negative Mixing linearly blends embeddings of the top-k nearest negatives to synthesize new, more challenging negatives. If the reported results hold, the claim is that a simple geometry-based hard-negative recipe gives consistent gains over SimCSE on semantic similarity and transfer tasks, and also outperforms prompted LLaMA2-7B variants on STS.","feed_headline":"Hard-negative mixing pushes sentence embeddings past SimCSE","feed_subtitle":"Two mixup tricks—one for positives, one for negatives—raise STS averages to 78.4 and beat prompted LLaMA2.","key_machinery":"Two named mechanisms carry the argument. Positive Mixing (PM) is the conditional interpolation rule of Eq. (2): it compares the query–positive similarity d1 with the query–hardest-negative similarity d2 and, depending on the gap, replaces the positive target s+1 with a weighted mixture w1s+1+w2s−1 (or with a ratio-weighted mixture when d1<d2), so the positive moves toward the query and away from the confusing negative. Hard Negative Mixing (HNM) is the linear-combination generator of Eq. (7): from the top-k nearest neighbours of the query in a previous minibatch it forms m new negatives uo = αGi + (1−α)Gj, with α adapted to the similarity of Gi and Gj, L2-normalizes them, and adds them to the negative bank of the InfoNCE loss. The paper's Appendix A offers additional formalizations (hard-negative threshold, loss, gradients, embedding-space distance change) meant to show why mixing should improve discriminability.","core_discovery":"The central discovery is that hard negatives can be used in both directions: to repair weak positives and to create stronger negatives. For each query, the method computes cosine distances d1 to its dropout positive and d2 to the most similar in-batch sentence. When d1 and d2 are close or d1 < d2, Eq. (2) mixes the positive embedding with the hardest negative to produce a new positive target. Separately, Eq. (7) takes the top-k nearest neighbours of the query from a previous minibatch and creates new hard negatives by random or adaptive linear combination αGi+(1−α)Gj followed by L2 normalization. The paper reports Spearman averages of 78.38 (positive mixing) and 78.27 (hard negative mixing) on seven STS benchmarks with BERT-base, against 76.16–76.25 for SimCSE baselines, with larger BERT variants reaching 79.13 and 79.20.","pith_inferences":["A reader-level implication the paper does not state: the same cosine-proximity rule could be used to filter or re-weight negatives in other contrastive text objectives, not just SimCSE, because the mechanisms only require an embedding space and a batch.","The positive-mixing branch may function partly as a regularizer that smooths the embedding geometry, so part of the gain could survive even if nearest-by-cosine negatives are not semantically hard; a test would compare against mixing with random negatives or with true paraphrase negatives.","A natural extension is to define hardness by semantic similarity (e.g., entailment or paraphrase scores) rather than by cosine distance, and then measure whether the gains increase; the paper's threshold-based mixing rule supports such a plug-in change."],"forward_implications":["If HNCSE is right, unsupervised sentence embedding can be improved by a cheap, data-free extension of SimCSE: no new data or supervision, only interpolation rules applied during training.","Both HNCSE-PM and HNCSE-HNM beat the SimCSE baselines they reproduce on the average of seven STS benchmarks, and on the majority of individual STS and transfer tasks.","With BERT-large, the same mechanisms push average STS scores to about 79.2, suggesting the gains do not vanish as model capacity grows.","The reported results also imply that a small contrastively trained encoder can outperform LLaMA2-7B with prompt-engineering strategies on STS, i.e., task-specific contrastive training can beat generic generative prompting for similarity measurement."],"supporting_citations":[{"why":"Defines the unsupervised SimCSE objective and training protocol that HNCSE builds on and compares against.","marker":"Gao et al. (2021)"},{"why":"Supplies the hard-negative-mixing idea that HNCSE adapts into its Hard Negative Mixing component.","marker":"Kalantidis et al. (2020)"},{"why":"Supplies the InfoNCE contrastive loss formulation and the in-batch negative bank convention used in the objective.","marker":"Chen et al. (2020)"},{"why":"Supports the contrastive loss framework and momentum-style negative handling referenced in Eq. (4).","marker":"He et al. (2020)"},{"why":"Supplies the BERT encoder and pretrained checkpoints used to produce the sentence embeddings.","marker":"Devlin et al. (2018)"},{"why":"Supplies the SentEval evaluation toolkit and the transfer-task benchmark suite used for measuring generalization.","marker":"Conneau and Kiela (2018)"}],"fun_headline_variants":["Hard-negative mixups lift sentence embeddings past SimCSE","HNCSE: hybrid hard-negative contrastive learning beats SimCSE","Hard negatives repair weak positives and forge strong negatives","STS scores jump to 78.4 with hard-negative mixing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the nearest in-batch sentence by cosine similarity is genuinely the semantically hardest negative, and that linear interpolation in embedding space produces a semantically meaningful positive or negative; if geometric closeness does not track semantic difficulty, both mixup rules could still help for unrelated reasons.","fun_headline_variants_meta":{"raw":{"variants":["Hard-negative mixups lift sentence embeddings past SimCSE","HNCSE: hybrid hard-negative contrastive learning beats SimCSE","Hard negatives repair weak positives and forge strong negatives","STS scores jump to 78.4 with hard-negative mixing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000652,"raw_usage":{"total_tokens":2971,"prompt_tokens":907,"completion_tokens":2064,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1995}},"tokens_in":523,"tokens_out":2064,"duration_ms":15699,"temperature":1.0,"reasoning_tokens":1995,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:51:10.945512+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain HNCSE with the same hyperparameters but with the 'hardest negative' replaced by a random in-batch sentence; if the average STS Spearman score stays near 78, the claimed effect is not due to hardness. Alternatively, compare against a variant that defines hardness by a paraphrase model instead of cosine distance: if the paraphrase-based hard negatives do not improve on the cosine-based ones, the geometric assumption is the active ingredient; if they do, the paper's mechanism is a proxy rather than the cause.","supporting_citations":[{"cited_title":"Hard negative mixing for contrastive learning","cited_arxiv_id":null,"evidence_quote":"Supplies the hard-negative-mixing idea that HNCSE adapts into its Hard Negative Mixing component."}],"review_version":1}