{"id":"9b54bf6f-30b2-4e8e-b86c-3791d2a0f2b4","arxiv_id":"1908.06178","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Distributional negative sampling, which selects corrupted entities by cosine similarity to the original entity, improves several knowledge-base completion metrics for TransE and RESCAL on three benchmarks, though not universally.","lead":"Knowledge-base completion models learn by seeing true facts and made-up false ones; this paper proposes generating the false ones from entities that are semantically similar to the true entity, using the model's own embeddings. The idea is plausible and improves several benchmark metrics, but the paper's headline claim of universal improvement is contradicted by its own table on one of the six settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own Table 2 contradicts the central claim: TransE DNS underperforms TransE RNS on FB15k MRR (43.0 vs 46.3) and Hits@10 (63.9 vs 74.9).","rationale":"I read the paper as claiming that DNS improves KBC training by replacing random negatives with distributionally similar ones. The mechanism is plausible and Algorithm 1 is explicit, but the central claim is empirical and depends entirely on Table 2. That table is the least secure part of the argument: the TransE/FB15k row is a direct counterexample to the 'outperforms on all' claim, and the RNS baselines are not generated in-house, so differences may reflect implementation rather than the sampling method. The reader's weakest_assumption focused on the cosine-similarity proxy; my concern is prior to that. Even if the proxy is perfect, the published numbers do not support the headline. A controlled rerun is the minimal check that would settle the issue. I do not see grounds for accepting the paper without it; but retaining the REJECT verdict is appropriate, not because DNS is impossible, but because the submitted evidence is internally inconsistent and uncontrolled.","tokens_in":7411,"tokens_out":3401,"duration_ms":34618,"concrete_test":"Reproduce Table 2's TransE rows on FB15k using one codebase for both samplers: same implementation, embedding dimension 100, margin 10.0, Adam defaults, same early-stopping rule, and at least five random seeds. Report mean and standard deviation for filtered MRR, Hits@10, and Hits@1. If TransE(DNS) still falls below TransE(RNS) on MRR or Hits@10, the central claim is contradicted even under controlled conditions; if it matches or exceeds RNS, the original table's RNS baseline was not comparable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires DNS to improve KBC accuracy over RNS, but the paper's own evaluation fails to establish this. Section 4 states 'DNS outperforms RNS on all the considered KBC algorithms and across all benchmarks and evaluation metrics,' then Table 2 shows the opposite for TransE on FB15k: DNS MRR 43.0 vs RNS 46.3 and Hits@10 63.9 vs 74.9. Footnote 5 acknowledges this exception but does not investigate it. The comparison is also not controlled: RNS numbers for TransE and RESCAL are borrowed from Nickel et al. (2016b), and the WN18RR TransE RNS row is missing MRR and Hits@1. With no error bars, no same-code baseline, and an acknowledged counterexample to the universal claim, the abstract's 'significant improvement' is unsupported. The method may still work in some settings, but the strongest reading of the paper—that DNS reliably improves training across benchmarks and algorithms—rests on the validity of this Table 2 comparison, which is internally contradicted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Distributional Negative Sampling (DNS), a training-time negative sampler for knowledge base completion. Instead of corrupting a positive triple with a uniformly random entity, DNS computes cosine similarities between the entity being replaced and all other entities using the model's current embeddings, and accepts each candidate entity as a negative with probability equal to its (clamped) cosine similarity. The authors evaluate DNS against random negative sampling (RNS) for TransE and RESCAL on FB15k, FB15k-237, and WN18RR, reporting filtered MRR, Hits@10, and Hits@1. They claim in the abstract and Section 4 that DNS gives significant improvements over RNS, and in Section 5 they argue qualitatively that DNS converges faster and produces more meaningful negatives.","tokens_in":7789,"tokens_out":3557,"duration_ms":36891,"significance":"If the central claim were established, DNS would be a simple and appealing drop-in replacement for random negative sampling in embedding-based KBC, with no additional data or supervision required. The idea is intuitive and the qualitative example in Table 3 is instructive. The paper also has the merit of explicitly acknowledging one counterexample to its own headline claim. However, the evaluation is not controlled: RNS baselines are borrowed from a different paper, one baseline row is incomplete, no variance or significance information is reported, and the method's own table contradicts the universal improvement claim. The distributional hypothesis underlying the sampler is plausible, but the evidence offered is anecdotal and limited to a single query entity. Because the central empirical claim fails on the paper's own numbers, the contribution as stated is not established.","major_comments":[{"comment":"The statement in Section 4 that 'DNS outperforms RNS on all the considered KBC algorithms and across all benchmarks and evaluation metrics' is directly contradicted by Table 2. On FB15k, TransE(DNS) achieves MRR 43.0 and Hits@10 63.9, while TransE(RNS) achieves MRR 46.3 and Hits@10 74.9. Footnote 5 acknowledges this exception but does not investigate it, and the abstract's claim of a 'significant improvement' in MRR is therefore unsupported for this setting. The paper should either restrict its claim to the settings where DNS wins, or explain why the FB15k TransE result does not undermine the general claim.","section":"Section 4, Table 2"},{"comment":"The RNS baselines for TransE and RESCAL are borrowed from Nickel et al. (2016b) rather than produced with the same code, hyperparameters, and experimental pipeline as the DNS runs. In addition, the WN18RR TransE(RNS) row reports only Hits@10 and lacks MRR and Hits@1. With no error bars, no multiple-seed runs, and no same-code controlled baseline, it is impossible to determine whether the differences in Table 2 are due to the sampling strategy or to implementation details, hyperparameter choices, or randomness. A controlled comparison under identical training conditions is essential to support the paper's central claim.","section":"Section 4, Table 2"},{"comment":"DNS uses the model's own embeddings, at the current training state, to select negatives for that same model. The paper asserts that this creates a 'virtuous circle' and that the similarity signal becomes meaningful over time, but the only supporting evidence is Table 3, which shows nearest neighbors of a single entity (DVD) on one dataset (FB15k-237). No quantitative measure is given of how well embedding similarity tracks type similarity, or of how the sampler behaves in the first epochs when embeddings are not yet informative. The possibility that DNS could amplify early embedding errors is not addressed. The faster-convergence claim in Section 5 is based on one figure for RESCAL on WN18RR over the first 50 epochs; more systematic evidence is needed.","section":"Section 3, Algorithm 1 and Section 5"},{"comment":"The number of negative samples per positive triple is an uncontrolled variable between DNS and RNS. RNS is described in Section 3 as using a fixed number C of negatives, whereas Algorithm 1 accepts each candidate entity with probability max(0, cosine similarity). The expected number of DNS negatives therefore varies with the similarity distribution and changes during training. If the DNS runs effectively use a different negatives-to-positives ratio than the RNS runs, the comparison in Table 2 is confounded. The paper should either match the number of negatives across conditions, or report the actual number of negatives generated per triple and analyze sensitivity to C for both samplers.","section":"Section 3, Equation (1) and Algorithm 1"}],"minor_comments":[{"comment":"The GAN-based negative sampling reference in Section 2 appears as '?' and the Fast-TransX implementation in Table 2 has no citation; these references need to be completed.","section":"Related Work and Table 2"},{"comment":"The Introduction states that DNS gives 'consistent and significant improvement in five out of six settings', while Section 4 claims it outperforms RNS in all settings. The paper should state the claim consistently and precisely, acknowledging the FB15k TransE exception.","section":"Abstract and Introduction"},{"comment":"The caption and text of Table 3 are ambiguous about whether the numbers shown are cosine similarities or softmax probabilities. The text later computes the 'odds' of VHS via a softmax, but the table values appear to be raw similarities; please clarify.","section":"Section 5, Table 3"},{"comment":"The sentence 'as per LCWA this assertion is counted as less positive' should read 'counted as negative' or 'counted as unseen', since the Local Closed World Assumption treats unobserved triples as negative.","section":"Section 5"},{"comment":"The comment 'Use bernoulli sampling to conclude that t should be corrupted' is implementation-specific and appears inside the pseudocode; the Bernoulli choice of whether to corrupt the head or tail should be described in the text rather than embedded as a code comment.","section":"Algorithm 1, line 4"}],"recommendation":"reject","confidential_remarks":"The paper is an early preprint with missing citations and incomplete experimental reporting. The decisive issue is that the main empirical claim is contradicted by the paper's own Table 2, and the baseline comparison is not controlled. Substantial new experiments with matched implementations and a revised, more modest claim would be needed before the contribution could be evaluated fairly; I do not think this can be achieved within a routine revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll cut to the chase: the paper has a genuinely neat idea—DNS, sampling negatives by cosine similarity using the very model being trained—but it hurts its own case. Its own Table 2 shows TransE with DNS underperforming TransE with random sampling on FB15k (MRR 43.0 vs 46.3, Hits@10 63.9 vs 74.9), while Section 4 claims DNS wins everywhere. That's not a minor blemish; it directly contradicts the abstract's 'significant improvement.'\n\nDNS is new, as far as I can tell: stochastic, self-referential, no external type data, no pre-trained selector. That alone is worth something. The intuition—hard negatives that are plausible but false—is sensible, and the convergence plots on WN18RR (Figure 2) suggest it can help RESCAL. The idea deserves follow-up.\n\nThe problems are all in the evidence. RNS baselines for TransE and RESCAL are borrowed from Nickel et al. (2016b) rather than re-run with the same code, hyperparameters, and early stopping; the WN18RR TransE RNS row is missing MRR and Hits@1; there are no error bars; and unresolved citation placeholders (for the GAN and Fast-TransX) make reproducibility harder. The reader's circularity worry doesn't land—the test triples are held out, so the success metric isn't circular—but the universal claim is overreach even with footnote 5. A controlled comparison would settle whether DNS is actually better, not just different.\n\nThis is for someone working on negative sampling or contrastive learning who wants a simple trick to try. It's not a paper to trust for benchmarks. If I were the editor, I wouldn't desk reject it—the idea is plausible and fixable—but I'd send it to a referee with instructions to demand a same-code RNS baseline, complete results, and a rewritten abstract. As is, it's a workshop-level draft.","headline":"The DNS negative-sampling trick is clever and plausibly useful, but the paper's own Table 2 contradicts its universal improvement claim and the baselines are not controlled.","tokens_in":8148,"tokens_out":2757,"would_cite":false,"duration_ms":26285,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training knowledge-base models on plausible false statements instead of random negatives improves link prediction accuracy.","keywords":["knowledge base completion","negative sampling","link prediction","distributional similarity","knowledge graph embeddings","TransE","RESCAL","mean reciprocal rank"],"falsifier":"Measure, at an early and a late epoch, the fraction of DNS-generated negatives that actually occur as true triples in the validation or test set, and compare that fraction with the same fraction for random negatives. DNS's argument predicts this fraction is low; if DNS negatives are true triples as often as or more often than random ones, then the sampler is not producing 'highly likely false' statements and the reported gains must come from some other effect.","tokens_in":7213,"feed_emoji":"🔗","tokens_out":4068,"duration_ms":40060,"temperature":0.7,"pith_summary":"The paper argues that random negative sampling in knowledge base completion generates mostly nonsensical assertions that give the model little to learn from, and proposes Distributional Negative Sampling (DNS) to replace it. DNS corrupts a triple by replacing an entity with another entity whose learned embedding is similar, so the resulting negative is a plausible but false statement. The authors claim that using DNS in place of random sampling improves Mean Reciprocal Rank and Hits@1 for two KBC models, TransE and RESCAL, across three benchmarks, and that training converges in fewer epochs. A sympathetic reader would care because this is a drop-in change to the training loop that needs no extra supervision and makes better use of the model's own learned representations.","feed_headline":"Plausible false triples beat random negatives for knowledge graphs","feed_subtitle":"Training on meaningful false statements instead of nonsense raises Hits@1 and speeds convergence in five of six tests.","key_machinery":"The load-bearing mechanism is Algorithm 1. For each training triple, DNS picks whether to corrupt the head or tail entity by a Bernoulli draw, then computes the cosine similarity between the chosen entity and every other entity in the KB. For each candidate entity, it accepts the candidate as a negative with probability max(0, cosine similarity), skipping any candidate that would form a true triple already in the training set. Because the acceptance probability is proportional to similarity, the negative set concentrates on plausible alternatives; because the step is stochastic, there remains a small chance of sampling unrelated entities, which preserves exploration. The same network's embeddings are reused to compute similarity, so the sampler sharpens as training proceeds, and the number of negatives per positive emerges automatically from the accept-reject step instead of being a tuned hyperparameter.","core_discovery":"The paper's central claim is that meaningful negatives, meaning plausible false assertions produced by substituting an entity with a distributionally similar one, provide a stronger training signal than random negatives, and that the similarity signal can be read off from the entity embeddings the KBC model is already learning. DNS therefore creates a positive feedback loop: better embeddings make similar entities more likely to be sampled as negatives, and training on those harder negatives makes the embeddings better. The paper reports that DNS beats random sampling on nearly every metric in five of the six algorithm-dataset settings, with the largest gains in Hits@1, and that DNS reaches a given filtered MRR in fewer epochs.","pith_inferences":["A natural stress test is to split negatives by relation type: DNS's similarity ignores the relation of the triple being corrupted, so for relations where similar entities are often valid alternatives, DNS may push down true facts; conditioning similarity on the relation is a testable extension.","The reported convergence speedup may be larger on datasets with many entities per type, where random sampling is most likely to produce nonsense; on small or sparse KBs the benefit could shrink.","The accept-reject idea could be applied to any task that builds negatives by corrupting discrete units, but the cosine-similarity proxy would need to be replaced by a unit-level similarity appropriate to that task.","DNS's gains are measured only on two model families; the same sampler might behave differently on models with very different embedding geometries, such as those that use complex-valued or graph-neural representations."],"forward_implications":["If DNS is correct, any embedding-based KBC model can be improved by swapping the negative sampler, with no architectural change and no new labels.","DNS should reduce the number of training epochs needed to reach a given accuracy, since hard negatives continue to contribute nonzero gradient when random negatives would already satisfy the margin.","DNS removes the need to cross-validate the number of negative samples per positive, because the accept-reject mechanism sets it automatically.","Because the paper reports the largest Hits@1 gains, DNS most helps applications that need the top-ranked prediction to be right, not just a high rank among ten.","The paper explicitly suggests that the same sampling idea could transfer to other tasks that corrupt units into negatives, such as relation extraction, where random corruption currently dominates."],"supporting_citations":[{"why":"Supplies the TransE algorithm and the FB15K benchmark used in the DNS experiments.","marker":"Bordes et al. [2013]"},{"why":"Supplies the RESCAL tensor-factorization algorithm used as the second KBC model.","marker":"Nickel et al. [2011]"},{"why":"Provides the harder FB15k-237 benchmark that removes near-duplicate and inverse relations.","marker":"Toutanova and Chen [2015]"},{"why":"Provides the WN18RR benchmark used to evaluate DNS on WordNet data.","marker":"Dettmers et al. [2017]"},{"why":"Offers a type-based and nearest-neighbor negative sampling approach that DNS contrasts with, since DNS avoids external type information.","marker":"Kotnis and Nastase [2017]"},{"why":"Supplies the RNS baseline results for TransE and RESCAL that DNS is compared against in the evaluation table.","marker":"Nickel et al. [2016b]"},{"why":"Introduces the Bernoulli sampling procedure that DNS uses to decide whether to corrupt the head or the tail entity.","marker":"Wang et al. [2014]"}],"fun_headline_variants":["Hard negatives from similar entities boost knowledge graph accuracy","Distributional sampling: plausible false triples beat random noise","Knowledge base training: similar entities make better negatives","DNS: use embedding similarity to generate meaningful false triples","Plausible fakes outperform random negatives for KB completion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that cosine similarity between the KBC model's own entity embeddings is a reliable measure of whether two entities would make a plausible false statement, even in the early epochs when those embeddings are barely trained.","fun_headline_variants_meta":{"raw":{"variants":["Hard negatives from similar entities boost knowledge graph accuracy","Distributional sampling: plausible false triples beat random noise","Knowledge base training: similar entities make better negatives","DNS: use embedding similarity to generate meaningful false triples","Plausible fakes outperform random negatives for KB completion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1471,"prompt_tokens":790,"completion_tokens":681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":604}},"tokens_in":406,"tokens_out":681,"duration_ms":7474,"temperature":1.0,"reasoning_tokens":604,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:53:14.965817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, at an early and a late epoch, the fraction of DNS-generated negatives that actually occur as true triples in the validation or test set, and compare that fraction with the same fraction for random negatives. DNS's argument predicts this fraction is low; if DNS negatives are true triples as often as or more often than random ones, then the sampler is not producing 'highly likely false' statements and the reported gains must come from some other effect.","supporting_citations":[{"cited_title":"A three-way model for collective learning on multi-relational data","cited_arxiv_id":null,"evidence_quote":"Supplies the RESCAL tensor-factorization algorithm used as the second KBC model."},{"cited_title":"Observed versus latent features for knowledge base and text inference","cited_arxiv_id":null,"evidence_quote":"Provides the harder FB15k-237 benchmark that removes near-duplicate and inverse relations."},{"cited_title":"Knowledge graph embedding by translating on hyperplanes","cited_arxiv_id":null,"evidence_quote":"Introduces the Bernoulli sampling procedure that DNS uses to decide whether to corrupt the head or the tail entity."}],"review_version":1}