{"id":"604809a6-f83b-4430-a692-d1776d3b1a3f","arxiv_id":"2506.01602","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An MMD-based variable selection method ranks words whose embedding dimensions shifted most between time periods, with qualitative evidence from Japanese news and American English.","lead":"This paper applies a statistics tool called Maximum Mean Discrepancy to word embeddings to find which word meanings shifted over time, and tests it on 18 years of Japanese news and 200 years of American English. It identifies likely sense-shifting words such as disaster victim in 2011 and health condition in 2020 by comparing word vectors across years.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The word-level score in Eq. (4) has no null baseline; it will rank any word whose vector moves along the MMD-selected global shift directions, so the reported top words do not establish word-specific sense change.","rationale":"The reader's weakest assumption was embedding-space alignment, but that is substantially addressed by the use of PMI-SVD joint embeddings, which are explicitly designed to place all time periods in a shared vector space. The more load-bearing weakness is that the word scoring function in Eq. (4) is not calibrated against any null. Selection of S_{t,t'} by MMD maximizes the aggregate distribution difference between periods, so the selected variables are precisely those along which the whole population of word vectors moved. A word will receive a high score if its vector moved along these global directions, even if its sense is unchanged; the paper provides no comparison to the average shift of all words or to a permutation baseline. The near-saturated scores of 2.0 for all top words in 2011 in Table 1 are a red flag: the same maximal cosine distance across every pairwise period is implausible for three distinct words and instead suggests that the score is dominated by a single global component. If the proposed standardization test changes the rankings, then the paper's empirical demonstration, which is already qualitative and self-referential, does not support the claim that the method identifies words undergoing sense shifts. The paper explicitly lists benchmark evaluation as future work, and the code is available, so a conditional verdict remains appropriate; my concern adds a specific falsifiable check that should be required before the central claim is accepted.","tokens_in":8227,"tokens_out":10564,"duration_ms":120224,"concrete_test":"Recompute the Table 1 rankings after standardizing Eq. (4): for each period t and each word v, subtract the mean cosine distance over all words and divide by the standard deviation, or, more robustly, build a null distribution by permuting word identities between t and t'. If 不況, 被災, and 具合 no longer appear in the top ranks, the unnormalized score is confounded by global shift and the central word-level claim is unsupported. Use the released code with the same PMI-SVD joint embeddings from [9] so only the scoring step changes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 defines score_v^t as the average cosine distance between e^S_{v,t} and e^S_{v,t'} over t', where S_{t,t'} are the variables selected to maximize the aggregate MMD between the full distributions of all words in periods t and t'. Because S is chosen to capture the largest global distribution shift, every word that participates in that shift, regardless of whether its own sense changed, will have a large cosine distance along S. The score is not centered or normalized by the distribution of distances over all words, and no word-level permutation null is provided. The saturated 2.0 scores for all top words in 2011 (Table 1) are consistent with a single global direction dominating the selected variables rather than with three independent sense changes. Consequently, the qualitative matches to earthquakes/COVID in Table 1 and Figure 3 do not demonstrate that the method identifies words undergoing sense shifts; they may just identify words whose embeddings moved most under the same topical shift. This is load-bearing because the paper's central claim is word-level identification, and this concern is independent of the alignment question, which the PMI-SVD joint model is designed to address.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MMD-Sense-Analysis, a method that applies Maximum Mean Discrepancy (MMD) variable selection to diachronic word embeddings in order to (i) select embedding dimensions ('sense change variables') that differ between time periods and (ii) score individual words by the average cosine distance over those dimensions across periods. The author demonstrates the method on a Japanese Mainichi Shimbun corpus (2003–2020) and on CCOHA (1810–2010), interpreting high-scoring words and period-pair heatmaps in relation to events such as the 2011 Great East Japan Earthquake, the Lehman shock, and the COVID-19 pandemic. The paper claims to be the first application of MMD to word sense change detection and states that the empirical results demonstrate the method's effectiveness.","tokens_in":8397,"tokens_out":6675,"duration_ms":69120,"significance":"The idea of using MMD-based variable selection for sense change detection is novel and potentially useful: it addresses the recognized problem that cosine distance in full embedding space mixes sense-relevant and sense-irrelevant dimensions. The paper is transparent about its code and about some limitations (e.g., the small-vocabulary CCOHA result). However, the central claim of effectiveness is not quantitatively supported. There is no benchmark evaluation, no comparison to existing sense change detection methods, and no word-level null model. The word score in Eq. (4) is constructed from variables that were selected to maximize the global distribution shift, so it may rank words merely for participating in that global shift rather than for word-specific sense change. As presented, the paper is an exploratory case study rather than a validated method; the stated future-work plan to evaluate on a benchmark confirms this gap.","major_comments":[{"comment":"The paper explicitly states 'As future work, I plan to evaluate MMD-Sense-Analysis on a benchmark dataset for the sense change detection.' This admission, combined with the absence of any comparison to existing sense change methods, gold-standard labels, or quantitative metrics (precision/recall, AP, or similar) in Sections 3.1 and 3.2, means the abstract's claim that 'Empirical assessment results demonstrate the effectiveness of the proposed approach' is not supported. The current experiments are qualitative narratives around selected words and p-value heatmaps, which do not establish effectiveness in the sense used by the field.","section":"Section 4"},{"comment":"The word-level score score_v^t averages cosine distances between e^{S_{t,t'}}_{v,t} and e^{S_{t,t'}}_{v,t'}, where S_{t,t'} are the variables selected to maximize the aggregate MMD between the full word distributions of periods t and t'. Because the variable sets are chosen to capture the largest global distribution shift, any word whose embedding moves along that global direction will receive a high score, regardless of whether its own sense changed independently. The score has no null baseline, is not centered or normalized over words, and is not compared to a distribution expected under a word-level null hypothesis. The saturated value 2.0 for all top words in 2011 (Table 1) is consistent with a single dominant topical shift rather than with three distinct word-specific sense changes. A word-level permutation or a comparison to scores from randomly selected variable subsets is needed to support the claim that the method identifies words undergoing sense shifts.","section":"Section 2.2, Eq. (4)"},{"comment":"The description of the permutation test is incomplete. The manuscript does not specify how the permutation is performed (e.g., permuting period labels of word vectors, or permuting variable labels), which statistic is recomputed after variable selection, or whether any multiple-testing correction is applied across the 153 pairwise comparisons. Without these details, the p-value heatmaps in Figures 1 and 2 are not reproducible, and the threshold p<0.05 may yield false positives simply because of the large number of tests. This directly affects the claims that 2020 and 2009–2011 are significantly distinct from other periods.","section":"Section 3.1 (permutation test)"},{"comment":"The second experiment (CCOHA) is presented as a demonstration, yet the authors state that the selected words for 1980s and 1990s are 'insufficient to explain the social background' and attribute this to the small vocabulary (505 words). This shows that the method's ability to 'explain the evolution over multiple historical periods', a central claim of the abstract, is not reliably achieved when the vocabulary is limited. Since the paper later generalizes (Section 4) that the method 'selects words concerning the semantic aware variables' that 'can clearly describe the social backgrounds', the CCOHA result undermines this generalization unless the scope conditions are explicitly stated and addressed.","section":"Section 3.2"}],"minor_comments":[{"comment":"The phrase 'qualifying similarities' in the Introduction and conclusion should be 'quantifying similarities'; the abstract uses 'quantify' but the body uses 'qualify' in several places.","section":"Abstract / Section 4"},{"comment":"The notation in Eq. (4) is confusing: T′ is defined as 'the number of time periods without t, t′ ∈ {1, ..., T′}', but the summation index and the range over which t′ runs are not clearly specified. Please rewrite the definition to state explicitly that the sum runs over all t′ != t, and define the cosine distance (e.g., 1 − cosine similarity).","section":"Section 2.2, Eq. (4)"},{"comment":"In the paragraph before Table 3, the text says 'Table 1 represents the top three words...' but the displayed table is numbered 'Table 3'. Please correct the cross-reference.","section":"Section 3.2"},{"comment":"The description of the permutation test is vague: 'The p-value represents the degree to which the observation data pair Et, Et′ is significantly against the null hypothesis' is not a standard statistical statement. Provide a concrete permutation procedure and the exact test statistic.","section":"Section 2.2"},{"comment":"The paper reports no error bars or repeated runs over the random selection of 2,300 vocabularies and the 2,000/300 split. Given that the top-word lists (Table 1) are a main result, their stability under different random seeds should be reported or at least discussed.","section":"Section 3.1"},{"comment":"The ARD kernel in Eq. (2) includes constants γ_d said to 'unit-normalize each variable', but the paper does not state how these are computed beyond citing [16, Appendix C]. A brief summary would make the method self-contained.","section":"Section 2.1"},{"comment":"The paper uses PMI-SVD joint embeddings to ensure comparability across time periods, but does not explicitly explain why these embeddings are in a shared vector space. Please add a sentence citing the joint factorization property from [2] or [9] to justify the use of cosine distance in Eq. (4).","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper depends heavily on the author's own prior work [16] for the MMD variable selection, and the PMI-SVD joint embeddings come from [2,9], which are from the same research group (with the author also acknowledging Hajime Kiyama for providing the models). This is not a disqualifying issue, but it means there is little independent validation of the building blocks. The manuscript reads more like a workshop-level technical report than a journal paper: the idea is interesting but the evaluation is anecdotal. If the scope of the journal permits exploratory methodological papers, a thorough revision with benchmark evaluation and null baselines could make it publishable; otherwise, it may be better suited to a workshop or as a resource paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. The genuinely new piece is applying MMD variable selection to diachronic embeddings, plus the Eq. (4) word-level scoring function; the rest of the machinery comes from the author's own earlier MMD variable selection paper. The empirical section never measures effectiveness: no baselines, no benchmark scores, no error bars, and benchmark evaluation is explicitly left for future work. The abstract says results \"demonstrate effectiveness,\" which the body does not support.\n\nWhat the paper does well: it is clearly written, the math is standard and mostly correct, and the choice of PMI-SVD joint embeddings partly addresses the alignment concern that always hangs over diachronic work. The qualitative findings have face validity: 2020 differs from every other year in Japanese news, 2011 words like disaster victim and evacuation saturate the score, and the \"family\" result on CCOHA replicates earlier work. The author also honestly notes the CCOHA vocabulary is too small to interpret well. Code is available.\n\nThe soft spots, in order of size. The load-bearing problem is the self-referential loop in word scoring. Variables S_{t,t'} are selected to maximize the aggregate MMD between the full word distributions of two periods, and words are then scored by their movement along exactly those variables. A word does not need its own sense to have changed; it just needs to participate in the global topical shift. The saturated 2.0 scores for all top words in 2011 are a red flag: they look like one dominant direction rather than three independent sense shifts. The paper provides no word-level null distribution or normalization to show the top words stand out beyond the global shift. This directly weakens the central claim of identifying words undergoing sense shifts. The second issue is the absence of any comparison to existing sense change detection methods, so there is no evidence MMD selection is competitive. The alignment assumption is real but less troubling because the joint PMI-SVD model is designed to put periods in a common space; still, a quick validation of that would cost little.\n\nWho is this for? Researchers working on interpretable diachronic analysis for historical and social questions, not someone looking for state-of-the-art detection. I would not cite it yet, but I would read a revised version with benchmark results. My recommendation: do not send it to peer review in its current form. Invite a revision that adds a proper quantitative evaluation against existing methods and a word-level significance test; then it deserves a serious referee.","headline":"A clear new application of MMD variable selection to sense change detection, but the evaluation is qualitative and the word scores are partly circular; needs benchmark validation before it can be cited as effective.","tokens_in":8967,"tokens_out":4403,"would_cite":false,"duration_ms":50460,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","62G10"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes MMD-Sense-Analysis, a method that uses Maximum Mean Discrepancy variable selection to identify which words change meaning across time and to explain the change through the embedding dimensions that drive it.","keywords":["word sense change detection","Maximum Mean Discrepancy","MMD variable selection","diachronic embeddings","PMI-SVD joint","semantic shift detection","kernel two-sample test","interpretable NLP"],"falsifier":"Run MMD-Sense-Analysis on a synthetic corpus with a known injected shift: give one word a new sense in one period by changing only a single embedding dimension, then check whether that dimension is recovered by MMD-CV-AGG and whether the word tops the global-time score; failing that recovery would directly contradict the method's central mechanism.","tokens_in":7965,"feed_emoji":"🕰️","tokens_out":4905,"duration_ms":45837,"temperature":0.7,"pith_summary":"This paper proposes a method, MMD-Sense-Analysis, for detecting when a word's meaning changes across time and for explaining the change in terms of specific embedding dimensions. The paper's central claim is that Maximum Mean Discrepancy (MMD), a kernel-based distance between probability distributions, can be repurposed as a variable-selection tool that picks out the dimensions of word embeddings that carry sense change, and that word scores built from those dimensions separate genuinely shifted words from stable ones. The author argues this is the first application of MMD to word sense change detection. Demonstrated on Japanese news text (2003–2020) and American English (1810–2010), the method surfaces periods and words that align with known historical events, such as the COVID-19 pandemic in 2020 and the Great East Japan Earthquake in 2011. If correct, the method gives a parameter-light, interpretable route from raw diachronic embeddings to both word rankings and period-level comparisons.","feed_headline":"MMD picks the words whose meaning shifted across eras","feed_subtitle":"It isolates sense-aware dimensions in word embeddings, flagging eras like 2020 and 2011 in news text.","key_machinery":"The load-bearing object is the ARD-kernel MMD estimator with sparse weights. Maximum Mean Discrepancy gives a kernel distance between two time-period embedding distributions; the ARD kernel assigns per-dimension weights whose L1 penalty drives most weights to zero, so the surviving dimensions index the sense change variables. MMD-CV-AGG avoids choosing the sparsity constant by aggregating selections over many lambda values. Words are then scored by projecting each word vector onto the selected dimensions and averaging cosine similarity across time-period pairs, so a high score means the word's embedding moves within the sense-relevant subspace.","core_discovery":"On its own terms, the paper establishes that sense change detection can be cast as a two-sample distribution shift problem in which the relevant signal is carried by a sparse subset of embedding dimensions. The algorithm first runs MMD-CV-AGG, an L1-regularized ARD-kernel variable selection procedure, on every pair of time periods to identify sense change variables. It then runs a permutation test on the selected variables to decide whether two periods differ significantly, and scores each word by the average cosine similarity between its sub-vectors restricted to the selected variables across periods. The paper reports that on two corpora, the resulting heatmaps and word rankings track documented social events, and that interpreting the output requires enough vocabulary coverage.","pith_inferences":["A natural next test would be a benchmark with ground-truth sense changes; the author lists this as future work, and a positive result there would separate the MMD mechanism from the evocative historical coincidences in the demonstration.","The dimension-level output suggests a testable extension: comparing the selected dimensions across different word pairs could reveal whether a single dominant sense shift drives a whole period, or whether changes are word-specific.","The method assumes aligned embedding spaces; an informative stress test would be to train embeddings separately per period and apply the algorithm with and without post-hoc alignment, to see how much of the signal survives alignment artifacts.","Because MMD is kernel-based, the procedure could be applied to contextualized embeddings from language models by projecting token representations of a target word into a shared space, potentially giving a fine-grained sense-change detector for individual words."],"forward_implications":["If MMD-Sense-Analysis is correct, word sense change detection can be done without a learned classifier: a distribution-distance test plus sparse dimension selection is enough to rank shifted words.","The same pipeline transfers across languages and corpus types, since it only needs aligned embedding matrices per period; the paper demonstrates it on Japanese news and historical American English.","Period-level p-values from the permutation test give a direct, interpretable map of which years are linguistically exceptional, useful for linking corpus change to external historical events.","Because it returns dimensions rather than just scores, the method opens the way to explaining a word's change by which co-occurrence dimensions moved, not merely by ranking the word.","The method's reliance on abundant vocabulary is a design constraint: with too few words, the selected variables do not support social interpretation."],"supporting_citations":[{"why":"Supplies the MMD-CV-AGG variable selection algorithm that identifies sense change variables and is the methodological core of the paper.","marker":"[16]"},{"why":"Defines the kernel-based Maximum Mean Discrepancy estimator that the variable selection optimizes and the permutation test relies on.","marker":"[6]"},{"why":"Provides the PMI-SVD joint word embedding model used to produce the diachronic vectors in both experiments.","marker":"[2]"},{"why":"Gives the two goals of sense change detection and the suggestion for building PMI-SVD joint embeddings on which the experimental setup is based.","marker":"[9]"},{"why":"Introduces the distinction between sense-aware and uniform embedding dimensions that MMD-Sense-Analysis is designed to exploit.","marker":"[1]"},{"why":"Supplies the CCOHA historical American English corpus used in the second empirical demonstration.","marker":"[3]"},{"why":"Underlies the ARD-kernel weight optimization by test power that the variable selection approach adapts with L1 regularization.","marker":"[23]"}],"fun_headline_variants":["MMD spots sense shifts via sparse embedding dimensions","First MMD-based method for word sense change detection","Sense change found by testing embedding distribution shifts","MMD zeroes in on dimensions that reveal meaning drift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the PMI-SVD joint word vectors from different time periods are directly comparable, so a cosine between their sense-selected sub-vectors measures semantic change rather than alignment noise; if the embedding spaces drift or are misaligned, every score and heatmap inherits that artifact.","fun_headline_variants_meta":{"raw":{"variants":["MMD spots sense shifts via sparse embedding dimensions","First MMD-based method for word sense change detection","Sense change found by testing embedding distribution shifts","MMD zeroes in on dimensions that reveal meaning drift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1353,"prompt_tokens":775,"completion_tokens":578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":391,"completion_tokens_details":{"reasoning_tokens":517}},"tokens_in":391,"tokens_out":578,"duration_ms":6372,"temperature":1.0,"reasoning_tokens":517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:37:09.233464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MMD-Sense-Analysis on a synthetic corpus with a known injected shift: give one word a new sense in one period by changing only a single embedding dimension, then check whether that dimension is recovered by MMD-CV-AGG and whether the word tops the global-time score; failing that recovery would directly contradict the method's central mechanism.","supporting_citations":[{"cited_title":"Variable selec- tion in maximum mean discrepancy for interpretable distribution comparison","cited_arxiv_id":null,"evidence_quote":"Supplies the MMD-CV-AGG variable selection algorithm that identifies sense change variables and is the methodological core of the paper."},{"cited_title":"A comprehensive analysis of PMI-based models for measuring semantic differences","cited_arxiv_id":null,"evidence_quote":"Provides the PMI-SVD joint word embedding model used to produce the diachronic vectors in both experiments."},{"cited_title":"Analyzing continuous semantic shifts with diachronic word similarity matrices","cited_arxiv_id":null,"evidence_quote":"Gives the two goals of sense change detection and the suggestion for building PMI-SVD joint embeddings on which the experimental setup is based."},{"cited_title":"A semantic distance metric learning approach for lexical semantic change detection","cited_arxiv_id":null,"evidence_quote":"Introduces the distinction between sense-aware and uniform embedding dimensions that MMD-Sense-Analysis is designed to exploit."},{"cited_title":"CCOHA: Clean corpus of historical American English","cited_arxiv_id":null,"evidence_quote":"Supplies the CCOHA historical American English corpus used in the second empirical demonstration."},{"cited_title":"Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton","cited_arxiv_id":null,"evidence_quote":"Underlies the ARD-kernel weight optimization by test power that the variable selection approach adapts with L1 regularization."}],"review_version":1}