{"id":"270411ca-b386-47d9-8321-981b07e6b922","arxiv_id":"2506.06169","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A library and demo reveal that masked language models encode the dative construction's person-like versus place-like reading of ambiguous recipients.","lead":"This paper introduces a software library and demo that project language model word representations into human-readable semantic feature spaces. A case study shows that three language models treat 'London' as more person-like in 'I sent London the letter' than in 'I sent the letter to London.'","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Projector generalization is unvalidated: the central DO–PO semantic effect in Fig. 2 could be an artifact of the trained MLP mapping rather than LM sensitivity, since no held-out evaluation on individual contexts or human baseline is provided.","rationale":"The reader's weakest-assumption analysis correctly identifies the missing validation of the MLP projectors as the most load-bearing issue. The paper's central claim depends entirely on the projection being semantic rather than superficial, and the manuscript provides no held-out evaluation, no human baseline, and no statistical test linking the projected feature differences to actual human interpretation. This is not merely a disagreement with consensus; it is an internal evidential gap: the result in Figure 2 is uninterpretable without knowing whether the probe is reliable on the specific ambiguous-name contexts it is asked to classify. The reader's CONDITIONAL verdict is appropriate, and my stress-test does not change it. I recommend no verdict adjustment: the paper should be accepted only if the missing validation is supplied or if the presentation is reframed as a tool demonstration rather than a confirmed semantic finding. The concrete test described above would settle the concern directly and would also address the non-independence of the stimulus sample by reporting cluster-level uncertainty.","tokens_in":100,"tokens_out":2553,"duration_ms":41641,"concrete_test":"Run a held-out validation study on the 450 experimental sentence pairs: collect human judgments (e.g., forced choice: is the recipient more likely a person or a place in the DO version, and in the PO version) for a representative subset, then compute the correlation between the human DO–PO personhood/placehood difference and the model-predicted Binder feature difference for each pair. If the predicted differences do not correlate with human judgments, or do not exceed a permutation baseline where DO/PO labels are randomly shuffled within pairs, the Figure 2 result cannot be attributed to LM semantic sensitivity. Report cluster-bootstrap confidence intervals over the 15 recipient names to account for non-independence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that the observed DO-vs-PO differences in Binder features reflect genuine contextual semantic sensitivity of masked LMs (Fig. 2 and Section 3). That claim depends on the trained MLP projectors faithfully mapping individual contextual embeddings into Binder feature space. The projectors in Section 2 are trained on context-averaged word embeddings from the BNC against static Binder norms, and no validation is reported on held-out individual contexts of the experimental stimuli. In particular, the 450 test sentences use 15 proper nouns that are ambiguous between person and place readings, exactly the cases where the context-averaged training signal is least informative: for a name like 'London', the static target norm is predominantly place-like, so the projector may be biased toward place features whenever the input is lexical rather than truly contextual. Because the comparison is between two sentence frames, any systematic sensitivity of the projector to word order, surrounding function words, or sentence position could produce the reported positive differences without the LM encoding the intended semantic distinction. The paper also reports 'almost all' model-layer combinations as 33 of 36, but the 450 sentence pairs are generated from only 15 recipient names, 6 verbs, and 5 agents, so the non-independence of the stimulus sample means this count overstates the evidence. No statistical test, confidence interval, or human judgment baseline is provided to connect the projected differences to actual person-place interpretation. Without such validation, Figure 2 is compatible with an alternative explanation: the probe transforms contextual embeddings in a way that tracks surface form rather than semantic construal. This is the load-bearing gap identified by the reader, and it remains unresolved in the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces semantic-features, a library for projecting contextual word embeddings from masked language models into interpretable semantic feature spaces such as the Binder norms, together with an interactive Gradio demo. The central case study tests whether the dative alternation affects the semantic construal of the recipient argument: for sentence pairs like 'I sent London the letter' (DO) versus 'I sent the letter to London' (PO), the authors hypothesize that the recipient is more animate in the DO and more place-like in the PO. Using projectors trained on BNC context-averaged embeddings, they report that BERT, RoBERTa, and ALBERT show the expected direction of change in 33 of 36 model-layer combinations (Figure 2).","tokens_in":6129,"tokens_out":2298,"duration_ms":26784,"significance":"If the empirical result holds, the paper provides both a useful open-source tool for interpretable analysis of contextual embeddings and evidence that masked language models encode construction-specific semantic constraints, aligning with theoretical accounts of the dative alternation. The paper's explicit strengths are the public code repository, the reproducible training pipeline for the projection models, and the accessible interactive demo. However, the validity of the central empirical claim depends critically on whether the trained projectors generalize to individual unseen contexts, which is not demonstrated; the current evidence is also weakened by the small effective stimulus sample and the absence of statistical uncertainty quantification.","major_comments":[{"comment":"The central empirical claim depends on the trained MLP projectors faithfully mapping individual contextual embeddings from the experimental sentences into Binder feature space, but the projectors are trained only on context-averaged embeddings per word from BNC (Section 2), and no validation is reported on held-out individual contexts. For a word like 'London', the training target is a single static Binder norm vector, so the projector may learn a lexical mapping that is biased toward the dominant reading rather than sensitive to construction-level context. A concrete test would be to evaluate projected feature values for the 450 experimental sentences against human feature-annotation judgments, or at least to report projector accuracy on held-out individual contexts from a corpus, and to show that the DO-PO difference is not reproduced by a projector trained on shuffled or context-free inputs. Without such validation, the observed differences in Figure 2 could be artifacts of the probe rather than genuine LM semantic sensitivity.","section":"Section 3, stimulus construction and Figure 2"},{"comment":"The 450 sentence pairs are generated from only 15 recipient names, 6 verbs, and 5 agents, so the effective independent sample is much smaller than 450. The claim that the pattern holds in '33 of 36' model-layer combinations therefore overstates the strength of the evidence: the 15 recipient names are the critical random factor, and each name contributes 30 sentence pairs. The paper should report by-item variance, include a permutation or mixed-effects analysis that treats recipient name as a random effect, and provide confidence intervals or significance tests for the average changes shown in Figure 2. Without this, the observed consistency across layers could reflect idiosyncratic properties of a few names rather than a general phenomenon.","section":"Section 3, Results"},{"comment":"The training procedure reports an 80-20 train-validation split and selection by validation loss, but no held-out test evaluation is described for the projection models. The fact that all 117 models used 2 layers with 50% dropout and early stopping (Appendix B) does not by itself establish that the learned projections are accurate. Since the case study interprets absolute differences in predicted feature values, the paper should report at least one quantitative accuracy measure (e.g., correlation or mean absolute error) on a held-out set of individual contexts, and ideally on the same classes of ambiguous person/place nouns used in the experiment.","section":"Section 2, Model training; Appendix B"}],"minor_comments":[{"comment":"There are several typos: 'incontext' should be 'in context', 'a interpetable' should be 'an interpretable', and 'To what extend' in Section 3 should be 'To what extent'.","section":"Section 1 and Section 3"},{"comment":"The hand-selected feature subsets for person-hood and place-hood are a central design choice, but the paper does not discuss how sensitive the results are to adding or removing individual features (e.g., excluding 'Biomotion' or including 'Vision'). A brief sensitivity analysis would strengthen confidence that the animacy and place-hood contrasts are not driven by a single feature.","section":"Section 3, Table 1 and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially a tool paper with a case-study demonstration. The tool contribution is clear and the code is likely to be useful to the community, but the empirical case study as currently presented would need substantially stronger validation to support the central claim in a journal venue. The most important missing piece is a held-out evaluation of the projector on individual contexts, along with uncertainty quantification that respects the small number of recipient names."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey [Name],\n\nQuick take: this is a useful tool paper with a case study that is suggestive but not yet convincing. The new contribution is the semantic-features library itself—a clean, extensible pipeline for projecting contextual embeddings into interpretable norm spaces—plus an interactive demo and a new dataset of 450 dative sentence pairs. If you work with interpretable embeddings or dative alternation, the library is worth a look.\n\nWhat it does well: the code and demo are real, the hyperparameter details are in the appendix, and the feature selection for person/place is principled (using Binder feature definitions). The authors are transparent that the projection method comes from Chronis et al. (2023); they are not claiming to invent the core method. The case study directionality is consistent: 33 of 36 model-layer combinations show more animacy in DO than PO, and more place-hood in PO than DO.\n\nThe soft spots are in the empirical support. The trained MLP projectors are fit on context-averaged word embeddings against static Binder norms, with no held-out evaluation on the individual contexts used in the test stimuli. Since the test sentences deliberately use ambiguous person/place names, this is exactly where the projector's generalization is most uncertain. The stress-test note has this right: without a validation set of individual-context projections or a human baseline, Figure 2 could reflect probe artifacts (e.g., sensitivity to surrounding function words) rather than the LM's semantic construal. The 450 pairs are generated from only 15 names, 6 verbs, and 5 agents, so the effective sample is small; the 33/36 count overstates the evidence. No confidence intervals or significance tests are reported.\n\nThese issues are not disqualifying for a tool paper. The case study is a demonstration, not a standalone empirical contribution. But the conclusion's wording—'contextually sensitive distributional semantic embeddings capture subtle changes'—goes beyond what the current evidence supports.\n\nWho it's for: NLP researchers using the demo or building on the library; linguists interested in dative alternation might cite the dataset. It deserves a serious referee; with projector validation, released stimuli, and variance estimates, it would be solid. I'd send it to review but ask for those additions before acceptance.","headline":"A genuinely useful tool library with a plausible but unvalidated case study; the projector's generalization on individual contexts is the key missing piece.","tokens_in":6658,"tokens_out":2073,"would_cite":false,"duration_ms":17076,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Masked language models assign a dative recipient higher animacy in the double-object construction and higher place-hood in the prepositional construction.","keywords":["contextual word embeddings","interpretable semantic spaces","dative alternation","Binder norms","language models","semantic features","masked language models","semantic probing"],"falsifier":"Collect human judgments of animacy and place-hood for the 450 experimental sentences, then compare them to the projected feature scores; if the projections do not align with human intuitions, or if the same names in neutral contexts also show the same projection pattern, the claimed construction-specific sensitivity would be called into question.","tokens_in":1475,"feed_emoji":"🧠","tokens_out":11709,"duration_ms":165328,"temperature":0.7,"pith_summary":"This paper introduces an open-source library that projects contextual word embeddings—the internal representations language models compute for a word in a sentence—into a human-interpretable semantic feature space. It applies the tool to a classic linguistic question: whether the choice between two dative constructions changes how a recipient word is interpreted. The paper builds a dataset of 450 sentence pairs that differ only in construction, using names that can be either a person or a place, and shows that three masked language models consistently produce higher animacy-related feature values for the recipient in the double-object form (\"I sent London the letter\") and higher place-related values in the prepositional form (\"I sent the letter to London\"). If correct, this means models encode the subtle semantic constraint that double-object recipients must be animate, and the library makes such questions easy to test.","feed_headline":"Language models encode dative recipient animacy by construction","feed_subtitle":"A new open-source tool projects word embeddings into interpretable features, showing models treat dative recipients as more animate in…","key_machinery":"The central mechanism is a trained multi-layer perceptron that maps a contextual word embedding to a vector of Binder feature values. The network is trained on context-averaged embeddings from the British National Corpus against human-annotated feature norms; once trained, it can be applied to the embedding of any target word in a new sentence. The Binder space is the key because its features have concrete definitions that cleanly separate person-hood (Biomotion, Body, Human, Face, Speech) from place-hood (Landmark, Scene), allowing the authors to quantify how the model's representation shifts across constructions.","core_discovery":"The central empirical discovery is that the contextual representation of an ambiguous proper noun changes with the dative construction: projection into the Binder feature space—a set of human-annotated semantic properties such as Human, Landmark, and Scene—shows a rise in person/animacy features (Human, Body, Face, Biomotion, Speech) for the recipient in the double-object construction, and a rise in place features (Landmark, Scene) in the prepositional construction. This pattern holds for BERT, RoBERTa, and ALBERT across most layers, with the strongest effects in intermediate layers (6–9). The paper argues this indicates that contextually sensitive distributional embeddings capture subtle construction-level semantic changes.","pith_inferences":["The observed double-object versus prepositional-object difference might partly be an artifact of the projection probe rather than a genuine LM semantic representation; a probe trained on averaged contexts could systematically bias ambiguous names toward one reading. Testing the same sentences with a probe trained on randomized context embeddings would help isolate the LM's contribution.","The method's success on dative recipients suggests a general strategy for diagnosing 'constructional meaning' in LMs: choose a construction whose alternation has a clear semantic contrast, build a balanced minimal pair dataset, and read off the relevant feature values. This could be extended to other argument-structure alternations, such as the spray/load alternation or the causative/inchoative al","If the effect is robust, it implies that LMs do not treat words as having a single fixed sense but dynamically adjust their semantic features to fit the argument structure—a property that could be exploited for controlled paraphrase generation or for detecting when models rely on surface form rather than meaning."],"forward_implications":["The library can be used to test whether other grammatical constructions that impose semantic constraints on arguments, such as locative alternations or passive voice, are reflected in contextual embeddings.","The finding that middle layers (6–9) carry the strongest semantic sensitivity suggests that layer-wise probing with interpretable spaces can localize where constructional semantics are encoded.","The open-source tool and interactive demo lower the barrier for linguists and cognitive scientists to run hypothesis-driven semantic analyses on transformer LMs without writing custom code.","Because the method works with any masked LM, it provides a standardized route to compare the semantic competences of future models across languages and architectures."],"supporting_citations":[{"why":"Supplies the original projection method and the choice of semantic feature spaces.","marker":"Chronis et al. (2023)"},{"why":"Provides the feature norms and definitions that define the interpretable space.","marker":"Binder et al. (2016)"},{"why":"Establishes the linguistic constraint that double-object recipients are animate, grounding the hypothesis.","marker":"Beavers (2011)"},{"why":"Gives the construction grammar account of dative semantics (caused possession vs. caused motion).","marker":"Goldberg (1995)"},{"why":"Provides a balanced dative dataset that motivates the need for a new, animacy-focused dataset.","marker":"Hawkins et al. (2020)"},{"why":"BERT is one of the three models tested.","marker":"Devlin et al. (2019)"},{"why":"RoBERTa is one of the three models tested.","marker":"Liu et al. (2019)"},{"why":"ALBERT is one of the three models tested.","marker":"Lan et al. (2020)"},{"why":"minicons is used for extracting contextual embeddings.","marker":"Misra (2022)"}],"fun_headline_variants":["Dative construction flips animacy in language model embeddings","Tool shows LMs read double-object recipients as more human","LMs see 'London' as a person in double-object dative","Interpretable embeddings reveal dative effect on animacy","From place to person: dative construction biases LM meaning"],"cache_read_input_tokens":8832,"weakest_assumption_plain":"The trained projectors, fit on context-averaged embeddings from a general corpus, are assumed to generalize faithfully to the individual, unseen contexts of the experimental sentences, yet the paper does not evaluate projection accuracy on those specific stimuli.","fun_headline_variants_meta":{"raw":{"variants":["Dative construction flips animacy in language model embeddings","Tool shows LMs read double-object recipients as more human","LMs see 'London' as a person in double-object dative","Interpretable embeddings reveal dative effect on animacy","From place to person: dative construction biases LM meaning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1515,"prompt_tokens":870,"completion_tokens":645,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":562}},"tokens_in":486,"tokens_out":645,"duration_ms":6652,"temperature":1.0,"reasoning_tokens":562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:59:17.510320+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect human judgments of animacy and place-hood for the 450 experimental sentences, then compare them to the projected feature scores; if the projections do not align with human intuitions, or if the same names in neutral contexts also show the same projection pattern, the claimed construction-specific sensitivity would be called into question.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the linguistic constraint that double-object recipients are animate, grounding the hypothesis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the construction grammar account of dative semantics (caused possession vs. caused motion)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ALBERT is one of the three models tested."}],"review_version":1}