pith. machine review for the scientific record. sign in

arxiv: 2605.07409 · v1 · submitted 2026-05-08 · 💻 cs.CL · cs.LG· stat.AP

Recognition: no theorem link

The Proxy Presumption: From Semantic Embeddings to Valid Social Measures

Baishi Li , Ta Yu , Kelvin J.L. Koa , Ke-Wei Huang
Authors on Pith no claims yet

Pith reviewed 2026-05-11 02:05 UTC · model grok-4.3

classification 💻 cs.CL cs.LGstat.AP
keywords proxy presumptionsemantic embeddingsconstruct validitycounterfactual neutralizationcomputational social sciencepsychometricsvalidity testingcausal representation learning
0
0 comments X

The pith

Semantic embeddings mix target social constructs with confounders unless explicitly neutralized and validated through a new protocol.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper contends that researchers commonly treat geometric properties of language model embeddings, such as cosine distances, as direct measures of social concepts like novelty or bias, yet this Proxy Presumption fails without validation. Unsupervised embeddings typically entangle the intended construct with unrelated attributes such as topic, style, and authorship. To resolve this, the authors introduce the Construct Validity Protocol, which adapts principles from causal representation learning and psychometrics into a pipeline that moves from clear conceptualization to quantitative checks. The protocol features Counterfactual Neutralization, a technique that employs large language models to diminish confounding influences while retaining the core construct. It also supplies a Validity Suite containing tests for discriminant, incremental, and predictive validity to turn heuristic proxies into defensible instruments.

Core claim

The central claim is that without explicit validation, unsupervised representations remain entangled mixtures of the target construct C and confounding attributes Z, rendering them unsuitable as social measures. The Construct Validity Protocol addresses this by offering a rigorous pipeline from conceptualization to quantitative verification, augmented by Counterfactual Neutralization that uses large language models to reduce confounding in embedding space and a standardized Validity Suite for testing discriminant, incremental, and predictive validity.

What carries the argument

The Construct Validity Protocol (CVP), a pipeline that integrates causal representation learning and psychometrics to move from conceptualizing a social construct to verifying embedding-based measures through neutralization and validity tests.

If this is right

  • Researchers using embeddings for social science can apply the Validity Suite to establish discriminant, incremental, and predictive validity for their measures.
  • Counterfactual Neutralization provides a concrete method to isolate the target construct by reducing the influence of attributes like style and authorship.
  • The protocol supplies a standardized process that replaces ad-hoc proxy use with verifiable steps from conceptualization onward.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the protocol succeeds, similar validation steps could be adapted for embeddings in domains outside computational social science, such as legal or medical text analysis.
  • Widespread adoption might encourage embedding developers to include built-in neutralization options for common confounders.
  • The approach suggests a template for testing other unsupervised representations, including those from vision models, against analogous validity criteria.

Load-bearing premise

Large language models can reliably neutralize confounding attributes in embedding space while preserving the target construct without introducing new distortions or biases.

What would settle it

An experiment demonstrating that embeddings produced after Counterfactual Neutralization remain significantly correlated with known confounders such as topic or authorship would undermine the protocol's core neutralization step.

read the original abstract

Natural Language Processing is rapidly evolving into a primary instrument for Computational Social Science, with researchers increasingly using embeddings to measure latent constructs such as novelty, creativity, and bias. However, this transition faces a fundamental validity challenge: the ''Proxy Presumption,'' or the reliance on geometric properties (e.g., cosine distance) as direct measures of social concepts. We argue that without explicit validation, unsupervised representations remain entangled mixtures of the target construct ($C$) and confounding attributes ($Z$) like topic, style, and authorship. To bridge the gap between semantic embeddings and valid social measures, we introduce the Construct Validity Protocol (CVP). Drawing on causal representation learning and psychometrics, the CVP offers a rigorous pipeline from conceptualization to quantitative verification. We further propose Counterfactual Neutralization, a novel method using LLMs to reduce confounding in embedding space. By providing a standardized Validity Suite -- including tests for discriminant, incremental, and predictive validity -- this work offers the community a toolkit to transform heuristic proxies into robust, scientifically defensible instruments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 3 minor

Summary. The manuscript identifies the 'Proxy Presumption' as the invalid reliance on geometric properties of unsupervised semantic embeddings (e.g., cosine distance) as direct measures of latent social constructs such as novelty, creativity, or bias. It argues that these embeddings entangle the target construct C with confounding attributes Z (topic, style, authorship) and proposes the Construct Validity Protocol (CVP) as a pipeline from conceptualization through causal-representation-learning and psychometric principles to quantitative verification. The CVP incorporates a novel Counterfactual Neutralization step that uses LLMs to reduce confounding in embedding space, together with a Validity Suite of tests for discriminant, incremental, and predictive validity.

Significance. If the CVP and its LLM-based neutralization step can be shown to isolate C from Z without new artifacts or loss of construct information, the work would supply a much-needed standardized validation framework for embedding-based social measures, raising the evidentiary standard in computational social science and enabling more defensible use of NLP representations as scientific instruments.

major comments (3)
  1. [Abstract and §3] Abstract and §3 (Counterfactual Neutralization): the central claim that CVP converts heuristic proxies into valid measures rests on the untested assumption that LLM-driven neutralization removes only Z directions while leaving C intact and without injecting LLM-specific distortions; no formal invertibility argument with respect to C, no controlled experiment on a dataset with ground-truth C and Z labels, and no before/after comparison of validity metrics are supplied.
  2. [§4] §4 (Validity Suite): because the discriminant, incremental, and predictive validity tests are described as downstream of the neutralization procedure, the absence of any empirical demonstration that post-neutralization embeddings outperform raw embeddings on known ground-truth data renders the suite's claimed utility unverified and the overall pipeline's soundness dependent on an unproven modeling step.
  3. [§2] §2 (Proxy Presumption and entanglement argument): the assertion that unsupervised representations 'remain entangled mixtures' is presented as a foundational motivation, yet no quantitative measure of entanglement (e.g., mutual information between embedding directions and known Z attributes) or comparison against supervised disentanglement baselines is provided to establish the severity of the problem the CVP is intended to solve.
minor comments (3)
  1. [Abstract and §1] Notation: the symbols C and Z are introduced in the abstract but should be given explicit mathematical definitions (e.g., as random variables or subspaces) at first use in the main text and used consistently thereafter.
  2. [Abstract and §1] Terminology: 'Proxy Presumption' is placed in quotation marks as if newly coined; the manuscript should clarify whether this is an original term or a reference to prior literature on proxy validity in psychometrics or causal inference.
  3. [Figures] Figure clarity: any diagrams illustrating the CVP pipeline or the neutralization operation should include explicit arrows or labels showing the hypothesized causal paths from C and Z to the observed embedding and the intended removal of Z.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their insightful and constructive comments. We address each major point below, clarifying the manuscript's scope as a methodological proposal while committing to revisions that strengthen the empirical grounding where feasible.

read point-by-point responses

  1. Referee: [Abstract and §3] Abstract and §3 (Counterfactual Neutralization): the central claim that CVP converts heuristic proxies into valid measures rests on the untested assumption that LLM-driven neutralization removes only Z directions while leaving C intact and without injecting LLM-specific distortions; no formal invertibility argument with respect to C, no controlled experiment on a dataset with ground-truth C and Z labels, and no before/after comparison of validity metrics are supplied.

    Authors: The referee correctly identifies that the current manuscript does not include a formal invertibility proof or controlled experiments with ground-truth labels for C and Z. The CVP is presented as a conceptual pipeline drawing on causal representation learning, with Counterfactual Neutralization motivated by established LLM capabilities in attribute manipulation rather than as a fully validated procedure. We agree this leaves the central claim partially untested. In revision, we will add a dedicated limitations subsection discussing potential LLM-induced distortions and the absence of invertibility guarantees, along with a proposed experimental protocol (including synthetic data generation and a small-scale demonstration on a dataset with known labels) and before/after validity comparisons to be implemented in future work. revision: yes

  2. Referee: [§4] §4 (Validity Suite): because the discriminant, incremental, and predictive validity tests are described as downstream of the neutralization procedure, the absence of any empirical demonstration that post-neutralization embeddings outperform raw embeddings on known ground-truth data renders the suite's claimed utility unverified and the overall pipeline's soundness dependent on an unproven modeling step.

    Authors: We acknowledge that the Validity Suite's utility remains unverified without direct empirical comparison on ground-truth data. The manuscript positions the suite as a standardized toolkit to be applied post-neutralization, with the tests derived from psychometric principles. To address the concern, the revised manuscript will include a preliminary empirical illustration using an existing public dataset (e.g., one with annotated constructs and confounders) to show measurable improvements in discriminant and predictive validity metrics after neutralization, thereby providing initial evidence for the pipeline's soundness. revision: yes

  3. Referee: [§2] §2 (Proxy Presumption and entanglement argument): the assertion that unsupervised representations 'remain entangled mixtures' is presented as a foundational motivation, yet no quantitative measure of entanglement (e.g., mutual information between embedding directions and known Z attributes) or comparison against supervised disentanglement baselines is provided to establish the severity of the problem the CVP is intended to solve.

    Authors: The entanglement claim draws on extensive prior literature in representation learning, but the referee is right that the manuscript does not supply a new quantitative demonstration within this work. We will revise §2 to incorporate a brief quantitative analysis: computing mutual information or correlation statistics between embedding dimensions and known confounding attributes (Z) on standard embedding models, and briefly contrasting this with supervised disentanglement approaches to better motivate the need for the CVP. revision: yes

Circularity Check

0 steps flagged

No circularity: methodological proposal without self-referential derivations or fitted predictions.

full rationale

The manuscript introduces the Construct Validity Protocol (CVP) and Counterfactual Neutralization as novel methodological contributions drawing on causal representation learning and psychometrics. No equations, parameter fits, or derivations appear in the abstract or described pipeline. The central claims are presented as a standardized validity suite and LLM-based procedure rather than predictions that reduce to inputs by construction. No self-citation load-bearing steps, uniqueness theorems, or ansatz smuggling are quoted or evident. The derivation chain is self-contained as an independent protocol proposal.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

The claims rest on the assumption that embeddings can be disentangled via LLMs and that the new protocol will produce valid measures, with no free parameters or external evidence shown.

axioms (1)
  • domain assumption Semantic embeddings contain entangled mixtures of target constructs and confounders that can be separated using LLMs and causal methods.
    Core premise enabling Counterfactual Neutralization and the CVP pipeline.
invented entities (2)
  • Construct Validity Protocol (CVP) no independent evidence
    purpose: Standardized pipeline from conceptualization to quantitative verification of embedding validity.
    Newly introduced framework without demonstrated external validation in the abstract.
  • Counterfactual Neutralization no independent evidence
    purpose: LLM-based method to reduce confounding attributes in embedding space.
    Novel technique proposed to address the proxy presumption.

pith-pipeline@v0.9.0 · 5487 in / 1322 out tokens · 64953 ms · 2026-05-11T02:05:46.246406+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

137 extracted references · 137 canonical work pages

  1. [1]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    SocialGaze: Improving the integration of human social norms in large language models , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    work page 2024

  2. [2]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Culturally aware natural language inference , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    work page 2023

  3. [3]

    international conference on machine learning , pages=

    Challenging common assumptions in the unsupervised learning of disentangled representations , author=. international conference on machine learning , pages=. 2019 , organization=

    work page 2019

  4. [4]

    Proceedings of the IEEE , volume=

    Toward causal representation learning , author=. Proceedings of the IEEE , volume=. 2021 , publisher=

    work page 2021

  5. [5]

    IEEE transactions on pattern analysis and machine intelligence , volume=

    Representation learning: A review and new perspectives , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2013 , publisher=

    work page 2013

  6. [6]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=

    Adversarial removal of demographic attributes from text data , author=. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2018

  7. [7]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    Null it out: Guarding protected attributes by iterative nullspace projection , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    work page

  8. [8]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages=

    Problems with cosine as a measure of embedding similarity for high frequency words , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages=

    work page

  9. [9]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Large language models for automated open-domain scientific hypotheses discovery , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    work page 2024

  10. [10]

    , author=

    Construct validity in psychological tests. , author=. Psychological bulletin , volume=. 1955 , publisher=

    work page 1955

  11. [11]

    , author=

    Convergent and discriminant validation by the multitrait-multimethod matrix. , author=. Psychological bulletin , volume=. 1959 , publisher=

    work page 1959

  12. [12]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    Language (technology) is power: A critical survey of “bias” in NLP , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    work page

  13. [13]

    American political science review , volume=

    Measurement validity: A shared standard for qualitative and quantitative research , author=. American political science review , volume=. 2001 , publisher=

    work page 2001

  14. [14]

    , title =

    DeVellis, Robert F. , title =

    work page

  15. [15]

    , author=

    A quantitative approach to content validity. , author=. Personnel psychology , volume=

    work page

  16. [16]

    , author=

    Content validity in psychological assessment: A functional approach to concepts and methods. , author=. Psychological assessment , volume=. 1995 , publisher=

    work page 1995

  17. [17]

    , author=

    Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. , author=. American psychologist , volume=. 1995 , publisher=

    work page 1995

  18. [18]

    Political analysis , volume=

    Text as data: The promise and pitfalls of automatic content analysis methods for political texts , author=. Political analysis , volume=. 2013 , publisher=

    work page 2013

  19. [19]

    Journal of Economic Literature , volume=

    Text as data , author=. Journal of Economic Literature , volume=. 2019 , publisher=

    work page 2019

  20. [20]

    , author=

    Intraclass correlations: uses in assessing rater reliability. , author=. Psychological bulletin , volume=. 1979 , publisher=

    work page 1979

  21. [21]

    Journal of chiropractic medicine , volume=

    A guideline of selecting and reporting intraclass correlation coefficients for reliability research , author=. Journal of chiropractic medicine , volume=. 2016 , publisher=

    work page 2016

  22. [22]

    Cohen, Jacob , title =

    work page

  23. [23]

    2009 , publisher=

    The elements of statistical learning , author=. 2009 , publisher=

    work page 2009

  24. [24]

    The American Journal of Psychology , year =

    Spearman, Charles , title =. The American Journal of Psychology , year =

    work page

  25. [25]

    Epidemiology , volume=

    Negative controls: a tool for detecting confounding and bias in observational studies , author=. Epidemiology , volume=. 2010 , publisher=

    work page 2010

  26. [26]

    Standards for Educational and Psychological Testing , publisher =

    work page

  27. [27]

    1994 , publisher=

    Psychometric Theory , author=. 1994 , publisher=

    work page 1994

  28. [28]

    , author=

    The incremental validity of psychological testing and assessment: conceptual, methodological, and statistical issues. , author=. Psychological assessment , volume=. 2003 , publisher=

    work page 2003

  29. [29]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Evaluating n-gram novelty of language models using rusty-dawg , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2024

  30. [30]

    ICScore : Metrics for Evaluating Interestingness and Creativity of Stories

    Lee, Junha and Cho, Jaeshin and Cho, Youngjin and Jin, Hyewon and Lee, Hyemin and Song, Min. ICScore : Metrics for Evaluating Interestingness and Creativity of Stories. 2024

    work page 2024

  31. [31]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Measuring political bias in large language models: What is said and how it is said , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  32. [32]

    Science , volume=

    Semantics derived automatically from language corpora contain human-like biases , author=. Science , volume=. 2017 , publisher=

    work page 2017

  33. [33]

    On measuring social biases in sentence encoders , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages=

    work page 2019

  34. [34]

    Advances in Neural Information Processing Systems , volume=

    Man is to computer programmer as woman is to homemaker? debiasing word embeddings , author=. Advances in Neural Information Processing Systems , volume=

    work page

  35. [35]

    Proceedings of the National Academy of Sciences , volume=

    Word embeddings quantify 100 years of gender and ethnic stereotypes , author=. Proceedings of the National Academy of Sciences , volume=. 2018 , publisher=

    work page 2018

  36. [36]

    American Sociological Review , volume=

    The geometry of culture: Analyzing the meanings of class through word embeddings , author=. American Sociological Review , volume=. 2019 , publisher=

    work page 2019

  37. [37]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    GoEmotions: A dataset of fine-grained emotions , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    work page

  38. [38]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

    We can detect your bias: Predicting the political ideology of news articles , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2020

  39. [39]

    Changes in European solidarity before and during COVID-19: Evidence from a large crowd-and expert-annotated Twitter dataset , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

    work page

  40. [40]

    Measuring conversational uptake: A case study on student-teacher interactions , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages=

    work page

  41. [41]

    Learning from the worst: Dynamically generated datasets to improve online hate detection , author=. Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (volume 1: long papers) , pages=

    work page

  42. [42]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    Latent hatred: A benchmark for understanding implicit hate speech , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2021

  43. [43]

    Findings of the Association for Computational Linguistics: NAACL 2022 , pages=

    POLITICS: Pretraining with same-story article comparison for ideology prediction and stance detection , author=. Findings of the Association for Computational Linguistics: NAACL 2022 , pages=

    work page 2022

  44. [44]

    Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Political ideology and polarization: A multi-dimensional approach , author=. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    work page 2022

  45. [45]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Misinfo reaction frames: Reasoning about readers’ reactions to news headlines , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  46. [46]

    Social Psychological and Personality Science , volume=

    Moral foundations twitter corpus: A collection of 35k tweets annotated for moral sentiment , author=. Social Psychological and Personality Science , volume=. 2020 , publisher=

    work page 2020

  47. [47]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  48. [48]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Sociocultural norm similarities and differences via situational alignment and explainable textual entailment , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2023

  49. [49]

    Measuring Psychological Depth in Language Models

    Harel-Canada, Fabrice Y and Zhou, Hanyu and Muppalla, Sreya and Yildiz, Zeynep Senahan and Kim, Miryung and Sahai, Amit and Peng, Nanyun. Measuring Psychological Depth in Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024

    work page 2024

  50. [50]

    SAFARI : Cross-lingual Bias and Factuality Detection in News Media and News Articles

    Azizov, Dilshod and Mujahid, Zain Muhammad and AlQuabeh, Hilal and Nakov, Preslav and Liang, Shangsong. SAFARI : Cross-lingual Bias and Factuality Detection in News Media and News Articles. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024

    work page 2024

  51. [51]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Only a little to the left: A theory-grounded measure of political bias in large language models , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  52. [52]

    POW : Political Overton Windows of Large Language Models

    Azzopardi, Leif and Moshfeghi, Yashar. POW : Political Overton Windows of Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025

    work page 2025

  53. [53]

    Transactions of the Association for Computational Linguistics , volume=

    Data statements for natural language processing: Toward mitigating system bias and enabling better science , author=. Transactions of the Association for Computational Linguistics , volume=. 2018 , publisher=

    work page 2018

  54. [54]

    Proceedings of the 2022 conference on Empirical Methods in Natural Language Processing , pages=

    The “problem” of human label variation: On ground truth in data, modeling and evaluation , author=. Proceedings of the 2022 conference on Empirical Methods in Natural Language Processing , pages=

    work page 2022

  55. [55]

    Proceedings of the 2020 conference on fairness, accountability, and transparency , pages=

    Garbage in, garbage out? Do machine learning application papers in social computing report where human-labeled training data comes from? , author=. Proceedings of the 2020 conference on fairness, accountability, and transparency , pages=

    work page 2020

  56. [56]

    Transactions of the Association for Computational Linguistics , volume=

    Inherent disagreements in human textual inferences , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher=

    work page 2019

  57. [57]

    Transactions of the Association for Computational Linguistics , volume=

    Dealing with disagreements: Looking beyond the majority vote in subjective annotations , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher=

    work page 2022

  58. [58]

    AI magazine , volume=

    Truth is a lie: Crowd truth and the seven myths of human annotation , author=. AI magazine , volume=

    work page

  59. [59]

    Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages=

    Annotation artifacts in natural language inference data , author=. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages=

    work page 2018

  60. [60]

    Proceedings of the seventh joint conference on lexical and computational semantics , pages=

    Hypothesis only baselines in natural language inference , author=. Proceedings of the seventh joint conference on lexical and computational semantics , pages=

    work page

  61. [61]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    work page

  62. [62]

    Proceedings of the 57th annual meeting of the Association for Computational Linguistics , pages=

    The risk of racial bias in hate speech detection , author=. Proceedings of the 57th annual meeting of the Association for Computational Linguistics , pages=

    work page

  63. [63]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    Beyond accuracy: Behavioral testing of NLP models with CheckList , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    work page

  64. [64]

    Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    Evaluating models’ local decision boundaries via contrast sets , author=. Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    work page 2020

  65. [65]

    Science Advances , volume=

    How to make causal inferences using texts , author=. Science Advances , volume=. 2022 , publisher=

    work page 2022

  66. [66]

    Advances in Neural Information Processing Systems , volume=

    Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models , author=. Advances in Neural Information Processing Systems , volume=

    work page

  67. [67]

    Annual Review of Political Science , volume=

    Testing causal theories with learned proxies , author=. Annual Review of Political Science , volume=. 2022 , publisher=

    work page 2022

  68. [68]

    1927 , publisher=

    Interpretation of educational measurements , author=. 1927 , publisher=

    work page 1927

  69. [69]

    Sentence-bert: Sentence embeddings using siamese bert-networks , author=. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages=

    work page 2019

  70. [70]

    Robert Adcock and David Collier. 2001. Measurement validity: A shared standard for qualitative and quantitative research. American political science review, 95(3):529--546

    work page 2001

  71. [71]

    American Educational Research Association , American Psychological Association , and National Council on Measurement in Education . 2014. Standards for Educational and Psychological Testing. American Educational Research Association

    work page 2014

  72. [72]

    Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI magazine, 36(1):15--24

    work page 2015

  73. [73]

    Dilshod Azizov, Zain Muhammad Mujahid, Hilal AlQuabeh, Preslav Nakov, and Shangsong Liang. 2024. SAFARI : Cross-lingual bias and factuality detection in news media and news articles. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12217--12231. Association for Computational Linguistics

    work page 2024

  74. [74]

    Leif Azzopardi and Yashar Moshfeghi. 2025. POW : Political overton windows of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 24767--24773. Association for Computational Linguistics

    work page 2025

  75. [75]

    Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4982--4991

    work page 2020

  76. [76]

    Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. Measuring political bias in large language models: What is said and how it is said. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11142--11159

    work page 2024

  77. [77]

    Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587--604

    work page 2018

  78. [78]

    Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798--1828

    work page 2013

  79. [79]

    Su Lin Blodgett, Solon Barocas, Hal Daum \'e Iii, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in nlp. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5454--5476

    work page 2020

  80. [80]

    Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in Neural Information Processing Systems, 29

    work page 2016

Showing first 80 references.