Pith. sign in

REVIEW 3 major objections 5 minor 51 references

Position: Contextual Integrity is Inadequately Applied to Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM privacy evaluations are systematically under-applying Contextual Integrity, this paper argues.

desk verdict A useful systematization of CI misuse in LLM papers, but the central charge overreaches by treating legal and crowdsourced proxies as categorically invalid. read the letter →

arxiv 2501.19173 v2 pith:VEGVJE45 submitted 2025-01-31 cs.CY

Yan Shvartzshnaider , Vasisht Duddu This is my paper · ORCID
classification cs.CY
keywords ContextualIntegrityLLMprivacynormsCIheuristicpromptsensitivitypositionbiasevaluationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Contextual Integrity (CI) defines privacy as the appropriate flow of information, governed by privacy norms in a given social context. This position paper claims that nine recent studies using CI to evaluate large language models do not adhere to the theory's four core tenets: they substitute other privacy notions such as data minimization or secrecy, use legal statutes and crowdsourced preferences as proxies for privacy norms, sometimes omit the five CI parameters, and none apply the CI heuristic for normative legitimacy. The authors contend that partial or superficial use of CI can lead to incorrect privacy conclusions and flawed privacy-preserving system designs, and that current CI-based LLM experiments also overlook non-adversarial robustness issues such as prompt sensitivity and positional bias. A sympathetic reader takes away that CI-based privacy claims about LLMs are only as trustworthy as the theory's tenets they actually carry.

What carries the argument

The evaluative machinery is the four-tenet checklist derived from CI theory, applied as a rubric to prior work. Each tenet does specific work: T1 fixes the object of analysis as flow appropriateness rather than data protection; T2 anchors appropriateness in contextual privacy norms rather than preferences or statutes; T3 makes the flow description complete and unambiguous; T4 supplies the normative step, the CI heuristic, that judges whether a norm-breaching flow is legitimate given affected interests, societal values, and contextual purposes. The paper's argument runs by marking each surveyed work against this rubric and showing that nearly all fail at least one tenet.

What would settle it

A controlled comparison could settle it: for a fixed set of LLM-agent information flows, compare privacy judgments obtained from the descriptive CI steps alone with judgments from a full CI heuristic run by an expert deliberative panel; if flows the descriptive steps flag as inappropriate are consistently judged legitimate by the heuristic, the paper's claim that skipping T4 yields flawed conclusions is confirmed, while full agreement would weaken it.

Watch

Extended reading notes

Core claim

On the authors' account, privacy under CI is not determined by data type, minimization, or legal compliance: T1 says privacy is appropriate information flow, T2 says appropriate flows conform with privacy norms, T3 says flows must be specified by all five parameters (sender, subject, recipient, information type, transmission principle), and T4 says the ethical legitimacy of a norm breach is assessed by the CI heuristic. Surveying nine works, the paper finds that eight deviate from T1, all nine deviate from T2, two deviate from T3, and all nine deviate from T4. Consequently, the paper concludes that these works borrow CI terminology without applying CI as a theory, making their privacy evaluations at best descriptive and at worst misleading for governance decisions.

Load-bearing premise

The critique assumes that a genuine CI application must satisfy all four tenets with privacy norms kept separate from legal statutes and crowdsourced preferences; if partial use is a legitimate way to borrow the framework, the inadequate-application charge loses much of its force.

Editorial extensions

If this is right

  • Legal-compliance and preference-based LLM privacy evaluations should be described as measuring compliance or preference alignment, not privacy under CI, unless the normative step is carried out.
  • Future CI-based LLM benchmarks should require all five parameters and an explicit T4 evaluation; otherwise privacy-violation labels remain ambiguous.
  • LLM privacy measurements should report stability under paraphrased prompts and reordered answer options, since positional and prompt-induced biases can flip appropriateness judgments.
  • LLM agents that arbitrate information flows need a process for assessing the legitimacy of novel flows, not just a rule set, because a flow that breaches an existing norm may still be appropriate by contextual values.
  • Researchers who deliberately use only part of CI should state that their results are CI-inspired, avoiding the stronger claim that they evaluate privacy as defined by CI.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the same under-application critique likely applies to CI-based privacy evaluations of systems beyond LLMs, such as IoT and data-sharing platforms, wherever legal rules or preference surveys stand in for norms.
  • If the position holds, legal statutes are not a safe shortcut for norms: privacy regulation and community norms can diverge, so a compliance-based privacy test could both over- and under-count violations.
  • A testable extension would be to annotate the same LLM information-flow scenarios with a full CI heuristic using expert deliberation and with the proxies criticized in the paper; the disagreement rate would quantify how much the under-application distorts conclusions.
  • The experimental-hygiene critique suggests that prompt and position sensitivity should be treated as first-class measurement axes in any LLM-based privacy study, not as noise to be averaged away.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This position paper argues that recent LLM privacy evaluations that invoke Contextual Integrity (CI) do not actually apply the theory, because they fail to uphold its four tenets (T1-T4). The authors clarify the tenets, survey nine prior works, tabulate deviations in Table 1, and add a call for experimental hygiene around prompt sensitivity and position bias. Their central claim is that inadequate application risks incorrect conclusions and flawed privacy-preserving designs.

Significance. The paper makes a timely and useful corrective: it gives the ML community a compact statement of CI's core tenets, and Table 1 is a clear systematization that will help readers see where partial uses diverge. Section 4's emphasis on prompt sensitivity and position bias in CI-based vignette studies is a concrete, actionable contribution. The force of the central claim, however, is conditional on a strict standard for what counts as legitimate norm identification and normative assessment; that standard is currently stated too categorically in Section 3.

major comments (3)
  1. [Section 3, 'Legal Statutes are not Privacy Norms' and Table 2] The categorical claim that legal statutes are not privacy norms is overbroad and is internally inconsistent with the paper's own position in Table 2, which states that 'Governing institutions can be designed to support and reflect privacy norms.' This concession implies that legal instruments can encode settled contextual expectations, as HIPAA does for health contexts. The text, however, uses the categorical claim to mark all surveyed works as ✗ on T2 merely because they draw on legal statutes or crowdsourced preferences. This conflates 'source or evidence of a norm' with 'ground truth.' If these sources can legitimately serve as evidence of norms, the all-✗ T2 row overreaches, and the inference from 'uses proxies' to 'inadequately applies CI' is unsupported. The authors should restate T2 as requiring that norm sources be validated and subjected to normative assessment, rather than prohibiting legal or survey-based inputs outright.
  2. [Section 3, T4: CI Heuristic] The blanket ✗ for T4 is based on the observation that the surveyed works 'defer to laws, policies, and collective preferences as proxies' for ethical legitimacy. This is a non sequitur: using legal or survey input as evidence within a CI heuristic, for example at Level 1 where interests and preferences are examined, is not the same as treating that input as final ethical authority. The paper does not show that any of the nine works considered and rejected the heuristic; it shows that they did not mention it. That absence is a valid observation, but the inference to 'failure' requires a standard that distinguishes omission from affirmative wrong treatment. Without that distinction, the T4 column overstates the case.
  3. [Section 3, T3: Five Parameters] Mireshghallah et al. (2023) are marked ✗ for using only three CI parameters, yet the paper itself acknowledges that Martin & Nissenbaum (2016), the authoritative source for the survey template they follow, deliberately simplified to three parameters while acknowledging that a full operationalization would require five. Given this, the T3 criterion needs to address whether partial parameterization is always inconclusive or whether it can be a legitimate, acknowledged simplification. As written, the paper applies a stricter standard to Mireshghallah et al. than to Martin & Nissenbaum without explaining the difference, which weakens the force of that particular ✗ in Table 1.
minor comments (5)
  1. [Section 4, 'Same Prompt Sensitivity'] The label 'same prompt sensitivity' is confusing; the phenomenon described is variation across repeated queries of the same prompt, so 'same-prompt variation' or 'sampling sensitivity' would be clearer.
  2. [Section 4, 'Paraphrasing Prompt Sensitivity'] The sentence 'None of the prior works ... have accounted for prompt variation' lists only Mireshghallah et al., Ghalebikesabi et al., and Shao et al.; since the systematic claim appears to cover all nine surveyed works, the scope of this claim should be specified more precisely.
  3. [Section 2, Figure 1 and T4 text] Figure 1 labels Level 1 as 'Interests & preferences of affected parties,' while the T4 text describes Level 1 as identifying 'winners and losers'; these descriptions should be aligned for consistency.
  4. [References] In Section 4, 'Cao et al., 2024' is cited twice consecutively in the same parenthetical list; the duplicate citation should be removed.
  5. [Footnote 1] The footnote states that the most accurate account of CI is the Privacy in Context book, but the tenets T1-T4 are drawn from later work (Nissenbaum 2019); the paper should clarify which source is authoritative for the four-tenet formulation.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper is an interpretive position critique, not a derivation; the only self-citation is an empirical input in Section 4 and is not load-bearing.

full rationale

This is a position paper, not a derivation or empirical fitting exercise. Its central claim—that existing LLM literature inadequately applies Contextual Integrity because it does not support the four core tenets—is an interpretive argument grounded in the authors' reading of Nissenbaum's CI theory. The four tenets are presented as attributed to Nissenbaum, and the audit in Table 1 marks deviations from those tenets. One could dispute the categorical claims that 'Legal Statutes are not Privacy Norms' and 'Crowdsourced Preferences are not Privacy Norms,' but that is a substantive disagreement about how to interpret CI, not a circularity in the paper's own reasoning. There are no equations, fitted parameters, or predictions derived from inputs. The only self-referential support is the citation of the authors' prior study (Shvartzshnaider & Duddu, 2025) in Section 4, which reports empirical prompt-sensitivity results for CI-based prompts. That study is used as an input to highlight experimental hygiene issues, not to establish the paper's central critical position or to justify the T1-T4 framework. Thus, the cited prior work is not load-bearing for the main claim. Even if one regards the four-tenet checklist as contestable, that contestability does not make the argument circular: the paper does not define 'adequate CI application' solely in terms of its own prior work, and its conclusion does not reduce by construction to its premises. Overall, the paper is self-contained in its argumentative structure and the minor self-citation does not create circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim is non-quantitative, so there are no fitted parameters. The analysis rests on an interpretive standard: the four tenets as stated, the norm versus legal/preference distinction, and the adequacy of the nine-paper sample. These are assumptions drawn from CI theory and the authors' framing rather than independently measured facts.

assumptions (3)
  • domain assumption The four CI tenets T1-T4 as stated in Section 2 are the correct and complete standard for any CI-based privacy analysis.
    The paper evaluates all nine prior works against this decomposition and treats it as ground truth. The source is Nissenbaum, but the completeness of the four-tenet framing is an interpretive premise.
  • domain assumption Legal statutes and crowdsourced preferences cannot serve as privacy norms in a CI analysis.
    Used to mark T2 failures in Fan, Shao, Li, Cheng, and others. This is a substantive normative claim from CI theory, not an empirical fact, and it is load-bearing for the paper's systematization.
  • domain assumption The nine surveyed works are a sufficient or representative sample of the existing LLM-CI literature.
    The paper generalizes to 'existing LLM literature' from a hand-selected set of nine works, with no systematic search or inclusion protocol described in Section 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position: Contextual Integrity is Inadequately Applied to Language Models." pith.science (2026). https://pith.science/paper/VEGVJE45

@misc{pith2026250119173,
  author       = {Pith},
  title        = {Pith review of: Position: Contextual Integrity is Inadequately Applied to Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VEGVJE45}},
  note         = {Machine review of arXiv:2501.19173}
}
read the original abstract

Machine learning community is discovering Contextual Integrity (CI) as a useful framework to assess the privacy implications of large language models (LLMs). This is an encouraging development. The CI theory emphasizes sharing information in accordance with privacy norms and can bridge the social, legal, political, and technical aspects essential for evaluating privacy in LLMs. However, this is also a good point to reflect on use of CI for LLMs. This position paper argues that existing literature inadequately applies CI for LLMs without embracing the theory's fundamental tenets. Inadequate applications of CI could lead to incorrect conclusions and flawed privacy-preserving designs. We clarify the four fundamental tenets of CI theory, systematize prior work on whether they deviate from these tenets, and highlight overlooked issues in experimental hygiene for LLMs (e.g., prompt sensitivity, positional bias).

Figures

Figures reproduced from arXiv: 2501.19173 by the authors.

Figure 1
Figure 1. summarizes the various steps for CI-based privacy analysis: Step ❶ (identifying the information flows), Step ❷ (identifying the established privacy norms), Step ❸ (con￾ducting a breach analysis by checking for deviation from socially acceptable norms as a baseline), and Step ❹ (revis￾iting the legitimacy of a norm-breaching information flow by examining moral, political, and economic implications). Descriptive CI An… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 22 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Air Gap : Protecting Privacy - Conscious Conversational Agents , May 2024

    Bagdasaryan, E., Yi, R., Ghalebikesabi, S., Kairouz, P., Gruteser, M., Oh, S., Balle, B., and Ramage, D. Air Gap : Protecting Privacy - Conscious Conversational Agents , May 2024. URL http://arxiv.org/abs/2405.05175. arXiv:2405.05175 [cs]

    arXiv 2024

  4. [4]

    C., and Nissenbaum, H

    Barth, A., Datta, A., Mitchell, J. C., and Nissenbaum, H. Privacy and contextual integrity: Framework and applications. In 2006 IEEE Symposium on Security and Privacy (S&P'06), pp.\ 15. IEEE, 2006

    work page 2006

  5. [5]

    Contextual integrity through the lens of computer science

    Benthall, S., G\" u rses, S., and Nissenbaum, H. Contextual integrity through the lens of computer science. Found. Trends Priv. Secur., 2 0 (1): 0 1–69, December 2017. ISSN 2474-1558. doi:10.1561/3300000016. URL https://doi.org/10.1561/3300000016

    work page doi:10.1561/3300000016 2017

  6. [6]

    What does it mean for a language model to preserve privacy? In ACM Conference on Fairness, Accountability, and Transparency, pp.\ 2280--2292, 2022

    Brown, H., Lee, K., Mireshghallah, F., Shokri, R., and Tram\` e r, F. What does it mean for a language model to preserve privacy? In ACM Conference on Fairness, Accountability, and Transparency, pp.\ 2280--2292, 2022. URL https://doi.org/10.1145/3531146.3534642

    arXiv 2022

  7. [7]

    On the worst prompt performance of large language models

    Cao, B., Cai, D., Zhang, Z., Zou, Y., and Lam, W. On the worst prompt performance of large language models. arXiv preprint arXiv:2406.10248, 2024

    arXiv 2024

  8. [8]

    Ci-bench: Benchmarking contextual integrity of ai assistants on synthetic data

    Cheng, Z., Wan, D., Abueg, M., Ghalebikesabi, S., Yi, R., Bagdasarian, E., Balle, B., Mellem, S., and O'Banion, S. Ci-bench: Benchmarking contextual integrity of ai assistants on synthetic data. arXiv preprint arXiv:2409.13903, 2024

    arXiv 2024

Show all 51 references
  1. [9]

    Logical specification of the glba and hipaa privacy laws

    DeYoung, H., Garg, D., Kaynar, D., and Datta, A. Logical specification of the glba and hipaa privacy laws. Technical Report CMU-CyLab-10-007, Carnegie Mellon University, Pittsburgh, PA, USA, 2010

    2010

  2. [10]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Law's empire

    Dworkin, R. Law's empire. Pravovedenie, pp.\ 195, 2013

    2013

  4. [12]

    What did i do wrong? quantifying llms' sensitivity and consistency to prompt engineering

    Errica, F., Siracusano, G., Sanvito, D., and Bifulco, R. What did i do wrong? quantifying llms' sensitivity and consistency to prompt engineering. arXiv preprint arXiv:2406.12334, 2024

    2024 arXiv

  5. [13]

    Goldcoin: Grounding large language models in privacy laws via contextual integrity theory

    Fan, W., Li, H., Deng, Z., Wang, W., and Song, Y. Goldcoin: Grounding large language models in privacy laws via contextual integrity theory. arXiv preprint arXiv:2406.11149, 2024

    2024 arXiv

  6. [14]

    and Mori, T

    Gan, C. and Mori, T. Sensitivity and robustness of large language models to prompt template in J apanese text classification tasks. In Huang, C.-R., Harada, Y., Kim, J.-B., Chen, S., Hsu, Y.-Y., Chersoni, E., A, P., Zeng, W. H., Peng, B., Li, Y., and Li, J. (eds.), Proceedings...

    2023

  7. [15]

    Pal: Program-aided language models

    Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided language models. In International Conference on Machine Learning, pp.\ 10764--10799. PMLR, 2023

    2023

  8. [16]

    Privacy and the limits of law

    Gavison, R. Privacy and the limits of law. The Yale law journal, 89 0 (3): 0 421--471, 1980

    1980

  9. [17]

    Gemini: a family of highly capable multimodal models

    Gemini-Team. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  10. [18]

    L., Kreuter, F., and Zins, S

    Gerdon, F., Nissenbaum, H., Bach, R. L., Kreuter, F., and Zins, S. Individual acceptance of using health data for private and public benefit: Changes during the covid-19 pandemic. Harvard Data Science Review, 2020

    2020

  11. [19]

    Operationalizing contextual integrity in privacy-conscious assistants

    Ghalebikesabi, S., Bagdasaryan, E., Yi, R., Yona, I., Shumailov, I., Pappu, A., Shi, C., Weidinger, L., Stanforth, R., Berrada, L., et al. Operationalizing contextual integrity in privacy-conscious assistants. arXiv preprint arXiv:2408.02373, 2024

    2024 arXiv

  12. [20]

    Can llms get help from other llms without revealing private information? arXiv preprint arXiv:2404.01041, 2024

    Hartmann, F., Tran, D.-H., Kairouz, P., C a rbune, V., et al. Can llms get help from other llms without revealing private information? arXiv preprint arXiv:2404.01041, 2024

    2024 arXiv

  13. [21]

    Social bias evaluation for large language models requires prompt variations

    Hida, R., Kaneko, M., and Okazaki, N. Social bias evaluation for large language models requires prompt variations. arXiv preprint arXiv:2407.03129, 2024

    2024 arXiv

  14. [22]

    T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al

    Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. Found in the middle: Calibrating positional attention bias improves long context utilization. arXiv preprint arXiv:2406.16008, 2024

    2024 arXiv

  15. [23]

    Internet-augmented dialogue generation

    Komeili, M., Shuster, K., and Weston, J. Internet-augmented dialogue generation. In Annual Meeting of the Association for Computational Linguistics, 2021. URL https://api.semanticscholar.org/CorpusID:236034557

    2021

  16. [24]

    Privacy checklist: Privacy violation detection grounding on contextual integrity theory

    Li, H., Fan, W., Chen, Y., Cheng, J., Chu, T., Zhou, X., Hu, P., and Song, Y. Privacy checklist: Privacy violation detection grounding on contextual integrity theory. arXiv preprint arXiv:2408.10053, 2024

    2024 arXiv

  17. [25]

    Exploring the sensitivity of LLM s ' decision-making capabilities: Insights from prompt variations and hyperparameters

    Loya, M., Sinha, D., and Futrell, R. Exploring the sensitivity of LLM s ' decision-making capabilities: Insights from prompt variations and hyperparameters. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp...

    2023 doi

  18. [26]

    How are prompts different in terms of sensitivity? In Duh, K., Gomez, H., and Bethard, S

    Lu, S., Schuff, H., and Gurevych, I. How are prompts different in terms of sensitivity? In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies...

    2024 doi

  19. [27]

    and Nissenbaum, H

    Martin, K. and Nissenbaum, H. Measuring privacy: An empirical test using context to expose confounding variables. Colum. Sci. & Tech. L. Rev., 18: 0 176, 2016

    2016

  20. [28]

    Can llms keep a secret? testing privacy implications of language models via contextual integrity theory

    Mireshghallah, N., Kim, H., Zhou, X., Tsvetkov, Y., Sap, M., Shokri, R., and Choi, Y. Can llms keep a secret? testing privacy implications of language models via contextual integrity theory. In International Conference on Learning Representation, 2023

    2023

  21. [29]

    State of what art? a call for multi-prompt llm evaluation

    Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., and Stanovsky, G. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12: 0 933--949, 2024

    2024

  22. [30]

    C., Kadhe, S., Wang, H., Murugesan, K., Weisz, J

    Ngong, I. C., Kadhe, S., Wang, H., Murugesan, K., Weisz, J. D., Dhurandhar, A., and Ramamurthy, K. N. Protecting users from themselves: Safeguarding contextual privacy in interactions with conversational agents. In Workshop on Socially Responsible Language Modelling Research, ...

    2024

  23. [31]

    Privacy as contextual integrity

    Nissenbaum, H. Privacy as contextual integrity. Wash. L. Rev., 79: 0 119, 2004

    2004

  24. [32]

    Privacy in context: Technology, policy, and the integrity of social life

    Nissenbaum, H. Privacy in context: Technology, policy, and the integrity of social life. Privacy in Context, November 2009. doi:10.1515/9780804772891

    2009 doi

  25. [33]

    Respect for context as a benchmark for privacy online: What it is and isn't

    Nissenbaum, H. Respect for context as a benchmark for privacy online: What it is and isn't. In Roessler, B. and Mokrosinska, D. (eds.), Social Dimensions of Privacy : Interdisciplinary Perspectives . Cambridge University Press, Cambridge, UK, 2015

    2015

  26. [34]

    Contextual integrity up and down the data food chain

    Nissenbaum, H. Contextual integrity up and down the data food chain. Theoretical inquiries in law, 20 0 (1): 0 221--256, 2019

    2019

  27. [35]

    Foreword by Helen Nissenbaum

    Nissenbaum, H. Foreword by Helen Nissenbaum . Modern Socio-Technical Perspectives on Privacy . Springer Nature, 2022

    2022

  28. [36]

    Talm: Tool augmented language models

    Parisi, A., Zhao, Y., and Fiedel, N. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022

    2022 arXiv

  29. [37]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  30. [38]

    Toolformer: Language models can teach themselves to use tools

    Schick, T., Dwivedi-Yu, J., Dess \` , R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024

    2024

  31. [39]

    Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting

    Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, 2024. URL https://op...

    2024

  32. [40]

    Privacylens: Evaluating privacy norm awareness of language models in action

    Shao, Y., Li, T., Shi, W., Liu, Y., and Yang, D. Privacylens: Evaluating privacy norm awareness of language models in action. arXiv preprint arXiv:2409.00138, 2024

    2024 arXiv

  33. [41]

    Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms

    Shi, L., Ma, C., Liang, W., Ma, W., and Vosoughi, S. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. arXiv preprint arXiv:2406.07791, 2024

    2024

  34. [42]

    and Duddu, V

    Shvartzshnaider, Y. and Duddu, V. Investigating privacy bias in training data of language models, 2025. URL https://arxiv.org/abs/2409.03735

    2025

  35. [43]

    Learning privacy expectations by crowdsourcing contextual informational norms

    Shvartzshnaider, Y., Tong, S., Wies, T., Kift, P., Nissenbaum, H., Subramanian, L., and Mittal, P. Learning privacy expectations by crowdsourcing contextual informational norms. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 4, pp.\ 209--218, 2016

    2016

  36. [44]

    VACCINE : Using Contextual Integrity ForData Leakage Detection

    Shvartzshnaider, Y., Pavlinovic, Z., Balashankar, A., Wies, T., Subramanian, L., Nissenbaum, H., and Mittal, P. VACCINE : Using Contextual Integrity ForData Leakage Detection . In The World Wide Web Conference , WWW '19, pp.\ 1702--1712, New York, NY, USA, 2019. ACM. ISBN 978-...

    2019

  37. [45]

    and Bonotti, M

    Susser, D. and Bonotti, M. Privacy mini-publics: A deliberative democratic approach to understanding informational norms. In Annual Symposium on Applications of Contextual Integrity, 2024

    2024

  38. [46]

    M., Herbert, F., Schaewitz, L., Degeling, M., and D \"u rmuth, M

    Utz, C., Becker, S., Schnitzler, T., Farke, F. M., Herbert, F., Schaewitz, L., Degeling, M., and D \"u rmuth, M. Apps against the spread: Privacy implications and user acceptance of covid-19-related smartphone apps on three continents. In Proceedings of the 2021 chi conference...

    2021

  39. [47]

    and Zimmer, M

    Vitak, J. and Zimmer, M. More than just privacy: Using contextual integrity to evaluate the long-term risks from covid-19 surveillance technologies. Social media+ society, 6 0 (3): 0 2056305120948250, 2020

    2020

  40. [48]

    Mitigate position bias in large language models via scaling a single dimension

    Yu, Y., Jiang, H., Luo, X., Wu, Q., Lin, C.-Y., Li, D., Yang, Y., Huang, Y., and Qiu, L. Mitigate position bias in large language models via scaling a single dimension. arXiv preprint arXiv:2406.02536, 2024

    2024 arXiv

  41. [49]

    Can we instruct llms to compensate for position bias? In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 12545--12556, 01 2024

    Zhang, M., Meng, Z., and Collier, N. Can we instruct llms to compensate for position bias? In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 12545--12556, 01 2024. doi:10.18653/v1/2024.findings-emnlp.732

    2024 doi

  42. [50]

    On large language models' selection bias in multi-choice questions

    Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. On large language models' selection bias in multi-choice questions. arXiv preprint arXiv:2309.03882, 2023

    2023 arXiv

  43. [51]

    P ro SA : Assessing and understanding the prompt sensitivity of LLM s

    Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., and Chen, K. P ro SA : Assessing and understanding the prompt sensitivity of LLM s. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 1950--1976, ...

    2024 doi

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.