Pith. sign in

REVIEW 4 major objections 4 minor 38 references

LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LangFair is an open-source Python package that audits LLM bias and fairness for a practitioner's own prompts, not generic benchmarks.

desk verdict LangFair is a useful BYOP fairness-metrics wrapper, but the paper needs validation of its external classifiers before I'd trust its numbers for audits. read the letter →

arxiv 2501.03112 v1 pith:ASOYHR2O submitted 2025-01-06 cs.CL cs.AIcs.CYcs.LG

Dylan Bouchard , Mohit Singh Chauhan , David Skarbrevik , Viren Bajaj , Zeya Ahmad This is my paper · ORCID
classification cs.CLcs.AIcs.CYcs.LG
keywords LangFairlargelanguagemodelsbiasevaluationfairnessmetricscounterfactualtoxicitydetectionstereotypebringyourownprompts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces LangFair, an open-source Python package for evaluating bias and fairness risks in concrete LLM use cases. The package lets practitioners supply their own prompts, generates LLM responses to those prompts, and computes metrics for toxicity, stereotypes, counterfactual unfairness, and classification disparities. It is built around a bring-your-own-prompts approach, so the assessment reflects the actual task a deployed system performs rather than a generic benchmark. A companion decision framework guides which metrics to apply, based in part on whether the prompts mention protected attribute groups. If the package works as described, it gives application developers and auditors a practical, output-only route to fairness assessments without needing access to model internals.

What carries the argument

The load-bearing mechanism is the bring-your-own-prompts (BYOP) evaluation loop: ResponseGenerator and CounterfactualGenerator turn a user's prompt list into a response dataset, and the metric classes then summarize those responses. The selection logic is driven by a fairness-through-unawareness (FTU) check, which parses prompts for mentions of protected attribute groups; when FTU is satisfied only classification-style metrics apply, and when it is not, counterfactual prompt pairs are generated and compared. AutoEval orchestrates the whole loop by checking FTU, generating responses and counterfactual responses, and then reporting the applicable toxicity, stereotype, and counterfactual metrics. The individual metrics lean on external pre-trained classifiers and similarity measures, so LangFair's own contribution is the workflow, dataset-generation layer, and metric-categorization system rather than new detection models.

What would settle it

Run LangFair's AutoEval on a fixed set of LLM responses and have independent human annotators label the same responses for toxicity and stereotypes; if the package's rankings of models do not match the human rankings, such as a model with a lower toxicity score being judged more toxic by people, the central claim that LangFair measures bias and fairness risk fails.

Watch

Extended reading notes

Core claim

The central claim is that a practical fairness audit of an LLM use case can be built from the model's generated responses alone, and that LangFair implements that audit. The package wraps the user's chosen LLM client to generate responses asynchronously from user-supplied prompts, checks whether prompts satisfy fairness through unawareness, builds counterfactual prompt pairs when they do not, and then computes metrics grouped by risk: toxicity metrics from pre-trained toxicity classifiers, stereotype metrics from word cooccurrence and a stereotype classifier, counterfactual fairness metrics from sentiment and text-similarity comparisons, recommendation fairness metrics from list similarity, and classification fairness metrics from disparities in prevalence, false positives, and false negatives. An AutoEval class combines these steps into a semi-automated pipeline. The paper argues this output-only design is more practical for real-world systems than metrics requiring internal access, and cites prior evidence that output-based metrics can be more reliable than intrinsic ones.

Load-bearing premise

LangFair's scores are only as trustworthy as the external classifiers used to label model outputs, and the paper assumes those classifiers behave correctly on arbitrary LLM text without testing that assumption.

Editorial extensions

If this is right

  • A deployed LLM application can be audited for bias and fairness using only its prompts and outputs, without access to model weights, embeddings, or token probabilities.
  • Practitioners can tailor evaluation to their specific task: the same model can score differently on toxicity, stereotype, counterfactual, recommendation, and classification metrics depending on which prompts are used.
  • If fairness through unawareness is violated, counterfactual prompt pairs give a concrete way to measure treatment differences across protected attribute groups.
  • Governance teams can run semi-automated audits through AutoEval, making the assessment repeatable across model versions and prompt updates.
  • Traditional ML classification fairness metrics, expressed as pairwise differences or ratios, become available for LLM-as-classifier use cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, LangFair's practical value hinges on prompt sampling: the same model could look fair or unfair depending on which prompts the user supplies, so the package would be strengthened by guidance or defaults for curating a representative prompt list.
  • Beyond the paper, the output-only design could be tested against intrinsic measures on the same model, comparing LangFair's scores with embedding- or probability-based bias metrics to see where they agree and disagree.
  • Beyond the paper, an empirical validation study comparing LangFair's metric values with human judgments of toxicity and stereotype on diverse LLM outputs would show whether the reported scores track perceived harm; the paper does not report such a study.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper introduces LangFair, an open-source Python package for assessing bias and fairness in LLM-based systems. The package follows a 'bring your own prompts' approach: users supply prompts and a LangChain LLM, LangFair generates responses (including counterfactual pairs), and computes metrics for toxicity, stereotypes, counterfactual fairness, recommendation, and classification. It also provides an AutoEval class that automates metric selection, response generation, and metric computation for text generation use cases. The manuscript describes the package's modules and classes, gives one code example, and refers to a companion paper for the metric-selection decision framework. The central claim is that LangFair equips practitioners with tools to evaluate bias and fairness risks relevant to their use cases.

Significance. If the package performs as described, it would fill a real gap: many existing LLM fairness tools rely on static benchmark prompts, whereas LangFair targets task-specific, prompt-based evaluation for governance audits. The open-source release, with documentation, tutorial notebooks, and a technical companion paper, is a practical contribution that lowers the barrier for practitioners. The design choices are sensible: response-only metrics sidestep difficulties of accessing internal model states, and the modular architecture allowing custom classifiers is a strength. However, the manuscript is purely a software description with no empirical evaluation, no numerical examples, and no comparison with existing toolkits, so the scientific contribution currently rests on the repository's existence and on the prior validation of the external models it wraps. The paper's utility for governance audits depends on measurement validity, which is not demonstrated.

major comments (4)
  1. [Section 3 (ToxicityMetrics, StereotypeMetrics, CounterfactualMetrics)] Every metric described in Section 3 is computed from an external classifier or similarity measure (detoxify, Sentence-Level-Stereotype-Detector, VADER, and generic text similarity), but the paper provides no calibration, error analysis, or robustness evidence for these instruments on LLM outputs. Since LangFair's stated purpose is to support governance audits, a false fairness signal caused by classifier error correlated with protected attributes would be a material failure. This concern is load-bearing for the central claim that practitioners can 'evaluate bias and fairness risks' with LangFair. The authors should add a validation study (e.g., comparing metric values against human judgments or against known biased/unbiased response sets) or, at minimum, include an explicit and prominent limitation statement.
  2. [Sections 3-4 (no evaluation section)] The manuscript contains no test outputs, no numerical examples, no unit-test results, and no comparison with existing toolkits such as AIF360, Aequitas, or FaiRLLM. Section 3 lists metric classes without reporting any computed values, and Section 4 shows only a code listing (Listing 1) with no returned results. For a software paper, this makes the central existence and functionality claim unverifiable from the manuscript alone. Please include at least one end-to-end example with actual metric values on a small prompt set, and ideally a reproducibility statement pointing to a specific repository commit, test suite, and CI status.
  3. [Section 4 and abstract] The abstract promises that 'LangFair offers an actionable decision framework,' but the decision framework is not described in this paper; Section 4 simply states that metric selection follows Bouchard (2024) and Table 1 references that companion paper. This is not circular, but it prevents the manuscript from standing alone. Please summarize the decision rules (for example, the FTU check, the risk taxonomy of toxicity, stereotypes, counterfactual fairness, and allocational harms, and the mapping to applicable tasks) so a reader can apply the framework without consulting the companion paper.
  4. [Section 2 (CounterfactualGenerator)] The fairness-through-unawareness (FTU) check is described as parsing prompts for mentions of protected attribute groups using off-the-shelf word mappings for gender and race/ethnicity, but the paper gives no details on the coverage of these word lists, their false-positive/false-negative behavior, or how custom mappings are applied. Since FTU determines whether counterfactual metrics are computed in the AutoEval flow (Section 4), this is a load-bearing step for the package's correctness. Please clarify the implementation and provide evidence or tests of its behavior.
minor comments (4)
  1. [Throughout] There is a typo in the header 'LangF air' and a misspelling 'satsified' in Section 4; these should be corrected.
  2. [Section 1] The claim that output-based metrics are 'potentially more reliable' than embedding- or probability-based metrics is supported only by two citations; the paper should clarify that this is an assumption drawn from prior work, not a result validated in LangFair.
  3. [Section 4 (Listing 1)] The code example calls an async method but does not show the surrounding async context or how 'prompts' is defined; adding a complete, runnable snippet would improve reproducibility.
  4. [References] Several references are incomplete (e.g., 'TrustGPT' is cited as a pre-print with a numeric identifier, and 'Huggingface' has no version or release date); please add arXiv IDs and version information where available.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: LangFair is a software paper that implements and packages externally defined metrics rather than deriving a result from its own definitions.

full rationale

LangFair's central claim is that the package exists and can compute specified bias and fairness metrics from user-provided prompts using external classifiers and standard metric definitions. There is no derivation chain in which an input is reintroduced as a prediction: toxicity metrics wrap detoxify/evaluate/toxigen, stereotype metrics wrap the Sentence-Level-Stereotype-Detector, counterfactual sentiment metrics wrap VADER, and the classification and similarity metrics reuse standard definitions from AIF360, Aequitas, and text-similarity measures. These are external instruments, so using them is not circular; the paper never claims to derive or validate the classifiers themselves. The repeated citations to Bouchard (2024), a companion paper by one of the authors, point to a separately written metric-selection framework and metric taxonomy. Those citations are self-references, but they are not load-bearing in the sense of forcing a mathematical or empirical result: the package's calculations are implemented over external model outputs rather than being entailed by the companion paper. The main weakness, that off-the-shelf classifiers may be unreliable on arbitrary LLM outputs, is a measurement-validity concern rather than a circularity in the paper's reasoning. No circular step can be exhibited from the manuscript's own equations or definitions, so the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted in the paper. The core of the system is the assumption that output-only measurements via borrowed classifiers are valid proxies for bias and fairness, plus the assumption that word-list based FTU checks are sufficient. The metric taxonomy is imported from the authors' companion paper, which is a form of self-citation at the design level.

assumptions (3)
  • domain assumption Output-only fairness metrics are more reliable than embedding or token probability based metrics.
    Invoked in Section 1 with citations to Goldfarb-Tarrant et al. and Delobelle et al.; if false, the design choice to avoid internal model states may reduce measurement quality.
  • domain assumption Pre-trained toxicity, stereotype, and sentiment classifiers provide valid measurements for LLM outputs.
    Section 3 wraps detoxify, wu981526092/Sentence-Level-Stereotype-Detector, and VADER without validating them on LLM-generated text.
  • domain assumption Fairness through unawareness can be detected by parsing prompts for protected attribute words.
    Section 2 CounterfactualGenerator: the off-the-shelf FTU check uses word lists for gender and race, which may miss contextual or indirect mentions of protected attributes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases." pith.science (2026). https://pith.science/paper/ASOYHR2O

@misc{pith2026250103112,
  author       = {Pith},
  title        = {Pith review of: LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ASOYHR2O}},
  note         = {Machine review of arXiv:2501.03112}
}
read the original abstract

Large Language Models (LLMs) have been observed to exhibit bias in numerous ways, potentially creating or worsening outcomes for specific groups identified by protected attributes such as sex, race, sexual orientation, or age. To help address this gap, we introduce LangFair, an open-source Python package that aims to equip LLM practitioners with the tools to evaluate bias and fairness risks relevant to their specific use cases. The package offers functionality to easily generate evaluation datasets, comprised of LLM responses to use-case-specific prompts, and subsequently calculate applicable metrics for the practitioner's use case. To guide in metric selection, LangFair offers an actionable decision framework.

Figures

Figures reproduced from arXiv: 2501.03112 by the authors.

Figure 1
Figure 1. Flowchart of internal design of AutoEval.evaluate method. 5. Conclusions In this paper, we introduced langfair, an open-source Python package that aims to equip LLM practitioners with the tools to evaluate bias and fairness risks relevant to their specific use cases. The package offers functionality to easily generate evaluation datasets, comprised of LLM responses to use-case-specific prompts, and subsequently calc… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 9 canonical work pages

  1. [1]

    Redditbias: A real-world resource for bias evaluation and debiasing of conversational language models, 2021

    Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. Redditbias: A real-world resource for bias evaluation and debiasing of conversational language models, 2021. URL https://arxiv.org/abs/2106.03521

    arXiv 2021

  2. [2]

    Unmasking contextual stereotypes: Measuring and mitigating bert's gender bias

    Marion Bartl, Malvina Nissim, and Albert Gatt. Unmasking contextual stereotypes: Measuring and mitigating bert's gender bias. In Marta R. Costa-jussà, Christian Hardmeier, Kellie Webster, and Will Radford, editors, Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, 2020

    work page 2020

  3. [3]

    Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang. AI Fairness 360: An extensible too...

    arXiv 2018

  4. [4]

    An actionable framework for assessing bias and fairness in large language model use cases, 2024

    Dylan Bouchard. An actionable framework for assessing bias and fairness in large language model use cases, 2024. URL https://arxiv.org/abs/2407.10853

    arXiv 2024

  5. [5]

    Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models

    Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa...

    2022

  6. [6]

    Bold: Dataset and metrics for measuring biases in open-ended language generation

    Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 862–872, New York, NY, USA, 2021. Association for Computing ...

    arXiv 2021

  7. [7]

    Felkner, Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May

    Virginia K. Felkner, Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May. Winoqueer: A community-in-the-loop benchmark for anti-lgbtq+ bias in large language models, 2024. URL https://arxiv.org/abs/2306.15087

    arXiv 2024

  8. [8]

    Gallegos, Ryan A

    Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: A survey, 2024. URL https://arxiv.org/abs/2309.00770

    arXiv 2024

Show all 38 references
  1. [9]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  2. [10]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings, 2020. URL https://api.semanticscholar.org/CorpusID:221878771

    2020

  3. [11]

    Intrinsic bias metrics do not correlate with application bias, 2021

    Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sanchez, Mugdha Pandya, and Adam Lopez. Intrinsic bias metrics do not correlate with application bias, 2021. URL https://arxiv.org/abs/2012.15859

    2021 arXiv

  4. [12]

    Reducing sentiment bias in language models via counterfactual evaluation, 2020

    Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfactual evaluation, 2020. URL https://arxiv.org/abs/1911.03064

    2020 arXiv

  5. [13]

    Y, and Lichao Sun

    Yue Huang, Qihui Zhang, Philip S. Y, and Lichao Sun. Trustgpt: A benchmark for trustworthy and responsible large language models, 2023. URL https://arxiv.org/abs/2306.11507

    2023 arXiv

  6. [14]

    Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P....

    2024

  7. [15]

    Github - huggingface/evaluate: Evaluate: A library for easily evaluating machine learning models and datasets., 2022

    Huggingface. Github - huggingface/evaluate: Evaluate: A library for easily evaluating machine learning models and datasets., 2022. URL https://github.com/huggingface/evaluate

    2022

  8. [16]

    Mohammad

    Svetlana Kiritchenko and Saif M. Mohammad. Examining gender and race bias in two hundred sentiment analysis systems, 2018. URL https://arxiv.org/abs/1805.04508

    2018 arXiv

  9. [17]

    Grep-biasir: A dataset for investigating gender representation bias in information retrieval results

    Klara Krieg, Emilia Parada-Cabaleiro, Gertraud Medicus, Oleg Lesota, Markus Schedl, and Navid Rekabsaz. Grep-biasir: A dataset for investigating gender representation bias in information retrieval results. In Proceedings of the 2023 Conference on Human Information Interaction ...

    2023

  10. [18]

    Collecting a large-scale gender bias dataset for coreference resolution and machine translation, 2021

    Shahar Levy, Koren Lazar, and Gabriel Stanovsky. Collecting a large-scale gender bias dataset for coreference resolution and machine translation, 2021

    2021

  11. [19]

    UNQOVER ing stereotyping biases via underspecified questions

    Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. UNQOVER ing stereotyping biases via underspecified questions. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3475--3489, O...

    2020 doi

  12. [20]

    Manning, Christopher Ré, Diana Acosta-Navas, Drew A

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas...

    2023 arXiv

  13. [21]

    Stereoset: Measuring stereotypical bias in pretrained language models, 2020

    Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models, 2020

    2020

  14. [22]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online, November 2020. Associa...

    2020

  15. [23]

    LangTest: A comprehensive evaluation library for custom LLM and NLP models

    Arshaan Nazir, Thadaka Kalyan Chakravarthy, David Amore Cecchini, Thadaka Kalyan Chakravarthy, Rakshit Khajuria, Prikshit Sharma, Ali Tarik Mirik, Veysel Kocaman, and David Talby. LangTest: A comprehensive evaluation library for custom LLM and NLP models . Software Impacts, 19...

    2024

  16. [24]

    HONEST : Measuring hurtful sentence completion in language models

    Debora Nozza, Federico Bianchi, and Dirk Hovy. " HONEST : Measuring hurtful sentence completion in language models". In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2398--2...

    2021 doi

  17. [25]

    BBQ : A hand-built bias benchmark for question answering

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ : A hand-built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Associ...

    2022 doi

  18. [26]

    Perturbation augmentation for fairer nlp, 2022

    Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams. Perturbation augmentation for fairer nlp, 2022. URL https://arxiv.org/abs/2205.12586

    2022 arXiv

  19. [27]

    Gender bias in coreference resolution

    Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme . Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Lou...

    2018

  20. [28]

    Aequitas: A bias and fairness audit toolkit

    Pedro Saleiro, Benedict Kuester, Abby Stevens, Ari Anisfeld, Loren Hinkson, Jesse London, and Rayid Ghani. Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577, 2018

    2018 arXiv

  21. [29]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri `a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint...

    2022 doi arXiv

  22. [30]

    GitHub - tensorflow/fairness-indicators: Tensorflow's Fairness Evaluation and Visualization Toolkit , 2020

    Tensorflow. GitHub - tensorflow/fairness-indicators: Tensorflow's Fairness Evaluation and Visualization Toolkit , 2020. URL https://github.com/tensorflow/fairness-indicators

    2020

  23. [31]

    LiFT : A scalable framework for measuring fairness in ml applications

    Sriram Vasudevan and Krishnaram Kenthapadi. LiFT : A scalable framework for measuring fairness in ml applications. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management, CIKM '20, 2020

    2020

  24. [32]

    Decodingtrust: A comprehensive assessment of trustworthiness in gpt models

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. 2023

    2023

  25. [33]

    Mind the GAP : A balanced corpus of gendered ambiguous pronouns

    Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. Mind the GAP : A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6: 0 605--617, 2018. doi:10.1162/tacl_a_00240. URL https://aclanthology.org/Q18-1042

    2018 doi

  26. [34]

    Fairlearn: Assessing and Improving Fairness of AI Systems

    Hilde Weerts, Miroslav Dudík, Richard Edgar, Adrin Jalali, Roman Lutz, and Michael Madaio. Fairlearn: Assessing and Improving Fairness of AI Systems . Journal of Machine Learning Research, 24, 2023. URL http://jmlr.org/papers/v24/23-0389.html

    2023

  27. [35]

    Vi \' e gas, and Jimbo Wilson

    James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda B. Vi \' e gas, and Jimbo Wilson. The what-if tool: Interactive probing of machine learning models. CoRR, abs/1907.04135, 2019. URL http://arxiv.org/abs/1907.04135

    1907 arXiv

  28. [36]

    Towards auditing large language models: Improving text-based stereotype detection, 2023

    Wu Zekun, Sahan Bulathwela, and Adriano Soares Koshiyama. Towards auditing large language models: Improving text-based stereotype detection, 2023. URL https://arxiv.org/abs/2311.14126

    2023 arXiv

  29. [37]

    Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation

    Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, volume 2012 of RecSys ’23, page 993–99...

    2012

  30. [38]

    Gender Bias in Coreference Resolution: Evaluation and Debiasing methods , 4 2018

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender Bias in Coreference Resolution: Evaluation and Debiasing methods , 4 2018. URL https://arxiv.org/abs/1804.06876

    2018 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.