Pith. sign in

REVIEW 4 cited by

Bias Testing and Mitigation in LLM-based Code Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.14345 v4 pith:BGPUMWDO submitted 2023-09-03 cs.SE cs.AI

Dong Huang , Jie M. Zhang , Qingwen Bu , Xiaofei Xie , Junjie Chen , Heming Cui This is my paper · ORCID
classification cs.SEcs.AI
keywords codebiasgeneratedbiasesgenerationllmsfiveframework
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

As the adoption of LLMs becomes more widespread in software coding ecosystems, a pressing issue has emerged: does the generated code contain social bias and unfairness, such as those related to age, gender, and race? This issue concerns the integrity, fairness, and ethical foundation of software applications that depend on the code generated by these models but are underexplored in the literature. This paper presents a novel bias testing framework that is specifically designed for code generation tasks. Based on this framework, we conduct an extensive empirical study on the biases in code generated by five widely studied LLMs (i.e., PALM-2-CodeChat-bison, Claude-instant-1, GPT-3.5-turbo, GPT-4-turbo, and GPT-4). Our findings reveal that biases are prevalent. For example, 13.47% to 49.10% of the codes generated by these LLMs have biased behaviors towards gender. Moreover, we study five bias mitigation prompt strategies that are commonly used in current code generation scenarios, i.e., zero-shot, one-shot, few-shot, and two Chain-of-Thought (CoT) prompts, with and without provided feedback-driven refinement. Our evaluation results illustrate that using direct prompt engineering strategies has limited effectiveness in mitigating bias, but our test execution feedback can help to reduce the ratio of code biases to a large extent (e.g., from 59.88% to 4.79% for GPT-4).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation

    cs.SE 2026-08 conditional novelty 6.0 of 10

    On a new 1,440-screenshot fill-in-the-blank benchmark, five frontier multimodal models default to repeated UI patterns instead of visual deviations, with mean bias rates from 69.8% on card widths to 80.2% on font sizes.

  2. BiasFilter: An Inference-Time Debiasing Framework for Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    BiasFilter filters low-fairness segments during LLM generation using a reward model trained on a GPT-4-scored preference dataset, cutting bias on CEB and FairMT.

  3. Uncovering Intention through LLM-Driven Code Snippet Description Generation

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A manual and LLM-based study of 400 npm code snippet descriptions finds example-based descriptions dominate and that Llama's classifications and generated descriptions need human oversight.

  4. LFTF: Locating First and Then Fine-Tuning for Mitigating Gender Bias in Large Language Models

    cs.CL 2025-05 reject novelty 4.0 of 10

    A block-localizing fine-tuning method for gender debiasing is presented, but its stated loss is inconsistent with its reported behavior and the evaluation tables contain duplicate rows.

Pith tools