Pith. sign in

REVIEW 5 cited by

JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.02050 v4 pith:JYQ5ZJWK submitted 2024-06-04 cs.CL

Hitomi Yanaka , Namgi Han , Ryoma Kumon , Jie Lu , Masashi Takeshita , Ryo Sekizawa , Taisei Kato , Hiromi Arai This is my paper · ORCID
classification cs.CL
keywords biasesjapanesesocialllmsbiasjbbqbenchmarkdataset
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

With the development of large language models (LLMs), social biases in these LLMs have become a pressing issue. Although there are various benchmarks for social biases across languages, the extent to which Japanese LLMs exhibit social biases has not been fully investigated. In this study, we construct the Japanese Bias Benchmark dataset for Question Answering (JBBQ) based on the English bias benchmark BBQ, with analysis of social biases in Japanese LLMs. The results show that while current open Japanese LLMs with more parameters show improved accuracies on JBBQ, their bias scores increase. In addition, prompts with a warning about social biases and chain-of-thought prompting reduce the effect of biases in model outputs, but there is room for improvement in extracting the correct evidence from contexts in Japanese. Our dataset is available at https://github.com/ynklab/JBBQ_data.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SESGO: Spanish Evaluation of Stereotypical Generative Outputs

    cs.CY 2025-09 conditional novelty 6.0 of 10

    A culturally grounded Spanish benchmark of 4,156 prompts shows commercial LLMs exhibit different and often stronger bias in Spanish than in English, and that bias persists across sampling temperatures.

  2. McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Models

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A new Chinese bias benchmark with 4,077 instances and five tasks indicates larger language models are less biased than smaller ones when bias is measured through understanding tasks.

  3. Intersectional Bias in Japanese Large Language Models from a Contextualized Perspective

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Using the new inter-JBBQ benchmark, biased responses of Japanese LLMs vary with scenario context even when similar social attribute combinations are tested.

  4. AnswerCarefully: A Dataset for Improving the Safety of Japanese LLM Output

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A Japanese safety dataset with reference answers improves LLM safety through fine-tuning and provides a benchmark that reveals large safety differences across 12 models.

  5. GG-BBQ: German Gender Bias Benchmark for Question Answering

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A manually corrected German translation of the BBQ gender bias QA dataset shows that all evaluated German LLMs exhibit measurable gender bias.

Pith tools