Safety

BBQ

BBQ tests whether models pick the correct answer to multiple choice questions about people from different social groups, asking each question once with too little context to answer it and once with the context that settles it.

2,000items

81subjects

unknownlicense

safetydomain

textmodality

item-level responses released

Saturation status: Yes

Full data on Hugging Face Back to the gallery

Paper and credit

2022

BBQ: A hand-built bias benchmark for question answering (opens in a new tab)

Original source

Authors

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar +4 moreShow all

01Alicia Parrish
02Angelica Chen
03Nikita Nangia
04Vishakh Padmakumar
05Jason Phang
06Jana Thompson
07Phu Mon Htut
08Samuel R. Bowman

Citation

BibTeX citationShow

BibTeX

1@inproceedings{parrish-etal-2022-bbq,
2  title={{BBQ}: A hand-built bias benchmark for question answering},
3  author={Parrish, Alicia and Chen, Angelica and Nangia, Nikita and Padmakumar, Vishakh and Phang, Jason and Thompson, Jana and Htut, Phu Mon and Bowman, Samuel R.},
4  booktitle={Findings of the Association for Computational Linguistics: ACL 2022},
5  year={2022},
6  publisher={Association for Computational Linguistics},
7  pages={2086--2105},
8  doi={10.18653/v1/2022.findings-acl.165},
9  url={https://aclanthology.org/2022.findings-acl.165/},
10}

Response matrix

Responses come from Stanford CRFM's HELM Safety leaderboard (v1.17.0), which re-ran BBQ 0-shot as forced multiple choice at temperature 0 -- not the original evaluation of Parrish et al.

Loading response matrix…

Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect