Knowledge

MMLU-Pro

Released model predictions on multiple-choice questions across fourteen academic domains.

14,109items

48subjects

Apache-2.0 (captured upstream repository LICENSE).license

knowledgedomain

reasoningdomain

textmodality

item-level responses released

Saturation status: Yes

Full data on Hugging Face Back to the gallery

Paper and credit

2024

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (opens in a new tab)

Original source

Authors

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni +13 moreShow all

01Yubo Wang
02Xueguang Ma
03Ge Zhang
04Yuansheng Ni
05Abhranil Chandra
06Shiguang Guo
07Weiming Ren
08Aaran Arulraj
09Xuan He
10Ziyan Jiang
11Tianle Li
12Max Ku
13Kai Wang
14Alex Zhuang
15Rongqi Fan
16Xiang Yue
17Wenhu Chen

Citation

BibTeX citationShow

BibTeX

1@misc{wang2024mmluprorobustchallengingmultitask,
2      title={MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark},
3      author={Yubo Wang and Xueguang Ma and Ge Zhang and Yuansheng Ni and Abhranil Chandra and Shiguang Guo and Weiming Ren and Aaran Arulraj and Xuan He and Ziyan Jiang and Tianle Li and Max Ku and Kai Wang and Alex Zhuang and Rongqi Fan and Xiang Yue and Wenhu Chen},
4      year={2024},
5      eprint={2406.01574},
6      archivePrefix={arXiv},
7      primaryClass={cs.CL},
8      url={https://arxiv.org/abs/2406.01574},
9}

Response matrix

leaderboard per-category aggregates

Loading response matrix…

Correct (1)Incorrect (0)Unobserved

Scale: 1 = correct · 0 = incorrect