<p>Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding<sup><CitationRef CitationID="CR1">1</CitationRef></sup>, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity’s Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable but cannot be quickly answered by internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a marked gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at <a href="https://lastexam.ai">https://lastexam.ai</a>.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A benchmark of expert-level academic questions to assess AI capabilities

  • Long Phan,
  • Alice Gatti,
  • Nathaniel Li,
  • Adam Khoja,
  • Ryan Kim,
  • Richard Ren,
  • Jason Hausenloy,
  • Oliver Zhang,
  • Mantas Mazeika,
  • Dan Hendrycks,
  • Ziwen Han,
  • Josephina Hu,
  • Hugh Zhang,
  • Chen Bo Calvin Zhang,
  • Mohamed Shaaban,
  • John Ling,
  • Sean Shi,
  • Michael Choi,
  • Anish Agrawal,
  • Arnav Chopra,
  • Aakaash Nattanmai,
  • Gordon McKellips,
  • Anish Cheraku,
  • Asim Suhail,
  • Ethan Luo,
  • Marvin Deng,
  • Jason Luo,
  • Ashley Zhang,
  • Kavin Jindel,
  • Jay Paek,
  • Kasper Halevy,
  • Allen Baranov,
  • Michael Liu,
  • Advaith Avadhanam,
  • David Zhang,
  • Vincent Cheng,
  • Brad Ma,
  • Evan Fu,
  • Liam Do,
  • Joshua Lass,
  • Hubert Yang,
  • Surya Sunkari,
  • Vishruth Bharath,
  • Violet Ai,
  • James Leung,
  • Rishit Agrawal,
  • Alan Zhou,
  • Kevin Chen,
  • Tejas Kalpathi,
  • Ziqi Xu,
  • Gavin Wang,
  • Tyler Xiao,
  • Erik Maung,
  • Sam Lee,
  • Ryan Yang,
  • Roy Yue,
  • Ben Zhao,
  • Julia Yoon,
  • Xiangwan Sun,
  • Aryan Singh,
  • Clark Peng,
  • Tyler Osbey,
  • Taozhi Wang,
  • Daryl Echeazu,
  • Timothy Wu,
  • Spandan Patel,
  • Vidhi Kulkarni,
  • Vijaykaarti Sundarapandiyan,
  • Andrew Le,
  • Zafir Nasim,
  • Srikar Yalam,
  • Ritesh Kasamsetty,
  • Soham Samal,
  • David Sun,
  • Nihar Shah,
  • Abhijeet Saha,
  • Alex Zhang,
  • Leon Nguyen,
  • Laasya Nagumalli,
  • Kaixin Wang,
  • Aidan Wu,
  • Anwith Telluri,
  • Summer Yue,
  • Alexandr Wang,
  • Dmitry Dodonov,
  • Tung Nguyen,
  • Jaeho Lee,
  • Daron Anderson,
  • Mikhail Doroshenko,
  • Alun Cennyth Stokes,
  • Mobeen Mahmood,
  • Oleksandr Pokutnyi,
  • Oleg Iskra,
  • Jessica P. Wang,
  • John-Clark Levin,
  • Mstyslav Kazakov,
  • Fiona Feng,
  • Steven Y. Feng,
  • Haoran Zhao,
  • Michael Yu,
  • Varun Gangal,
  • Chelsea Zou,
  • Zihan Wang,
  • Serguei Popov,
  • Robert Gerbicz,
  • Geoff Galgon,
  • Johannes Schmitt,
  • Will Yeadon,
  • Yongki Lee,
  • Scott Sauers,
  • Alvaro Sanchez,
  • Fabian Giska,
  • Marc Roth,
  • Søren Riis,
  • Saiteja Utpala,
  • Noah Burns,
  • Gashaw M. Goshu,
  • Mohinder Maheshbhai Naiya,
  • Chidozie Agu,
  • Zachary Giboney,
  • Antrell Cheatom,
  • Francesco Fournier-Facio,
  • Sarah-Jane Crowson,
  • Lennart Finke,
  • Zerui Cheng,
  • Jennifer Zampese,
  • Ryan G. Hoerr,
  • Mark Nandor,
  • Hyunwoo Park,
  • Tim Gehrunger,
  • Jiaqi Cai,
  • Ben McCarty,
  • Alexis C. Garretson,
  • Edwin Taylor,
  • Damien Sileo,
  • Qiuyu Ren,
  • Usman Qazi,
  • Lianghui Li,
  • Jungbae Nam,
  • John B. Wydallis,
  • Pavel Arkhipov,
  • Jack Wei Lun Shi,
  • Aras Bacho,
  • Chris G. Willcocks,
  • Hangrui Cao,
  • Sumeet Motwani,
  • Emily de Oliveira Santos,
  • Johannes Veith,
  • Edward Vendrow,
  • Doru Cojoc,
  • Kengo Zenitani,
  • Joshua Robinson,
  • Longke Tang,
  • Yuqi Li,
  • Joshua Vendrow,
  • Natanael Wildner Fraga,
  • Vladyslav Kuchkin,
  • Andrey Pupasov Maksimov,
  • Pierre Marion,
  • Denis Efremov,
  • Jayson Lynch,
  • Kaiqu Liang,
  • Aleksandar Mikov,
  • Andrew Gritsevskiy,
  • Julien Guillod,
  • Gözdenur Demir,
  • Dakotah Martinez,
  • Ben Pageler,
  • Kevin Zhou,
  • Saeed Soori,
  • Ori Press,
  • Henry Tang,
  • Paolo Rissone,
  • Sean R. Green,
  • Lina Brüssel,
  • Moon Twayana,
  • Aymeric Dieuleveut,
  • Joseph Marvin Imperial,
  • Ameya Prabhu,
  • Jinzhou Yang,
  • Nick Crispino,
  • Arun Rao,
  • Dimitri Zvonkine,
  • Gabriel Loiseau,
  • Mikhail Kalinin,
  • Marco Lukas,
  • Ciprian Manolescu,
  • Nate Stambaugh,
  • Subrata Mishra,
  • Tad Hogg,
  • Carlo Bosio,
  • Brian P. Coppola,
  • Julian Salazar,
  • Jaehyeok Jin,
  • Rafael Sayous,
  • Stefan Ivanov,
  • Philippe Schwaller,
  • Shaipranesh Senthilkumar,
  • Andres M. Bran,
  • Andres Algaba,
  • Kelsey Van den Houte,
  • Lynn Van Der Sypt,
  • Brecht Verbeken,
  • David Noever,
  • Alexei Kopylov,
  • Benjamin Myklebust,
  • Bikun Li,
  • Lisa Schut,
  • Evgenii Zheltonozhskii,
  • Qiaochu Yuan,
  • Derek Lim,
  • Richard Stanley,
  • Tong Yang,
  • John Maar,
  • Julian Wykowski,
  • Mart Oller,
  • Anmol Sahu,
  • Cesare Giulio Ardito,
  • Yuzheng Hu,
  • Ariel Ghislain Kemogne Kamdoum,
  • Alvin Jin,
  • Tobias Garcia Vilchis,
  • Yuexuan Zu,
  • Martin Lackner,
  • James Koppel,
  • Gongbo Sun,
  • Daniil S. Antonenko,
  • Steffi Chern,
  • Bingchen Zhao,
  • Pierrot Arsene,
  • Joseph M. Cavanagh,
  • Daofeng Li,
  • Jiawei Shen,
  • Donato Crisostomi,
  • Wenjin Zhang,
  • Ali Dehghan,
  • Sergey Ivanov,
  • David Perrella,
  • Nurdin Kaparov,
  • Allen Zang,
  • Ilia Sucholutsky,
  • Arina Kharlamova,
  • Daniil Orel,
  • Vladislav Poritski,
  • Shalev Ben-David,
  • Zachary Berger,
  • Parker Whitfill,
  • Michael Foster,
  • Daniel Munro,
  • Linh Ho,
  • Shankar Sivarajan,
  • Dan Bar Hava,
  • Aleksey Kuchkin,
  • David Holmes,
  • Alexandra Rodriguez-Romero,
  • Frank Sommerhage,
  • Anji Zhang,
  • Richard Moat,
  • Keith Schneider,
  • Zakayo Kazibwe,
  • Don Clarke,
  • Dae Hyun Kim,
  • Felipe Meneguitti Dias,
  • Sara Fish,
  • Veit Elser,
  • Tobias Kreiman,
  • Victor Efren Guadarrama Vilchis,
  • Immo Klose,
  • Ujjwala Anantheswaran,
  • Adam Zweiger,
  • Kaivalya Rawal,
  • Jeffery Li,
  • Jeremy Nguyen,
  • Nicolas Daans,
  • Haline Heidinger,
  • Maksim Radionov,
  • Václav Rozhoň,
  • Vincent Ginis,
  • Christian Stump,
  • Niv Cohen,
  • Rafał Poświata,
  • Josef Tkadlec,
  • Alan Goldfarb,
  • Chenguang Wang,
  • Piotr Padlewski,
  • Stanislaw Barzowski,
  • Kyle Montgomery,
  • Ryan Stendall,
  • Jamie Tucker-Foltz,
  • Jack Stade,
  • T. Ryan Rogers,
  • Tom Goertzen,
  • Declan Grabb,
  • Abhishek Shukla,
  • Alan Givré,
  • John Arnold Ambay,
  • Archan Sen,
  • Muhammad Fayez Aziz,
  • Mark H. Inlow,
  • Hao He,
  • Ling Zhang,
  • Younesse Kaddar,
  • Ivar Ängquist,
  • Yanxu Chen,
  • Harrison K. Wang,
  • Kalyan Ramakrishnan,
  • Elliott Thornley,
  • Antonio Terpin,
  • Hailey Schoelkopf,
  • Eric Zheng,
  • Avishy Carmi,
  • Ethan D. L. Brown,
  • Kelin Zhu,
  • Max Bartolo,
  • Richard Wheeler,
  • Martin Stehberger,
  • Peter Bradshaw,
  • JP Heimonen,
  • Kaustubh Sridhar,
  • Ido Akov,
  • Jennifer Sandlin,
  • Yury Makarychev,
  • Joanna Tam,
  • Hieu Hoang,
  • David M. Cunningham,
  • Vladimir Goryachev,
  • Demosthenes Patramanis,
  • Michael Krause,
  • Andrew Redenti,
  • David Aldous,
  • Jesyin Lai,
  • Shannon Coleman,
  • Jiangnan Xu,
  • Sangwon Lee,
  • Ilias Magoulas,
  • Sandy Zhao,
  • Ning Tang,
  • Michael K. Cohen,
  • Orr Paradise,
  • Jan Hendrik Kirchner,
  • Maksym Ovchynnikov,
  • Jason O. Matos,
  • Adithya Shenoy,
  • Michael Wang,
  • Yuzhou Nie,
  • Anna Sztyber-Betley,
  • Paolo Faraboschi,
  • Robin Riblet,
  • Jonathan Crozier,
  • Shiv Halasyamani,
  • Shreyas Verma,
  • Prashant Joshi,
  • Eli Meril,
  • Ziqiao Ma,
  • Jérémy Andréoletti,
  • Raghav Singhal,
  • Jacob Platnick,
  • Volodymyr Nevirkovets,
  • Luke Basler,
  • Alexander Ivanov,
  • Seri Khoury,
  • Nils Gustafsson,
  • Marco Piccardo,
  • Hamid Mostaghimi,
  • Qijia Chen,
  • Virendra Singh,
  • Tran Quoc Khánh,
  • Paul Rosu,
  • Hannah Szlyk,
  • Zachary Brown,
  • Himanshu Narayan,
  • Aline Menezes,
  • Jonathan Roberts,
  • William Alley,
  • Kunyang Sun,
  • Arkil Patel,
  • Max Lamparth,
  • Anka Reuel,
  • Linwei Xin,
  • Hanmeng Xu,
  • Jacob Loader,
  • Freddie Martin,
  • Zixuan Wang,
  • Andrea Achilleos,
  • Thomas Preu,
  • Tomek Korbak,
  • Ida Bosio,
  • Fereshteh Kazemi,
  • Ziye Chen,
  • Biró Bálint,
  • Eve J. Y. Lo,
  • Jiaqi Wang,
  • Maria Inês S. Nunes,
  • Jeremiah Milbauer,
  • M. Saiful Bari,
  • Zihao Wang,
  • Behzad Ansarinejad,
  • Yewen Sun,
  • Stephane Durand,
  • Hossam Elgnainy,
  • Guillaume Douville,
  • Daniel Tordera,
  • George Balabanian,
  • Hew Wolff,
  • Lynna Kvistad,
  • Hsiaoyun Milliron,
  • Ahmad Sakor,
  • Murat Eron,
  • Andrew Favre,
  • Shailesh Shah,
  • Xiaoxiang Zhou,
  • Firuz Kamalov,
  • Sherwin Abdoli,
  • Tim Santens,
  • Shaul Barkan,
  • Allison Tee,
  • Robin Zhang,
  • Alessandro Tomasiello,
  • G. Bruno De Luca,
  • Shi-Zhuo Looi,
  • Vinh-Kha Le,
  • Noam Kolt,
  • Jiayi Pan,
  • Emma Rodman,
  • Jacob Drori,
  • Carl J. Fossum,
  • Niklas Muennighoff,
  • Milind Jagota,
  • Ronak Pradeep,
  • Honglu Fan,
  • Jonathan Eicher,
  • Michael Chen,
  • Kushal Thaman,
  • William Merrill,
  • Moritz Firsching,
  • Carter Harris,
  • Stefan Ciobâcă,
  • Jason Gross,
  • Rohan Pandey,
  • Ilya Gusev,
  • Adam Jones,
  • Shashank Agnihotri,
  • Pavel Zhelnov,
  • Mohammadreza Mofayezi,
  • Alexander Piperski,
  • David K. Zhang,
  • Kostiantyn Dobarskyi,
  • Roman Leventov,
  • Ignat Soroko,
  • Joshua Duersch,
  • Vage Taamazyan,
  • Andrew Ho,
  • Wenjie Ma,
  • William Held,
  • Ruicheng Xian,
  • Armel Randy Zebaze,
  • Mohanad Mohamed,
  • Julian Noah Leser,
  • Michelle X. Yuan,
  • Laila Yacar,
  • Johannes Lengler,
  • Katarzyna Olszewska,
  • Claudio Di Fratta,
  • Edson Oliveira,
  • Joseph W. Jackson,
  • Andy Zou,
  • Muthu Chidambaram,
  • Timothy Manik,
  • Hector Haffenden,
  • Dashiell Stander,
  • Ali Dasouqi,
  • Alexander Shen,
  • Bita Golshani,
  • David Stap,
  • Egor Kretov,
  • Mikalai Uzhou,
  • Alina Borisovna Zhidkovskaya,
  • Nick Winter,
  • Miguel Orbegozo Rodriguez,
  • Robert Lauff,
  • Dustin Wehr,
  • Colin Tang,
  • Zaki Hossain,
  • Shaun Phillips,
  • Fortuna Samuele,
  • Fredrik Ekström,
  • Angela Hammon,
  • Oam Patel,
  • Faraz Farhidi,
  • George Medley,
  • Forough Mohammadzadeh,
  • Madellene Peñaflor,
  • Haile Kassahun,
  • Alena Friedrich,
  • Rayner Hernandez Perez,
  • Daniel Pyda,
  • Taom Sakal,
  • Omkar Dhamane,
  • Ali Khajegili Mirabadi,
  • Eric Hallman,
  • Kenchi Okutsu,
  • Mike Battaglia,
  • Mohammad Maghsoudimehrabani,
  • Alon Amit,
  • Dave Hulbert,
  • Roberto Pereira,
  • Simon Weber,
  • Handoko,
  • Anton Peristyy,
  • Stephen Malina,
  • Mustafa Mehkary,
  • Rami Aly,
  • Frank Reidegeld,
  • Anna-Katharina Dick,
  • Cary Friday,
  • Mukhwinder Singh,
  • Hassan Shapourian,
  • Wanyoung Kim,
  • Mariana Costa,
  • Hubeyb Gurdogan,
  • Harsh Kumar,
  • Chiara Ceconello,
  • Chao Zhuang,
  • Haon Park,
  • Micah Carroll,
  • Andrew R. Tawfeek,
  • Stefan Steinerberger,
  • Daattavya Aggarwal,
  • Michael Kirchhof,
  • Linjie Dai,
  • Evan Kim,
  • Johan Ferret,
  • Jainam Shah,
  • Yuzhou Wang,
  • Minghao Yan,
  • Krzysztof Burdzy,
  • Lixin Zhang,
  • Antonio Franca,
  • Diana T. Pham,
  • Kang Yong Loh,
  • Joshua Robinson,
  • Abram Jackson,
  • Paolo Giordano,
  • Philipp Petersen,
  • Adrian Cosma,
  • Jesus Colino,
  • Colin White,
  • Jacob Votava,
  • Vladimir Vinnikov,
  • Ethan Delaney,
  • Petr Spelda,
  • Vit Stritecky,
  • Syed M. Shahid,
  • Jean-Christophe Mourrat,
  • Lavr Vetoshkin,
  • Koen Sponselee,
  • Renas Bacho,
  • Zheng-Xin Yong,
  • Florencia de la Rosa,
  • Nathan Cho,
  • Xiuyu Li,
  • Guillaume Malod,
  • Orion Weller,
  • Guglielmo Albani,
  • Leon Lang,
  • Julien Laurendeau,
  • Dmitry Kazakov,
  • Fatimah Adesanya,
  • Julien Portier,
  • Lawrence Hollom,
  • Victor Souza,
  • Yuchen Anna Zhou,
  • Julien Degorre,
  • Yiğit Yaln,
  • Gbenga Daniel Obikoya,
  • Rai Michael Pokorny,
  • Filippo Bigi,
  • M. C. Boscá,
  • Oleg Shumar,
  • Kaniuar Bacho,
  • Gabriel Recchia,
  • Mara Popescu,
  • Nikita Shulga,
  • Ngefor Mildred Tanwie,
  • Thomas C. H. Lux,
  • Ben Rank,
  • Colin Ni,
  • Matthew Brooks,
  • Alesia Yakimchyk,
  • Huanxu Quinn Liu,
  • Stefano Cavalleri,
  • Olle Häggström,
  • Emil Verkama,
  • Joshua Newbould,
  • Hans Gundlach,
  • Leonor Brito-Santana,
  • Brian Amaro,
  • Vivek Vajipey,
  • Rynaa Grover,
  • Ting Wang,
  • Yosi Kratish,
  • Wen-Ding Li,
  • Sivakanth Gopi,
  • Andrea Caciolai,
  • Christian Schroeder de Witt,
  • Pablo Hernández-Cámara,
  • Emanuele Rodolà,
  • Jules Robins,
  • Dominic Williamson,
  • Brad Raynor,
  • Hao Qi,
  • Ben Segev,
  • Jingxuan Fan,
  • Sarah Martinson,
  • Erik Y. Wang,
  • Kaylie Hausknecht,
  • Michael P. Brenner,
  • Mao Mao,
  • Christoph Demian,
  • Peyman Kassani,
  • Xinyu Zhang,
  • David Avagian,
  • Eshawn Jessica Scipio,
  • Alon Ragoler,
  • Justin Tan,
  • Blake Sims,
  • Rebeka Plecnik,
  • Aaron Kirtland,
  • Omer Faruk Bodur,
  • D. P. Shinde,
  • Yan Carlos Leyva Labrador,
  • Zahra Adoul,
  • Mohamed Zekry,
  • Ali Karakoc,
  • Tania C. B. Santos,
  • Samir Shamseldeen,
  • Loukmane Karim,
  • Anna Liakhovitskaia,
  • Nate Resman,
  • Nicholas Farina,
  • Juan Carlos Gonzalez,
  • Gabe Maayan,
  • Earth Anderson,
  • Rodrigo De Oliveira Pena,
  • Elizabeth Kelley,
  • Hodjat Mariji,
  • Rasoul Pouriamanesh,
  • Wentao Wu,
  • Ross Finocchio,
  • Ismail Alarab,
  • Joshua Cole,
  • Danyelle Ferreira,
  • Bryan Johnson,
  • Mohammad Safdari,
  • Liangti Dai,
  • Siriphan Arthornthurasuk,
  • Isaac C. McAlister,
  • Alejandro José Moyano,
  • Alexey Pronin,
  • Jing Fan,
  • Angel Ramirez-Trinidad,
  • Yana Malysheva,
  • Daphiny Pottmaier,
  • Omid Taheri,
  • Stanley Stepanic,
  • Samuel Perry,
  • Luke Askew,
  • Raúl Adrián Huerta Rodrguez,
  • Ali M. R. Minissi,
  • Ricardo Lorena,
  • Krishnamurthy Iyer,
  • Arshad Anil Fasiludeen,
  • Ronald Clark,
  • Josh Ducey,
  • Matheus Piza,
  • Maja Somrak,
  • Eric Vergo,
  • Juehang Qin,
  • Benjámin Borbás,
  • Eric Chu,
  • Jack Lindsey,
  • Antoine Jallon,
  • I. M. J. McInnis,
  • Evan Chen,
  • Avi Semler,
  • Luk Gloor,
  • Tej Shah,
  • Marc Carauleanu,
  • Pascal Lauer,
  • Tran Duc Huy,
  • Hossein Shahrtash,
  • Emilien Duc,
  • Lukas Lewark,
  • Assaf Brown,
  • Samuel Albanie,
  • Brian Weber,
  • Warren S. Vaz,
  • Pierre Clavier,
  • Yiyang Fan,
  • Gabriel Poesia Reis e Silva,
  • Long Tony Lian,
  • Marcus Abramovitch,
  • Xi Jiang,
  • Sandra Mendoza,
  • Murat Islam,
  • Juan Gonzalez,
  • Vasilios Mavroudis,
  • Justin Xu,
  • Pawan Kumar,
  • Laxman Prasad Goswami,
  • Daniel Bugas,
  • Nasser Heydari,
  • Ferenc Jeanplong,
  • Thorben Jansen,
  • Antonella Pinto,
  • Archimedes Apronti,
  • Abdallah Galal,
  • Ng Ze-An,
  • Ankit Singh,
  • Tong Jiang,
  • Joan of Arc Xavier,
  • Kanu Priya Agarwal,
  • Mohammed Berkani,
  • Gang Zhang,
  • Zhehang Du,
  • Benedito Alves de Oliveira Junior,
  • Dmitry Malishev,
  • Nicolas Remy,
  • Taylor D. Hartman,
  • Tim Tarver,
  • Stephen Mensah,
  • Gautier Abou Loume,
  • Wiktor Morak,
  • Farzad Habibi,
  • Sarah Hoback,
  • Will Cai,
  • Javier Gimenez,
  • Roselynn Grace Montecillo,
  • Jakub Łucki,
  • Russell Campbell,
  • Asankhaya Sharma,
  • Khalida Meer,
  • Shreen Gul,
  • Daniel Espinosa Gonzalez,
  • Xavier Alapont,
  • Alex Hoover,
  • Gunjan Chhablani,
  • Freddie Vargus,
  • Arunim Agarwal,
  • Yibo Jiang,
  • Deepakkumar Patil,
  • David Outevsky,
  • Kevin Joseph Scaria,
  • Rajat Maheshwari,
  • Abdelkader Dendane,
  • Priti Shukla,
  • Ashley Cartwright,
  • Sergei Bogdanov,
  • Niels Mündler,
  • Sören Möller,
  • Luca Arnaboldi,
  • Kunvar Thaman,
  • Muhammad Rehan Siddiqi,
  • Prajvi Saxena,
  • Himanshu Gupta,
  • Tony Fruhauff,
  • Glen Sherman,
  • Mátyás Vincze,
  • Siranut Usawasutsakorn,
  • Dylan Ler,
  • Anil Radhakrishnan,
  • Innocent Enyekwe,
  • Sk Md Salauddin,
  • Jiang Muzhen,
  • Aleksandr Maksapetyan,
  • Vivien Rossbach,
  • Chris Harjadi,
  • Mohsen Bahaloohoreh,
  • Claire Sparrow,
  • Jasdeep Sidhu,
  • Sam Ali,
  • Song Bian,
  • John Lai,
  • Eric Singer,
  • Justine Leon Uro,
  • Greg Bateman,
  • Mohamed Sayed,
  • Ahmed Menshawy,
  • Darling Duclosel,
  • Dario Bezzi,
  • Yashaswini Jain,
  • Ashley Aaron,
  • Murat Tiryakioglu,
  • Sheeshram Siddh,
  • Keith Krenek,
  • Imad Ali Shah,
  • Jun Jin,
  • Scott Creighton,
  • Denis Peskoff,
  • Zienab EL-Wasif,
  • Ragavendran P V,
  • Michael Richmond,
  • Joseph McGowan,
  • Tejal Patwardhan,
  • Hao-Yu Sun,
  • Ting Sun,
  • Nikola Zubić,
  • Samuele Sala,
  • Stephen Ebert,
  • Jean Kaddour,
  • Manuel Schottdorf,
  • Dianzhuo Wang,
  • Gerol Petruzella,
  • Alex Meiburg,
  • Tilen Medved,
  • Ali ElSheikh,
  • S. Ashwin Hebbar,
  • Lorenzo Vaquero,
  • Xianjun Yang,
  • Jason Poulos,
  • Vilém Zouhar,
  • Sergey Bogdanik,
  • Mingfang Zhang,
  • Jorge Sanz-Ros,
  • David Anugraha,
  • Yinwei Dai,
  • Anh N. Nhu,
  • Xue Wang,
  • Ali Anil Demircali,
  • Zhibai Jia,
  • Yuyin Zhou,
  • Juncheng Wu,
  • Mike He,
  • Nitin Chandok,
  • Aarush Sinha,
  • Gaoxiang Luo,
  • Long Le,
  • Mickaël Noyé,
  • Michał Perełkiewicz,
  • Ioannis Pantidis,
  • Tianbo Qi,
  • Soham Sachin Purohit,
  • Letitia Parcalabescu,
  • Thai-Hoa Nguyen,
  • Genta Indra Winata,
  • Edoardo M. Ponti,
  • Hanchen Li,
  • Kaustubh Dhole,
  • Jongee Park,
  • Dario Abbondanza,
  • Yuanli Wang,
  • Anupam Nayak,
  • Diogo M. Caetano,
  • Antonio A. W. L. Wong,
  • Maria del Rio-Chanona,
  • Dániel Kondor,
  • Pieter Francois,
  • Ed Chalstrey,
  • Jakob Zsambok,
  • Dan Hoyer,
  • Jenny Reddish,
  • Jakob Hauser,
  • Francisco-Javier Rodrigo-Ginés,
  • Suchandra Datta,
  • Maxwell Shepherd,
  • Thom Kamphuis,
  • Qizheng Zhang,
  • Hyunjun Kim,
  • Ruiji Sun,
  • Jianzhu Yao,
  • Franck Dernoncourt,
  • Satyapriya Krishna,
  • Sina Rismanchian,
  • Bonan Pu,
  • Francesco Pinto,
  • Yingheng Wang,
  • Kumar Shridhar,
  • Kalon J. Overholt,
  • Glib Briia,
  • Hieu Nguyen,
  • David Quod Soler Bartomeu,
  • Tony CY Pang,
  • Adam Wecker,
  • Yifan Xiong,
  • Fanfei Li,
  • Lukas S. Huber,
  • Joshua Jaeger,
  • Romano De Maddalena,
  • Xing Han Lù,
  • Yuhui Zhang,
  • Claas Beger,
  • Patrick Tser Jern Kon,
  • Sean Li,
  • Vivek Sanker,
  • Ming Yin,
  • Yihao Liang,
  • Xinlu Zhang,
  • Ankit Agrawal,
  • Li S. Yifei,
  • Zechen Zhang,
  • Mu Cai,
  • Yasin Sonmez,
  • Costin Cozianu,
  • Changhao Li,
  • Alex Slen,
  • Shoubin Yu,
  • Hyun Kyu Park,
  • Gabriele Sarti,
  • Marcin Briański,
  • Alessandro Stolfo,
  • Truong An Nguyen,
  • Mike Zhang,
  • Yotam Perlitz,
  • Jose Hernandez-Orallo,
  • Runjia Li,
  • Amin Shabani,
  • Felix Juefei-Xu,
  • Shikhar Dhingra,
  • Orr Zohar,
  • My Chiffon Nguyen,
  • Alexander Pondaven,
  • Abdurrahim Yilmaz,
  • Xuandong Zhao,
  • Chuanyang Jin,
  • Muyan Jiang,
  • Stefan Todoran,
  • Xinyao Han,
  • Jules Kreuer,
  • Brian Rabern,
  • Anna Plassart,
  • Martino Maggetti,
  • Luther Yap,
  • Robert Geirhos,
  • Jonathon Kean,
  • Dingsu Wang,
  • Sina Mollaei,
  • Chenkai Sun,
  • Yifan Yin,
  • Shiqi Wang,
  • Rui Li,
  • Yaowen Chang,
  • Anjiang Wei,
  • Alice Bizeul,
  • Xiaohan Wang,
  • Alexandre Oliveira Arrais,
  • Kushin Mukherjee,
  • Jorge Chamorro-Padial,
  • Jiachen Liu,
  • Xingyu Qu,
  • Junyi Guan,
  • Adam Bouyamourn,
  • Shuyu Wu,
  • Martyna Plomecka,
  • Junda Chen,
  • Mengze Tang,
  • Jiaqi Deng,
  • Shreyas Subramanian,
  • Haocheng Xi,
  • Haoxuan Chen,
  • Weizhi Zhang,
  • Yinuo Ren,
  • Haoqin Tu,
  • Sejong Kim,
  • Yushun Chen,
  • Sara Vera Marjanović,
  • Junwoo Ha,
  • Grzegorz Luczyna,
  • Jeff J. Ma,
  • Zewen Shen,
  • Dawn Song,
  • Cedegao E. Zhang,
  • Zhun Wang,
  • Gaël Gendron,
  • Yunze Xiao,
  • Leo Smucker,
  • Erica Weng,
  • Kwok Hao Lee,
  • Zhe Ye,
  • Stefano Ermon,
  • Ignacio D. Lopez-Miguel,
  • Theo Knights,
  • Anthony Gitter,
  • Namkyu Park,
  • Boyi Wei,
  • Hongzheng Chen,
  • Kunal Pai,
  • Ahmed Elkhanany,
  • Han Lin,
  • Philipp D. Siedler,
  • Jichao Fang,
  • Ritwik Mishra,
  • Károly Zsolnai-Fehér,
  • Xilin Jiang,
  • Shadab Khan,
  • Jun Yuan,
  • Rishab Kumar Jain,
  • Xi Lin,
  • Mike Peterson,
  • Zhe Wang,
  • Aditya Malusare,
  • Maosen Tang,
  • Isha Gupta,
  • Ivan Fosin,
  • Timothy Kang,
  • Barbara Dworakowska,
  • Kazuki Matsumoto,
  • Guangyao Zheng,
  • Gerben Sewuster,
  • Jorge Pretel Villanueva,
  • Ivan Rannev,
  • Igor Chernyavsky,
  • Jiale Chen,
  • Deepayan Banik,
  • Ben Racz,
  • Wenchao Dong,
  • Jianxin Wang,
  • Laila Bashmal,
  • Duarte V. Gonçalves,
  • Wei Hu,
  • Kaushik Bar,
  • Ondrej Bohdal,
  • Atharv Singh Patlan,
  • Shehzaad Dhuliawala,
  • Caroline Geirhos,
  • Julien Wist,
  • Yuval Kansal,
  • Bingsen Chen,
  • Kutay Tire,
  • Atak Talay Yücel,
  • Brandon Christof,
  • Veerupaksh Singla,
  • Zijian Song,
  • Sanxing Chen,
  • Jiaxin Ge,
  • Kaustubh Ponkshe,
  • Isaac Park,
  • Tianneng Shi,
  • Martin Q. Ma,
  • Joshua Mak,
  • Sherwin Lai,
  • Antoine Moulin,
  • Zhuo Cheng,
  • Zhanda Zhu,
  • Ziyi Zhang,
  • Vaidehi Patil,
  • Ketan Jha,
  • Qiutong Men,
  • Jiaxuan Wu,
  • Tianchi Zhang,
  • Bruno Hebling Vieira,
  • Alham Fikri Aji,
  • Jae-Won Chung,
  • Mohammed Mahfoud,
  • Ha Thi Hoang,
  • Marc Sperzel,
  • Wei Hao,
  • Kristof Meding,
  • Sihan Xu,
  • Vassilis Kostakos,
  • Davide Manini,
  • Yueying Liu,
  • Christopher Toukmaji,
  • Eunmi Yu,
  • Arif Engin Demircali,
  • Zhiyi Sun,
  • Ivan Dewerpe,
  • Hongsen Qin,
  • Roman Pflugfelder,
  • James Bailey,
  • Johnathan Morris,
  • Ville Heilala,
  • Sybille Rosset,
  • Zishun Yu,
  • Peter E. Chen,
  • Woongyeong Yeo,
  • Eeshaan Jain,
  • Sreekar Chigurupati,
  • Julia Chernyavsky,
  • Sai Prajwal Reddy,
  • Subhashini Venugopalan,
  • Hunar Batra,
  • Core Francisco Park,
  • Hieu Tran,
  • Guilherme Maximiano,
  • Genghan Zhang,
  • Yizhuo Liang,
  • Hu Shiyu,
  • Rongwu Xu,
  • Rui Pan,
  • Siddharth Suresh,
  • Ziqi Liu,
  • Samaksh Gulati,
  • Songyang Zhang,
  • Peter Turchin,
  • Christopher W. Bartlett,
  • Christopher R. Scotese,
  • Phuong M. Cao,
  • Ben Wu,
  • Jacek Karwowski,
  • Davide Scaramuzza

摘要

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding1, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity’s Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable but cannot be quickly answered by internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a marked gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.