A realistic web environment for developing autonomous agents. GPT-4 agent achieves 14.41% success rate vs 78.24% human performance.

A benchmark for measuring how well AI agents perform at machine learning engineering.

A benchmark for software engineering tasks.

A benchmark for evaluating AI systems on visual software engineering tasks with JavaScript.

A comprehensive benchmark to evaluate LLMs as agents (ICLR'24)

A benchmark for evaluating AI agents' performance in real-world settings with dynamic interaction.

A pioneering, cross-domain dataset for evaluating text-to-SQL models on large-scale databases.

A collaboratively built benchmark for measuring legal reasoning in large language models.

A benchmark for evaluating semantic equivalence between text snippets.

A large-scale dataset for benchmarking information retrieval systems.

A comprehensive framework for evaluating language models across various scenarios.

A benchmark for evaluating tool-augmented LLMs with 73 API tools and 314 dialogues.

A dataset of 7,787 grade-school level, multiple-choice science questions for advanced QA research.

A benchmark for testing physical situation reasoning with harder endings and longer contexts.

Evaluates functional correctness of code generation based on docstrings.

A comprehensive benchmark evaluating models on multiple choice questions across 57 subjects.

A rigorous benchmark for language understanding with eight challenging tasks.

A benchmark for evaluating model truthfulness with 817 questions across 38 categories.

A benchmark for evaluating emotional intelligence in Large Language Models.

A benchmark for evaluating cybersecurity risks and capabilities in Large Language Models.

A comprehensive benchmark for evaluating Speculative Decoding methods across diverse scenarios.

A comprehensive benchmark for evaluating LLMs & LMMs performance and resource consumption on mobile devices.

A comprehensive multimodal evaluation suite with 505 real-world tasks and diverse output formats.

A user-centric interactive platform for embodied AI and robotic task completion in simulated environments.

A diverse set of tasks designed to measure AI's general capabilities across reasoning, common sense, creativity, and many other areas.

A benchmark for evaluating models on various programming and software engineering tasks such as code completion, bug detection, and code translation.

A heterogeneous benchmark suite for evaluating the performance of retrieval models across various datasets and domains.

A benchmark for optical character recognition tasks, evaluating models on text detection, recognition, and end-to-end reading comprehension.

A dataset of 113K Wikipedia-based questions requiring multi-hop reasoning and supporting fact identification.

A large-scale QA dataset with 950K question-answer pairs from 662K Wikipedia and web documents.

A comprehensive benchmark for evaluating function/tool calling capabilities of language models across single-turn, multi-turn and multi-step scenarios.

A comprehensive benchmark suite for evaluating function calling, tool use, and agent capabilities of language models.

A comprehensive hallucination evaluation benchmark with 15K samples from real-world domains including finance and medicine.

A comprehensive benchmark for evaluating multi-LLM routing systems with 405k+ inference outcomes.

A stable benchmark for evaluating LLMs' tool learning capabilities with virtual API system and solvable queries.

A comprehensive framework for evaluating LLMs in task automation across decomposition, tool selection, and parameter prediction.

A benchmark evaluating LMMs' image understanding through text-to-image generation.

A benchmark for evaluating LLMs' ability to process and reason over super long contexts (100K+ tokens).

A benchmark for evaluating LLMs' ability to process and reason over super long contexts using a needle-in-a-haystack approach.

A comprehensive benchmark for evaluating long-context language models across retrieval, RAG, SQL, and more.

A comprehensive benchmark for evaluating long-context language models across seven diverse categories.

APPS is a dataset for evaluating model performance on Python programming tasks across three difficulty levels consisting of 1,000 at introductory, 3,000 at interview, and 1,000 at competition level. The dataset consists of an additional 5,000 training samples, for a total of 10,000 total samples. We evaluate on questions from the test split, which consists of programming problems commonly found in coding interviews.

Python coding benchmark with 1,140 diverse questions drawing on numerous python libraries.

Evaluate how well an LLM Agent is at computationally reproducing the results of a set of scientific papers.

Evaluates LLMs on class-level code generation with 100 tasks constructed over 500 person-hours. The study shows that LLMs perform worse on class-level tasks compared to method-level tasks.

Code generation benchmark with a thousand data science problems spanning seven Python libraries.

Measures the functional correctness of synthesizing programs from docstrings.

Measures the ability to synthesize short Python programs from natural language descriptions.

Machine learning tasks drawn from 75 Kaggle competitions.

Software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories.

SciCode tests the ability of language models to generate code to solve scientific research problems. It assesses models on 65 problems from mathematics, physics, chemistry, biology, and materials science.

Challenging programming problems from USA Computing Olympiad competitions, spread across four difficulty levels.

Tests whether AI agents can perform real-world time-consuming tasks on the web.

Proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs.

A dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website.

Benchmark for multimodal agents for open-ended tasks in real computer environments.

Evaluate sycophancy of language models across a variety of free-form text-generation tasks.

A benchmark that evaluates the susceptibility of LLMs to various jailbreak prompts.

Evaluates Large Language Models for cybersecurity risk to third parties, application developers and end users.

40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF competitions, chosen to be recent, meaningful, and spanning a wide range of difficulties.

Datasets containing 80, 500, 2000 and 10000 multiple-choice questions, designed to evaluate understanding across nine domains within cybersecurity

Evaluates Large Language Models for risky capabilities in cybersecurity.

CTF challenges covering web app vulnerabilities, off-the-shelf exploits, databases, Linux privilege escalation, password cracking and spraying. Demonstrates tool use and sandboxing untrusted model code.

Measure expertise in coding, cryptography (i.e. binary exploitation, forensics), reverse engineering, and recognizing security vulnerabilities. Demonstrates tool use and sandboxing untrusted model code.

Designed for analyzing cybersecurity incidents, which is comprised of two primary task categories: understanding and generation, with a further breakdown into 28 subcategories of tasks.

"Security Question Answering" dataset to assess LLMs' understanding and application of security principles. SecQA has "v1" and "v2" datasets of multiple-choice questions that aim to provide two levels of cybersecurity evaluation criteria. The questions were generated by GPT-4 based on the "Computer Systems Security: Planning for Success" textbook and vetted by humans.

Diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment.

Tests LLMs and LLM-augmented agents abilities to answer questions on scientific research workflows in domains like chemistry, biology, materials science, as well as more general science tasks

A dataset of 3,668 multiple-choice questions developed by a consortium of academics and technical consultants that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security.

A benchmark for evaluating AI's ability to solve challenging mathematics problems from AIME - a prestigious high school mathematics competition.

Measures how effectively language models solve realistic, linguistically rich math word problems suitable for grade-school-level mathematics.

Dataset of 12,500 challenging competition mathematics problems. Demonstrates fewshot prompting and custom scorers.

Extends the original GSM8K dataset by translating 250 of its problems into 10 typologically diverse languages.

Diverse mathematical and visual tasks that require fine-grained, deep visual understanding and compositional reasoning. Demonstrates multimodal inputs and custom scorers.

Dataset of natural, grade-school science multiple-choice questions (authored for human tests).

Suite of 23 challenging BIG-Bench tasks for which prior language model evaluations did not outperform the average human-rater.

Reading comprehension dataset that queries for complex, non-factoid information, and require difficult entailment-like inference to solve.

Evaluates reading comprehension where models must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them (such as addition, counting, or sorting).

DocVQA is a Visual Question Answering benchmark that consists of 50,000 questions covering 12,000+ document images. This implementation solves and scores the "validation" split.

Evaluates the ability to follow a set of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword of AI at least 3 times. Demonstrates custom scoring.

Multimodal questions from college exams, quizzes, and textbooks, covering six core disciplinestasks, demanding college-level subject knowledge and deliberate reasoning. Demonstrates multimodel inputs.

Evaluating models on multistep soft reasoning tasks in the form of free text narratives.

NIAH evaluates in-context retrieval ability of long context LLMs by testing a model's ability to extract factual information from long-context inputs.

Evaluating models on the task of paraphrase detection by providing pairs of sentences that are either paraphrases or not.

Measure physical commonsense reasoning (e.g. "To apply eyeshadow without a brush, should I use a cotton swab or a toothpick?")

Reading comprehension tasks collected from the English exams for middle and high school Chinese students in the age range between 12 to 18.

Set of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage.

V*Bench is a visual question & answer benchmark that evaluates MLLMs in their ability to process high-resolution and visually crowded images to find and focus on small details.

Set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations.

LLM benchmark featuring an average data length surpassing 100K tokens. Comprises synthetic and realistic tasks spanning diverse domains in English and Chinese.

AGIEval is a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.

A safety benchmark evaluating language models against risk categories derived from government regulations and company policies.

Measure question answering with commonsense prior knowledge.

Challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry (experts at PhD level in the corresponding domains reach 65% accuracy).

Humanity's Last Exam (HLE) is a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. Humanity's Last Exam consists of 3,000 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading.

An enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options.

Questions and answers from the Ordinary National Educational Test (O-NET), administered annually by the National Institute of Educational Testing Service to Matthayom 6 (Grade 12 / ISCED 3) students in Thailand. The exam contains six subjects: English language, math, science, social knowledge, and Thai language. There are questions with multiple-choice and true/false answers. Questions can be in either English or Thai.

Biomedical question answering (QA) dataset collected from PubMed abstracts.

A benchmark that evaluates the ability of language models to answer short, fact-seeking questions.

Measure whether a language model is truthful in generating answers to questions using questions that some humans would answer falsely due to a false belief or misconception.

Dataset with 250 safe prompts across ten prompt types that well-calibrated models should not refuse, and 200 unsafe prompts as contrasts that models, for most applications, should refuse.