A suite of curated benchmarks for evaluating Large Language Models (LLMs).
- ✅ Validated: Benchmarks implemented in this suite are required to have reproduced published results.
- ⚖️ Legal: The data is legal to exist and to use. Some datasets are subject to third-party licenses.
- 📐 Metrics: Bayesian loglikelihood accuracy with a length correction fitted from the data (paper), BitsPerByte on answers with corpus- and prior-level variants, and the distributional-correctness score (Burns, 2025).
- 🤷 Abstention variants:
ARC_IDK,MMLU_IDK,MMLU_PRO_IDK,PIQA_IDKandGoldenSwag_IDKadd a scored "I don't know" answer, so a model can decline instead of guessing. These tasks and the loglikelihood metrics above need the Python API and a model served by vLLM; the CLI, including--self-host, cannot run them yet. - 🌍 European languages: EllaMind translations of ARC, CommonsenseQA, GPQA, GSM8K, HellaSwag, PIQA and more, in German, French, Spanish and Italian.
- 📚 250+ tasks: including BigCodeBench, GoldenSwag, LBPP and MultiPL-E. Browse the benchmarks source directory.
The quickstart runs a small model on your machine with llama.cpp. No API key is needed. It requires uv, which also provides a suitable Python.
Install eval-framework, and llama.cpp, which provides the llama-server program that serves the model:
uv tool install git+https://github.com/Aleph-Alpha-Research/eval-framework
brew install llama.cpp # macOS and Linux with HomebrewOn other platforms, install llama.cpp with the llama.cpp install guide. To work from a clone of the repository, see the installation guide.
Then run a benchmark:
eval_framework \
--self-host \
--llm-args model_name=Qwen/Qwen3-0.6B-GGUF \
--task-name MMLU_COT \
--task-subjects abstract_algebra \
--num-samples 10 \
--max-tokens 1024 \
--output-dir ./eval_resultsWhat happens here?
--self-hoststarts the model with llama.cpp and stops it when the run ends.- It downloads the MMLU dataset and asks the model 10 abstract-algebra questions (the first run also downloads the model).
- It writes the scores to
aggregated_results.jsonand every prompt and answer tooutput.jsonl, in a run folder under./eval_results.
Example output (from aggregated_results.json, shortened):
{
"Average Accuracy Completion": 0.2,
"ErrorFreeRatio Accuracy Completion": 1.0
}Scores are fractions between 0 and 1. This 0.6B model is only a smoke test that the pipeline works: it often runs out of tokens or misses the required answer format, so expect a low score.
For all options, see the CLI Usage Guide. For full documentation, visit our Docs Page.
Hosted or self-hosted, any OpenAI-compatible endpoint works:
# OpenAI
export OPENAI_API_KEY=...
eval_framework --llm-args model_name=gpt-4o-mini \
--task-name MMLU_COT --task-subjects abstract_algebra --num-samples 10
# Your own server, e.g. vLLM
eval_framework --llm-args model_name=<served model> base_url=http://localhost:8000/v1 \
--task-name MMLU_COT --task-subjects abstract_algebra --num-samples 10eval-framework can also be called from your own Python code instead of the command line. This is needed for custom
model classes and for the loglikelihood metrics, which also require a model served by vLLM. Add it to your own project
with uv add eval-framework.
from pathlib import Path
from eval_framework.llm.openai import OpenAIModel
from eval_framework.main import main
from eval_framework.tasks.eval_config import EvalConfig
config = EvalConfig(
llm_class=OpenAIModel,
llm_args={"model_name": "gpt-4o-mini"}, # plus e.g. "base_url" for any OpenAI-compatible endpoint
task_name="MMLU_COT",
task_subjects=["abstract_algebra"],
num_samples=10, # per subject; None evaluates the whole split
output_dir=Path("outputs"),
)
results = main(llm=config.llm_class(**config.llm_args), config=config)If you use eval-framework in your research, please cite:
@software{eval_framework,
author={Aleph Alpha Research},
title={Aleph Alpha Eval Framework},
year={2026},
version = {x.y.z},
url={https://github.com/Aleph-Alpha-Research/eval-framework}
}This project is licensed under the Apache License 2.0.
The constituent tasks' datasets can be subject to specific, more restrictive license terms by third parties. By using eval-framework you agree to adhere to and be bound by such license terms in the individual case.
This project has received funding from the European Union’s Digital Europe Programme under grant agreement No. 101195233 (OpenEuroLLM).
The contents of this publication are the sole responsibility of the OpenEuroLLM consortium and do not necessarily reflect the opinion of the European Union.

