This guide covers best practices for building environments with `verifiers` and using them to train and evaluate LLMs. It is downloaded automatically using the setup script below (which has likely already been run if you're reading this). See `environments/AGENTS.md` for more details.
Sign in to like and favorite skills
This guide covers best practices for building environments with
verifiersenvironments/AGENTS.mdVerifiers is our library for creating environments to train and evaluate LLMs.
Environments contain everything required to run and evaluate a model on a particular task:
Environments can be used for training models with reinforcement learning (RL), evaluating capabilities, generating synthetic data, experimenting with agent harnesses, and more.
Verifiers is tightly integrated with the Environments Hub, as well as our training framework prime-rl and our Hosted Training platform.
Ensure you have
uvprime# install uv curl -LsSf https://astral.sh/uv/install.sh | sh # install the prime CLI uv tool install prime # log in to the Prime Intellect platform prime login
To set up a new workspace for developing environments, do:
# ~/dev/my-lab prime lab setup
This sets up a Python project if needed (with
uv initverifiersuv add verifiersconfigs/ ├── endpoints.py # OpenAI-compatible API endpoint configuration └── lab/ # Example configs for Hosted Training environments/ └── AGENTS.md # Documentation for AI coding agents AGENTS.md # Top-level documentation for AI coding agents CLAUDE.md # Claude-specific pointer to AGENTS.md
Alternatively, add
verifiersuv add verifiers && prime lab setup --skip-install
Environments built with Verifiers are self-contained Python modules. To initialize a fresh environment template, do:
prime env init my-env # creates a new template in ./environments/my_env
This will create a new module called
my_envenvironments/my_env/ ├── my_env.py # Main implementation ├── pyproject.toml # Dependencies and metadata └── README.md # Documentation
Environment modules should expose a
load_environment# my_env.py import verifiers as vf def load_environment(dataset_name: str = 'gsm8k') -> vf.Environment: dataset = vf.load_example_dataset(dataset_name) # 'question' async def correct_answer(completion, answer) -> float: completion_ans = completion[-1]['content'] return 1.0 if completion_ans == answer else 0.0 rubric = Rubric(funcs=[correct_answer]) env = vf.SingleTurnEnv(dataset=dataset, rubric=rubric) return env
To install the environment module into your project, do:
prime env install my-env # installs from ./environments/my_env
To install an environment from the Environments Hub into your project, do:
prime env install primeintellect/math-python
To run a local evaluation with any OpenAI-compatible model, do:
prime eval run my-env -m gpt-5-nano # run and save eval results locally
Evaluations use Prime Inference by default; configure your own API endpoints in
./configs/endpoints.pyView local evaluation results in the terminal UI:
prime eval tui
In the TUI, press
ccTo publish the environment to the Environments Hub, do:
prime env push --path ./environments/my_env
To run an evaluation directly from the Environments Hub, do:
prime eval run primeintellect/math-python
Environments — Create datasets, rubrics, and custom multi-turn interaction protocols.
Evaluation - Evaluate models using your environments.
Training — Train models in your environments with reinforcement learning.
Development — Contributing to verifiers
API Reference — Understanding the API and data structures
FAQs - Other frequently asked questions.