Judgeval
Open-source SDK for tracing agents and scoring them with agent judges, so failures in production can be found and fixes checked.
Updated
✨ Try AgentScout's free AI image generator →Good for: building AI agents, evaluating agents, monitoring agents in production
Use cases
About Judgeval
Judgeval is listed in our evals & observability category and is aimed at builders. Open-source SDK for tracing agents and scoring them with agent judges, so failures in production can be found and fixes checked. Pricing, as the maker states it: Open source + cloud. Maturity: Active. We last checked the official link on 2026-10-18.
Judgeval at a glance
- Category: Evals & observability
- Built for: Developers and teams building with AI agents
- Pricing: Open source + cloud (as the maker states it)
- Licence: Apache License 2.0 (OSI-approved open source)
- GitHub: JudgmentLabs/judgeval · 1,068 stars (as of Oct 21, 2026)
- Latest release: v1.3.3, Oct 1, 2026
- Last commit: Oct 1, 2026
- Official site: judgmentlabs.ai
- Listed on AgentScout: Oct 18, 2026
Judgeval FAQ
What is Judgeval?
Open-source SDK for tracing agents and scoring them with agent judges, so failures in production can be found and fixes checked.
How much does Judgeval cost?
Judgeval's pricing, as the maker states it: Open source + cloud. Open-source tools are free to use, though you may still pay for model API usage or a hosted plan. Check the official site for current plans.
Is Judgeval open source?
Yes. Judgeval's GitHub repository is published under the Apache License 2.0, according to the licence file in the repo.
Where can I find Judgeval's source code?
On GitHub at JudgmentLabs/judgeval, which had 1,068 stars as of Oct 21, 2026. The latest release is v1.3.3 (Oct 1, 2026).
What are alternatives to Judgeval?
Similar tools include MLflow, Arize Phoenix, Langfuse, Tracely, Kore.ai Autoloop and AgentEnv Framework.
Facts from our link-checked listing, the GitHub repository and the official site. See something wrong? Suggest an edit.
Judgeval ratings and reviews
Be the first to rate Judgeval: tap 1 to 5 stars below. A written review is optional.
Judgeval alternatives
Other evals & observability tools in our directory. See all evals & observability tools →
MLflow
Open-source platform for tracing, evaluating and managing agents, LLM apps and ML models.
Arize Phoenix
Open-source tracing and evaluation for LLM apps.
Langfuse
Open-source LLM observability, tracing and evals.
Tracely
Tracely is open-source, trace-native CI/CD for AI agents: it grades every production trace with evaluators, clusters failures into issues, freezes bad runs into replayable test cases and blocks pull requests that would reintroduce them.
Kore.ai Autoloop
Optimization engine in Kore.ai Agent Platform Artemis that evaluates agents against business goals, diagnoses failures from traces, applies targeted fixes, and re-verifies; GA Oct 2026.
AgentEnv Framework
AgentEnv Framework is Scale AI's open-source Python SDK and CLI for building RL environments for agents: containerized worlds with realistic apps and data behind an MCP gateway, plus tasks (as DAGs) that grade any agent in any sandbox.
Popular searches with this tool
🏆 See also: Best AI Agent Observability & Eval Tools · All best-of lists
Badge for makers
Built Judgeval? Add this badge to your README or site. It links back to this page. About badges
