Best tools for evaluating agents (2026)
Evaluation tools for scoring agent outputs, catching regressions and comparing prompts and models. We list 121 tools for evaluating agents below, each with a link-checked page, pricing as the maker states it and alternatives.
Updated
121 tools: 59 open source, 8 source-available, 27 free or freemium, 12 paid, usage-based or enterprise, 15 pricing on the maker's site.
Tracing and observability (46)
Openlayer MCP
Openlayer's MCP server, letting coding agents pull evaluation results, failing tests and monitoring data from Openlayer so they can debug and improve AI systems from the IDE.
Added
Beacon
An open-source memory layer that captures traces from coding agents across harnesses and turns them into self-improving memory and visibility into what your agents did.
Added
OpenOSINT
An open-source OSINT engine with about 20 passive investigation tools usable from a CLI, a web UI or an MCP server for AI assistants, with hosted monitoring and due-diligence plans.
Added
Agent Barn
Agent Barn is an open-source (Apache-2.0) control plane for running a fleet of AI agents on your own Kubernetes: create, configure, run and monitor agents across Slack, Microsoft Teams and Telegram from one place.
Added
sem
Semantic layer on top of Git that diffs, blames and traces impact by function and class instead of by line, with an MCP server for coding agents.
Added
Opik
Comet's open-source platform for tracing, evaluating and monitoring LLM apps, RAG systems and agent workflows, self-hosted or in the cloud.
Added
Latitude
Observability platform for AI agents that captures traces, groups failures into signals and verifies fixes against real traces.
Added
Laminar
Open-source observability platform for AI agents with OpenTelemetry tracing, plain-English behavior tracking and alerts.
Added
Pezzo
Cloud-native LLMOps platform for prompt design, versioning, delivery, observability and cost monitoring.
Added
Future AGI
Open-source platform for evaluating, tracing, simulating and guarding LLM and AI agent applications, self-hostable or cloud.
Added
OpenInference
OpenTelemetry-based instrumentation that traces LLM and agent framework calls so any observability backend can show them.
Added
Judgeval
Open-source SDK for tracing agents and scoring them with agent judges, so failures in production can be found and fixes checked.
Added
Arize AI
Agent observability and evaluation platform with tracing, evals and experiments for improving AI agents.
Added
Confident AI
Enterprise platform for AI evals and observability that gives teams one standard for measuring and monitoring AI quality.
Added
Jarvis Registry
Open-source MCP and A2A agent gateway that connects copilots and agents to enterprise tools with identity, access control and observability.
Added
Zeron
Native control plane for running and monitoring Claude Code, Codex, Cursor, Devin and other coding agents locally.
Added
Workflow SDK
Vercel's TypeScript SDK for building durable, reliable and observable apps and AI agents.
Added
ADR
Uber's Agentic AI Detection and Response system that secures employee and customer-facing AI agents with observability and threat detection.
Added
Sentry MCP
Sentry's MCP server and CLI that give coding agents access to Sentry issues, errors and traces for debugging.
Added
UptimeRobot MCP
UptimeRobot's MCP server for creating monitors, triaging incidents and querying uptime stats for sites, APIs and servers.
Added
Aimdoc
AI agents that connect to your product and systems to deliver work for your customers, metered and monitored.
Added
Auquan
AI agents for institutional finance that handle deal screening, IC memos, portfolio monitoring and LP reporting.
Added
Axiom MCP
Axiom's MCP server for listing datasets and schemas and running APL queries for log exploration and monitoring.
Added
Bland AI
Enterprise voice AI platform for building, deploying and monitoring phone agents with self-hosted models.
Added
CodeRabbit
AI code review for pull requests, with PR triage, change explanations and continuous security monitoring.
Added
Frase MCP
Frase's MCP server with SEO, GEO and AI visibility tools to research, write, optimize, publish and monitor content.
Added
Kadoa
AI agents that build, monitor and maintain web data pipelines producing verified datasets for investors.
Added
Picsellia MCP Server
Official MCP server for Picsellia's computer vision MLOps platform, giving AI assistants access to datasets, experiments, models, deployments and monitoring.
Added
MLflow
Open-source platform for tracing, evaluating and managing agents, LLM apps and ML models.
Added
Arize Phoenix
Open-source tracing and evaluation for LLM apps.
Added
Agenta
Open-source LLMOps platform for prompt management, evaluation and observability.
Added
AgentOps
Monitoring and session replay for AI agents.
Added
Braintrust
Evals and observability platform for AI products.
Added
Helicone
Open-source LLM observability and gateway.
Added
Kore.ai Autoloop
Optimization engine in Kore.ai Agent Platform Artemis that evaluates agents against business goals, diagnoses failures from traces, applies targeted fixes, and re-verifies; GA Oct 2026.
Added
Langfuse
Open-source LLM observability, tracing and evals.
Added
LangSmith
Tracing, evals and monitoring for LLM apps and agents.
Added
Respan
Observability and evaluation platform for LLM apps and agents.
Added
W&B Weave
Weights & Biases toolkit for tracing and evaluating agents.
Added
agentgateway
Proxy for AI agents and MCP servers with security and observability.
Added
Cloudflare MCP Servers
Cloudflare's collection of MCP servers for Workers, observability, DNS analytics and other Cloudflare services.
Added
abtop
htop-style monitor for Claude Code and Codex sessions and token use.
Added
New Relic MCP
New Relic's MCP server for querying metrics, logs, traces and alerts through an agent.
Added
Blockscope
AI on-chain investigator that decodes transactions, profiles wallets and traces stolen funds across 8 chains.
Added
TradeOS AI
Build AI trading agents from your own technical-analysis rules that monitor markets 24/7 and flag setups (stocks, forex, commodities; crypto secondary).
Added
Wayfinder
Autonomous agents for cross-chain execution that you deploy, monitor and earn with, no scripting.
Added
Evals and benchmarks (23) See the full list →
Tov
Tov is an experimental programming language designed to be written by coding agents. It uses TypeScript syntax minus a few traps (no any, null or loose equality, separate int and float types, and failing calls marked with try), so agents already know how to write it, and its compiler returns machine-readable fixes the agent can apply after every edit. Programs compile to small, fast native binaries, with published benchmarks against Rust, Bun and Node. Agents can learn it from the site's llms.txt.
Added
Tracely
Tracely is open-source, trace-native CI/CD for AI agents: it grades every production trace with evaluators, clusters failures into issues, freezes bad runs into replayable test cases and blocks pull requests that would reintroduce them.
Added
SWE-bench
Benchmark that tests whether language models and coding agents can resolve real GitHub issues, with multimodal and verified variants.
Added
Kiln
Free desktop app and open-source library for building AI systems with evals, RAG, agents, fine-tuning and synthetic data in one workbench.
Added
OSWorld
Benchmark for multimodal computer-use agents on open-ended tasks in real desktop environments.
Added
MCPJam Inspector
Open-source testing and evals platform for MCP servers, MCP apps and ChatGPT apps across many clients and models.
Added
BrowserGym
ServiceNow's Gym environment for building, testing and evaluating web agents across browser benchmarks.
Added
Coval
Testing and evaluation platform for voice agents, with simulations, production evals and human QA.
Added
Agents CLI
Google's CLI and skills that teach coding assistants to build, evaluate and deploy agents on Google Cloud and the Gemini platform.
Added
EvoAgentX
Framework for building, evaluating and automatically evolving LLM agents and agentic workflows.
Added
Qwen-UI-Agent
Tongyi foundation GUI agent model that operates mobile, computer and browser interfaces, trained and evaluated on real devices.
Added
Hume AI
Voice AI company focused on emotionally intelligent speech, with voice-native evaluation and human judgment tooling for voice agents.
Added
Apache Maka
Incubating Apache agent harness and workspace that records everything it does and publishes benchmarked run results.
Added
tRPC-Agent-Go
Go framework for production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation and OpenTelemetry.
Added
AIDE ML
Tree-search agent that drafts, debugs and benchmarks machine-learning code until a target metric is optimized.
Added
Agent Memory Leaderboard
Public leaderboard and open evaluation protocol, run by a group of universities and labs, for comparing long-term memory systems for agents in recurring submission rounds.
Added
FiftyOne MCP Server
Voxel51's MCP server and skills that let AI agents load, explore, curate and evaluate computer vision datasets and run FiftyOne operators and the App from natural language.
Added
YRIKKA Text-to-Model
YRIKKA's ORION workflow that turns a plain-language mission description and a few unlabeled images into generated training data, candidate models and an evaluated, selected vision model.
Added
TruLens
Evaluation and tracking for LLM experiments and AI agents.
Added
OpenLIT
Open-source platform to trace, evaluate and guard agents and LLM apps.
Added
Oumi
Open platform for fine-tuning, evaluating and deploying open agentic LLMs and VLMs.
Added
fast-agent
Framework to build, run and evaluate agents with strong MCP, ACP and A2A support.
Added
Giza
Self-custodial autonomous agents that evaluate markets and execute on-chain capital strategies (Giza World, ARMA).
Added
Guardrails and policies (6)
OpenAPPA
OpenAPPA is a deterministic guardrail that sits between an AI agent and its tools and tracks how data flows across tool calls, instead of asking a second model to judge each call. Policies decide whether untrusted or sensitive data may reach a given action, which blocks data exfiltration caused by prompt injection or hallucination while letting agents finish their tasks. It plugs into Claude Code, Pi, OpenCode, kAgent and Archestra, comes with reusable policy "batteries", validation, observability and reporting, and is backed by a NeurIPS workshop paper with published benchmarks. It is in preview as an open RFC.
Added
LiteLLM
Open-source AI gateway and Python SDK that calls 100+ LLM providers in the OpenAI format, with cost tracking, guardrails, load balancing and logging.
Added
coucou
Mac notch and phone companion that monitors your Claude Code, Codex, Cursor and Gemini CLI sessions and lets you approve permissions on the go.
Added
Metorial
Open-source identity and access layer that connects AI agents to 1,200+ integrations via MCP, CLI or API with auth, permissions and observability.
Added
Altmetric MCP
Altmetric's MCP server for tracking research attention across news, policy documents and social media.
Added
NVIDIA Open Agent Safety Platform
Reference stack pairing OpenShell with NVIDIA Sentry on BlueField-4 DPUs for hardware-level agent monitoring.
Added
Penetration and security testing (5)
AI Beat
An open-source AI red-teaming suite: PromptBeat attacks models and prompts, and AgentBeat runs black-box security evaluations against HTTP-exposed agents.
Added
Rogue
Agent evaluator and red-team tool that probes your agent with test conversations and attacks and reports pass or fail against your policies.
Added
Almanax
AI security engineer that finds vulnerabilities in applications and smart contracts and reviews pull requests; acquired by depthfirst (June 2026).
Added
Octane
Agentic security analysis that finds exploitable vulnerabilities in mission-critical code, including DeFi smart contracts.
Added
Sherlock AI
Sherlock's AI auditor for smart-contract vulnerabilities.
Added
Sandboxes and secure execution (4)
AgentEnv Framework
AgentEnv Framework is Scale AI's open-source Python SDK and CLI for building RL environments for agents: containerized worlds with realistic apps and data behind an MCP gateway, plus tasks (as DAGs) that grade any agent in any sandbox.
Added
ToolHive
Stacklok platform that runs MCP servers in isolated containers with per-request identity, access policy and observability.
Added
CoreWeave Forge
One product to run, observe, evaluate and post-train agents (W&B, OpenPipe, sandboxes).
Added
Cua
Open-source infrastructure for computer-use agents: sandboxes, drivers across operating systems and benchmarks.
Added
More tools (37)
Superlog
Superlog runs AI agents on routine engineering work around production. You set up automations with triggers and connectors, for example rating each new Sentry issue, posting a plain-language summary to Slack and opening a pull request with the fix, comparing error rates and latency against their normal range every hour, adding missing logs and spans to hard-to-debug code paths, or answering community questions from your code and docs. You choose the model and agent harness and bring your own tokens or subscriptions. The core is open source (Apache-2.0) with a hosted version.
Added
Reticle
Verification layer for AI coding agents: a dev-only SDK inside your web or desktop app lets the agent drive real flows and read network calls, app state and console, then returns a pass, fail or cannot-tell verdict with the file and line to fix; saved flows replay without an AI model and can gate CI.
Added
agent-qa (Vostride)
A self-improving AI QA agent with persistent memory that tests web, Android and iOS apps from plain-English test steps, for developers and coding agents.
Added
Firedrill
A simulation platform for testing AI agents before launch: it runs your agent through realistic scenarios with simulated users and tools, scores results against checks or rubrics and shows where it fails. Includes a remote MCP server.
Added
Bottally
Bottally is an AI marketing agent for SEO and AEO that runs on autopilot. Once you connect your blog (WordPress, Webflow, Ghost, Shopify, Wix, Notion, Framer, Next.js or a webhook), it researches keywords and competitor gaps, then writes and publishes one researched article a day with images. It also offers an opt-in backlink exchange network and tracks weekly whether ChatGPT, Google AI Mode, Google AI Overviews and Perplexity recommend your business.
Added
Moli
Lightweight Rust headless browser for AI agents that fetches pages, searches and extracts content, usable via CLI, CDP or WebDriver.
Added
Manifest
Open-source LLM gateway that routes each request from agents and apps to the right model across API keys, subscriptions and local models, with spend tracking.
Added
Sparrow
Open-source tool for structured data extraction from documents and images with ML, LLMs and vision LLMs, plus agentic workflows.
Added
golutra
Multi-agent workspace that turns Codex, Claude Code and OpenClaw CLIs into one orchestrated team with parallel execution and task tracking.
Added
Antigravity SDK
Google's Python SDK for building stateful AI agents powered by Antigravity and Gemini, abstracting the agent loop and runtime.
Added
webclaw
Rust web extraction tool that turns sites into clean Markdown or JSON for agents, as a CLI, MCP server and REST API, with a hosted API.
Added
Kagi MCP
Official Kagi MCP server with search and page-extraction tools backed by the Kagi API.
Added
OpenZeppelin MCP
OpenZeppelin's official MCP servers that let agents generate secure smart contracts from audited OpenZeppelin libraries for Solidity, Cairo, Stellar and more.
Added
PandaDoc MCP
PandaDoc's MCP server for creating, sending, signing and tracking documents from AI assistants.
Added
AfterShip MCP
AfterShip's MCP server for tracking packages across 1,300+ carriers with real-time status.
Added
Extend MCP
Extend's MCP server for turning documents into structured data: parse, extract, classify, split and fill PDF forms.
Added
LlamaParse
LlamaIndex's document processing platform that parses PDFs, scans and Office files into markdown or JSON and extracts, classifies and indexes them for agents.
Added
SE Ranking MCP
SE Ranking's MCP server for keyword, backlink, SERP, audit, rank-tracking and AI search visibility data.
Added
CVAT AI Agents
CVAT feature that runs your own detection, segmentation or tracking model as an agent on your infrastructure and feeds its auto-annotations back into CVAT tasks.
Added
Etherscan MCP
Etherscan's official MCP server for read-only onchain data across EVM chains, such as balances, transactions, token transfers, contracts and logs, usable with or without an API key.
Added
EyePop.ai
SDK-first vision platform where you define what to detect and it builds detection, tracking, OCR and VLM pipelines, with a Video Agent and agent skills, deployable to cloud, edge or on-prem.
Added
TwelveLabs Jockey MCP
TwelveLabs' hosted MCP server for Jockey, a video-understanding agent that searches, tracks entities and answers questions across your videos and images with clip citations.
Added
Galileo Agent Leaderboard
Galileo's leaderboard that ranks how well LLMs perform as agents on tool use and multi-step tasks.
Added
Basalt
Captures customer behavior so your AI agent can learn and improve from real interactions.
Added
Hamming AI
Testing and QA platform for voice and chat agents that generates scenarios and replays real calls.
Added
Beads
Issue-tracking memory for coding agents so they can keep track of long-running work.
Added
Yao
Workspace for running and tracking all your agents and tasks across devices.
Added
Unstract
LLM-powered platform for extracting structured data from documents via APIs and ETL pipelines.
Added
Mission Control
Self-hosted control plane for dispatching tasks to agents, reviewing runs and tracking spend.
Added
Accordio AI
Adds time tracking, contract signing, invoicing and payments to Claude through a free MCP connector.
Added
aixbt
Autonomous agent tracking crypto narratives and mindshare from X.
Added
ChainGPT
Suite of crypto AI tools: chatbot, smart-contract generator and auditor.
Added
Friends & Fables
AI game master for tabletop-style RPGs that plays with groups on Discord and the web, tracking characters, rules and world lore.
Added
Laika AI
Polymetric by Laika AI: AI research, whale tracking and a screener for Polymarket prediction markets (formerly a Web3 research extension).
Added
Nora
Deep-reasoning AI coding agent for building and securing Web3 apps and smart contracts.
Added
Tavily
Search and extraction API tuned for agents.
Added
Qodo
AI code review and testing agents that work in the IDE, the terminal and pull requests to catch issues before merge.
Added
GitHub repos (73)
| Repo | GitHub stars | Licence | Last push |
|---|---|---|---|
| LiteLLM ↗ | 61,465 | MIT | 2026-10-21 |
| Cua ↗ | 29,966 | MIT | 2026-10-21 |
| MLflow ↗ | 28,418 | Apache License 2.0 | 2026-10-21 |
| Beads ↗ | 27,943 | MIT | 2026-10-21 |
| Opik ↗ | 22,700 | Apache License 2.0 | 2026-10-21 |
| Moli ↗ | 19,349 | — | 2026-10-21 |
| CVAT AI Agents ↗ | 16,943 | MIT | 2026-10-21 |
| Qodo ↗ | 13,415 | MIT | 2026-10-21 |
| Arize Phoenix ↗ | 11,828 | Elastic License 2.0 (source-available) | 2026-10-21 |
| Oumi ↗ | 9,400 | Apache License 2.0 | 2026-10-21 |
| Yao ↗ | 8,134 | Modified Apache License 2.0 with commercial-use conditions | 2026-10-19 |
| Manifest ↗ | 7,595 | MIT | 2026-10-21 |
| Unstract ↗ | 7,285 | GNU AGPL v3.0 | 2026-10-21 |
| Mission Control ↗ | 6,358 | MIT | 2026-09-22 |
| Agents CLI ↗ | 6,137 | Apache License 2.0 | 2026-10-20 |
Top 15 of 73 by stars. Stars and licence come from each repo, checked when the listing was last verified.
Related tasks
Popular with: AI engineers · QA engineers · Related searches: Free AI agent evaluation tools
Tools are matched to tasks by category, use cases and what each tool says it does. Missing one? Submit it free.