Free AI agent evaluation tools (2026)

AI agent evaluation tools that you can use without paying: open-source projects plus tools with a free plan. Here are the top 25 of 83 from the AgentScout directory.

Updated October 21, 2026

How we picked and ranked: tools are matched by category, the pricing each maker states and what each tool says it does. They are ranked by GitHub stars as recorded on each listing (link checks 2026-10-21); tools without a tracked repo follow, by listing maturity. No paid placements.

1. 

   1. LiteLLM

   agent infraOpen source + cloud★ 61,465 GitHub stars

   Open-source AI gateway and Python SDK that calls 100+ LLM providers in the OpenAI format, with cost tracking, guardrails, load balancing and logging.

   Pricing (as stated): Open source + cloud · Mature

2. 

   2. MLflow

   evals & observabilityOpen source + cloud★ 28,418 GitHub stars

   Open-source platform for tracing, evaluating and managing agents, LLM apps and ML models.

   Pricing (as stated): Open source + cloud · Mature

3. 

   3. Beads

   codingOpen source★ 27,943 GitHub stars

   Issue-tracking memory for coding agents so they can keep track of long-running work.

   Pricing (as stated): Open source · Mature

4. 

   4. Opik

   evals & observabilityOpen source + cloud★ 22,700 GitHub stars

   Comet's open-source platform for tracing, evaluating and monitoring LLM apps, RAG systems and agent workflows, self-hosted or in the cloud.

   Pricing (as stated): Open source + cloud · Mature

5. 

   5. Moli

   browser/computer useOpen source★ 19,349 GitHub stars

   Lightweight Rust headless browser for AI agents that fetches pages, searches and extracts content, usable via CLI, CDP or WebDriver.

   Pricing (as stated): Open source · Mature

6. 

   6. CVAT AI Agents

   visionFreemium★ 16,943 GitHub stars

   CVAT feature that runs your own detection, segmentation or tracking model as an agent on your infrastructure and feeds its auto-annotations back into CVAT tasks.

   Pricing (as stated): Freemium · Active

7. 

   7. Oumi

   models & local runtimesOpen source★ 9,400 GitHub stars

   Open platform for fine-tuning, evaluating and deploying open agentic LLMs and VLMs.

   Pricing (as stated): Open source · Mature

8. 

   8. Manifest

   agent infraOpen source + cloud★ 7,595 GitHub stars

   Open-source LLM gateway that routes each request from agents and apps to the right model across API keys, subscriptions and local models, with spend tracking.

   Pricing (as stated): Self-hosted free (MIT); Manifest Cloud Free plan until Nov 1, 2026, then Cloud $19/month (Pro) · Mature

9. 

   9. Unstract

   no-code buildersOpen source + cloud★ 7,285 GitHub stars

   LLM-powered platform for extracting structured data from documents via APIs and ETL pipelines.

   Pricing (as stated): Open source + cloud · Mature

10. 

    10. Mission Control

    codingOpen source★ 6,358 GitHub stars

    Self-hosted control plane for dispatching tasks to agents, reviewing runs and tracking spend.

    Pricing (as stated): Open source · Mature

11. 

    11. Agents CLI

    frameworksOpen source★ 6,137 GitHub stars

    Google's CLI and skills that teach coding assistants to build, evaluate and deploy agents on Google Cloud and the Gemini platform.

    Pricing (as stated): Free and open source (Apache-2.0); Google Cloud usage billed separately · Mature

12. 

    12. SWE-bench

    evals & observabilityOpen source★ 6,067 GitHub stars

    Benchmark that tests whether language models and coding agents can resolve real GitHub issues, with multimodal and verified variants.

    Pricing (as stated): Free and open source (MIT) · Mature

13. 

    13. coucou

    codingOpen source★ 5,468 GitHub stars

    Mac notch and phone companion that monitors your Claude Code, Codex, Cursor and Gemini CLI sessions and lets you approve permissions on the go.

    Pricing (as stated): Open source · Mature

14. 

    14. agentgateway

    MCP serversOpen source★ 5,373 GitHub stars

    Proxy for AI agents and MCP servers with security and observability.

    Pricing (as stated): Open source · Mature

15. 

    15. Sparrow

    visionOpen source★ 5,225 GitHub stars

    Open-source tool for structured data extraction from documents and images with ML, LLMs and vision LLMs, plus agentic workflows.

    Pricing (as stated): Open source · Mature

16. 

    16. Agenta

    evals & observabilityOpen source + cloud★ 4,840 GitHub stars

    Open-source LLMOps platform for prompt management, evaluation and observability.

    Pricing (as stated): Open source + cloud · Active

17. 

    17. Latitude

    evals & observabilityOpen source + cloud★ 4,723 GitHub stars

    Observability platform for AI agents that captures traces, groups failures into signals and verifies fixes against real traces.

    Pricing (as stated): Open source + cloud · Active

18. 

    18. Cloudflare MCP Servers

    MCP serversOpen source★ 4,387 GitHub stars

    Cloudflare's collection of MCP servers for Workers, observability, DNS analytics and other Cloudflare services.

    Pricing (as stated): Open source · Active

19. 

    19. fast-agent

    frameworksOpen source★ 3,935 GitHub stars

    Framework to build, run and evaluate agents with strong MCP, ACP and A2A support.

    Pricing (as stated): Open source · Active

20. 

    20. Antigravity SDK

    frameworksOpen source★ 3,768 GitHub stars

    Google's Python SDK for building stateful AI agents powered by Antigravity and Gemini, abstracting the agent loop and runtime.

    Pricing (as stated): Open source · Active

21. 

    21. abtop

    codingOpen source★ 3,752 GitHub stars

    htop-style monitor for Claude Code and Codex sessions and token use.

    Pricing (as stated): Open source · Active

22. 

    22. Jarvis Registry

    MCP serversOpen source★ 3,648 GitHub stars

    Open-source MCP and A2A agent gateway that connects copilots and agents to enterprise tools with identity, access control and observability.

    Pricing (as stated): Open source · Active

23. 

    23. TruLens

    evals & observabilityOpen source★ 3,602 GitHub stars

    Evaluation and tracking for LLM experiments and AI agents.

    Pricing (as stated): Open source · Active

24. 

    24. sem

    codingOpen source★ 3,415 GitHub stars

    Semantic layer on top of Git that diffs, blames and traces impact by function and class instead of by line, with an MCP server for coding agents.

    Pricing (as stated): Open source · Active

25. 

    25. Laminar

    evals & observabilityOpen source + cloud★ 3,400 GitHub stars

    Open-source observability platform for AI agents with OpenTelemetry tracing, plain-English behavior tracking and alerts.

    Pricing (as stated): Open source + cloud · Active

FAQ

What are the best free AI agent evaluation tools?

Ranked by tracked GitHub stars, the top free AI agent evaluation tools in our directory are LiteLLM, MLflow, Beads, Opik, Moli. Stars measure popularity, not fit, so compare a few against your own task.

How many free AI agent evaluation tools does AgentScout list?

83 tools match. This page shows the top 25. Pricing mix on this page: Open source (16), Open source + cloud (8), Freemium (1).

Are these tools really free?

Pricing is what each maker states on their site. Open-source tools are free to self-host but may need a paid model API key or offer paid cloud plans; freemium tools have a free plan with limits. Check each official site.

Where can I see every tool for evaluating agents?

Our task page lists all 115 tools we match to evaluating agents.

Related: All tools for evaluating agents · evals & observability (33) · MCP servers (10) · coding (8) · frameworks (5)

By role: For AI engineers · For QA engineers

All popular searches · AI agents by role · Free tools

Best-of lists: Best AI Agent Observability & Eval Tools · All best lists

Related categories: Evals & observability · Coding · Frameworks · MCP servers

Facts come from our directory listings; pricing is as each maker states it and can change. Check official sites before you choose.