Open-source LLM observability and eval tools (2026)
Open-source tracing, evaluation and monitoring tools for LLM apps and agents that you can self-host. Here are all 23 from the AgentScout directory.
Updated
How we picked and ranked: tools are matched by category, the pricing each maker states and what each tool says it does. They are ranked by GitHub stars as recorded on each listing (link checks 2026-10-21); tools without a tracked repo follow, by listing maturity. No paid placements.
1. MLflow
Open-source platform for tracing, evaluating and managing agents, LLM apps and ML models.
Pricing (as stated): Open source + cloud · Mature
2. Opik
Comet's open-source platform for tracing, evaluating and monitoring LLM apps, RAG systems and agent workflows, self-hosted or in the cloud.
Pricing (as stated): Open source + cloud · Mature
3. SWE-bench
Benchmark that tests whether language models and coding agents can resolve real GitHub issues, with multimodal and verified variants.
Pricing (as stated): Free and open source (MIT) · Mature
4. Agenta
Open-source LLMOps platform for prompt management, evaluation and observability.
Pricing (as stated): Open source + cloud · Active
5. Latitude
Observability platform for AI agents that captures traces, groups failures into signals and verifies fixes against real traces.
Pricing (as stated): Open source + cloud · Active
6. TruLens
Evaluation and tracking for LLM experiments and AI agents.
Pricing (as stated): Open source · Active
7. Laminar
Open-source observability platform for AI agents with OpenTelemetry tracing, plain-English behavior tracking and alerts.
Pricing (as stated): Open source + cloud · Active
8. Pezzo
Cloud-native LLMOps platform for prompt design, versioning, delivery, observability and cost monitoring.
Pricing (as stated): Open source · Active
9. OSWorld
Benchmark for multimodal computer-use agents on open-ended tasks in real desktop environments.
Pricing (as stated): Open source · Active
10. OpenLIT
Open-source platform to trace, evaluate and guard agents and LLM apps.
Pricing (as stated): Open source · Active
11. Future AGI
Open-source platform for evaluating, tracing, simulating and guarding LLM and AI agent applications, self-hostable or cloud.
Pricing (as stated): Open source + cloud · Active
12. Superlog
Superlog runs AI agents on routine engineering work around production. You set up automations with triggers and connectors, for example rating each new Sentry issue, posting a plain-language summary to Slack and opening a pull request with the fix, comparing error rates and latency against their normal range every hour, adding missing logs and spans to hard-to-debug code paths, or answering community questions from your code and docs. You choose the model and agent harness and bring your own tokens or subscriptions. The core is open source (Apache-2.0) with a hosted version.
Pricing (as stated): Open source (Apache-2.0) to self-host; hosted plans and enterprise on the pricing page; bring your own model tokens or subscriptions · Active
13. BrowserGym
ServiceNow's Gym environment for building, testing and evaluating web agents across browser benchmarks.
Pricing (as stated): Open source · Active
14. AI Beat
An open-source AI red-teaming suite: PromptBeat attacks models and prompts, and AgentBeat runs black-box security evaluations against HTTP-exposed agents.
Pricing (as stated): Open source · Active
15. OpenInference
OpenTelemetry-based instrumentation that traces LLM and agent framework calls so any observability backend can show them.
Pricing (as stated): Open source · Active
16. Tracely
Tracely is open-source, trace-native CI/CD for AI agents: it grades every production trace with evaluators, clusters failures into issues, freezes bad runs into replayable test cases and blocks pull requests that would reintroduce them.
Pricing (as stated): Self-host free (MIT); hosted Free $0 (20k traces/month), Team $49/mo; LLM judges run on your own OpenRouter key · Active
17. Agent Memory Leaderboard
Public leaderboard and open evaluation protocol, run by a group of universities and labs, for comparing long-term memory systems for agents in recurring submission rounds.
Pricing (as stated): Open source · Active
18. Rogue
Agent evaluator and red-team tool that probes your agent with test conversations and attacks and reports pass or fail against your policies.
Pricing (as stated): Open source · Active
19. Judgeval
Open-source SDK for tracing agents and scoring them with agent judges, so failures in production can be found and fixes checked.
Pricing (as stated): Open source + cloud · Active
20. AgentEnv Framework
AgentEnv Framework is Scale AI's open-source Python SDK and CLI for building RL environments for agents: containerized worlds with realistic apps and data behind an MCP gateway, plus tasks (as DAGs) that grade any agent in any sandbox.
Pricing (as stated): Free and open source (Apache-2.0); pip install agentenv-framework · Active
21. Galileo Agent Leaderboard
Galileo's leaderboard that ranks how well LLMs perform as agents on tool use and multi-step tasks.
Pricing (as stated): Open source · Active
22. Helicone
evals & observabilityOpen source + cloudOpen-source LLM observability and gateway.
Pricing (as stated): Open source + paid cloud · Mature
23. Langfuse
evals & observabilityOpen source + cloudOpen-source LLM observability, tracing and evals.
Pricing (as stated): Open source + paid cloud · Mature
FAQ
What are the best open-source LLM observability?
Ranked by tracked GitHub stars, the top open-source LLM observability in our directory are MLflow, Opik, SWE-bench, Agenta, Latitude. Stars measure popularity, not fit, so compare a few against your own task.
How many open-source LLM observability does AgentScout list?
23 tools match. This page shows the top 23. Pricing mix on this page: Open source (12), Open source + cloud (11).
How much do these tools cost?
Pricing is what each maker states on their site. Open-source tools are free to self-host but may need a paid model API key or offer paid cloud plans; freemium tools have a free plan with limits. Check each official site.
Where can I see every tool for monitoring agents in production?
Our task page lists all 34 tools we match to monitoring agents in production.
Related: All tools for monitoring agents in production · Best AI Agent Observability & Eval Tools · evals & observability (23)
By role: For AI engineers · For DevOps and SRE engineers · For startup CTOs
All popular searches · AI agents by role · Free tools
Related categories: Evals & observability
Facts come from our directory listings; pricing is as each maker states it and can change. Check official sites before you choose.