Tracely
Tracely is open-source, trace-native CI/CD for AI agents: it grades every production trace with evaluators, clusters failures into issues, freezes bad runs into replayable test cases and blocks pull requests that would reintroduce them.
Updated
✨ Try AgentScout's free AI image generator →Good for: reviewing pull requests, evaluating agents, testing apps
Use cases
About Tracely
Tracely is listed in our evals & observability category and is aimed at builders. Tracely is open-source, trace-native CI/CD for AI agents: it grades every production trace with evaluators, clusters failures into issues, freezes bad runs into replayable test cases and blocks pull requests that would reintroduce them. Pricing, as the maker states it: Self-host free (MIT); hosted Free $0 (20k traces/month), Team $49/mo; LLM judges run on your own OpenRouter key. Maturity: Active. We last checked the official link on 2026-10-20.
Tracely at a glance
- Category: Evals & observability
- Built for: Developers and teams building with AI agents
- Pricing: Self-host free (MIT); hosted Free $0 (20k traces/month), Team $49/mo; LLM judges run on your own OpenRouter key (as the maker states it)
- GitHub: jwuthri-cloud/Tracely-ai · 1,160 stars (as of Oct 21, 2026)
- Latest release: sdk-v0.0.1: chore(sdk): publish as trac…, Oct 12, 2026
- Last commit: Oct 15, 2026
- Official site: tracely-ai.com
- Listed on AgentScout: Oct 20, 2026
Tracely FAQ
What is Tracely?
Tracely is open-source, trace-native CI/CD for AI agents: it grades every production trace with evaluators, clusters failures into issues, freezes bad runs into replayable test cases and blocks pull requests that would reintroduce them.
How much does Tracely cost?
Tracely's pricing, as the maker states it: Self-host free (MIT); hosted Free $0 (20k traces/month), Team $49/mo; LLM judges run on your own OpenRouter key. Check the official site for current plans.
Where can I find Tracely's source code?
On GitHub at jwuthri-cloud/Tracely-ai, which had 1,160 stars as of Oct 21, 2026. The latest release is sdk-v0.0.1: chore(sdk): publish as tracelyai-sdk, version 0.0.1 (Oct 12, 2026).
What are alternatives to Tracely?
Similar tools include OpenLIT, Latitude, OpenInference, Judgeval, Rogue and SWE-bench.
Facts from our link-checked listing, the GitHub repository and the official site. See something wrong? Suggest an edit.
Tracely ratings and reviews
Be the first to rate Tracely: tap 1 to 5 stars below. A written review is optional.
Tracely alternatives
Other evals & observability tools in our directory. See all evals & observability tools →
OpenLIT
Open-source platform to trace, evaluate and guard agents and LLM apps.
Latitude
Observability platform for AI agents that captures traces, groups failures into signals and verifies fixes against real traces.
OpenInference
OpenTelemetry-based instrumentation that traces LLM and agent framework calls so any observability backend can show them.
Judgeval
Open-source SDK for tracing agents and scoring them with agent judges, so failures in production can be found and fixes checked.
Rogue
Agent evaluator and red-team tool that probes your agent with test conversations and attacks and reports pass or fail against your policies.
SWE-bench
Benchmark that tests whether language models and coding agents can resolve real GitHub issues, with multimodal and verified variants.
Popular searches with this tool
🏆 Ranked in: Best AI Agent Observability & Eval Tools
🧭 Task guides: AI eval and benchmark tools
Badge for makers
Built Tracely? Add this badge to your README or site. It links back to this page. About badges
