Evals & observability AI agent tools

Superlog runs AI agents on routine engineering work around production. You set up automations with triggers and connectors, for example rating each new Sentry issue, posting a plain-language summary to Slack and opening a pull request with the fix, comparing error rates and latency against their normal range every hour, adding missing logs and spans to hard-to-debug code paths, or answering community questions from your code and docs. You choose the model and agent harness and bring your own tokens or subscriptions. The core is open source (Apache-2.0) with a hosted version.

Added Oct 21, 2026

Tracely

Tracely is open-source, trace-native CI/CD for AI agents: it grades every production trace with evaluators, clusters failures into issues, freezes bad runs into replayable test cases and blocks pull requests that would reintroduce them.

Added Oct 20, 2026

AgentEnv Framework

Newevals & observabilityFree

AgentEnv Framework is Scale AI's open-source Python SDK and CLI for building RL environments for agents: containerized worlds with realistic apps and data behind an MCP gateway, plus tasks (as DAGs) that grade any agent in any sandbox.

Added Oct 20, 2026

Firedrill

A simulation platform for testing AI agents before launch: it runs your agent through realistic scenarios with simulated users and tools, scores results against checks or rubrics and shows where it fails. Includes a remote MCP server.

Added Oct 20, 2026

Openlayer MCP

Newevals & observabilityFreemium

Openlayer's MCP server, letting coding agents pull evaluation results, failing tests and monitoring data from Openlayer so they can debug and improve AI systems from the IDE.

Added Oct 20, 2026

AI Beat

Newevals & observabilitySee site

An open-source AI red-teaming suite: PromptBeat attacks models and prompts, and AgentBeat runs black-box security evaluations against HTTP-exposed agents.

Added Oct 20, 2026

agent-qa (Vostride)

A self-improving AI QA agent with persistent memory that tests web, Android and iOS apps from plain-English test steps, for developers and coding agents.

Added Oct 20, 2026

Rogue

Newevals & observabilitySource-available

Agent evaluator and red-team tool that probes your agent with test conversations and attacks and reports pass or fail against your policies.

Added Oct 19, 2026

Kiln

Free desktop app and open-source library for building AI systems with evals, RAG, agents, fine-tuning and synthetic data in one workbench.

Added Oct 18, 2026

Laminar

Newevals & observabilitySource-available + cloud

Open-source observability platform for AI agents with OpenTelemetry tracing, plain-English behavior tracking and alerts.

Added Oct 18, 2026

MCPJam Inspector

Open-source testing and evals platform for MCP servers, MCP apps and ChatGPT apps across many clients and models.

Added Oct 18, 2026

Future AGI

Open-source platform for evaluating, tracing, simulating and guarding LLM and AI agent applications, self-hostable or cloud.

Added Oct 18, 2026

Opik

Comet's open-source platform for tracing, evaluating and monitoring LLM apps, RAG systems and agent workflows, self-hosted or in the cloud.

Added Oct 18, 2026

SWE-bench

Benchmark that tests whether language models and coding agents can resolve real GitHub issues, with multimodal and verified variants.

Added Oct 18, 2026

Latitude

Observability platform for AI agents that captures traces, groups failures into signals and verifies fixes against real traces.

Added Oct 18, 2026

Pezzo

Cloud-native LLMOps platform for prompt design, versioning, delivery, observability and cost monitoring.

Added Oct 18, 2026

OSWorld

Benchmark for multimodal computer-use agents on open-ended tasks in real desktop environments.

Added Oct 18, 2026

BrowserGym

Newevals & observabilityFreemium

ServiceNow's Gym environment for building, testing and evaluating web agents across browser benchmarks.

Added Oct 18, 2026

Confident AI

Enterprise platform for AI evals and observability that gives teams one standard for measuring and monitoring AI quality.

Added Oct 18, 2026

Coval

Newevals & observabilityPaid

Testing and evaluation platform for voice agents, with simulations, production evals and human QA.

Added Oct 18, 2026

Arize AI

Newevals & observabilityFreemium

Agent observability and evaluation platform with tracing, evals and experiments for improving AI agents.

Added Oct 18, 2026

OpenInference

OpenTelemetry-based instrumentation that traces LLM and agent framework calls so any observability backend can show them.

Added Oct 18, 2026

Judgeval

Open-source SDK for tracing agents and scoring them with agent judges, so failures in production can be found and fixes checked.

Added Oct 18, 2026

Agent Memory Leaderboard

evals & observabilityEnterprise

Public leaderboard and open evaluation protocol, run by a group of universities and labs, for comparing long-term memory systems for agents in recurring submission rounds.

Added Oct 17, 2026

Kore.ai Autoloop

Optimization engine in Kore.ai Agent Platform Artemis that evaluates agents against business goals, diagnoses failures from traces, applies targeted fixes, and re-verifies; GA Oct 2026.

Added Oct 14, 2026

CoreWeave Forge

One product to run, observe, evaluate and post-train agents (W&B, OpenPipe, sandboxes).

Added Oct 14, 2026

LangSmith

Tracing, evals and monitoring for LLM apps and agents.

Added Oct 14, 2026

Langfuse

Open-source LLM observability, tracing and evals.

Added Oct 14, 2026

Braintrust

evals & observabilitySource-available + cloud

Evals and observability platform for AI products.

Added Oct 14, 2026

Arize Phoenix

Open-source tracing and evaluation for LLM apps.

Added Oct 14, 2026

W&B Weave

Weights & Biases toolkit for tracing and evaluating agents.

Added Oct 14, 2026

AgentOps

Monitoring and session replay for AI agents.

Added Oct 14, 2026

Helicone

Open-source LLM observability and gateway.

Added Oct 14, 2026

Respan

Observability and evaluation platform for LLM apps and agents.

Added Oct 16, 2026

MLflow

Open-source platform for tracing, evaluating and managing agents, LLM apps and ML models.

Added Oct 16, 2026

Agenta