Rogue

AI Beat

An open-source AI red-teaming suite: PromptBeat attacks models and prompts, and AgentBeat runs black-box security evaluations against HTTP-exposed agents.

Tracely

Tracely is open-source, trace-native CI/CD for AI agents: it grades every production trace with evaluators, clusters failures into issues, freezes bad runs into replayable test cases and blocks pull requests that would reintroduce them.

SWE-bench

Benchmark that tests whether language models and coding agents can resolve real GitHub issues, with multimodal and verified variants.

Firedrill

A simulation platform for testing AI agents before launch: it runs your agent through realistic scenarios with simulated users and tools, scores results against checks or rubrics and shows where it fails. Includes a remote MCP server.

Superlog

Superlog runs AI agents on routine engineering work around production. You set up automations with triggers and connectors, for example rating each new Sentry issue, posting a plain-language summary to Slack and opening a pull request with the fix, comparing error rates and latency against their normal range every hour, adding missing logs and spans to hard-to-debug code paths, or answering community questions from your code and docs. You choose the model and agent harness and bring your own tokens or subscriptions. The core is open source (Apache-2.0) with a hosted version.

CoreWeave Forge

One product to run, observe, evaluate and post-train agents (W&B, OpenPipe, sandboxes).