Qualifire AI Releases Rogue: An End-to-End Agentic AI Testing Framework, Evaluating The Effectivity Of AI Brokers

Agentic strategies are stochastic, context-dependent, and policy-bounded. Typical QA—unit checks, static prompts, or scalar “LLM-as-a-judge” scores—fails to indicate multi-turn vulnerabilities and gives weak audit trails. Developer teams need protocol-accurate conversations, particular protection checks, and machine-readable proof that will gate releases with confidence.

Qualifire AI has open-sourced Rogue, a Python framework that evaluates AI brokers over the Agent-to-Agent (A2A) protocol. Rogue converts enterprise insurance coverage insurance policies into executable eventualities, drives multi-turn interactions in direction of a aim agent, and outputs deterministic critiques acceptable for CI/CD and compliance critiques.

Quick Start

Stipulations

• uvx – If not put in, observe uv arrange info
• Python 3.10+
• An API key for an LLM provider (e.g., OpenAI, Google, Anthropic).

Arrange

Alternative 1: Quick Arrange (Advisable)

Use our automated arrange script to face up and working shortly:

# TUI
uvx rogue-ai
# Web UI
uvx rogue-ai ui
# CLI / CI/CD
uvx rogue-ai cli

Alternative 2: Handbook Arrange

(a) Clone the repository:

git clone https://github.com/qualifire-dev/rogue.git
cd rogue

(b) Arrange dependencies:

In the event you’re using uv:

Or, in the event you’re using pip:

(c) OPTIONALLY: Organize your environment variables: Create a .env file inside the root itemizing and add your API keys. Rogue makes use of LiteLLM, so that you’ll have the ability to set keys for diverse suppliers.

OPENAI_API_KEY="sk-..."
ANTHROPIC_API_KEY="sk-..."
GOOGLE_API_KEY="..."

Working Rogue

Rogue operates on a client-server construction the place the core evaluation logic runs in a backend server, and various customers hook up with it for varied interfaces.

Default Habits

In the event you run uvx rogue-ai with none mode specified, it:

1. Begins the Rogue server inside the background
2. Launches the TUI (Terminal Shopper Interface) shopper

On the market Modes

• Default (Server + TUI): uvx rogue-ai – Begins server in background + TUI shopper
• Server: uvx rogue-ai server – Runs solely the backend server
• TUI: uvx rogue-ai tui – Runs solely the TUI shopper (requires server working)
• Web UI: uvx rogue-ai ui – Runs solely the Gradio web interface shopper (requires server working)
• CLI: uvx rogue-ai cli – Runs non-interactive command-line evaluation (requires server working, glorious for CI/CD)

Mode Arguments

Server Mode

uvx rogue-ai server [OPTIONS]

Decisions:

• –host HOST – Host to run the server on (default: 127.0.0.1 or HOST env var)
• –port PORT – Port to run the server on (default: 8000 or PORT env var)
• –debug – Enable debug logging

TUI Mode

uvx rogue-ai tui [OPTIONS]
Web UI Mode
uvx rogue-ai ui [OPTIONS]

Decisions:

• –rogue-server-url URL – Rogue server URL (default: http://localhost:8000)
• –port PORT – Port to run the UI on
• –workdir WORKDIR – Working itemizing (default: ./.rogue)
• –debug – Enable debug logging

Occasion: Testing the T-Shirt Retailer Agent

This repository contains a simple occasion agent that sells T-shirts. It’s essential use it to see Rogue in movement.

Arrange occasion dependencies:

In the event you’re using uv:

or, in the event you’re using pip:

pip arrange -e .[examples]

(a) Start the occasion agent server in a separate terminal:

In the event you’re using uv:

uv run examples/tshirt_store_agent

If not:

python examples/tshirt_store_agent

This may occasionally start the agent on http://localhost:10001.

(b) Configure Rogue inside the UI to degree to the occasion agent:

• Agent URL: http://localhost:10001
• Authentication: no-auth

(c) Run the evaluation and watch Rogue check out the T-Shirt agent’s insurance coverage insurance policies!

It’s essential use each the TUI (uvx rogue-ai) or Web UI (uvx rogue-ai ui) mode.

The place Rogue Fits: Smart Use Circumstances

• Safety & Compliance Hardening: Validate PII/PHI coping with, refusal habits, secret-leak prevention, and regulated-domain insurance coverage insurance policies with transcript-anchored proof.
• E-Commerce & Help Brokers: Implement OTP-gated reductions, refund pointers, SLA-aware escalation, and tool-use correctness (order lookup, ticketing) beneath adversarial and failure conditions.
• Developer/DevOps Brokers: Assess code-mod and CLI copilots for workspace confinement, rollback semantics, rate-limit/backoff habits, and unsafe command prevention.
• Multi-Agent Applications: Verify planner↔executor contracts, performance negotiation, and schema conformance over A2A; contemplate interoperability all through heterogeneous frameworks.
• Regression & Drift Monitoring: Nightly suites in direction of new model variations or rapid modifications; detect behavioral drift and implement policy-critical transfer requirements sooner than launch.

What Exactly Is Rogue—and Why Must Agent Dev Teams Care?

Rogue is an end-to-end testing framework designed to evaluate the effectivity, compliance, and reliability of AI brokers. Rogue synthesizes enterprise context and menace into structured checks with clear objectives, methods and success requirements. The EvaluatorAgent runs protocol proper conversations in fast single flip or deep multi flip adversarial modes. Carry your particular person model, or let Rogue use Qualifire’s bespoke SLM judges to drive the checks. Streaming observability and deterministic artifacts: keep transcripts,transfer/fail verdicts, rationales tied to transcript spans, timing and model/mannequin lineage.

Beneath the Hood: How Rogue Is Constructed

Rogue operates on a client-server construction:

• Rogue Server: Incorporates the core evaluation logic
• Shopper Interfaces: Various interfaces that hook up with the server:
  • TUI (Terminal UI): Stylish terminal interface constructed with Go and Bubble Tea
  • Web UI: Gradio-based web interface
  • CLI: Command-line interface for automated evaluation and CI/CD

This construction permits for versatile deployment and utilization patterns, the place the server can run independently and various customers can hook up with it concurrently.

Summary

Rogue helps developer teams check out agent habits one of the simplest ways it actually runs in manufacturing. It turns written insurance coverage insurance policies into concrete eventualities, exercises these eventualities over A2A, and data what occurred with transcripts you’ll have the ability to audit. The end result’s a clear, repeatable signal it’s best to use in CI/CD to catch protection breaks and regressions sooner than they ship.

On account of the Qualifire crew for the thought administration/ Property for this textual content. Qualifire crew has supported this content material materials/article.

🙌 Adjust to MARKTECHPOST: Add us as a hottest provide on Google.