Agent Health
Open-source AI Agent Evaluation & Observability
Agent Health helps you evaluate, monitor, and optimize AI agents of any kind — coding assistants, ops/RCA agents, customer-support agents, data-analysis agents, retrieval/discovery agents, and multi-agent workflows. It provides real-time execution streaming, LLM-based evaluation with trajectory comparison, batch experiments, and deep observability through OpenTelemetry traces — stored locally out of the box, or in OpenSearch when you want a shared cluster.
[](LICENSE.txt)
[](https://www.npmjs.com/package/@opensearch-project/agent-health)
[](https://github.com/opensearch-project/agent-health/actions/workflows/ci.yml)
[](docs/SDK.md)
Website •
Slack •
Twitter/X •
Demo Video •
Documentation •
Changelog
Side-by-side comparison of agent evaluation runs with pass rate, accuracy, cost, and performance metrics over time.
---
## What is Agent Health?
Agent Health is an evaluation and observability framework for AI agents, built on [OpenSearch](https://opensearch.org). It helps you measure agent performance through **"Golden Path" trajectory comparison** — where an LLM judge evaluates agent actions against expected outcomes — and provides deep observability into agent execution via OpenTelemetry traces.
**Who uses Agent Health:**
- AI teams building autonomous agents (RCA, customer support, data analysis, retrieval/discovery)
- Teams comparing coding agents and multi-agent workflows across models, prompts, and context strategies
- QA engineers testing agent behavior across scenarios
- Platform teams monitoring agent performance in production
- Developers using AI coding agents who want visibility into usage, costs, and productivity
> **See it in action:** Watch the [demo video on YouTube](https://www.youtube.com/watch?v=MU3tTv4lKtc)
---
## AI Agent Skills
Agent Health ships with built-in skill files for *Claude Code* and *Kiro* that teach your AI coding agent how to work with this project effectively. Copy the relevant directory into your workspace to unlock project-aware assistance:
| Skill | Claude Code | Kiro | What it does |
|-------|-------------|------|--------------|
| **Add Connector** | [`.claude/skills/add-connector/SKILL.md`](./.claude/skills/add-connector/SKILL.md) | [`.kiro/steering/add-connector.md`](./.kiro/steering/add-connector.md) | Guides creation of custom agent connectors |
| **Write Test** | [`.claude/skills/write-test/SKILL.md`](./.claude/skills/write-test/SKILL.md) | [`.kiro/steering/write-test.md`](./.kiro/steering/write-test.md) | Project test conventions, mocking patterns, coverage thresholds |
| **Create PR** | [`.claude/skills/create-pr/SKILL.md`](./.claude/skills/create-pr/SKILL.md) | [`.kiro/steering/create-pr.md`](./.kiro/steering/create-pr.md) | PR workflow with DCO signoff and CI compliance |
| **Config & Auth** | [`.claude/skills/config-auth/SKILL.md`](./.claude/skills/config-auth/SKILL.md) | — | Config loading, AWS auth, multi-profile setup |
| **Instrument with OTel** | [`.claude/skills/instrument-otel/SKILL.md`](./.claude/skills/instrument-otel/SKILL.md) | — | OpenTelemetry GenAI span structure + config for Agent Health |
| **Agent Health** | [`.claude/skills/agent-health/SKILL.md`](./.claude/skills/agent-health/SKILL.md) | [`.kiro/steering/agent-health.md`](./.kiro/steering/agent-health.md) | Evaluate, benchmark & improve agents with the agent-health CLI/APIs |
**To use these skills:**
- *Claude Code* — Skills in `.claude/skills/` are auto-discovered when the directory exists in your workspace root. No extra setup needed.
- *Kiro* — Copy `.kiro/steering/` to your workspace root. Kiro loads steering files automatically.
---
## Installation
Get Agent Health running in minutes. Choose the option that best suits your needs:
### Option 1: NPX (Fastest — No Setup)
```bash
# Start Agent Health with demo data (no configuration needed)
npx @opensearch-project/agent-health
```
Opens http://localhost:4001 with pre-loaded sample data for exploration. If port 4001 is already in use, the server automatically tries the next available port (4002, 4003, etc., up to 10 attempts).
### Option 2: Docker Compose
For the full observability stack with OpenSearch, OpenTelemetry Collector, and Data Prepper for trace ingestion:
**Quick start (one command):**
```bash
curl -fsSL https://raw.githubusercontent.com/opensearch-project/agent-health/main/scripts/install.sh | bash
```
This clones the repo, starts the Docker stack, waits for OpenSearch, auto-configures `agent-health.config.json`, and launches Agent Health.
Or step-by-step:
```bash
# Clone the repository
git clone https://github.com/opensearch-project/agent-health.git
cd agent-health
# Start the OpenSearch observability stack
docker compose up -d
# Copy Docker environment configuration
cp .env.docker .env
# Start Agent Health (connects to local OpenSearch automatically)
npx @opensearch-project/agent-health
```
This brings up:
- **OpenSearch** — Stores traces, test cases, benchmarks, and evaluation results
- **OpenTelemetry Collector** — Receives telemetry data via OTLP (ports 4317/4318)
- **Data Prepper** — Transforms and enriches traces before OpenSearch ingestion
> **Prerequisites:** Docker Desktop with 4GB+ memory allocated. See [docker-compose.yml](./docker-compose.yml) for configuration options.
### Option 3: AWS CloudFormation (Managed OpenSearch)
Deploy a fully managed observability backend using the included CloudFormation template:
```bash
aws cloudformation create-stack \
--stack-name AgentHealthObservability \
--template-body file://deployment/cloudformation/agent-health-observability.yaml \
--capabilities CAPABILITY_NAMED_IAM
```
This deploys:
- **Amazon OpenSearch Service** domain or **OpenSearch Serverless** collection for trace storage
- **OpenSearch Ingestion (OSIS)** pipeline for OTLP data collection
- **IAM roles** for pipeline execution and agent telemetry ingestion
> **Both Amazon OpenSearch Service domains and OpenSearch Serverless collections are supported.** Set `OPENSEARCH_STORAGE_AWS_SERVICE=es` for managed domains or `OPENSEARCH_STORAGE_AWS_SERVICE=aoss` for Serverless collections. Both use SigV4 authentication (`OPENSEARCH_STORAGE_AUTH_TYPE=sigv4`). See [docs/CONFIGURATION.md](./docs/CONFIGURATION.md) for details.
After deployment, connect it to Agent Health:
```bash
npx @opensearch-project/agent-health configure --from-stack AgentHealthObservability
```
Or manually copy the `AgentHealthConfigJSON` stack output into your `agent-health.config.json`. See [deployment/cloudformation/](./deployment/cloudformation/) for details and regional Launch Stack URLs.
### Next Steps
- [Getting Started Guide](./GETTING_STARTED.md) — Step-by-step walkthrough from install to first evaluation
- [Configuration Guide](./docs/CONFIGURATION.md) — Connect your own agent and configure the environment
- [CLI Reference](./docs/CLI.md) — Full command-line documentation
---
## Features
### Agent Evaluation & Observability
| Feature | Description |
|---------|-------------|
| **Evals** | Real-time agent evaluation with trajectory streaming |
| **Experiments** | Batch evaluation runs with configurable parameters |
| **Compare** | Side-by-side trace comparison with aligned and merged views |
| **Agent Traces** | Table-based trace view with latency histogram, filtering, and detailed flyout |
| **Live Traces** | Real-time trace monitoring with auto-refresh and filtering |
| **Trace Views** | Timeline and Flow visualizations for debugging |
| **Reports** | Evaluation reports with LLM judge reasoning |
| **Connectors** | Pluggable protocol adapters (AG-UI SSE, REST, CLI, Claude Code) |
### Coding Agent Analytics
A unified dashboard for monitoring AI coding agent usage across **Claude Code**, **Kiro**, and **Codex CLI**. Zero configuration — just run `agent-health` and it auto-detects installed agents.
- **Multi-agent dashboard**: Session history, cost estimation, tool usage, activity patterns, and efficiency metrics
- **9 analytics tabs**: Overview, Sessions, Projects, Costs, Activity, Efficiency, Tools, Advanced, and Workspace management
- **Interactive drill-downs**: Click any chart, card, or metric to drill into filtered session views
- **Workspace management**: View and edit Claude Code memory files, plans, tasks; browse Kiro MCP servers, agents, and extensions
- **Privacy-first**: All data stays local — reads directly from `~/.claude/`, `~/.kiro/`, `~/.codex/`
[Full Coding Agent Analytics documentation](./docs/CODING_AGENT_ANALYTICS.md)
### Supported Connectors
| Connector | Protocol | Description |
|-----------|----------|-------------|
| `agui-streaming` | AG-UI SSE | ML-Commons agents (default) |
| `rest` | HTTP POST | Non-streaming REST APIs |
| `openai-compatible` | OpenAI Chat | LiteLLM, Ollama, vLLM |
| `strands` | Bedrock Agent Runtime | Amazon Strands agents (server-only) |
| `langgraph` | LangGraph REST | Non-AG-UI LangGraph instances |
| `subprocess` | CLI | Command-line tools |
| `claude-code` | Claude CLI | Claude Code agent comparison |
| `kiro` | Kiro CLI | Kiro coding agent |
| `pi` | Pi CLI | Pi coding agent |
| `mock` | In-memory | Demo and testing |
For creating custom connectors, see [docs/CONNECTORS.md](./docs/CONNECTORS.md).
### Observio Sample Agent
Agent Health includes **Observio**, a reference ReAct agent you can use as a practice target for evaluating and improving agent performance:
```bash
cd observio-sample-agent && npm install && npm run start:ag-ui
npx @opensearch-project/agent-health run -t demo-otel-001 -a observio
```
See the [Observio README](./observio-sample-agent/README.md) for details.
---
## Architecture
Agent Health uses a client-server architecture where all clients (UI, CLI) access OpenSearch through a unified HTTP API. The server handles agent communication via pluggable connectors and proxies LLM judge calls to AWS Bedrock.
For detailed architecture documentation, see [docs/ARCHITECTURE.md](./docs/ARCHITECTURE.md).
---
## Quick Configuration
Agent Health works out-of-the-box with demo data. Configure when you're ready to connect your own agent:
```bash
# Generate a config file with examples
npx @opensearch-project/agent-health init
```
```typescript
// agent-health.config.ts
export default {
agents: [
{
key: "my-agent",
name: "My Agent",
endpoint: "http://localhost:8000/agent",
connectorType: "rest", // or "agui-streaming", "langgraph", "strands", "subprocess"
models: ["claude-sonnet-4"],
useTraces: true, // Enable OpenTelemetry trace collection (default: false)
}
],
};
```
> **Tip:** Run `npx @opensearch-project/agent-health doctor` to verify your configuration is loaded correctly.
For full configuration options including authentication hooks and environment variables, see [CONFIGURATION.md](./docs/CONFIGURATION.md).
---
## Star History
If you find Agent Health useful, please consider giving us a star! Your support helps us grow our community and continue improving the project.
[](https://github.com/opensearch-project/agent-health)
---
## Contributing
We welcome contributions! There are many ways to get involved:
- [Report a Bug](https://github.com/opensearch-project/agent-health/issues/new/choose) — Found something broken? Let us know
- [Request a Feature](https://github.com/opensearch-project/agent-health/issues/new/choose) — Have an idea? We'd love to hear it
- [Submit a Pull Request](https://github.com/opensearch-project/agent-health/pulls) — Code contributions are always welcome
- [Join the Discussion](https://opensearch.org/slack.html) — Chat with us on the OpenSearch Slack
### Development Quick Start
```bash
git clone https://github.com/opensearch-project/agent-health.git
cd agent-health
npm install
npm run dev # Frontend on port 4000
npm run dev:server # Backend on port 4001
```
> **Port conflicts:** If port 4001 is already in use, the backend server automatically tries 4002, 4003, etc. (up to 10 attempts). The actual port is displayed in the console output.
All commits require DCO signoff (`git commit -s`) and all PRs must pass CI checks.
For detailed development setup, testing, CI pipeline, debugging, and troubleshooting, see the [Developer Guide](./DEVELOPER_GUIDE.md). For full contribution guidelines, see [CONTRIBUTING.md](./CONTRIBUTING.md).
---
## Documentation
| Guide | Description |
|-------|-------------|
| [Getting Started](./GETTING_STARTED.md) | Step-by-step walkthrough from install to first evaluation |
| [Configuration](./docs/CONFIGURATION.md) | Connect your agent and configure the environment |
| [CLI Reference](./docs/CLI.md) | Command-line interface documentation |
| [Code-Based SDK](./docs/SDK.md) | Write evaluations as `.eval.js` / `.eval.ts` test files (experimental) |
| [Skill Evaluator](./docs/SKILLS.md) | A/B-benchmark and improve a `SKILL.md` |
| [Instrument with OTel](./docs/INSTRUMENT_WITH_OTEL.md) | OpenTelemetry instrumentation for Agent Health |
| [Coding Agent Analytics](./docs/CODING_AGENT_ANALYTICS.md) | Multi-agent dashboard and remote server monitoring |
| [Observio Sample Agent](./observio-sample-agent/) | Reference agent for practicing evaluations |
| [Developer Guide](./DEVELOPER_GUIDE.md) | Development setup, testing, CI, debugging |
| [Connectors Guide](./docs/CONNECTORS.md) | Create custom connectors for your agent type |
| [Architecture](./docs/ARCHITECTURE.md) | System design and patterns |
| [ML-Commons Setup](./docs/ML-COMMONS-SETUP.md) | OpenSearch ML-Commons integration |
---
Made with care by the OpenSearch community