# M365 Copilot Agent Evaluations (CLI)

This CLI evaluates DA responses from a configured M365 Copilot endpoint (copilotApi) and scores them locally using Azure AI Evaluation services.

Current evaluation metrics:
- Relevance (1–5)
- Coherence (1–5)
- Groundedness (1–5)
- Citations (count with pass/fail based on presence)

## 📋 Prerequisites

- Python 3.13.10+ - https://www.python.org/downloads/
- Azure subscription with Azure OpenAI access (for evaluation metrics only)

## 🚀 Quick Start (Run CLI)

### 1. Clone and Install Dependencies

```bash
git clone https://github.com/microsoft/M365-Copilot-Agent-Evals.git
cd M365-Copilot-Agent-Evals/src/clients/cli
python -m venv .venv # Create virtual Python environment.
.\.venv\Scripts\Activate.ps1 # Activate the virtual environment.
pip install -r requirements.txt
```

### 2. Set Up Environment Variables

Create a `.env` file in the `src/clients/cli` directory (or export them). Use interactive WAM auth (Windows) to authenticate with the Copilot API.

```bash
# Azure OpenAI (evaluation models)
AZURE_AI_OPENAI_ENDPOINT="<Your_Azure_AI_OpenAI_Endpoint>"
AZURE_AI_API_KEY="<azure-openai-key>"
AZURE_AI_API_VERSION="2024-12-01-preview"
AZURE_AI_MODEL_NAME="gpt-4o-mini"

# Your Tenant ID (or use TEAMS_APP_TENANT_ID from ATK .env.local)
TENANT_ID="<aad-tenant-id>"

# Optional: default agent id (overridable via --m365-agent-id)
M365_AGENT_ID="00000000-0000-0000-0000-000000000000"

# Optional: INFO/DEBUG console message truncation (defaults shown)
# WARNING and ERROR messages are never truncated.
RUNEVALS_LOG_TRUNCATE="true"
RUNEVALS_LOG_MAX_LENGTH="250"
```

### 3. Run the Agent Evaluation

The CLI supports multiple ways to run evaluations:

#### Basic Usage (with default prompt set)
```bash
python main.py
```

#### Command Line Prompts
```bash
# Single prompt and expected response
python main.py --prompts "What is Microsoft Graph?" --expected "Microsoft Graph is a gateway to data and intelligence in Microsoft 365."

# Multiple prompts and expected responses
python main.py --prompts "What is Microsoft Graph?" "How does authentication work?" --expected "Microsoft Graph is a gateway..." "Authentication works by..."

# Override the agent configured in environment variables
python main.py --m365-agent-id "00000000-0000-0000-0000-000000000000" --prompts "What is Microsoft Graph?"
```

#### Using Prompts from File
```bash
# JSON file with prompts and expected responses
python main.py --prompts-file ../../../schema/v1/examples/valid/example_prompts.json

# Output to JSON file
python main.py --prompts-file ../../../schema/v1/examples/valid/example_prompts.json --output results.json

# Output to CSV file
python main.py --prompts-file ../../../schema/v1/examples/valid/example_prompts.json --output results.csv

# Output to HTML file (opens in browser)
python main.py --prompts-file ../../../schema/v1/examples/valid/example_prompts.json --output report.html
```

#### Evaluating Captured Responses
```bash
# Score responses already present in a standard v1 eval document.
# This does not invoke or authenticate to the M365 agent.
python main.py --evaluate-only captured-responses.json

# Write the normal schema-compliant evaluation report.
python main.py --evaluate-only captured-responses.json --output results.json
```

`--evaluate-only` cannot be combined with `--prompts`, `--prompts-file`, or
`--interactive`. Every item or turn in the input document must contain a
`response`. Azure/Foundry or GitHub Copilot judge configuration remains
required; WorkIQ/A2A environment variables are not required.

#### Interactive Mode
```bash
# Enter prompts interactively
python main.py --interactive
```

#### Additional Options
```bash
# Logging verbosity (canonical control surface)
python main.py --log-level debug
python main.py --log-level info
python main.py --log-level warning
python main.py --log-level error

# Bare flag resolves to info
python main.py --log-level

# Legacy flags (no longer supported; use --log-level instead)
# The following will fail with "unrecognized arguments" errors:
python main.py --verbose
python main.py --quiet

# Share diagnostics with support: redirect console output to a file (recommended)
 # PowerShell: '*>' redirects all streams (stdout + stderr)
 python main.py --log-level debug --prompts-file ../../../schema/v1/examples/valid/example_prompts.json *> my_run.log
 # bash/zsh: merge stderr into stdout
 python main.py --log-level debug --prompts-file ../../../schema/v1/examples/valid/example_prompts.json > my_run.log 2>&1

# Get help and see all options
python main.py --help

# Specify / override the Agent ID (takes precedence over M365_AGENT_ID env var)
python main.py --m365-agent-id "00000000-0000-0000-0000-000000000000"
```

#### File Format Examples

**JSON Format 1 (Array of Objects):**
```json
[
  {
    "prompt": "What is Microsoft Graph?",
    "expected_response": "Microsoft Graph is a gateway to data and intelligence in Microsoft 365."
  },
  {
    "prompt": "How do I authenticate with Microsoft Graph?",
    "expected_response": "You can authenticate using OAuth 2.0..."
  }
]
```

**JSON Format 2 (Separate Arrays):**
```json
{
  "prompts": [
    "What is Microsoft Graph?",
    "How do I authenticate with Microsoft Graph?"
  ],
  "expected_responses": [
    "Microsoft Graph is a gateway to data and intelligence in Microsoft 365.",
    "You can authenticate using OAuth 2.0..."
  ]
}
```

## 🔧 Configuration

### Getting Azure AI Foundry Configuration Values

#### 1. Azure OpenAI Endpoint (`AZURE_AI_OPENAI_ENDPOINT`)

1. In [Azure AI Foundry](https://ai.azure.com/)
2. Go to **Models + endpoints** under My assets
3. Select your model. Create one if you don't have one available - Use `gpt-4o-mini` for better compatibility with the evaluators
4. Copy the endpoint URL (format: `https://your-resource.openai.azure.com/`)

#### 2. Azure OpenAI API Key (`AZURE_AI_API_KEY`)

1. In [Azure AI Foundry](https://ai.azure.com/)
2. Go to **Models + endpoints** under My assets
3. Select your model. Create one if you don't have one available - Use `gpt-4o-mini` for better compatibility with the evaluators
4. Copy the key

#### 3. Azure OpenAI Model Name (`AZURE_AI_MODEL_NAME`)

1. In [Azure AI Foundry](https://ai.azure.com/)
2. Go to **Models + endpoints** under My assets
3. Select your model. Create one if you don't have one available - Use `gpt-4o-mini` for better compatibility with the evaluators
4. Copy the model/deployment name (e.g., "gpt-4o-mini", "gpt-4", "gpt-35-turbo")

#### 4. API Version (`AZURE_AI_API_VERSION`)

Use the API version: `2024-12-01-preview`

For the most current version, check the [Azure OpenAI API reference](https://docs.microsoft.com/en-us/azure/cognitive-services/openai/reference).


## 📁 Project Structure

```
M365-Copilot-Agent-Evals/
├── schema/
│   └── v1/
│       └── examples/
│           ├── valid/                 # Runnable, schema-valid eval documents
│           └── invalid/               # Negative schema-validation fixtures
├── src/
│   └── clients/
│       └── cli/
│           ├── main.py              # Main evaluation script
│           ├── response_extractor.py # Enhanced response parsing
│           ├── generate_report.py   # HTML report generation with aggregates
│           ├── requirements.txt     # Python dependencies
│           ├── readme.md           # This file
│           ├── CHANGELOG.md        # Version history and changes
│           ├── .env.template       # Environment variables template
│           ├── .env                # Environment variables (create this)
│           └── custom_evaluators/  # Custom evaluation modules
│               ├── __init__.py
│               └── citations_evaluator.py  # Citation detection evaluator
├── tests/                          # Test files
├── CODE_OF_CONDUCT.md             # Code of conduct
├── LICENSE                        # License file
├── README.md                      # Root project README
├── SECURITY.md                    # Security guidelines
└── SUPPORT.md                     # Support information
```

## 📊 Features

- **Chat Invocation**: Sends prompts via the WorkIQ API
- **Evaluation Metrics**: Relevance, Coherence, Groundedness, Citations
  - **Multi-format Citation Detection**: Supports OAI Unicode, legacy bracket, and markdown link (`[N](targetLink#suffix)`) formats
- **Enhanced Response Extraction**: Detailed parsing of results, and message flow
- **Aggregate Statistics**: Summary metrics across multiple prompts with pass/fail rates
- **Colorized Console Output**
- **Multiple Output Formats**: JSON, CSV, HTML with aggregate dashboards
- **Flexible Prompt Input**: Command line, file, interactive

## 🔒 Security Best Practices

- ✅ All sensitive information is stored in environment variables
- ✅ No hardcoded credentials in source code
- ✅ `.env` files are excluded from version control

## 🔑 Authentication Requirements

- WorkIQ A2A API: WAM interactive auth via `WORK_IQ_A2A_CLIENT_ID`, `TENANT_ID`, and `WORK_IQ_A2A_SCOPES`
- Evaluators: Azure OpenAI key (`AZURE_AI_API_KEY`)

## 📚 Useful Resources

- [Azure AI Foundry Documentation](https://docs.microsoft.com/en-us/azure/ai-services/ai-foundry/)
- [Azure AI Projects Python SDK](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/README.md)
- [Azure AI Evaluation Documentation](https://docs.microsoft.com/en-us/azure/ai-services/ai-foundry/how-to/evaluate-models)
- [Azure OpenAI Service Documentation](https://docs.microsoft.com/en-us/azure/cognitive-services/openai/)
- [Azure Identity Library](https://docs.microsoft.com/en-us/python/api/overview/azure/identity-readme)

## 🐛 Troubleshooting

### Common Issues

1. **Authentication Errors**: Ensure your Azure credentials are properly configured and you have access to the resources.

2. **HTTP 401 / 403**: Confirm `WORK_IQ_A2A_CLIENT_ID`, `TENANT_ID`, and `WORK_IQ_A2A_SCOPES` are correct. Re-run `--signout` to clear cached tokens and re-authenticate.

3. **Endpoint Issues**: Confirm `WORK_IQ_A2A_ENDPOINT` is reachable (try `curl` / `Invoke-WebRequest`).

### Capturing Run Output for Troubleshooting

Redirect the console output to a file with `--log-level debug` — this is the recommended way to capture output for troubleshooting or a support handoff. Redirect both stdout and stderr so the plain-text log includes the human-readable results and the structured diagnostic lines:

```powershell
# Windows PowerShell — '*>' redirects all streams (stdout + stderr)
python main.py --log-level debug *> my_run.log
```

```bash
# macOS/Linux (bash/zsh) — merge stderr into stdout
python main.py --log-level debug > my_run.log 2>&1
```

> **⚠️ Review before sharing:** debug output may include raw API payloads. Redaction is pattern-based and will not catch arbitrary PII or custom credentials — review the file before sending it.

### Getting Help

- Check the [Azure AI Foundry troubleshooting guide](https://docs.microsoft.com/en-us/azure/ai-services/ai-foundry/troubleshooting)
- Review the Azure AI Projects SDK [troubleshooting section](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/README.md#troubleshooting)

## 📋 Version History

For detailed information about changes, new features, and breaking changes, see [CHANGELOG.md](CHANGELOG.md).

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Ensure all sensitive information uses environment variables
5. Submit a pull request

## 📄 License

This project is licensed under the MIT License - see the LICENSE file for details.