# @qianxude/ai

A TypeScript LLM client with provider-model-task architecture, designed for the Cechat knowledge base toolset.

## Overview

This package provides a modular architecture for making LLM calls. It features:

1. **Provider-Model-Task Architecture** - Flexible configuration for multiple LLM providers
2. **OpenAI-compatible API** - Works with CloudEdge, SiliconFlow, OpenAI, and other compatible services

## Installation

```bash
bun add @qianxude/ai
```

## Architecture

### Package Structure

```
┌─────────────────────────────────────────────────────────────────────────┐
│                         @qianxude/ai                                    │
├──────────────────────────┬──────────────────────────────────────────────┤
│    Client                │      Types                                   │
│  (@qianxude/ai/client)   │   (@qianxude/ai)                             │
├──────────────────────────┼──────────────────────────────────────────────┤
│ LLMClient                │                                              │
│ ConfigLoader             │   CompletionOptions                          │
│ OpenAICompatibleProvider │                                              │
│ Provider-Model-Task      │   LLMConfig                                  │
└──────────────────────────┴──────────────────────────────────────────────┘
```

### Provider-Model-Task Architecture

The new LLM client uses a three-layer configuration architecture that separates concerns and enables flexible model management:

```
┌────────────────────────────────────────────────────────────────┐
│                         Task Layer                             │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐            │
│  │classification│ │  reasoning   │ │ generation   │            │
│  └──────┬───────┘ └──────┬───────┘ └──────┬───────┘            │
└─────────┼────────────────┼────────────────┼────────────────────┘
          │                │                │
          ▼                ▼                ▼
┌────────────────────────────────────────────────────────────────┐
│                        Model Layer                             │
│  ┌──────────────────┐        ┌──────────────────┐              │
│  │  deepseek-chat   │        │ deepseek-reasoner│              │
│  │  temperature: 0.3│        │ temperature: 0.1 │              │
│  │  maxTokens: 2000 │        │ maxTokens: 4096  │              │
│  └──────┬───────────┘        └──────┬───────────┘              │
└─────────┼───────────────────────────┼──────────────────────────┘
          │                           │
          └───────────┬───────────────┘
                      │
                      ▼
┌────────────────────────────────────────────────────────────────┐
│                       Provider Layer                           │
│  ┌──────────────────────────────────────────────────────┐      │
│  │                      ce                              │      │
│  │  apiKeyEnv: CE_API_KEY                               │      │
│  │  baseUrlEnv: CE_BASE_URL                             │      │
│  └──────────────────────────────────────────────────────┘      │
└────────────────────────────────────────────────────────────────┘
```

#### How It Works

1. **Task Layer** - Semantic abstraction for model selection
   - Tasks represent high-level use cases (e.g., `classification`, `reasoning`)
   - Tasks map to specific models in the configuration
   - Allows changing models without modifying code

2. **Model Layer** - Model-specific configuration
   - Each model defines its parameters (temperature, maxTokens, etc.)
   - Models reference a provider for API connectivity
   - Multiple models can use the same provider

3. **Provider Layer** - API endpoint configuration
   - Defines environment variable names for API keys and URLs
   - All providers use OpenAI-compatible API protocol
   - Easy to switch between different LLM services

### Core Components

1. **LLM Client** (`@qianxude/ai/client`) - Provider-model-task based client with multi-provider support
2. **Type Definitions** (`@qianxude/ai`) - Shared TypeScript types and types

## Configuration

### Using llm.manifest.json (Required)

The LLM client requires a `llm.manifest.json` configuration file. You must set the `LLM_MANIFEST` environment variable to point to this file.

#### 1. Create llm.manifest.json

```json
{
  "version": "1.0",
  "providers": {
    "ce": {
      "name": "CloudEdge",
      "apiKeyEnv": "CE_API_KEY",
      "baseUrlEnv": "CE_BASE_URL"
    }
  },
  "models": {
    "deepseek-chat": {
      "provider": "ce",
      "model": "deepseek-v3.2-671b",
      "temperature": 0.3,
      "maxTokens": 2000
    },
    "deepseek-reasoner": {
      "provider": "ce",
      "model": "DeepSeek-R1-671B",
      "temperature": 0.1,
      "maxTokens": 4096
    }
  },
  "tasks": {
    "classification": "deepseek-chat",
    "generation": "deepseek-chat",
    "reasoning": "deepseek-reasoner",
    "summarization": "deepseek-chat"
  }
}
```

#### 2. Set Environment Variables

| Variable              | Required | Description                                        |
| --------------------- | -------- | -------------------------------------------------- |
| `LLM_MANIFEST` | **Yes**  | Path to your `llm.manifest.json` file                    |
| `CE_API_KEY`          | Yes\*    | API authentication key (required by ce provider)   |
| `CE_BASE_URL`         | Yes\*    | Base URL for the LLM API (required by ce provider) |

\* Required only if using the `ce` provider. Each provider defines its own required environment variables in `llm.manifest.json` via `apiKeyEnv` and `baseUrlEnv`.

#### Example

```bash
export LLM_MANIFEST=./llm.manifest.json
export CE_API_KEY=your_api_key
export CE_BASE_URL=https://api.ce.example.com
```

### Configuration Schema

#### `providers`

Providers define API endpoints and credential sources:

```json
{
  "providers": {
    "ce": {
      "name": "CloudEdge",
      "apiKeyEnv": "CE_API_KEY",
      "baseUrlEnv": "CE_BASE_URL"
    },
    "sf": {
      "name": "SiliconFlow",
      "apiKeyEnv": "SF_API_KEY",
      "baseUrlEnv": "SF_BASE_URL"
    }
  }
}
```

Each provider defines:

- `name`: Human-readable name for logging
- `apiKeyEnv`: Environment variable name containing the API key
- `baseUrlEnv`: Environment variable name containing the base URL

#### `models`

Models define LLM configurations and reference providers:

```json
{
  "models": {
    "deepseek-chat": {
      "provider": "ce",
      "model": "deepseek-v3.2-671b",
      "temperature": 0.3,
      "maxTokens": 2000,
      "topP": 0.9
    },
    "deepseek-reasoner": {
      "provider": "ce",
      "model": "DeepSeek-R1-671B",
      "temperature": 0.1,
      "maxTokens": 4096
    }
  }
}
```

Each model defines:

- `provider`: Reference to provider key (must exist in `providers`)
- `model`: Actual model name/id for the API
- `temperature`: Default sampling temperature (0-2)
- `maxTokens`: Default maximum tokens per request
- `topP`: Optional nucleus sampling parameter (0-1)

#### `tasks`

Tasks provide semantic model selection:

```json
{
  "tasks": {
    "classification": "deepseek-chat",
    "generation": "deepseek-chat",
    "reasoning": "deepseek-reasoner",
    "summarization": "deepseek-chat",
    "extraction": "deepseek-chat",
    "translation": "deepseek-chat"
  }
}
```

Task-to-model mappings:

- Key: Task name representing a use case
- Value: Either a simple model string or an object with model and API options

**Simple format (backward compatible):**

```json
{
  "tasks": {
    "classification": "deepseek-chat"
  }
}
```

**Object format with API options:**

```json
{
  "tasks": {
    "generation": {
      "model": "deepseek-chat",
      "temperature": 0.7,
      "maxTokens": 2000
    },
    "reasoning": {
      "model": "deepseek-reasoner",
      "temperature": 0.05
    }
  }
}
```

When using the object format:
- `model` (required): Model ID from the `models` section
- `temperature`, `maxTokens`, `topK`, `topP` (optional): Override model-level defaults for this specific task

This abstraction allows:

1. **Model switching without code changes** - Update the task mapping to use a different model
2. **A/B testing** - Route specific tasks to different models for comparison
3. **Cost optimization** - Use cheaper models for simple tasks, expensive ones for complex tasks
4. **Fallback strategies** - Define multiple tasks for the same use case with different model tiers

### Advanced Configuration Examples

#### Multi-Provider Setup

```json
{
  "version": "1.0",
  "providers": {
    "ce": {
      "name": "CloudEdge",
      "apiKeyEnv": "CE_API_KEY",
      "baseUrlEnv": "CE_BASE_URL"
    },
    "sf": {
      "name": "SiliconFlow",
      "apiKeyEnv": "SF_API_KEY",
      "baseUrlEnv": "SF_BASE_URL"
    }
  },
  "models": {
    "deepseek-chat": {
      "provider": "ce",
      "model": "deepseek-v3.2-671b",
      "temperature": 0.3,
      "maxTokens": 2000
    },
    "qwen-max": {
      "provider": "sf",
      "model": "qwen-max",
      "temperature": 0.3,
      "maxTokens": 2000
    }
  },
  "tasks": {
    "classification": "deepseek-chat",
    "generation": "qwen-max"
  }
}
```

#### Task-Based Model Selection by Complexity

```json
{
  "version": "1.0",
  "providers": {
    "ce": {
      "name": "CloudEdge",
      "apiKeyEnv": "CE_API_KEY",
      "baseUrlEnv": "CE_BASE_URL"
    }
  },
  "models": {
    "deepseek-chat": {
      "provider": "ce",
      "model": "deepseek-v3.2-671b",
      "temperature": 0.3,
      "maxTokens": 1000
    },
    "deepseek-reasoner": {
      "provider": "ce",
      "model": "DeepSeek-R1-671B",
      "temperature": 0.1,
      "maxTokens": 4000
    }
  },
  "tasks": {
    "simple_classification": "deepseek-chat",
    "complex_reasoning": "deepseek-reasoner",
    "code_generation": "deepseek-reasoner",
    "text_summarization": "deepseek-chat"
  }
}
```

## Usage

### Basic LLM Client

```typescript
import { createLLMClient } from '@qianxude/ai/client';

// Create client (uses LLM_MANIFEST env var)
const llm = createLLMClient();
await llm.initialize();

// Simple completion (uses first available model)
const response = await llm.complete('Explain TypeScript generics');
console.log(response.content);

// Task-based model selection
const result = await llm.complete('Classify this text', { task: 'classification' });

// Direct model selection
const result2 = await llm.complete('Write a function', {
  model: 'deepseek-reasoner',
  temperature: 0.7,
  maxTokens: 1000,
});
```

### With Custom Config Path (Overrides Env Var)

```typescript
const llm = createLLMClient({ manifestPath: './config/llm.manifest.json' });
await llm.initialize();
```

### With Logger

```typescript
const llm = createLLMClient({ logger: consoleLogger });
await llm.initialize();
```

### Complete with Messages

```typescript
import { LLMMessage } from '@qianxude/ai/client';

const messages: LLMMessage[] = [
  { role: 'system', content: 'You are a helpful assistant' },
  { role: 'user', content: 'Hello!' },
];

const response = await llm.complete(messages, { task: 'generation' });
```

### Accessing Configuration

```typescript
// Get model for a task
const modelId = llm.getModelForTask('reasoning'); // 'deepseek-reasoner'

// Get full config
const config = llm.getConfig();

// Get tasks using a specific model
const tasks = configLoader.getTasksForModel('deepseek-chat');
// Returns: ['classification', 'generation', 'summarization']
```

### Runtime Model Resolution

When you call `complete()`, the client resolves the model using this priority:

1. **Explicit model ID** - If `options.model` is provided, use it directly
2. **Task-based selection** - If `options.task` is provided, look up the model in config
3. **First available model** - Use the first model defined in `llm.manifest.json`

```typescript
// Priority 1: Direct model selection
await llm.complete('Hello', { model: 'deepseek-reasoner' });

// Priority 2: Task-based selection
await llm.complete('Hello', { task: 'classification' });

// Priority 3: First available model
await llm.complete('Hello');
```

This resolution happens at runtime, allowing dynamic model selection based on context.

### Migrating from Legacy to New Architecture

If you were previously using the legacy client, here are the key changes:

**Legacy approach (removed):**

```typescript
// This code no longer works - legacy client has been removed
const llm = createLegacyLLMClient(console);
const response = await llm.complete('Hello'); // Returns string
```

**New approach:**

```typescript
import { createLLMClient } from '@qianxude/ai/client';

const llm = createLLMClient({ logger: console });
await llm.initialize();
const response = await llm.complete('Hello'); // Returns LLMResponse
console.log(response.content);
```

**Key differences:**

1. New client requires `initialize()` call before use
2. New client returns `LLMResponse` objects with `content` and `usage`
3. New client supports task-based model selection via `llm.manifest.json`
4. New client uses options object instead of logger as first parameter
5. New client requires a `llm.manifest.json` configuration file

## API Reference

### TaskBasedLLMClient (New Interface)

The primary interface for the new provider-model-task architecture:

```typescript
interface TaskBasedLLMClient {
  initialize(): Promise<void>;
  complete(input: string | LLMMessage[], options?: CompletionOptions): Promise<LLMResponse>;
  stream(input: string | LLMMessage[], options?: CompletionOptions): AsyncGenerator<LLMStreamChunk>;
  getModelForTask(task: string): string;
  getConfig(): LLMConfig;
}
```

**Methods:**

- `initialize()` - Load and validate configuration from `llm.manifest.json`
- `complete()` - Single prompt completion or multi-turn conversation completion with task-based model selection. Accepts either a string prompt or an array of messages.
- `stream()` - Stream completion with task-based model selection. Accepts either a string prompt or an array of messages.
- `getModelForTask()` - Get model ID for a configured task
- `getConfig()` - Get full configuration object

### LLMClient

Implementation of `TaskBasedLLMClient`:

```typescript
class LLMClient implements TaskBasedLLMClient {
  constructor(options?: LLMClientOptions);

  initialize(): Promise<void>;
  complete(input: string | LLMMessage[], options?: CompletionOptions): Promise<LLMResponse>;
  stream(input: string | LLMMessage[], options?: CompletionOptions): AsyncGenerator<LLMStreamChunk>;
  getModelForTask(task: string): string;
  getConfig(): LLMConfig;
}
```

### ConfigLoader

```typescript
class ConfigLoader {
  constructor(configPath: string);

  load(): Promise<LLMConfig>;
  resolveModel(modelId: string): ResolvedModelConfig;
  getModelForTask(task: string): string;
  getTasksForModel(modelId: string): string[];
  getConfig(): LLMConfig;
}
```

### OpenAICompatibleProvider

```typescript
class OpenAICompatibleProvider implements ProviderClient {
  complete(config: ResolvedModelConfig, call: LLMRequest): Promise<LLMResponse>;
  stream(config: ResolvedModelConfig, call: LLMRequest): AsyncGenerator<LLMStreamChunk>;
}
```

## Types

### Core Types

```typescript
interface CompletionOptions {
  task?: string; // Task-based model selection (e.g., 'classification')
  model?: string; // Direct model selection (e.g., 'deepseek-chat')
  systemPrompt?: string; // System prompt for the conversation
  temperature?: number; // Sampling temperature (0-2)
  topP?: number; // Nucleus sampling (0-1)
  maxTokens?: number; // Maximum tokens to generate
}

interface LLMResponse {
  content: string;
  usage?: {
    promptTokens: number;
    completionTokens: number;
    totalTokens: number;
  };
  model: string;
}

interface LLMConfig {
  version: string;
  providers: Record<string, LLMProvider>;
  models: Record<string, LLMModel>;
  tasks: Record<string, string>;
}

interface LLMProvider {
  name: string;
  apiKeyEnv: string;
  baseUrlEnv: string;
}

interface LLMModel {
  provider: string;
  model: string;
  temperature: number;
  maxTokens: number;
  topP?: number;
}

interface TaskBasedLLMClient {
  initialize(): Promise<void>;
  complete(input: string | LLMMessage[], options?: CompletionOptions): Promise<LLMResponse>;
  stream(input: string | LLMMessage[], options?: CompletionOptions): AsyncGenerator<LLMStreamChunk>;
  getModelForTask(task: string): string;
  getConfig(): LLMConfig;
}

interface LLMMessage {
  role: 'system' | 'user' | 'assistant';
  content: string;
}
```

### Error Types

```typescript
class LLMError extends Error {
  readonly code: string;
  readonly cause?: unknown;
}

class ConfigurationError extends LLMError {
  // Configuration-related errors
}

class ProviderError extends LLMError {
  readonly provider: string;
  // Provider API errors
}
```

## Error Handling

```typescript
import { createLLMClient, ConfigurationError, ProviderError } from '@qianxude/ai/client';

const llm = createLLMClient();

try {
  await llm.initialize();
  const response = await llm.complete('Hello');
} catch (error) {
  if (error instanceof ConfigurationError) {
    console.error('Config error:', error.message);
  } else if (error instanceof ProviderError) {
    console.error('Provider error:', error.message, 'Provider:', error.provider);
  } else {
    console.error('Unexpected error:', error);
  }
}
```

## License

Private package for Cechat internal use.