# ๐ฏ model-orch-sdk
[](https://www.npmjs.com/package/model-orch-sdk)
[](LICENSE)
[](package.json)
[](https://www.typescriptlang.org/)
**Enterprise-Grade Multi-Model Orchestration** - Route requests intelligently across 19+ LLM providers with sophisticated strategies, ensemble logic, automatic failover, cost optimization, and production-ready observability.
---
## ๐ Overview
`model-orch-sdk` is a comprehensive TypeScript SDK for orchestrating multiple LLM and AI model endpoints. It enables teams to register many model providers (OpenAI, Anthropic, Google, Azure, custom endpoints), configure sophisticated routing and ensemble strategies, manage credentials securely, and operate models reliably with monitoring, cost controls, fallback policies, and A/B/canary experiments.
### Key Capabilities
- **๐ Multi-Model Support**: Register any LLM API (OpenAI, Anthropic, Google, Azure, Cohere, HuggingFace, Replicate, custom HTTP/gRPC, local models)
- **๐ฏ Intelligent Routing**: Percent splits, conditional routing, confidence cascading, cost/latency optimization
- **๐ค Ensemble Strategies**: Voting, confidence-weighted aggregation, rankers, synthesizers
- **๐ Reliability**: Circuit breakers, automatic fallbacks, retry with exponential backoff
- **๐ฐ Cost Control**: Per-request and per-project budgets, cost estimation, quota management
- **๐ Observability**: Distributed tracing, metrics, telemetry, audit logs
- **๐ Security**: Encrypted credential vault, RBAC, data residency rules, rotation policies
- **๐งช Experimentation**: A/B tests, canary deployments, traffic splits with sticky sessions
- **โก Performance**: Rate limiting, token buckets, caching, concurrent request management
---
## ๐ Prerequisites
- **Node.js** 18 or higher
- **TypeScript** 5.0+ (for development)
---
## ๐ Installation
```bash
npm install model-orch-sdk
```
Or with yarn:
```bash
yarn add model-orch-sdk
```
---
## ๐ฎ Quick Start
### Basic Usage
```typescript
import {
CredentialVault,
TokenBucketRateLimiter,
CircuitBreaker
} from 'model-orch-sdk';
// Initialize credential vault
const vault = new CredentialVault({
encryptionKey: process.env.ENCRYPTION_KEY!,
enableAuditLog: true,
maxAccessLogSize: 1000,
});
// Store API credentials
await vault.storeCredential({
projectId: 'proj_123',
name: 'OpenAI Key',
type: 'api_key',
value: process.env.OPENAI_API_KEY!,
createdBy: 'admin',
});
// Set up rate limiting
const rateLimiter = new TokenBucketRateLimiter(
100, // capacity
10 // refill rate per second
);
// Create circuit breaker
const breaker = new CircuitBreaker(
5, // failure threshold
2, // success threshold
30000 // reset timeout (30s)
);
```
### Routing Configuration
```typescript
import type {
PercentSplitConfig,
RoutingPolicyType
} from 'model-orch-sdk';
// Define a percent split routing policy
const routingPolicy: PercentSplitConfig = {
type: RoutingPolicyType.PERCENT_SPLIT,
splits: [
{ connectorId: 'gpt-4', percentage: 70, weight: 0.7 },
{ connectorId: 'claude-3', percentage: 30, weight: 0.3 },
],
stickySession: true,
sessionKey: 'userId',
};
```
### Ensemble Strategy
```typescript
import type {
EnsembleRoutingConfig,
VotingConfig,
EnsembleStrategyType
} from 'model-orch-sdk';
// Configure ensemble with voting
const ensembleConfig: EnsembleRoutingConfig = {
type: RoutingPolicyType.ENSEMBLE,
connectorIds: ['gpt-4', 'claude-3', 'gemini-pro'],
executionMode: 'parallel',
aggregationStrategy: EnsembleStrategyType.VOTING,
aggregationConfig: {
type: EnsembleStrategyType.VOTING,
votingMethod: 'majority',
normalization: 'semantic',
tieBreaker: 'highest_confidence',
} as VotingConfig,
timeout: 30000,
minSuccessful: 2,
};
```
---
## ๐ Core Concepts
### 1. **Project**
Top-level workspace grouping models, policies, quotas, and telemetry. Each organization can have multiple projects with isolated configurations.
### 2. **Model Connector**
Registered endpoint with metadata (type, endpoint URL, credentials, rate limits, cost info). Supports all major LLM providers and custom endpoints.
### 3. **Model Pool**
Logical group of connectors serving a single role (e.g., `high_quality_llms`, `cheap_fallback`, `embedding_engines`).
### 4. **Routing Policy**
Declarative rule set determining which model(s) to call for incoming requests:
- **Percent Split**: Distribute traffic by percentages (70/30, A/B tests)
- **Conditional**: Route based on input attributes (language, length, topic)
- **Confidence Cascade**: Try cheaper models first, escalate if confidence is low
- **Cost/Latency Based**: Optimize for budget or speed
- **Ensemble**: Call multiple models and aggregate responses
### 5. **Ensemble Strategy**
How to combine responses when multiple models are invoked:
- **Voting**: Majority, plurality, or unanimous voting
- **Confidence-Weighted**: Weight outputs by model confidence scores
- **Ranker**: Use learned model to score and rank outputs
- **Synthesizer**: Call a high-quality model to merge outputs
- **Diversity**: Select diverse candidates for creative tasks
- **Earliest**: Return first acceptable response
### 6. **Fallback Policy**
Automatic failover when primary model fails, times out, or exceeds budget. Define fallback chains with multiple backup options.
### 7. **Circuit Breaker**
Automatically disable failing connectors to prevent cascading failures. Auto-heal when service recovers.
### 8. **Credential Vault**
Secure, encrypted storage for API keys with audit logs, rotation policies, and RBAC.
---
## ๐๏ธ Architecture
### High-Level Architecture
The SDK follows a layered architecture:
- **API Layer**: REST, GraphQL, and WebSocket interfaces for client applications
- **Core Orchestrator**: Central coordination engine for request routing and execution
- **Middleware Layer**: Caching, rate limiting, cost tracking, load balancing, circuit breakers, and retry logic
- **Policy Engine**: Intelligent routing, security, and compliance policies
- **Provider Layer**: Integration with multiple LLM providers (OpenAI, Anthropic, Cohere, Google, Groq, Together AI, etc.)
- **Storage Layer**: Redis cache, PostgreSQL database, and MongoDB for persistent storage
- **Monitoring**: Comprehensive logging, metrics, and observability
### Request Processing Flow
Every request goes through a comprehensive processing pipeline:
1. **Authentication**: Verify API keys and user credentials
2. **Rate Limiting**: Check and enforce rate limits
3. **Cache Check**: Look for cached responses (3-tier cache)
4. **Policy Evaluation**: Apply routing and security policies
5. **Provider Selection**: Choose optimal model based on policies
6. **Load Balancing**: Distribute load across available providers
7. **Circuit Breaker**: Prevent requests to failing services
8. **Provider Request**: Execute request with retry logic
### Policy Engine
The Policy Engine supports multiple policy types:
- **Routing Policies**: Cost-based, latency-based, quality-based routing
- **Security Policies**: Authentication, authorization, data filtering
- **Cost Policies**: Budget limits, cost optimization, quota management
- **Rate Limit Policies**: Per-user, per-model, per-tenant limits
- **Custom Policies**: Business rules, compliance, user-defined logic
### Caching Strategy
Multi-layer caching for optimal performance:
- **L1 (Memory Cache)**: In-process LRU cache (~10ms latency)
- **L2 (Redis Cache)**: Distributed cache (~50ms latency)
- **L3 (Semantic Cache)**: Similar query matching (~100ms latency)
### Cloud Deployment
Production-ready deployment on AWS with Kubernetes:
- **Load Balancing**: Application Load Balancer with health checks
- **Kubernetes**: Multi-replica services for high availability
- **Storage**: Redis cluster, PostgreSQL RDS, S3, ElastiCache
- **Monitoring**: Prometheus, Grafana, Jaeger, ELK Stack, AlertManager
- **CI/CD**: GitHub Actions with automated testing and deployment
- **Infrastructure**: Terraform for infrastructure as code
---
## ๐ง Configuration
### Environment Variables
Create a `.env` file (see `.env.example`):
```bash
# Model API Keys
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_AI_API_KEY=...
AZURE_OPENAI_API_KEY=...
COHERE_API_KEY=...
# Security
CREDENTIAL_ENCRYPTION_KEY=<64-char-hex-string>
# Observability
ENABLE_TELEMETRY=true
METRICS_EXPORT_INTERVAL=60000
```
Generate encryption key:
```typescript
import { CredentialVault } from 'model-orch-sdk';
const key = CredentialVault.generateKey();
console.log('CREDENTIAL_ENCRYPTION_KEY=' + key);
```
---
## ๐ API Documentation
### Type Definitions
All types are exported from the main package:
```typescript
import type {
// Core
Project,
ModelConnector,
ModelPool,
RoutingPolicy,
// Requests
OrchestrationRequest,
RequestPayload,
RequestContext,
// Responses
OrchestrationResponse,
ModelResponse,
ExecutionTrace,
// Configs
PercentSplitConfig,
ConditionalRoutingConfig,
ConfidenceCascadeConfig,
EnsembleRoutingConfig,
// Enums
ModelEndpointType,
RoutingPolicyType,
EnsembleStrategyType,
ConnectorStatus,
CircuitBreakerState,
} from 'model-orch-sdk';
```
### Core Classes
#### CredentialVault
```typescript
const vault = new CredentialVault({
encryptionKey: string,
enableAuditLog: boolean,
maxAccessLogSize: number,
});
// Store credential
await vault.storeCredential({
projectId: string,
name: string,
type: 'api_key' | 'oauth' | 'basic_auth' | 'custom',
value: string,
createdBy: string,
});
// Get credential
const apiKey = await vault.getCredential(id, userId, ipAddress);
// Rotate credential
await vault.rotateCredential(id, newValue, userId, ipAddress);
// Check rotation needed
const needsRotation = vault.needsRotation(id);
```
#### TokenBucketRateLimiter
```typescript
const limiter = new TokenBucketRateLimiter(capacity, refillRate);
// Try to consume tokens
if (limiter.tryConsume(1)) {
// Request allowed
} else {
// Rate limit exceeded
}
// Get available tokens
const available = limiter.getAvailableTokens();
```
#### CircuitBreaker
```typescript
const breaker = new CircuitBreaker(failureThreshold, successThreshold, resetTimeoutMs);
// Check if requests allowed
if (breaker.allowRequest()) {
try {
// Make request
breaker.recordSuccess();
} catch (error) {
breaker.recordFailure();
}
}
// Get state
const { state, failures, successes } = breaker.getState();
```
#### SimpleCache
```typescript
const cache = new SimpleCache(maxSize, ttlSeconds);
// Set value
cache.set(key, value);
// Get value
const cached = cache.get(key);
// Check existence
if (cache.has(key)) {
// Use cached value
}
```
---
## ๐งช Testing
The SDK includes comprehensive utilities for testing:
```bash
# Run all tests
npm test
# Watch mode
npm run test:watch
# Coverage report
npm run test:coverage
```
---
## ๐ก๏ธ Security
- **Encryption**: All credentials encrypted at rest using AES-256-GCM
- **Audit Logs**: Complete trail of credential access and modifications
- **RBAC**: Role-based access control for projects and resources
- **Rotation**: Automatic credential rotation with configurable policies
- **Data Residency**: Control which vendors/regions can process data
- **No Plaintext**: Credentials never logged or stored in plaintext
---
## ๐ Observability
The SDK provides comprehensive telemetry:
- **Request Tracing**: Distributed traces for every orchestration request
- **Metrics**: Latency (p50/p95/p99), error rates, token usage, costs
- **Audit Logs**: Complete history of policy changes, credential access
- **Health Checks**: Automatic monitoring of connector health
- **Circuit Breaker Events**: Track when connectors open/close
- **Experiment Metrics**: A/B test results and statistical significance
---
## ๐ค Contributing
Contributions are welcome! Please see our contribution guidelines:
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'feat: add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
### Development Setup
```bash
git clone https://github.com/gyash1512/model-orch-sdk.git
cd model-orch-sdk
npm install
npm run build
npm test
```
### Code Style
- **TypeScript**: Strict mode enabled, no `any` types allowed
- **Linting**: ESLint with TypeScript rules
- **Formatting**: Prettier for consistent code style
- **Testing**: Jest for unit and integration tests
---
## ๐ License
MIT License - See [LICENSE](LICENSE) file for details.
---
## ๐ Acknowledgments
- OpenAI, Anthropic, Google, and other LLM providers for their APIs
- The TypeScript community for excellent tooling
- Contributors and users of this SDK
---
## ๐ Support
- **Issues**: [GitHub Issues](https://github.com/gyash1512/model-orch-sdk/issues)
- **Discussions**: [GitHub Discussions](https://github.com/gyash1512/model-orch-sdk/discussions)
- **Email**: [Support](mailto:support@example.com)
---
**Built with โค๏ธ using TypeScript**
For detailed API documentation, examples, and guides, visit the [documentation](https://github.com/gyash1512/model-orch-sdk/tree/main/docs).