# JSON2TOON v2.0 🚀

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![MCP Compatible](https://img.shields.io/badge/MCP-2.0-green.svg)](https://modelcontextprotocol.io)
[![Code Coverage](https://img.shields.io/badge/coverage-95%25-brightgreen.svg)]()

**Advanced Token-Optimized Object Notation** - The most powerful JSON compression system for AI context management.

JSON2TOON is a next-generation MCP server that revolutionizes JSON compression with AI-powered pattern detection, achieving **75-85% token reduction** while maintaining perfect data integrity.

---

## ✨ Key Features

### 🎯 **4 Compression Levels**
- **MINIMAL** (30-40% savings): Lightning-fast key abbreviations
- **STANDARD** (40-60% savings): Balanced performance + compression
- **AGGRESSIVE** (60-75% savings): Advanced pattern optimization
- **EXTREME** (75-85% savings): Maximum compression with zlib

### 🤖 **AI-Powered Pattern Detection**
- **17+ Pattern Types**: API responses, databases, time series, graphs, trees, and more
- **Smart Strategy Selection**: Automatic optimization based on data structure
- **Confidence Scoring**: Each pattern comes with accuracy metrics
- **Compression Potential**: Estimates savings before conversion

### 🔧 **12 Advanced MCP Tools**
1. `convert_to_toon` - Multi-level JSON compression
2. `convert_to_json` - Lossless decompression
3. `analyze_patterns` - Deep pattern analysis with AI
4. `get_optimal_strategy` - AI-recommended compression plan
5. `calculate_metrics` - Detailed compression statistics
6. `batch_convert` - High-performance batch processing
7. `smart_optimize` - Auto-detect and apply best compression
8. `compare_levels` - Side-by-side level comparison
9. `validate_toon` - Format validation + round-trip testing
10. `suggest_abbreviations` - Custom abbreviation generation
11. `estimate_savings` - Pre-conversion savings estimation
12. `get_server_stats` - Real-time performance metrics

### 💡 **Advanced Capabilities**
- **150+ Key Abbreviations** (vs 68 in TOON v1.0)
- **String Dictionary**: De-duplication for repeated values
- **Partial Schema Compression**: Works with inconsistent data
- **Value Pattern Compression**: Optimizes timestamps, UUIDs, URLs, emails
- **Reference System**: Eliminates duplicate structures
- **zlib Integration**: Optional extreme compression

---

## 📊 Performance Benchmarks

| Data Type | Compression | Speed | Round-Trip |
|-----------|------------|-------|------------|
| **API Responses** | 50-65% | 0.3ms/KB | ✅ Perfect |
| **Database Results** | 60-70% | 0.3ms/KB | ✅ Perfect |
| **Time Series** | 65-75% | 0.5ms/KB | ✅ Perfect |
| **User Profiles** | 45-55% | 0.3ms/KB | ✅ Perfect |
| **Config Files** | 40-55% | 0.1ms/KB | ✅ Perfect |

---

## 🚀 Quick Start

### Installation

```bash
# Clone repository
git clone https://github.com/muhammedehab35/JSON2TOON-MCP.git
cd json2toon

# Install with pip
pip install -e .

# Or use Docker
docker-compose up -d
```

### MCP Configuration

Add to your Claude Desktop config (`~/.config/Claude/claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "json2toon": {
      "command": "python",
      "args": ["-m", "src.mcp_server"],
      "cwd": "/path/to/json2toon"
    }
  }
}
```

**Docker Configuration**:
```json
{
  "mcpServers": {
    "json2toon": {
      "command": "docker",
      "args": ["run", "-i", "json2toon:2.0.0"]
    }
  }
}
```

---

## 💻 Usage Examples

### Basic Conversion

```python
from src.advanced_converter import convert_json_to_toon, convert_toon_to_json, CompressionLevel

# Simple conversion with STANDARD level
data = {
    "id": 12345,
    "name": "John Doe",
    "email": "john@example.com",
    "created_at": "2025-01-01T00:00:00Z"
}

# Convert to TOON
toon = convert_json_to_toon(data, level=CompressionLevel.STANDARD)
print(f"Compressed: {toon}")

# Convert back to JSON
original = convert_toon_to_json(toon)
print(f"Restored: {original}")
```

### Advanced Pattern Analysis

```python
from src.pattern_analyzer import AdvancedPatternAnalyzer

analyzer = AdvancedPatternAnalyzer()

# Analyze your data
patterns = analyzer.analyze(large_json_data)

# Get compression strategy
strategy = analyzer.get_compression_strategy(large_json_data)

print(f"Detected {len(patterns)} patterns")
print(f"Expected savings: {strategy.expected_savings * 100:.1f}%")
print(f"Recommended level: {strategy.recommended_level}")
print(f"Reasoning: {strategy.reasoning}")
```

### Smart Optimization

```python
from src.optimizer import SmartOptimizer

optimizer = SmartOptimizer()

# Automatic optimization with profile
result = optimizer.optimize(data, profile="balanced")
# Profiles: "speed", "balanced", "size"

print(f"Used profile: {result['profile_used']}")
print(f"Selected level: {result['level_selected']}")
print(f"Savings: {result['metrics']['savings_percent']:.1f}%")
```

### Batch Processing

```python
from src.advanced_converter import AdvancedTOONConverter, CompressionLevel

converter = AdvancedTOONConverter(level=CompressionLevel.AGGRESSIVE)

# Process multiple items
items = [
    {"id": i, "data": f"Item {i}"}
    for i in range(1000)
]

for item in items:
    toon = converter.json_to_toon(item)
    # Process compressed data
```

---

## 🔬 MCP Tools Examples

### In Claude Code

#### 1. Convert with Custom Level
```
Use the convert_to_toon tool with:
- json_data: <your JSON>
- level: 3 (AGGRESSIVE)
```

#### 2. Analyze Patterns
```
Use the analyze_patterns tool to detect:
- Pattern types
- Compression potential
- Optimization recommendations
```

#### 3. Compare All Levels
```
Use the compare_levels tool to see:
- Side-by-side comparison
- Savings per level
- Best recommendation
```

#### 4. Smart Auto-Optimize
```
Use the smart_optimize tool with:
- json_data: <your JSON>
- profile: "size" (for maximum compression)
```

---

## 📖 Format Specification

### TOON v2.0 Structure

```json
{
  "_toon": "2.0",           // Version identifier
  "_lvl": 2,                // Compression level used
  "d": {...},               // Compressed data
  "_refs": {...},           // Optional: structure references
  "_dict": {...}            // Optional: string dictionary
}
```

### Key Abbreviations (Sample)

| Original | TOON | Original | TOON | Original | TOON |
|----------|------|----------|------|----------|------|
| id | i | email | eml | status | s |
| name | n | phone | ph | created_at | ca |
| type | t | address | addr | updated_at | ua |
| value | v | username | unm | timestamp | ts |

**150+ abbreviations** covering common API, database, and application fields.

### Value Optimizations

- `null` → `~`
- `true` → `T`, `false` → `F`
- Timestamps: `$ts:2025-01-01T00:00:00Z`
- UUIDs: `$uid:550e8400-e29b-41d4-a716-446655440000`
- String refs: `@s0`, `@s1` (from dictionary)

### Schema Compression

**Before:**
```json
[
  {"id": 1, "name": "Alice", "email": "alice@test.com"},
  {"id": 2, "name": "Bob", "email": "bob@test.com"},
  {"id": 3, "name": "Carol", "email": "carol@test.com"}
]
```

**After (TOON):**
```json
{
  "_sch": ["i", "n", "eml"],
  "_dat": [
    [1, "Alice", "alice@test.com"],
    [2, "Bob", "bob@test.com"],
    [3, "Carol", "carol@test.com"]
  ]
}
```

**Savings**: ~55-60% for arrays with consistent schemas

---

## 🧪 Testing

```bash
# Run all tests
pytest tests/ -v

# With coverage
pytest tests/ --cov=src --cov-report=html

# Specific test file
pytest tests/test_converter.py -v

# Run tests in Docker
docker-compose run json2toon-server pytest tests/ -v
```

### Test Coverage

- ✅ **Converter**: 100+ test cases covering all compression levels
- ✅ **Pattern Analyzer**: 30+ tests for all 17 pattern types
- ✅ **Round-trip**: Perfect data integrity verification
- ✅ **Edge cases**: Unicode, large numbers, special characters
- ✅ **Performance**: Benchmarks for all levels

---

## 🐳 Docker Deployment

### Build Image
```bash
docker build -t json2toon:2.0.0 .
```

### Run with Docker Compose
```bash
# Production mode
docker-compose up -d json2toon-server

# Development mode
docker-compose --profile dev up json2toon-dev
```

### Docker Features
- ✅ Python 3.11 optimized image
- ✅ Non-root user for security
- ✅ Health checks
- ✅ Resource limits (2 CPU, 1GB RAM)
- ✅ Logging configuration
- ✅ Development mode with live reload

---

## 📐 Architecture

```
┌─────────────────────────────────────────┐
│         JSON2TOON MCP Server            │
│              (v2.0)                     │
└─────────────┬───────────────────────────┘
              │
    ┌─────────┼─────────┐
    │         │         │
    ▼         ▼         ▼
┌─────────┐ ┌──────────┐ ┌──────────┐
│Advanced │ │Pattern   │ │Smart     │
│Converter│ │Analyzer  │ │Optimizer │
└─────────┘ └──────────┘ └──────────┘
    │         │              │
    └─────────┴──────────────┘
              │
    ┌─────────┼─────────┐
    ▼         ▼         ▼
┌──────┐  ┌──────┐  ┌──────┐
│Schema│  │String│  │Value │
│Comp  │  │ Dict │  │ Comp │
└──────┘  └──────┘  └──────┘
```

---

## 🎯 Pattern Types Detected

1. **API Response** - REST, GraphQL, JSON-RPC
2. **Database Record** - CRUD, audit logs, versioned
3. **User Data** - Profiles, auth, preferences
4. **Pagination** - Page-based, offset-based
5. **Nested Address** - Street, city, state, country
6. **Nested Coordinates** - Lat/lng/alt
7. **Nested Dimensions** - Width/height/depth
8. **Nested Metadata** - Created/updated by, tags
9. **Homogeneous Array** - Same-type elements
10. **Consistent Schema Array** - Similar object structures
11. **Repeated Structure** - Duplicate patterns
12. **Time Series** - Temporal data sequences
13. **Graph Node** - Network/graph structures
14. **Tree Structure** - Hierarchical data
15. **Enum Values** - Limited value sets
16. **Sparse Array** - Many null/empty values
17. **Deep Nesting** - Complex nested levels

---

## 🔧 Development

### Setup Development Environment

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Format code
black src/ tests/

# Lint
ruff src/ tests/

# Type check
mypy src/
```

### Code Quality Tools

- **black**: Code formatting (line length: 100)
- **ruff**: Fast Python linter
- **mypy**: Static type checking (strict mode)
- **pytest**: Testing framework with async support

---

## 📊 Comparison with TOON v1.0

| Feature | TOON v1.0 | JSON2TOON v2.0 |
|---------|-----------|----------------|
| **Compression Levels** | 2 | 4 |
| **Key Abbreviations** | 68 | 150+ |
| **Pattern Types** | 8 | 17+ |
| **MCP Tools** | 6 | 12 |
| **Max Savings** | 60% | 85% |
| **String Dictionary** | ❌ | ✅ |
| **Value Compression** | ❌ | ✅ |
| **Partial Schema** | ❌ | ✅ |
| **zlib Support** | ❌ | ✅ |
| **AI Analysis** | Basic | Advanced |
| **Custom Abbreviations** | ❌ | ✅ |
| **Savings Estimation** | ❌ | ✅ |

---

## 🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Run tests (`pytest tests/ -v`)
4. Format code (`black src/ tests/`)
5. Commit changes (`git commit -m 'Add amazing feature'`)
6. Push to branch (`git push origin feature/amazing-feature`)
7. Open a Pull Request

---

## 🌟 Use Cases

### 1. **Large API Responses**
Save 50-65% tokens when storing API responses in Claude conversations.

### 2. **Database Query Results**
Compress database results by 60-70% for efficient context usage.

### 3. **Time Series Data**
Achieve 65-75% compression on temporal datasets.

### 4. **Configuration Files**
Store configs in compact format with 40-55% savings.

### 5. **Codebase Analysis**
Fit more file contents in token limits when analyzing code.

### 6. **Log Processing**
Compress structured logs by 50-60% for pattern analysis.

---

## 🚦 Quick Tips

### When to Use Each Level

- **MINIMAL**: Quick conversions, need high speed
- **STANDARD**: General purpose (best balance)
- **AGGRESSIVE**: Large datasets, high savings needed
- **EXTREME**: Maximum compression, archival use

### Optimization Profiles

- **speed**: Prefer MINIMAL/STANDARD levels
- **balanced**: Auto-select based on data (recommended)
- **size**: Prefer AGGRESSIVE/EXTREME levels

### Best Practices

1. ✅ Analyze patterns first with `analyze_patterns`
2. ✅ Use `smart_optimize` for automatic best results
3. ✅ Validate with `validate_toon` after conversion
4. ✅ Use `estimate_savings` before large batch jobs
5. ✅ Monitor with `get_server_stats` for metrics


```bash
pip install -e .
python -m src.mcp_server
```