# ICGYAD - I Can Get You A Date

**Automated appointment booking engine for high-demand government services in Uruguay**

## 🎯 Overview

ICGYAD is a sophisticated automation system designed to detect and book appointments for government services with limited availability and brief release windows. The system monitors appointment availability, solves security challenges using AI, and automatically notifies users or books appointments when slots become available.

### 🔐 **NEW: API Key Authentication System**
ICGYAD now includes a comprehensive API key authentication system for secure access control:
- **API Key Management**: Create, list, and revoke API keys
- **Strategy Permissions**: Control which strategies each API key can access
- **Public/Private Strategies**: Configure strategies as public or require authentication
- **Usage Tracking**: Monitor API key usage and access patterns
- **Enterprise Ready**: Designed for integration into third-party APIs and services

### Initial Use Case
- **Service**: Inscripción de partidas extranjeras - Registro Civil Uruguay
- **Platform**: SAE (Sistema de Agenda Electrónica)
- **Target**: Sede Uruguay 933 appointments that are released during night hours (typically 20:45-00:00)

## 🚀 Features

- **🔐 API Key Authentication**: Secure access control with comprehensive permission management
- **Multi-Strategy Engine**: Extensible framework for different government services
- **Intelligent Detection**: Multiple availability detection methods (DOM, XHR, hidden inputs, API interception)
- **AI-Powered Security**: Automated resolution of security questions using LLM
- **Smart Notifications**: Email alerts with detailed availability information + **immediate notifications**
- **Auto-Booking**: Optional automatic appointment confirmation
- **Robust Scheduling**: Configurable time windows and intervals
- **Comprehensive Logging**: Full audit trail with screenshots and performance metrics
- **Stealth Mode**: Anti-detection measures for reliable operation
- **📦 Library Integration**: Can be used as NPM package for API integration

## 🆕 Recent Updates (v2.1 - Enhanced Retry Loop & CLI Parameters)

### New CLI Parameters & Testing Features
- **⏱️ `--delay` Parameter**: Configure retry delay in seconds (default: 300)
  - Fast testing: `--delay 30` for 30-second intervals
  - Custom intervals: `--delay 120` for 2-minute delays
  - Respects server load while allowing flexible timing
- **📧 `--notify-always` Flag**: Test notification system without waiting for real slots
  - Sends email notifications even when no appointments available
  - Perfect for testing email delivery and notification configuration
  - Validates notification pipeline functionality

### Enhanced Uruguay Strategy (v2.1)
- **🔄 Improved Retry Loop**: Internal retry logic with configurable delays
  - Robust cycle: Security Question → Calendar → Availability Check → Volver → Delay
  - Consistent availability detection between calendar navigation and polling
  - Proper "Volver" button navigation using specific href patterns
  - Comprehensive error handling and logging for debugging
- **🔍 Stable Form Selectors**: Using `name` attributes for maximum reliability
  - `select[name="titular_pais"]`, `input[name="titular_nombres"]`, `input[name="titular_apellidos"]`
  - `select[name="TipoDocumento"]`, `input[name="NroDocumento"]:not([disabled])`
- **🔐 Security Question Detection**: `label[id^="lblpreg_"]` → `input[name="pregunta"]`
- **📍 Location & Schedule**: Role-based selectors for "Box 1 y 2" and "Cualquier horario"
- **⚡ Real-time API Interception**: `agenda_sae_api_disponibilidades` endpoint monitoring
- **🚨 Immediate Notifications**: Instant alerts upon availability detection (`hasSlots === true`)
- **🎯 Enhanced Availability Detection**: 
  - `input[name="sin_fechas_disponibles"]` (no slots signal)
  - `input[name="fechas_disponibles"]` (available dates)
  - `input[name="horas_disponibles_chk"]` (time slot radios)
- **✅ Booking Confirmation**: Extract `num_serie_reserva`, `cod_cancelacion_reserva`, `cod_trazabilidad_reserva`

## 📋 Requirements

- **Node.js**: 18.0.0 or higher
- **Database**: MongoDB (local or cloud)
- **Email Service**: SendGrid, AWS SES, or SMTP
- **AI Service**: OpenAI, Google Gemini, or Anthropic Claude

## 🛠 Installation

1. **Clone the repository**
```bash
git clone <repository-url>
cd icgyad
```

2. **Install dependencies**
```bash
npm install
```

3. **Install Playwright browsers**
```bash
npx playwright install chromium
```

4. **Configure environment**
```bash
cp .env.example .env
# Edit .env with your configuration
```

5. **Build the application**
```bash
npm run build
```

## ⚙️ Configuration

### Required Environment Variables

#### Database
```env
DATABASE_URL=mongodb://localhost:27017/icgyad
```

#### AI Service (choose one)
```env
GENAI_PROVIDER=openai
OPENAI_API_KEY=your_openai_api_key
```

#### Email Service (choose one)
```env
# SendGrid
MAIL_PROVIDER=sendgrid
SENDGRID_API_KEY=your_sendgrid_api_key
SENDGRID_FROM_EMAIL=noreply@yourdomain.com

# AWS SES
MAIL_PROVIDER=ses
AWS_ACCESS_KEY_ID=your_aws_key
AWS_SECRET_ACCESS_KEY=your_aws_secret
AWS_REGION=us-east-1
SES_FROM_EMAIL=noreply@yourdomain.com

# SMTP
MAIL_PROVIDER=smtp
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=your_email@gmail.com
SMTP_PASS=your_app_password
SMTP_FROM=your_email@gmail.com
```

### Optional Configuration
```env
# Browser settings
BROWSER_HEADLESS=true
BROWSER_TIMEOUT=30000
MAX_ATTEMPTS_PER_NIGHT=50

# Feature flags
ENABLE_AUTO_BOOKING=false
ENABLE_SCREENSHOTS=true
ENABLE_HAR_CAPTURE=true

# Notification settings
NOTIFY_ON=availability  # availability | booking | both

# API Configuration
ICGYAD_API_URL=http://localhost:3000  # Base URL for API key validation endpoint
```

## 📖 Usage

### 🔐 API Key Authentication

**Note**: API key management (creation, listing, revocation) is handled directly in the database by administrators. The CLI only validates API keys through the REST API endpoint.

#### Using API Keys with Strategies

To run a strategy with API key authentication:

```bash
# Run a private strategy with API key
npm run start:prod run UruguayInscripcionDePartidasExtrangerasRegistroCivil \
  --api-key "your-api-key-here" \
  --user-data ./user-data.json

# Run a public strategy (no API key required)
npm run start:prod run PublicStrategyId \
  --user-data ./user-data.json
```

#### API Key Validation Flow

1. **Private Strategies**: Require a valid API key that has access to the specific strategy
2. **Public Strategies**: Can be executed without an API key
3. **Validation**: API keys are validated through the REST API endpoint `GET /:apiKey/scopes`

### CLI Commands

#### 1. List Available Strategies
```bash
npm run start:prod list
```

#### 2. Generate User Data Template
```bash
# For all strategies
npm run start:prod template

# For specific strategy
npm run start:prod template UruguayInscripcionDePartidasExtrangerasRegistroCivil
```

#### 3. Run Strategy (Public - No API Key Required)
```bash
# If strategy is configured as public
npm run start:prod run UruguayInscripcionDePartidasExtrangerasRegistroCivil \
  --user-data ./user-data.json \
  --dry-run \
  --headful
```

#### 4. Run Strategy (With API Key Authentication)
```bash
# For private strategies or API key validation
npm run start:prod run UruguayInscripcionDePartidasExtrangerasRegistroCivil \
  --api-key "icgyad_abc123def456..." \
  --user-data ./user-data.json \
  --dry-run \
  --headful
```

#### 5. Run Strategy (Scheduled with API Key)
```bash
npm run start:prod run UruguayInscripcionDePartidasExtrangerasRegistroCivil \
  --api-key "icgyad_abc123def456..." \
  --window "20:45-00:00" \
  --interval 5 \
  --user-data ./user-data.json \
  --auto-book
```

#### 5. Run Strategy (Custom Retry Delay)
```bash
# Fast retry for testing (30 seconds between attempts)
npm run start:prod run UruguayInscripcionDePartidasExtrangerasRegistroCivil \
  --user-data ./user-data.json \
  --delay 30 \
  --headful

# Custom retry delay (2 minutes between attempts)
npm run start:prod run UruguayInscripcionDePartidasExtrangerasRegistroCivil \
  --user-data ./user-data.json \
  --delay 120 \
  --auto-book
```

#### 6. Test Notifications (Development)
```bash
# Test notification system - sends email even when no slots found
npm run start:prod run UruguayInscripcionDePartidasExtrangerasRegistroCivil \
  --user-data ./user-data.json \
  --delay 30 \
  --notify-always \
  --headful
```

#### 7. Check Run Status
```bash
npm run start:prod status <run-id>
```

### User Data Format

Create a `user-data.json` file with your information:

```json
{
  "titular_pais": "UY",
  "titular_nombres": "Juan Carlos",
  "titular_apellidos": "García López",
  "tipo_partida": "PN",
  "TipoDocumento": "CI",
  "NroDocumento": "12345678",
  "apellidos": "García López", 
  "nombres": "Juan Carlos",
  "Mail": "juan.garcia@email.com",
  "Telefono": "099123456",
  "gestoria": false,
  "email": "juan.garcia@email.com"
}
```

### 📦 Library Integration (for API Development)

You can import ICGYAD as a library to integrate appointment booking into your own applications:

```typescript
import { StrategyRunnerService, AuthService, AppModule } from 'icgyad';
import { NestFactory } from '@nestjs/core';

// Initialize the application
const app = await NestFactory.createApplicationContext(AppModule);
const strategyRunner = app.get(StrategyRunnerService);
const authService = app.get(AuthService);

// Create API key for your application
const { apiKey, keyId } = await authService.createApiKey({
  name: 'My App Integration',
  issuer: 'myapp',
  allowedStrategies: ['UruguayInscripcionDePartidasExtrangerasRegistroCivil']
});

// Run strategy with API key authentication
const runId = await strategyRunner.runStrategy({
  strategyId: 'UruguayInscripcionDePartidasExtrangerasRegistroCivil',
  apiKey: apiKey,
  window: { start: '20:45', end: '00:00', intervalMin: 5 },
  userData: {
    titular_pais: 'UY',
    titular_nombres: 'Juan Carlos',
    // ... other required fields
  },
  options: {
    dryRun: false,
    autoBook: true,
    notifyOn: 'availability'
  }
});

console.log(`Strategy execution started: ${runId}`);
```

## 🔧 Command Line Options

| Option | Description | Example | Default |
|--------|-------------|---------|---------|
| `--api-key` | **🔐 NEW** API key for authentication | `--api-key "icgyad_abc123..."` | None (checks if strategy is public) |
| `--window` | Time window for checking (HH:mm-HH:mm) | `--window "20:45-00:00"` | One-time execution |
| `--interval` | Check interval in minutes | `--interval 5` | 5 minutes |
| `--dry-run` | Test mode (no actual booking) | `--dry-run` | false |
| `--headful` | Show browser window | `--headful` | false |
| `--max-attempts` | Maximum attempts per night | `--max-attempts 50` | 10 |
| `--auto-book` | Enable automatic booking | `--auto-book` | false |
| `--delay` | Retry delay in seconds | `--delay 30` | 300 (5 minutes) |
| `--notify-always` | Send test notifications on no slots | `--notify-always` | false |
| `--user-data` | Path to user data file | `--user-data ./data.json` | Strategy-specific file |
| `--config` | Path to configuration file | `--config ./config.json` | - |
| `--api-key` | 🔐 API key for authentication | `--api-key "your-key"` | Required for private strategies |

### 🆕 New Parameters (v2.1)

#### `--delay <seconds>`
- **Purpose**: Configure retry delay between attempts in the Uruguay strategy
- **Usage**: `--delay 30` (30 seconds), `--delay 300` (5 minutes)
- **Benefits**: 
  - ⚡ **Fast testing**: Use `--delay 30` for quick development testing
  - 🔄 **Custom intervals**: Adjust retry frequency based on your needs
  - ⚖️ **Server respect**: Avoid overwhelming the target server

#### `--notify-always`
- **Purpose**: Send email notifications even when no slots are found (for testing notification system)
- **Usage**: `--notify-always` (flag, no value needed)
- **Benefits**:
  - 📧 **Test email delivery** without waiting for real availability
  - 🔧 **Debug notification service** configuration
  - ✅ **Verify email templates** and formatting
  - 🚀 **Quick validation** of notification pipeline

## 📊 Monitoring

### Logs and Screenshots
- **Logs**: `./logs/` directory with structured JSON logs
- **Screenshots**: `./logs/screenshots/<run-id>/` for debugging
- **HAR Files**: `./logs/traces/` for network analysis

### Database Collections
- **runs**: Execution records
- **attempts**: Step-by-step attempt logs  
- **availabilities**: Detected appointment slots
- **bookings**: Successful reservations
- **notifications**: Email delivery records

## 🔐 Security Considerations

### API Key Security
1. **🔑 API Key Storage**: Store API keys securely, never commit to version control
2. **🔄 Key Rotation**: Regularly rotate API keys and revoke unused ones
3. **📊 Usage Monitoring**: Monitor API key usage patterns for anomalies
4. **🛡️ Permissions**: Follow principle of least privilege - only grant necessary strategy access
5. **⏰ Expiration**: Set appropriate expiration dates for API keys

### General Security
1. **Environment Variables**: Never commit sensitive data to version control
2. **Rate Limiting**: Respects reasonable intervals to avoid detection
3. **User Agents**: Rotates browser fingerprints
4. **Terms of Service**: Review and comply with website terms
5. **Legal Compliance**: Ensure compliance with local automation laws

### API Key Best Practices
```bash
# ✅ DO: Store API keys in environment variables
export ICGYAD_API_KEY="icgyad_abc123def456..."

# ✅ DO: Use different API keys for different environments
export ICGYAD_API_KEY_DEV="icgyad_dev_..."
export ICGYAD_API_KEY_PROD="icgyad_prod_..."

# ❌ DON'T: Hardcode API keys in source code
const apiKey = "icgyad_abc123..."; // NEVER DO THIS

# ✅ DO: Revoke compromised keys immediately
npm run start:prod revoke-key --key-id "compromised_key_id"
```

## 🎨 Architecture

```
CLI Interface / Library API
    ↓
🔐 API Key Authentication
    ↓
Strategy Runner
    ↓
┌─────────────────────────────────────────────────────┐
│                Strategy Engine                       │
├─────────────────────────────────────────────────────┤
│ • AuthService (API Key Management & Validation)    │
│ • PlaywrightService (Browser Automation)           │
│ • GenAISolverService (Security Questions)          │  
│ • AvailabilityDetector (Multi-method Detection)    │
│ • NotifierService (Email Notifications)            │
└─────────────────────────────────────────────────────┘
    ↓
Database (MongoDB)
├── Strategies (with permissions)
├── API Keys (with usage tracking)
├── Runs & Attempts
└── Notifications & Bookings
```

## 🔄 Adding New Strategies

1. **Create Strategy Class**
```typescript
@Strategy({
  id: 'YourNewStrategy',
  name: 'Your Service Name',
  version: '1.0.0',
  enabled: true
})
export class YourNewStrategy implements IStrategy {
  // Implement required methods
}
```

2. **Register in AppModule**
```typescript
this.strategyFactory.register(YourNewStrategy, metadata);
```

3. **Configure Detection Rules**
```typescript
private readonly detectionConfig: DetectionConfig = {
  noSlotsText: ['No appointments available'],
  availabilitySelectors: ['.available-slot'],
  // ... other selectors
};
```

## 🚀 Deployment

### Local Development
```bash
npm run start:dev
```

### Production
```bash
# Build
npm run build

# Start
npm run start:prod

# Or with PM2
pm2 start ecosystem.config.js
```

### Docker
```bash
# Build image
docker build -t icgyad .

# Run container
docker run -d \
  --name icgyad \
  -v $(pwd)/.env:/app/.env \
  -v $(pwd)/logs:/app/logs \
  icgyad
```

### Cloud Deployment

#### Google Cloud Run
```bash
# Deploy as a job
gcloud run jobs create icgyad \
  --image gcr.io/PROJECT/icgyad \
  --region us-central1 \
  --set-env-vars DATABASE_URL=... 
```

#### AWS Lambda (with Container)
```bash
# Package and deploy
aws lambda create-function \
  --function-name icgyad \
  --package-type Image \
  --code ImageUri=ACCOUNT.dkr.ecr.region.amazonaws.com/icgyad:latest
```

## 🧪 Testing

### Unit Tests
```bash
npm test
```

### End-to-End Tests
```bash
npm run test:e2e
```

### Manual Testing
```bash
# Test with dry-run mode (default 5-minute delays)
npm run start:prod run UruguayInscripcionDePartidasExtrangerasRegistroCivil \
  --dry-run \
  --headful \
  --user-data ./test-data.json

# Fast testing with 30-second retry delays
npm run start:prod run UruguayInscripcionDePartidasExtrangerasRegistroCivil \
  --dry-run \
  --headful \
  --delay 30 \
  --user-data ./test-data.json

# Test notification system (sends emails even on no slots)
npm run start:prod run UruguayInscripcionDePartidasExtrangerasRegistroCivil \
  --dry-run \
  --headful \
  --delay 30 \
  --notify-always \
  --user-data ./test-data.json
```

## 🔍 Troubleshooting

### Common Issues

#### "Strategy not found"
- Check strategy ID spelling
- Run `npm run start:prod list` to see available strategies

#### "Invalid user data"
- Generate template: `npm run start:prod template <strategy-id>`
- Verify all required fields are filled

#### "Browser timeout"
- Increase `BROWSER_TIMEOUT` in environment
- Check internet connection and website availability

#### "Security question failed"
- Verify AI service API key is correct
- Check AI service credits/quota

#### "Email notification failed"
- Verify email service configuration
- Check spam folder for test emails

### Debug Mode
```bash
# Run with debug logging
LOG_LEVEL=debug npm run start:prod run ...
```

### Logs Analysis
```bash
# View structured logs
tail -f logs/app.log | jq '.'

# Filter by run ID
grep "run-id-here" logs/app.log | jq '.'
```

## 📚 API Reference

### Strategy Interface
```typescript
interface IStrategy {
  bootstrap(ctx: ExecutionContext): Promise<void>;
  fillForms(ctx: ExecutionContext, userData: Record<string, unknown>): Promise<void>;
  solveSecurityCheck(ctx: ExecutionContext, solver: SecurityQuestionSolver): Promise<void>;
  reachCalendar(ctx: ExecutionContext): Promise<void>;
  pollAvailability(ctx: ExecutionContext): Promise<AvailabilityResult>;
  book(ctx: ExecutionContext, slot: Slot, userData: Record<string, unknown>): Promise<BookResult>;
  notify(ctx: ExecutionContext, event: NotificationEvent): Promise<void>;
}
```

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

## 📄 License

MIT License - see LICENSE file for details

## ⚠️ Disclaimer

This software is provided for educational and personal use only. Users are responsible for:
- Complying with website terms of service
- Respecting rate limits and robot policies  
- Following local laws regarding automation
- Using the system ethically and responsibly

The authors are not responsible for any misuse or consequences arising from the use of this software.

## 📞 Support

- **Issues**: Use GitHub Issues for bug reports
- **Discussions**: Use GitHub Discussions for questions
- **Email**: For security issues only

---

**Made with ❤️ for the Uruguay community**