# logstack

Production-ready Node.js package for automated hourly API logs processing and cloud storage uploads with MongoDB and cron jobs.

## ✨ Features

- **� Automated Processing**: Hourly cron jobs for API logs processing and uploads
- **☁️ Multi-Cloud Storage**: AWS S3, Google Cloud, Azure Blob, and local file storage
- **�️ Database Flexibility**: Custom collection names to avoid conflicts
- **📁 Organized Structure**: Clean output directories and file organization
- **⚡ High Performance**: Batch processing for large datasets (163+ records/sec)
- **🎯 TypeScript Support**: Full type safety and IntelliSense
- **�️ Production Ready**: Comprehensive error handling and retry logic

## 🚀 Installation

```bash
npm install logstack
```

## 📋 Quick Start

```typescript
import { init, createDailyJobs } from "logstack";

// Initialize with your preferred configuration
await init({
  dbUri: "mongodb://localhost:27017/cronlog",
  uploadProvider: "s3", // 'local', 's3', 'gcs', 'azure'
  fileFormat: "json",
  retryAttempts: 3,

  // 📁 Organized output directory (keeps project clean)
  outputDirectory: "production-logs",

  // 🗃️ Custom collection names (avoid database conflicts)
  collections: {
    jobsCollectionName: "myapp_jobs", // Instead of default "jobs"
    logsCollectionName: "myapp_logs", // Instead of default "logs"
    apiLogsCollectionName: "myapp_apilogs", // Instead of default "apilogs"
  },

  s3: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: "us-east-1",
    bucket: "my-logs-bucket",
  },
  logging: {
    level: "info",
    enableConsole: true,
    enableFile: true,
  },
});

// Create daily jobs (automatically done via cron)
await createDailyJobs();
```

## 🔧 Configuration Options

### Storage Providers

#### AWS S3

```typescript
{
  uploadProvider: "s3",
  outputDirectory: "production-logs", // Organized S3 structure
  collections: {
    jobsCollectionName: "prod_jobs",   // Avoid conflicts
    logsCollectionName: "prod_logs"
  },
  s3: {
    accessKeyId: "your-access-key-id",
    secretAccessKey: "your-secret-access-key",
    region: "us-east-1",
    bucket: "your-bucket-name",
    endpoint: "https://custom-s3-endpoint.com" // Optional for S3-compatible services
  }
}
```

**🧪 Test AWS Setup:**

```bash
npm run test:aws              # Basic AWS S3 testing
npm run test:aws-performance  # Performance testing
npm run test:aws-regions      # Multi-region testing
```

**📁 S3 File Structure:**

```
s3://your-bucket/
├── production-logs/2025-08-25/14-15.json
├── dev-logs/2025-08-25/14-15.json
└── organized-structure/
```

#### Google Cloud Storage

```typescript
{
  uploadProvider: "gcs",
  gcs: {
    projectId: "your-gcp-project-id",
    keyFilename: "./path/to/service-account-key.json", // or use credentials
    bucket: "your-gcs-bucket-name"
  }
}
```

#### Azure Blob Storage

```typescript
{
  uploadProvider: "azure",
  azure: {
    connectionString: "DefaultEndpointsProtocol=https;AccountName=...",
    containerName: "your-container-name"
  }
}
```

#### Local Storage

```typescript
{
  uploadProvider: "local"; // Files saved to local filesystem
}
```

### Complete Configuration Interface

```typescript
interface Config {
  dbUri: string;
  uploadProvider: "local" | "s3" | "gcs" | "azure";
  fileFormat?: "json" | "csv" | "txt";
  retryAttempts?: number;
  dailyCron?: string; // Default: '0 0 * * *' (midnight)
  hourlyCron?: string; // Default: '0 * * * *' (every hour)
  timezone?: string; // Default: 'UTC'

  // 📁 Organized output directory for clean project structure
  outputDirectory?: string; // Default: "uploads"

  // 🗃️ Custom collection names to avoid database conflicts
  collections?: {
    jobsCollectionName?: string; // Default: "jobs"
    logsCollectionName?: string; // Default: "logs"
    apiLogsCollectionName?: string; // Default: "apilogs"
  };

  // 📊 API Logs Configuration (for existing collections)
  apiLogs?: {
    existingCollection?: {
      name: string; // Your existing collection name
      timestampField: string; // Field to filter by time
      requiredFields?: {
        /* field mappings */
      };
    };
    createNew?: {
      collectionName?: string;
      enableMiddleware?: boolean;
      autoFields?: boolean;
    };
  };

  logging?: {
    level: "debug" | "info" | "warn" | "error";
    enableConsole?: boolean;
    enableFile?: boolean;
    logFilePath?: string;
  };

  // Cloud storage configurations
  s3?: {
    /* AWS S3 config */
  };
  gcs?: {
    /* Google Cloud config */
  };
  azure?: {
    /* Azure Blob config */
  };
}
```

### 📁 Organized Output Directory

Keep your project clean by organizing generated files in a dedicated directory:

```typescript
await init({
  dbUri: "mongodb://localhost:27017/your-db",
  uploadProvider: "local",

  // Organize files in a dedicated directory
  outputDirectory: "logs", // Files saved in logs/{date}/

  // Before: Files created in project root: 2025-08-25/00-01.json
  // After:  Files created in: logs/2025-08-25/00-01.json
});
```

### 🗂️ Custom Collection Names

Avoid database conflicts by using custom collection names:```typescript
await init({
dbUri: "mongodb://localhost:27017/your-db",
uploadProvider: "local",

// Use custom collection names to prevent conflicts
collections: {
jobsCollectionName: "my_file_processing_jobs", // Instead of "jobs"
logsCollectionName: "my_file_processing_logs", // Instead of "logs"
},
});

````

This is especially useful when:

- Running multiple instances of the service
- Avoiding conflicts with existing collections
- Organizing collections with meaningful names
- Separating environments (dev/staging/prod)

## 🚀 Complete Node.js Implementation

### 📦 1. Installation & Setup

```bash
# Install the package
npm install logstack

# Install required dependencies
npm install mongodb mongoose
npm install dotenv  # For environment variables

# Optional: AWS SDK for S3
npm install aws-sdk

# Optional: Express.js for API middleware
npm install express @types/express
```

### 🔧 2. Environment Configuration

Create a `.env` file in your project root:

```env
# Database Configuration
DB_URI=mongodb://localhost:27017/your-app-name

# AWS S3 Configuration (if using S3)
AWS_ACCESS_KEY_ID=your-access-key-id
AWS_SECRET_ACCESS_KEY=your-secret-access-key
AWS_REGION=us-east-1
S3_BUCKET=your-bucket-name

# Application Configuration
NODE_ENV=production
LOG_LEVEL=info
```

### 💻 3. Basic Implementation (CommonJS)

```javascript
// basic-setup.js
const { init, createDailyJobs, processSpecificHour } = require('logstack');
require('dotenv').config();

async function setupCronLogService() {
  try {
    // Basic configuration
    const config = {
      dbUri: process.env.DB_URI || 'mongodb://localhost:27017/myapp',
      uploadProvider: 'local', // Start with local storage
      fileFormat: 'json',

      // Organized file structure
      outputDirectory: 'api-logs',

      // Custom collections to avoid conflicts
      collections: {
        jobsCollectionName: 'myapp_jobs',
        logsCollectionName: 'myapp_logs',
        apiLogsCollectionName: 'myapp_apilogs'
      },

      logging: {
        level: 'info',
        enableConsole: true,
        enableFile: true
      }
    };

    // Initialize the service
    console.log('🚀 Initializing logstack...');
    await init(config);
    console.log('✅ Service initialized successfully');

    // Create daily jobs for today
    const today = new Date().toISOString().split('T')[0];
    const job = await createDailyJobs(today, config);
    console.log(`📅 Created daily job with ${job.hours.length} hour slots`);

    // Process specific hour (example)
    const currentHour = new Date().getHours();
    await processSpecificHour(today, currentHour, config);
    console.log(`⚡ Processed hour ${currentHour}:00-${currentHour + 1}:00`);

  } catch (error) {
    console.error('❌ Setup failed:', error);
    process.exit(1);
  }
}

// Run the setup
setupCronLogService();
```

### 🎯 4. Advanced Implementation (ES Modules)

```javascript
// advanced-setup.mjs
import { init, createDailyJobs, processSpecificHour, saveApiLog } from 'logstack';
import dotenv from 'dotenv';

dotenv.config();

class CronLogManager {
  constructor() {
    this.config = {
      dbUri: process.env.DB_URI,
      uploadProvider: process.env.UPLOAD_PROVIDER || 'local',
      fileFormat: 'json',

      // Production-ready configuration
      outputDirectory: process.env.OUTPUT_DIR || 'production-logs',

      // Environment-specific collections
      collections: {
        jobsCollectionName: `${process.env.NODE_ENV || 'dev'}_jobs`,
        logsCollectionName: `${process.env.NODE_ENV || 'dev'}_logs`,
        apiLogsCollectionName: `${process.env.NODE_ENV || 'dev'}_apilogs`
      },

      // AWS S3 configuration
      s3: process.env.UPLOAD_PROVIDER === 's3' ? {
        accessKeyId: process.env.AWS_ACCESS_KEY_ID,
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
        region: process.env.AWS_REGION || 'us-east-1',
        bucket: process.env.S3_BUCKET
      } : undefined,

      logging: {
        level: process.env.LOG_LEVEL || 'info',
        enableConsole: true,
        enableFile: true,
        logFilePath: './logs/cron-service.log'
      }
    };
  }

  async initialize() {
    try {
      console.log('🚀 Initializing CronLogManager...');
      await init(this.config);
      console.log('✅ CronLogManager initialized successfully');

      // Log configuration summary
      console.log('📋 Configuration Summary:');
      console.log(`   📁 Output Directory: ${this.config.outputDirectory}`);
      console.log(`   ☁️  Upload Provider: ${this.config.uploadProvider}`);
      console.log(`   🗃️  Collections: ${Object.values(this.config.collections).join(', ')}`);

      return true;
    } catch (error) {
      console.error('❌ Initialization failed:', error);
      throw error;
    }
  }

  async createJobsForDateRange(startDate, endDate) {
    const jobs = [];
    const start = new Date(startDate);
    const end = new Date(endDate);

    for (let d = start; d <= end; d.setDate(d.getDate() + 1)) {
      const dateStr = d.toISOString().split('T')[0];
      const job = await createDailyJobs(dateStr, this.config);
      jobs.push(job);
      console.log(`📅 Created job for ${dateStr} with ${job.hours.length} hours`);
    }

    return jobs;
  }

  async processHourRange(date, startHour, endHour) {
    const results = [];

    for (let hour = startHour; hour <= endHour; hour++) {
      try {
        console.log(`⚡ Processing ${date} ${hour}:00-${hour + 1}:00...`);
        const result = await processSpecificHour(date, hour, this.config);
        results.push({ hour, status: 'success', result });
        console.log(`✅ Hour ${hour} processed successfully`);
      } catch (error) {
        console.error(`❌ Hour ${hour} failed:`, error.message);
        results.push({ hour, status: 'failed', error: error.message });
      }
    }

    return results;
  }

  async addApiLog(logData) {
    return await saveApiLog(logData, this.config);
  }

  getConfig() {
    return { ...this.config };
  }
}

// Usage example
async function main() {
  const manager = new CronLogManager();

  try {
    // Initialize
    await manager.initialize();

    // Create jobs for the next 7 days
    const today = new Date();
    const nextWeek = new Date(today.getTime() + 7 * 24 * 60 * 60 * 1000);
    await manager.createJobsForDateRange(
      today.toISOString().split('T')[0],
      nextWeek.toISOString().split('T')[0]
    );

    // Process today's business hours (9 AM to 5 PM)
    const todayStr = today.toISOString().split('T')[0];
    await manager.processHourRange(todayStr, 9, 17);

    console.log('🎉 Setup completed successfully!');

  } catch (error) {
    console.error('❌ Setup failed:', error);
    process.exit(1);
  }
}

// Run if this file is executed directly
if (import.meta.url === `file://${process.argv[1]}`) {
  main();
}

export default CronLogManager;
```

### 🌐 5. Express.js Integration

```javascript
// express-integration.js
const express = require('express');
const { init, createApiLogMiddleware, getApiLogs } = require('logstack');
require('dotenv').config();

const app = express();
app.use(express.json());

async function setupExpressApp() {
  // Initialize logstack
  const config = {
    dbUri: process.env.DB_URI,
    uploadProvider: 'local',
    outputDirectory: 'api-logs',
    collections: {
      jobsCollectionName: 'express_jobs',
      logsCollectionName: 'express_logs',
      apiLogsCollectionName: 'express_apilogs'
    }
  };

  await init(config);

  // Add API logging middleware
  app.use(createApiLogMiddleware(config));

  // Your API routes
  app.get('/api/users', (req, res) => {
    res.json({
      users: [
        { id: 1, name: 'John Doe', email: 'john@example.com' },
        { id: 2, name: 'Jane Smith', email: 'jane@example.com' }
      ]
    });
  });

  app.post('/api/users', (req, res) => {
    const { name, email } = req.body;
    const newUser = { id: Date.now(), name, email };
    res.status(201).json({ user: newUser, message: 'User created successfully' });
  });

  // API logs endpoint
  app.get('/api/logs', async (req, res) => {
    try {
      const { method, startDate, endDate, limit = 100 } = req.query;

      const filters = {};
      if (method) filters.method = method;
      if (startDate) filters.startDate = new Date(startDate);
      if (endDate) filters.endDate = new Date(endDate);
      filters.limit = parseInt(limit);

      const logs = await getApiLogs(filters, config);
      res.json({ logs, count: logs.length });
    } catch (error) {
      res.status(500).json({ error: error.message });
    }
  });

  // Health check endpoint
  app.get('/health', (req, res) => {
    res.json({
      status: 'healthy',
      timestamp: new Date().toISOString(),
      service: 'logstack-express'
    });
  });

  const PORT = process.env.PORT || 3000;
  app.listen(PORT, () => {
    console.log(`🚀 Express server running on port ${PORT}`);
    console.log(`📊 API logs automatically captured`);
    console.log(`🔗 View logs: http://localhost:${PORT}/api/logs`);
  });
}

setupExpressApp().catch(console.error);
```

### ☁️ 6. AWS S3 Production Setup

```javascript
// aws-production.js
const { init, createDailyJobs, processSpecificHour } = require('logstack');
require('dotenv').config();

class AWSProductionSetup {
  constructor() {
    this.config = {
      dbUri: process.env.DB_URI,
      uploadProvider: 's3',
      fileFormat: 'json',

      // Production AWS configuration
      outputDirectory: 'production-api-logs',

      collections: {
        jobsCollectionName: 'prod_jobs',
        logsCollectionName: 'prod_logs',
        apiLogsCollectionName: 'prod_apilogs'
      },

      s3: {
        accessKeyId: process.env.AWS_ACCESS_KEY_ID,
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
        region: process.env.AWS_REGION || 'us-east-1',
        bucket: process.env.S3_BUCKET
      },

      retryAttempts: 5,

      logging: {
        level: 'info',
        enableConsole: true,
        enableFile: true,
        logFilePath: '/var/log/cron-service/production.log'
      }
    };
  }

  async validateAWSCredentials() {
    console.log('🔐 Validating AWS credentials...');

    const required = ['AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY', 'S3_BUCKET'];
    const missing = required.filter(key => !process.env[key]);

    if (missing.length > 0) {
      throw new Error(`Missing AWS environment variables: ${missing.join(', ')}`);
    }

    console.log('✅ AWS credentials validated');
  }

  async initialize() {
    await this.validateAWSCredentials();

    console.log('🚀 Initializing AWS Production Setup...');
    await init(this.config);
    console.log('✅ Production setup initialized');

    console.log('📋 Production Configuration:');
    console.log(`   🪣 S3 Bucket: ${this.config.s3.bucket}`);
    console.log(`   🌍 Region: ${this.config.s3.region}`);
    console.log(`   📁 Output Directory: ${this.config.outputDirectory}`);
  }

  async setupDailyProcessing() {
    // Setup for the next 30 days
    const jobs = [];
    const today = new Date();

    for (let i = 0; i < 30; i++) {
      const date = new Date(today.getTime() + i * 24 * 60 * 60 * 1000);
      const dateStr = date.toISOString().split('T')[0];

      const job = await createDailyJobs(dateStr, this.config);
      jobs.push(job);

      if (i % 7 === 0) {
        console.log(`📅 Created jobs for week starting ${dateStr}`);
      }
    }

    console.log(`🎉 Created ${jobs.length} daily job configurations`);
    return jobs;
  }

  async processBacklog(startDate, endDate) {
    console.log(`⏮️ Processing backlog from ${startDate} to ${endDate}`);

    const start = new Date(startDate);
    const end = new Date(endDate);
    const results = [];

    for (let d = start; d <= end; d.setDate(d.getDate() + 1)) {
      const dateStr = d.toISOString().split('T')[0];

      // Process all 24 hours for each day
      for (let hour = 0; hour < 24; hour++) {
        try {
          await processSpecificHour(dateStr, hour, this.config);
          results.push({ date: dateStr, hour, status: 'success' });
        } catch (error) {
          console.error(`❌ Failed ${dateStr} ${hour}:00:`, error.message);
          results.push({ date: dateStr, hour, status: 'failed', error: error.message });
        }
      }

      console.log(`✅ Processed all hours for ${dateStr}`);
    }

    const successful = results.filter(r => r.status === 'success').length;
    const failed = results.filter(r => r.status === 'failed').length;

    console.log(`📊 Backlog processing complete:`);
    console.log(`   ✅ Successful: ${successful}`);
    console.log(`   ❌ Failed: ${failed}`);
    console.log(`   📈 Success rate: ${((successful / (successful + failed)) * 100).toFixed(2)}%`);

    return results;
  }
}

// Production deployment script
async function deployProduction() {
  const setup = new AWSProductionSetup();

  try {
    // Initialize
    await setup.initialize();

    // Setup daily processing
    await setup.setupDailyProcessing();

    // Process any backlog (last 7 days)
    const today = new Date();
    const weekAgo = new Date(today.getTime() - 7 * 24 * 60 * 60 * 1000);
    await setup.processBacklog(
      weekAgo.toISOString().split('T')[0],
      today.toISOString().split('T')[0]
    );

    console.log('🎉 AWS Production deployment completed successfully!');
    console.log('🔄 Cron jobs will now run automatically every hour');

  } catch (error) {
    console.error('❌ Production deployment failed:', error);
    process.exit(1);
  }
}

// Run deployment
if (require.main === module) {
  deployProduction();
}

module.exports = AWSProductionSetup;
```

### 📊 7. Performance Testing & Monitoring

```javascript
// performance-test.js
const { init, processSpecificHour, getApiLogs } = require('logstack');
require('dotenv').config();

class PerformanceMonitor {
  constructor() {
    this.metrics = {
      processedRecords: 0,
      failedRecords: 0,
      totalDuration: 0,
      averagePerRecord: 0,
      memoryUsage: []
    };
  }

  async testPerformance() {
    const config = {
      dbUri: process.env.DB_URI,
      uploadProvider: 'local',
      outputDirectory: 'performance-test',
      collections: {
        jobsCollectionName: 'perf_jobs',
        logsCollectionName: 'perf_logs',
        apiLogsCollectionName: 'apilogs' // Use your real data
      }
    };

    await init(config);

    console.log('⚡ Starting performance test...');
    const startTime = Date.now();

    // Test processing multiple hours in parallel
    const today = new Date().toISOString().split('T')[0];
    const hours = [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18];

    const promises = hours.map(async (hour) => {
      const hourStart = Date.now();

      try {
        await processSpecificHour(today, hour, config);
        const duration = Date.now() - hourStart;

        this.metrics.processedRecords++;
        console.log(`✅ Hour ${hour}: ${duration}ms`);

        return { hour, duration, status: 'success' };
      } catch (error) {
        this.metrics.failedRecords++;
        console.error(`❌ Hour ${hour}: ${error.message}`);

        return { hour, status: 'failed', error: error.message };
      }
    });

    const results = await Promise.all(promises);

    const endTime = Date.now();
    this.metrics.totalDuration = endTime - startTime;
    this.metrics.averagePerRecord = this.metrics.totalDuration / this.metrics.processedRecords;

    console.log('\n📊 Performance Test Results:');
    console.log(`⏱️  Total Duration: ${this.metrics.totalDuration}ms`);
    console.log(`✅ Processed Records: ${this.metrics.processedRecords}`);
    console.log(`❌ Failed Records: ${this.metrics.failedRecords}`);
    console.log(`📈 Average per Record: ${this.metrics.averagePerRecord.toFixed(2)}ms`);
    console.log(`🎯 Success Rate: ${((this.metrics.processedRecords / (this.metrics.processedRecords + this.metrics.failedRecords)) * 100).toFixed(2)}%`);

    const memUsage = process.memoryUsage();
    console.log(`💾 Memory Usage: ${(memUsage.heapUsed / 1024 / 1024).toFixed(2)} MB`);

    return results;
  }
}

// Run performance test
if (require.main === module) {
  const monitor = new PerformanceMonitor();
  monitor.testPerformance().catch(console.error);
}
```

### 🧪 8. Testing Your Implementation

```bash
# Run comprehensive tests
npm run test:final-local              # Complete local test
npm run test:final-local:quick        # Quick local test
npm run test:final-local:analytics    # Analytics only
npm run test:final-local:validate     # Validate existing files

# AWS testing
npm run test:aws                      # Basic AWS test
npm run test:aws-performance          # Performance test
npm run test:aws-regions              # Multi-region test

# Custom examples
npm run example:custom-collections    # Custom collections demo
npm run example:organized-output      # Organized directories demo
```

### 📈 9. Production Deployment Checklist

```bash
# 1. Environment setup
cp .env.example .env
# Edit .env with your credentials

# 2. Database setup
mongosh your-mongodb-uri
# Verify connection and permissions

# 3. AWS setup (if using S3)
aws configure list
aws s3 ls s3://your-bucket-name

# 4. Install and test
npm install
npm run test:final-local

# 5. Deploy to production
node production-setup.js

# 6. Monitor logs
tail -f /var/log/cron-service/production.log
```

### 🎯 10. Real-World Usage Scenarios

**Scenario 1: E-commerce API Logs**
```javascript
const ecommerceConfig = {
  dbUri: process.env.DB_URI,
  uploadProvider: 's3',
  outputDirectory: 'ecommerce-api-logs',
  collections: {
    jobsCollectionName: 'ecommerce_jobs',
    logsCollectionName: 'ecommerce_logs',
    apiLogsCollectionName: 'api_requests' // Your existing collection
  },
  s3: { /* AWS config */ }
};
```

**Scenario 2: Multi-tenant SaaS**
```javascript
const createTenantConfig = (tenantId) => ({
  dbUri: process.env.DB_URI,
  uploadProvider: 'local',
  outputDirectory: `tenant-${tenantId}-logs`,
  collections: {
    jobsCollectionName: `tenant_${tenantId}_jobs`,
    logsCollectionName: `tenant_${tenantId}_logs`,
    apiLogsCollectionName: `tenant_${tenantId}_apilogs`
  }
});
```

**Scenario 3: Microservices Architecture**
```javascript
const services = ['user-service', 'order-service', 'payment-service'];

services.forEach(async (service) => {
  const config = {
    dbUri: process.env.DB_URI,
    uploadProvider: 's3',
    outputDirectory: `${service}-logs`,
    collections: {
      jobsCollectionName: `${service.replace('-', '_')}_jobs`,
      logsCollectionName: `${service.replace('-', '_')}_logs`,
      apiLogsCollectionName: `${service.replace('-', '_')}_apilogs`
    }
  };

  await init(config);
});
```

## 🎯 API Request Logging

Built-in Express.js middleware for automatic API request logging:

```typescript
import express from "express";
import { createApiLogMiddleware, getApiLogs, init } from "logstack";

const app = express();

// Initialize the service
await init({
  dbUri: "mongodb://localhost:27017/cronlog",
  uploadProvider: "local",
});

// Add API logging middleware
app.use(createApiLogMiddleware());

// Your API routes
app.get("/api/users", (req, res) => {
  res.json({ users: [] });
});

// Query API logs
app.get("/api/logs", async (req, res) => {
  const logs = await getApiLogs({
    method: "GET",
    startDate: new Date("2025-08-25T00:00:00Z"),
    endDate: new Date("2025-08-25T23:59:59Z"),
    limit: 100,
  });
  res.json(logs);
});
````

## 📊 Job Management

### Manual Operations

```typescript
import {
  createDailyJobs,
  runHourlyJob,
  getLogs,
  getJobStatus,
  retryFailedJobs,
} from "logstack";

// Create jobs for specific date
await createDailyJobs("2025-08-25");

// Manually run hourly job
await runHourlyJob(config);

// Get job status
const status = await getJobStatus("2025-08-25", "09-10");

// Get logs
const logs = await getLogs("2025-08-25");

// Retry failed jobs
await retryFailedJobs(config);
```

## 🧪 Testing & Validation

### 📊 Comprehensive Test Results

Our final comprehensive test processed **500 API logs** with perfect results:

```
🎉 Final Test Results:
⏱️  Total Duration: 3.06 seconds
📄 Records Processed: 500
📁 Files Created: 52
💾 Total Size: 0.55 MB
⚡ Processing Rate: 163.29 records/second
🧠 Memory Usage: 1025.97 MB
📈 Success Rate: 100.00%
```

### 🗂️ Generated File Structure

```
final-test-outputs/
├── 2025-08-25/
│   ├── api-logs-06-00.json (20 records)
│   ├── api-logs-07-00.json (20 records)
│   └── ... (all 24 hours)
├── 2025-08-26/
│   ├── 09-10.json (21 records)
│   ├── 12-13.json (9 records)
│   └── api-logs-{hour}-00.json
├── streaming/
│   ├── 2025-08-25/stream-api-logs-{hour}-00.json
│   └── 2025-08-26/stream-api-logs-{hour}-00.json
└── analytics/
    └── final-test-analytics.json
```

### 🧪 Running Tests

```bash
# Complete final test (recommended)
npm run test:final-local

# Quick tests
npm run test:final-local:quick        # Faster with fewer records
npm run test:final-local:analytics    # Analytics generation only
npm run test:final-local:validate     # Validate existing files

# AWS S3 testing
npm run test:aws                      # Basic AWS functionality
npm run test:aws-performance          # High-performance AWS test
npm run test:aws-regions              # Multi-region AWS test

# Feature-specific examples
npm run example:custom-collections    # Custom collection names
npm run example:organized-output      # Organized directory structure

# API logging examples
npm run test:apilogs                  # API logs processing
npm run inspect:apilogs               # Inspect existing API logs
```

### ✅ Test Coverage

- **✅ Local File Storage**: 100% success rate with organized structure
- **✅ AWS S3 Integration**: Multi-environment testing with real credentials
- **✅ High-Performance Processing**: 163+ records/second processing rate
- **✅ Memory Efficiency**: Streaming prevents memory overflow
- **✅ Custom Collections**: No database conflicts with custom names
- **✅ Error Handling**: Robust retry mechanisms and error recovery
- **✅ File Validation**: 100% JSON validation success rate
- **✅ Analytics Reporting**: Comprehensive performance metrics

## 📁 Examples & Use Cases

### 🚀 Quick Start Examples

```bash
# Basic local setup
npm run example:custom-collections    # Custom collection names demo
npm run example:organized-output      # Organized directory structure

# API logging examples
npm run test:apilogs                  # Process API logs from database
npm run inspect:apilogs               # Inspect existing API logs data

# Production examples
npm run practical:setup               # Production-ready setup
npm run practical:cloud               # Cloud storage configuration
npm run practical:inspect             # Inspect production data

# Job processing examples
npm run job:create-and-process         # Create and process jobs
npm run job:current-hour               # Process current hour
npm run job:date-range                 # Process date range

# Multi-provider examples
npm run example:s3                     # AWS S3 example
npm run example:gcs                    # Google Cloud Storage example
npm run example:azure                  # Azure Blob Storage example
```

### 💡 Real-World Implementation Examples

**1. E-commerce Platform:**

```javascript
// E-commerce API logs processing
const ecommerceConfig = {
  dbUri: process.env.DB_URI,
  uploadProvider: "s3",
  outputDirectory: "ecommerce-api-logs",
  collections: {
    jobsCollectionName: "ecommerce_jobs",
    logsCollectionName: "ecommerce_logs",
    apiLogsCollectionName: "api_requests",
  },
  s3: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: "us-east-1",
    bucket: "ecommerce-logs-bucket",
  },
};
```

**2. Multi-tenant SaaS:**

```javascript
// Separate logs per tenant
const createTenantConfig = (tenantId) => ({
  dbUri: process.env.DB_URI,
  uploadProvider: "local",
  outputDirectory: `tenant-${tenantId}-logs`,
  collections: {
    jobsCollectionName: `tenant_${tenantId}_jobs`,
    logsCollectionName: `tenant_${tenantId}_logs`,
    apiLogsCollectionName: `tenant_${tenantId}_apilogs`,
  },
});
```

**3. Microservices Architecture:**

```javascript
// Separate configs per service
const services = ["user-service", "order-service", "payment-service"];

services.forEach(async (service) => {
  const config = {
    outputDirectory: `${service}-logs`,
    collections: {
      jobsCollectionName: `${service.replace("-", "_")}_jobs`,
      logsCollectionName: `${service.replace("-", "_")}_logs`,
      apiLogsCollectionName: `${service.replace("-", "_")}_apilogs`,
    },
  };
  await init(config);
});
```

## 🏗️ Project Structure

```
logstack/
├── src/
│   ├── main.ts           # Main initialization
│   ├── jobs.ts           # Job management logic
│   ├── logs.ts           # Log querying
│   └── apiLogs.ts        # API logging functionality
├── lib/
│   ├── db.ts             # Database connection
│   ├── cron.ts           # Cron job setup
│   ├── fileProcessor.ts  # File processing logic
│   ├── fileWriters.ts    # File format writers
│   ├── uploadProviders.ts # Cloud storage providers
│   ├── validation.ts     # Configuration validation
│   ├── logger.ts         # Winston logging setup
│   └── userDataProvider.ts # Custom data provider
├── models/
│   ├── job.ts            # Job schema
│   ├── log.ts            # Log schema
│   └── apiLog.ts         # API log schema
├── types/
│   └── config.d.ts       # TypeScript definitions
├── examples/
│   ├── basic.ts          # Basic usage
│   ├── advanced.ts       # Advanced configuration
│   ├── multiProvider.ts  # Multi-cloud examples
│   └── apiLogs.ts        # API logging examples
└── test/
    ├── jobs.test.ts      # Job tests
    └── comprehensive.test.ts # Full test suite
```

## 🔧 Custom Data Provider

Override the default data provider to use your own data source:

```typescript
// lib/userDataProvider.ts
export async function getDataForHour(
  date: string,
  hourRange: string
): Promise<any> {
  // Your custom data fetching logic
  const logs = await YourDatabase.find({
    timestamp: {
      $gte: new Date(`${date}T${hourRange.split("-")[0]}:00:00Z`),
      $lt: new Date(`${date}T${hourRange.split("-")[1]}:00:00Z`),
    },
  });

  return logs;
}
```

## � Production Deployment

### 📋 Pre-deployment Checklist

```bash
# 1. Environment Configuration
cp .env.example .env
# Configure: DB_URI, AWS credentials, etc.

# 2. Test Local Setup
npm run test:final-local
# Verify: 100% success rate, file generation working

# 3. Test Cloud Storage (if using)
npm run test:aws
# Verify: S3 uploads working, credentials valid

# 4. Validate Database Connection
npm run inspect:apilogs
# Verify: Can access your API logs collection

# 5. Performance Testing
npm run test:final-local:quick
# Verify: Processing speed meets requirements
```

### 🌟 Production Performance Metrics

Based on comprehensive testing with **real API logs data**:

| Metric                 | Result                 | Status       |
| ---------------------- | ---------------------- | ------------ |
| **Processing Rate**    | 163+ records/sec       | ✅ Excellent |
| **Memory Usage**       | ~1GB for 500 records   | ✅ Efficient |
| **File Generation**    | 52 files in 3.06 sec   | ✅ Fast      |
| **Success Rate**       | 100%                   | ✅ Perfect   |
| **Storage Efficiency** | 0.55MB for 500 records | ✅ Optimized |

### �🔄 Cron Schedule & Automation

The package automatically sets up these production cron jobs:

```javascript
// Automatic cron jobs (no manual setup required)
{
  dailyCron: '0 0 * * *',    // Create daily jobs at midnight
  hourlyCron: '0 * * * *',   // Process each hour
  retryCron: '*/30 * * * *'  // Retry failed jobs every 30 min
}
```

**What happens automatically:**

1. **Midnight**: Creates 24 hourly job slots for the new day
2. **Every Hour**: Processes the current hour's API logs
3. **Every 30 Minutes**: Retries any failed processing jobs
4. **Continuous**: Monitors and logs all operations

### 📊 Monitoring & Analytics

**Real-time Monitoring:**

```bash
# View live logs
tail -f logs/cron-service.log

# Check analytics
npm run test:final-local:analytics

# Validate generated files
npm run test:final-local:validate
```

**Analytics Include:**

- Processing performance metrics
- Success/failure rates
- Memory usage patterns
- File size and count statistics
- Method and status code distributions

## 🏗️ Project Structure

```
your-project/
├── node_modules/
│   └── logstack/
├── final-test-outputs/          # Generated files (organized)
│   ├── 2025-08-25/
│   │   ├── api-logs-09-00.json  # Hour 9-10 API logs
│   │   ├── api-logs-10-00.json  # Hour 10-11 API logs
│   │   └── 09-10.json           # Job-generated files
│   ├── streaming/               # Streaming processor output
│   └── analytics/               # Performance reports
├── logs/                        # Service logs
│   └── cron-service.log
├── .env                         # Environment configuration
├── package.json
└── your-app.js                  # Your implementation
```

## 🎯 Best Practices

### 🗃️ Database Organization

```javascript
// ✅ Good: Environment-specific collections
const prodConfig = {
  collections: {
    jobsCollectionName: "prod_jobs",
    logsCollectionName: "prod_logs",
    apiLogsCollectionName: "prod_apilogs",
  },
};

const devConfig = {
  collections: {
    jobsCollectionName: "dev_jobs",
    logsCollectionName: "dev_logs",
    apiLogsCollectionName: "dev_apilogs",
  },
};

// ❌ Avoid: Using default collection names in production
const badConfig = {
  // Will conflict with other instances
  // collections: undefined  // Uses defaults: 'jobs', 'logs', 'apilogs'
};
```

### 📁 File Organization

```javascript
// ✅ Good: Organized directory structure
const organizedConfig = {
  outputDirectory: "production-api-logs", // Clean separation
  uploadProvider: "s3",
  s3: { bucket: "company-api-logs-prod" },
};

// ❌ Avoid: Files in project root
const messyConfig = {
  // outputDirectory: undefined,  // Files in project root
  uploadProvider: "local",
};
```

### ⚡ Performance Optimization

```javascript
// ✅ Good: Batch processing for large datasets
const performantConfig = {
  retryAttempts: 5, // Handle transient failures
  fileFormat: "json", // Most efficient for APIs
  logging: { level: "info" }, // Reduce log noise
};

// Configure batch size for your data volume
const batchSize = process.env.NODE_ENV === "production" ? 1000 : 100;
```

## 🔧 Configuration Best Practices

### 🌍 Environment-Specific Configs

```javascript
// config/production.js
module.exports = {
  dbUri: process.env.PROD_DB_URI,
  uploadProvider: "s3",
  outputDirectory: "production-api-logs",
  collections: {
    jobsCollectionName: "prod_jobs",
    logsCollectionName: "prod_logs",
    apiLogsCollectionName: "prod_apilogs",
  },
  s3: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: process.env.AWS_REGION,
    bucket: process.env.PROD_S3_BUCKET,
  },
  retryAttempts: 5,
  logging: { level: "info", enableFile: true },
};

// config/development.js
module.exports = {
  dbUri: process.env.DEV_DB_URI,
  uploadProvider: "local",
  outputDirectory: "dev-api-logs",
  collections: {
    jobsCollectionName: "dev_jobs",
    logsCollectionName: "dev_logs",
    apiLogsCollectionName: "dev_apilogs",
  },
  retryAttempts: 3,
  logging: { level: "debug", enableConsole: true },
};
```

## 🐛 Troubleshooting

### Common Issues

1. **MongoDB Connection**: Ensure MongoDB is running and the connection string is correct
2. **Cloud Storage Permissions**: Verify credentials have proper read/write permissions
3. **File Path Issues**: Check that the application has write permissions for local storage
4. **Timezone Issues**: Set the correct timezone in configuration

### Debug Logging

Enable debug logging for detailed information:

```typescript
await init({
  // ... other config
  logging: {
    level: "debug",
    enableConsole: true,
    enableFile: true,
    logFilePath: "./debug.log",
  },
});
```

## 📄 License

MIT License - see LICENSE file for details

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Submit a pull request

## � Complete Documentation

### 🚀 Getting Started

- **[Quick Start Guide](./examples/quickStart.ts)** - Copy-paste examples for immediate use
- **[Implementation Checklist](./IMPLEMENTATION_CHECKLIST.md)** - Step-by-step setup guide
- **[Complete Step-by-Step Guide](./COMPLETE_STEP_BY_STEP_GUIDE.md)** - Comprehensive usage documentation

### 🎯 Key Features

- **[Custom Collection Names](./CUSTOM_COLLECTIONS.md)** - Avoid database conflicts
- **[Organized Output Directory](./OUTPUT_DIRECTORY.md)** - Clean project structure
- **[Implementation Complete](./IMPLEMENTATION_COMPLETE.md)** - Features overview

### 🧪 Testing & Examples

- **[Testing Guide](./TESTING_GUIDE.md)** - How to test your setup
- **[API Logs Testing](./APILOGS_TESTING_GUIDE.md)** - API logging validation
- **[Examples Folder](./examples/)** - Working code samples

### 📋 Reference

- **[Configuration Options](./FLEXIBLE_API_LOGS.md)** - All available settings
- **[Usage Guide](./COMPLETE_USAGE_GUIDE.md)** - Detailed API reference

## 🎯 Quick Implementation

**Choose your setup:**

1. **Basic Setup** (Recommended for beginners):

   ```typescript
   const config = {
     dbUri: "mongodb://localhost:27017/myapp",
     uploadProvider: "local",
     outputDirectory: "api-logs",
     collections: {
       jobsCollectionName: "myapp_jobs",
       logsCollectionName: "myapp_logs",
     },
   };
   ```

2. **Production Setup**:

   ```typescript
   const config = {
     dbUri: process.env.DB_URI,
     uploadProvider: "s3",
     outputDirectory: "production-logs",
     collections: {
       jobsCollectionName: "prod_jobs",
       logsCollectionName: "prod_logs",
     },
     s3: {
       /* S3 credentials */
     },
   };
   ```

3. **Multi-Service Setup**:
   ```typescript
   const createConfig = (serviceName) => ({
     dbUri: process.env.DB_URI,
     uploadProvider: "local",
     outputDirectory: `${serviceName}-logs`,
     collections: {
       jobsCollectionName: `${serviceName}_jobs`,
       logsCollectionName: `${serviceName}_logs`,
     },
   });
   ```

**Start here:** [Implementation Checklist](./IMPLEMENTATION_CHECKLIST.md) ✅

## 📞 Support

- **📖 Documentation**: Check the comprehensive guides above
- **🐛 Issues**: Create an issue on GitHub for bugs or feature requests
- **💡 Examples**: Review the examples folder for usage patterns
- **🧪 Testing**: Use the testing guides to validate your setup

````

#### Local Storage
```typescript
{
  uploadProvider: "local" // Files saved to local filesystem
}
````

### Complete Configuration Interface

```typescript
interface Config {
  dbUri: string;
  uploadProvider: "local" | "s3" | "gcs" | "azure";
  fileFormat?: "json" | "csv" | "txt";
  retryAttempts?: number;
  dailyCron?: string; // Default: '0 0 * * *' (midnight)
  hourlyCron?: string; // Default: '0 * * * *' (every hour)
  timezone?: string; // Default: 'UTC'
  logging?: {
    level: "debug" | "info" | "warn" | "error";
    enableConsole?: boolean;
    enableFile?: boolean;
    logFilePath?: string;
  };
  // ... provider-specific configs
}
```

// Manually trigger daily job creation
await createDailyJobs();

// Manually trigger hourly job
await runHourlyJob();

// Fetch logs
const logs = await getLogs("2025-08-25", "09-10");

// Fetch job status
const status = await getJobStatus("2025-08-25", "09-10");

```

## Custom Data Provider

Override `lib/userDataProvider.ts` to provide your own data for file generation.

## Running Tests

This package uses Jest for testing. To run tests:

```

npm run build
npx jest

```

## Examples

See the `examples/` folder for usage samples.

To run the basic example:

```

npm run build
node examples/basic.js

```

Or use ts-node for TypeScript:

```

npx ts-node examples/basic.ts

```

## License

MIT
```