Troubleshooting - LogStack Documentation

🚨 Common Issues

❌ "Cannot connect to database"

Error Message:

Error: MongoNetworkError: connection ECONNREFUSED

🔧 Solutions:

Check MongoDB Service:

# Check if MongoDB is running
sudo systemctl status mongodb

# Start MongoDB if not running
sudo systemctl start mongodb

# For Windows (MongoDB service)
net start MongoDB

Verify Connection String:

# Test connection manually
mongo "mongodb://localhost:27017/logstack"

# For remote databases
mongo "mongodb://user:password@host:27017/database"

Check Network/Firewall:
- Ensure port 27017 is open
- Check firewall rules
- Verify network connectivity

❌ "AWS credentials not found"

Error Message:

Error: AWS credentials not found or invalid

🔧 Solutions:

Set Environment Variables:

# Set AWS credentials
export AWS_ACCESS_KEY_ID="your_access_key"
export AWS_SECRET_ACCESS_KEY="your_secret_key"
export AWS_REGION="us-east-1"

# Verify they are set
echo $AWS_ACCESS_KEY_ID

Use AWS CLI to configure:

# Configure AWS CLI
aws configure

# Test configuration
aws sts get-caller-identity

Check IAM Permissions:

Ensure your AWS user has the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket",
        "arn:aws:s3:::your-bucket/*"
      ]
    }
  ]
}

❌ "Job failed with timeout"

Error Message:

Error: Job processing timeout after 30000ms

🔧 Solutions:

Increase Timeout:

const config = {
  // ... other config
  processing: {
    timeoutMs: 60000, // Increase to 60 seconds
    batchSize: 500,   // Reduce batch size
    maxRetries: 5
  }
};

Optimize Database Queries:
- Add indexes on timestamp fields
- Use smaller batch sizes
- Consider pagination for large datasets

Check System Resources:

# Check memory usage
free -h

# Check disk space
df -h

# Check CPU usage
top

🗄️ Database Problems

Collection Not Found

Problem: LogStack cannot find collections

Quick Fix:

// Verify collection names in config
const config = {
  collections: {
    jobsCollectionName: 'jobs',
    logsCollectionName: 'logs', 
    apiLogsCollectionName: 'api_logs'
  }
};

Index Creation Failed

Problem: Database index creation errors

Quick Fix:

# Connect to MongoDB and create indexes manually
mongo your_database
db.api_logs.createIndex({"timestamp": 1})
db.api_logs.createIndex({"method": 1, "path": 1})

Authentication Failed

Problem: Database authentication errors

Quick Fix:

// Use correct connection string format
mongodb://username:password@host:port/database?authSource=admin

Connection Pool Exhausted

Problem: Too many concurrent connections

Quick Fix:

// Increase connection pool size
const mongoOptions = {
  maxPoolSize: 20,
  minPoolSize: 5,
  maxIdleTimeMS: 30000
};

☁️ Storage Issues

AWS S3 Issues

Access Denied

AccessDenied: User is not authorized to perform: s3:PutObject

Fix: Update IAM policy to include required permissions

{
  "Effect": "Allow",
  "Action": [
    "s3:PutObject",
    "s3:GetObject", 
    "s3:DeleteObject"
  ],
  "Resource": "arn:aws:s3:::your-bucket/*"
}

Bucket Does Not Exist

NoSuchBucket: The specified bucket does not exist

Fix: Create the bucket or check the name

# Create bucket using AWS CLI
aws s3 mb s3://your-logs-bucket --region us-east-1

Region Mismatch

PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint

Fix: Use correct region in configuration

const config = {
  s3: {
    region: 'us-west-2', // Match bucket region
    bucket: 'your-logs-bucket'
  }
};

Local Storage Issues

Permission Denied

EACCES: permission denied, mkdir '/var/logs/'

Fix: Check file permissions or use accessible directory

# Fix permissions
sudo chmod 755 /var/logs/

# Or use user directory
const config = {
  outputDirectory: './logs' // Use relative path
};

⚡ Performance Issues

Slow Log Processing

📊 Symptoms:

Jobs taking longer than expected
High memory usage
CPU spikes during processing

🚀 Optimizations:

1. Reduce Batch Size

const config = {
  processing: {
    batchSize: 500,    // Reduce from default 1000
    maxMemoryMB: 256   // Set memory limit
  }
};

2. Add Database Indexes

# Add indexes for common queries
db.api_logs.createIndex({"timestamp": 1})
db.api_logs.createIndex({"method": 1, "statusCode": 1})
db.api_logs.createIndex({"path": 1})

3. Enable Compression

const config = {
  compression: {
    enabled: true,
    format: 'gzip',
    level: 6  // Balance between speed and compression
  }
};

4. Optimize Data Masking

const config = {
  dataMasking: {
    enabled: true,
    // Only mask what's necessary
    maskPasswords: true,
    maskEmails: false,  // Disable if not needed
    customFields: ['token'] // Be specific
  }
};

📈 Performance Monitoring

const { init } = require('logstack');

const config = {
  // ... other config
  monitoring: {
    enabled: true,
    logLevel: 'info',
    metricsInterval: 60000, // Log metrics every minute
    
    // Performance thresholds
    thresholds: {
      maxProcessingTime: 30000,  // 30 seconds
      maxMemoryMB: 512,          // 512 MB
      maxBatchSize: 1000
    }
  }
};

await init(config);

// Listen for performance events
process.on('logstack:performance', (metrics) => {
  console.log('Performance metrics:', metrics);
  
  if (metrics.processingTime > 30000) {
    console.warn('Slow processing detected!');
  }
});

🐛 Debugging

Enable Debug Mode

# Enable debug logging
export LOG_LEVEL=debug
export NODE_ENV=development

# Run your application
node your-app.js

Debug Configuration Issues

const { validateConfig } = require('logstack');

const config = {
  dbUri: 'mongodb://localhost:27017/logs',
  uploadProvider: 's3'
  // ... rest of config
};

// Validate before initializing
const validation = validateConfig(config);
if (!validation.isValid) {
  console.error('Configuration errors:', validation.errors);
  validation.errors.forEach(error => {
    console.error(`- ${error.field}: ${error.message}`);
  });
} else {
  console.log('✅ Configuration is valid');
}

Monitor Job Execution

const logstack = require('logstack');

// Listen to all job events
logstack.on('job:started', (job) => {
  console.log(`🚀 Job ${job._id} started at ${new Date()}`);
});

logstack.on('job:progress', (job, progress) => {
  console.log(`📊 Job ${job._id}: ${progress.completed}/${progress.total} records processed`);
});

logstack.on('job:completed', (job, result) => {
  console.log(`✅ Job ${job._id} completed in ${result.processingTime}ms`);
  console.log(`   Records processed: ${result.recordsProcessed}`);
  console.log(`   Files generated: ${result.filesGenerated.length}`);
});

logstack.on('job:failed', (job, error) => {
  console.error(`❌ Job ${job._id} failed:`, error);
  console.error(`   Attempt: ${job.attempts}/${job.maxAttempts}`);
  console.error(`   Error: ${error.message}`);
});

logstack.on('upload:progress', (filename, progress) => {
  console.log(`📤 Uploading ${filename}: ${progress}%`);
});

logstack.on('upload:completed', (filename, result) => {
  console.log(`✅ Upload completed: ${filename}`);
});

Database Query Debugging

// Enable MongoDB query logging
const mongoose = require('mongoose');

// Log all queries in development
if (process.env.NODE_ENV === 'development') {
  mongoose.set('debug', true);
}

// Custom query logging
mongoose.set('debug', function (collectionName, method, query, doc) {
  console.log(`${collectionName}.${method}`, JSON.stringify(query), doc);
});

Memory Leak Detection

// Monitor memory usage
setInterval(() => {
  const used = process.memoryUsage();
  console.log('Memory usage:');
  for (let key in used) {
    console.log(`${key}: ${Math.round(used[key] / 1024 / 1024 * 100) / 100} MB`);
  }
}, 30000); // Every 30 seconds

// Set memory limit warnings
const maxMemoryMB = 512;
setInterval(() => {
  const heapUsed = process.memoryUsage().heapUsed / 1024 / 1024;
  if (heapUsed > maxMemoryMB) {
    console.warn(`⚠️ High memory usage: ${heapUsed.toFixed(2)} MB`);
  }
}, 60000);

📋 Error Codes

LOGSTACK_001

Message: Database connection failed
Cause: Invalid connection string or database not available
Solution: Check database connection and credentials

LOGSTACK_002

Message: Storage provider configuration invalid
Cause: Missing or incorrect storage credentials
Solution: Verify storage provider configuration

LOGSTACK_003

Message: Job processing timeout
Cause: Job took longer than configured timeout
Solution: Increase timeout or optimize processing

LOGSTACK_004

Message: File upload failed
Cause: Network issues or storage provider errors
Solution: Check network connectivity and storage permissions

LOGSTACK_005

Message: Data validation failed
Cause: Invalid log data format
Solution: Ensure log data matches expected schema

✅ Best Practices

🔒 Security

Always enable data masking in production
Use environment variables for sensitive credentials
Regularly rotate access keys and passwords
Implement proper IAM policies with minimal permissions
Enable encryption for data at rest and in transit

⚡ Performance

Monitor memory usage and set appropriate limits
Use appropriate batch sizes for your data volume
Add database indexes on frequently queried fields
Enable compression for large log files
Consider using multiple workers for high-volume processing

🗄️ Data Management

Set appropriate retention policies for your use case
Regularly monitor storage costs and usage
Implement backup strategies for critical logs
Use lifecycle policies for cloud storage
Archive old logs to cheaper storage tiers

📊 Monitoring

Set up alerting for failed jobs
Monitor processing times and success rates
Track storage usage and costs
Log application metrics for troubleshooting
Use health checks for production deployments

📞 Get Support

🐛 Bug Reports

Found a bug? Report it on GitHub Issues with:

LogStack version
Node.js version
Operating system
Steps to reproduce
Error messages and logs

Report Bug →

💡 Feature Requests

Have an idea for improvement? Create a feature request with:

Use case description
Expected behavior
Benefits and impact
Alternative solutions considered

Request Feature →

💬 Community Help

Need help with implementation? Join the discussion:

GitHub Discussions
Stack Overflow (tag: logstack)
Community forums

Get Help →

📧 Direct Contact

For urgent issues or enterprise support:

Email: support@logstack.dev
Enterprise: enterprise@logstack.dev

Contact Us →

📋 Before Asking for Help

I've checked this troubleshooting guide I've searched existing GitHub issues I've tried the suggested solutions I have error logs and reproduction steps I've included my LogStack configuration (without credentials)