Getting Started - LogStack Documentation

📦 Installation

Install LogStack via npm or yarn:

                                    npm install logstack
                                
                                    yarn add logstack
                                
                                    pnpm add logstack

💡 Requirements

Node.js 16+ (recommended: 18+)
MongoDB, MySQL, PostgreSQL, or SQLite
AWS S3 or local storage

⚡ Quick Setup

Get started with LogStack in under 5 minutes:

1

Create Environment File

# .env
MONGODB_URI=mongodb://localhost:27017/logstack
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
S3_BUCKET=your-log-bucket
NODE_ENV=production

2

Basic Implementation

const { init, createDailyJobs } = require('logstack');

async function setupLogStack() {
  const config = {
    dbUri: process.env.MONGODB_URI,
    uploadProvider: 's3',
    retention: {
      dbRetentionDays: 14,
      fileRetentionDays: 180
    },
    dataMasking: {
      maskPasswords: true,
      maskEmails: false
    }
  };

  await init(config);
  await createDailyJobs();
  
  console.log('✅ LogStack initialized successfully!');
}

setupLogStack().catch(console.error);

3

Run Your Application

node your-app.js

🎉 Success!

LogStack is now processing your logs automatically with:

📁 Daily folder structure
🔒 Password masking enabled
☁️ S3 storage configured
🗑️ Automatic retention cleanup

⚙️ Configuration Options

Complete configuration reference for all LogStack features:

🗄️ Database Configuration

const config = {
  // MongoDB (recommended)
  dbUri: 'mongodb://localhost:27017/logstack',
  
  // MySQL
  dbUri: 'mysql://user:password@localhost:3306/logstack',
  
  // PostgreSQL
  dbUri: 'postgresql://user:password@localhost:5432/logstack',
  
  // SQLite
  dbUri: 'sqlite:./logs.db',
  
  // Custom collection names
  collections: {
    jobsCollectionName: 'production_jobs',
    logsCollectionName: 'production_logs',
    apiLogsCollectionName: 'production_api_logs'
  }
};

☁️ Storage Provider Configuration

const config = {
  uploadProvider: 's3',
  s3: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    region: 'us-east-1',
    bucket: 'your-log-bucket',
    serverSideEncryption: 'AES256',
    storageClass: 'STANDARD'
  }
};

const config = {
  uploadProvider: 'local',
  outputDirectory: './logs',
  preserveLocalFiles: true
};

📁 Folder Structure Configuration

const config = {
  folderStructure: {
    type: 'daily',                    // 'daily', 'monthly', 'yearly'
    subFolders: {
      enabled: true,
      byHour: true,                   // hour-14-15/
      byStatus: true,                 // success/failed/
      custom: ['processed', 'raw']    // Custom sub-folders
    },
    naming: {
      prefix: 'production-logs',      // production-logs_2025-09-02
      dateFormat: 'YYYY-MM-DD',       // Date format
      includeTime: false              // Include time in folder names
    }
  }
};

🗄️ Database Setup

Step-by-step guide for setting up different databases:

MongoDB Setup

# Install MongoDB
sudo apt-get install mongodb

# Start MongoDB service
sudo systemctl start mongodb

# Create database and user
mongo
use logstack
db.createUser({
  user: "logstack_user",
  pwd: "secure_password",
  roles: ["readWrite"]
})

# Connection string
MONGODB_URI=mongodb://logstack_user:secure_password@localhost:27017/logstack

PostgreSQL Setup

# Install PostgreSQL
sudo apt-get install postgresql

# Create database and user
sudo -u postgres psql
CREATE DATABASE logstack;
CREATE USER logstack_user WITH PASSWORD 'secure_password';
GRANT ALL PRIVILEGES ON DATABASE logstack TO logstack_user;

# Connection string
DB_URI=postgresql://logstack_user:secure_password@localhost:5432/logstack

☁️ Storage Provider Setup

Configure your preferred storage provider:

🚀 AWS S3 Setup

1. Create S3 Bucket

aws s3 mb s3://your-log-bucket --region us-east-1

2. Set Lifecycle Policy

{
  "Rules": [{
    "ID": "LogRetentionPolicy",
    "Status": "Enabled",
    "Filter": {"Prefix": "logs/"},
    "Expiration": {"Days": 180}
  }]
}

3. IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject"
    ],
    "Resource": "arn:aws:s3:::your-log-bucket/*"
  }]
}

🔒 Security Features

Configure data masking and security settings:

Data Masking Configuration

const config = {
  dataMasking: {
    enabled: true,                    // Enable data masking
    maskEmails: true,                 // Mask email addresses
    maskIPs: false,                   // Keep IPs for debugging
    maskPasswords: true,              // Mask password fields
    showLastChars: 0,                 // Show last N characters (0 = full mask)
    customFields: [                   // Additional fields to mask
      'token', 'secret', 'key', 
      'authorization', 'cookie'
    ],
    maskingChar: '*',                 // Character to use for masking
    patterns: {                       // Custom masking patterns
      creditCard: /\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}/g,
      phone: /\+?[\d\s-()]{10,}/g
    }
  }
};

Example: Before and After Masking

❌ Before Masking

{
  "email": "user@example.com",
  "password": "mySecretPassword123",
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9",
  "creditCard": "4532-1234-5678-9012"
}

✅ After Masking

{
  "email": "****@example.com",
  "password": "********",
  "token": "********",
  "creditCard": "****-****-****-9012"
}

🚀 Production Deployment

Best practices for deploying LogStack in production:

🐳 Docker Deployment

# Dockerfile
FROM node:18-slim

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

COPY . .
EXPOSE 3000

CMD ["node", "production-implementation.js"]

# docker-compose.yml
version: '3.8'
services:
  logstack:
    build: .
    environment:
      - NODE_ENV=production
      - MONGODB_URI=${MONGODB_URI}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - S3_BUCKET=${S3_BUCKET}
    restart: unless-stopped

📊 PM2 Process Manager

# Install PM2
npm install -g pm2

# ecosystem.config.js
module.exports = {
  apps: [{
    name: 'logstack-production',
    script: 'production-implementation.js',
    instances: 'max',
    exec_mode: 'cluster',
    env: {
      NODE_ENV: 'production'
    },
    error_file: './logs/pm2-errors.log',
    out_file: './logs/pm2-output.log',
    log_file: './logs/pm2-combined.log'
  }]
};

# Start with PM2
pm2 start ecosystem.config.js
pm2 save
pm2 startup

☸️ Kubernetes Deployment

# logstack-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstack
spec:
  replicas: 3
  selector:
    matchLabels:
      app: logstack
  template:
    metadata:
      labels:
        app: logstack
    spec:
      containers:
      - name: logstack
        image: your-registry/logstack:latest
        env:
        - name: MONGODB_URI
          valueFrom:
            secretKeyRef:
              name: logstack-secrets
              key: mongodb-uri
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: logstack-secrets
              key: aws-access-key
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"

📊 Monitoring & Health Checks

Set up monitoring and health checks for your LogStack deployment:

Health Check Endpoint

const express = require('express');
const app = express();

app.get('/health', async (req, res) => {
  try {
    // Check database connection
    await mongoose.connection.db.admin().ping();
    
    // Check S3 connectivity
    await s3.headBucket({ Bucket: process.env.S3_BUCKET }).promise();
    
    res.json({
      status: 'healthy',
      timestamp: new Date().toISOString(),
      database: 'connected',
      storage: 'accessible',
      uptime: process.uptime()
    });
  } catch (error) {
    res.status(503).json({
      status: 'unhealthy',
      error: error.message
    });
  }
});

app.listen(3000);

Getting Started with LogStack