# SQMViews Plugin - WordPress Page Views Tracking

## Overview

**SQMViews** is a plugin that collects, stores, and analyzes page view statistics for content. It provides real-time tracking, automated data processing, and interactive analytics dashboard for understanding user engagement with your WordPress site.

---

## Purpose

The plugin serves multiple purposes:

1. **Track User Engagement** - Monitor page views, time on page, and user activity patterns
2. **Analyze Content Performance** - Identify popular content, trending posts, and underperforming pages
3. **Understand Traffic Patterns** - Analyze when users visit, how long they stay, and how they navigate
4. **Data-Driven Decisions** - Provide actionable insights through interactive dashboards and REST API

---

## Core Functionality

### 1. **Client-Side Tracking**
- **JavaScript Tracker** (`artifacts/sqm-views-pages-{version}.js`)
  - Lightweight, async-loaded script injected into tracked pages
  - Tracks pageviews, user activity (init, ping, exit, timeout events)
  - Monitors time on page, active engagement, interaction frequency
  - Sends encrypted tracking data to server endpoints
  - Configurable inline vs external loading (minified or full version)

### 2. **Server-Side Processing**
- **Rapid Processor** (`SQMViewsRapidProcessor.php`)
  - Ultra-fast endpoint (`sqm-views-pages.php`) bypassing WordPress load
  - Validates and sanitizes tracking requests
  - Writes raw JSON Lines files for batch processing
  - Request size limiting and security validation
  - Diagnostic testing capabilities

- **Statistics Processor** (`sqm-views-process-statistics.php`)
  - Batch processes raw tracking data into database
  - Aggregates metrics: count, active time, high/low frequency interactions
  - Handles incomplete records (sessions still in progress)
  - Archives processed and invalid data
  - Runs via WordPress Cron or WP-CLI

### 3. **Data Model**

**Tables:**
- `wp_sqm_views_trackables` - Trackable entities (posts, taxonomies, pages)
- `wp_sqm_views_events` - Event types (pageview, click, etc.)
- `wp_sqm_views_records` - Raw individual event records
- `wp_sqm_views_daily` - Daily aggregated statistics (optimized for queries)

**Key Metrics:**
- `count` - Total number of views/events (usually 1 per pageview)
- `on_page` - Total time spent on page (seconds)
- `active` - Active engagement time (user actively interacting)
- `high_freq` - High-frequency interactions (rapid user actions like scrolling, mouse moves, dragging)
- `low_freq` - Low-frequency interactions (more important but less numerous events like clicks, touches, typing)

### 4. **REST API**
- **Tracking Endpoint:** `POST /wp-json/sqm-views/v1/track`
  - Receives tracking data from JavaScript client
  - Falls back to rapid processor for performance

- **Charts Endpoint:** `GET /wp-json/sqm-views/v1/charts`
  - Provides aggregated data for dashboard visualizations
  - Supports filters: date range, event type, trackable ID, metric, granularity
  - Returns time series data formatted for D3.js charts
  - Requires `manage_options` capability

### 5. **WordPress Admin Interface**
- **Settings Pages:**
  - **Tracking Settings** - Configure trackable post types and taxonomies
  - **Processing Settings** - Configure cron intervals, handlers (fast/API)
  - **Statistics Dashboard** - Interactive charts and analytics

- **WP-CLI Commands** (`SQMViewsCLI.php`)
  - Manual statistics processing
  - Diagnostics and debugging
  - Bulk operations

### 6. **Cron System** (`SQMViewsCron.php`)
- Scheduled statistics processing (15 min, hourly, 6 hours, daily, weekly)
- Automatic data aggregation and archiving
- Configurable intervals via admin settings
- Activation/deactivation hooks

---

## Architecture

### Data Flow

```
User Visits Page
     ↓
[Tracker Injected] (tracker.php)
     ↓
[JavaScript Tracker] (artifacts/sqm-views-pages-{version}.js)
     ↓
[Tracking Events] (init → ping → exit/timeout)
     ↓
[Fast Endpoint] (sqm-views-pages.php) OR [REST API]
     ↓
[Rapid Processor] (SQMViewsRapidProcessor.php)
     ↓
[Raw JSONL Files] (wp-content/uploads/sqm-views/raw/*.jsonl)
     ↓
[WordPress Cron] (SQMViewsCron.php)
     ↓
[Statistics Processor] (sqm-views-process-statistics.php)
     ↓
[Database Tables] (wp_sqm_views*)
     ↓
[REST API /charts] (charts-api.php)
     ↓
[Dashboard] (artifacts/dashboard-{version}.bundle.js)
     ↓
[Admin Sees Analytics]
```

### File Structure

```
sqm-views/
├── sqm-views.php                          # Main plugin file (entry point)
├── composer.json                        # PHP autoloader configuration
├── sqm-views-pages.php                    # Fast tracking endpoint (bypasses WP)
├── sqm-views-process-statistics.php       # Batch statistics processor
├── uninstall.php                        # Cleanup on plugin deletion
│
├── artifacts/                           # Built JavaScript bundles (versioned)
│   ├── sqm-views-pages-{version}.js       # JavaScript tracker (full version)
│   ├── sqm-views-pages-{version}.min.js   # JavaScript tracker (minified)
│   ├── dashboard-{version}.bundle.js      # Dashboard UI (full version)
│   ├── dashboard-{version}.bundle.min.js  # Dashboard UI (minified)
│   └── dashboard-{version}.bundle.libs.js # Dashboard dependencies (D3.js, etc.)
│
├── assets/                              # WordPress.org assets
│   ├── screenshot-1.png                   # Dashboard screenshot
│   ├── screenshot-2.png                   # Settings screenshot
│   ├── screenshot-3.png                   # Processing screenshot
│   └── screenshot-4.png                   # Statistics screenshot
│
├── languages/                           # Translation files
│   └── sqm-views.pot                      # Translation template
│
├── src/                                 # PHP source code
│   ├── SQMViewsSettings.php              # Admin pages and settings
│   ├── SQMViewsCron.php                  # Cron scheduling and management
│   ├── SQMViewsCLI.php                   # WP-CLI commands
│   ├── SQMViewsRapidProcessor.php        # Fast tracking request handler
│   ├── SQMViewsRecord.php                # Record accumulation and validation
│   ├── SQMViewsEvent.php                 # Event model/helper
│   ├── ProcessorResponse.php             # Response wrapper for processor
│   ├── RequestValidator.php              # Request schema validation
│   ├── DocumentSchemas.php               # JSON schema definitions
│   ├── Logger.php                       # PSR-3 style logging utility
│   │
│   └── includes/                        # Utility functions and hooks
│       ├── activation.php               # Plugin activation/deactivation hooks
│       ├── rest-api.php                 # REST API route registration
│       ├── charts-api.php               # Charts data endpoint logic
│       ├── tracker.php                  # Tracking code injection
│       └── utils.php                    # Helper functions and constants
│
└── vendor/                              # Composer dependencies
```

---

## Key Components

### 1. **Tracker Injection** (`src/includes/tracker.php`)
**Function:** `sqm_views_tracker()`

**Responsibilities:**
- Determines what content is being viewed (post, taxonomy, archive, home, search)
- Encrypts tracking metadata
- Injects meta tag with tracking configuration
- Includes JavaScript tracker script (inline or external)

**Trackable Content Types:**
- Posts and custom post types (configurable)
- Taxonomy archives (categories, tags, custom taxonomies)
- Author archives
- Date archives (year, month, day)
- Home page
- Search results

### 2. **Rapid Processor** (`src/SQMViewsRapidProcessor.php`)

**Purpose:** High-performance tracking endpoint

**Features:**
- Request size validation (hard limit: configurable)
- JSON payload parsing and validation
- Event type validation (init, ping, exit, timeout, test)
- JSONL file writing for batch processing
- Diagnostic mode for testing
- CORS headers for cross-origin requests

**Security:**
- Input sanitization
- Content-length limits
- Action whitelisting
- Remote IP logging
- User-agent tracking

### 3. **Statistics Processor** (`sqm-views-process-statistics.php`)

**Purpose:** Convert raw tracking data into aggregated statistics

**Process:**
1. Scans raw JSONL files in `sqm-views_files/raw/`
2. Parses and validates tracking records
3. Groups events by session (gid — Group of events ID)
4. Calculates metrics (on_page, active, high_freq, low_freq, count)
5. Inserts/updates records in database
6. Aggregates daily statistics
7. Archives processed files
8. Handles incomplete sessions (active users)

**Metrics Calculation:**
- **on_page:** Total time between init and exit/timeout
- **active:** Sum of time periods with high user activity
- **high_freq:** Count of rapid user interactions (scroll, mouse move, drag)
- **low_freq:** Count of slower interactions (click, touch, keypress)
- **count:** Total number of events (usually 1 per pageview)

### 4. **Cron System** (`src/SQMViewsCron.php`)

**Singleton class** managing automated processing

**Features:**
- Custom cron intervals (15min, hourly, 6 hours, daily, weekly)
- Automatic scheduling on plugin activation
- Interval change detection and rescheduling
- Admin settings integration
- Next run time display

**Hook:** `sqm_views_process_statistics_cron`

### 5. **REST API Endpoints**

#### Tracking: `POST /wp-json/sqm-views/v1/track`
- **Permission:** Public (no authentication)
- **Handler:** `SQMViewsRapidProcessor::process()`
- **Usage:** Fallback for JavaScript tracker when fast endpoint unavailable

#### Charts: `GET /wp-json/sqm-views/v1/charts`
- **Permission:** `manage_options` capability required
- **Handler:** `get_charts_data()` in `charts-api.php`

**Parameters:**
- `startDate` (YYYY-MM-DD) - Start of date range
- `endDate` (YYYY-MM-DD) - End of date range
- `eid` (int, optional) - Event ID filter
- `tid` (int, optional) - Trackable ID filter
- `metric` (string) - count | on_page | active | high_freq | low_freq
- `granularity` (string) - daily | by_content
- `limit` (int) - Max number of series for by_content

**Response Format:**
```json
{
  "series": [
    {
      "name": "Total",
      "values": [
        { "x": "2024-01-01", "y": 42 },
        { "x": "2024-01-02", "y": 56 }
      ]
    }
  ],
  "xLabel": "Date",
  "yLabel": "Count"
}
```

Note: The `name` field can be either `"Total"` for aggregated data or `"tid X"` where X is the trackable ID for individual content series.

### 6. **Admin Settings** (`src/SQMViewsSettings.php`)

**Admin Menu Structure:**
- SQMViews (parent menu)
  - Tracking Settings (trackable post types, taxonomies, data taxonomies)
  - Processing Settings (cron interval, handler type, JS options)
  - Statistics Dashboard (interactive charts via dashboard.bundle.js)

**Settings:**
- `sqm_views_trackable_post_types` - Array of post types to track
- `sqm_views_trackable_taxonomies` - Array of taxonomies pages to track
- `sqm_views_data_taxonomies` - Taxonomies tags/categories to add as metadata to pageviews
- `sqm_views_encryption_key` - Base64 encoded encryption key
- `sqm_views_cron_interval` - Cron frequency
- `sqm_views_handler` - fast | api | no (was not autodetect for some reason)
- `sqm_views_inline_js` - Boolean, inline vs external JS
- `sqm_views_min_js` - Boolean, minified vs source file JS

---

## Installation & Setup

1. Install from WordPress Plugin Directory or upload the plugin zip file
2. Activate plugin in WordPress admin
3. Plugin auto-creates database tables and directories
4. Configure trackable post types in **SQMViews → Tracking Settings**
5. Set cron interval in **SQMViews → Processing Settings**
6. View analytics in **SQMViews → Statistics Dashboard**

### Requirements
- WordPress 6.0+
- PHP 8.1+
- MySQL/MariaDB
- Write permissions for `wp-content/uploads/sqm-views_files/`

---

## Configuration Options

### Tracking Handler Types

**Fast Handler** (`sqm-views-pages.php`):
- Bypasses WordPress load for maximum performance
- Requires symlink: `wp-content/plugins/sqm-views/sqm-views-pages.php` → `/sqm-views-pages.php`
- Best for high-traffic sites
- Set via: `update_option(SQMVIEWS_OPTION_HANDLER, SQMVIEWS_HANDLERS_FAST)`

**API Handler** (REST API):
- Uses WordPress REST API endpoint
- Slower but easier to setup
- WordPress authentication and nonces available
- Set via: `update_option(SQMVIEWS_OPTION_HANDLER, SQMVIEWS_HANDLERS_SLOW)`

**No Handler**:
- Disables tracking
- Set via: `update_option(SQMVIEWS_OPTION_HANDLER, 'no')`

### JavaScript Options

**Inline vs External:**
- Inline: JS embedded in HTML (faster initial load, larger HTML)
- External: Separate `<script src>` tag (cached, async loading)
- Controlled by: `sqm_views_inline_js` option

**Minified vs Full:**
- Minified: Production-ready, smaller file size
- Full: Development, easier debugging
- Controlled by: `sqm_views_min_js` option

---

## WP-CLI Commands

```bash
# Process statistics manually
wp sqm-views process [--verbose]

# Run diagnostics
wp sqm-views test

# Clear cache/data
wp sqm-views clear

# View cron status
wp sqm-views cron-status
```

---

## Data Storage

### Directories (in `wp-content/uploads/sqm-views_files/`)
- `raw/` - Unprocessed JSONL files from tracking
- `processed/` - Successfully processed records not yet written to database (usually empty)
- `archive/` - processed raw files grouped by day are moved here for archiving
- `stale/` - incomplete sessions are moved here for archiving and troubleshooting

### Database Tables
- `wp_sqm_views_trackables` - Catalog of tracked entities
- `wp_sqm_views_events` - Event type definitions
- `wp_sqm_views_records` - Individual tracking records
- `wp_sqm_views_daily` - Daily aggregated statistics (primary query table)

**Indexes:**
- Optimized for date range queries
- Efficient filtering by trackable and event
- Fast aggregation operations

---

## Security Considerations

### Client-Side
- Tracking data encrypted before transmission
- Meta tag content sanitized with `esc_attr()`
- JavaScript loaded asynchronously (non-blocking)

### Server-Side
- Request size limits prevent DoS
- JSON validation prevents injection
- Action whitelisting prevents unauthorized operations
- REST API requires `manage_options` for charts
- WordPress nonces supported
- Input sanitization on all parameters
- SQL prepared statements prevent SQL injection

### File System
- Separate directory per day for raw files
- Unique file IDs prevent collisions
- Archive old files automatically
- Write-only access for tracking endpoint

---

## Performance Optimization

### Fast Endpoint
- Bypasses WordPress load (~200ms saved per request)
- Minimal PHP execution
- Direct file writes (no database I/O during tracking)
- CORS headers for CDN support

### Batch Processing
- Processes hundreds of records per cron run
- Single database connection for entire batch
- Bulk inserts where possible
- Transactions for data integrity

### Query Optimization
- Daily aggregation table reduces query complexity
- Composite indexes on filter columns
- Date partitioning for large datasets
- Metric column indexing

### Caching
- Dashboard bundle cached by browser
- REST API responses cacheable
- Static assets served via WordPress

---

## Troubleshooting

### No Tracking Data
1. Check tracking handler setting (fast vs API)
2. Verify JavaScript loaded on page (`View Source`)
3. Check browser console for errors
4. Test fast endpoint: `/wp-content/uploads/pvc.php`
5. Check file permissions on `sqm-views_files/raw/`

### Statistics Not Processing
1. Check cron scheduled: `wp cron event list`
2. Run manual processing: `wp sqm-views process --verbose`
3. Check `sqm-views_files/raw/` for pending files
4. Review Logger output for errors
5. Verify database tables exist

### Dashboard Not Loading
1. Verify `dashboard.bundle.js` exists in plugin directory
2. Check browser console for JS errors
3. Test REST API endpoint: `/wp-json/sqm-views/v1/charts?startDate=...`
4. Verify user has `manage_options` capability
5. Check WordPress nonce configuration

### High Database Usage
1. Reduce cron frequency
2. Implement data retention policy (delete old records)
3. Archive historical data to separate tables
4. Optimize indexes on large tables
5. Consider read replicas for reporting

---

## Dashboard Integration

The plugin includes an interactive analytics dashboard built with D3.js:

1. **REST API Endpoint:** `/wp-json/sqm-views/v1/charts` provides chart data
2. **Admin Interface:** Accessible via **SQMViews → Statistics Dashboard**
3. **Visualizations:** Interactive time-series charts with filtering options
4. **Data Export:** Query data via REST API or WP-CLI commands

---

## Debugging

- Enable verbose logging: `Logger::getInstance()->setVerbose(true)`
- Check `Logger` buffer output in admin notices
- Use `wp_debug_log()` for WordPress debugging
- Browser DevTools for JavaScript tracking issues
- WP-CLI commands for manual testing and diagnostics

---

## Constants Defined

(Defined in `src/includes/utils.php` and `src/includes/activation.php`)

**Directories:**
- `SQMVIEWS_DIR`            - Base directory for data files
- `SQMVIEWS_RAW_DIR`        - Unprocessed tracking files
- `SQMVIEWS_PROCESSED_DIR`  - Processed records
- `SQMVIEWS_ARCHIVE_DIR`    - Invalid/error records
- `SQMVIEWS_STALE_DIR`      - Old archived files

**Options:**
- `SQMVIEWS_OPTION_*`       - WordPress option keys for settings

**Events:**
- `SQMVIEWS_EVENTS_INIT`    - Page load event
- `SQMVIEWS_EVENTS_PING`    - Activity ping
- `SQMVIEWS_EVENTS_EXIT`    - User exit
- `SQMVIEWS_EVENTS_TIMEOUT` - Session timeout
- `SQMVIEWS_EVENTS_TEST`    - Diagnostic test

**Handlers:**
- `SQMVIEWS_HANDLERS_FAST`  - Fast endpoint mode
- `SQMVIEWS_HANDLERS_SLOW`  - REST API mode

**Limits:**
- `SQMVIEWS_REQUEST_LENGTH_HARD_LIMIT` - Max request size

---

## Plugin Extensibility & Hooks

SQMViews provides **30+ action and filter hooks** for developers to extend and customize plugin functionality without modifying core files.

### Hook Availability

The plugin supports two tracking endpoints with different hook availability:

| Endpoint | URL | Hook Availability | Performance |
|----------|-----|-------------------|-------------|
| **REST API** | `/wp-json/sqm-views/v1/track` | ✅ All hooks | Slower (full WP init) |
| **Fast Endpoint** | `/sqm-views-pages.php` | ⚠️ Most hooks* | Faster (minimal init) |

**\*Note:** Fast endpoint has limited availability for `sqm_views_session_timeout` and `sqm_views_ping_interval` hooks only (falls back to constants).

### Available Hook Categories

#### Tracker & Frontend Hooks (15 hooks)
- Control tracking behavior and what gets tracked
- Customize JavaScript loading and minification
- Modify tracking data before encryption
- Filter trackable post types and taxonomies

**Example - Exclude Admin Users:**
```php
add_filter( 'sqm_views_should_track', function( $should_track, $tracking_data ) {
    return ! current_user_can( 'manage_options' );
}, 10, 2 );
```

#### Data Processing Hooks (8 hooks)
- Monitor and modify record processing
- Filter raw records before processing
- Customize metrics calculation
- Add custom engagement scores

**Example - Add Custom Metrics:**
```php
add_filter( 'sqm_views_calculated_metrics', function( $metrics, $records ) {
    // Add engagement score based on active time
    $metrics['engagement_score'] = ( $metrics['active'] / max( 1, $metrics['on_page'] ) ) * 100;
    return $metrics;
}, 10, 2 );
```

#### REST API Hooks (3 hooks)
- Customize API permissions
- Transform chart data before response
- Modify allowed query parameters

**Example - Allow Editors to View Analytics:**
```php
add_filter( 'sqm_views_rest_permissions', function( $capability ) {
    return 'edit_posts'; // Instead of manage_options
} );
```

#### Activation & Configuration Hooks (4 hooks)
- Modify default settings
- Customize data storage location
- Control fast endpoint creation
- Execute actions on activation

**Example - Use Custom Storage Location:**
```php
add_filter( 'sqm_views_data_directory', function( $directory ) {
    return WP_CONTENT_DIR . '/custom-analytics-data';
} );
```

### Key Filter Hooks

- `sqm_views_should_track` - Control whether to track current page
- `sqm_views_tracking_data` - Modify tracking data before encryption
- `sqm_views_trackable_post_types` - Customize trackable post types
- `sqm_views_trackable_taxonomies` - Customize trackable taxonomies
- `sqm_views_raw_record` - Filter raw records (e.g., bot filtering)
- `sqm_views_calculated_metrics` - Modify calculated metrics
- `sqm_views_rest_chart_data` - Transform chart data
- `sqm_views_default_settings` - Modify default plugin settings
- `sqm_views_data_directory` - Customize data storage location

### Key Action Hooks

- `sqm_views_activated` - After plugin activation
- `sqm_views_before_tracker_output` - Before tracking code output
- `sqm_views_after_tracker_output` - After tracking code output
- `sqm_views_before_processing` - Before statistics processing
- `sqm_views_after_processing` - After statistics processing
- `sqm_views_record_processed` - After each record processed
- `sqm_views_daily_aggregated` - After daily aggregation

### Common Use Cases

**1. Filter Bot Traffic:**
```php
add_filter( 'sqm_views_raw_record', function( $data ) {
    $bot_patterns = array( 'bot', 'crawler', 'spider' );
    $user_agent = $data['server']['HTTP_USER_AGENT'] ?? '';

    foreach ( $bot_patterns as $pattern ) {
        if ( stripos( $user_agent, $pattern ) !== false ) {
            return null; // Skip this record
        }
    }
    return $data;
} );
```

**2. Track Custom Post Types:**
```php
add_filter( 'sqm_views_trackable_post_types', function( $post_types ) {
    $post_types[] = 'product';
    $post_types[] = 'course';
    return $post_types;
} );
```

**3. Add Custom Metadata:**
```php
add_filter( 'sqm_views_tracking_data', function( $tracking_data ) {
    if ( is_singular( 'post' ) ) {
        $tracking_data['author_name'] = get_the_author();
        $tracking_data['word_count'] = str_word_count( get_the_content() );
    }
    return $tracking_data;
} );
```

**4. Send Notifications:**
```php
add_action( 'sqm_views_activated', function() {
    wp_mail(
        get_option( 'admin_email' ),
        'SQMViews Activated',
        'Analytics tracking is now active.'
    );
} );
```

### Documentation

For complete hook reference with all parameters and examples, see the plugin's extensibility documentation.

### Benefits

- ✅ No core file modifications needed
- ✅ Customizations survive updates
- ✅ Clean separation of custom code
- ✅ WordPress coding standards compliant
- ✅ Easy to test and maintain

---

## Future Enhancements (Potential)

- Real-time analytics (WebSocket support)
- User segmentation and cohort analysis
- A/B testing integration
- Conversion tracking
- Heatmap visualization
- Session recording
- Export to Google Analytics, Matomo, etc.
- Privacy controls (GDPR compliance features)
- Bot detection and filtering
- Referrer tracking and attribution

---

## Questions to Clarify with Maintainer

1. **Encryption:** What encryption algorithm is used for tracker payload? Is it just base64 or actual encryption?
2. **Data Retention:** Is there an automatic data cleanup/archival policy?
3. **Trackable Types:** Can users track custom entities beyond post types and taxonomies?
4. **Privacy:** How does the plugin handle PII? Is there IP anonymization?
5. **Scaling:** What's the expected traffic volume? Any horizontal scaling considerations?
6. **Bot Filtering:** Is there bot detection to exclude crawler traffic?
7. **Fast Endpoint Setup:** Is the symlink created automatically or requires manual setup?
8. **Metrics Definition:** Can you clarify exact calculation for high_freq vs low_freq?

---

## Versioning

This plugin follows [Semantic Versioning](https://semver.org/) (MAJOR.MINOR.PATCH).

**Current Version:** 1.2.4

### Version Check in Code

```php
// Using constant (available everywhere after utils.php is loaded)
$version = SQMVIEWS_BUILD_GLOBAL_VERSION;

// Using WordPress plugin data
if ( function_exists( 'get_plugin_data' ) ) {
    require_once ABSPATH . 'wp-admin/includes/plugin.php';
    $plugin_file = __DIR__ . '/sqm-views.php';
    $plugin_data = get_plugin_data( $plugin_file );
    $version = $plugin_data['Version'];
}
```

---

**Version:** 1.2.4
**Author:** Pavel Khloponin
**License:** GPLv3 or later
**Text Domain:** sqm-views

**Last Updated:** 2025-10-26
**Purpose:** Plugin for page view tracking and analytics