=== VigIA - AI Crawler Activity Analytics & Control === Contributors: fernandot, ayudawp Tags: ai, analytics, gpt, claude, llms Requires at least: 6.2 Tested up to: 6.9 Requires PHP: 7.4 Stable tag: 1.2.2 License: GPLv2 or later License URI: https://www.gnu.org/licenses/gpl-2.0.html Track, analyze, block or control AI crawler visits to your site. Monitor 50+ AI crawlers with blocking, robots.txt management, and llms.txt generation == Description == **VigIA** (Spanish for "lookout" or "watchman", incorporating "IA" - Spanish for "AI") is a comprehensive WordPress plugin that tracks, analyzes, and helps you control AI crawler activity on your website. = What does VigIA do? = * **Tracks AI crawlers** visiting your site (GPTBot, ClaudeBot, PerplexityBot, and 50+ others) * **Provides detailed analytics** with charts, statistics, and exportable reports * **Blocks unwanted crawlers** via PHP (403 response) * **Manages robots.txt rules** for AI crawlers with compliance monitoring * **Sends email alerts** about crawler activity (daily, weekly, or monthly) * **Generates llms.txt files** to help AI systems understand your site = Key Features = **Analytics Dashboard** * Total visits, unique crawlers, and pages crawled statistics * Timeline chart with daily breakdown * Category distribution (AI Training, AI Search, AI Assistant, Data Scraper) * Top crawlers and most crawled pages tables * Recent activity log with filters * Period comparison functionality * CSV export **Crawler Blocking** * Block crawlers via PHP with 403 Forbidden response * Quick block dropdown in analytics dashboard * Manage blocks from Extras page * Works on any server (Apache, Nginx, LiteSpeed, etc.) **Robots.txt Management** * Add Disallow rules for AI crawlers * Visual preview of your robots.txt * Compliance monitoring: see which crawlers ignore your rules * One-click blocking for non-compliant crawlers * Works with both physical and virtual robots.txt **Email Alerts** * Daily, weekly, or monthly reports * Three detail levels: Minimal, Normal, Complete * Non-compliant crawler warnings * Activity comparison with previous period **LLMs.txt Generator** * Select content by post type with one click * Filter by taxonomies (categories, tags, custom) * Manual include/exclude with AJAX search * Exclude by URL patterns (wildcards supported) * SEO plugin integration (auto-exclude noindex content) * Auto-regeneration (daily, weekly, monthly) * Robots.txt integration (add llms.txt and llms-full.txt references) * Generate llms.txt and llms-full.txt files * Full content or excerpt mode * Compatible with Yoast SEO, Rank Math, All in One SEO, SEOPress, and The SEO Framework = Supported AI Crawlers = VigIA monitors 50+ AI crawlers including: * **OpenAI**: GPTBot, OAI-SearchBot, ChatGPT-User * **Anthropic**: ClaudeBot, Claude-Web, Claude-SearchBot * **Google**: Google-Extended, GoogleOther, Gemini-Deep-Research * **Perplexity**: PerplexityBot, Perplexity-User * **Meta**: Meta-ExternalAgent, FacebookBot * **Microsoft**: BingBot * **ByteDance**: Bytespider * **Amazon**: Amazonbot * **Apple**: Applebot-Extended * **And many more...** = Privacy Focused = VigIA stores visitor data locally in your WordPress database. No data is sent to external servers. == Installation == 1. Upload the plugin files to `/wp-content/plugins/vigia/` or install through the WordPress plugins screen 2. Activate the plugin through the 'Plugins' screen in WordPress 3. Navigate to **VigIA** in the admin menu to view your AI crawler analytics 4. Configure blocking rules and alerts in **VigIA > Extras** == Frequently Asked Questions == = Does this plugin slow down my site? = No. VigIA adds minimal overhead by checking the User-Agent string on each request. The check is very fast and only writes to the database when an AI crawler is detected. = What's the difference between robots.txt blocking and PHP blocking? = **Robots.txt** is advisory - crawlers should respect it but may choose to ignore it. **PHP blocking** returns a 403 Forbidden response, effectively preventing access regardless of whether the crawler respects robots.txt. = Will blocking crawlers affect my SEO? = Blocking AI training crawlers (like GPTBot or ClaudeBot) will not affect traditional search engine rankings. However, blocking AI search crawlers might affect how your content appears in AI-powered search results. = Can I add custom crawlers to monitor? = Yes! In the main analytics page, scroll down to "Custom crawlers" and add your own User-Agent patterns to track. = Where is the data stored? = All data is stored in your WordPress database in a custom table (`wp_vigia_visits`). No data leaves your server. = What is llms.txt? = The llms.txt file is a standard for helping AI systems understand your website's content and structure. It provides a machine-readable overview of your site that AI can use to better represent your content. Learn more at llmstxt.org. = Which SEO plugins are supported for noindex detection? = VigIA supports automatic noindex detection from: Yoast SEO, Rank Math, All in One SEO, SEOPress, and The SEO Framework. == Screenshots == 1. Main analytics dashboard with charts and statistics 2. Timeline showing crawler activity over time 3. Top crawlers and most crawled pages tables 4. Recent activity log with filtering 5. Custom crawler configuration 6. Dashboard widget 7. Extras page - Robots.txt management and blocking 8. Extras page - Email alerts configuration 9. Extras page - LLMs.txt generator with content type selection == Changelog == = 1.2.2 = * Improved: Added PHPCS ignore comments for intentional direct database queries (cache bypass) * Improved: Added translator comments for all placeholder strings in LLMs generator = 1.2.1 = * Fix: LLMs.txt generator now correctly handles taxonomy filters for each post type * Fix: Taxonomy filters (categories, tags) no longer incorrectly exclude pages or other post types * Fix: Resolved object cache conflicts on hosts with Memcached/Redis (SiteGround, etc.) * Fix: Settings now persist correctly after saving and regenerating * Fix: Robots.txt references only added when llms files actually exist * Fix: Robots.txt integration reads fresh data bypassing object cache * Improved: Added "Save settings" button to save options without generating files * Improved: Direct database queries for settings to avoid cache inconsistencies * Improved: Better boolean normalization for checkbox values = 1.2.0 = * NEW: Redesigned LLMs.txt generator with improved content selection * NEW: Select content by post type (posts, pages, custom post types) * NEW: Filter content by taxonomies (categories, tags, custom taxonomies) * NEW: Manual include with AJAX-powered search * NEW: Manual exclude with AJAX-powered search * NEW: Exclude content by URL patterns with wildcard support * NEW: SEO plugin integration - auto-exclude noindex content * NEW: Supported SEO plugins: Yoast, Rank Math, AIOSEO, SEOPress, The SEO Framework * NEW: Auto-regeneration options (daily, weekly, monthly) * NEW: Add llms.txt and llms-full.txt references to robots.txt * Improved: Scalable architecture for sites with thousands of posts * Improved: Better UI/UX for content selection = 1.1.1 = * Fix: Version dump to improve texts. = 1.1.0 = * NEW: PHP-based crawler blocking (403 Forbidden response) * NEW: Robots.txt management for AI crawlers * NEW: Compliance monitoring (detect crawlers ignoring robots.txt) * NEW: Email alerts system (daily/weekly/monthly reports) * NEW: LLMs.txt and llms-full.txt file generator * NEW: Extras page with all new management tools * NEW: Quick block dropdown in crawlers table * Improved: Block actions directly from analytics dashboard = 1.0.0 = * Initial release * AI crawler detection and tracking * Analytics dashboard with charts * CSV data export * Custom crawler support * Dashboard widget == Upgrade Notice == = 1.2.1 = Bug fix release. Fixes taxonomy filter issues in LLMs.txt generator and resolves object cache conflicts on managed hosting. Recommended update for all users. = 1.2.0 = Major update to LLMs.txt generator with post type selection, taxonomy filters, SEO plugin integration, and auto-regeneration. Your existing settings will be preserved. = 1.1.1 = Update to support correct translations. = 1.1.0 = Major update with new blocking features, email alerts, and llms.txt generation. Check the new Extras page after updating. == Support == Need help or have suggestions? * [Official website](https://servicios.ayudawp.com) * [WordPress support forum](https://wordpress.org/support/plugin/vigia/) * [YouTube channel](https://www.youtube.com/AyudaWordPressES) * [Documentation and tutorials](https://ayudawp.com) Love the plugin? Please leave us a 5-star review and help spread the word! == About AyudaWP == We are specialists in WordPress security, SEO, and performance optimization plugins. We create tools that solve real problems for WordPress site owners while maintaining the highest coding standards and accessibility requirements.