# Shadowdog 🐾 drawing **Shadowdog** is a Node.js package for generating artifacts as derivative processes of files in your project. Whether you need to generate static assets, precompiled resources, or any other transformations, Shadowdog makes it easy and powerful with its supercharged feature set. ## Features πŸš€ - **Flexible Input Detection:** Automatically detects changes in files and processes only the updated ones. - **Configurable Pipelines:** Define artifact generation workflows using an intuitive JSON configuration file. - **Plugin Support:** Extend functionality with custom or community-built plugins. - **Blazing Fast Performance:** Optimized for speed, even with large repositories. - **Watch Mode:** Automatically regenerate artifacts when source files change. --- ## Installation Install Shadowdog via npm: ```bash npm install shadowdog --save-dev ``` --- ## Getting started πŸ• Shadowdog uses a configuration file (`shadowdog.json`) to define workflows for generating artifacts. Here’s an example: ```json { "$schema": "https://raw.githubusercontent.com/factorialco/shadowdog/refs/heads/main/schema.json", "plugins": [], "watchers": [ { "files": ["example.txt"], "commands": [ { "artifacts": [ { "output": "example.output.txt" } ], "command": "cp example.txt example.output.txt" } ] } ] } ``` ### Key fields - **`$schema`**: Provides schema validation for the configuration. - **`plugins`**: An array of plugin names to extend Shadowdog's functionality. - **`watchers`**: Defines file watchers that trigger artifact generation commands. - `files`: An array of file paths or glob patterns to watch. - `commands`: Commands to execute when changes are detected. - `artifacts`: Specifies the output files generated by the command. - `command`: The shell command to run. --- ## CLI commands Shadowdog provides a variety of commands to simplify your workflows: - **Generate artifacts**: ```bash npx shadowdog ``` - **Watch mode** (includes MCP server for external tool integration): ```bash npx shadowdog --watch ``` --- ## Available plugins 🧩 Enhance Shadowdog with these powerful plugins: - **`shadowdog-local-cache`** Implements a local caching mechanism to speed up repeated artifact generation. Environment variables: - `SHADOWDOG_DISABLE_LOCAL_CACHE`: When `true`, disables local cache completely - `SHADOWDOG_LOCAL_CACHE_READ`: When set, overrides the plugin's read cache configuration (`true`/`false`) - `SHADOWDOG_LOCAL_CACHE_WRITE`: When set, overrides the plugin's write cache configuration (`true`/`false`) - `SHADOWDOG_LOCAL_CACHE_PATH`: When set, overrides the plugin's cache directory path - **`shadowdog-remote-aws-s3-cache`** Enables remote caching with AWS S3 for distributed workflows. Environment variables: - `SHADOWDOG_DISABLE_REMOTE_CACHE`: When `true`, disables remote cache completely - `SHADOWDOG_REMOTE_CACHE_READ`: When set, overrides the plugin's read cache configuration (`true`/`false`) - `SHADOWDOG_REMOTE_CACHE_WRITE`: When set, overrides the plugin's write cache configuration (`true`/`false`) - `SHADOWDOG_REMOTE_CACHE_EXTRA`: When set, adds extra information to the cache key in S3 - `AWS_PROFILE`: AWS profile to use for authentication (optional) - `AWS_ACCESS_KEY_ID`: AWS access key ID (required if AWS_PROFILE not set) - `AWS_SECRET_ACCESS_KEY`: AWS secret access key (required if AWS_PROFILE not set) - `AWS_REGION`: AWS region (required if AWS_PROFILE not set) - **`shadowdog-tag`** Adds tagging capabilities to filter specific commands. Environment variables: - `SHADOWDOG_TAG`: When set, only runs commands with matching tag - **`shadowdog-lock`** Generates a `shadowdog-lock.json` file that tracks artifact metadata and dependencies for reproducible builds. The lock file includes: - Shadowdog version used for generation - Node.js version for environment consistency - Artifact cache identifiers and file manifests - Watched files with wildcard expansion - Environment variable values used for invalidation - Command details for each artifact Features: - **Deterministic output**: All arrays are sorted for consistent lock files - **Partial updates**: Only updates changed artifacts in watch mode - **Race condition safe**: Handles concurrent task execution - **Wildcard support**: Expands file patterns like `src/*.ts` automatically - **Relative paths**: All file paths are relative to project root - **`shadowdog-git`** Handles git rebases and merges smoothly pausing the watcher and resuming it after the rebase is done. Internal configuration: - Checks for rebase every 2000ms (INTERVAL_TIME) - Uses `.git/rebase-merge` to detect rebase state - **`shadowdog-socket`** Provides an external communication channel for interacting with Shadowdog. No configurable environment variables. Uses socket events: - `CHANGED_FILE`: Emitted when a file changes - `ERROR`: Emitted on errors - `INITIALIZED`: Emitted on startup - `CLEAR`: Emitted on cleanup - **`shadowdog-tree`** Generate a dependency tree structure between commands to run different commands that depend on each other. No configurable environment variables. Uses internal dependency graph algorithm. - **`shadowdog-rake`** Optimize multiple `bundle exec rake` commands into a single command. No configurable environment variables. Automatically detects and combines rake tasks. - **`shadowdog-mcp`** Provides a Model Context Protocol (MCP) server for external tools (like Cursor AI) to interact with Shadowdog programmatically. Environment variables: - `SHADOWDOG_MCP_PORT`: HTTP port for MCP server (default: `8473`) - `SHADOWDOG_MCP_HOST`: HTTP host for MCP server (default: `localhost`) Available MCP Tools: - `pause-shadowdog`: Pauses shadowdog in watch mode to prevent artifact generation during code changes (properly integrated with daemon) - `resume-shadowdog`: Resumes shadowdog after being paused (properly integrated with daemon) - `get-artifacts`: Retrieves information about all artifacts, including status, last update time, and associated files - `compute-artifact`: Generates a specific artifact using the same task runner and middleware as the daemon - `get-shadowdog-status`: Gets the current status of shadowdog including daemon availability and configuration summary **Connection**: HTTP endpoint at `http://localhost:8473/mcp` (configurable via environment variables) **Cursor Integration**: To connect with Cursor, add this to your MCP configuration: ```json { "mcpServers": { "shadowdog": { "url": "http://localhost:8473/mcp" } } } ``` For detailed Cursor MCP setup instructions, see: [Cursor MCP Installation Guide](https://cursor.com/docs/context/mcp/install-links) ### Using plugins To use a plugin, add it to the `plugins` section of your `shadowdog.json` configuration file. For example: Update your configuration: ```json { ... "plugins": [ { "name": "shadowdog-local-cache", }, { "name" : "shadowdog-tree" } ] ... } ``` Take into account that the order of plugins is important. The plugins will be executed in the order they are defined in the configuration file. --- ## MCP Integration πŸ€– Shadowdog includes a built-in Model Context Protocol (MCP) server that allows AI tools like Cursor to interact with your build system programmatically. The MCP server starts automatically when you run Shadowdog in watch mode. [![Install MCP Server](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/en/install-mcp?name=shadowdog&config=eyJ1cmwiOiJodHRwOi8vbG9jYWxob3N0Ojg0NzMvbWNwIn0%3D) ### Quick Setup 1. **Add the MCP plugin to your configuration:** ```json { "plugins": [ { "name": "shadowdog-mcp" } ] } ``` 2. **Start in watch mode** (MCP server starts automatically): ```bash npx shadowdog --watch ``` 3. **Configure Cursor:** Add this to your Cursor MCP configuration: ```json { "mcpServers": { "shadowdog": { "url": "http://localhost:8473/mcp" } } } ``` ### Available MCP Tools Once connected, you can use these tools in Cursor: - **`pause-shadowdog`** - Pause artifact generation during code changes (properly integrated with daemon) - **`resume-shadowdog`** - Resume artifact generation after changes (properly integrated with daemon) - **`get-artifacts`** - Query artifact status and information - **`compute-artifact`** - Generate specific artifacts on demand using the same task runner as the daemon - **`get-shadowdog-status`** - Check shadowdog's current state and daemon availability ### Advanced Configuration - **Custom Port**: Set `SHADOWDOG_MCP_PORT=9000` to use a different port - **Custom Host**: Set `SHADOWDOG_MCP_HOST=0.0.0.0` to allow external connections For detailed Cursor MCP setup instructions, see: [Cursor MCP Installation Guide](https://cursor.com/docs/context/mcp/install-links) --- ## How Caching Works πŸ”„ Shadowdog uses a sophisticated caching system to determine when artifacts need to be regenerated. Understanding how cache keys are computed is crucial for optimizing your workflows. ### Cache Key Computation Cache identifiers are generated using the `computeCache` function, which creates deterministic hashes based on: 1. **File Contents**: All watched files are read and their contents included in the hash 2. **File Paths**: Absolute file paths are used for consistency across environments 3. **Environment Variables**: Only specified environment variables are included 4. **Command**: The exact command string is hashed 5. **Shadowdog Version**: Ensures cache invalidation when Shadowdog is updated ### Key Features - **Deterministic**: Same inputs always produce the same cache key - **Sorted Files**: File paths are sorted alphabetically for consistency - **Wildcard Expansion**: Glob patterns like `src/*.ts` are expanded before hashing - **Environment Aware**: Only specified environment variables affect the cache - **Version Sensitive**: Cache invalidates when Shadowdog version changes ### Best Practices - **Use specific file patterns**: Avoid overly broad glob patterns that might include unwanted files - **Environment variables**: Only include variables that actually affect your build process - **File organization**: Keep related files together for better cache efficiency - **Version pinning**: Use specific Shadowdog versions in CI/CD for reproducible builds --- ## License πŸ“„ Shadowdog is open source and available under the [MIT License](./LICENSE). --- ## Feedback & Support ❀️ If you encounter any issues, have questions, or want to suggest features, please open an issue or join the discussions. --- Enjoy artifact generation, **supercharged**! 🐾