{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"","title":"llm.energy","text":"

Extract documentation for AI agents. Fetch llms.txt and install.md from any documentation site, organized into downloadable markdown ready for Claude, ChatGPT, and other AI assistants.

Get Started View on GitHub

"},{"location":"#features","title":"Features","text":""},{"location":"#quick-start","title":"Quick Start","text":"Web AppMCP ServerAPI
  1. Visit llm.energy
  2. Enter a documentation URL
  3. Download the extracted files

Add to your MCP client configuration:

{\n  \"mcpServers\": {\n    \"llm-energy\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"@llm-energy/mcp-server\"]\n    }\n  }\n}\n

See MCP Server Installation for detailed setup.

curl -X POST https://llm-energy-lp642k3kpa-uc.a.run.app/api/extract \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"url\": \"docs.anthropic.com\"}'\n

See API Reference for all endpoints.

"},{"location":"#supported-standards","title":"Supported Standards","text":"

llm.energy supports two complementary standards for LLM-friendly documentation:

Standard Description Learn More llms.txt Machine-readable documentation at /llms.txt llms.txt Standard install.md LLM-executable installation instructions install.md Standard"},{"location":"#links","title":"Links","text":""},{"location":"api-reference/","title":"API Reference","text":"

REST API for llm.energy.

"},{"location":"api-reference/#base-url","title":"Base URL","text":"
https://llm-energy-lp642k3kpa-uc.a.run.app/api\n
"},{"location":"api-reference/#endpoints","title":"Endpoints","text":""},{"location":"api-reference/#extract-documentation","title":"Extract Documentation","text":"
POST /extract\n

Request Body

{\n  \"url\": \"docs.anthropic.com\"\n}\n

Response

{\n  \"success\": true,\n  \"source\": \"llms-full.txt\",\n  \"documents\": [\n    {\n      \"name\": \"getting-started.md\",\n      \"title\": \"Getting Started\",\n      \"tokens\": 1250\n    }\n  ],\n  \"stats\": {\n    \"totalDocuments\": 12,\n    \"totalTokens\": 45000,\n    \"processingTimeMs\": 234\n  }\n}\n
"},{"location":"api-reference/#fetch-raw-content","title":"Fetch Raw Content","text":"
GET /fetch?url={url}&full={boolean}\n

Parameters

Name Type Default Description url string required Documentation site URL full boolean true Fetch llms-full.txt

Response

{\n  \"success\": true,\n  \"content\": \"# Documentation\\n\\n...\",\n  \"source\": \"llms-full.txt\"\n}\n
"},{"location":"api-reference/#error-responses","title":"Error Responses","text":"
{\n  \"success\": false,\n  \"error\": \"No llms.txt found at the specified URL\"\n}\n
Status Description 400 Invalid request parameters 404 No llms.txt found 500 Server error"},{"location":"api-reference/#validate-installmd","title":"Validate install.md","text":"
POST /validate-install\n

Request Body

{\n  \"content\": \"# my-tool\\n\\nOBJECTIVE: Install the tool...\"\n}\n

Or validate from URL:

{\n  \"url\": \"https://example.com/install.md\"\n}\n

Response

{\n  \"valid\": true,\n  \"parsed\": {\n    \"productName\": \"my-tool\",\n    \"objective\": \"Install the tool\",\n    \"doneWhen\": \"Tool is working\",\n    \"todoItems\": [...],\n    \"steps\": [...]\n  }\n}\n
"},{"location":"api-reference/#check-installmd-existence","title":"Check install.md Existence","text":"
GET /validate-install?url={url}\n

Response

{\n  \"exists\": true,\n  \"url\": \"https://example.com/install.md\",\n  \"valid\": true,\n  \"productName\": \"my-tool\"\n}\n
"},{"location":"examples/","title":"Examples","text":"

Real-world examples of using llm.energy.

"},{"location":"examples/#extracting-anthropic-docs","title":"Extracting Anthropic Docs","text":"
# Web App\nhttps://llm-energy-lp642k3kpa-uc.a.run.app\n\n# Enter URL\ndocs.anthropic.com\n

Result: 15+ markdown files covering Claude API, prompt engineering, and more.

"},{"location":"examples/#extracting-mcp-docs","title":"Extracting MCP Docs","text":"
# URL\nmodelcontextprotocol.io\n

Result: Complete MCP specification, server implementation guides, and client examples.

"},{"location":"examples/#extracting-stripe-docs","title":"Extracting Stripe Docs","text":"
# URL\ndocs.stripe.com\n

Result: API reference, integration guides, and webhook documentation.

"},{"location":"examples/#using-with-claude","title":"Using with Claude","text":"
  1. Extract documentation using llm.energy
  2. Download the ZIP archive
  3. In Claude, start a new conversation
  4. Upload llms-full.md or individual files
  5. Ask questions about the documentation

Example prompt:

\"Based on the uploaded documentation, how do I implement streaming responses?\"

"},{"location":"examples/#using-with-chatgpt","title":"Using with ChatGPT","text":"
  1. Extract documentation
  2. Download files
  3. Upload to ChatGPT (GPT-4 with file upload)
  4. Reference the docs in your prompts
"},{"location":"examples/#using-installmd","title":"Using install.md","text":""},{"location":"examples/#extracting-installmd","title":"Extracting install.md","text":"

When extracting from a site that has install.md, llm.energy will automatically detect and include it:

# URL with install.md\npython.langchain.com\n

Result: Documentation files + install.md with installation instructions.

"},{"location":"examples/#creating-installmd","title":"Creating install.md","text":"

Use the install.md Generator to create your own:

  1. Go to the Generator page
  2. Fill in product name, objective, and done criteria
  3. Add TODO items (checklist for the LLM)
  4. Add installation steps with code blocks
  5. Download the generated install.md
"},{"location":"examples/#using-installmd-with-ai-agents","title":"Using install.md with AI Agents","text":"

Give an AI agent the install.md file and ask it to install the software:

Example prompt:

\"Follow the instructions in install.md to set up this tool on my machine\"

The agent will:

  1. Read the OBJECTIVE to understand the goal
  2. Execute each step in the TODO list
  3. Verify completion against DONE WHEN criteria
"},{"location":"examples/#using-the-mcp-server","title":"Using the MCP Server","text":"

With the MCP server, AI assistants can extract docs programmatically:

Claude Desktop

You: \"Extract the Vercel documentation and tell me about edge functions\"\n\nClaude: [Uses extract_documentation tool]\n        [Reads relevant sections]\n        \"Based on the Vercel docs, edge functions are...\"\n

Example conversation:

You: \"What MCP tools do you have for documentation?\"\n\nClaude: \"I have access to llm-energy with these tools:\n        - extract_documentation: Extract docs from a URL\n        - fetch_llms_txt: Get raw llms.txt content\n        - get_document: Retrieve a cached document\n        - list_documents: List all cached docs\"\n\nYou: \"Extract docs from docs.anthropic.com\"\n\nClaude: [Extracts and summarizes the documentation]\n
"},{"location":"examples/#supported-sites","title":"Supported Sites","text":"

Sites known to have llms.txt support:

Site URL Anthropic docs.anthropic.com MCP modelcontextprotocol.io Stripe docs.stripe.com Vercel vercel.com/docs Supabase supabase.com/docs Mintlify mintlify.com/docs

Check any site

Try adding /llms.txt or /llms-full.txt to any documentation URL to see if it's supported.

"},{"location":"install-md-standard/","title":"The install.md Standard","text":"

Understanding the install.md standard for LLM-executable installation instructions.

"},{"location":"install-md-standard/#what-is-installmd","title":"What is install.md?","text":"

The install.md standard is a convention for providing installation instructions in a format designed for AI agents to execute autonomously.

Similar to how llms.txt provides documentation context for LLMs, install.md provides structured installation instructions that LLMs can follow step-by-step.

"},{"location":"install-md-standard/#why-installmd","title":"Why install.md?","text":"

The Problem:

Traditional installation documentation is written for humans. It uses prose, assumes context, and often requires interpretation. AI agents need explicit, step-by-step instructions with clear success criteria.

The Solution:

install.md provides:

"},{"location":"install-md-standard/#file-location","title":"File Location","text":"File Location /install.md Root of documentation or project /docs/install.md Documentation subdirectory"},{"location":"install-md-standard/#format","title":"Format","text":"

install.md uses a specific format with keywords that guide LLM execution:

# product-name\n\n> Description of the product\n\nI want you to install [Product] for me. Execute all the steps below autonomously.\n\nOBJECTIVE: What the installation should achieve.\n\nDONE WHEN: Specific verification criteria.\n\n## TODO\n\n- [ ] First step to complete\n- [ ] Second step to complete\n- [ ] Verify installation\n\n## Prerequisites\n\nInstructions for prerequisites...\n\n```bash\nnode --version\n
"},{"location":"install-md-standard/#install","title":"Install","text":"

Installation instructions...

npm install -g product-name\n
"},{"location":"install-md-standard/#verify","title":"Verify","text":"

Verification instructions...

product-name --version\n

EXECUTE NOW: Complete the above TODO list to achieve: [objective]. 

---\n\n## Required Elements\n\n| Element | Purpose |\n|---------|---------|\n| `# product-name` | H1 header with lowercase-hyphenated product name |\n| `OBJECTIVE:` | What the installation should achieve |\n| `DONE WHEN:` | Specific verification criteria |\n| `## TODO` | Markdown checkbox list of steps |\n| `EXECUTE NOW:` | Final call-to-action referencing TODO and objective |\n\n---\n\n## Optional Elements\n\n| Element | Purpose |\n|---------|---------|\n| `> description` | Blockquote describing the product |\n| Action prompt | \"I want you to install...\" instruction |\n| Step sections | `## Prerequisites`, `## Install`, etc. |\n| Code blocks | Commands in fenced code blocks |\n| Platform variants | Multiple options for different environments |\n\n---\n\n## Usage\n\nUsers can pipe install.md directly to an AI agent:\n\n```bash\n# Using Claude\ncurl -fsSL https://docs.example.com/install.md | claude\n\n# Using other AI tools\ncurl -fsSL https://docs.example.com/install.md | ai-agent\n

Or paste the contents into any LLM conversation.

"},{"location":"install-md-standard/#relationship-to-llmstxt","title":"Relationship to llms.txt","text":"File Purpose llms.txt Helps LLMs understand your software install.md Tells LLMs how to install your software

Your install.md can link to llms.txt for troubleshooting, configuration details, or additional context during installation.

"},{"location":"install-md-standard/#best-practices","title":"Best Practices","text":"

1. Be Explicit

Write for an agent that has no prior context. State exactly what needs to happen.

2. Include Verification

Every installation should have a way to verify success. Include commands that return expected output.

3. Handle Alternatives

Provide options for different package managers, operating systems, or configurations.

4. Keep TODO Focused

Each TODO item should be a clear, completable task.

5. Use Code Blocks

Always use fenced code blocks with language hints for commands.

"},{"location":"install-md-standard/#llmenergy-support","title":"llm.energy Support","text":"

llm.energy automatically:

"},{"location":"install-md-standard/#learn-more","title":"Learn More","text":""},{"location":"llms-txt-standard/","title":"The llms.txt Standard","text":"

Understanding the llms.txt standard for AI-readable documentation.

"},{"location":"llms-txt-standard/#what-is-llmstxt","title":"What is llms.txt?","text":"

The llms.txt standard is a convention for providing documentation in a format optimized for Large Language Models (LLMs).

Similar to how robots.txt tells search engines how to crawl a site, llms.txt provides AI assistants with structured documentation.

"},{"location":"llms-txt-standard/#file-locations","title":"File Locations","text":"File Purpose /llms.txt Concise overview and key information /llms-full.txt Complete documentation"},{"location":"llms-txt-standard/#format","title":"Format","text":"

llms.txt files are plain markdown:

# Project Name\n\n> Brief description\n\n## Section 1\n\nContent...\n\n## Section 2\n\nContent...\n
"},{"location":"llms-txt-standard/#why-it-matters","title":"Why It Matters","text":"

For AI Assistants:

For Developers:

"},{"location":"llms-txt-standard/#adoption","title":"Adoption","text":"

Sites with llms.txt support include:

"},{"location":"llms-txt-standard/#adding-llmstxt-to-your-site","title":"Adding llms.txt to Your Site","text":"

Option 1: Manual

Create /public/llms.txt or /public/llms-full.txt with your documentation in markdown format.

Option 2: Automated

Documentation platforms like Mintlify automatically generate llms.txt from your docs.

Best Practices:

"},{"location":"llms-txt-standard/#learn-more","title":"Learn More","text":""},{"location":"overview/","title":"Overview","text":"

How llm.energy works.

"},{"location":"overview/#supported-standards","title":"Supported Standards","text":"

llm.energy supports two complementary standards for LLM-friendly documentation:

"},{"location":"overview/#llmstxt-standard","title":"llms.txt Standard","text":"

Many documentation sites provide their content in a machine-readable format at /llms.txt or /llms-full.txt. This format is designed for AI consumption.

Endpoint Description /llms.txt Concise version /llms-full.txt Complete documentation"},{"location":"overview/#installmd-standard","title":"install.md Standard","text":"

The install.md standard provides structured installation instructions that LLMs can execute autonomously.

Endpoint Description /install.md LLM-executable installation guide /docs/install.md Alternative location

install.md files include:

"},{"location":"overview/#how-it-works","title":"How It Works","text":"
flowchart LR\n    A[Enter URL] --> B[Fetch llms.txt]\n    B --> C[Fetch install.md]\n    C --> D[Parse Content]\n    D --> E[Generate Files]\n    E --> F[Download]
"},{"location":"overview/#1-fetch","title":"1. Fetch","text":"

llm.energy fetches:

  1. {url}/llms-full.txt (complete documentation)
  2. Falls back to {url}/llms.txt if full version unavailable
  3. {url}/install.md (installation instructions)
  4. Also checks /docs/install.md and docs.{domain}/install.md
"},{"location":"overview/#2-parse","title":"2. Parse","text":"

Content is split into sections based on markdown headers:

"},{"location":"overview/#3-generate","title":"3. Generate","text":"

Four types of output:

File Description *.md Individual section files llms-full.md Consolidated document with TOC AGENT-GUIDE.md Instructions for AI assistants install.md Installation instructions (if available)"},{"location":"overview/#supported-sites","title":"Supported Sites","text":"

Any site that provides /llms.txt or /llms-full.txt will work:

"},{"location":"self-hosting/","title":"Self Hosting","text":"

Deploy llm.energy on your own infrastructure.

"},{"location":"self-hosting/#requirements","title":"Requirements","text":""},{"location":"self-hosting/#clone-install","title":"Clone & Install","text":"
git clone https://github.com/nirholas/extract-llms-docs.git\ncd extract-llms-docs\npnpm install\n
"},{"location":"self-hosting/#development","title":"Development","text":"
pnpm dev\n

Open http://localhost:3001

"},{"location":"self-hosting/#production-build","title":"Production Build","text":"
pnpm build\npnpm start\n
"},{"location":"self-hosting/#deploy-to-vercel","title":"Deploy to Vercel","text":"

Or via CLI:

vercel --prod\n
"},{"location":"self-hosting/#environment-variables","title":"Environment Variables","text":"Variable Description Default PORT Server port 3001"},{"location":"self-hosting/#mcp-server","title":"MCP Server","text":"

To run the MCP server locally:

cd llms-forge/mcp-server\npnpm install\npnpm build\npnpm start\n
"},{"location":"mcp-server/installation/","title":"Installation","text":"

Set up the llm.energy MCP server for AI assistants.

"},{"location":"mcp-server/installation/#what-is-mcp","title":"What is MCP?","text":"

The Model Context Protocol (MCP) allows AI assistants to interact with external tools and data sources.

"},{"location":"mcp-server/installation/#quick-setup","title":"Quick Setup","text":"npx (Recommended)Global Install

No installation required:

{\n  \"mcpServers\": {\n    \"llm-energy\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"@llm-energy/mcp-server\"]\n    }\n  }\n}\n
npm install -g @llm-energy/mcp-server\n
{\n  \"mcpServers\": {\n    \"llm-energy\": {\n      \"command\": \"llm-energy-mcp\"\n    }\n  }\n}\n
"},{"location":"mcp-server/installation/#configuration","title":"Configuration","text":"Claude DesktopVS Code / Cursor

Add to your Claude Desktop config:

OS Path macOS ~/Library/Application Support/Claude/claude_desktop_config.json Windows %APPDATA%\\Claude\\claude_desktop_config.json

Add to workspace settings or MCP configuration:

{\n  \"llm-energy\": {\n    \"command\": \"npx\",\n    \"args\": [\"-y\", \"@llm-energy/mcp-server\"]\n  }\n}\n
"},{"location":"mcp-server/installation/#verify-installation","title":"Verify Installation","text":"

After configuration, restart your AI client. Ask:

\"What tools does llm-energy provide?\"

The assistant should list the available extraction tools.

"},{"location":"mcp-server/tools/","title":"Tools Reference","text":"

Available MCP tools for llm.energy.

"},{"location":"mcp-server/tools/#extract_documentation","title":"extract_documentation","text":"

Extract and parse documentation from a website with llms.txt support.

Parameters

Name Type Required Description url string Yes Documentation site URL

Example

{\n  \"url\": \"docs.anthropic.com\"\n}\n

Output

Caching

After extraction, documentation is cached for quick access via other tools.

"},{"location":"mcp-server/tools/#fetch_llms_txt","title":"fetch_llms_txt","text":"

Fetch raw llms.txt content without parsing.

Parameters

Name Type Required Description url string Yes Documentation site URL full boolean No Fetch llms-full.txt instead (default: true)

Example

{\n  \"url\": \"modelcontextprotocol.io\",\n  \"full\": true\n}\n
"},{"location":"mcp-server/tools/#get_document","title":"get_document","text":"

Retrieve a specific document from the cache.

Parameters

Name Type Required Description name string Yes Document filename

Example

{\n  \"name\": \"getting-started.md\"\n}\n
"},{"location":"mcp-server/tools/#list_documents","title":"list_documents","text":"

List all cached documents.

Output

Array of document names with metadata:

"}]}