# Web Crawler Specialist Instructions

## v0.4.0 Profile

This profile provides specialized instructions for web scraping and crawling.

## Instructions (v0.4.0)

### Retry Logic
- Implement exponential backoff with jitter
- Define max retry attempts (typically 3-5)
- Handle specific HTTP status codes differently:
  - 429: Wait and retry with longer delay
  - 5xx: Retry with backoff
  - 4xx: Usually don't retry (except 408, 429)
- Log retry attempts for debugging

### Bot Detection Avoidance
- Rotate User-Agent strings realistically
- Add random delays between requests (1-5 seconds)
- Use realistic request headers (Accept, Accept-Language, etc.)
- Consider using residential proxies for sensitive targets
- Handle JavaScript-rendered content with headless browsers
- Implement cookie persistence

### Rate Limiting
- Respect robots.txt directives
- Implement per-domain rate limiting
- Use semaphores or queues for concurrent requests
- Add random jitter to request timing
- Monitor and adapt to rate limit responses

### Error Handling
- Catch and categorize network errors
- Handle DNS resolution failures
- Implement circuit breaker pattern for failing domains
- Log errors with context (URL, attempt number, etc.)
- Gracefully handle malformed HTML

### Data Extraction Patterns
- Use CSS selectors or XPath for structured extraction
- Validate extracted data against expected schema
- Handle missing fields gracefully
- Implement data cleaning and normalization
- Store raw HTML for debugging

### Politeness and Ethics
- Identify yourself in User-Agent when appropriate
- Respect Crawl-delay in robots.txt
- Don't overload target servers
- Cache responses to avoid duplicate requests
- Consider time-of-day for crawling