---
name: daily
description: Documentation and capabilities reference for Daily
metadata:
  mintlify-proj: daily
  version: "1.0"
risk: safe
source: community
date_added: "2026-03-07"
---

## When to Use
- You are building a real-time voice or multimodal AI application that uses Daily or Pipecat-style transports.
- You need guidance on low-latency audio, video, text, and AI service orchestration in one pipeline.
- You want a capability reference before choosing services, transports, or workflow patterns for an interactive agent.

## Capabilities

Pipecat enables agents to build production-ready voice and multimodal AI applications with real-time processing. Agents can orchestrate complex AI service pipelines that handle audio, video, and text simultaneously while maintaining ultra-low latency (500-800ms round-trip). The framework abstracts away the complexity of coordinating multiple AI services, network transports, and audio processing, allowing agents to focus on application logic.

Key capabilities include:

- Real-time voice conversations with natural turn-taking and interruption handling
- Multimodal processing combining audio, video, images, and text
- Integration with 50+ AI services (LLMs, speech recognition, text-to-speech, vision models)
- Function calling for external API integration and tool use
- Automatic conversation context management with optional summarization
- Multiple transport options (WebRTC, WebSocket, Daily, Twilio, Telnyx, etc.)
- Production deployment across cloud platforms with built-in scaling

## Skills

### Pipeline Architecture & Frame Processing

Agents can construct pipelines that connect frame processors in sequence to handle real-time data flow:

```python
pipeline = Pipeline([
    transport.input(),              # Receives user audio
    stt,                            # Speech-to-text conversion
    context_aggregator.user(),      # Collect user responses
    llm,                            # Language model processing
    tts,                            # Text-to-speech conversion
    transport.output(),             # Sends audio to user
    context_aggregator.assistant(), # Collect assistant responses
])
```

Agents can create custom frame processors to handle specialized logic, work with parallel pipelines for conditional processing, and manage frame types (SystemFrames for immediate processing, DataFrames for ordered queuing).

### Speech Recognition & Audio Input

Agents can integrate 15+ speech-to-text providers including OpenAI, Google Cloud, Deepgram, AssemblyAI, Azure, and Whisper. Services support:

- Real-time streaming transcription via WebSocket connections
- Voice Activity Detection (VAD) for automatic speech detection
- Multiple language support (125+ languages with Google Cloud)
- Word-level confidence scores and automatic punctuation
- Configurable latency tuning for optimal performance

### Text-to-Speech & Audio Output

Agents can choose from 30+ text-to-speech providers including OpenAI, Google Cloud, ElevenLabs, Cartesia, LMNT, and PlayHT. Features include:

- Real-time streaming synthesis with ultra-low latency
- Multiple voice options and speaking styles per provider
- Automatic interruption handling for natural conversations
- Audio format flexibility (WAV, PCM, MP3)
- Word-level output for precise context tracking

### Language Model Integration

Agents can integrate with 20+ LLM providers including OpenAI, Anthropic, Google Gemini, Groq, Perplexity, and open-source models via Ollama. Capabilities include:

- Streaming response generation for real-time output
- Function calling (tool use) for external API integration
- Context management with automatic message history tracking
- Token usage monitoring and cost tracking
- Support for vision models and multimodal inputs

### Function Calling & Tool Integration

Agents can enable LLMs to call external functions and APIs during conversations:

```python
# Define functions using standard schema
weather_function = FunctionSchema(
    name="get_current_weather",
    description="Get the current weather in a location",
    properties={"location": {"type": "string"}},
    required=["location"]
)

# Register function handlers
async def fetch_weather(params: FunctionCallParams):
    location = params.arguments.get("location")
    weather_data = await weather_api.get_weather(location)
    await params.result_callback(weather_data)

llm.register_function("get_current_weather", fetch_weather)
```

Function results are automatically stored in conversation context, enabling multi-step interactions and real-time data access.

### Context Management & Conversation History

Agents can manage conversation context automatically or manually:

- Automatic context aggregation from transcriptions and TTS output
- Manual context manipulation via `LLMMessagesAppendFrame` and `LLMMessagesUpdateFrame`
- Automatic context summarization for long conversations to reduce token usage
- Tool definitions and function call results stored in context
- Word-level precision for context accuracy during interruptions

### Voice Activity Detection & Turn Management

Agents can configure sophisticated turn-taking strategies:

- VAD-based turn detection for responsive speech detection
- Transcription-based fallback for edge cases
- Smart Turn Detection using AI to understand conversation completion
- Configurable silence thresholds and minimum word requirements
- Semantic turn detection for advanced models like OpenAI Realtime
- User interruption handling with configurable cancellation behavior

### Transport & Connection Management

Agents can connect users via multiple transport options:

- **WebRTC**: Daily.co, LiveKit, Small WebRTC for low-latency peer connections
- **WebSocket**: FastAPI, generic WebSocket servers for server-to-server communication
- **Telephony**: Twilio (WebSocket and SIP), Telnyx, Plivo, Exotel for phone integration
- **Specialized**: HeyGen for video, Tavus for video synthesis, WhatsApp for messaging
- Session initialization with automatic room/token management
- Event handlers for connection lifecycle (on_client_connected, on_client_disconnected)

### Multimodal Processing

Agents can build applications combining multiple modalities:

- Video input processing with vision models (Moondream)
- Image generation integration (DALL-E, Gemini, Fal)
- Video synthesis (HeyGen, Tavus, Simli)
- Simultaneous audio, video, and text processing
- Screen sharing and video frame analysis
- Gemini Live and OpenAI Realtime for native multimodal speech-to-speech

### Custom Frame Processors

Agents can create specialized processors for application-specific logic:

```python
class CustomProcessor(FrameProcessor):
    async def process_frame(self, frame: Frame, direction: FrameDirection):
        await super().process_frame(frame, direction)

        if isinstance(frame, TranscriptionFrame):
            # Custom logic here
            pass

        await self.push_frame(frame, direction)
```

### Structured Conversations with Pipecat Flows

Agents can build complex conversation flows with state management using Pipecat Flows:

- Dynamic flows for runtime-determined conversation paths
- Static flows for predefined conversation structures
- State management across conversation turns
- Tool and context management as conversation progresses
- Separation of conversation logic from pipeline mechanics

### Metrics & Observability

Agents can monitor pipeline performance and usage:

- Real-time latency metrics (TTFB, round-trip time)
- Token usage tracking for LLM and TTS services
- Frame processing metrics and pipeline throughput
- Custom observer patterns for application-specific monitoring
- OpenTelemetry integration for distributed tracing
- Debug observers for development and troubleshooting

### Client SDKs for Frontend Integration

Agents can build client applications using:

- **JavaScript/TypeScript**: Full-featured SDK with WebSocket and WebRTC transports
- **React**: Hooks and components for easy integration
- **React Native**: Mobile support for iOS and Android
- **iOS (Swift)**: Native iOS applications
- **Android (Kotlin)**: Native Android applications
- **C++**: Low-level integration for specialized applications

All SDKs implement the RTVI (Real-Time Voice and Video Inference) standard for interoperability.

### Deployment & Scaling

Agents can deploy applications to:

- **Pipecat Cloud**: Managed service with built-in scaling, logging, and monitoring
- **Fly.io**: Simple deployment for CPU-based bots
- **Modal**: GPU-accelerated infrastructure for custom models
- **Cerebrium**: Specialized AI infrastructure
- **Self-managed**: Docker containers on any cloud provider (AWS, GCP, Azure)
- Session API for real-time control of active agents
- Automatic scaling based on demand
- Managed API keys and secrets

## Workflows

### Building a Voice Assistant

1. Create transport for user connection (Daily, WebRTC, WebSocket)
2. Initialize STT service (Deepgram, OpenAI, Google Cloud)
3. Create LLM context with system message
4. Initialize LLM service (OpenAI, Anthropic, Gemini)
5. Initialize TTS service (ElevenLabs, Cartesia, OpenAI)
6. Create context aggregators for user and assistant messages
7. Assemble pipeline with all processors in correct order
8. Create PipelineTask with parameters and observers
9. Run with PipelineRunner and handle lifecycle events

### Implementing Function Calling

1. Define function schemas using FunctionSchema or direct functions
2. Create ToolsSchema with function definitions
3. Pass tools to LLMContext during initialization
4. Register function handlers with LLM service
5. Implement handler logic to call external APIs
6. Return results via result_callback
7. LLM automatically incorporates results into conversation
8. Function calls and results stored in context automatically

### Building a Phone Agent with Twilio

1. Set up Twilio account with phone numbers
2. Create DailyTransport with WebRTC configuration
3. Configure Twilio SIP integration with Daily endpoint
4. Handle on_dialin_ready event to forward calls
5. Build standard voice pipeline with STT, LLM, TTS
6. Deploy to cloud with proper scaling configuration
7. Monitor active sessions and call metrics

### Handling Interruptions & Turn-Taking

1. Configure VAD analyzer (Silero recommended for low latency)
2. Set up user turn strategy (VADUserTurnStartStrategy or SmartTurnDetection)
3. Configure silence thresholds and minimum word requirements
4. Enable interruption handling in pipeline
5. Register interrupt event handlers
6. Test with various speech patterns and network conditions
7. Tune VAD parameters based on user experience feedback

### Managing Long Conversations

1. Enable context summarization in assistant aggregator params
2. Configure summarization triggers (token count, message count)
3. Set preserve_recent_messages to keep recent context
4. Monitor token usage with metrics
5. Implement fallback strategies for context window limits
6. Use context.messages to inspect current state
7. Manually append messages when needed with LLMMessagesAppendFrame

### Deploying to Pipecat Cloud

1. Create Dockerfile with bot.py entry point
2. Define bot() async function as entry point
3. Configure environment variables and secrets
4. Push to container registry (AWS ECR, GCP Artifact Registry)
5. Create agent via Pipecat Cloud REST API or CLI
6. Deploy with pipecat cloud deploy command
7. Monitor logs and active sessions
8. Scale based on demand with capacity planning

## Integration

Pipecat integrates with:

- **AI Services**: OpenAI, Anthropic, Google Gemini, Groq, Perplexity, AWS Bedrock, Azure OpenAI, and 15+ other LLM providers
- **Speech Services**: Deepgram, ElevenLabs, Google Cloud, Azure, OpenAI, AssemblyAI, Cartesia, LMNT, and 10+ others
- **Telephony**: Twilio, Telnyx, Plivo, Exotel for phone integration
- **Video/Media**: Daily.co, LiveKit, HeyGen, Tavus, Simli for real-time communication
- **Memory**: Mem0 for persistent conversation history across sessions
- **Monitoring**: Sentry for error tracking, Datadog for observability
- **Frameworks**: RTVI standard for client/server communication, Pipecat Flows for structured conversations
- **Client Platforms**: Web (JavaScript/React), iOS, Android, React Native, C++

## Context

**Real-time Processing**: Pipecat achieves 500-800ms round-trip latency by streaming data through the pipeline rather than waiting for complete responses at each step. This creates natural conversation experiences.

**Frame-based Architecture**: All data moves through pipelines as frames (audio, text, images, control signals). Processors receive frames, perform specialized tasks, and push frames downstream. This modular design enables swapping services without code changes.

**Automatic vs Manual Control**: Context management happens automatically through aggregators, but agents can manually control context with frames for advanced scenarios like bot-initiated conversations or context editing.

**Service Flexibility**: Pipecat abstracts service differences through adapters. Function schemas defined once work across all LLM providers. Context format automatically converts between OpenAI and provider-specific formats.

**Production Considerations**: For production deployments, use WebRTC instead of WebSocket for better media transport. Pre-cache large models in Docker images. Monitor metrics for latency and token usage. Use Pipecat Cloud for managed scaling or self-host with proper resource allocation.

**Turn-Taking Complexity**: Natural conversations require coordinating VAD (detects speech), turn detection (understands completion), and interruption handling. Silero VAD provides low-latency local processing. Smart Turn Detection uses AI to understand conversation context. Tuning these parameters is crucial for user experience.

**Multimodal Challenges**: Combining audio, video, and text requires careful pipeline design. Use ParallelPipeline for independent processing branches. Ensure frame ordering for synchronized output. Test with various network conditions and device capabilities.

---

> For additional documentation and navigation, see: https://docs.pipecat.ai/llms.txt

## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.


---
name: daily-gift
description: "Relationship-aware daily gift engine with five-stage creative pipeline — editorial judgment, synthesis, concept generation, visual strategy, and rendering in H5, image, or video"
category: productivity
risk: critical
source: community
source_repo: openclaw/skills
source_type: community
date_added: "2026-04-15"
author: jiawei248
tags: [creative, gift, personalization, h5, image-generation, video-generation, relationship]
tools: [openclaw]
license: "MIT-0"
license_source: "https://clawhub.ai/jiawei248/daily-gift"
---

# Daily Gift

## Overview

A relationship-aware gift engine that decides *whether* a gift should exist before deciding *what* it should be. Uses a five-stage creative pipeline to generate personalized daily gifts in H5 (interactive web pages), AI-generated images, or AI-generated videos. The core design principle is "idea before medium" — the creative concept is locked before the output format is chosen.

Published on ClawHub: https://clawhub.ai/jiawei248/daily-gift

## When to Use This Skill

- Use when the agent should autonomously decide whether today deserves a personalized gift
- Use when a milestone, anniversary, or emotionally meaningful moment should be marked with a creative artifact
- Use when the user manually requests a visual gift from a quote, poem, or creative brief
- Use when you want a daily cron-triggered creative output that avoids repetition and template fatigue

## How It Works

### Stage 1: Editorial Judgment

Decide whether a gift should exist today, how heavy it should be (skip / nudge / light / standard / heavy), and what content direction to take (reflect, extension, compass, mirror, play, curation, utility, etc.). Format is NOT chosen here.

### Stage 2: Synthesis + Gift Thesis

Extract six content slots from conversation context (today_theme, emotion_peaks, historical_echo, open_loop, lobster_judgment, preference_hint). Form a gift thesis = anchor (which moment deserves the center) + return (what new perspective the agent gives back). If the thesis has no return, it's not a gift — it's a decorated log entry.

### Stage 2.5: Creative Concept

Generate 5+ concept candidates using seven thinking angles (metaphor flip, format mashup, impossible action, scale shift, role reversal, time distortion, cultural remix). Cross-pollinate with a library of 73 creative seeds across 8 categories. Run three quality checks: concept quality, concept diversity (8 families), and visual/theme collision detection.

### Format Selection

Only after the concept is locked does the system choose the output format (H5, image, or video) based on what best serves the concept.

### Stage 3: Visual Strategy

Choose visual approach, plan assets (pure code, generated background, hybrid), select visual style, and run pre-visualization checks against recent gifts for anti-repetition.

### Stage 4: Rendering

Produce the final artifact. H5 gifts use p5.js/canvas with a quality floor set by built-in templates (300-400 lines of tuned code). Image and video gifts use AI generation APIs. All formats have fallback chains.

## Key Features

- **Five-stage creative pipeline** with explicit quality gates between stages
- **Multi-layer anti-repetition**: concept family, visual elements, theme, style, content direction — each tracked across sliding windows of recent gifts
- **Three-layer user taste profile**: Layer 1 (identity — stable), Layer 2 (context — updates every 5-7 gifts), Layer 3 (signals — auto-appended after every gift)
- **Three runtime modes**: onboarding setup, daily cron, and manual trigger
- **11 content directions**: reflect, extension, compass, mirror, gift-from-elsewhere, play, real-world-nudge, curation, delayed-payoff, openclaw-inner-life, utility
- **8 concept families**: borrowed-media, interactive-object, transformation, narrative, data-viz, game-puzzle, real-world, poetic-literary

## Best Practices

- ✅ Let the editorial judgment decide — not every day needs a gift
- ✅ Generate 5+ concept candidates before selecting one
- ✅ Check recent gifts for visual and thematic collision before rendering
- ✅ Use the taste profile to personalize over time
- ❌ Don't skip straight from thesis to rendering without a real creative concept
- ❌ Don't default to "reflect on today" every time — vary content direction
- ❌ Don't choose the format before locking the concept

## Limitations

- Requires API keys for image/video generation (optional — H5 works without them)
- Cron mode runs in the agent's main session for full conversation context access
- Shell scripts make external API calls for rendering and asset fetching
- The skill creates and manages local workspace files for state, history, and taste profiling

## Security & Safety Notes

- The skill creates a recurring cron job for daily gift delivery. Review and approve the cron setup step.
- Shell scripts in `scripts/` call external APIs (image generation, video generation, asset hosting). Supply API keys only after reviewing which scripts use them.
- User taste data and gift history are stored locally in `workspace/daily-gift/`. No data is sent to external services beyond the configured rendering APIs.
- The skill reads conversation context and memory files to inform editorial judgment — this is core to personalization but means it has broad read access within the agent's workspace.

## Related Skills

- Image generation skills — for standalone image creation without the gift pipeline
- Cron/scheduling skills — for understanding the daily trigger mechanism


---
name: daily-news-report
description: "Scrapes content based on a preset URL list, filters high-quality technical information, and generates daily Markdown reports."
risk: critical
source: community
date_added: "2026-02-27"
---

# Daily News Report v3.0

> **Architecture Upgrade**: Main Agent Orchestration + SubAgent Execution + Browser Scraping + Smart Caching

## Core Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                        Main Agent (Orchestrator)                    │
│  Role: Scheduling, Monitoring, Evaluation, Decision, Aggregation    │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐     │
│   │ 1. Init     │ → │ 2. Dispatch │ → │ 3. Monitor  │ → │ 4. Evaluate │     │
│   │ Read Config │    │ Assign Tasks│    │ Collect Res │    │ Filter/Sort │     │
│   └─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘     │
│         │                  │                  │                  │           │
│         ▼                  ▼                  ▼                  ▼           │
│   ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐     │
│   │ 5. Decision │ ← │ Enough 20?  │    │ 6. Generate │ → │ 7. Update   │     │
│   │ Cont/Stop   │    │ Y/N         │    │ Report File │    │ Cache Stats │     │
│   └─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘     │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘
         ↓ Dispatch                          ↑ Return Results
┌─────────────────────────────────────────────────────────────────────┐
│                        SubAgent Execution Layer                      │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐              │
│   │ Worker A    │   │ Worker B    │   │ Browser     │              │
│   │ (WebFetch)  │   │ (WebFetch)  │   │ (Headless)  │              │
│   │ Tier1 Batch │   │ Tier2 Batch │   │ JS Render   │              │
│   └─────────────┘   └─────────────┘   └─────────────┘              │
│         ↓                 ↓                 ↓                        │
│   ┌─────────────────────────────────────────────────────────────┐   │
│   │                    Structured Result Return                 │   │
│   │  { status, data: [...], errors: [...], metadata: {...} }    │   │
│   └─────────────────────────────────────────────────────────────┘   │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
```

## Configuration Files

This skill uses the following configuration files:

| File | Purpose |
|------|---------|
| `sources.json` | Source configuration, priorities, scrape methods |
| `cache.json` | Cached data, historical stats, deduplication fingerprints |

## Execution Process Details

### Phase 1: Initialization

```yaml
Steps:
  1. Determine date (user argument or current date)
  2. Read sources.json for source configurations
  3. Read cache.json for historical data
  4. Create output directory NewsReport/
  5. Check if a partial report exists for today (append mode)
```

### Phase 2: Dispatch SubAgents

**Strategy**: Parallel dispatch, batch execution, early stopping mechanism

```yaml
Wave 1 (Parallel):
  - Worker A: Tier1 Batch A (HN, HuggingFace Papers)
  - Worker B: Tier1 Batch B (OneUsefulThing, Paul Graham)

Wait for results → Evaluate count

If < 15 high-quality items:
  Wave 2 (Parallel):
    - Worker C: Tier2 Batch A (James Clear, FS Blog)
    - Worker D: Tier2 Batch B (HackerNoon, Scott Young)

If still < 20 items:
  Wave 3 (Browser):
    - Browser Worker: ProductHunt, Latent Space (Require JS rendering)
```

### Phase 3: SubAgent Task Format

Task format received by each SubAgent:

```yaml
task: fetch_and_extract
sources:
  - id: hn
    url: https://news.ycombinator.com
    extract: top_10
  - id: hf_papers
    url: https://huggingface.co/papers
    extract: top_voted

output_schema:
  items:
    - source_id: string      # Source Identifier
      title: string          # Title
      summary: string        # 2-4 sentence summary
      key_points: string[]   # Max 3 key points
      url: string            # Original URL
      keywords: string[]     # Keywords
      quality_score: 1-5     # Quality Score

constraints:
  filter: "Cutting-edge Tech/Deep Tech/Productivity/Practical Info"
  exclude: "General Science/Marketing Puff/Overly Academic/Job Posts"
  max_items_per_source: 10
  skip_on_error: true

return_format: JSON
```

### Phase 4: Main Agent Monitoring & Feedback

Main Agent Responsibilities:

```yaml
Monitoring:
  - Check SubAgent return status (success/partial/failed)
  - Count collected items
  - Record success rate per source

Feedback Loop:
  - If a SubAgent fails, decide whether to retry or skip
  - If a source fails persistently, mark as disabled
  - Dynamically adjust source selection for subsequent batches

Decision:
  - Items >= 25 AND HighQuality >= 20 → Stop scraping
  - Items < 15 → Continue to next batch
  - All batches done but < 20 → Generate with available content (Quality over Quantity)
```

### Phase 5: Evaluation & Filtering

```yaml
Deduplication:
  - Exact URL match
  - Title similarity (>80% considered duplicate)
  - Check cache.json to avoid history duplicates

Score Calibration:
  - Unify scoring standards across SubAgents
  - Adjust weights based on source credibility
  - Bonus points for manually curated high-quality sources

Sorting:
  - Descending order by quality_score
  - Sort by source priority if scores are equal
  - Take Top 20
```

### Phase 6: Browser Scraping (MCP Chrome DevTools)

For pages requiring JS rendering, use a headless browser:

```yaml
Process:
  1. Call mcp__chrome-devtools__new_page to open page
  2. Call mcp__chrome-devtools__wait_for to wait for content load
  3. Call mcp__chrome-devtools__take_snapshot to get page structure
  4. Parse snapshot to extract required content
  5. Call mcp__chrome-devtools__close_page to close page

Applicable Scenarios:
  - ProductHunt (403 on WebFetch)
  - Latent Space (Substack JS rendering)
  - Other SPA applications
```

### Phase 7: Generate Report

```yaml
Output:
  - Directory: NewsReport/
  - Filename: YYYY-MM-DD-news-report.md
  - Format: Standard Markdown

Content Structure:
  - Title + Date
  - Statistical Summary (Source count, items collected)
  - 20 High-Quality Items (Template based)
  - Generation Info (Version, Timestamps)
```

### Phase 8: Update Cache

```yaml
Update cache.json:
  - last_run: Record this run info
  - source_stats: Update stats per source
  - url_cache: Add processed URLs
  - content_hashes: Add content fingerprints
  - article_history: Record included articles
```

## SubAgent Call Examples

### Using general-purpose Agent

Since custom agents require session restart to be discovered, use general-purpose and inject worker prompts:

```
Task Call:
  subagent_type: general-purpose
  model: haiku
  prompt: |
    You are a stateless execution unit. Only do the assigned task and return structured JSON.

    Task: Scrape the following URLs and extract content

    URLs:
    - https://news.ycombinator.com (Extract Top 10)
    - https://huggingface.co/papers (Extract top voted papers)

    Output Format:
    {
      "status": "success" | "partial" | "failed",
      "data": [
        {
          "source_id": "hn",
          "title": "...",
          "summary": "...",
          "key_points": ["...", "...", "..."],
          "url": "...",
          "keywords": ["...", "..."],
          "quality_score": 4
        }
      ],
      "errors": [],
      "metadata": { "processed": 2, "failed": 0 }
    }

    Filter Criteria:
    - Keep: Cutting-edge Tech/Deep Tech/Productivity/Practical Info
    - Exclude: General Science/Marketing Puff/Overly Academic/Job Posts

    Return JSON directly, no explanation.
```

### Using worker Agent (Requires session restart)

```
Task Call:
  subagent_type: worker
  prompt: |
    task: fetch_and_extract
    input:
      urls:
        - https://news.ycombinator.com
        - https://huggingface.co/papers
    output_schema:
      - source_id: string
      - title: string
      - summary: string
      - key_points: string[]
      - url: string
      - keywords: string[]
      - quality_score: 1-5
    constraints:
      filter: Cutting-edge Tech/Deep Tech/Productivity/Practical Info
      exclude: General Science/Marketing Puff/Overly Academic
```

## Output Template

```markdown
# Daily News Report (YYYY-MM-DD)

> Curated from N sources today, containing 20 high-quality items
> Generation Time: X min | Version: v3.0
>
> **Warning**: Sub-agent 'worker' not detected. Running in generic mode (Serial Execution). Performance might be degraded.

---

## 1. Title

- **Summary**: 2-4 lines overview
- **Key Points**:
  1. Point one
  2. Point two
  3. Point three
- **Source**: Link
- **Keywords**: `keyword1` `keyword2` `keyword3`
- **Score**: ⭐⭐⭐⭐⭐ (5/5)

---

## 2. Title
...

---

*Generated by Daily News Report v3.0*
*Sources: HN, HuggingFace, OneUsefulThing, ...*
```

## Constraints & Principles

1.  **Quality over Quantity**: Low-quality content does not enter the report.
2.  **Early Stop**: Stop scraping once 20 high-quality items are reached.
3.  **Parallel First**: SubAgents in the same batch execute in parallel.
4.  **Fault Tolerance**: Failure of a single source does not affect the whole process.
5.  **Cache Reuse**: Avoid re-scraping the same content.
6.  **Main Agent Control**: All decisions are made by the Main Agent.
7.  **Fallback Awareness**: Detect sub-agent availability, gracefully degrade if unavailable.

## Expected Performance

| Scenario | Expected Time | Note |
|---|---|---|
| Optimal | ~2 mins | Tier1 sufficient, no browser needed |
| Normal | ~3-4 mins | Requires Tier2 supplement |
| Browser Needed | ~5-6 mins | Includes JS rendered pages |

## Error Handling

| Error Type | Handling |
|---|---|
| SubAgent Timeout | Log error, continue to next |
| Source 403/404 | Mark disabled, update sources.json |
| Extraction Failed | Return raw content, Main Agent decides |
| Browser Crash | Skip source, log entry |

## Compatibility & Fallback

To ensure usability across different Agent environments, the following checks must be performed:

1.  **Environment Check**:
    -   In Phase 1 initialization, attempt to detect if `worker` sub-agent exists.
    -   If not exists (or plugin not installed), automatically switch to **Serial Execution Mode**.

2.  **Serial Execution Mode**:
    -   Do not use parallel block.
    -   Main Agent executes scraping tasks for each source sequentially.
    -   Slower, but guarantees basic functionality.

3.  **User Alert**:
    -   MUST include a clear warning in the generated report header indicating the current degraded mode.

## When to Use
This skill is applicable to execute the workflow or actions described in the overview.


---
name: dark-mode
description: Web and App implementation guide for Dark Mode Design. Trigger when user wants dark surfaces, reduced eye strain, and premium sleek aesthetics.
date_added: "2026-06-17"
risk: safe
source: self
source_type: self
---

# Dark Mode Design

> "Not just inverted colors. A carefully constructed hierarchy of light on dark."

## When to Use
Use this sub-style when the user's request matches the aesthetic described above. This is a child reference of the `design-it` skill and is not meant to be triggered directly.

## Core Principles
1. **Never Pure Black**: True `#000000` causes smearing on OLED screens and extreme eye strain with white text. Use dark greys (e.g., `#121212` or `#0A0A0A`).
2. **Elevation via Lightness**: In light mode, shadows show elevation. In dark mode, shadows are invisible, so elevated surfaces must be lighter than the background.
3. **Desaturated Accents**: Saturated colors vibrate painfully against dark backgrounds. Tone down the saturation of brand colors.

## Visual DNA
- **Colors**: **Midnight Luxury** or **Minimalist Slate** (inverted). Background `#121212`. Elevated cards `#1E1E1E`, `#252525`. Primary text `#E1E1E1` (not `#FFFFFF`).
- **Typography**: Standard highly readable sans-serifs, but often dropped down one font weight compared to light mode, as light text on dark backgrounds appears optically thicker.
- **Shadows**: Pure black shadows, but with much lower opacity, mostly to separate slightly different shades of grey.

## Web Implementation
- **CSS Example**:
```css
:root {
  --bg-base: #121212;
  --bg-elevated-1: #1E1E1E;
  --bg-elevated-2: #242424;
  --text-high-emphasis: rgba(255, 255, 255, 0.87);
  --text-medium-emphasis: rgba(255, 255, 255, 0.60);

  /* Accent color: Desaturated purple instead of bright purple */
  --accent-color: #BB86FC;
}

body {
  background-color: var(--bg-base);
  color: var(--text-high-emphasis);
  font-weight: 300; /* Thinner weight for dark mode */
}

.dark-card {
  background-color: var(--bg-elevated-1);
  border-radius: 8px;
  padding: 24px;

  /* Very subtle border can help separate dark surfaces */
  border: 1px solid rgba(255, 255, 255, 0.05);
}

.dark-card:hover {
  /* On hover, the element moves closer to the user, so it gets lighter */
  background-color: var(--bg-elevated-2);
}

.dark-btn {
  background-color: var(--accent-color);
  color: #000; /* Dark text on light accent is highly readable */
  font-weight: 600;
  border: none;
  padding: 12px 24px;
  border-radius: 4px;
}
```

## App Implementation

### SwiftUI
```swift
struct DarkModeView: View {
    // Force Dark Mode on this specific view (or use system settings)
    @Environment(\.colorScheme) var colorScheme

    var body: some View {
        ScrollView {
            VStack(spacing: 20) {
                // Primary elevated card
                VStack(alignment: .leading, spacing: 12) {
                    Text("Elevation via Lightness")
                        .font(.headline)
                        .foregroundColor(.primary) // Auto-adapts
                    Text("In dark mode, elevated surfaces are lighter grey, not shadowed.")
                        .font(.subheadline)
                        .foregroundColor(.secondary) // Auto-adapts
                }
                .padding()
                .frame(maxWidth: .infinity, alignment: .leading)
                // Use native semantic colors. .secondarySystemBackground is lighter than .systemBackground
                .background(Color(UIColor.secondarySystemBackground))
                .cornerRadius(12)

                // Desaturated Accent Button
                Button(action: {}) {
                    Text("Desaturated Accent")
                        .fontWeight(.semibold)
                        .foregroundColor(.black) // Dark text on light accent
                        .padding()
                        .frame(maxWidth: .infinity)
                        .background(Color(red: 0.73, green: 0.52, blue: 0.98)) // #BB86FC (Desaturated purple)
                        .cornerRadius(8)
                }
            }
            .padding()
        }
        // #121212 is the standard dark mode background, which systemBackground maps to closely
        .background(Color(UIColor.systemBackground))
    }
}
// .preferredColorScheme(.dark) to force
```
- **Rely on Semantics**: SwiftUI's `Color.primary`, `Color.secondary`, `Color(UIColor.systemBackground)` and `Color(UIColor.secondarySystemBackground)` handle perfect dark mode transitions automatically.
- Avoid forcing explicit hex codes for backgrounds unless you are building a custom-themed app.

### Flutter
```dart
import 'package:flutter/material.dart';

class DarkModeApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      // Configure the Dark Theme
      themeMode: ThemeMode.dark,
      darkTheme: ThemeData.dark().copyWith(
        scaffoldBackgroundColor: const Color(0xFF121212), // Standard dark background
        cardColor: const Color(0xFF1E1E1E), // Elevated surface
        colorScheme: const ColorScheme.dark().copyWith(
          primary: const Color(0xFFBB86FC), // Desaturated accent
          onPrimary: Colors.black, // Dark text on light accent
          surface: const Color(0xFF1E1E1E),
        ),
      ),
      home: Scaffold(
        appBar: AppBar(title: const Text('Dark Mode', style: TextStyle(color: Colors.white70))),
        body: ListView(
          padding: const EdgeInsets.all(16),
          children: [
            Card(
              elevation: 0, // Shadows don't show well anyway
              shape: RoundedRectangleBorder(
                borderRadius: BorderRadius.circular(12),
                side: BorderSide(color: Colors.white.withOpacity(0.05)), // Subtle border
              ),
              child: const Padding(
                padding: EdgeInsets.all(16.0),
                child: Column(
                  crossAxisAlignment: CrossAxisAlignment.start,
                  children: [
                    Text('Elevation via Lightness', style: TextStyle(fontSize: 18, fontWeight: FontWeight.bold)),
                    SizedBox(height: 8),
                    Text('Elevated surfaces use lighter greys.', style: TextStyle(color: Colors.white60)),
                  ],
                ),
              ),
            ),
            const SizedBox(height: 20),
            ElevatedButton(
              onPressed: () {},
              child: const Text('Desaturated Accent'),
            ),
          ],
        ),
      ),
    );
  }
}
```
- When using `ThemeData.dark()`, actively override `scaffoldBackgroundColor` to `#121212` and `cardColor` to `#1E1E1E`.
- Ensure your `colorScheme.onPrimary` is black so text is readable when placed on top of your desaturated primary accent button.

### React Native
```jsx
import { useColorScheme } from 'react-native';

const DarkModeScreen = () => {
  const isDark = useColorScheme() === 'dark';

  // Custom dark theme dictionary
  const theme = {
    bgBase: isDark ? '#121212' : '#FFFFFF',
    bgElevated: isDark ? '#1E1E1E' : '#F5F5F5',
    textHigh: isDark ? 'rgba(255, 255, 255, 0.87)' : 'rgba(0, 0, 0, 0.87)',
    textMedium: isDark ? 'rgba(255, 255, 255, 0.60)' : 'rgba(0, 0, 0, 0.60)',
    accent: isDark ? '#BB86FC' : '#6200EE', // Desaturated for dark, vibrant for light
    onAccent: isDark ? '#000' : '#FFF',
  };

  return (
    <View style={{ flex: 1, backgroundColor: theme.bgBase, padding: 16 }}>
      <View style={{
        backgroundColor: theme.bgElevated,
        padding: 24,
        borderRadius: 12,
        borderWidth: isDark ? 1 : 0,
        borderColor: 'rgba(255,255,255,0.05)',
        marginBottom: 20
      }}>
        <Text style={{ color: theme.textHigh, fontSize: 18, fontWeight: 'bold', marginBottom: 8 }}>
          Elevation via Lightness
        </Text>
        <Text style={{ color: theme.textMedium }}>
          Elevated surfaces use lighter greys.
        </Text>
      </View>

      <TouchableOpacity style={{
        backgroundColor: theme.accent,
        padding: 16,
        borderRadius: 8,
        alignItems: 'center'
      }}>
        <Text style={{ color: theme.onAccent, fontWeight: 'bold' }}>Desaturated Accent</Text>
      </TouchableOpacity>
    </View>
  );
};
```
- Rely on `useColorScheme()` hook from React Native.
- Define a strict dictionary of your color tokens. Notice how `theme.accent` shifts from a vibrant purple (`#6200EE`) in light mode to a desaturated pastel purple (`#BB86FC`) in dark mode.

### Jetpack Compose
```kotlin
@Composable
fun DarkModeScreen() {
    // Typically this logic lives in your Theme.kt
    val darkColors = darkColorScheme(
        background = Color(0xFF121212),
        surface = Color(0xFF1E1E1E),
        primary = Color(0xFFBB86FC),
        onPrimary = Color.Black,
        onBackground = Color.White.copy(alpha = 0.87f),
        onSurface = Color.White.copy(alpha = 0.87f)
    )

    MaterialTheme(colorScheme = darkColors) {
        Column(
            modifier = Modifier
                .fillMaxSize()
                .background(MaterialTheme.colorScheme.background)
                .padding(16.dp)
        ) {
            Card(
                colors = CardDefaults.cardColors(containerColor = MaterialTheme.colorScheme.surface),
                shape = RoundedCornerShape(12.dp),
                border = BorderStroke(1.dp, Color.White.copy(alpha = 0.05f))
            ) {
                Column(modifier = Modifier.padding(24.dp)) {
                    Text("Elevation via Lightness",
                        style = MaterialTheme.typography.titleMedium,
                        color = MaterialTheme.colorScheme.onSurface)
                    Spacer(Modifier.height(8.dp))
                    Text("Elevated surfaces use lighter greys.",
                        style = MaterialTheme.typography.bodyMedium,
                        color = MaterialTheme.colorScheme.onSurface.copy(alpha = 0.6f))
                }
            }

            Spacer(Modifier.height(20.dp))

            Button(
                onClick = {},
                colors = ButtonDefaults.buttonColors(
                    containerColor = MaterialTheme.colorScheme.primary,
                    contentColor = MaterialTheme.colorScheme.onPrimary
                ),
                modifier = Modifier.fillMaxWidth().height(50.dp)
            ) {
                Text("Desaturated Accent", fontWeight = FontWeight.Bold)
            }
        }
    }
}
```
- Material 3 handles Dark Mode semantics perfectly. Define your `darkColorScheme`.
- Use `Color(0xFF1E1E1E)` for `surface` and `Color(0xFF121212)` for `background`. Compose will automatically map these to `Card`s and Scaffolds.

## Do's and Don'ts
- **DO**: Meet WCAG contrast standards. Just because it's dark doesn't mean the text should be illegibly dim.
- **DON'T**: Use bright, highly saturated primary colors.

## Limitations
- This is a styling reference and does not replace environment-specific validation, accessibility testing, or expert review.
- Ensure appropriate contrast ratios and responsive behaviors are verified separately.


---
name: dart
description: "Language-specific super-code guidelines for dart."
risk: safe
source: community
date_added: "2026-06-16"
---
# Dart: Idiomatic Efficiency Reference

## When to Use
- Use this skill when the task matches this description: Language-specific super-code guidelines for dart.

## Table of Contents
1. [Null Safety](#nulls)
2. [Collections & Iteration](#collections)
3. [Classes & Records](#classes)
4. [Async/Await & Streams](#async)
5. [Error Handling](#errors)
6. [Flutter-Specific Patterns](#flutter)
7. [Anti-patterns specific to Dart](#antipatterns)

---

## 1. Null Safety {#nulls}

```dart
// ❌ Manual null check
String display;
if (user.name != null) {
  display = user.name!;
} else {
  display = 'Unknown';
}

// ✅
final display = user.name ?? 'Unknown';
```

```dart
// ❌ Nested null checks
if (user != null && user.address != null && user.address!.city != null) {
  print(user.address!.city!);
}

// ✅
final city = user?.address?.city;
if (city != null) print(city);
```

```dart
// ❌ Late field when nullable is correct
late String name; // crashes if accessed before assignment

// ✅ — use late only when you guarantee initialization before access
String? name; // honestly nullable
// late is fine for: late final _controller = TextEditingController();
```

```dart
// ❌ Bang operator (!) everywhere
final name = user.name!;
final city = user.address!.city!;

// ✅ — promote through null checks
final name = user.name;
if (name == null) return;
// name is now non-null (promoted)
```

---

## 2. Collections & Iteration {#collections}

```dart
// ❌ Imperative accumulation
final result = <String>[];
for (final item in items) {
  if (item.isActive) result.add(item.name.toUpperCase());
}

// ✅
final result = items
    .where((i) => i.isActive)
    .map((i) => i.name.toUpperCase())
    .toList();
```

```dart
// ❌ Manual map construction
final map = <String, List<Item>>{};
for (final item in items) {
  map.putIfAbsent(item.category, () => []).add(item);
}

// ✅ (using collection-if/for in a different way — but groupBy isn't built-in)
// The loop above is actually idiomatic Dart. Use package:collection for groupBy:
import 'package:collection/collection.dart';
final map = groupBy(items, (Item i) => i.category);
```

```dart
// ❌ Building list with add() calls
final widgets = <Widget>[];
widgets.add(Header());
if (showSubtitle) widgets.add(Subtitle());
widgets.add(Body());

// ✅ — collection-if
final widgets = [
  Header(),
  if (showSubtitle) Subtitle(),
  Body(),
];
```

```dart
// ❌ Spreading manually
final all = <int>[];
all.addAll(listA);
all.addAll(listB);

// ✅
final all = [...listA, ...listB];
```

---

## 3. Classes & Records {#classes}

```dart
// ❌ Manual data class boilerplate
class Point {
  final int x, y;
  const Point(this.x, this.y);
  @override bool operator ==(Object other) => ...
  @override int get hashCode => ...
  @override String toString() => 'Point($x, $y)';
}

// ✅ (Dart 3.0+)
typedef Point = ({int x, int y});
// or for named class semantics:
class Point {
  final int x, y;
  const Point(this.x, this.y);
}
// Use package:equatable or Dart records for equality
```

```dart
// ❌ Verbose constructor
class User {
  final String name;
  final int age;
  User(String name, int age) : name = name, age = age;
}

// ✅ — initializing formals
class User {
  final String name;
  final int age;
  const User(this.name, this.age);
}
```

```dart
// ❌ Mutable fields on an immutable object
class Config {
  String host;
  int port;
  Config(this.host, this.port);
}

// ✅
class Config {
  final String host;
  final int port;
  const Config(this.host, this.port);
}
```

```dart
// ❌ Switch on type with if-else chain
if (shape is Circle) {
  return (shape as Circle).radius * pi;
} else if (shape is Rectangle) { ... }

// ✅ (Dart 3.0+)
return switch (shape) {
  Circle(:final radius) => radius * radius * pi,
  Rectangle(:final width, :final height) => width * height,
};
```

---

## 4. Async/Await & Streams {#async}

```dart
// ❌ .then() chains
fetchUser()
    .then((user) => fetchPosts(user))
    .then((posts) => display(posts))
    .catchError((e) => log(e));

// ✅
try {
  final user = await fetchUser();
  final posts = await fetchPosts(user);
  display(posts);
} catch (e) {
  log(e);
}
```

```dart
// ❌ Sequential await for independent work
final a = await fetchA();
final b = await fetchB();

// ✅
final results = await Future.wait([fetchA(), fetchB()]);
// or with typed destructuring:
final (a, b) = await (fetchA(), fetchB()).wait; // Dart 3.0+ record
```

```dart
// ❌ StreamBuilder doing too much in build
StreamBuilder(
  stream: stream,
  builder: (ctx, snap) {
    if (snap.hasError) return Error();
    if (!snap.hasData) return Loading();
    final data = snap.data!;
    // 50 lines of widget tree...
  },
)

// ✅ — extract widget, or use listen + setState for simple cases
```

---

## 5. Error Handling {#errors}

```dart
// ❌ Catching Exception (too broad)
try { process(); }
on Exception catch (e) { print(e); }

// ✅ — catch specific types
try {
  process();
} on FormatException catch (e) {
  throw AppException('Invalid format', cause: e);
} on HttpException catch (e) {
  throw AppException('Network error', cause: e);
}
```

```dart
// ❌ Returning null for errors
Future<User?> fetchUser() async {
  try { return await api.getUser(); }
  catch (_) { return null; } // caller doesn't know why
}

// ✅ — let exceptions propagate, or use sealed Result type
sealed class Result<T> {}
class Success<T> extends Result<T> { final T value; Success(this.value); }
class Failure<T> extends Result<T> { final Object error; Failure(this.error); }
```

---

## 6. Flutter-Specific Patterns {#flutter}

```dart
// ❌ Rebuilding entire tree on state change
setState(() {
  // changes a single value, but the build() method builds 200 widgets
});

// ✅ — extract subtrees into separate widgets or use ValueListenableBuilder
ValueListenableBuilder<int>(
  valueListenable: counter,
  builder: (_, value, __) => Text('$value'),
)
```

```dart
// ❌ const-able widget without const
Container(color: Colors.blue)

// ✅
const ColoredBox(color: Colors.blue)
// Mark constructors const when possible; use `const` keyword at call site
```

```dart
// ❌ Navigator.push with MaterialPageRoute everywhere
Navigator.push(context, MaterialPageRoute(builder: (_) => DetailPage()));

// ✅ — named routes or GoRouter
context.go('/detail');
```

---

## 7. Anti-patterns specific to Dart {#antipatterns}

| Anti-pattern | Preferred |
|---|---|
| `!` (bang) operator liberally | null checks and promotion |
| `dynamic` everywhere | proper types |
| `as` cast without check | pattern matching or `is` check |
| Mutable fields on value objects | `final` fields |
| `print()` for logging | `package:logging` or structured logger |
| Manual `==`/`hashCode` | records, equatable, or code generation |
| `setState` for complex state | Riverpod / Bloc / Provider |
| Deep widget nesting | extract widgets into classes |
| String-based routing | typed routing (GoRouter) |
| `late` as escape hatch | nullable types or proper initialization |
| `Future.delayed` for debounce | `Timer` or proper debounce utility |

## Limitations
- These are language-specific guidelines and do not cover overall architectural decisions.
- Over-compression might reduce readability; apply judgement.


---
name: dashboard-design
description: Web and App implementation guide for Dashboard Design. Trigger when user wants analytics-focused layouts, data visualization, and modular overview screens.
date_added: "2026-06-17"
risk: safe
source: self
source_type: self
---

# Dashboard Design

> "Data at a glance. Organized, scannable, and highly functional."

## When to Use
Use this sub-style when the user's request matches the aesthetic described above. This is a child reference of the `design-it` skill and is not meant to be triggered directly.

## Core Principles
1. **Modular Grid**: The screen is broken down into functional "widgets" or cards. Usually a sidebar on the left and a top nav.
2. **Data Hierarchy**: The most important numbers (KPIs) are large and usually at the top. Charts take up the middle, and lists/tables are at the bottom.
3. **Muted Backgrounds**: A soft grey or off-white background so the white data cards stand out clearly.

## Visual DNA
- **Colors**: **Minimalist Slate** or **Earth-Grounded Elegance**. Avoid too many colors. Use red/green strictly for positive/negative trends.
- **Typography**: Clean, tabular sans-serifs (`Inter`, `Roboto Mono` for numbers).
- **Styling**: Very subtle shadows or 1px borders to separate cards.

## Web Implementation
- Use CSS Grid for the macro layout (Sidebar, Header, Main).
- **CSS Example**:
```css
body {
  background-color: #F8F9FA;
  color: #212529;
  font-family: 'Inter', sans-serif;
  margin: 0;
}

.dashboard-layout {
  display: grid;
  grid-template-columns: 250px 1fr;
  grid-template-rows: 70px 1fr;
  height: 100vh;
}

.sidebar {
  grid-row: 1 / 3;
  background-color: #ffffff;
  border-right: 1px solid #e9ecef;
  padding: 20px;
}

.header {
  background-color: #ffffff;
  border-bottom: 1px solid #e9ecef;
  padding: 0 30px;
  display: flex;
  align-items: center;
}

.main-content {
  padding: 30px;
  display: grid;
  grid-template-columns: repeat(4, 1fr);
  gap: 20px;
  overflow-y: auto;
}

/* KPI Card */
.kpi-card {
  background: #fff;
  border-radius: 8px;
  padding: 24px;
  border: 1px solid #e9ecef;
  box-shadow: 0 2px 4px rgba(0,0,0,0.02);
}

.kpi-title { font-size: 0.9rem; color: #6c757d; }
.kpi-value { font-size: 2rem; font-weight: 700; margin-top: 8px; }
.kpi-trend.positive { color: #28a745; }
```

## App Implementation

### SwiftUI
```swift
struct DashboardView: View {
    // For iPad/Mac, NavigationSplitView is ideal.
    // For iPhone, we use a scrolling VGrid.
    let columns = [
        GridItem(.adaptive(minimum: 150), spacing: 16)
    ]

    var body: some View {
        NavigationView {
            ScrollView {
                LazyVGrid(columns: columns, spacing: 16) {
                    KPICard(title: "Revenue", value: "$45,231", trend: "+12.5%", isPositive: true)
                    KPICard(title: "Active Users", value: "2,405", trend: "+4.1%", isPositive: true)
                    KPICard(title: "Churn Rate", value: "1.2%", trend: "-0.4%", isPositive: false)
                    KPICard(title: "Avg. Session", value: "4m 12s", trend: "+0.1%", isPositive: true)
                }
                .padding()

                // Placeholder for Chart
                RoundedRectangle(cornerRadius: 12)
                    .fill(Color.white)
                    .frame(height: 250)
                    .overlay(Text("Chart Area").foregroundColor(.gray))
                    .padding(.horizontal)
            }
            .background(Color(UIColor.systemGroupedBackground))
            .navigationTitle("Overview")
        }
    }
}

struct KPICard: View {
    let title: String
    let value: String
    let trend: String
    let isPositive: Bool

    var body: some View {
        VStack(alignment: .leading, spacing: 8) {
            Text(title).font(.subheadline).foregroundColor(.secondary)
            Text(value).font(.title2).fontWeight(.bold)
            Text(trend)
                .font(.caption)
                .fontWeight(.semibold)
                .foregroundColor(isPositive ? .green : .red)
        }
        .padding()
        .frame(maxWidth: .infinity, alignment: .leading)
        .background(Color.white)
        .cornerRadius(12)
        .shadow(color: Color.black.opacity(0.02), radius: 4, y: 2)
    }
}
```
- Dashboards require `.adaptive` grids. `LazyVGrid` handles rearranging 4 cards in a row on iPad down to 2 cards on iPhone automatically.
- Use `Color(UIColor.systemGroupedBackground)` to provide that subtle off-white contrast against stark white cards.

### Flutter
```dart
class DashboardScreen extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      backgroundColor: const Color(0xFFF8F9FA),
      appBar: AppBar(
        title: const Text('Overview', style: TextStyle(color: Colors.black)),
        backgroundColor: Colors.white,
        elevation: 1,
      ),
      // On tablets, use a Row with NavigationRail. On mobile, use Drawer.
      drawer: const Drawer(),
      body: CustomScrollView(
        slivers: [
          SliverPadding(
            padding: const EdgeInsets.all(16),
            sliver: SliverGrid.extent(
              maxCrossAxisExtent: 200, // Adapts layout based on width
              mainAxisSpacing: 16,
              crossAxisSpacing: 16,
              childAspectRatio: 1.5,
              children: [
                _buildKPI('Revenue', '\$45,231', '+12.5%', true),
                _buildKPI('Active Users', '2,405', '+4.1%', true),
                _buildKPI('Churn Rate', '1.2%', '-0.4%', false),
                _buildKPI('Avg. Session', '4m 12s', '+0.1%', true),
              ],
            ),
          ),
          SliverPadding(
            padding: const EdgeInsets.symmetric(horizontal: 16),
            sliver: SliverToBoxAdapter(
              child: Container(
                height: 250,
                decoration: BoxDecoration(color: Colors.white, borderRadius: BorderRadius.circular(12)),
                child: const Center(child: Text('Chart Area', style: TextStyle(color: Colors.grey))),
              ),
            ),
          ),
        ],
      ),
    );
  }

  Widget _buildKPI(String title, String value, String trend, bool isPositive) {
    return Container(
      padding: const EdgeInsets.all(16),
      decoration: BoxDecoration(
        color: Colors.white,
        borderRadius: BorderRadius.circular(12),
        border: Border.all(color: Colors.grey[200]!),
      ),
      child: Column(
        crossAxisAlignment: CrossAxisAlignment.start,
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          Text(title, style: const TextStyle(color: Colors.grey, fontSize: 14)),
          const SizedBox(height: 8),
          Text(value, style: const TextStyle(fontSize: 24, fontWeight: FontWeight.bold)),
          Text(trend, style: TextStyle(color: isPositive ? Colors.green : Colors.red, fontWeight: FontWeight.w600)),
        ],
      ),
    );
  }
}
```
- `SliverGrid.extent` with a `maxCrossAxisExtent` is the responsive magic bullet for Flutter dashboards. It handles varying screen widths flawlessly.
- For charts, the `fl_chart` package is the gold standard in Flutter.

### React Native
```jsx
const DashboardScreen = () => {
  return (
    <ScrollView style={{ flex: 1, backgroundColor: '#F8F9FA' }}>
      <View style={{ padding: 16, flexDirection: 'row', flexWrap: 'wrap', gap: 16 }}>
        <KPICard title="Revenue" value="$45,231" trend="+12.5%" isPositive={true} />
        <KPICard title="Users" value="2,405" trend="+4.1%" isPositive={true} />
        <KPICard title="Churn" value="1.2%" trend="-0.4%" isPositive={false} />
        <KPICard title="Session" value="4m 12s" trend="+0.1%" isPositive={true} />
      </View>

      <View style={{ paddingHorizontal: 16, paddingBottom: 32 }}>
        <View style={{ height: 250, backgroundColor: '#FFF', borderRadius: 12, justifyContent: 'center', alignItems: 'center' }}>
          <Text style={{ color: '#999' }}>Chart Area</Text>
        </View>
      </View>
    </ScrollView>
  );
};

const KPICard = ({ title, value, trend, isPositive }) => (
  <View style={{
    backgroundColor: '#FFF',
    padding: 16,
    borderRadius: 8,
    borderWidth: 1,
    borderColor: '#E9ECEF',
    flexBasis: '47%', // roughly half width minus gap
    minWidth: 150
  }}>
    <Text style={{ color: '#6C757D', fontSize: 14, marginBottom: 4 }}>{title}</Text>
    <Text style={{ fontSize: 22, fontWeight: '700', marginBottom: 4 }}>{value}</Text>
    <Text style={{ color: isPositive ? '#28A745' : '#DC3545', fontWeight: '600', fontSize: 12 }}>{trend}</Text>
  </View>
);
```
- To make responsive flex grids in React Native, use `flexDirection: 'row'`, `flexWrap: 'wrap'`, and set the children to `flexBasis: '47%'`.
- For heavy data visualization, look into `victory-native` or `@shopify/react-native-skia`.

### Jetpack Compose
```kotlin
@Composable
fun DashboardScreen() {
    Column(
        modifier = Modifier
            .fillMaxSize()
            .background(Color(0xFFF8F9FA))
            .verticalScroll(rememberScrollState())
    ) {
        // Top App Bar substitute
        Text(
            text = "Overview",
            style = MaterialTheme.typography.headlineSmall,
            fontWeight = FontWeight.Bold,
            modifier = Modifier.padding(16.dp)
        )

        // KPI Grid
        LazyVerticalGrid(
            columns = GridCells.Adaptive(minSize = 150.dp),
            contentPadding = PaddingValues(horizontal = 16.dp),
            horizontalArrangement = Arrangement.spacedBy(16.dp),
            verticalArrangement = Arrangement.spacedBy(16.dp),
            modifier = Modifier.heightIn(max = 400.dp) // Bound the grid height in scrollview
        ) {
            item { KPICard("Revenue", "$45,231", "+12.5%", true) }
            item { KPICard("Active Users", "2,405", "+4.1%", true) }
            item { KPICard("Churn Rate", "1.2%", "-0.4%", false) }
            item { KPICard("Avg. Session", "4m 12s", "+0.1%", true) }
        }

        Spacer(Modifier.height(16.dp))

        // Chart Area
        Box(
            modifier = Modifier
                .fillMaxWidth()
                .padding(horizontal = 16.dp)
                .height(250.dp)
                .background(Color.White, RoundedCornerShape(12.dp))
                .border(1.dp, Color(0xFFE9ECEF), RoundedCornerShape(12.dp)),
            contentAlignment = Alignment.Center
        ) {
            Text("Chart Area", color = Color.Gray)
        }

        Spacer(Modifier.height(32.dp))
    }
}

@Composable
fun KPICard(title: String, value: String, trend: String, isPositive: Boolean) {
    Column(
        modifier = Modifier
            .background(Color.White, RoundedCornerShape(8.dp))
            .border(1.dp, Color(0xFFE9ECEF), RoundedCornerShape(8.dp))
            .padding(16.dp)
    ) {
        Text(title, color = Color.Gray, fontSize = 14.sp)
        Spacer(Modifier.height(4.dp))
        Text(value, fontSize = 22.sp, fontWeight = FontWeight.Bold)
        Spacer(Modifier.height(4.dp))
        Text(trend, color = if (isPositive) Color(0xFF28A745) else Color(0xFFDC3545), fontWeight = FontWeight.SemiBold, fontSize = 12.sp)
    }
}
```
- `GridCells.Adaptive(minSize = 150.dp)` creates the responsive card layout automatically.
- Warning: Nesting `LazyVerticalGrid` inside a `Column` with `.verticalScroll` can cause height calculation issues. You must use `.heightIn(max=...)` on the grid, or completely convert the entire layout to a single `LazyVerticalGrid` where the chart is just a `GridItemSpan(maxLineSpan)` element.

## Do's and Don'ts
- **DO**: Right-align numbers in tables so they are easier to scan and compare.
- **DON'T**: Clutter cards with unnecessary decorative images. The data is the decoration.

## Limitations
- This is a styling reference and does not replace environment-specific validation, accessibility testing, or expert review.
- Ensure appropriate contrast ratios and responsive behaviors are verified separately.


---
name: data-dense-design
description: Web and App implementation guide for Data-Dense Design. Trigger when user wants professional tools, maximum information density, and expert interfaces (like Bloomberg terminals or IDEs).
date_added: "2026-06-17"
risk: safe
source: self
source_type: self
---

# Data-Dense Design

> "Density is a feature. For expert users, reducing clicks is more important than whitespace."

## When to Use
Use this sub-style when the user's request matches the aesthetic described above. This is a child reference of the `design-it` skill and is not meant to be triggered directly.

## Core Principles
1. **Compact Layouts**: Extremely tight margins and padding (often 2px to 4px).
2. **Monospace & Tabular Data**: Numbers must align vertically perfectly.
3. **High Utility**: Every pixel serves a functional purpose. Minimal purely decorative elements.

## Visual DNA
- **Colors**: **Industrial Chic** (high contrast) or **Minimalist Slate**. Avoid bright backgrounds. Dark themes are heavily preferred to reduce eye strain over 8-hour sessions.
- **Typography**: Small base sizes (`11px` - `13px`). Strict use of monospace fonts (`Fira Code`, `JetBrains Mono`) for data.
- **Borders**: Thin `1px` borders (`#333` or `#e0e0e0`) are used extensively to separate tiny cells of data.

## Web Implementation
- Tables, CSS Grid, and Flexbox with zero gap.
- **CSS Example**:
```css
body {
  background-color: #1e1e1e; /* IDE Dark */
  color: #cccccc;
  font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
  font-size: 12px; /* Very small */
  margin: 0;
}

.dense-toolbar {
  display: flex;
  background-color: #2d2d2d;
  border-bottom: 1px solid #3c3c3c;
  padding: 2px 4px;
}

.dense-btn {
  background: transparent;
  color: #ccc;
  border: 1px solid transparent;
  padding: 2px 8px;
  border-radius: 2px;
  cursor: pointer;
}
.dense-btn:hover {
  background-color: #3c3c3c;
  border-color: #555;
}

/* Dense Data Table */
.data-table {
  width: 100%;
  border-collapse: collapse;
  font-family: 'JetBrains Mono', monospace;
}

.data-table th, .data-table td {
  padding: 4px 8px;
  border: 1px solid #3c3c3c;
  text-align: right; /* Numbers align right */
}

.data-table tr:nth-child(even) { background-color: #252526; }
.data-table tr:hover { background-color: #094771; color: #fff; } /* Selection highlight */
```

## App Implementation

### SwiftUI
```swift
struct DataDenseView: View {
    var body: some View {
        ScrollView([.horizontal, .vertical]) {
            Grid(horizontalSpacing: 0, verticalSpacing: 0) {
                // Header Row
                GridRow {
                    HeaderCell("SYM")
                    HeaderCell("BID")
                    HeaderCell("ASK")
                    HeaderCell("CHG")
                }

                // Data Rows
                DataRow(sym: "AAPL", bid: "173.40", ask: "173.45", chg: "+0.12", isPos: true)
                DataRow(sym: "MSFT", bid: "320.10", ask: "320.15", chg: "-0.45", isPos: false)
                DataRow(sym: "GOOG", bid: "135.20", ask: "135.30", chg: "+0.02", isPos: true)
            }
            .border(Color.gray.opacity(0.3), width: 1)
        }
        .background(Color(white: 0.12)) // Dark IDE background
    }
}

struct HeaderCell: View {
    let text: String
    init(_ text: String) { self.text = text }
    var body: some View {
        Text(text)
            .font(.system(size: 11, weight: .bold, design: .monospaced))
            .foregroundColor(.gray)
            .padding(4)
            .frame(minWidth: 60, alignment: .leading)
            .border(Color.gray.opacity(0.3), width: 0.5)
            .background(Color(white: 0.18))
    }
}

struct DataRow: View {
    let sym, bid, ask, chg: String
    let isPos: Bool
    var body: some View {
        GridRow {
            Cell(sym, color: .white)
            Cell(bid, color: .white, align: .trailing)
            Cell(ask, color: .white, align: .trailing)
            Cell(chg, color: isPos ? .green : .red, align: .trailing)
        }
    }
}

struct Cell: View {
    let text: String
    let color: Color
    let align: Alignment
    init(_ text: String, color: Color, align: Alignment = .leading) {
        self.text = text; self.color = color; self.align = align
    }
    var body: some View {
        Text(text)
            .font(.system(size: 12, design: .monospaced))
            .foregroundColor(color)
            .padding(4)
            .frame(minWidth: 60, alignment: align)
            .border(Color.gray.opacity(0.3), width: 0.5)
    }
}
```
- Use `Grid` with `0` spacing.
- Font must be `.system(..., design: .monospaced)`.
- Use a `.border()` with `0.5` width on every single cell to recreate the dense spreadsheet look.

### Flutter
```dart
class DataDenseScreen extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      backgroundColor: const Color(0xFF1E1E1E),
      body: SingleChildScrollView(
        scrollDirection: Axis.vertical,
        child: SingleChildScrollView(
          scrollDirection: Axis.horizontal,
          child: Theme(
            // Override theme specifically to make the table hyper-dense
            data: Theme.of(context).copyWith(
              dividerColor: Colors.grey[800],
            ),
            child: DataTable(
              headingRowHeight: 28, // Extremely dense
              dataRowMinHeight: 24,
              dataRowMaxHeight: 24, // Extremely dense
              columnSpacing: 16,
              border: TableBorder.all(color: Colors.grey[800]!, width: 1),
              columns: const [
                DataColumn(label: Text('SYM', style: TextStyle(color: Colors.grey, fontSize: 11, fontFamily: 'RobotoMono'))),
                DataColumn(label: Text('BID', style: TextStyle(color: Colors.grey, fontSize: 11, fontFamily: 'RobotoMono')), numeric: true),
                DataColumn(label: Text('ASK', style: TextStyle(color: Colors.grey, fontSize: 11, fontFamily: 'RobotoMono')), numeric: true),
              ],
              rows: [
                _buildRow('AAPL', '173.40', '173.45'),
                _buildRow('MSFT', '320.10', '320.15'),
                _buildRow('GOOG', '135.20', '135.30'),
              ],
            ),
          ),
        ),
      ),
    );
  }

  DataRow _buildRow(String sym, String bid, String ask) {
    const style = TextStyle(color: Colors.white, fontSize: 12, fontFamily: 'RobotoMono');
    return DataRow(
      cells: [
        DataCell(Text(sym, style: style)),
        DataCell(Text(bid, style: style)),
        DataCell(Text(ask, style: style)),
      ],
    );
  }
}
```
- Flutter's `DataTable` is perfect, but you must manually crush the `headingRowHeight` and `dataRowHeight` down from their Material defaults (which are huge).
- Wrap in dual `SingleChildScrollView` to allow panning around large data sets.

### React Native
```jsx
const DataDenseScreen = () => {
  return (
    <ScrollView style={{ backgroundColor: '#1E1E1E', flex: 1 }}>
      <ScrollView horizontal>
        <View style={{ borderWidth: 1, borderColor: '#333' }}>
          {/* Header */}
          <View style={{ flexDirection: 'row', backgroundColor: '#2D2D2D' }}>
            <HeaderCell text="SYM" width={60} />
            <HeaderCell text="BID" width={80} align="right" />
            <HeaderCell text="ASK" width={80} align="right" />
            <HeaderCell text="CHG" width={60} align="right" />
          </View>

          {/* Rows */}
          <DataRow sym="AAPL" bid="173.40" ask="173.45" chg="+0.12" isPos={true} />
          <DataRow sym="MSFT" bid="320.10" ask="320.15" chg="-0.45" isPos={false} />
        </View>
      </ScrollView>
    </ScrollView>
  );
};

const HeaderCell = ({ text, width, align = 'left' }) => (
  <View style={{ width, padding: 4, borderWidth: 0.5, borderColor: '#333' }}>
    <Text style={{ color: '#999', fontSize: 11, fontFamily: 'monospace', textAlign: align, fontWeight: 'bold' }}>
      {text}
    </Text>
  </View>
);

const DataRow = ({ sym, bid, ask, chg, isPos }) => (
  <View style={{ flexDirection: 'row' }}>
    <Cell text={sym} width={60} />
    <Cell text={bid} width={80} align="right" />
    <Cell text={ask} width={80} align="right" />
    <Cell text={chg} width={60} align="right" color={isPos ? '#4CAF50' : '#F44336'} />
  </View>
);

const Cell = ({ text, width, align = 'left', color = '#CCC' }) => (
  <View style={{ width, padding: 4, borderWidth: 0.5, borderColor: '#333' }}>
    <Text style={{ color, fontSize: 12, fontFamily: 'monospace', textAlign: align }}>
      {text}
    </Text>
  </View>
);
```
- React Native doesn't have a native Table component, so you must construct a grid using `flexDirection: 'row'` and strict `width` properties on cells.
- Double scroll views (one vertical, one horizontal inside it) are standard for mobile data tables.

### Jetpack Compose
```kotlin
@Composable
fun DataDenseScreen() {
    val scrollStateHorizontal = rememberScrollState()
    val scrollStateVertical = rememberScrollState()

    Box(
        modifier = Modifier
            .fillMaxSize()
            .background(Color(0xFF1E1E1E))
            .verticalScroll(scrollStateVertical)
            .horizontalScroll(scrollStateHorizontal)
            .padding(8.dp)
    ) {
        Column(modifier = Modifier.border(1.dp, Color(0xFF333333))) {
            // Header
            Row(modifier = Modifier.background(Color(0xFF2D2D2D))) {
                Cell("SYM", 60.dp, Color.Gray, true)
                Cell("BID", 80.dp, Color.Gray, true, TextAlign.End)
                Cell("ASK", 80.dp, Color.Gray, true, TextAlign.End)
            }

            // Rows
            DataRow("AAPL", "173.40", "173.45")
            DataRow("MSFT", "320.10", "320.15")
        }
    }
}

@Composable
fun DataRow(sym: String, bid: String, ask: String) {
    Row {
        Cell(sym, 60.dp, Color.White, false)
        Cell(bid, 80.dp, Color.White, false, TextAlign.End)
        Cell(ask, 80.dp, Color.White, false, TextAlign.End)
    }
}

@Composable
fun Cell(text: String, width: Dp, color: Color, isHeader: Boolean, align: TextAlign = TextAlign.Start) {
    Text(
        text = text,
        color = color,
        fontSize = if (isHeader) 11.sp else 12.sp,
        fontWeight = if (isHeader) FontWeight.Bold else FontWeight.Normal,
        fontFamily = FontFamily.Monospace,
        textAlign = align,
        modifier = Modifier
            .width(width)
            .border(0.5.dp, Color(0xFF333333))
            .padding(4.dp)
    )
}
```
- Chain `.verticalScroll()` and `.horizontalScroll()` on the root `Box`.
- Compose allows `.border()` directly on `Text` modifiers, making spreadsheet grids very clean to implement without wrapping everything in `Box`es.

## Do's and Don'ts
- **DO**: Use color coding (red/green) for data deltas, but keep the saturation muted to prevent eye fatigue.
- **DON'T**: Use large padding or giant H1 headers. Expert users already know what screen they are on.

## Limitations
- This is a styling reference and does not replace environment-specific validation, accessibility testing, or expert review.
- Ensure appropriate contrast ratios and responsive behaviors are verified separately.


---
name: data-engineer
description: Build scalable data pipelines, modern data warehouses, and real-time streaming architectures. Implements Apache Spark, dbt, Airflow, and cloud-native data platforms.
risk: critical
source: community
date_added: '2026-02-27'
---
You are a data engineer specializing in scalable data pipelines, modern data architecture, and analytics infrastructure.

## Use this skill when

- Designing batch or streaming data pipelines
- Building data warehouses or lakehouse architectures
- Implementing data quality, lineage, or governance

## Do not use this skill when

- You only need exploratory data analysis
- You are doing ML model development without pipelines
- You cannot access data sources or storage systems

## Instructions

1. Define sources, SLAs, and data contracts.
2. Choose architecture, storage, and orchestration tools.
3. Implement ingestion, transformation, and validation.
4. Monitor quality, costs, and operational reliability.

## Safety

- Protect PII and enforce least-privilege access.
- Validate data before writing to production sinks.

## Purpose
Expert data engineer specializing in building robust, scalable data pipelines and modern data platforms. Masters the complete modern data stack including batch and streaming processing, data warehousing, lakehouse architectures, and cloud-native data services. Focuses on reliable, performant, and cost-effective data solutions.

## Capabilities

### Modern Data Stack & Architecture
- Data lakehouse architectures with Delta Lake, Apache Iceberg, and Apache Hudi
- Cloud data warehouses: Snowflake, BigQuery, Redshift, Databricks SQL
- Data lakes: AWS S3, Azure Data Lake, Google Cloud Storage with structured organization
- Modern data stack integration: Fivetran/Airbyte + dbt + Snowflake/BigQuery + BI tools
- Data mesh architectures with domain-driven data ownership
- Real-time analytics with Apache Pinot, ClickHouse, Apache Druid
- OLAP engines: Presto/Trino, Apache Spark SQL, Databricks Runtime

### Batch Processing & ETL/ELT
- Apache Spark 4.0 with optimized Catalyst engine and columnar processing
- dbt Core/Cloud for data transformations with version control and testing
- Apache Airflow for complex workflow orchestration and dependency management
- Databricks for unified analytics platform with collaborative notebooks
- AWS Glue, Azure Synapse Analytics, Google Dataflow for cloud ETL
- Custom Python/Scala data processing with pandas, Polars, Ray
- Data validation and quality monitoring with Great Expectations
- Data profiling and discovery with Apache Atlas, DataHub, Amundsen

### Real-Time Streaming & Event Processing
- Apache Kafka and Confluent Platform for event streaming
- Apache Pulsar for geo-replicated messaging and multi-tenancy
- Apache Flink and Kafka Streams for complex event processing
- AWS Kinesis, Azure Event Hubs, Google Pub/Sub for cloud streaming
- Real-time data pipelines with change data capture (CDC)
- Stream processing with windowing, aggregations, and joins
- Event-driven architectures with schema evolution and compatibility
- Real-time feature engineering for ML applications

### Workflow Orchestration & Pipeline Management
- Apache Airflow with custom operators and dynamic DAG generation
- Prefect for modern workflow orchestration with dynamic execution
- Dagster for asset-based data pipeline orchestration
- Azure Data Factory and AWS Step Functions for cloud workflows
- GitHub Actions and GitLab CI/CD for data pipeline automation
- Kubernetes CronJobs and Argo Workflows for container-native scheduling
- Pipeline monitoring, alerting, and failure recovery mechanisms
- Data lineage tracking and impact analysis

### Data Modeling & Warehousing
- Dimensional modeling: star schema, snowflake schema design
- Data vault modeling for enterprise data warehousing
- One Big Table (OBT) and wide table approaches for analytics
- Slowly changing dimensions (SCD) implementation strategies
- Data partitioning and clustering strategies for performance
- Incremental data loading and change data capture patterns
- Data archiving and retention policy implementation
- Performance tuning: indexing, materialized views, query optimization

### Cloud Data Platforms & Services

#### AWS Data Engineering Stack
- Amazon S3 for data lake with intelligent tiering and lifecycle policies
- AWS Glue for serverless ETL with automatic schema discovery
- Amazon Redshift and Redshift Spectrum for data warehousing
- Amazon EMR and EMR Serverless for big data processing
- Amazon Kinesis for real-time streaming and analytics
- AWS Lake Formation for data lake governance and security
- Amazon Athena for serverless SQL queries on S3 data
- AWS DataBrew for visual data preparation

#### Azure Data Engineering Stack
- Azure Data Lake Storage Gen2 for hierarchical data lake
- Azure Synapse Analytics for unified analytics platform
- Azure Data Factory for cloud-native data integration
- Azure Databricks for collaborative analytics and ML
- Azure Stream Analytics for real-time stream processing
- Azure Purview for unified data governance and catalog
- Azure SQL Database and Cosmos DB for operational data stores
- Power BI integration for self-service analytics

#### GCP Data Engineering Stack
- Google Cloud Storage for object storage and data lake
- BigQuery for serverless data warehouse with ML capabilities
- Cloud Dataflow for stream and batch data processing
- Cloud Composer (managed Airflow) for workflow orchestration
- Cloud Pub/Sub for messaging and event ingestion
- Cloud Data Fusion for visual data integration
- Cloud Dataproc for managed Hadoop and Spark clusters
- Looker integration for business intelligence

### Data Quality & Governance
- Data quality frameworks with Great Expectations and custom validators
- Data lineage tracking with DataHub, Apache Atlas, Collibra
- Data catalog implementation with metadata management
- Data privacy and compliance: GDPR, CCPA, HIPAA considerations
- Data masking and anonymization techniques
- Access control and row-level security implementation
- Data monitoring and alerting for quality issues
- Schema evolution and backward compatibility management

### Performance Optimization & Scaling
- Query optimization techniques across different engines
- Partitioning and clustering strategies for large datasets
- Caching and materialized view optimization
- Resource allocation and cost optimization for cloud workloads
- Auto-scaling and spot instance utilization for batch jobs
- Performance monitoring and bottleneck identification
- Data compression and columnar storage optimization
- Distributed processing optimization with appropriate parallelism

### Database Technologies & Integration
- Relational databases: PostgreSQL, MySQL, SQL Server integration
- NoSQL databases: MongoDB, Cassandra, DynamoDB for diverse data types
- Time-series databases: InfluxDB, TimescaleDB for IoT and monitoring data
- Graph databases: Neo4j, Amazon Neptune for relationship analysis
- Search engines: Elasticsearch, OpenSearch for full-text search
- Vector databases: Pinecone, Qdrant for AI/ML applications
- Database replication, CDC, and synchronization patterns
- Multi-database query federation and virtualization

### Infrastructure & DevOps for Data
- Infrastructure as Code with Terraform, CloudFormation, Bicep
- Containerization with Docker and Kubernetes for data applications
- CI/CD pipelines for data infrastructure and code deployment
- Version control strategies for data code, schemas, and configurations
- Environment management: dev, staging, production data environments
- Secrets management and secure credential handling
- Monitoring and logging with Prometheus, Grafana, ELK stack
- Disaster recovery and backup strategies for data systems

### Data Security & Compliance
- Encryption at rest and in transit for all data movement
- Identity and access management (IAM) for data resources
- Network security and VPC configuration for data platforms
- Audit logging and compliance reporting automation
- Data classification and sensitivity labeling
- Privacy-preserving techniques: differential privacy, k-anonymity
- Secure data sharing and collaboration patterns
- Compliance automation and policy enforcement

### Integration & API Development
- RESTful APIs for data access and metadata management
- GraphQL APIs for flexible data querying and federation
- Real-time APIs with WebSockets and Server-Sent Events
- Data API gateways and rate limiting implementation
- Event-driven integration patterns with message queues
- Third-party data source integration: APIs, databases, SaaS platforms
- Data synchronization and conflict resolution strategies
- API documentation and developer experience optimization

## Behavioral Traits
- Prioritizes data reliability and consistency over quick fixes
- Implements comprehensive monitoring and alerting from the start
- Focuses on scalable and maintainable data architecture decisions
- Emphasizes cost optimization while maintaining performance requirements
- Plans for data governance and compliance from the design phase
- Uses infrastructure as code for reproducible deployments
- Implements thorough testing for data pipelines and transformations
- Documents data schemas, lineage, and business logic clearly
- Stays current with evolving data technologies and best practices
- Balances performance optimization with operational simplicity

## Knowledge Base
- Modern data stack architectures and integration patterns
- Cloud-native data services and their optimization techniques
- Streaming and batch processing design patterns
- Data modeling techniques for different analytical use cases
- Performance tuning across various data processing engines
- Data governance and quality management best practices
- Cost optimization strategies for cloud data workloads
- Security and compliance requirements for data systems
- DevOps practices adapted for data engineering workflows
- Emerging trends in data architecture and tooling

## Response Approach
1. **Analyze data requirements** for scale, latency, and consistency needs
2. **Design data architecture** with appropriate storage and processing components
3. **Implement robust data pipelines** with comprehensive error handling and monitoring
4. **Include data quality checks** and validation throughout the pipeline
5. **Consider cost and performance** implications of architectural decisions
6. **Plan for data governance** and compliance requirements early
7. **Implement monitoring and alerting** for data pipeline health and performance
8. **Document data flows** and provide operational runbooks for maintenance

## Example Interactions
- "Design a real-time streaming pipeline that processes 1M events per second from Kafka to BigQuery"
- "Build a modern data stack with dbt, Snowflake, and Fivetran for dimensional modeling"
- "Implement a cost-optimized data lakehouse architecture using Delta Lake on AWS"
- "Create a data quality framework that monitors and alerts on data anomalies"
- "Design a multi-tenant data platform with proper isolation and governance"
- "Build a change data capture pipeline for real-time synchronization between databases"
- "Implement a data mesh architecture with domain-specific data products"
- "Create a scalable ETL pipeline that handles late-arriving and out-of-order data"

## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.


---
name: data-engineering-data-driven-feature
description: "Build features guided by data insights, A/B testing, and continuous measurement using specialized agents for analysis, implementation, and experimentation."
risk: critical
source: community
date_added: "2026-02-27"
---

# Data-Driven Feature Development

Build features guided by data insights, A/B testing, and continuous measurement using specialized agents for analysis, implementation, and experimentation.

[Extended thinking: This workflow orchestrates a comprehensive data-driven development process from initial data analysis and hypothesis formulation through feature implementation with integrated analytics, A/B testing infrastructure, and post-launch analysis. Each phase leverages specialized agents to ensure features are built based on data insights, properly instrumented for measurement, and validated through controlled experiments. The workflow emphasizes modern product analytics practices, statistical rigor in testing, and continuous learning from user behavior.]

## Use this skill when

- Working on data-driven feature development tasks or workflows
- Needing guidance, best practices, or checklists for data-driven feature development

## Do not use this skill when

- The task is unrelated to data-driven feature development
- You need a different domain or tool outside this scope

## Instructions

- Clarify goals, constraints, and required inputs.
- Apply relevant best practices and validate outcomes.
- Provide actionable steps and verification.
- If detailed examples are required, open `resources/implementation-playbook.md`.

## Phase 1: Data Analysis and Hypothesis Formation

### 1. Exploratory Data Analysis
- Use Task tool with subagent_type="machine-learning-ops::data-scientist"
- Prompt: "Perform exploratory data analysis for feature: $ARGUMENTS. Analyze existing user behavior data, identify patterns and opportunities, segment users by behavior, and calculate baseline metrics. Use modern analytics tools (Amplitude, Mixpanel, Segment) to understand current user journeys, conversion funnels, and engagement patterns."
- Output: EDA report with visualizations, user segments, behavioral patterns, baseline metrics

### 2. Business Hypothesis Development
- Use Task tool with subagent_type="business-analytics::business-analyst"
- Context: Data scientist's EDA findings and behavioral patterns
- Prompt: "Formulate business hypotheses for feature: $ARGUMENTS based on data analysis. Define clear success metrics, expected impact on key business KPIs, target user segments, and minimum detectable effects. Create measurable hypotheses using frameworks like ICE scoring or RICE prioritization."
- Output: Hypothesis document, success metrics definition, expected ROI calculations

### 3. Statistical Experiment Design
- Use Task tool with subagent_type="machine-learning-ops::data-scientist"
- Context: Business hypotheses and success metrics
- Prompt: "Design statistical experiment for feature: $ARGUMENTS. Calculate required sample size for statistical power, define control and treatment groups, specify randomization strategy, and plan for multiple testing corrections. Consider Bayesian A/B testing approaches for faster decision making. Design for both primary and guardrail metrics."
- Output: Experiment design document, power analysis, statistical test plan

## Phase 2: Feature Architecture and Analytics Design

### 4. Feature Architecture Planning
- Use Task tool with subagent_type="data-engineering::backend-architect"
- Context: Business requirements and experiment design
- Prompt: "Design feature architecture for: $ARGUMENTS with A/B testing capability. Include feature flag integration (LaunchDarkly, Split.io, or Optimizely), gradual rollout strategy, circuit breakers for safety, and clean separation between control and treatment logic. Ensure architecture supports real-time configuration updates."
- Output: Architecture diagrams, feature flag schema, rollout strategy

### 5. Analytics Instrumentation Design
- Use Task tool with subagent_type="data-engineering::data-engineer"
- Context: Feature architecture and success metrics
- Prompt: "Design comprehensive analytics instrumentation for: $ARGUMENTS. Define event schemas for user interactions, specify properties for segmentation and analysis, design funnel tracking and conversion events, plan cohort analysis capabilities. Implement using modern SDKs (Segment, Amplitude, Mixpanel) with proper event taxonomy."
- Output: Event tracking plan, analytics schema, instrumentation guide

### 6. Data Pipeline Architecture
- Use Task tool with subagent_type="data-engineering::data-engineer"
- Context: Analytics requirements and existing data infrastructure
- Prompt: "Design data pipelines for feature: $ARGUMENTS. Include real-time streaming for live metrics (Kafka, Kinesis), batch processing for detailed analysis, data warehouse integration (Snowflake, BigQuery), and feature store for ML if applicable. Ensure proper data governance and GDPR compliance."
- Output: Pipeline architecture, ETL/ELT specifications, data flow diagrams

## Phase 3: Implementation with Instrumentation

### 7. Backend Implementation
- Use Task tool with subagent_type="backend-development::backend-architect"
- Context: Architecture design and feature requirements
- Prompt: "Implement backend for feature: $ARGUMENTS with full instrumentation. Include feature flag checks at decision points, comprehensive event tracking for all user actions, performance metrics collection, error tracking and monitoring. Implement proper logging for experiment analysis."
- Output: Backend code with analytics, feature flag integration, monitoring setup

### 8. Frontend Implementation
- Use Task tool with subagent_type="frontend-mobile-development::frontend-developer"
- Context: Backend APIs and analytics requirements
- Prompt: "Build frontend for feature: $ARGUMENTS with analytics tracking. Implement event tracking for all user interactions, session recording integration if applicable, performance metrics (Core Web Vitals), and proper error boundaries. Ensure consistent experience between control and treatment groups."
- Output: Frontend code with analytics, A/B test variants, performance monitoring

### 9. ML Model Integration (if applicable)
- Use Task tool with subagent_type="machine-learning-ops::ml-engineer"
- Context: Feature requirements and data pipelines
- Prompt: "Integrate ML models for feature: $ARGUMENTS if needed. Implement online inference with low latency, A/B testing between model versions, model performance tracking, and automatic fallback mechanisms. Set up model monitoring for drift detection."
- Output: ML pipeline, model serving infrastructure, monitoring setup

## Phase 4: Pre-Launch Validation

### 10. Analytics Validation
- Use Task tool with subagent_type="data-engineering::data-engineer"
- Context: Implemented tracking and event schemas
- Prompt: "Validate analytics implementation for: $ARGUMENTS. Test all event tracking in staging, verify data quality and completeness, validate funnel definitions, ensure proper user identification and session tracking. Run end-to-end tests for data pipeline."
- Output: Validation report, data quality metrics, tracking coverage analysis

### 11. Experiment Setup
- Use Task tool with subagent_type="cloud-infrastructure::deployment-engineer"
- Context: Feature flags and experiment design
- Prompt: "Configure experiment infrastructure for: $ARGUMENTS. Set up feature flags with proper targeting rules, configure traffic allocation (start with 5-10%), implement kill switches, set up monitoring alerts for key metrics. Test randomization and assignment logic."
- Output: Experiment configuration, monitoring dashboards, rollout plan

## Phase 5: Launch and Experimentation

### 12. Gradual Rollout
- Use Task tool with subagent_type="cloud-infrastructure::deployment-engineer"
- Context: Experiment configuration and monitoring setup
- Prompt: "Execute gradual rollout for feature: $ARGUMENTS. Start with internal dogfooding, then beta users (1-5%), gradually increase to target traffic. Monitor error rates, performance metrics, and early indicators. Implement automated rollback on anomalies."
- Output: Rollout execution, monitoring alerts, health metrics

### 13. Real-time Monitoring
- Use Task tool with subagent_type="observability-monitoring::observability-engineer"
- Context: Deployed feature and success metrics
- Prompt: "Set up comprehensive monitoring for: $ARGUMENTS. Create real-time dashboards for experiment metrics, configure alerts for statistical significance, monitor guardrail metrics for negative impacts, track system performance and error rates. Use tools like Datadog, New Relic, or custom dashboards."
- Output: Monitoring dashboards, alert configurations, SLO definitions

## Phase 6: Analysis and Decision Making

### 14. Statistical Analysis
- Use Task tool with subagent_type="machine-learning-ops::data-scientist"
- Context: Experiment data and original hypotheses
- Prompt: "Analyze A/B test results for: $ARGUMENTS. Calculate statistical significance with confidence intervals, check for segment-level effects, analyze secondary metrics impact, investigate any unexpected patterns. Use both frequentist and Bayesian approaches. Account for multiple testing if applicable."
- Output: Statistical analysis report, significance tests, segment analysis

### 15. Business Impact Assessment
- Use Task tool with subagent_type="business-analytics::business-analyst"
- Context: Statistical analysis and business metrics
- Prompt: "Assess business impact of feature: $ARGUMENTS. Calculate actual vs expected ROI, analyze impact on key business metrics, evaluate cost-benefit including operational overhead, project long-term value. Make recommendation on full rollout, iteration, or rollback."
- Output: Business impact report, ROI analysis, recommendation document

### 16. Post-Launch Optimization
- Use Task tool with subagent_type="machine-learning-ops::data-scientist"
- Context: Launch results and user feedback
- Prompt: "Identify optimization opportunities for: $ARGUMENTS based on data. Analyze user behavior patterns in treatment group, identify friction points in user journey, suggest improvements based on data, plan follow-up experiments. Use cohort analysis for long-term impact."
- Output: Optimization recommendations, follow-up experiment plans

## Configuration Options

```yaml
experiment_config:
  min_sample_size: 10000
  confidence_level: 0.95
  runtime_days: 14
  traffic_allocation: "gradual"  # gradual, fixed, or adaptive

analytics_platforms:
  - amplitude
  - segment
  - mixpanel

feature_flags:
  provider: "launchdarkly"  # launchdarkly, split, optimizely, unleash

statistical_methods:
  - frequentist
  - bayesian

monitoring:
  - real_time_metrics: true
  - anomaly_detection: true
  - automatic_rollback: true
```

## Success Criteria

- **Data Coverage**: 100% of user interactions tracked with proper event schema
- **Experiment Validity**: Proper randomization, sufficient statistical power, no sample ratio mismatch
- **Statistical Rigor**: Clear significance testing, proper confidence intervals, multiple testing corrections
- **Business Impact**: Measurable improvement in target metrics without degrading guardrail metrics
- **Technical Performance**: No degradation in p95 latency, error rates below 0.1%
- **Decision Speed**: Clear go/no-go decision within planned experiment runtime
- **Learning Outcomes**: Documented insights for future feature development

## Coordination Notes

- Data scientists and business analysts collaborate on hypothesis formation
- Engineers implement with analytics as first-class requirement, not afterthought
- Feature flags enable safe experimentation without full deployments
- Real-time monitoring allows for quick iteration and rollback if needed
- Statistical rigor balanced with business practicality and speed to market
- Continuous learning loop feeds back into next feature development cycle

Feature to develop with data-driven approach: $ARGUMENTS

## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.