---
description: Data Engineering
alwaysApply: false
---

# Data Engineering

Guidelines for building robust, scalable data platforms and pipelines.

## Scope

- Batch pipelines (ETL/ELT) and stream processing
- Data warehouses, lakehouses, analytics engineering (DBT)
- Data quality and observability systems

## Key Principles

- **Idempotency is non-negotiable** — every pipeline must produce identical results on re-run
- **Data quality is a feature** — validate early, monitor continuously, alert proactively
- **Schema is a contract** — breaking changes require coordination and versioning
- **Observability over debugging** — instrument everything; never debug production with ad-hoc queries
- **Cost-aware engineering** — compute and storage have real costs; optimize deliberately

## Data Architecture Layers

- **Bronze/Raw** — exact copy of source, append-only (minutes-hours freshness)
- **Silver/Curated** — validated, typed, deduplicated (hours freshness)
- **Gold/Marts** — business-ready aggregates (hours-daily freshness)

## Pipeline Design Essentials

- Delete-then-insert or merge for idempotency — never blind append
- Use `execution_date`, not `current_timestamp()` for determinism
- Parameterize source/target tables and dates for testability
- Support backfill via parameterized execution dates
- Implement dead letter queues for invalid records

## Anti-Patterns

- Appending without clearing target partition (creates duplicates on re-run)
- Processing only today's data without a late-arrival window
- Hardcoded table names and connection strings in pipeline code

## Definition of Done

- [ ] Idempotency verified (re-run produces same result)
- [ ] Data quality checks implemented and passing
- [ ] Schema documented; monitoring and alerting configured
- [ ] Cost estimate understood