---
name: explore-data
description: Systematic EDA and data-quality assessment with explicit data-leakage detection before modeling.
keywords: eda, data, leakage, data quality, exploration, dataset
---

# Explore Data

> This skill borrows the investigative mindset of `superpowers:systematic-debugging` — form
> hypotheses about the data, verify each one with evidence, and document what you find.
> Do not skip a check because the data "looks clean".

---

## When to use this skill

- Gate 1, immediately after `frame-ml-problem` has defined the target and metric
- Any time a new dataset enters the project (new data source, updated extract, external dataset)
- Before re-training when data distribution changes are suspected

---

## Steps

### 1. Profile the dataset

Establish the basic facts before any analysis:
- **Shape:** row count, column count; confirm against expected data volume from the ticket
- **dtypes:** numeric, categorical, datetime, free text, binary — flag any type mismatches (e.g. numeric column stored as string)
- **Missing values:** per-column missing rate; distinguish structural missing (feature does not apply) from random missing (collection gap)
- **Duplicates:** exact duplicate rows; near-duplicate rows (same entity, slightly different values); duplicate IDs that should be unique
- **Target distribution:** class balance for classification (report majority-class baseline accuracy); value range, skew, and outliers for regression

### 2. Univariate and bivariate analysis

- **Univariate:** distribution shape, outliers, value ranges, cardinality of categoricals
- **Bivariate:** feature–target correlations or mutual information; flag suspiciously high correlations that may indicate leakage (see Step 3)
- **Temporal trends:** if a time column exists, plot target rate and key features over time to detect distribution shift

### 3. Leakage detection (mandatory)

Run all three checks explicitly and document the result of each — do not skip any.

**(a) Target leakage** — features derived from or computed after the target event:
- Identify any column that could not be known at prediction time (e.g. a "days to resolution" column when predicting whether a ticket will be resolved)
- Drop or exclude any such column before training
- Flag columns with suspiciously high predictive power; investigate their origin

**(b) Train/test contamination** — information bleeding between splits:
- Check for duplicate or near-duplicate rows that may span the train and test sets
- Verify that entity IDs (user IDs, order IDs) do not appear in both splits if the task requires generalising to unseen entities
- Confirm that scalers, encoders, and imputers will be fit on the training fold only — fitting on the full dataset before splitting is a hard error

**(c) Temporal leakage** — using future data to predict the past:
- If the data has a time dimension, verify that all features for a given row reflect information available strictly before the prediction timestamp
- Check for lag features or aggregation windows that inadvertently include the target period
- Time-series splits must respect chronological order; random splits are wrong for temporal data

### 4. Data-quality flags

Document any of the following that are present:
- **Inconsistent units:** mixed currencies, mixed date formats, mixed length units
- **Label noise:** suspiciously contradictory labels for near-identical features
- **Sampling bias:** is the historical sample representative of the production population? Check for selection effects (e.g. data collected only from active users)
- **Train / serving distribution drift:** if serving statistics are available, compare them to training distributions on key features; flag deviations above a sensible threshold

### 5. Record findings in the Gate 1 data report

Write the EDA results into the `ml-problem.md` file created by `frame-ml-problem`:
- Data shape, quality issues, and missing-value strategy
- Leakage assessment — one paragraph per leakage type stating what was checked and what was found
- Cleaning actions required before training (drop columns, impute, clip outliers, re-encode)

Output language: auto-detect from the ticket/task input — see `custom/rules/output-language.md` (Vietnamese input → Vietnamese output; otherwise English).

---

## Completion Checklist

- [ ] Dataset shape, dtypes, and missing-value rates profiled
- [ ] Duplicate rows and duplicate IDs checked
- [ ] Target distribution inspected (class balance or value distribution)
- [ ] Univariate and bivariate analysis completed; suspiciously high correlations investigated
- [ ] **Target leakage** checked and documented (no future-derived features in the feature set)
- [ ] **Train/test contamination** checked and documented (no ID overlap, no full-data fitting)
- [ ] **Temporal leakage** checked and documented (chronological integrity verified if applicable)
- [ ] Data-quality issues listed (units, label noise, sampling bias, drift)
- [ ] Required cleaning steps noted
- [ ] All findings written into `AK-Docs/04.Coding/01.Requirements/[functionId]/[ticketId].md` data report section (see `custom/rules/ml-conventions.md` — do NOT use the legacy `plan/[ticket-id]/` path)