{
  "version": 1,
  "id": "sql-cohort-analysis",
  "title": "Cohort Analysis: Retention, Churn and LTV",
  "summary": "Group users by first activity and measure retention and lifetime value over time in plain SQL.",
  "estimatedMinutes": 25,
  "prerequisites": [],
  "source": "courses-notebook DS301_databases_sql 10_cohort_analysis",
  "lesson": "Group users by acquisition month, track behaviour over time, and measure product health.\n\n```python\nimport sqlite3, pandas as pd, numpy as np, matplotlib.pyplot as plt\nprint('Cohort Analysis')\n```\n\n## Key Concepts\n\n### What is a Cohort?\n\nA cohort is a group of users sharing a common starting event \u2014 most commonly the month they first purchased or signed up (**acquisition cohort**).\n\n### Retention Matrix\n\n$$\\text{Retention}_{c,t} = \\frac{|\\text{users in cohort }c\\text{ active in period }t|}{|\\text{cohort }c\\text{ size}|}$$\n\nRows = cohorts, columns = periods since acquisition. Period 0 = 100% by definition.\n\n### Key Metrics\n\n**Month-1 retention** \u2014 the most actionable early signal. If fewer than 30% of users return after their first month, the product has a core value problem.\n\n**Churn rate at period** $t$: $1 - \\text{retention}_t$\n\n**Customer LTV**:\n\n$$\\text{LTV} = \\text{ARPU} \\times \\frac{1}{\\text{monthly churn rate}}$$\n\n### Reading the Heatmap\n\n- Diagonal unavailability is normal (recent cohorts have less history)\n- Horizontal improvement across all cohorts at the same period: product change\n- Sudden vertical drop: a specific month had a problem (seasonal, bug)\n\n```python\nimport sqlite3, pandas as pd, numpy as np, matplotlib.pyplot as plt\n\nrng = np.random.default_rng(42)\nconn = sqlite3.connect(\":memory:\")\nconn.execute(\"CREATE TABLE events (user_id INT, cohort TEXT, period INT)\")\n\nrows = []\nfor ci, cohort in enumerate([\"2023-01\",\"2023-02\",\"2023-03\",\"2023-04\",\"2023-05\",\"2023-06\"]):\n    n = rng.integers(80, 150)\n    for u in range(n):\n        base_ret = rng.uniform(0.75, 0.92)\n        for period in range(8):\n            prob = base_ret ** period\n            if rng.random() < prob:\n                rows.append((ci*150+u, cohort, period))\n\nconn.executemany(\"INSERT INTO events VALUES (?,?,?)\", rows)\nconn.commit()\n\ndf = pd.read_sql(\"\"\"\nWITH base AS (\n    SELECT cohort, COUNT(DISTINCT user_id) AS total FROM events WHERE period=0 GROUP BY cohort\n),\nact AS (\n    SELECT cohort, period, COUNT(DISTINCT user_id) AS active FROM events GROUP BY cohort, period\n)\nSELECT a.cohort, a.period, ROUND(100.0*a.active/b.total,1) AS retention\nFROM act a JOIN base b ON a.cohort=b.cohort\n\"\"\", conn)\n\nmat = df.pivot(index=\"cohort\", columns=\"period\", values=\"retention\")\n\nfig, axes = plt.subplots(1,2, figsize=(13,5))\nim = axes[0].imshow(mat.values, cmap=\"YlGn\", vmin=0, vmax=100, aspect=\"auto\")\naxes[0].set_xticks(range(len(mat.columns))); axes[0].set_xticklabels([f\"M+{c}\" for c in mat.columns])\naxes[0].set_yticks(range(len(mat.index))); axes[0].set_yticklabels(mat.index)\nfor i in range(len(mat.index)):\n    for j in range(len(mat.columns)):\n        v = mat.values[i,j]\n        if not np.isnan(v):\n            axes[0].text(j,i,f\"{v:.0f}%\",ha=\"center\",va=\"center\",fontsize=9,\n                        color=\"white\" if v<40 else \"black\")\nplt.colorbar(im, ax=axes[0], label=\"Retention %\")\naxes[0].set_title(\"Cohort Retention Heatmap\")\n\nfor cohort in mat.index:\n    row = mat.loc[cohort].dropna()\n    axes[1].plot(row.index, row.values, marker=\"o\", ms=5, lw=2, label=cohort)\naxes[1].set_xlabel(\"Period\"); axes[1].set_ylabel(\"Retention %\")\naxes[1].set_title(\"Retention Curves by Cohort\"); axes[1].legend(fontsize=8)\naxes[1].grid(alpha=0.3); axes[1].spines[\"top\"].set_visible(False); axes[1].spines[\"right\"].set_visible(False)\nplt.tight_layout(); plt.show()\nconn.close()\n```\n\n```python\n# Data analysis solution checker\n\nimport pandas as pd\nimport numpy as np\n\ndef summarise_df(df, label=\"DataFrame\"):\n    \"\"\"Quick sanity check for a cleaned dataframe.\"\"\"\n    print(f\"=== {label} ===\")\n    print(f\"  Shape: {df.shape}\")\n    print(f\"  Missing: {df.isnull().sum().sum()} values ({df.isnull().mean().mean():.1%})\")\n    print(f\"  Dtypes: {df.dtypes.value_counts().to_dict()}\")\n    if df.select_dtypes('number').shape[1] > 0:\n        num = df.select_dtypes('number')\n        print(f\"  Numeric range: [{num.values.min():.2f}, {num.values.max():.2f}]\")\n    dupes = df.duplicated().sum()\n    print(f\"  Duplicates: {dupes}\")\n\ndef check_clean(df):\n    \"\"\"Assert DataFrame is clean: no NaN, no dupes, correct dtypes.\"\"\"\n    issues = []\n    if df.isnull().any().any():\n        issues.append(f\"{df.isnull().sum().sum()} missing values remain\")\n    if df.duplicated().any():\n        issues.append(f\"{df.duplicated().sum()} duplicate rows remain\")\n    if issues:\n        print(\"\u2717 ISSUES: \" + \"; \".join(issues))\n    else:\n        print(\"\u2713 DataFrame is clean\")\n\n# Demo\nnp.random.seed(0)\ndf = pd.DataFrame({\n    'age':    [25, 30, np.nan, 22, 35, 30],\n    'salary': [50000, 60000, 75000, 45000, None, 60000],\n    'dept':   ['Eng', 'HR', 'Eng', 'Sales', 'HR', 'HR'],\n})\nsummarise_df(df, \"Raw\")\ndf_clean = df.dropna().drop_duplicates()\nsummarise_df(df_clean, \"Cleaned\")\ncheck_clean(df_clean)\n```\n\n```python\n# Advanced: Pandas performance patterns\n\nimport pandas as pd\nimport numpy as np\nimport timeit\n\nnp.random.seed(42)\nn = 100_000\ndf = pd.DataFrame({\n    'group':   np.random.choice(list('ABCDE'), n),\n    'value':   np.random.randn(n),\n    'flag':    np.random.randint(0, 2, n).astype(bool),\n    'amount':  np.random.exponential(100, n),\n})\n\ndef time_it(label, fn):\n    t = timeit.timeit(fn, number=10) / 10 * 1000\n    print(f\"  {label:<40}: {t:.2f}ms\")\n\nprint(\"Pandas performance patterns:\")\n\n# 1. apply vs vectorised operation\ntime_it(\"df['value'].apply(lambda x: x**2)\", lambda: df['value'].apply(lambda x: x**2))\ntime_it(\"df['value'] ** 2  (vectorised)\", lambda: df['value'] ** 2)\n\n# 2. iterrows vs vectorised groupby\ntime_it(\"groupby().mean()\", lambda: df.groupby('group')['amount'].mean())\ntime_it(\"groupby().agg({'amount': ['mean', 'std', 'count']})\",\n        lambda: df.groupby('group')['amount'].agg(['mean', 'std', 'count']))\n\n# 3. Boolean indexing vs query\ntime_it(\"df[df.flag & (df.value > 0)]\",\n        lambda: df[df['flag'] & (df['value'] > 0)])\ntime_it(\"df.query('flag and value > 0')\",\n        lambda: df.query('flag and value > 0'))\n\nprint(\"\\nKey insight: always prefer vectorised ops over .apply() or .iterrows()\")\n```\n\n## Databases & SQL Quick Reference\n\n**Query optimisation checklist:**\n1. `EXPLAIN ANALYZE` \u2014 find seq scans and high cost nodes\n2. Add index on filter/join columns\n3. Rewrite correlated subqueries as JOINs or CTEs\n4. Use `LIMIT` + keyset pagination instead of `OFFSET`\n5. Vacuum/analyse for stale statistics\n\n**Index decision:**\n| Situation | Index type |\n|-----------|-----------|\n| Equality + range | B-Tree (default) |\n| Full-text search | GIN with `tsvector` |\n| JSONB keys | GIN |\n| Geospatial | GiST |\n| Sequential insert, range scan | BRIN |\n\n**Transaction isolation levels:**\n```sql\nSET TRANSACTION ISOLATION LEVEL READ COMMITTED;  -- default Postgres\n-- REPEATABLE READ  \u2192 no phantom reads for single txn\n-- SERIALIZABLE     \u2192 full isolation, slowest\n```\n\n## Summary & Key Takeaways\n\n- Cohort analysis separates lifecycle effects (decay over time) from calendar effects (product changes, seasonality).\n- Period-1 retention is the most actionable early signal \u2014 it measures whether users find core value.\n- A horizontal line in the heatmap at a late period indicates a loyal retained core \u2014 a valuable insight for LTV modelling.\n- LTV = ARPU / churn rate; improving early retention compounds over the entire customer lifetime.\n- Always check cohort sizes \u2014 small cohorts produce noisy curves that look like improvements or regressions.\n\n## \ud83e\udde0 Quick Quiz\n\n**Q1.** A cohort in cohort analysis is defined by:\n- A) Users with the same demographic\n- B) **Users who share a common characteristic at a specific time \u2014 typically acquisition date** \u2713\n- C) Users in the same geographic region\n- D) Users with similar behaviour\n\n**Q2.** Retention rate in cohort analysis measures:\n- A) The proportion of users who paid\n- B) **What fraction of a cohort returned in a subsequent time period** \u2713\n- C) Average revenue per user\n- D) Churn rate\n\n**Q3.** The SQL pattern for cohort analysis uses:\n- A) GROUP BY only\n- B) **Self-join or window functions to compare each user's activity to their cohort's first event** \u2713\n- C) PIVOT tables\n- D) Recursive CTEs\n\n<details><summary>Answers</summary>1-B, 2-B, 3-B</details>\n",
  "quiz": []
}