{
  "version": 1,
  "id": "window-functions-in-sql",
  "title": "Window Functions",
  "summary": "Ranking, running totals and lag/lead without collapsing rows: PARTITION BY, ORDER BY and frames.",
  "estimatedMinutes": 35,
  "prerequisites": [],
  "source": "courses-notebook DS301_databases_sql 09_window_functions",
  "quiz": [
    {
      "prompt": "What is the difference between `RANK()` and `DENSE_RANK()` on ties?",
      "options": [
        "RANK leaves gaps after ties; DENSE_RANK does not",
        "DENSE_RANK leaves gaps; RANK does not",
        "They are identical",
        "Only DENSE_RANK supports PARTITION BY"
      ],
      "answer": 0
    },
    {
      "prompt": "Which window function returns the value from the previous row by order?",
      "options": [
        "LEAD",
        "LAG",
        "NTILE",
        "ROW_NUMBER"
      ],
      "answer": 1
    },
    {
      "prompt": "`SUM(amount) OVER (PARTITION BY region ORDER BY date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)` computes:",
      "options": [
        "The grand total of amount",
        "A running total of amount per region ordered by date",
        "The maximum amount per region",
        "The average amount per region"
      ],
      "answer": 1
    }
  ],
  "lesson": "Row-level analytics: ranking, running totals, and lag/lead comparisons without collapsing rows.\n\n```python\nimport sqlite3, pandas as pd, numpy as np, matplotlib.pyplot as plt\nconn = sqlite3.connect(':memory:')\nprint('Window Functions')\n```\n\n## Key Concepts\n\n### Syntax\n\n```sql\nfunction_name() OVER (\n    [PARTITION BY column, ...]\n    [ORDER BY column]\n    [ROWS BETWEEN ... AND ...]\n)\n```\n\n**PARTITION BY** resets the window per group (like GROUP BY but keeps all rows).  \n**ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW** = running total.\n\n### Ranking Functions\n\n| Function | Behaviour |\n|---|---|\n| `ROW_NUMBER()` | Unique 1,2,3 even for ties |\n| `RANK()` | Ties share rank; skips next (1,1,3) |\n| `DENSE_RANK()` | Ties share rank; no skipping (1,1,2) |\n| `NTILE(n)` | Divide into n equal buckets |\n\n### Offset Functions\n\n- `LAG(col, n)` \u2014 value n rows before current row\n- `LEAD(col, n)` \u2014 value n rows after current row\n- `FIRST_VALUE(col)`, `LAST_VALUE(col)` \u2014 first/last in the frame\n\n### Aggregate Window Functions\n\nAny aggregate (`SUM`, `AVG`, `MIN`, `MAX`) works as a window function:\n\n```sql\nSUM(revenue) OVER (\n    PARTITION BY region\n    ORDER BY month\n    ROWS UNBOUNDED PRECEDING\n) AS running_total\n```\n\n### GROUP BY vs OVER\n\n`GROUP BY` collapses rows; `OVER` adds a column while keeping all rows.\n\n```python\nimport sqlite3, pandas as pd, numpy as np, matplotlib.pyplot as plt\n\nconn = sqlite3.connect(\":memory:\")\nconn.executescript(\"\"\"\nCREATE TABLE sales (month TEXT, region TEXT, rep TEXT, revenue REAL);\nINSERT INTO sales VALUES\n('2024-01','North','Alice',12000),('2024-01','North','Bob',9500),\n('2024-01','South','Carol',15000),('2024-01','South','Dave',11000),\n('2024-02','North','Alice',13500),('2024-02','North','Bob',9500),\n('2024-02','South','Carol',14000),('2024-02','South','Dave',12500),\n('2024-03','North','Alice',11000),('2024-03','North','Bob',10500),\n('2024-03','South','Carol',16000),('2024-03','South','Dave',13000),\n('2024-04','North','Alice',14500),('2024-04','North','Bob',8500),\n('2024-04','South','Carol',15500),('2024-04','South','Dave',14000);\n\"\"\")\n\nrank_df = pd.read_sql(\"\"\"\nSELECT month, region, rep, revenue,\n       RANK()       OVER (PARTITION BY month ORDER BY revenue DESC) AS rnk,\n       ROW_NUMBER() OVER (PARTITION BY month ORDER BY revenue DESC) AS row_num\nFROM sales ORDER BY month, rnk\n\"\"\", conn)\nprint(rank_df.head(8).to_string(index=False))\n\nrunning_df = pd.read_sql(\"\"\"\nSELECT month, region,\n       SUM(revenue) AS monthly_rev,\n       SUM(SUM(revenue)) OVER (PARTITION BY region ORDER BY month\n                               ROWS UNBOUNDED PRECEDING) AS running_total,\n       LAG(SUM(revenue)) OVER (PARTITION BY region ORDER BY month) AS prev_month\nFROM sales GROUP BY month, region ORDER BY region, month\n\"\"\", conn)\nprint(running_df.to_string(index=False))\n\nfig, axes = plt.subplots(1,2, figsize=(12,4))\nfor region, grp in running_df.groupby(\"region\"):\n    axes[0].plot(grp[\"month\"], grp[\"running_total\"]/1000, marker=\"o\", label=region, lw=2)\naxes[0].set_title(\"Cumulative Revenue by Region\"); axes[0].set_ylabel(\"$k\")\naxes[0].legend(); axes[0].grid(alpha=0.3); axes[0].tick_params(axis=\"x\",rotation=30)\n\ntop1 = rank_df[rank_df[\"rnk\"]==1].groupby(\"rep\").size() / rank_df[\"month\"].nunique()\naxes[1].bar(top1.index, top1.values*100, color=\"#3498db\", alpha=0.8)\naxes[1].set_title(\"% of Months as Top Earner\"); axes[1].set_ylabel(\"%\")\naxes[1].grid(axis=\"y\", alpha=0.3); axes[1].spines[\"top\"].set_visible(False); axes[1].spines[\"right\"].set_visible(False)\nplt.tight_layout(); plt.show()\nconn.close()\n```\n\n```python\n# Data analysis solution checker\n\nimport pandas as pd\nimport numpy as np\n\ndef summarise_df(df, label=\"DataFrame\"):\n    \"\"\"Quick sanity check for a cleaned dataframe.\"\"\"\n    print(f\"=== {label} ===\")\n    print(f\"  Shape: {df.shape}\")\n    print(f\"  Missing: {df.isnull().sum().sum()} values ({df.isnull().mean().mean():.1%})\")\n    print(f\"  Dtypes: {df.dtypes.value_counts().to_dict()}\")\n    if df.select_dtypes('number').shape[1] > 0:\n        num = df.select_dtypes('number')\n        print(f\"  Numeric range: [{num.values.min():.2f}, {num.values.max():.2f}]\")\n    dupes = df.duplicated().sum()\n    print(f\"  Duplicates: {dupes}\")\n\ndef check_clean(df):\n    \"\"\"Assert DataFrame is clean: no NaN, no dupes, correct dtypes.\"\"\"\n    issues = []\n    if df.isnull().any().any():\n        issues.append(f\"{df.isnull().sum().sum()} missing values remain\")\n    if df.duplicated().any():\n        issues.append(f\"{df.duplicated().sum()} duplicate rows remain\")\n    if issues:\n        print(\"\u2717 ISSUES: \" + \"; \".join(issues))\n    else:\n        print(\"\u2713 DataFrame is clean\")\n\n# Demo\nnp.random.seed(0)\ndf = pd.DataFrame({\n    'age':    [25, 30, np.nan, 22, 35, 30],\n    'salary': [50000, 60000, 75000, 45000, None, 60000],\n    'dept':   ['Eng', 'HR', 'Eng', 'Sales', 'HR', 'HR'],\n})\nsummarise_df(df, \"Raw\")\ndf_clean = df.dropna().drop_duplicates()\nsummarise_df(df_clean, \"Cleaned\")\ncheck_clean(df_clean)\n```\n\n```python\n# Advanced: Pandas performance patterns\n\nimport pandas as pd\nimport numpy as np\nimport timeit\n\nnp.random.seed(42)\nn = 100_000\ndf = pd.DataFrame({\n    'group':   np.random.choice(list('ABCDE'), n),\n    'value':   np.random.randn(n),\n    'flag':    np.random.randint(0, 2, n).astype(bool),\n    'amount':  np.random.exponential(100, n),\n})\n\ndef time_it(label, fn):\n    t = timeit.timeit(fn, number=10) / 10 * 1000\n    print(f\"  {label:<40}: {t:.2f}ms\")\n\nprint(\"Pandas performance patterns:\")\n\n# 1. apply vs vectorised operation\ntime_it(\"df['value'].apply(lambda x: x**2)\", lambda: df['value'].apply(lambda x: x**2))\ntime_it(\"df['value'] ** 2  (vectorised)\", lambda: df['value'] ** 2)\n\n# 2. iterrows vs vectorised groupby\ntime_it(\"groupby().mean()\", lambda: df.groupby('group')['amount'].mean())\ntime_it(\"groupby().agg({'amount': ['mean', 'std', 'count']})\",\n        lambda: df.groupby('group')['amount'].agg(['mean', 'std', 'count']))\n\n# 3. Boolean indexing vs query\ntime_it(\"df[df.flag & (df.value > 0)]\",\n        lambda: df[df['flag'] & (df['value'] > 0)])\ntime_it(\"df.query('flag and value > 0')\",\n        lambda: df.query('flag and value > 0'))\n\nprint(\"\\nKey insight: always prefer vectorised ops over .apply() or .iterrows()\")\n```\n\n## Databases & SQL Quick Reference\n\n**Query optimisation checklist:**\n1. `EXPLAIN ANALYZE` \u2014 find seq scans and high cost nodes\n2. Add index on filter/join columns\n3. Rewrite correlated subqueries as JOINs or CTEs\n4. Use `LIMIT` + keyset pagination instead of `OFFSET`\n5. Vacuum/analyse for stale statistics\n\n**Index decision:**\n| Situation | Index type |\n|-----------|-----------|\n| Equality + range | B-Tree (default) |\n| Full-text search | GIN with `tsvector` |\n| JSONB keys | GIN |\n| Geospatial | GiST |\n| Sequential insert, range scan | BRIN |\n\n**Transaction isolation levels:**\n```sql\nSET TRANSACTION ISOLATION LEVEL READ COMMITTED;  -- default Postgres\n-- REPEATABLE READ  \u2192 no phantom reads for single txn\n-- SERIALIZABLE     \u2192 full isolation, slowest\n```\n\n## Summary & Key Takeaways\n\n- Window functions add a computed column without reducing row count \u2014 unlike GROUP BY.\n- PARTITION BY resets the window per group; omitting it uses the entire table as one window.\n- LAG/LEAD enable period-over-period comparisons in a single query \u2014 no self-join needed.\n- Running totals use `SUM(...) OVER (... ROWS UNBOUNDED PRECEDING)`.\n- Choose ROW_NUMBER for unique rankings, RANK when ties should share a rank with gaps, DENSE_RANK with no gaps.\n\n## \ud83e\udde0 Quick Quiz\n\n**Q1.** Window functions differ from GROUP BY because:\n- A) They are faster\n- B) They only work with aggregate functions\n- C) **They do not collapse rows \u2014 each row keeps its own result** \u2713\n- D) They require an ORDER BY clause\n\n**Q2.** `ROW_NUMBER() OVER (PARTITION BY dept ORDER BY salary DESC)` gives:\n- A) Salary rank across all employees\n- B) **Row rank within each department, ordered by salary descending** \u2713\n- C) Running total of salaries per department\n- D) Average salary per department\n\n**Q3.** `LAG(value, 1)` returns:\n- A) The next row's value\n- B) **The previous row's value** \u2713\n- C) The first row's value in the partition\n- D) The running minimum\n\n<details><summary>Answers</summary>1-C, 2-B, 3-B</details>\n"
}