{
  "version": 1,
  "id": "sql-time-series",
  "title": "Time-Based Analysis in SQL",
  "summary": "Date bucketing, moving averages and period-over-period comparisons.",
  "estimatedMinutes": 40,
  "prerequisites": [],
  "source": "courses-notebook DS301_databases_sql 12_time_series_sql",
  "lesson": "Period-over-period comparisons, rolling averages, and gap filling.\n\n```python\nimport sqlite3, pandas as pd, numpy as np, matplotlib.pyplot as plt\nprint('Time-Series SQL')\n```\n\n## Key Concepts\n\n### Date Truncation\n\n```sql\nstrftime('%Y-%m', date)  -- SQLite: truncate to month\nstrftime('%Y-W%W', date) -- SQLite: ISO week\nDATE_TRUNC('month', ts)  -- PostgreSQL/BigQuery\n```\n\n### Period-over-Period Comparison\n\n```sql\nSELECT month, revenue,\n       LAG(revenue,  1) OVER (ORDER BY month) AS prev_month,\n       LAG(revenue, 12) OVER (ORDER BY month) AS prev_year,\n       ROUND(100.0*(revenue - LAG(revenue,12) OVER(ORDER BY month))\n             / LAG(revenue,12) OVER(ORDER BY month),1) AS yoy_pct\nFROM monthly_revenue\n```\n\n### Rolling Average\n\n```sql\nAVG(revenue) OVER (\n    ORDER BY month\n    ROWS BETWEEN 2 PRECEDING AND CURRENT ROW\n) AS rolling_3m\n```\n\n### Gap Filling\n\nSQL only returns rows that exist. Fill missing dates with a calendar CTE + LEFT JOIN:\n\n```sql\nWITH calendar(date) AS (\n    SELECT '2024-01-01'\n    UNION ALL SELECT date(date, '+1 day') FROM calendar WHERE date < '2024-12-31'\n)\nSELECT c.date, COALESCE(d.revenue, 0)\nFROM calendar c LEFT JOIN daily_rev d ON c.date = d.date\n```\n\n### Time Zone Best Practice\n\nStore all timestamps in **UTC**; convert to local time only in the final SELECT display.\n\n```python\nimport sqlite3, pandas as pd, numpy as np, matplotlib.pyplot as plt\n\nrng = np.random.default_rng(42)\nconn = sqlite3.connect(\":memory:\")\ndates = pd.date_range(\"2022-01-01\",\"2024-03-31\",freq=\"D\")\ntrend = np.linspace(1000,1800,len(dates))\nseasonal = 200*np.sin(2*np.pi*np.arange(len(dates))/365.25)\nrev = trend + seasonal + rng.normal(0,80,len(dates))\nconn.execute(\"CREATE TABLE daily_rev (date TEXT, revenue REAL)\")\nconn.executemany(\"INSERT INTO daily_rev VALUES (?,?)\",\n                 [(d.strftime(\"%Y-%m-%d\"),max(0,r)) for d,r in zip(dates,rev)])\nconn.commit()\n\nanalysis = pd.read_sql(\"\"\"\nWITH monthly AS (\n    SELECT strftime('%Y-%m', date) AS month, SUM(revenue) AS rev\n    FROM daily_rev GROUP BY month\n)\nSELECT month, rev,\n       LAG(rev,12) OVER (ORDER BY month) AS prev_year,\n       AVG(rev)    OVER (ORDER BY month ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS rolling_3m,\n       ROUND(100.0*(rev - LAG(rev,12) OVER(ORDER BY month))\n             / LAG(rev,12) OVER(ORDER BY month),1) AS yoy_pct\nFROM monthly\n\"\"\", conn)\n\nfig, axes = plt.subplots(2,2,figsize=(13,8))\nfig.suptitle(\"Time-Based SQL Analysis\",fontsize=13)\n\nax=axes[0,0]\nax.plot(analysis[\"month\"], analysis[\"rev\"]/1000, alpha=0.4, color=\"#aaaaaa\", label=\"Monthly\")\nax.plot(analysis[\"month\"], analysis[\"rolling_3m\"]/1000, color=\"#3498db\", lw=2.5, label=\"3m rolling\")\nax.set_title(\"Monthly Revenue + 3m Rolling Avg\"); ax.set_ylabel(\"$k\"); ax.legend()\nax.grid(alpha=0.2); ax.tick_params(axis=\"x\",rotation=45); ax.set_xticks(ax.get_xticks()[::4])\n\nyoy = analysis.dropna(subset=[\"yoy_pct\"])\nax=axes[0,1]\ncolors = [\"#2ecc71\" if v>=0 else \"#e74c3c\" for v in yoy[\"yoy_pct\"]]\nax.bar(yoy[\"month\"], yoy[\"yoy_pct\"], color=colors, alpha=0.8)\nax.axhline(0,color=\"black\",lw=0.8)\nax.set_title(\"Year-over-Year Growth %\"); ax.set_ylabel(\"YoY %\")\nax.grid(axis=\"y\",alpha=0.2); ax.tick_params(axis=\"x\",rotation=45); ax.set_xticks(ax.get_xticks()[::3])\n\ngap = pd.read_sql(\"\"\"\nWITH calendar(date) AS (\n    SELECT '2024-01-01'\n    UNION ALL SELECT date(date,'+1 day') FROM calendar WHERE date < '2024-01-31'\n)\nSELECT c.date, COALESCE(d.revenue,0) AS revenue\nFROM calendar c LEFT JOIN daily_rev d ON c.date=d.date ORDER BY c.date\n\"\"\", conn)\nax=axes[1,0]\nax.fill_between(gap[\"date\"], gap[\"revenue\"]/1000, alpha=0.4, color=\"steelblue\")\nax.plot(gap[\"date\"], gap[\"revenue\"]/1000, color=\"steelblue\", lw=1.5)\nax.set_title(\"Gap Filling: Calendar LEFT JOIN (Jan 2024)\")\nax.set_ylabel(\"$k\"); ax.tick_params(axis=\"x\",rotation=45); ax.set_xticks(ax.get_xticks()[::5])\nax.grid(alpha=0.2)\n\nmom_pct = analysis[\"rev\"].pct_change()*100\nax=axes[1,1]\nax.hist(mom_pct.dropna(), bins=20, color=\"#9b59b6\", alpha=0.8, edgecolor=\"white\")\nax.axvline(mom_pct.mean(), color=\"red\", ls=\"--\", lw=2, label=f\"Mean: {mom_pct.mean():.1f}%\")\nax.set_title(\"MoM Growth Distribution\"); ax.set_xlabel(\"Growth %\"); ax.legend(); ax.grid(alpha=0.2)\nplt.tight_layout(); plt.show()\nconn.close()\n```\n\n```python\n# Data analysis solution checker\n\nimport pandas as pd\nimport numpy as np\n\ndef summarise_df(df, label=\"DataFrame\"):\n    \"\"\"Quick sanity check for a cleaned dataframe.\"\"\"\n    print(f\"=== {label} ===\")\n    print(f\"  Shape: {df.shape}\")\n    print(f\"  Missing: {df.isnull().sum().sum()} values ({df.isnull().mean().mean():.1%})\")\n    print(f\"  Dtypes: {df.dtypes.value_counts().to_dict()}\")\n    if df.select_dtypes('number').shape[1] > 0:\n        num = df.select_dtypes('number')\n        print(f\"  Numeric range: [{num.values.min():.2f}, {num.values.max():.2f}]\")\n    dupes = df.duplicated().sum()\n    print(f\"  Duplicates: {dupes}\")\n\ndef check_clean(df):\n    \"\"\"Assert DataFrame is clean: no NaN, no dupes, correct dtypes.\"\"\"\n    issues = []\n    if df.isnull().any().any():\n        issues.append(f\"{df.isnull().sum().sum()} missing values remain\")\n    if df.duplicated().any():\n        issues.append(f\"{df.duplicated().sum()} duplicate rows remain\")\n    if issues:\n        print(\"\u2717 ISSUES: \" + \"; \".join(issues))\n    else:\n        print(\"\u2713 DataFrame is clean\")\n\n# Demo\nnp.random.seed(0)\ndf = pd.DataFrame({\n    'age':    [25, 30, np.nan, 22, 35, 30],\n    'salary': [50000, 60000, 75000, 45000, None, 60000],\n    'dept':   ['Eng', 'HR', 'Eng', 'Sales', 'HR', 'HR'],\n})\nsummarise_df(df, \"Raw\")\ndf_clean = df.dropna().drop_duplicates()\nsummarise_df(df_clean, \"Cleaned\")\ncheck_clean(df_clean)\n```\n\n```python\n# Advanced: Pandas performance patterns\n\nimport pandas as pd\nimport numpy as np\nimport timeit\n\nnp.random.seed(42)\nn = 100_000\ndf = pd.DataFrame({\n    'group':   np.random.choice(list('ABCDE'), n),\n    'value':   np.random.randn(n),\n    'flag':    np.random.randint(0, 2, n).astype(bool),\n    'amount':  np.random.exponential(100, n),\n})\n\ndef time_it(label, fn):\n    t = timeit.timeit(fn, number=10) / 10 * 1000\n    print(f\"  {label:<40}: {t:.2f}ms\")\n\nprint(\"Pandas performance patterns:\")\n\n# 1. apply vs vectorised operation\ntime_it(\"df['value'].apply(lambda x: x**2)\", lambda: df['value'].apply(lambda x: x**2))\ntime_it(\"df['value'] ** 2  (vectorised)\", lambda: df['value'] ** 2)\n\n# 2. iterrows vs vectorised groupby\ntime_it(\"groupby().mean()\", lambda: df.groupby('group')['amount'].mean())\ntime_it(\"groupby().agg({'amount': ['mean', 'std', 'count']})\",\n        lambda: df.groupby('group')['amount'].agg(['mean', 'std', 'count']))\n\n# 3. Boolean indexing vs query\ntime_it(\"df[df.flag & (df.value > 0)]\",\n        lambda: df[df['flag'] & (df['value'] > 0)])\ntime_it(\"df.query('flag and value > 0')\",\n        lambda: df.query('flag and value > 0'))\n\nprint(\"\\nKey insight: always prefer vectorised ops over .apply() or .iterrows()\")\n```\n\n## SQL Quick Reference\n\n```sql\n-- Window functions\nSELECT name, salary,\n  RANK() OVER (PARTITION BY dept ORDER BY salary DESC) AS rank,\n  AVG(salary) OVER (PARTITION BY dept) AS dept_avg\nFROM employees;\n\n-- CTEs\nWITH ranked AS (\n  SELECT *, ROW_NUMBER() OVER (ORDER BY created_at) AS rn FROM orders\n)\nSELECT * FROM ranked WHERE rn <= 10;\n```\n\n**Index types:**\n| Type | Best for |\n|------|----------|\n| B-Tree | Equality + range queries |\n| Hash | Exact equality only |\n| GIN | Full-text, array, JSONB |\n| BRIN | Naturally ordered large tables |\n\n**Query plan reading:** `EXPLAIN ANALYZE SELECT ...`\n\n## Summary & Key Takeaways\n\n- strftime (SQLite) and DATE_TRUNC (PostgreSQL) bucket timestamps into time periods; always be explicit about granularity.\n- YoY comparison (LAG 12 months) removes seasonal effects; MoM is noisier but more timely.\n- Rolling averages require an explicit ROWS BETWEEN frame \u2014 without it, behaviour is undefined or database-specific.\n- Gap filling requires a calendar CTE + LEFT JOIN; omitting it hides zero-revenue periods from trend analysis.\n- Always store timestamps in UTC and convert to local time only at display layer.\n\n## \ud83e\udde0 Quick Quiz\n\n**Q1.** A date spine (or calendar table) is used in time series SQL to:\n- A) Store timezone conversions\n- B) **Ensure every date appears in results even when no data exists for that date** \u2713\n- C) Partition tables by month\n- D) Calculate date differences\n\n**Q2.** Rolling 7-day average in SQL uses:\n- A) GROUP BY with 7 rows\n- B) **`AVG() OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW)`** \u2713\n- C) A subquery per date\n- D) CASE WHEN with date arithmetic\n\n**Q3.** TimescaleDB extends PostgreSQL for time series by:\n- A) Adding OLAP features\n- B) **Automatically partitioning time-series tables into time-based \"chunks\"** \u2713\n- C) Providing a columnar storage engine\n- D) Adding streaming ingestion\n\n<details><summary>Answers</summary>1-B, 2-B, 3-B</details>\n",
  "quiz": []
}