{
  "version": 1,
  "id": "sql-joins-and-subqueries",
  "title": "Joins and Subqueries",
  "summary": "Inner, outer and cross joins, correlated subqueries and when each is the right tool.",
  "estimatedMinutes": 25,
  "prerequisites": [],
  "source": "courses-notebook DS301_databases_sql 03_joins_subqueries",
  "quiz": [
    {
      "prompt": "Which JOIN returns only rows with a matching key in both tables?",
      "options": [
        "LEFT JOIN",
        "INNER JOIN",
        "FULL OUTER JOIN",
        "CROSS JOIN"
      ],
      "answer": 1
    }
  ],
  "lesson": "## SQL joins subqueries filtering (Vault Research)\n\n**Insight 1:**\n> Byte Pair Encoding (BPE) tokenizers eliminate the out-of-vocabulary problem by recursively merging the most frequent character pairs in a corpus until\n\nSource: *Build a Large Language Model (From Scratch)* (Chunk 3167)\n\n**Insight 2:**\n> The size-overlap trade-off has a sweet spot: small chunks (200\u2013500 chars) improve precision but lose context; large chunks (1k\u20132k) carry context but d\n\nSource: *30 Agents Every AI Engineer Must Build - 1 Edition - Transform LLMs into autonomous decision-making vertical agents in healthcare, finance, and beyond - 30 Agents Every AI Engineer Must Build (for )* (Chunk 3629)\n\n**Insight 3:**\n> In LangGraph supervisor architectures, all subagent messages are typically appended to a shared message list so subagents see each other's work; alter\n\nSource: *Learning LangChain* (Chunk 6950)\n\n```python\nimport sqlite3\n\nconn = sqlite3.connect(':memory:')\nc = conn.cursor()\nc.executescript('''\nCREATE TABLE customers (id INTEGER PRIMARY KEY, name TEXT, city TEXT);\nCREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER, amount REAL, date TEXT);\nINSERT INTO customers VALUES (1,'Alice','NYC'),(2,'Bob','LA'),(3,'Charlie','NYC'),(4,'Diana','Chicago');\nINSERT INTO orders VALUES (1,1,150,'2024-01-15'),(2,1,200,'2024-02-20'),\n(3,2,75,'2024-01-10'),(4,3,300,'2024-03-05'),(5,1,125,'2024-03-15');\n''')\n\njoin_queries = [\n    ('INNER JOIN', '''SELECT c.name, o.amount, o.date\nFROM customers c INNER JOIN orders o ON c.id = o.customer_id'''),\n    ('LEFT JOIN (includes Diana with no orders)',\n     '''SELECT c.name, COALESCE(SUM(o.amount), 0) as total\nFROM customers c LEFT JOIN orders o ON c.id = o.customer_id\nGROUP BY c.name'''),\n    ('Subquery: customers with above-avg orders',\n     '''SELECT name FROM customers WHERE id IN\n(SELECT customer_id FROM orders WHERE amount > (SELECT AVG(amount) FROM orders))'''),\n    ('CTE: customer order summary',\n     '''WITH order_summary AS (\n    SELECT customer_id, COUNT(*) as n_orders, SUM(amount) as total\n    FROM orders GROUP BY customer_id\n)\nSELECT c.name, os.n_orders, os.total\nFROM customers c JOIN order_summary os ON c.id = os.customer_id\nORDER BY os.total DESC'''),\n]\n\nfor title, sql in join_queries:\n    print(f'\\n--- {title} ---')\n    for row in c.execute(sql).fetchall():\n        print(f'  {row}')\nconn.close()\n```\n\n## Exercises\n\n### Exercise 1 (\u2b50): Practice the queries from this chapter.\n### Exercise 2 (\u2b50\u2b50): Write more complex queries.\n---\n*Next chapter*\n\n## JOIN Types\n\n```sql\nINNER JOIN  -- only matching rows from both tables\nLEFT JOIN   -- all from left, NULLs where no right match\nRIGHT JOIN  -- all from right, NULLs where no left match\nFULL JOIN   -- all rows from both, NULLs for non-matches\nCROSS JOIN  -- cartesian product (every combination)\nSELF JOIN   -- join table to itself (hierarchy, adjacency)\n```\n\n```python\nimport sqlite3\n\nconn = sqlite3.connect(':memory:')\nconn.executescript(\"\"\"\n    CREATE TABLE emp (id INT, name TEXT, mgr_id INT, dept_id INT);\n    CREATE TABLE dept (id INT, name TEXT);\n    INSERT INTO dept VALUES (1,'Eng'),(2,'Mkt'),(3,'HR');\n    INSERT INTO emp VALUES\n        (1,'Alice',NULL,1),(2,'Bob',1,1),(3,'Charlie',1,2),\n        (4,'Diana',NULL,3),(5,'Eve',4,NULL);  -- Eve has no dept\n\"\"\")\n\n# INNER JOIN: only employees with a department\nprint('INNER JOIN (employees with depts):')\nfor r in conn.execute('SELECT e.name, d.name FROM emp e INNER JOIN dept d ON e.dept_id = d.id'):\n    print(f'  {r[0]:10s} -> {r[1]}')\n\n# LEFT JOIN: all employees, NULL if no dept\nprint('\\nLEFT JOIN (all employees):')\nfor r in conn.execute('SELECT e.name, d.name FROM emp e LEFT JOIN dept d ON e.dept_id = d.id'):\n    print(f'  {r[0]:10s} -> {r[1]}')\n\n# SELF JOIN: employee with their manager\nprint('\\nSELF JOIN (employee + manager):')\nfor r in conn.execute('SELECT e.name, m.name as manager FROM emp e LEFT JOIN emp m ON e.mgr_id = m.id'):\n    print(f'  {r[0]:10s} manager: {r[1]}')\n```\n\n```python\n# Subqueries: scalar, correlated, and EXISTS\n\nconn2 = sqlite3.connect(':memory:')\nconn2.executescript(\"\"\"\n    CREATE TABLE orders (id INT, customer_id INT, amount REAL, date TEXT);\n    INSERT INTO orders VALUES\n        (1,100,50.0,'2025-01'),(2,100,120.0,'2025-02'),\n        (3,200,30.0,'2025-01'),(4,300,200.0,'2025-02'),\n        (5,200,80.0,'2025-02');\n\"\"\")\n\n# Scalar subquery: orders above average\nprint('Orders above average amount:')\nfor r in conn2.execute('SELECT id, amount FROM orders WHERE amount > (SELECT AVG(amount) FROM orders)'):\n    print(f'  Order {r[0]}: ${r[1]}')\n\n# Correlated subquery: customers with more than 1 order\nprint('\\nCustomers with >1 order (correlated subquery):')\nq = '''\n    SELECT DISTINCT customer_id\n    FROM orders o1\n    WHERE (SELECT COUNT(*) FROM orders o2 WHERE o2.customer_id = o1.customer_id) > 1\n'''\nfor r in conn2.execute(q):\n    print(f'  customer_id: {r[0]}')\n\n# IN subquery vs JOIN (performance note)\nprint('\\nIN subquery vs JOIN:')\nq_in = 'SELECT id FROM orders WHERE customer_id IN (SELECT customer_id FROM orders WHERE date = \"2025-01\")'\nq_join = 'SELECT DISTINCT o2.id FROM orders o1 JOIN orders o2 ON o1.customer_id = o2.customer_id WHERE o1.date = \"2025-01\"'\nin_results = [r[0] for r in conn2.execute(q_in)]\njoin_results = sorted(set(r[0] for r in conn2.execute(q_join)))\nprint(f'  IN result:   {sorted(in_results)}')\nprint(f'  JOIN result: {join_results}')\n```\n\n## Further Reading\n\n**Books:**\n- *Python for Data Analysis* (McKinney) \u2014 pandas deep dive\n- *Storytelling with Data* (Nussbaumer Knaflic) \u2014 visualisation\n- *Designing Data-Intensive Applications* (Kleppmann) \u2014 systems\n\n**Resources:**\n- [Pandas docs](https://pandas.pydata.org/docs/) \u2014 API reference\n- [SQLZoo](https://sqlzoo.net) / [Mode Analytics SQL Tutorial](https://mode.com/sql-tutorial/)\n- [Kaggle Learn](https://kaggle.com/learn) \u2014 free data science courses\n\n## Performance: JOINs and Indexes\n\n| Operation | Without Index | With Index |\n|-----------|--------------|------------|\n| JOIN on foreign key | O(N\u00d7M) | O(N log M) |\n| WHERE on column | O(N) | O(log N) |\n| ORDER BY | O(N log N) | O(1) if indexed |\n\n```sql\nCREATE INDEX idx_emp_dept ON employees(dept_id);\nCREATE INDEX idx_orders_cust_date ON orders(customer_id, order_date);\n```\n\n**EXPLAIN QUERY PLAN** in SQLite or **EXPLAIN ANALYZE** in PostgreSQL shows if indexes are used.\n\n## Key Takeaways\n\n- **INNER JOIN**: Returns only rows where the join condition matches in both tables, discarding non-matching rows from either side.\n- **LEFT/RIGHT/FULL OUTER JOIN**: Preserves all rows from the left, right, or both tables respectively, filling NULLs where no match exists.\n- **Correlated Subqueries**: Subqueries that reference the outer query's columns and re-execute for each outer row \u2014 powerful but can be slow.\n- **EXISTS vs IN**: EXISTS short-circuits on first match and handles NULLs correctly; IN can be faster for small, static value sets.\n- **Index Utilization on JOINs**: Creating indexes on join columns (foreign keys) can reduce nested-loop join complexity from O(n\u00b2) to O(n log n).\n\n## \ud83e\udde0 Quick Quiz\n\n**Q1.** An INNER JOIN returns:\n- A) All rows from the left table\n- B) All rows from both tables\n- C) **Only rows with matching keys in both tables** \u2713\n- D) All rows from the right table\n\n**Q2.** A LEFT JOIN includes:\n- A) Only matched rows\n- B) **All rows from the left table, with NULLs for non-matching right rows** \u2713\n- C) All rows from the right table with NULLs for non-matching left rows\n- D) The Cartesian product\n\n**Q3.** A correlated subquery differs from a regular subquery because:\n- A) It is faster\n- B) It returns multiple rows\n- C) **It references the outer query and executes once per outer row** \u2713\n- D) It uses aggregate functions\n\n<details><summary>Answers</summary>1-C, 2-B, 3-C</details>\n"
}