{
  "version": 1,
  "id": "structured-logging",
  "title": "Structured Logging",
  "summary": "Why production logs should be structured, how severities and correlation ids work, and what to log.",
  "estimatedMinutes": 20,
  "prerequisites": [],
  "source": "courses-notebook SE303_observability_sre 03_structured_logging",
  "quiz": [
    {
      "prompt": "Which is the correct ascending order of standard log severities?",
      "options": [
        "DEBUG < INFO < WARNING < ERROR < CRITICAL",
        "INFO < DEBUG < WARNING < ERROR < CRITICAL",
        "DEBUG < INFO < ERROR < WARNING < CRITICAL",
        "WARNING < DEBUG < INFO < ERROR < CRITICAL"
      ],
      "answer": 0
    },
    {
      "prompt": "Why is structured (JSON) logging preferred over plain string logs in production?",
      "options": [
        "It can be parsed, filtered, and aggregated reliably across many services",
        "It is shorter on disk",
        "It bypasses log levels",
        "It removes the need for traces"
      ],
      "answer": 0
    }
  ],
  "lesson": "Topics: structured vs unstructured logs \u00b7 JSON logging \u00b7 correlation IDs \u00b7 log levels \u00b7 aggregation patterns\n\n## 1. Unstructured vs Structured Logs\n\n**Unstructured** (plain text):\n```\n2024-01-15 10:23:41 ERROR Failed to connect to DB: timeout after 5s\n```\nHard to query at scale \u2014 requires fragile regex parsing.\n\n**Structured** (JSON):\n```json\n{\"ts\":\"2024-01-15T10:23:41Z\",\"level\":\"error\",\"msg\":\"db_connection_failed\",\n \"error\":\"timeout\",\"duration_ms\":5000,\"service\":\"checkout\"}\n```\nMachine-readable, filterable by any field, aggregatable across services.\n\nModern log platforms (Loki, Elasticsearch, Datadog) index structured fields natively.\n\n```python\nimport json\nimport sys\nimport uuid\nimport datetime\nfrom typing import Any, Dict, List, Optional\n\n\nclass StructuredLogger:\n    \"\"\"Minimal structured logger emitting JSON to stdout.\"\"\"\n\n    LEVELS = {\"debug\": 10, \"info\": 20, \"warning\": 30, \"error\": 40, \"critical\": 50}\n\n    def __init__(self, service: str, min_level: str = \"info\"):\n        self.service   = service\n        self.min_level = self.LEVELS[min_level]\n        self._context: Dict[str, Any] = {}\n\n    def with_context(self, **kwargs: Any) -> \"StructuredLogger\":\n        \"\"\"Return a child logger with additional bound fields.\"\"\"\n        child = StructuredLogger(self.service)\n        child.min_level = self.min_level\n        child._context  = {**self._context, **kwargs}\n        return child\n\n    def _emit(self, level: str, msg: str, **kwargs: Any) -> None:\n        if self.LEVELS[level] < self.min_level:\n            return\n        record = {\n            \"ts\":      datetime.datetime.utcnow().isoformat() + \"Z\",\n            \"level\":   level,\n            \"service\": self.service,\n            \"msg\":     msg,\n            **self._context,\n            **kwargs,\n        }\n        print(json.dumps(record))\n\n    def debug(self,    msg: str, **kw) -> None: self._emit(\"debug\",    msg, **kw)\n    def info(self,     msg: str, **kw) -> None: self._emit(\"info\",     msg, **kw)\n    def warning(self,  msg: str, **kw) -> None: self._emit(\"warning\",  msg, **kw)\n    def error(self,    msg: str, **kw) -> None: self._emit(\"error\",    msg, **kw)\n    def critical(self, msg: str, **kw) -> None: self._emit(\"critical\", msg, **kw)\n\n\nlog = StructuredLogger(\"order-service\")\nlog.info(\"server_started\", port=8080)\nlog.warning(\"high_memory_usage\", rss_mb=780, threshold_mb=800)\nlog.error(\"payment_failed\", order_id=\"ord-123\", error=\"gateway_timeout\", duration_ms=3001)\n```\n\n## 2. Log Levels\n\n| Level | Numeric | When to use |\n|-------|---------|-------------|\n| DEBUG | 10 | Detailed diagnostics \u2014 disabled in production |\n| INFO | 20 | Normal lifecycle events (startup, requests) |\n| WARNING | 30 | Unexpected but handled (retry, fallback) |\n| ERROR | 40 | Failure requiring attention (but service still running) |\n| CRITICAL | 50 | System-level failure, imminent crash |\n\n**Anti-patterns**:\n- Log at ERROR for every 4xx (those are client errors, not service errors)\n- Log DEBUG in production \u2014 massive volume, performance impact\n- Inconsistent level semantics across services\n\n```python\n# Demonstrate level filtering\nprod_log = StructuredLogger(\"api-gateway\", min_level=\"warning\")\n\nprod_log.debug(\"request_parsed\", path=\"/health\")   # suppressed\nprod_log.info(\"request_handled\", status=200)        # suppressed\nprod_log.warning(\"rate_limit_approaching\", rps=950, limit=1000)\nprod_log.error(\"upstream_unavailable\", upstream=\"inventory-service\")\n```\n\n## 3. Correlation IDs\n\nIn distributed systems, a single user action may span many services.  \nA **correlation ID** (also called trace ID or request ID) ties all log lines from one request together.\n\nConvention:\n1. Edge service generates a UUID on entry (`X-Request-ID` header)\n2. Every downstream service propagates it in both HTTP headers and log fields\n3. Query `request_id = \"abc-123\"` in your log platform to see the full chain\n\n```python\nimport uuid\n\n\ndef handle_request(request_id: Optional[str] = None) -> None:\n    \"\"\"Simulate multi-service request flow with correlation ID.\"\"\"\n    rid = request_id or str(uuid.uuid4())\n\n    # Each service uses a child logger bound to this request\n    gateway_log    = StructuredLogger(\"api-gateway\").with_context(request_id=rid)\n    order_log      = StructuredLogger(\"order-svc\").with_context(request_id=rid)\n    inventory_log  = StructuredLogger(\"inventory-svc\").with_context(request_id=rid)\n\n    gateway_log.info(\"request_received\",   method=\"POST\", path=\"/orders\")\n    order_log.info(\"order_created\",        order_id=\"ord-999\", user_id=\"u-42\")\n    inventory_log.info(\"stock_reserved\",   sku=\"SKU-007\", qty=1)\n    order_log.info(\"payment_initiated\",    gateway=\"stripe\")\n    gateway_log.info(\"response_sent\",      status=201, duration_ms=142)\n\n\nprint(\"=== Correlated request log ===\")\nhandle_request(request_id=\"req-demo-abc123\")\n```\n\n## 4. Log Aggregation Patterns\n\nCommon architectures for collecting logs at scale:\n\n```\nService \u2192 Sidecar agent (Fluentbit/Vector)\n       \u2192 Message bus (Kafka / Cloud Pub/Sub)\n       \u2192 Storage + Index (Elasticsearch / Loki)\n       \u2192 Query UI (Grafana / Kibana)\n```\n\nKey concerns:\n- **Buffering** \u2014 handle log bursts without dropping events\n- **Backpressure** \u2014 slow consumers should not crash producers\n- **Sampling** \u2014 at very high volume, sample DEBUG/INFO (never ERROR)\n- **Retention** \u2014 short for debug logs, long for audit/compliance\n\n```python\n# Simple in-memory log buffer with sampling\nfrom collections import deque\nimport random as _random\n\n\nclass SamplingLogBuffer:\n    \"\"\"Buffer that samples low-level logs, always keeps errors.\"\"\"\n\n    def __init__(self, max_size: int = 1000, sample_rate: float = 0.1):\n        self.max_size    = max_size\n        self.sample_rate = sample_rate\n        self._buffer: deque = deque(maxlen=max_size)\n        self.stats = {\"accepted\": 0, \"sampled_out\": 0}\n\n    def write(self, record: Dict[str, Any]) -> None:\n        level = record.get(\"level\", \"info\")\n        # Always accept warnings and above\n        if level in (\"warning\", \"error\", \"critical\"):\n            self._buffer.append(record)\n            self.stats[\"accepted\"] += 1\n        elif _random.random() < self.sample_rate:\n            record = {**record, \"sampled\": True}\n            self._buffer.append(record)\n            self.stats[\"accepted\"] += 1\n        else:\n            self.stats[\"sampled_out\"] += 1\n\n    def flush(self) -> List[Dict]:\n        records = list(self._buffer)\n        self._buffer.clear()\n        return records\n\n\nbuf = SamplingLogBuffer(sample_rate=0.1)\n_random.seed(1)\n\nfor i in range(1000):\n    level = _random.choice([\"debug\", \"info\", \"info\", \"warning\", \"error\"])\n    buf.write({\"level\": level, \"msg\": f\"event_{i}\"})\n\nprint(f\"Accepted  : {buf.stats['accepted']}\")\nprint(f\"Sampled out: {buf.stats['sampled_out']}\")\nprint(f\"Buffer size: {len(buf._buffer)}\")\n```\n\n## Key Takeaways\n\n- Structured (JSON) logs are **machine-readable** \u2014 prefer them always\n- Use **correlation IDs** to link log lines across services for a single request\n- Set **log levels** deliberately \u2014 ERROR logs should wake someone up\n- At scale, **sample** high-volume low-severity logs; never sample errors\n- Include actionable fields: `request_id`, `user_id`, `duration_ms`, `error`\n\n## \ud83e\udde0 Quick Quiz\n\n**Q1.** Structured logs are preferable to plain text because:\n- A) They are shorter\n- B) **They are machine-parseable, enabling filtering, aggregation, and alerting** \u2713\n- C) They contain more information\n- D) They are faster to write\n\n**Q2.** A correlation ID in logs:\n- A) Identifies the log level\n- B) **Links all log entries across services for a single request** \u2713\n- C) Measures request latency\n- D) Identifies the logging framework\n\n**Q3.** Log sampling (not logging every event) is used to:\n- A) Improve log quality\n- B) Ensure all events are captured\n- C) **Reduce storage costs and noise at high traffic volumes** \u2713\n- D) Speed up the application\n\n<details><summary>Answers</summary>1-B, 2-B, 3-C</details>\n"
}