# Deployment-Readiness — Extraction Plan - **Sub-criterion:** 1.3 (AI-only) - **Weight:** 12.5 pts (4 Code AI × 12.5 = 50 pts half) - **Contenders:** Filesystem probes + Hadolint + **observability markers** (structured-logging library presence, metrics/tracing library presence, `/health` endpoint detection) + LLM README/Dockerfile read (layered) - **Source quote:** _"Can this actually ship? Docker, config, health checks, or clear instructions to run it outside localhost."_ Bands are **absolute** — per-signal thresholds in the _Banding_ section. Gates this recipe owns: **`DEMO`** (`deployment_readiness ≤ 1` when all 4 run-section/deploy signals are false OR README claims a deploy artifact that doesn't exist in the tree) and **`OBSERVABILITY`** (caps `deployment_readiness ≤ 3` when structured-logging library is absent AND metrics/tracing library is absent). Structured-logging detection: presence of `structlog` / `winston` / `pino` / `log-slog` / `zap` in deps OR JSON log format detected in code. Metrics/tracing detection: OpenTelemetry SDK / Prometheus client / Datadog agent / statsd client / a custom `/metrics` endpoint. Gate fires only when **both** categories are absent — a repo with structured logging but no metrics still ships above 3. Raw dump: `raw/deployment-readiness-observability-markers.json` with fields `{logging_lib: ""|null, metrics_lib: ""|null, has_health_endpoint: bool, json_log_detected: bool}`. > Five signals, integrated by the scorer. Serverless-native repos (no Dockerfile) can reach band 5 via PaaS/managed-platform path — see bands. No weighted-math internal split — the LLM emits a single 0–5 band and cites the most load-bearing signal. --- ## Extraction recipe ### Signal 1 — Containerization (Docker OR devcontainer OR compose-with-image) ```bash # Presence ls /Dockerfile \ /docker-compose.yml \ /docker-compose.yaml \ /compose.yaml \ /compose.override.yml \ /.dockerignore \ /.devcontainer/devcontainer.json \ /devcontainer.json 2>/dev/null # Dockerfile features (case-insensitive — Docker accepts `FROM ... as builder` too) grep -liE '^HEALTHCHECK' /Dockerfile 2>/dev/null # healthcheck grep -liE '^FROM[^#]*\bAS\b' /Dockerfile 2>/dev/null # multi-stage grep -liE '^USER[[:space:]]' /Dockerfile 2>/dev/null # non-root # Compose with inline image (no build:) → legitimate containerization without Dockerfile grep -lE '^\s*image:' /docker-compose.y*ml /compose.yaml 2>/dev/null # Lint (single Go binary; JSON out) — only if Dockerfile exists if [ -f /Dockerfile ]; then hadolint --format json /Dockerfile > /tmp/hadolint.json fi ``` Capture: - `has_dockerfile`, `has_compose`, `has_dockerignore`, `has_devcontainer` - `compose_has_inline_image` (compose without Dockerfile is legitimate) - `docker_has_healthcheck`, `docker_multi_stage`, `docker_non_root` - `hadolint_findings` (total), `hadolint_by_severity` (`error`/`warning`/`info`/`style`) - `has_container_artifact` = `has_dockerfile OR has_devcontainer OR compose_has_inline_image` ### Signal 2 — Config hygiene ```bash # env-example presence ls /{.env.example,.env.sample,.env.template} 2>/dev/null # .env gitignored? grep -E '^\.env(\.|$)' /.gitignore 2>/dev/null # Any tracked .env file? (should be zero) find -maxdepth 3 -name '.env' \ -not -path '*/node_modules/*' -not -path '*/.venv/*' ``` Capture: `has_env_example`, `env_in_gitignore`, `tracked_env_files` (list; should be empty). ### Signal 3 — Deploy artifacts (GHA / PaaS / IaC) ```bash # GHA deploy workflows ls /.github/workflows/ 2>/dev/null \ | grep -iE 'deploy|release|publish|ship' # PaaS configs — broad 2026 landscape ls /{\ vercel.json,netlify.toml,\ fly.toml,\ railway.json,railway.toml,\ render.yaml,\ Procfile,app.json,app.yaml,apprunner.yaml,\ wrangler.toml,wrangler.jsonc,\ serverless.yml,template.yaml,cdk.json,sst.config.ts,Pulumi.yaml,\ modal.toml,\ .replit,replit.nix,\ nixpacks.toml,project.toml,\ supabase/config.toml,\ .do/app.yaml,\ koyeb.yaml,zeabur.json,\ dagger.json,porter.yaml\ } 2>/dev/null # Cloudflare Pages markers (no single file; check for any) ls /{_routes.json,_headers,_redirects} 2>/dev/null # IaC directories ls -d /{k8s,kubernetes,manifests,helm,terraform,pulumi,cdk} 2>/dev/null # Helm chart detection ls /Chart.yaml 2>/dev/null # Kustomize find -maxdepth 3 -name 'kustomization.yaml' 2>/dev/null ``` Capture: - `deploy_workflow_files` (list) - `paas_configs` (list of present files; extend the list as new deploy targets appear) - `iac_dirs` (list of present dirs) - `has_any_deploy_artifact` = bool (OR across above + Signal 1 containerization) ### Signal 4 — Health checks + localhost counter-signal ```bash # Health route definitions (broad regex, all stacks) grep -rnE '["'\''/](health(z)?|readyz|livez|liveness|readiness|ping|status)["'\''/\s(]' \ /src /app /internal 2>/dev/null \ | head -20 # K8s liveness/readiness probes (if iac_dirs includes k8s/) grep -rE 'livenessProbe|readinessProbe' /k8s /helm 2>/dev/null # Localhost hardcoding in non-test source (counter-signal) # Exclude lines that reference env vars (fine: `process.env.X || 'localhost:3000'` is a dev default, not hardcoding) grep -rnE 'localhost:[0-9]+|127\.0\.0\.1:[0-9]+|http://localhost' \ /src /app \ --include='*.py' --include='*.ts' --include='*.tsx' \ --include='*.js' --include='*.jsx' --include='*.go' \ --exclude-dir=node_modules --exclude-dir=__tests__ \ --exclude-dir=.next --exclude-dir=dist --exclude-dir=build \ 2>/dev/null \ | grep -vE 'process\.env|os\.environ|ENV\[|getenv|import\.meta\.env|viper\.Get|os\.Getenv' \ | wc -l ``` Capture: - `health_route_hits` (count; 0 = none) - `k8s_probes_present` (bool) - `localhost_hardcoded_count` (non-test source, env-var-filtered) ### Signal 5 — README read (mechanical + LLM two-signal) **Mechanical — README section heading presence:** ```bash grep -iE '^##+\s*(Deploy|Deployment|Production|Hosting|Running in production|Getting Started|Installation|Quickstart|Usage|How to run|Setup)' \ /README.md /readme.md 2>/dev/null ``` Capture: `readme_has_run_section` (bool — any matching heading present). **LLM — quality read:** Read and score: - **README.md** (or `readme.md` / `README.rst`) - **Dockerfile** (if present) Score each on: - Does the README teach a stranger to run the app **outside localhost**? (deploy target named, deploy command given, env vars documented?) - Is the Dockerfile production-shaped (multi-stage, slim base, pinned versions, non-root, HEALTHCHECK)? - Anti-signals: only `npm run dev` / `python app.py`; no env docs; localhost-only language. Capture: - `llm_readme_score` (0–5) - `llm_dockerfile_score` (0–5 or `null` if absent) - `llm_has_run_section` (bool) - `llm_confidence` (`high`/`medium`/`low`) - `llm_notes` (one sentence) ### Signal 6 — Observability markers (OBSERVABILITY gate trigger) Ships-to-prod requires signal — structured logs and metrics/traces. This signal looks for evidence that either is wired into the app. **Both** must be absent for the gate to fire; one present means the repo has the basics. ```bash # Structured-logging library — check deps (manifest) + imports (code) + JSON-log # format heuristic. Any one match lets the repo off the gate's logging arm. LOG_LIBS="structlog|loguru|winston|pino|log-slog|uber-go/zap|go.uber.org/zap" # Dependency declaration — one of: if [ -f /pyproject.toml ]; then grep -iE "$LOG_LIBS" /pyproject.toml 2>/dev/null fi if [ -f /requirements.txt ]; then grep -iE "$LOG_LIBS" /requirements.txt 2>/dev/null fi if [ -f /package.json ]; then jq -r '(.dependencies // {}) + (.devDependencies // {}) | keys[]' \ /package.json 2>/dev/null | grep -iE "$LOG_LIBS" fi if [ -f /go.mod ]; then grep -iE "$LOG_LIBS" /go.mod 2>/dev/null fi # Import sighting in source — covers vendored or pinned-but-renamed deps. grep -rnE "import.*\b($LOG_LIBS)\b|from\s+($LOG_LIBS)" \ /src /app /internal \ --include='*.py' --include='*.ts' --include='*.tsx' \ --include='*.js' --include='*.jsx' --include='*.go' \ --exclude-dir=node_modules --exclude-dir=.venv \ 2>/dev/null | head -20 # JSON log-format heuristic — bare `json.dumps(...)` inside what looks like a # log call. Weak signal; participates only as a fallback. Evaluator should # confirm with LLM when this is the only logging-arm match. grep -rnE '(logger|log)\.(info|warn|error|debug)\([^)]*json\.(dumps|stringify)' \ /src /app \ --include='*.py' --include='*.ts' --include='*.js' --include='*.go' \ --exclude-dir=node_modules 2>/dev/null | head -5 # Metrics / tracing library — OpenTelemetry SDK, Prometheus client, # Datadog agent, statsd client, or a custom /metrics endpoint. METRICS_LIBS="opentelemetry|otel-|prom-client|prometheus_client|prometheus-client|ddtrace|datadog|statsd|hot-shots" # Dependency + import sighting (same pattern as logging) if [ -f /pyproject.toml ]; then grep -iE "$METRICS_LIBS" /pyproject.toml 2>/dev/null fi if [ -f /package.json ]; then jq -r '(.dependencies // {}) + (.devDependencies // {}) | keys[]' \ /package.json 2>/dev/null | grep -iE "$METRICS_LIBS" fi if [ -f /go.mod ]; then grep -iE "$METRICS_LIBS" /go.mod 2>/dev/null fi grep -rnE "import.*\b($METRICS_LIBS)\b|from\s+($METRICS_LIBS)" \ /src /app /internal \ --include='*.py' --include='*.ts' --include='*.tsx' \ --include='*.js' --include='*.jsx' --include='*.go' \ --exclude-dir=node_modules --exclude-dir=.venv \ 2>/dev/null | head -20 # Custom /metrics endpoint — Prometheus convention. Matches route definitions. grep -rnE '["/]metrics["/\s(]' /src /app \ /internal --include='*.py' --include='*.ts' \ --include='*.js' --include='*.go' \ --exclude-dir=node_modules 2>/dev/null | head -10 ``` Assemble observability markers JSON (the sub-agent writes the interpreted fields, not the raw grep): ```json { "logging_lib": "structlog" | null, "logging_detected_via": "manifest" | "import" | "json-format-heuristic" | null, "metrics_lib": "opentelemetry" | null, "metrics_detected_via": "manifest" | "import" | "metrics-endpoint" | null, "has_metrics_endpoint": true | false, "has_health_endpoint": true | false, "json_log_detected": true | false } ``` Capture: - `logging_lib` — name of the structured-logging library, or `null` if none found - `metrics_lib` — name of the metrics/tracing library, or `null` if none found - `has_metrics_endpoint` — bool (custom `/metrics` route found) - `has_json_log_format` — bool (heuristic match; low-confidence tiebreaker only) - `observability_logging_present` = `(logging_lib != null) OR has_json_log_format` - `observability_metrics_present` = `(metrics_lib != null) OR has_metrics_endpoint` **OBSERVABILITY gate trigger:** `observability_logging_present == false` AND `observability_metrics_present == false`. Both must be absent for the gate to fire — one present is enough to avoid the cap. Caps `deployment_readiness ≤ 3`. --- ## Raw dumps (flat — files under `raw/` named `deployment-readiness-*`) Each signal writes its raw output to the flat `raw/` directory at the workspace root. Absent files = not extracted / not applicable. Aggregates go to `scorecard.yaml` under `data.`. | File | Source signal | Shape / note | | --- | --- | --- | | `raw/deployment-readiness-artifacts.json` | Signals 1 + 3 + 4 | Combined filesystem probe: `{container: {dockerfile, compose, devcontainer, healthcheck, multi_stage, non_root}, deploy: {gha_workflows, paas_configs, iac_dirs, helm, kustomize}, env: {env_example, env_in_gitignore, tracked_env_files}, health: {routes, k8s_probes}}` | | `raw/deployment-readiness-hadolint.json` | Signal 1 | Raw `hadolint --format json` output (absent when no Dockerfile) | | `raw/deployment-readiness-localhost-refs.txt` | Signal 4 | `grep -rnE 'localhost:...\|127.0.0.1:...'` output with `file:line`, after env-var exclusion (absent when count = 0) | | `raw/deployment-readiness-llm-read.md` | Signal 5 | LLM's README + Dockerfile analysis: `{readme_score, dockerfile_score, has_run_section, confidence, one_sentence}` plus a short paragraph of rationale | | `raw/deployment-readiness-observability-markers.json` | Signal 6 | Structured object: `{logging_lib, logging_detected_via, metrics_lib, metrics_detected_via, has_metrics_endpoint, has_health_endpoint, json_log_detected}` — feeds the OBSERVABILITY gate decision | --- ## Aggregation → 0–5 band Scorer integrates the 5 signals. Rough directionality: | Signal | Pulls score UP | Pulls score DOWN | | --- | --- | --- | | `has_container_artifact` + (Hadolint-clean if Dockerfile) | Present + 0 Hadolint errors | Absent or > 5 Hadolint errors | | `docker_has_healthcheck` / `docker_non_root` / `docker_multi_stage` | All true | None true | | `has_env_example` + `env_in_gitignore` + zero tracked `.env` | All clean | Any violation | | `has_any_deploy_artifact` (container / GHA / PaaS / IaC) | ≥ 2 distinct artifacts | None | | `health_route_hits` or `k8s_probes_present` or Docker `HEALTHCHECK` | ≥ 1 present | 0 | | `localhost_hardcoded_count` | 0 in non-test source | > 3 | | `llm_readme_score` | 4–5 | 0–2 | | `llm_dockerfile_score` | 4–5 | 0–2 | **Bands (absolute):** | Band | Path A — container-first | Path B — serverless-native | | --- | --- | --- | | **5** | Dockerfile Hadolint-clean with multi-stage + USER + HEALTHCHECK + `.env.example` + health route + deploy workflow + README LLM ≥ 4 + zero localhost hardcoding | PaaS/managed-platform config + deploy workflow + `.env.example` + health route (HTTP or framework) + README LLM ≥ 4 + zero localhost hardcoding | | **4** | Strong container path, one weaker signal | Strong serverless path, one weaker signal | | **3** | Some container or platform artifact; env vars referenced; mixed other signals | | **2** | Minimal — one ship artifact present but weak README / no health / no env docs | | **1** | DEMO gate fires (see below); localhost-only repo | | **0** | Suppressed (DEMO gate caps at 1) | Serverless-native repos (Vercel/Fly/Cloudflare/Modal) reach band 5 via Path B — do not penalize absent Dockerfile if a PaaS config carries the deploy artifact. --- ## DEMO gate — two-signal agreement Gate fires only when **all three** are absent AND the LLM + mechanical README read agree: ``` has_container_artifact == false AND has_any_deploy_artifact == false (i.e. no GHA deploy, no PaaS config, no IaC) AND readme_has_run_section == false (mechanical heading grep) AND llm_has_run_section == false (LLM read) ``` **Why two-signal:** a single LLM judgment as a hard gate means one noisy call caps the score. Requiring heading grep + LLM agreement makes the gate robust to LLM noise. If `llm_confidence == low`, defer to mechanical-only (skip the LLM side of the AND). When fired, emit `gates_triggered: [DEMO]` and cap `deployment_readiness ≤ 1`. --- ## OBSERVABILITY gate — both arms absent Gate fires only when **both** observability arms are absent: ``` observability_logging_present == false AND observability_metrics_present == false ``` That is: the repo ships neither a structured-logging library (structlog / winston / pino / log-slog / zap / a detected JSON-log format) **nor** a metrics/tracing library (OpenTelemetry / Prometheus client / Datadog / statsd / a custom `/metrics` endpoint). **Why both-arms:** enterprise observability is "logs AND metrics", but a single-binary CLI or a thin FaaS handler may legitimately ship with just structured logs and skip metrics (or vice versa). Demanding both is punitive on small, valid shapes. The gate catches the repo that's shipping without _any_ production-grade signal — `print()`-only logging and no metrics — which is the failure we care about. When fired, emit `gates_triggered: [OBSERVABILITY]` and cap `deployment_readiness ≤ 3`. Absent arm names ship in the evidence string: `"no structured logger AND no metrics/tracing — cap at 3"`. --- ## Fallback — signals missing - No Dockerfile → drop Hadolint lint; rely on Signals 2, 3, 4, 5 + `has_devcontainer` / `compose_has_inline_image` as container signal. - Hadolint not installed → drop lint component; rely on LLM Dockerfile read. - Fully serverless (Vercel/Fly only, no Dockerfile) → Path B; PaaS config in Signal 3 carries deploy artifact weight. - LLM low-confidence → DEMO gate falls back to mechanical-only (README heading grep + artifact absence). Annotate evidence with `"Signals used: [...]"` and `"Path: A | B"` so the audit trail is explicit. --- ## Output (what the scorer emits) ```yaml sub_criterion: deployment_readiness score: 3 evidence: "Dockerfile present (multi-stage, non-root, no HEALTHCHECK; Hadolint 2 warnings); .env.example + .env gitignored; no deploy workflow; README has install steps but no outside-localhost deploy section; 4 hardcoded localhost refs. Path A." data: has_dockerfile: true has_compose: false has_dockerignore: true has_devcontainer: false compose_has_inline_image: false has_container_artifact: true docker_has_healthcheck: false docker_multi_stage: true docker_non_root: true hadolint_findings: 2 hadolint_by_severity: { error: 0, warning: 2, info: 0, style: 0 } has_env_example: true env_in_gitignore: true tracked_env_files: [] deploy_workflow_files: [] paas_configs: [] iac_dirs: [] has_any_deploy_artifact: true # Dockerfile counts as containerization health_route_hits: 0 k8s_probes_present: false localhost_hardcoded_count: 4 readme_has_run_section: true llm_readme_score: 2 llm_dockerfile_score: 3 llm_has_run_section: true llm_confidence: medium logging_lib: structlog logging_detected_via: manifest metrics_lib: null metrics_detected_via: null has_metrics_endpoint: false has_json_log_format: true observability_logging_present: true observability_metrics_present: false path: A signals_used: [filesystem, hadolint, llm, observability-markers] gate_demo_fired: false gate_observability_fired: false ``` --- ## Caveats - File presence ≠ working artifact. Only runtime verification (a separate manual pass) catches broken Dockerfiles / deploy workflows. - Hadolint catches form, not intent. A "clean" Dockerfile that installs nothing still lints green. - PaaS probe list needs periodic update — new deploy targets appear faster than the list evolves. - LLM sample is 1 README. Repos with multi-file docs (`docs/`, `CONTRIBUTING.md`, wiki) may be under-scored on the interpretive signal. Consider glob-and-concat for docs-heavy repos. - Localhost grep is a noisy counter-signal even with env-var exclusion — LLM notes should adjudicate when the count is > 0 but reasonable. - Two-signal DEMO gate may still false-negative on repos whose README has a heading like `## Getting Started` that only says `npm run dev` (LLM catches this; mechanical does not). That's why LLM is in the AND, not replaced.