{
  "version": 1,
  "id": "gitops-with-argocd",
  "title": "GitOps with Argo CD",
  "summary": "Desired state in Git, the reconcile loop, sync policies, prune and ApplicationSet promotion.",
  "estimatedMinutes": 20,
  "prerequisites": [],
  "source": "courses-notebook SYS401_kubernetes_devops 06_gitops_argocd",
  "quiz": [
    {
      "prompt": "In ArgoCD, what does `OutOfSync` indicate?",
      "options": [
        "The cluster is unreachable",
        "Actual cluster state differs from the desired state in the Git repository at the configured path/revision",
        "etcd is partitioned",
        "The ArgoCD controller itself is down"
      ],
      "answer": 1
    },
    {
      "prompt": "Which ArgoCD setting causes resources removed from Git to also be deleted from the cluster on sync?",
      "options": [
        "`selfHeal: true`",
        "`prune: true`",
        "`automated: false`",
        "`syncWaves: 0`"
      ],
      "answer": 1
    }
  ],
  "lesson": "**GitOps** treats Git as the single source of truth for cluster state. **ArgoCD** is the Kubernetes-native GitOps controller that continuously reconciles the actual cluster state with the desired state defined in Git \u2014 detecting drift, enabling auditable rollbacks via `git revert`, and automating deployments through pull requests.\n\n**Learning objectives**\n- Model the ArgoCD Application CRD and its sync/health status machine\n- Simulate the reconciliation loop: diff desired vs actual, apply deltas\n- Implement drift detection and self-heal mechanics\n- Model multi-environment promotion with ApplicationSets\n\n## Core Concepts\n\n| Concept | Definition | Key Detail |\n|---|---|---|\n| ArgoCD Application | CRD linking Git repo/path to cluster/namespace | The unit of GitOps management |\n| Sync status | Synced / OutOfSync / Unknown | Cluster matches Git or not |\n| Health status | Healthy / Progressing / Degraded / Missing | Actual workload health |\n| OutOfSync | Git differs from cluster state | Detected via hash comparison |\n| Sync | Apply Git state to cluster | Manual or automatic |\n| Auto-sync | ArgoCD syncs on Git change | Continuous delivery |\n| Prune | Delete orphaned cluster resources | Resources absent from Git |\n| Self-heal | Revert manual cluster changes | Enforces Git as only truth |\n| ApplicationSet | Template generating multiple Applications | Multi-env, multi-cluster |\n| Sync waves | Ordered sync of resource groups | Deploy CRDs before CRs |\n| App-of-apps | ArgoCD app managing other ArgoCD apps | Bootstrapping pattern |\n\n```python\nfrom dataclasses import dataclass, field\nfrom typing import Dict, List, Optional, Set\nfrom enum import Enum\nimport hashlib\nimport json\n\n# ---------------------------------------------------------------------------\n# ArgoCD core types\n# ---------------------------------------------------------------------------\n\nclass SyncStatus(Enum):\n    SYNCED       = 'Synced'\n    OUT_OF_SYNC  = 'OutOfSync'\n    UNKNOWN      = 'Unknown'\n\n\nclass HealthStatus(Enum):\n    HEALTHY      = 'Healthy'\n    PROGRESSING  = 'Progressing'\n    DEGRADED     = 'Degraded'\n    MISSING      = 'Missing'\n    SUSPENDED    = 'Suspended'\n\n\n@dataclass(frozen=True)\nclass K8sResource:\n    \"\"\"\n    Immutable representation of a Kubernetes resource.\n    Identity: (kind, namespace, name). Content: spec hash.\n    \"\"\"\n    kind:      str\n    name:      str\n    namespace: str\n    spec:      str  # JSON-serialized spec\n\n    @property\n    def key(self) -> str:\n        return f'{self.kind}/{self.namespace}/{self.name}'\n\n    @property\n    def content_hash(self) -> str:\n        return hashlib.sha256(self.spec.encode()).hexdigest()[:8]\n\n    def __repr__(self) -> str:\n        return f'{self.kind}/{self.name}@{self.content_hash}'\n\n\ndef make_resource(kind: str, name: str, ns: str, spec: dict) -> K8sResource:\n    return K8sResource(kind, name, ns, json.dumps(spec, sort_keys=True))\n\n\n@dataclass\nclass ResourceDiff:\n    creates: List[K8sResource] = field(default_factory=list)\n    updates: List[K8sResource] = field(default_factory=list)\n    deletes: List[K8sResource] = field(default_factory=list)\n\n    @property\n    def is_empty(self) -> bool:\n        return not (self.creates or self.updates or self.deletes)\n\n    def show(self) -> None:\n        if self.is_empty:\n            print('  (no diff \u2014 already synced)')\n            return\n        for r in self.creates:\n            print(f'  + CREATE {r}')\n        for r in self.updates:\n            print(f'  ~ UPDATE {r}')\n        for r in self.deletes:\n            print(f'  - DELETE {r}')\n\n\n@dataclass\nclass ArgoCDApp:\n    name:             str\n    repo_url:         str\n    target_revision:  str   # branch or commit SHA\n    path:             str   # path within repo\n    dest_namespace:   str\n    auto_sync:        bool = False\n    prune:            bool = False\n    self_heal:        bool = False\n    _desired: Dict[str, K8sResource] = field(default_factory=dict)  # from Git\n    _actual:  Dict[str, K8sResource] = field(default_factory=dict)  # from cluster\n    _sync_history: List[Dict] = field(default_factory=list)\n\n    # -------------------------\n    # Status properties\n    # -------------------------\n    @property\n    def sync_status(self) -> SyncStatus:\n        if not self._desired:\n            return SyncStatus.UNKNOWN\n        return SyncStatus.SYNCED if self._diff().is_empty else SyncStatus.OUT_OF_SYNC\n\n    @property\n    def health_status(self) -> HealthStatus:\n        # Simple model: healthy if synced and all expected resources exist\n        if not self._actual:\n            return HealthStatus.MISSING\n        if self.sync_status == SyncStatus.OUT_OF_SYNC:\n            return HealthStatus.PROGRESSING\n        return HealthStatus.HEALTHY\n\n    # -------------------------\n    # Core operations\n    # -------------------------\n    def load_from_git(self, resources: List[K8sResource], commit_sha: str = '') -> None:\n        \"\"\"Simulate ArgoCD polling Git and loading desired state.\"\"\"\n        self._desired = {r.key: r for r in resources}\n        if commit_sha:\n            self.target_revision = commit_sha\n\n    def _diff(self) -> ResourceDiff:\n        diff = ResourceDiff()\n        for key, desired in self._desired.items():\n            actual = self._actual.get(key)\n            if actual is None:\n                diff.creates.append(desired)\n            elif actual.content_hash != desired.content_hash:\n                diff.updates.append(desired)\n        if self.prune:\n            for key, actual in self._actual.items():\n                if key not in self._desired:\n                    diff.deletes.append(actual)\n        return diff\n\n    def sync(self, force: bool = False) -> ResourceDiff:\n        diff = self._diff()\n        if diff.is_empty and not force:\n            return diff\n        # Apply diff\n        new_actual = dict(self._actual)\n        for r in diff.creates + diff.updates:\n            new_actual[r.key] = r\n        for r in diff.deletes:\n            new_actual.pop(r.key, None)\n        self._actual = new_actual\n        self._sync_history.append({\n            'revision': self.target_revision,\n            'creates': len(diff.creates),\n            'updates': len(diff.updates),\n            'deletes': len(diff.deletes),\n        })\n        return diff\n\n    def simulate_drift(self, key: str, patched_spec: dict) -> None:\n        \"\"\"Someone ran kubectl edit \u2014 mutate actual state directly.\"\"\"\n        if key in self._actual:\n            old = self._actual[key]\n            drifted = K8sResource(old.kind, old.name, old.namespace,\n                                   json.dumps(patched_spec, sort_keys=True))\n            self._actual[key] = drifted\n\n    def status_line(self) -> str:\n        return (f'  {self.name}: sync={self.sync_status.value:12} '\n                f'health={self.health_status.value:12} '\n                f'resources={len(self._actual)}')\n\n\n# ---------------------------------------------------------------------------\n# Simulate a full ArgoCD application lifecycle\n# ---------------------------------------------------------------------------\napp = ArgoCDApp(\n    name='ml-inference-api',\n    repo_url='https://github.com/org/k8s-manifests',\n    target_revision='main',\n    path='apps/ml-inference-api',\n    dest_namespace='production',\n    auto_sync=True, prune=True, self_heal=True,\n)\n\n# --- Step 1: Initial deploy (Git has v1.0 manifests) ---\nv1_resources = [\n    make_resource('Deployment',  'ml-api',         'production', {'image': 'ml-api:1.0', 'replicas': 3}),\n    make_resource('Service',     'ml-api-svc',     'production', {'port': 80, 'type': 'ClusterIP'}),\n    make_resource('ConfigMap',   'ml-api-config',  'production', {'MODEL': 'resnet50', 'BATCH_SIZE': '32'}),\n]\napp.load_from_git(v1_resources, commit_sha='abc1234')\nprint('Step 1: Initial load from Git')\nprint(f'  Sync status before sync: {app.sync_status.value}')\ndiff = app.sync()\nprint('  Diff applied:')\ndiff.show()\nprint(app.status_line())\n\nprint()\n# --- Step 2: Developer pushes new image + HPA ---\nv2_resources = [\n    make_resource('Deployment',  'ml-api',         'production', {'image': 'ml-api:1.1', 'replicas': 3}),\n    make_resource('Service',     'ml-api-svc',     'production', {'port': 80, 'type': 'ClusterIP'}),\n    make_resource('ConfigMap',   'ml-api-config',  'production', {'MODEL': 'resnet50', 'BATCH_SIZE': '64'}),\n    make_resource('HPA',         'ml-api-hpa',     'production', {'min': 3, 'max': 15, 'target_cpu': 70}),\n]\napp.load_from_git(v2_resources, commit_sha='def5678')\nprint('Step 2: git push \u2014 image v1.1 + HPA + config change')\nprint(f'  Sync status: {app.sync_status.value}')\ndiff = app.sync()\nprint('  Diff applied:')\ndiff.show()\nprint(app.status_line())\n\nprint()\n# --- Step 3: Operator manually scales deployment (drift) ---\nprint('Step 3: Operator ran: kubectl scale deploy/ml-api --replicas=10')\napp.simulate_drift('Deployment/production/ml-api', {'image': 'ml-api:1.1', 'replicas': 10})\nprint(f'  Drift detected: sync={app.sync_status.value}')\n\n# Self-heal: ArgoCD detects drift and re-syncs\nif app.self_heal and app.sync_status == SyncStatus.OUT_OF_SYNC:\n    print('  Self-heal triggered: reverting to Git state')\n    diff = app.sync()\n    diff.show()\n    print(app.status_line())\n```\n\n## ApplicationSet \u2014 Multi-Environment Promotion\n\n```python\nfrom dataclasses import dataclass, field\nfrom typing import Dict, List, Any\n\n@dataclass\nclass AppSetGenerator:\n    \"\"\"List generator \u2014 simplest ApplicationSet generator type.\"\"\"\n    elements: List[Dict[str, Any]]\n\n\n@dataclass\nclass AppSetTemplate:\n    \"\"\"Template for generating ArgoCD Applications.\"\"\"\n    name_pattern: str          # e.g. 'ml-api-{{env}}'\n    repo_url: str\n    path_pattern: str          # e.g. 'envs/{{env}}'\n    dest_namespace_pattern: str  # e.g. '{{env}}'\n    auto_sync: bool = False\n    prune: bool = True\n\n\n@dataclass\nclass ApplicationSet:\n    name: str\n    generator: AppSetGenerator\n    template: AppSetTemplate\n\n    def generate_applications(self) -> List[ArgoCDApp]:\n        apps = []\n        for elem in self.generator.elements:\n            def render(pattern: str, params: dict) -> str:\n                result = pattern\n                for k, v in params.items():\n                    result = result.replace(f'{{{{{k}}}}}', str(v))\n                return result\n\n            apps.append(ArgoCDApp(\n                name=render(self.template.name_pattern, elem),\n                repo_url=self.template.repo_url,\n                target_revision=elem.get('revision', 'main'),\n                path=render(self.template.path_pattern, elem),\n                dest_namespace=render(self.template.dest_namespace_pattern, elem),\n                auto_sync=elem.get('auto_sync', self.template.auto_sync),\n                prune=self.template.prune,\n            ))\n        return apps\n\n\n# Define ApplicationSet for ml-api across environments\nappset = ApplicationSet(\n    name='ml-api-environments',\n    generator=AppSetGenerator([\n        {'env': 'dev',        'revision': 'main',    'auto_sync': True},\n        {'env': 'staging',    'revision': 'main',    'auto_sync': True},\n        {'env': 'production', 'revision': 'v1.5.0',  'auto_sync': False},  # manual in prod\n    ]),\n    template=AppSetTemplate(\n        name_pattern='ml-api-{{env}}',\n        repo_url='https://github.com/org/k8s-manifests',\n        path_pattern='envs/{{env}}/ml-api',\n        dest_namespace_pattern='{{env}}',\n        auto_sync=False,\n        prune=True,\n    ),\n)\n\ngenerated_apps = appset.generate_applications()\nprint(f'ApplicationSet \"{appset.name}\" generated {len(generated_apps)} Applications:')\nprint(f'{\"Name\":>25}  {\"Namespace\":>12}  {\"Revision\":>10}  {\"AutoSync\"}')\nprint('-' * 65)\nfor app_gen in generated_apps:\n    print(f'  {app_gen.name:>23}  {app_gen.dest_namespace:>12}  '\n          f'{app_gen.target_revision:>10}  {app_gen.auto_sync}')\n\n# Simulate promotion: dev auto-syncs on every commit, staging after PR merge,\n# production requires manual ArgoCD sync\nprint()\nprint('Promotion model:')\nprint('  dev        \u2192 auto-sync on every commit to main')\nprint('  staging    \u2192 auto-sync on merge to main (same branch)')\nprint('  production \u2192 manual sync after staging validation, pinned to release tag')\n\n# Simulate sync events\nresources_v1 = [\n    make_resource('Deployment', 'ml-api', env_app.dest_namespace,\n                  {'image': f'ml-api:1.5.0', 'replicas': 2})\n    for env_app in generated_apps\n]\n\nprint()\nprint('Sync all environments:')\nfor i, env_app in enumerate(generated_apps):\n    env_app.load_from_git([resources_v1[i]], commit_sha='sha_v1.5.0')\n    diff = env_app.sync()\n    print(env_app.status_line())\n```\n\n## Quick Quiz\n\n**Q1**: What is the difference between `prune: true` and `self_heal: true` in ArgoCD?\n> **A**: `prune: true` deletes cluster resources that exist in the cluster but are absent from Git (orphaned resources). `self_heal: true` reverts manual changes to resources that ARE in Git \u2014 someone ran `kubectl edit` and ArgoCD reverts the change on next reconcile.\n\n**Q2**: How does ArgoCD handle the case where a sync partially fails (3/5 resources applied)?\n> **A**: ArgoCD reports the application as `OutOfSync` with health `Degraded`. The 3 applied resources stay in the cluster; the 2 failed resources remain as their old versions. ArgoCD will retry the sync based on retry policy settings (`syncPolicy.retry`). The partial state is visible in the ArgoCD UI per-resource.\n\n**Q3**: How do sync waves help with resource ordering?\n> **A**: Resources annotated with `argocd.argoproj.io/sync-wave: \"0\"` sync before wave 1, which syncs before wave 2. This enables: CRDs (wave 0) \u2192 Namespaces (wave 1) \u2192 Deployments (wave 2). ArgoCD waits for all resources in wave N to be healthy before proceeding to wave N+1.\n\n## Key Takeaways\n\n- GitOps = Git as the single source of truth; ArgoCD continuously reconciles cluster to match Git\n- `OutOfSync` means Git differs from cluster \u2014 auto-sync applies Git state; manual sync requires approval\n- `prune` removes orphaned resources; `self_heal` reverts manual cluster changes\n- ApplicationSets generate multiple Applications from a template \u2014 enabling multi-env, multi-cluster deployments\n- Rollback = `git revert` \u2192 ArgoCD auto-syncs the reverted state \u2014 no kubectl required\n\n## Going Deeper\n\n1. **Exercise**: Implement sync waves in `ArgoCDApp.sync()` \u2014 annotate resources with wave numbers and enforce ordering by waiting for each wave before proceeding.\n2. **Exercise**: Add a `health_check(resource) -> HealthStatus` function that checks deployment rollout status (available replicas vs desired) and propagates health to the Application.\n3. **Exercise**: Model the `app-of-apps` pattern \u2014 an ArgoCD Application whose Git path contains other ArgoCD Application manifests. Bootstrap a new cluster from a single seed application.\n4. **Project**: Build a `GitOpsPromotion` simulator \u2014 commit to dev branch triggers dev sync; PR merge to main triggers staging sync; Git tag triggers production sync.\n5. **Research**: Compare ArgoCD to Flux CD \u2014 architecture differences (operator vs agent model), CRD design, and multi-tenancy security model.\n\n## Hands-on Lab\n\nThe simulator above models the ArgoCD reconciliation loop in Python. The companion lab runs the real thing: a local k3d cluster with ArgoCD installed from upstream, two `Application` CRs (`argoproj.io/v1alpha1`) pointing at the canonical `argocd-example-apps` repo, and a scripted drift scenario where you manually scale a managed Deployment and watch ArgoCD revert it \u2014 exactly the `self_heal` path simulated in Step 3 above.\n\nCluster boot to UI in under 5 minutes on an M1 with 8 GB RAM.\n\n- `make up` \u2014 k3d cluster + ArgoCD install\n- `make apply-apps` \u2014 register `guestbook` and `helm-guestbook` Applications\n- `make argocd-ui` \u2014 print admin password, port-forward UI to `https://localhost:8080`\n- `make drift-demo` \u2014 `kubectl scale` a managed Deployment to introduce drift\n- Watch the ArgoCD UI flip `OutOfSync` \u2192 `Synced` as self-heal reverts the change\n- `make down` \u2014 destroy the cluster\n\nSee [`lab_06_gitops_argocd/README.md`](lab_06_gitops_argocd/README.md) for the full walkthrough, expected output, and how to swap the upstream demo repo for your own fork.\n"
}