---
icon: material/heart-pulse
title: "Operational Procedures"
---

Monitoring L1ES plugin health in production, backup and disaster recovery for plugin indices, and safe rolling upgrades with version-pinned plugins.

??? tenx-operational "How do I monitor L1ES plugin health in production"

    **Health check endpoint:**

    ```bash
    curl http://localhost:9200/_l1es
    ```

    **Expected response:**

    ```json
    {
      "name": "l1es-plugin",
      "description": "L1x support for decoding inside elasticsearch",
      "version": {
        "l1es": "0.3.0",
        "l1es-snapshot": "0.3.0.es.8.17.0",
        "elasticsearch": "8.17.0"
      }
    }
    ```

    **Set up Elasticsearch Watcher alert (Kibana):**

    ```json
    POST _watcher/watch/l1es_plugin_health
    {
      "trigger": { "schedule": { "interval": "5m" } },
      "input": {
        "http": {
          "request": {
            "url": "http://localhost:9200/_l1es",
            "method": "GET"
          }
        }
      },
      "condition": {
        "compare": { "ctx.payload.version.l1es": { "eq": null } }
      },
      "actions": {
        "notify_team": {
          "email": {
            "to": "ops@company.com",
            "subject": "L1ES Plugin Health Alert"
          }
        }
      }
    }
    ```

    **Monitor template index health:**

    ```bash
    # Check shard status
    GET _cat/shards/l1es_dml?v
    # All shards should be GREEN (STARTED)

    # Check index stats
    GET l1es_dml/_stats
    # Look for: docs.count (template count), store.size_in_bytes
    ```

??? tenx-operational "Backup and disaster recovery for L1ES plugin"

    **Monthly backup procedure:**

    L1ES stores two critical indices: templates and field mappings.

    ```bash
    # Backup 1: Templates (monthly)
    curl -X GET "localhost:9200/l1es_dml/_search?size=10000&scroll=5m" \
      -H "Content-Type: application/json" > templates_backup_$(date +%Y%m%d).json

    # Backup 2: Field mappings (monthly)
    curl -X GET "localhost:9200/l1es_dml_indices/_search?size=1000" \
      -H "Content-Type: application/json" > mappings_backup_$(date +%Y%m%d).json

    # Store in S3 or backup system
    aws s3 cp templates_backup_*.json s3://my-backups/l1es/
    ```

    **Recovery: l1es_dml index deleted**

    1. **Check what's lost:**
       ```
       GET l1es_dml/_search
       # Should return 0 if index is gone
       ```

    2. **Restore from backup:**
       ```bash
       # Re-create and bulk-load templates from backup
       curl -X POST "localhost:9200/_bulk" \
         -H "Content-Type: application/json" \
         --data-binary @templates_backup_20260227.json
       ```

    3. **Verify restoration:**
       ```
       GET l1es_dml/_count
       # Should match pre-deletion count
       ```

    **Impact during recovery:** Queries will NOT expand until templates are restored (encoded events remain encoded). Restore time scales with the template count and your cluster's bulk-load throughput.

    **Recovery: l1es_dml_indices deleted**

    1. **Re-register field mappings:**
       ```json
       POST _l1es/add-dml-index
       {
         "index_name": "my-logs",
         "source": "message",
         "dest": "decoded_message"
       }
       ```

    2. **Verify:**
       ```
       GET l1es_dml_indices/_search
       # Should show your index mapping
       ```

    **Impact:** Expansion stops for that index until re-registered. Recovery is immediate (< 1 second).

??? tenx-operational "Rolling upgrade: Elasticsearch version bump with L1ES plugin"

    **For 3-node Elasticsearch cluster upgrade (e.g., 8.16 → 8.17):**

    ```bash
    # Step 1: Disable shard allocation (prevents rebalancing during upgrade)
    PUT _cluster/settings
    {
      "persistent": {
        "cluster.routing.allocation.enable": "none"
      }
    }

    # Step 2: Stop and upgrade node 1
    # (On node 1)
    curl -X POST "localhost:9200/_nodes/self/_shutdown"

    # Upgrade ES binary
    # (e.g., apt upgrade elasticsearch)

    # Upgrade L1ES plugin
    bin/elasticsearch-plugin remove l1es-plugin
    bin/elasticsearch-plugin install file:///path/to/l1es-plugin-0.3.0.es.8.17.0.zip

    # Start node 1
    systemctl start elasticsearch

    # Step 3: Wait for node 1 to rejoin cluster and recover shards
    GET _cluster/health
    # Watch until status=green, relocating_shards=0

    # Step 4: Re-enable shard allocation
    PUT _cluster/settings
    {
      "persistent": {
        "cluster.routing.allocation.enable": "all"
      }
    }

    # Step 5: Verify cluster is healthy
    GET _cluster/health
    # Should show: status=green, active_shards=<all shards>

    # Repeat for nodes 2 and 3
    ```

    **CRITICAL: Plugin version must match Elasticsearch version**

    Each plugin build is compiled against an exact Elasticsearch/OpenSearch version. An ES 8.17.0 plugin will not load on ES 8.16.x or 8.18.x.

    | L1ES | Elasticsearch | OpenSearch | Java | Lucene |
    |---|---|---|---|---|
    | 0.3.0 | 8.17.0 | 2.19.0 | 17+ | 9.12.0 |
    | 0.2.x | 7.10.0 | n/a | 11+ | 8.7.0 |

    **If plugin version mismatch:**
    - The plugin fails to load with a `Plugin version mismatch` error (or `UnsupportedClassVersionError` on the wrong Java version)
    - Always install the plugin build matching your ES/OS version exactly before starting the node