Manage Grepr integrations, datasets, and pipelines with Terraform.

# Manage Grepr resources with Terraform

Use the Grepr Terraform provider to create and manage integrations, datasets, and pipelines from your infrastructure configuration.

## Requirements \[#tf-provider-reqs]

To use the Grepr Terraform provider, you must have:

* Terraform or OpenTofu CLI version 1.0 or later.
* OAuth2 client credentials for authentication. To obtain the required credentials, create a service account in the Grepr UI. See [Manage Grepr service accounts](/admin/service-accounts).

## Install the Grepr provider

The Grepr provider is published to the [OpenTofu registry](https://search.opentofu.org/provider/grepr/grepr/). To install it, declare it in the `required_providers` block of your configuration. To use the OpenTofu CLI, use the following configuration:

```hcl
terraform {
  required_providers {
    grepr = {
      source  = "grepr/grepr"
      version = "~> 1.2"
    }
  }
}
```

To use the Terraform CLI, you must use the fully qualified source address `registry.opentofu.org/grepr/grepr`:

```hcl
terraform {
  required_providers {
    grepr = {
      source  = "registry.opentofu.org/grepr/grepr"
      version = "~> 1.2"
    }
  }
}
```

Then initialize your working directory to download the provider:

```bash
terraform init # Or tofu init
```

## Configure authentication for the Grepr Terraform provider

To configure the required OAuth2 credentials for authentication, add a provider block to your Terraform configuration with the `host`, `client_id`, and `client_secret` fields:

```hcl
provider "grepr" {
  host          = "https://<your-org>.app.grepr.ai"
  client_id     = var.grepr_client_id
  client_secret = var.grepr_client_secret
}
```

Replace:

* `<your-org>` with your organization name.
* `var.grepr_client_id` with a variable containing your service account client ID.
* `var.grepr_client_secret` with a sensitive variable containing your service account client secret.

Alternatively, you can use environment variables to set the required credentials. The provider automatically reads these environment variables if you set them:

```bash
export GREPR_HOST=https://<your-org>.app.grepr.ai
export GREPR_CLIENT_ID=<your-client-id>
export GREPR_CLIENT_SECRET=<your-client-secret>
```

## Manage integrations and datasets \[#tf-integrations-datasets]

The provider also manages the integrations and datasets your pipelines use, starting with provider version 1.2.0. Each integration type has its own resource:

| Resource | Manages |
| --- | --- |
| `grepr_datadog_integration` | A Datadog integration and its API keys |
| `grepr_splunk_integration` | A Splunk integration and its tokens |
| `grepr_newrelic_integration` | A New Relic integration and its keys |
| `grepr_sumo_integration` | A Sumo Logic integration and its collector URL |
| `grepr_otlp_integration` | An OTLP/HTTP destination and its secret headers |
| `grepr_data_warehouse_integration` | A Grepr-managed data warehouse |
| `grepr_s3_data_warehouse_integration` | A data warehouse in your S3 bucket |
| `grepr_dataset` | A dataset in a data warehouse |

Reference their IDs from a pipeline so Terraform creates them first:

```hcl
variable "datadog_api_key" {
  type      = string
  sensitive = true
}

resource "grepr_datadog_integration" "datadog" {
  name    = "terraform_datadog"
  site    = "datadoghq.com"
  api_key = var.datadog_api_key
}

resource "grepr_data_warehouse_integration" "warehouse" {
  name = "terraform_warehouse"
}

resource "grepr_dataset" "logs" {
  name           = "terraform_logs"
  integration_id = grepr_data_warehouse_integration.warehouse.id
}

resource "grepr_pipeline" "logs" {
  name = "terraform_logs"
  job_graph_json = jsonencode({
    vertices = [
      {
        type          = "datadog-log-agent-source"
        name          = "source"
        integrationId = grepr_datadog_integration.datadog.id
      },
      {
        type      = "logs-iceberg-table-sink"
        name      = "sink"
        datasetId = grepr_dataset.logs.id
      }
    ]
    edges = ["source -> sink"]
  })
}
```

Keep in mind:

* Secrets such as API keys are sensitive attributes, but Terraform stores them in state. Protect your state backend with encryption and access controls.
* Grepr never returns secrets, so Terraform writes them but can't read them back and doesn't detect a secret changed outside Terraform. Removing a secret from your configuration stops managing it without revoking it.
* Write values the way Grepr stores them, such as durations in hours (`PT24H` rather than `P1D`) and lowercase encodings. Grepr rewrites other spellings, and the next plan shows the difference.
* Deleting a dataset drops its stored data. Changing its `integration_id` replaces it, so the new dataset starts without the old data or table settings such as its promotion policy. Add `lifecycle { prevent_destroy = true }` to guard against accidental deletion. Table settings made outside Terraform are left alone.
* Import integrations and datasets by ID, for example `terraform import grepr_dataset.logs DATASET_ID`. Import can't recover secrets, so the next apply writes the secrets you configure. Writing a Datadog API key is refused while a pipeline uses the integration, so stop the pipeline first or leave the key out of the configuration.

See the [provider documentation](https://search.opentofu.org/provider/grepr/grepr/) for each resource's attributes.

## Import an existing pipeline

To bring an existing pipeline under Terraform management, use the `terraform import` command with the pipeline's ID or name:

```bash
terraform import grepr_pipeline.example <pipeline-id-or-name>
```

To automatically import an existing pipeline without using the `import` command, create a `grepr_pipeline` resource with the same name as the existing pipeline. The provider will detect and import the existing pipeline.

### Tags \[#tf-pipeline-tags]

Starting with provider version 1.3.0, the `tags` that you configure are the pipeline's complete set of tags. When you change them, Grepr updates the pipeline's tags without creating a new pipeline version or redeploying the pipeline. You can change a pipeline's tags only while the pipeline is running or stopped, so apply a tag change after any earlier update to the pipeline finishes. Tags that you leave out of the configuration are removed from the pipeline. To keep editing a pipeline that you created in the Grepr UI, include `grepr-ui-managed = "true"` in its tags. To keep a pipeline's existing tags, including for a pipeline that you import, leave `tags` unset.

A pipeline can have up to 10 tags. Each key can be up to 64 characters long, and each value can be up to 256 characters long. Tag keys that start with `__grepr__` are reserved.

## Example: create a pipeline

The following example shows a simple pipeline that uses a Datadog log agent source, a Grok parser, and an Iceberg table sink. To improve maintainability and avoid conflicts with Terraform reserved symbols or keywords, Grepr recommends defining your pipeline's job graph in a separate JSON file. You then reference this JSON file in your Terraform configuration. This example demonstrates how to define a pipeline job graph in a JSON file and reference the file in the Terraform configuration.

A common development workflow is to create a pipeline in the Grepr UI or API, test it and make changes until it meets your requirements, then copy the generated JSON to a file for use with your Terraform configuration. If you choose to create pipelines through the API, see [Create and manage jobs with the Grepr REST APIs](/apis/job-creation-guide).

After you've completed development of your pipeline, to find its JSON definition in the UI, click **Jobs** in the menu bar, click the three-dot menu in the **Actions** column for your pipeline, and click **Edit JSON** to view and copy the JSON. Save the copied JSON to a file in your Terraform project, for example, `pipeline.json`:

```json
{
  "vertices": [
    {
      "type": "datadog-log-agent-source",
      "name": "source",
      "integrationId": "<your-integration-id>"
    },
    {
      "type": "grok-parser",
      "name": "parser",
      "grokParsingRules": [
        "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
      ]
    },
    {
      "type": "logs-iceberg-table-sink",
      "name": "sink",
      "datasetId": "<your-dataset-id>"
    }
  ],
  "edges": ["source -> parser", "parser -> sink"]
}
```

`<your-integration-id>` and `<your-dataset-id>` are the identifiers of the Datadog integration and dataset used by this pipeline. If you copied the JSON from an existing pipeline, these fields are already set to the correct value.

Then reference the JSON file in your Terraform configuration:

```hcl
resource "grepr_pipeline" "example" {
  name           = "my_pipeline"
  job_graph_json = file("${path.module}/pipeline.json")
  desired_state  = "RUNNING"
}
```

Use `terraform init` to download the provider, `terraform plan` to preview your changes, and `terraform apply` to apply the configuration and create the pipeline. With the OpenTofu CLI, use the equivalent `tofu` commands:

```bash
terraform init
terraform plan
terraform apply
```

You can also define the job graph directly in the Terraform configuration using `jsonencode`:

```hcl
resource "grepr_pipeline" "example" {
  name = "my_pipeline"

  job_graph_json = jsonencode({
    vertices = [
      {
        type          = "datadog-log-agent-source"
        name          = "source"
        integrationId = var.datadog_integration_id
      },
      {
        type = "grok-parser"
        name = "parser"
        grokParsingRules = ["%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"]
      },
      {
        type      = "logs-iceberg-table-sink"
        name      = "sink"
        datasetId = var.dataset_id
      }
    ]
    edges = ["source -> parser", "parser -> sink"]
  })
}
```

## Resource attributes

The `grepr_pipeline` resource supports these attributes:

| Attribute | Type | Required | Description |
| --- | --- | --- | --- |
| `name` | String | Yes | The pipeline name. Must be lowercase alphanumeric with underscores, 1-128 characters. |
| `job_graph_json` | String | Yes | The job graph as a JSON string. Use `file()` to load from a JSON file or `jsonencode` to define inline. |
| `desired_state` | String | No | The desired state: `RUNNING` or `STOPPED`. Defaults to `RUNNING`. |
| `team_ids` | Set of Strings | No | Team IDs to associate with this pipeline. |
| `tags` | Map of Strings | No | Custom tags. See [Tags](#tf-pipeline-tags). |
| `wait_for_state` | Boolean | No | Wait for the pipeline to reach the desired state. Defaults to `true`. |
| `state_timeout` | Number | No | Timeout in seconds for state transitions. Defaults to `600`. |
| `rollback_enabled` | Boolean | No | Roll the pipeline back to its last stable configuration when an update fails. Defaults to `true`. |

### Read-only attributes

| Attribute | Type | Description |
| --- | --- | --- |
| `id` | String | The unique identifier of the pipeline. |
| `version` | Number | The current version of the pipeline. Increments on updates. |
| `state` | String | The actual current state of the pipeline. |
| `organization_id` | String | The organization ID that owns this pipeline. |
| `created_at` | String | The timestamp when the pipeline was created. |
| `updated_at` | String | The timestamp when the pipeline was last updated. |