Learn how to use a Datadog integration to ingest logs and traces into a Grepr pipeline for processing before forwarding to Datadog. # Configure Datadog connections with a Grepr integration The Grepr Datadog integration enables you to ingest telemetry data into Grepr using a Grepr source, process the data with a Grepr pipeline, and then send the processed data to Datadog using a Grepr sink. The Datadog integration supports logs and Application Performance Monitoring (APM) data, including traces and metrics. You can use the Grepr UI or REST API to create and manage Datadog integrations. This page describes using the UI. To use the REST API, see the [Datadog Integration specification](/apis/api-spec#tag/Datadog-Integration). To send logs or APM data to a Grepr pipeline for processing, change your collector configuration to send them to a Grepr endpoint URL instead of a Datadog endpoint. The collectors supported by the Datadog integration depend on the type of data: * Log ingestion is supported from Datadog-compatible collectors, including the Datadog Agent, Fluent Bit, and Fluentd. * APM data ingestion is supported from the Datadog Agent with the Datadog Trace Agent enabled. When forwarding processed data to Datadog, the integration supports sending traces or metrics. You can also optionally enable sending APM stats to Datadog. See [_](/pipelines/trace-pipelines#datadog-metrics-stats). This page includes example configurations for the supported collectors. You can also send logs or traces from the OpenTelemetry Collector to an OpenTelemetry integration that forwards them to a Datadog OTLP endpoint. See [_](/integrations/open-telemetry). > * Although the Grepr Datadog integration supports all Datadog sites, when you use the Datadog `US1` site in the `us-east-1` region, the Grepr SaaS offering uses an AWS PrivateLink connection to Datadog. > > * Although this page describes using an integration to create a pipeline for sourcing and sending data with a single observability platform, your pipelines can use multiple integrations to source and sink data for different platforms. For example, you can create a pipeline with a Datadog integration that's used to source data from Datadog agents, and a Splunk integration and sink that sends the pipeline output to Splunk. ## Requirements \[#datadog-requirements] Creating and configuring a Datadog integration requires the following: * A Datadog API key for your account. You must use the same API key in both your Grepr integration and collector configuration. * (Optional) A Datadog application key for your account. The required authorization scopes depend on which features you want to use: \| Feature | Required scopes | |---|---| | Selective reduction with automatic exception parsing | `monitors_read`, `dashboards_read`, `logs_read_config` | | Parsed query impact estimation against your indexed Datadog logs | `logs_read_data` | | Datadog as a [cloud source](/integrations/support-matrix/#supported-observability-integrations) | `logs_read_data` | To learn more about exception parsing, see [_](#enable-selective-reduction). To learn more about Datadog keys and scopes, see [API and Application Keys](https://docs.datadoghq.com/account_management/api-app-keys/) and [Authorization Scopes](https://docs.datadoghq.com/api/latest/scopes/). ## Enable selective reduction with a Datadog integration \[#enable-selective-reduction] When you include a Datadog application key in your integration configuration, Grepr automatically tracks all log monitors configured in your Datadog account. By parsing queries used in these monitors and making them available as exceptions to Grepr's log reducer, Grepr can determine which messages should not be processed by the reducer. The application key must have the required authorization scopes. See [_](#datadog-requirements). To learn more about viewing parsed queries and determining which should be used as exceptions, see [_](/integrations#passthrough-estimation). To learn more about Grepr log reduction, see [_](/transforms/reducer). ## Use a Datadog integration in a Grepr pipeline \[#use-datadog-integration] Use the following steps to create a Datadog integration in the Grepr UI and use it in a pipeline to source and sink data. ### Step 1: Create a Datadog integration To create a Datadog integration in the Grepr UI: 1. Go to the **Integrations** page in the Grepr UI and click **Add new** next to **Observability Vendors**. 2. In the **Add Observability Vendor** dialog: * In the **Type** menu, select **Datadog**. * In the **Name** field, enter a name for the integration. * In the **Site** menu, select the parameter matching your Datadog site. * In the **API Key** field, enter your Datadog API key. * Optional: In the **Application Key** field, enter your Datadog Application key. The application key is required only to import exceptions from Datadog, to estimate their impact against your indexed Datadog logs, or to use the integration as a cloud source. 3. Click **Create**. #### Optional: Grant team access to the integration When you have teams in your Grepr organization, you can grant access to the integration to one or more of those teams. To configure team access when you create or edit an integration: 1. In the **Grant Team Access** section, click the plus sign icon (+). 2. Select the team or teams that should be granted access to the integration. 3. Click **Create**. To learn more, see [Manage teams in the Grepr platform](/admin/manage-teams) and [Permissions in the Grepr platform](/grepr-platform/permissions). ### Step 2: Add a source to your pipeline To add a source to your pipeline that uses the Datadog integration, select the integration when creating the pipeline. For an existing pipeline, go to the pipeline details page, click **Sources** in the left-hand navigation menu, and select the integration. ### Step 3: Configure forwarding to Grepr \[#configure-forwarding-to-grepr] To send data to Grepr, you configure your collector or agent to send it to the ingestion URL for your Grepr integration, rather than to your observability vendor's platform. To find the ingestion URL when using the Grepr UI to configure your pipeline, go to the details page for the pipeline, click **Sources** in the left pane, and copy the **Ingest URL**. If you connect to Grepr through AWS PrivateLink, copy the private ingestion URL and use it in place of the public ingestion URL. To learn more about sending data to Grepr over AWS PrivateLink, see [Securely forward observability data to Grepr with AWS PrivateLink](/admin/private-link). When you use the API, you can construct the ingestion URL using the following format: `https://-.ingest.grepr.ai` Replace `` with the identifier of the vendor integration used by your pipeline and replace `` with the identifier for your organization. To use the PrivateLink endpoint service, insert `.private` after the `ingest` host segment. Although all sources created from the same integration share the same ingestion URL, you can use a single integration to create sources for multiple pipelines. The Grepr platform ensures events are routed to the correct pipelines. In the following examples, replace `` with the ingest URL, `` with its hostname, and `` with the Datadog API key you configured in your Grepr integration. Each example marks the settings needed for logs and the settings needed for APM data separately, so you can configure either one or both. Each tab also shows how to dual ship, which sends your data to both Datadog and Grepr so you can compare the two and confirm that Grepr-processed data is correct before you cut over. To see more information and configuration examples, select the tab for your preferred collector: To send logs and APM data to Grepr, add Agent environment variables in `spec.override.nodeAgent.env` as shown below. Restart the Agent for the changes to be deployed. To learn more about the fields of the custom resource, see the [DatadogAgent configuration reference](https://github.com/DataDog/datadog-operator/blob/main/docs/configuration.v2alpha1.md). * Logs: add `DD_LOGS_CONFIG_LOGS_DD_URL` set to the ingest URL and `DD_LOGS_CONFIG_USE_HTTP` set to `"true"`. Grepr accepts HTTP and HTTPS only. * APM: add `DD_APM_DD_URL` set to the ingest URL. ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: global: site: datadoghq.com credentials: apiKey: features: logCollection: enabled: true override: nodeAgent: env: # --- Logs — omit if you are forwarding APM data only --- - name: DD_LOGS_CONFIG_LOGS_DD_URL value: "" - name: DD_LOGS_CONFIG_USE_HTTP value: "true" # --- APM — omit if you are forwarding logs only --- - name: DD_APM_DD_URL value: "" ``` > Every `value` in the `spec.override.nodeAgent.env` list must be a string. Quote boolean and numeric values, such as `value: "true"`, because Kubernetes rejects non-string environment variable values. To dual ship to Datadog and Grepr, add Grepr as an additional endpoint. See [Dual Shipping](https://docs.datadoghq.com/agent/configuration/dual-shipping/). ```yaml # Leave the intake URLs unset so data keeps flowing to Datadog. spec: override: nodeAgent: env: # --- Logs — omit if you are forwarding APM data only --- - name: DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS value: '[{"api_key":"","Host":"","Port":443}]' # --- APM — omit if you are forwarding logs only --- - name: DD_APM_ADDITIONAL_ENDPOINTS value: '{"":[""]}' ``` To send logs and APM data to Grepr, add Agent environment variables to the `datadog.env` list as shown below. Restart the Agent for the changes to be deployed. To learn more about the chart values, see the [Datadog Helm chart](https://github.com/DataDog/helm-charts/tree/main/charts/datadog). * Logs: add `DD_LOGS_CONFIG_LOGS_DD_URL` set to the ingest URL and `DD_LOGS_CONFIG_USE_HTTP` set to `"true"`. Grepr accepts HTTP and HTTPS only. * APM: add `DD_APM_DD_URL` set to the ingest URL. ```yaml datadog: apiKey: site: datadoghq.com logs: enabled: true env: # --- Logs — omit if you are forwarding APM data only --- - name: DD_LOGS_CONFIG_LOGS_DD_URL value: "" - name: DD_LOGS_CONFIG_USE_HTTP value: "true" # --- APM — omit if you are forwarding logs only --- - name: DD_APM_DD_URL value: "" ``` > Every `value` in the `datadog.env` list must be a string. Quote boolean and numeric values, such as `value: "true"`, because the Helm chart renders the list directly into the agent pod specification and Kubernetes rejects non-string environment variable values. To dual ship to Datadog and Grepr, add Grepr as an additional endpoint. See [Dual Shipping](https://docs.datadoghq.com/agent/configuration/dual-shipping/). ```yaml # Leave the intake URLs unset so data keeps flowing to Datadog. datadog: env: # --- Logs — omit if you are forwarding APM data only --- - name: DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS value: '[{"api_key":"","Host":"","Port":443}]' # --- APM — omit if you are forwarding logs only --- - name: DD_APM_ADDITIONAL_ENDPOINTS value: '{"":[""]}' ``` To send logs to Grepr, update the Datadog output plugin as shown below. Restart Fluent Bit for the changes to be deployed. Fluent Bit does not collect APM data, so this configuration covers logs only. To learn more about configuration settings for Fluent Bit, see the documentation for the [Fluent Bit Datadog output plugin](https://docs.fluentbit.io/manual/data-pipeline/outputs/datadog). * In the `OUTPUT` section, set `Host` to the ingest hostname. The plugin expects a hostname without a scheme, and `TLS On` selects HTTPS. ```fluent [OUTPUT] Name datadog Match * Host TLS On apikey # Optional: Specify service, source, tags, host # dd_service # dd_source # dd_tags # dd_hostname ``` To dual ship to Datadog and Grepr, leave your existing `OUTPUT` section in place and add a second one for Grepr that matches the same tags. ```fluent [OUTPUT] Name datadog Match * Host http-intake.logs.datadoghq.com TLS On apikey [OUTPUT] Name datadog Match * Host TLS On apikey ``` To send logs to Grepr, update the `fluent-plugin-datadog` output plugin as shown below. Restart Fluentd for the changes to be deployed. Fluentd does not collect APM data, so this configuration covers logs only. To learn more about configuration settings for Fluentd, see the documentation for the [Fluentd Datadog output plugin](https://docs.datadoghq.com/integrations/fluentd/?tab=host). * In the `match` directive, set `host` to the ingest hostname. The plugin expects a hostname without a scheme, and it connects over SSL by default. ```fluent # Adjust your match pattern as needed @type datadog api_key host # Optional: Specify service, source, or tags # service # source # tags # Buffer configuration is recommended for production @type memory # Or file for persistence flush_interval # Adjust as needed chunk_limit_size # Adjust as needed # Add other buffer parameters as needed ``` Replace: * `your_tag.**` with the tag of the logs you want to forward to Grepr or use `**` to match all tags. To dual ship to Datadog and Grepr, use the `copy` output plugin with one `store` for each destination. Omitting `host` sends logs to the default Datadog intake. ```fluent @type copy @type datadog api_key @type datadog api_key host ``` To send logs and APM data to Grepr, update `datadog.yaml` as shown below. Restart the Agent for the changes to be deployed. To learn more about standalone agent configuration, see [Host Agent Log collection](https://docs.datadoghq.com/agent/logs/). * Logs: set `logs_config.logs_dd_url` to the ingest URL and `logs_config.force_use_http` to `true`. Grepr accepts HTTP and HTTPS only. * APM: set `apm_config.apm_dd_url` to the ingest URL. ```yaml api_key: # --- Logs — omit if you are forwarding APM data only --- logs_enabled: true logs_config: logs_dd_url: force_use_http: true # --- APM — omit if you are forwarding logs only --- apm_config: apm_dd_url: ``` To dual ship to Datadog and Grepr, add Grepr as an additional endpoint. See [Dual Shipping](https://docs.datadoghq.com/agent/configuration/dual-shipping/). ```yaml # Leave the intake URLs unset so data keeps flowing to Datadog. # --- Logs — omit if you are forwarding APM data only --- logs_config: additional_endpoints: - api_key: Host: Port: 443 # --- APM — omit if you are forwarding logs only --- apm_config: additional_endpoints: '': - ``` The Datadog Forwarder is an AWS Lambda function that forwards logs from AWS services to Datadog. You can instead forward logs to Grepr for processing by modifying the Lambda Forwarder's configuration. These instructions detail how to use AWS CloudFormation, recommended by Datadog, to configure and deploy the forwarder. See [Datadog Forwarder](https://docs.datadoghq.com/logs/guide/forwarder). > Grepr requires the Datadog Forwarder version 5.0.0 or later. To deploy the Datadog Forwarder using CloudFormation, follow the [Datadog Forwarder installation instructions](https://docs.datadoghq.com/logs/guide/forwarder/?tab=cloudformation#installation). Before creating the CloudFormation stack, configure the following parameters to send logs to Grepr: | CloudFormation Parameter | Lambda Environment Variable | Value | Description | | --- | --- | --- | --- | | `ddUrl` | `DD_URL` | `` | The Grepr ingestion URL without the `https://` prefix. | | `ddPort` | `DD_PORT` | `443` | Port for HTTPS connections to Grepr. Set this value to `443`. | | `ddApiKey` | `DD_API_KEY` | `` | Your Datadog API key. | | `ddLogLevel` | `DD_LOG_LEVEL` | `DEBUG` (optional) | Set to `DEBUG` to enable detailed logging. | > * The `ddUrl` parameter must not include the `https://` prefix. Use only the hostname portion of your ingestion URL. > * The `` value must be the same Datadog API key you configured in your Grepr Datadog integration. Alternatively, configure `DdApiKeySecretArn` if the key is stored in AWS Secrets Manager or `DdApiKeySsmParameterName` if the key is in AWS Systems Manager Parameter Store. > * Parameter names use camelCase when configuring the CloudFormation stack and UPPERCASE_UNDERSCORE when viewing or editing the Lambda function's environment variables in the AWS Console. To send logs through the Datadog Forwarder after deployment, you set up triggers on your AWS services to run the forwarder whenever a new batch of log events is available. See [Send AWS Services Logs With The Datadog Lambda Function](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function/?tab=awsconsole). To dual ship to Datadog and Grepr, deploy a second forwarder configured for Grepr, then set `DD_ADDITIONAL_TARGET_LAMBDAS` on your existing forwarder to the ARN of the new one. Your existing forwarder and its triggers stay unchanged: ```bash DD_ADDITIONAL_TARGET_LAMBDAS=arn:aws:lambda:us-east-1:123456789012:function:datadog-forwarder-grepr ``` > To specify multiple additional forwarders, separate ARNs with commas. > If logs are not appearing in Grepr, the following are suggested troubleshooting steps: > > * Check the Lambda function's CloudWatch Logs for errors. > * Set `ddLogLevel`/`DD_LOG_LEVEL` to `DEBUG` for more detailed logging. > * Verify the ingestion URL does not include the `https://` prefix. > * Confirm the API key matches the one in your Grepr Datadog integration. > * Ensure you are using Datadog Forwarder version 5.0.0 or later. To send logs and traces to Grepr, configure the Datadog exporter as shown below. Restart the Collector for the changes to be deployed. See [Install and Configure the OpenTelemetry Collector](https://docs.datadoghq.com/opentelemetry/setup/collector_exporter/). * Logs: set `logs.endpoint` to the ingest URL and add the exporter to a `logs` pipeline. * Traces: set `traces.endpoint` to the ingest URL and add the exporter to a `traces` pipeline. ```yaml exporters: datadog/grepr: api: key: fail_on_invalid_key: false # --- Logs — omit if you are forwarding traces only --- logs: endpoint: # --- Traces — omit if you are forwarding logs only --- traces: endpoint: service: pipelines: # --- Logs — omit if you are forwarding traces only --- logs: receivers: [otlp] # Replace with your actual receiver(s) processors: [batch] # Replace/add your actual processor(s) exporters: [datadog/grepr] # --- Traces — omit if you are forwarding logs only --- traces: receivers: [otlp] processors: [batch] exporters: [datadog/grepr] ``` To dual ship to Datadog and Grepr, leave your existing Datadog exporter in place and list both exporters in the pipeline. ```yaml exporters: datadog/datadog: api: key: fail_on_invalid_key: false service: pipelines: logs: receivers: [otlp] processors: [batch] exporters: [datadog/datadog, datadog/grepr] ``` To send logs and APM data to Grepr, pass the Agent settings as environment variables on the container as shown below. To assign a custom name to the container, use the `--name` argument. To learn more about configuring Docker and other container agents, see [Container Monitoring](https://docs.datadoghq.com/containers/). * Logs: set `DD_LOGS_CONFIG_LOGS_DD_URL` to the ingest URL and `DD_LOGS_CONFIG_USE_HTTP` to `true`. Grepr accepts HTTP and HTTPS only. * APM: set `DD_APM_DD_URL` to the ingest URL. ```bash # Omit the DD_LOGS_* variables if you are forwarding APM data only, # or the DD_APM_* variables if you are forwarding logs only. docker run -d --rm --name \ -e DD_API_KEY= \ -e DD_SITE="datadoghq.com" \ -v /var/run/docker.sock:/var/run/docker.sock:ro \ -v /proc/:/host/proc/:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e DD_LOGS_ENABLED=true \ -e DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true \ -e DD_LOGS_CONFIG_LOGS_DD_URL= \ -e DD_LOGS_CONFIG_USE_HTTP="true" \ -e DD_APM_ENABLED="true" \ -e DD_APM_DD_URL= \ gcr.io/datadoghq/agent:latest ``` To dual ship to Datadog and Grepr, add Grepr as an additional endpoint. See [Dual Shipping](https://docs.datadoghq.com/agent/configuration/dual-shipping/). ```bash # Leave the intake URLs unset so data keeps flowing to Datadog. # Omit the DD_LOGS_* variable if you are forwarding APM data only, # or the DD_APM_* variable if you are forwarding logs only. -e DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS='[{"api_key":"","Host":"","Port":443}]' \ -e DD_APM_ADDITIONAL_ENDPOINTS='{"":[""]}' ``` ## Forward events from an Azure event hub to Grepr If you have an existing integration that sends events from an Azure event hub to Datadog, you can modify it to forward events to Grepr for processing before sending them to Datadog. An example use of this event hub-to-Datadog integration is when the event hub is used to queue log events from other Azure services. These log events are then forwarded to a Datadog endpoint for processing. You can include Grepr in this integration by changing the destination URL from the Datadog endpoint to a Grepr ingestion URL, allowing you to use Grepr's log processing features, such as log reduction, parsing, and enrichment on the events before they reach Datadog. Adding Grepr to an event hub-to-Datadog integration requires: * An existing Azure Event Hubs-to-Datadog integration. The integration must be implemented to send events to a Datadog HTTP endpoint. Grepr does not support TCP. * A Datadog API key for your account. When sending events to Grepr, you must use the same API key that's used for the direct Datadog integration. ### Configure event forwarding to Grepr To integrate Grepr into your Event Hubs-to-Datadog integration: 1. Create a Grepr Datadog integration, add the integration to a Grepr pipeline, and create a source. For step-by-step instructions, see [_](#use-datadog-integration). 2. Get the ingestion URL for the Grepr integration by following the instructions in [_](#configure-forwarding-to-grepr), then skip the instructions for configuring collectors. 3. In the Azure component that forwards events to Datadog, such as an Azure Function, replace the Datadog endpoint URL with the Grepr ingestion URL and include the path from the Datadog URL. For example, replace: ```cs var eventsEndpoint = "https://http-intake.logs.datadoghq.com/api/v2/logs"; ``` with: ```cs var eventsEndpoint = "https://1234567890123-yourorg.ingest.grepr.ai/api/v2/logs"; ``` 4. Ensure the existing Datadog API key is also included in the request sent to Grepr. After you make this configuration change, you can use your existing Grepr pipelines or create new pipelines to process the events before they are forwarded to Datadog. ## Limitations * The Datadog integration supports only HTTP/HTTPS connections. TCP connections are not supported. * The Datadog integration does not support the use of [Datadog log pipelines](https://docs.datadoghq.com/logs/processing/pipelines/). Grepr processes and routes all logs before sending them to Datadog. ## Frequently Asked Questions ### How do I send Datadog logs to Grepr? Configure your log collector (Datadog Agent, Fluent Bit, Fluentd, or OpenTelemetry Collector) to forward logs to your Grepr ingest endpoint URL. ### What is the Grepr Datadog ingestion endpoint format? The endpoint format is https\://\-\.ingest.grepr.ai ### How do I reduce Datadog logging costs with Grepr? Use Grepr's log reduction and filtering to lower log volume before sending to Datadog. ### Can I use Grepr with the Datadog Agent? Yes, configure logs_config.logs_dd_url to your Grepr endpoint and set force_use_http to true. ### How do I send logs to Grepr when I deploy the Datadog Agent with the Helm chart? Add DD_LOGS_CONFIG_LOGS_DD_URL and DD_LOGS_CONFIG_USE_HTTP to the datadog.env list in your values file, setting the URL to your Grepr ingestion URL. ### How do I send logs to Grepr when I deploy the Datadog Agent with the Datadog Operator? Add DD_LOGS_CONFIG_LOGS_DD_URL and DD_LOGS_CONFIG_USE_HTTP to spec.override.nodeAgent.env in your DatadogAgent custom resource, setting the URL to your Grepr ingestion URL. ### How do I configure Fluent Bit to send logs to Grepr instead of Datadog? In the Fluent Bit OUTPUT section, set the Host to your Grepr ingestion hostname, without the https\:// scheme, and include your Datadog API key. ### Does Grepr support the OpenTelemetry Collector with Datadog? Yes, configure the Datadog exporter with your Grepr endpoint in the logs.endpoint setting. ### What is selective reduction with Datadog? When you provide a Datadog application key, Grepr automatically tracks your Datadog log monitors and excludes matching logs from reduction. ### Can I forward Azure Event Hub logs through Grepr to Datadog? Yes, replace the Datadog endpoint URL in your Azure function with your Grepr ingestion URL. ### Does Grepr support the Datadog Forwarder Lambda? Yes, Grepr supports Datadog Forwarder version 5.0.0 or later. Set DD_URL to your Grepr ingestion hostname. ### What protocols does the Grepr Datadog integration support? Only HTTP and HTTPS. TCP connections are not supported. ### Does the Grepr Datadog integration support traces and APM data? Yes. The integration ingests APM data, including traces and metrics, from the Datadog Agent with the Datadog Trace Agent enabled, and can forward processed traces or metrics to Datadog. ### How do I send Datadog traces to Grepr? Enable the Datadog Trace Agent and set apm_config.apm_dd_url, or the DD_APM_DD_URL environment variable, to your Grepr ingestion URL. The agent then sends trace data to Grepr at the /api/v0.2/traces path. ### Can Grepr forward Datadog APM stats? Yes. When you use a Datadog source, you can choose whether Grepr forwards Datadog APM stats, the Rate, Errors, and Duration (RED) metrics that the Datadog Agent computes before sampling, to Datadog unmodified so that your APM views stay accurate even when traces are sampled.