Learn how to safely test data transformation changes to a running production pipeline with the Grepr live edit feature.

# View pipeline data flows and test changes with live edit

The Grepr live edit feature allows you to:

* Safely test changes to data transformations in a running production pipeline. Using live edit, you can make and test changes to the pipeline's data processing and transformations without impacting the processing of live data and the flow of that data to your observability tools. After you've validated your changes, you can apply those changes to the production pipeline.
* See a live view of how data flows and is processed by the running production pipeline. You can use this capability to monitor the data flowing through your pipeline in real time, even if you're not making changes to the pipeline.

The live edit feature works by creating a temporary copy of the current pipeline but excludes the sink, preventing duplicate data from being sent to your observability platform. Your changes run in the copy of the pipeline. The state of the data at each step of the copied pipeline can be seen in the live edit view. You can also use this view to compare the results of processing between the original and copy pipelines, allowing you to get fast feedback on your changes.

> To ensure that the data passing through the pipeline copy doesn't exceed the UI's capacity, the pipeline copy receives a sampling of the pipeline data. To ensure the pipeline copy is receiving the data you need and the output data contains only the fields you need, you can use filter functionality in the live edit UI. See [Create a subset of data for testing using filters](#filter-source-data).

Any changes you make that are not applied to the production pipeline are visible only in the pipeline copy, and no data is sent to configured sinks or data lakes. When your changes are tested and ready to apply to production, you can apply them to the original pipeline.

## Using live edit with your pipelines

To use live edit:

1. In the Grepr UI, go to the details page for your pipeline.

2. In the lower right corner of the page, click the expand icon to open the **Pipeline Log Viewer**.

3. To create and start a temporary copy of your pipeline, in the expanded **Pipeline Log Viewer** pane, click the play icon.

4. Grepr creates a temporary copy of your pipeline and runs it using log events sampled from the original pipeline's source data. The **Pipeline Log Viewer** pane updates to show the status of the pipeline copy as it's created and started.

5. When the temporary pipeline copy is provisioned and started, the **Pipeline Log Viewer** pane displays the raw logs sampled from the original pipeline's source data. You can use this view to examine logs at different stages of processing, and to test and validate changes to processing.

To examine logs at different stages of processing, use the **Left table link** or **Right table link** menus to select the stage of processing to view.

To view details of a log event, click the event in the **Pipeline Log Viewer** pane to open the log detail pane. The log detail pane displays the contents of the selected log event at the selected processing steps. The top bar of the log detail pane displays the event type, such as `INFO`, `ERROR`, or `DEBUG`, and the timestamp of the log message. To view the state of the event in the selected processing step in the left table, click **LEFT**. To view the state of the event in the selected processing step in the right table, click **RIGHT**. Click **DIFF** to view a JSON comparison of the event state in the left and right panels.

## Create a subset of data for testing using filters \[#filter-source-data]

Because the pipeline copy is receiving a sampling of the source data to ensure that the UI's capacity is not exceeded, the **Pipeline Log Viewer** supports functionality to filter source data. This filtering functionality ensures you get the correct data required for your testing. You can also use this feature to select specific columns to include in tables created from the filtered data.

To use the filter functionality, click the filter icon next to the stop icon. In the filter dialogue, select your preferred query syntax from the **Query language** menu, and enter your filter query in the **Sampler query** text box. To select the columns you want to see in a table created from the filtered data, add and remove column names in the **Table Columns** input.

## Example: Use live edit to add and test a pipeline filter

You can use a local file containing logs for testing instead of streaming live data from the pipeline. This feature can be useful for tasks such as

> * This example uses the pipeline created in the [Build your first Grepr pipeline](/tutorials/first-pipeline) tutorial. If you want to follow the example, first complete that tutorial.
> * Although this example uses Datadog, you can use the live edit feature with any Grepr-supported observability platform. For the complete list of supported observability vendors, see [Supported integrations](/integrations/support-matrix).

In this example, you use the following steps to test an action that drops any log messages received by your Grepr pipeline that are assigned the hostname `my-test-host-1`.

1. In the Grepr UI, go to the details page for your running pipeline.
2. In the left pane, click **Pre-Parsing Actions**, click **Add Action**, and select **Condition (If/Else)**.
3. Make sure the query language is set to **Datadog**, and enter the following query in the **IF** row:

```bash
host:my-test-host-1
```

1. End the **IF** branch with **Drop**, and leave the **ELSE** branch ending with the next pipeline step.

2. *Do not* click **Apply Changes**. Before applying the changes to the production pipeline, use the live edit feature to test and validate the action.

3. In the lower right corner of the page, click the expand icon to open the **Pipeline Log Viewer** and click the play icon to create and start a temporary copy of your pipeline with the action added.

4. In the **Pipeline Log Viewer** left table, click a log message from `my-test-host-1`. The log detail pane opens. Because the action drops the log messages from the pipeline, only the **LEFT** table view is displayed.

5. When you complete validation of your change, click **Apply Changes** to save the new configuration to the production pipeline.

## Use a local file for testing

You can use log events in a local file for testing instead of streaming live data from the pipeline. This can be useful when you want to test pipeline changes against a consistent set of log events, or if you want to test with specific log events that are not currently flowing through the pipeline.

> The log file must be formatted as JSON or plain text, and must not exceed 10 MB in size.

To upload a file for testing:

1. In the **Pipeline Log Viewer** pane, next to the play icon, click **Live** and click the **UPLOAD** tab.
2. Drag and drop the file to upload, or click the upload area to select the file.
3. Click **Apply**

## Frequently Asked Questions

### What is the Grepr live edit feature?

Live edit is a feature that allows you to safely test changes to data transformations in a running production pipeline without impacting live data. It creates a temporary copy of your pipeline that excludes sinks, preventing duplicate data from being sent to your observability platform while you test changes.

### How do I start using live edit?

To use live edit, go to the details page for your running pipeline, click the expand icon in the lower right corner to open the Pipeline Log Viewer, and then click the play icon to create and start a temporary copy of your pipeline with sampled data.

### Will live edit changes affect my production pipeline?

No. Changes made in live edit are only visible in the temporary pipeline copy and do not affect production. No data is sent to configured sinks or data lakes until you explicitly apply the changes to the production pipeline.

### What is the Pipeline Log Viewer?

The Pipeline Log Viewer is a pane in the pipeline details that displays logs sampled from the original pipeline's source data. It allows you to examine logs at different processing stages, see how transformations affect data, and compare results between the original and test pipelines.

### Can I see how data flows through each stage of my pipeline?

Yes. Use the Pipeline Log Viewer to examine logs at different stages of processing. Select the processing step from the Left table link or Right table menu to see data at that point. Click a log event to open the detail pane and view its state at different processing steps.

### Why does live edit use sampled data instead of all pipeline data?

Live edit uses sampled data to ensure the pipeline copy doesn't exceed the UI's capacity. This prevents performance issues while still providing representative data for testing. You can use filter functionality to select specific data needed for your testing.

### How do I filter the data shown in Pipeline Log Viewer?

Click the filter icon next to the stop icon in the Pipeline Log Viewer. In the filter dialog, select your preferred query language syntax and enter your filter query in the Sampler query text box. You can also add or remove column names in the Table Columns input to customize the displayed data.

### What does DIFF view show in the log detail pane?

DIFF view shows a JSON comparison of an event's state between the left and right panels of the Pipeline Log Viewer. This allows you to see exactly what changed when processing logs through different pipeline stages or comparing original versus modified pipelines.

### How do I apply tested changes to my production pipeline?

After you've validated your changes in live edit, click 'Apply Changes' in the pipeline editor to save the new configuration to the production pipeline. Your changes will be applied to the running pipeline.

### Can I use live edit to monitor data without making changes?

Yes. You can use live edit to see a live view of how data flows and is processed by your running production pipeline without making any modifications. This allows you to monitor pipeline behavior and performance in real-time.