Use actions in a Grepr pipeline to drop or keep events by condition, drop late-arriving events, and process events with SQL. # Filter and transform events with actions You can use *actions* in a Grepr pipeline to decide what happens to log events at a specific point in the pipeline. With actions, you can: * Drop events, or keep only the events that match a condition. * Drop late-arriving events. * Process all or some events with SQL, and send the results to a later pipeline step. Conditions use familiar query syntaxes, such as the Datadog-like syntax or the New Relic log query-like syntax. To learn more about the supported query syntaxes, see [Query log data in Grepr](/queries). ## Where actions run in a pipeline \[#locations] Actions are available at three locations in a pipeline. Each location is named for the pipeline step it runs before: | Location | Runs before | CLI phase | | --- | --- | --- | | **Pre-Parsing Actions** | Parsing | `pre-parser` | | **Pre-Warehouse Actions** | Masking and the Data Lake | `pre-warehouse` | | **Pre-Reducer Actions** | Exceptions and the Reducer | `pre-exceptions` | Choose the earliest location that has the fields you need. An event dropped at **Pre-Parsing Actions** isn't parsed, stored, or reduced. An event dropped at **Pre-Warehouse Actions** is parsed but isn't stored in the data lake. An event dropped at **Pre-Reducer Actions** is stored in the data lake but isn't reduced or sent to your sinks. > **Pre-Warehouse Actions** run before masking. Conditions and SQL at this location see the original values of any fields that your masking rules replace. ## Add actions to a pipeline \[#add] 1. In the Grepr UI, go to the details page for your pipeline. 2. In the left pane, select **Pre-Parsing Actions**, **Pre-Warehouse Actions**, or **Pre-Reducer Actions**. 3. Click **Add Action**, and then select one of the following: * **Condition (If/Else)**: Sends events down different branches based on a query. See [Drop or keep events with conditions](#conditions). * **SQL**: Processes events with SQL. See [Process events with SQL](#sql). * **Drop late-arriving events**: Drops events that arrive too long after their timestamp. See [Drop late-arriving events](#late-data). 4. To save the change to your pipeline, click **Apply Changes**. Before you apply a change, you can test it against live data with the Pipeline Log Viewer. See [Test pipeline changes with live edit](/tutorials/live-edit). ## Drop or keep events with conditions \[#conditions] A condition sends each event down exactly one branch. The first branch whose query matches the event handles it, and an event that matches no query goes to the **ELSE** branch. * To add a condition, enter a query in the **IF** row. * To test events against more queries, add **ELSE IF** branches. Branches are checked from top to bottom, so you can reorder them to change which query handles an event first. * Each branch ends with either **Drop**, which removes the events from the pipeline, or the next pipeline step, which sends the events on. To change how a branch ends, click its ending and select the other option. * To process the events on one branch with SQL, insert a SQL action into that branch. For example, to drop debug events and keep everything else, add a condition with the query `status:debug`, end the **IF** branch with **Drop**, and end the **ELSE** branch with the next pipeline step. ## Process events with SQL \[#sql] A SQL action processes events with ANSI standard SQL queries. Each SQL action contains one or more statements. Each statement has a name and a SQL query that reads input events from the `logs` table or from an earlier statement by its name. To send the results of a statement to a pipeline step, connect the statement to that step. Depending on the location of the action, you can send results to steps such as the parsers, the Data Lake, the Reducer, or the pipeline sinks. Statements that aren't connected to a step are available only to later statements. A SQL action doesn't remove events from the pipeline. Events continue to the rest of the branch, and the results of connected statements are sent to the connected steps in addition. > The results of a SQL statement contain only the fields you select. To keep all fields from the original event, include the wildcard character (`*`) in your `SELECT` statement. For example, to transform the severity while keeping all other fields: `SELECT *, CASE WHEN severity > 16 THEN 21 ELSE severity END as severity FROM logs`. To learn more about using SQL in Grepr pipelines, see [Transform events with SQL](/transforms/sql-transform). ## Drop late-arriving events \[#late-data] You can drop events whose received time is later than their event timestamp by more than a maximum duration. Enter the maximum duration in the ISO-8601 format, such as `PT48H` for 48 hours. This is useful when timely data is critical, and late-arriving events could affect analysis or reporting. The late-data rule applies to every event at that location, before any condition is checked. ## Frequently Asked Questions ### What are actions in a Grepr pipeline? Actions decide what happens to log events at a specific point in a pipeline. You can use actions to drop events, keep only events that match a condition, drop late-arriving events, or process events with SQL and send the results to a later pipeline step. ### Where are actions located in a Grepr pipeline? Actions are available at three locations, each named for the step it runs before: Pre-Parsing Actions run before parsing, Pre-Warehouse Actions run before masking and the data lake, and Pre-Reducer Actions run before exceptions and the reducer. In the Grepr CLI, these locations are the pre-parser, pre-warehouse, and pre-exceptions phases. ### What happened to pipeline filters? The pipeline steps named Pre-parsing filter, Post-parsing filter, and Post-warehouse filter are named Pre-Parsing Actions, Pre-Warehouse Actions, and Pre-Reducer Actions. They run at the same locations in the pipeline. ### How do I drop events I don't need? Add a Condition (If/Else) action with a query that matches the events to drop, end the IF branch with Drop, and end the ELSE branch with the next pipeline step. ### How do I filter out late-arriving events? Add the Drop late-arriving events action and enter a maximum duration in the ISO-8601 format. Events whose received time is later than their timestamp by more than this duration are dropped. ### What query languages does Grepr support for conditions? Conditions support multiple query syntaxes, including the Datadog-like syntax and the New Relic log query-like syntax. SQL actions use ANSI standard SQL. ### Does a SQL action remove events from the pipeline? No. Events continue to the rest of the branch after a SQL action. The results of each SQL statement that you connect to a pipeline step are sent to that step in addition. ### Can I route different event types to different pipeline paths? Yes. Use a condition with a branch for each event type, or add several SQL statements and connect each one to a different pipeline step.