Learn how Grepr indexes log attributes and tags as columns in the data lake, and the rules that decide which fields each dataset indexes.

# Indexed columns

Grepr stores each raw log's attributes as one JSON document and its tags as a map. A search that filters on one attribute reads every log's whole attribute document to find a match.

An *indexed column* removes that cost for one field. Grepr writes the field into its own column in the dataset's data lake table, so a search that filters on the field reads that column and skips the files that can't contain a match. See [The Grepr data lake](/grepr-platform/data-lake).

Searches don't change. The same search returns the same results whether or not a field is indexed, and runs faster when it is.

## What Grepr indexes

Each dataset's raw log table has indexing settings for two groups of fields:

* Attributes, named by their dotted path, such as `http.method` or `kube.pod.name`. An indexed path has a column for its value, and a path that holds numbers can also have a numeric column, so that range filters skip files too.
* Tags, named by their key, such as `region` or `pod_name`. An indexed tag key has a column for its value.

Grepr always indexes the `service` and `host` tags, whatever the settings say.

For each group, you choose whether Grepr discovers new fields, which fields it always indexes, and which fields it never indexes.

## Discovery \[#discovery]

When discovery is on, Grepr adds a column for a new field it finds in your logs, until the group has as many columns as its *discovery cap*. Your organization sets the largest cap you can choose. The tag cap doesn't count `service` and `host`.

For attributes, the cap counts every attribute column, including the ones for paths on the always-index list, so to make room for more paths, raise the cap.

Discovery is off for a new dataset, so until you turn it on, Grepr indexes only `service`, `host`, and the tag keys on the always-index list. Attribute paths on the always-index list get columns after you turn discovery on. Discovery then also indexes other attribute paths it finds, so to index only the paths you list, turn on **Exclude all by default** as well. See [Always index and never index](#lists).

When discovery is on and Grepr finds a number at an indexed path, the path also gets a numeric column. A numeric column doesn't count toward the cap.

Turning discovery off stops Grepr from adding columns for new fields. Columns the table already has keep being written. To stop writing a column, add the field to the never-index list.

## Always index and never index \[#lists]

Grepr stops writing a field on a never-index list.

For attribute paths:

* A path matches exactly and is case-sensitive. Write the path as it appears in the log.
* An entry that includes nested paths covers the path and every path under it, split at dots. For example, `kube` with nested paths covers `kube` and `kube.pod.name`, but not `kubernetes`. Wildcards aren't supported.
* A path on the always-index list counts toward the cap. An exact path gets its column when you save, if the table is under its cap; if it isn't, the path gets its column when you save a higher cap. Paths under a nested entry are discovered like any other path, so they get columns as pipelines find them, up to the cap.
* An always-index entry marked numeric gets its numeric column when you save, even when the table has reached its cap, as long as the path has its value column. Use it for values that are numbers written as strings. Only an exact path can be marked numeric: a path under a nested entry gets its numeric column when Grepr finds a number there.
* Always index wins over never index. To stop indexing a subtree except for one path, add the subtree to the never-index list with nested paths, and add the one path to the always-index list. An always-index entry can't cover a never-index entry entirely, because the never-index entry would then never take effect.
* To index only the paths on the always-index list, turn on **Exclude all by default**. Columns the table already has for other paths stop being written and are removed from the table on the same schedule as a field on a never-index list. See [Excluded fields and removed columns](#excluded). Turning it off before the removal keeps them. The never-index list must be empty while it's on, because every other path is already excluded. To stop discovering new paths but keep writing the columns you have, set the cap to the number of columns the table has instead.

For tag keys:

* Grepr indexes a key on the always-index list whether discovery is on or off, and whether or not the group has reached its cap.
* A key matches exactly.
* A key can't be on both lists.
* `service`, `host`, and any key the table is partitioned on can't be on the never-index list.
* To partition a table on a tag key, the key must be on the always-index list, and it stays there while the table is partitioned on it.

## When changes take effect

A change reaches running pipelines within about a minute, without a restart. A tag key you add to the always-index list gets its column when you save, so you can partition on it immediately. An exact attribute path you add gets its column when you save, if the table is under its cap. A discovered path gets its column after pipelines find it in more than one log, while the table is under its cap.

## Excluded fields and removed columns \[#excluded]

When you add a field to a never-index list, Grepr stops writing the field's column within about a minute, and searches read the field from the log's attributes or tags instead, so they return the same results.

Grepr schedules the column for removal from the table 6 hours after you exclude the field, and the table's indexing settings show that time under the field. While the pipeline is receiving logs, the column is removed at that time or within about an hour after it. A pipeline that receives no logs removes it once logs arrive again. Until then the column still counts toward the discovery cap; once it's removed, its place is free. Logs written before the removal stay searchable through their attributes and tags, but searches on that field no longer skip files using the column.

If you take the field off the never-index list before then, the removal is cancelled: Grepr goes back to writing the same column, and everything already indexed in it stays. After the removal, the field gets a new column the same way a field that was never indexed does: from the always-index list when you save, if the table is under its cap for an attribute path, or from discovery if the table is under its discovery cap. The new column covers only logs written from then on.

## Collect file statistics on every column \[#file-statistics]

A search skips a data file on an indexed column by using the file's statistics for that column: the smallest and largest values the column holds in the file, and how many of them are null. By default, a data lake table collects these statistics on its first 100 columns only, counting top-level columns before nested fields. On a table with many indexed columns, the columns and fields past the first 100 have no statistics, so searches that filter on them read every data file.

To collect statistics on every column of a dataset's raw log table or raw span table, in the dataset's settings, expand **Advanced**, turn on **Raw logs table** or **Raw spans table**, and click **Save**. Both are off for a new dataset. Turning one off returns the table to collecting statistics on its first 100 columns. The raw span table's built-in span fields keep their statistics either way. Grepr pins their statistics mode when a pipeline writing the dataset starts, and keeps any mode already set on the table for one of those fields.

Statistics on every column add about 18 bytes of table metadata for each indexed column in each data file.

A change applies to the data files that pipelines write after their next restart, and earlier when the table gets a new indexed column. Data files already written keep the statistics they were written with.

## Change indexing through the API

Reading a table's configuration returns its indexing settings with a `revision`. To change them, send the complete settings for both groups to the promotion endpoint, which replaces what's stored. Include the `revision` you read: if the settings changed since then, the request fails with a 409 error, and you read them again before you retry. Without a `revision`, the request replaces the stored settings whatever they are. See the [Datasets specification](/apis/api-spec#tag/Datasets).

Each excluded attribute entry, and each excluded tag key under `tags.scheduledRemovalAt`, comes back with `scheduledRemovalAt`: when Grepr removes its columns. While `attributes.excludeAllByDefault` is on, `attributes.excludeAllScheduledRemovalAt` is when Grepr removes the columns it excludes. Grepr sets it when the entry is first saved and ignores any value you send. To change how long Grepr waits before removing an excluded column, set `columnRemovalDelay` to an ISO-8601 duration from `PT2M` (2 minutes) to `P30D` (30 days). Without it, the delay is 6 hours. A new delay applies to entries you exclude from then on.

To turn file statistics on every column on or off for the `logs_raw` or `spans_raw` table, send `{"enabled": true}` or `{"enabled": false}` to the table's column metrics endpoint. Reading the endpoint returns whether the setting is on. See [Collect file statistics on every column](#file-statistics).

## Frequently Asked Questions

### What is an indexed column in Grepr?

An indexed column is a log attribute path or tag key that Grepr writes into its own column in the dataset's data lake table, instead of leaving it only in the log's attribute document or tags. Searches that filter on the field read that column and skip files that can't match, which makes them faster.

### Do I have to list the fields I want indexed?

No, if you turn on discovery. Grepr then adds a column for each new field it finds, up to the discovery cap. Discovery is off for a new dataset. List a tag key on the always-index list to index it whether or not discovery is on. An attribute path on the always-index list counts toward the cap, so it gets a column only while the table is under its cap.

### Do my searches change when a field is indexed?

No. The same search returns the same results whether or not a field is indexed, and runs faster when it is.

### What happens when I turn discovery off?

Grepr stops adding columns for new fields, including for attribute paths on the always-index list. Every column the table already has keeps being written.

### How do I stop Grepr from writing a column?

Add the field to the never-index list. Turning discovery off doesn't do it, because that only stops new columns. Grepr stops writing the column within about a minute and removes it from the table 6 hours later. Take the field off the list before then to keep the column. The data already written stays searchable.

### What if an attribute path is on both the always-index and never-index lists?

Always index wins. You can exclude a subtree with a nested-paths entry and keep one path inside it indexed. An always-index entry can't cover a never-index entry entirely, because the never-index entry would then never take effect. A tag key can't be on both lists.

### How do I index only the attribute paths I list?

Turn on Exclude all by default. Every path that isn't on the always-index list stops being indexed, including paths the table already has columns for, and the never-index list must be empty. Grepr removes those columns from the table 6 hours later, unless you turn it off before then.

### When do indexing changes take effect?

Within about a minute on running pipelines, without a restart. A tag key you add gets its column when you save. An exact attribute path you add gets its column when you save, if the table is under its cap. A discovered path gets its column after pipelines find it in more than one log.

### Can I delete an indexed column?

Yes. Add the field to the never-index list. Grepr stops writing the column within about a minute and removes it from the table 6 hours later, which frees its place under the discovery cap. Logs already written stay searchable through their attributes and tags.

### Can I stop indexing service and host?

No. Grepr always indexes the service and host tags, and neither key can be on the never-index list.

### Why do searches on some indexed columns read every data file?

By default, a data lake table collects file statistics on its first 100 columns only, and a search skips data files on a column only by using those statistics. To collect them on every column of the raw log or raw span table, turn on Raw logs table or Raw spans table under Advanced in the dataset's settings. The change applies to data files written after the pipelines writing the dataset restart.

### What does marking an always-index path numeric do?

The path gets its numeric column when you save, even when the table has reached its cap, as long as the path has its value column. Use it for values that are numbers written as strings. Only an exact path can be marked numeric.