--- title: "TenXTemplate Loader" description: "import joint JSON schemas files to expand events into well-defined TenXObjects." source: "https://github.com/log-10x/modules/tree/main/pipelines/run/units/template/unit.yaml" icon: "material/forest-outline" --- The 10x Engine employs an optimization model inspired by Chrome V8’s [_intelligent design_](https://v8.dev/docs/hidden-classes#why-have-hidden-classes%3F){target="\_blank"} principle. Rather than treating each log event as a "bag of properties," the 10x Engine assumes only a finite number of object shapes will emerge and that these shapes will follow stereotypical usage `templates`. The 10x Engine utilizes [symbol library](https://doc.log10x.com/compile/link/#symbol-library) files to assign a shared [TenXTemplate](https://doc.log10x.com/api/js/#TenXTemplate "The TenXEngine class implements an optimization model inspired by Chrome V8's") (i.e., _hidden class_) to input log/trace events with the same [structure](https://doc.log10x.com/run/transform/structure "Structure input events into well-defined TenXObjects"), creating a cached optimized schema for each event type. Operating on typed TenXObjects enables direct access to [symbol](https://doc.log10x.com/run/transform/structure/#symbols) and [variable](https://doc.log10x.com/run/transform/structure/#variables) values at runtime without repeatedly parsing JSON structures or evaluating complex, brittle regular expressions for each event. ## :material-code-json: Template Files Template files describes the structure of [TenXTemplates](https://doc.log10x.com/engine/design/#optimization-model) generated during previous execution cycles, similar to a `.proto` file which describes the structure of serialized records. Each template instance is read as a single line JSON object which contains a `templateHash` and `template` fields into an in-memory map to enable [expanding](https://doc.log10x.com/run/transform/#expand) of previously compact instances read from [inputs](https://doc.log10x.com/run/input/ "Read events into the pipeline from local and remote sources"). For example, a template .json file or data read from an input stream may contain the following template objects: ```json {"templateHash":"7C9 /a$3Lv","template":"{\"log\":\".:$\\n\",\"stream\":\"stdout\",\"docker\":{\"container_id\":\"$\"},\"kubernetes\":{\"container_name\":\"coredns\",\"namespace_name\":\"kube-system\",\"pod_name\":\"coredns-$-$\",\"container_image\":\"registry.k8s.io//coredns//coredns:v1.$.$\",\"container_image_id\":\"docker-$:////registry.k8s.io//coredns//coredns@sha256:$\",\"pod_id\":\"$-$-$-$$\",\"pod_ip\":\"$.$.$.$\",\"host\":\"minikube\",\"labels\":{\"k8s-app\":\"kube-dns\",\"pod-template-hash\":\"$\"}},\"$_tag\":\"kubernetes.var.log.containers.coredns-$-$_kube-system_coredns-$.log\"}"} {"templateHash":"-EqO[AdeT!#","template":"{\"log\":\"[INFO] plugin//reload: Running configuration SHA512 = $\\n\",\"stream\":\"stdout\",\"docker\":{\"container_id\":\"$\"},\"kubernetes\":{\"container_name\":\"coredns\",\"namespace_name\":\"kube-system\",\"pod_name\":\"coredns-$-$\",\"container_image\":\"registry.k8s.io//coredns//coredns:v1.$.$\",\"container_image_id\":\"docker-$:////registry.k8s.io//coredns//coredns@sha256:$\",\"pod_id\":\"$-$-$-$$\",\"pod_ip\":\"$.$.$.$\",\"host\":\"minikube\",\"labels\":{\"k8s-app\":\"kube-dns\",\"pod-template-hash\":\"$\"}},\"$_tag\":\"kubernetes.var.log.containers.coredns-$-$_kube-system_coredns-$.log\"}"} {"templateHash":"-*pAW5IsbiJ","template":"{\"log\":\"[INFO] $.$.$.$:$ - $ \\\"AAAA IN github.com.default.svc.cluster.local. udp $ false $\\\" NXDOMAIN qr,aa,rd $ $7.$\\n\",\"stream\":\"stdout\",\"docker\":{\"container_id\":\"$\"},\"kubernetes\":{\"container_name\":\"coredns\",\"namespace_name\":\"kube-system\",\"pod_name\":\"coredns-$-$\",\"container_image\":\"registry.k8s.io//coredns//coredns:v1.$.$\",\"container_image_id\":\"docker-pullable:////registry.k8s.io//coredns//coredns@sha256:$\",\"pod_id\":\"$-$-$-$$\",\"pod_ip\":\"$.$.$.$\",\"host\":\"minikube\",\"labels\":{\"k8s-app\":\"kube-dns\",\"pod-template-hash\":\"$\"}},\"$_tag\":\"kubernetes.var.log.containers.coredns-$-$_kube-system_coredns-$.log\"}"} ``` The last template in this example whose hash is `-*pAW5IsbiJ` can be used to expand the contents of the following [compact event](https://doc.log10x.com/run/transform/#compact) from an input stream: ``` ~-*pAW5IsbiJ,10,244,0,102,45981,64767,77,1232,147,000221854s,4c195cfdbf7e41f640631629970b9af2d8a1f40f63dcffd15edca84e2e2e497e,7db6d8ff4d,pddxj,11,1,1eeb4c7316bacb1d4c8ead65571cd92dd21e27359f0d4917f1a5822a73b75db1,38b91d65,ba47,4d0f,a689,-711056955842,10,244,0,99,7db6d8ff4d,10x,7db6d8ff4d,pddxj,4c195cfdbf7e41f640631629970b9af2d8a1f40f63dcffd15edca84e2e2e497e ``` Given both template and compact event information, the 10x Engine will automatically expand the event to its original full-text form: ```json {"log":"[INFO] 10.244.0.102:45981 - 64767 \"AAAA IN github.com.default.svc.cluster.local. udp 77 false 1232\" NXDOMAIN qr,aa,rd 147 0.000221854s\n","stream":"stdout","docker":{"container_id":"4c195cfdbf7e41f640631629970b9af2d8a1f40f63dcffd15edca84e2e2e497e"},"kubernetes":{"container_name":"coredns","namespace_name":"kube-system","pod_name":"coredns-7db6d8ff4d-pddxj","container_image":"registry.k8s.io/coredns/coredns:v1.11.1","container_image_id":"docker-pullable://registry.k8s.io/coredns/coredns@sha256:1eeb4c7316bacb1d4c8ead65571cd92dd21e27359f0d4917f1a5822a73b75db1","pod_id":"38b91d65-ba47-4d0f-a689-711056955842","pod_ip":"10.244.0.99","host":"minikube","labels":{"k8s-app":"kube-dns","pod-template-hash":"7db6d8ff4d"}},"tenx_tag":"kubernetes.var.log.containers.coredns-7db6d8ff4d-pddxj_kube-system_coredns-4c195cfdbf7e41f640631629970b9af2d8a1f40f63dcffd15edca84e2e2e497e.log"} ``` At runtime, matching TenXTemplate JSON objects from pipeline [inputs](https://doc.log10x.com/run/input/ "Read events into the pipeline from local and remote sources") are automatically added to the available templates for expanding. This allows compact files and streams to include template definitions followed by their compact instances. ## :material-wrench-outline: Config Files To configure the TenXTemplate Loader unit, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [template/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/template/config.yaml "template/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ 'run' template file configuration # Load TenXTemplates .json files that define the structure/schema of TenXObjects. # To learn more see https://doc.log10x.com/run/template/ # Set the 10x pipeline to 'run' tenx: run # =============================== Template =================================== template: # 'files' specifies GLOB pattern for finding JSON-encoded TenXTemplates. files: - $=path("data/templates/*.json") - $=path("data/sample/output/*.json") # 'cacheSize' controls the maximum total byte size of templates held # in the in-memory cache vs. on disk. Set to 0 to disable pruning. cacheSize: $=parseBytes("10MB") # =============================== Variable =================================== var: # 'placeholder' specifies a character to use when encoding a TenXTemplate # to signify the location of a runtime variable value. placeholder: "$" # 'maxRecurIndexes' controls the maximum number of variable values to reuse. maxRecurIndexes: 10 # =============================== Timestamp ================================== timestamp: # 'prefix' specifies a prefix for a TenXTemplate's timestamp tokens. prefix: ( # 'postfix' specifies a postfix for a TenXTemplate's timestamp tokens. postfix: ) ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the TenXTemplate Loader: |Name|Description|Category| |---|---|---| |[templateFiles](#templatefiles "input files or glob patterns containing JSON-encoded TenXTemplates")|Input files or glob patterns containing JSON-encoded TenXTemplates|Load| |[varPlaceholder](#varplaceholder "placeholder character for notating TenXObject instance variables")|Placeholder character for notating TenXObject instance variables|Parse| |[timestampPrefix](#timestampprefix "template timestamp prefix")|Template timestamp prefix|Parse| |[timestampPostfix](#timestamppostfix "template timestamp postfix")|Template timestamp postfix|Parse| |[varMaxRecurIndexes](#varmaxrecurindexes "maximum number of template reusable variable value")|Maximum number of template reusable variable value|Parse| |[templateCacheSize](#templatecachesize "maximum total byte size of in-memory cached templates")|Maximum total byte size of in-memory cached templates|Advanced| ### Load #### :material-menu-right-outline:**`templateFiles`** Input files or glob patterns containing JSON-encoded TenXTemplates. |Type|Default|Category| |---|---|---| |List|\[path("data/sample/templates/\*.json")\]|Load| Input files or glob patterns from which to load JSON-encoded TenXTemplates. ### Parse #### :material-menu-right-outline:**`varPlaceholder`** Placeholder character for notating TenXObject instance variables. |Type|Default|Category| |---|---|---| |String|$|Parse| Specifies a character to use when encoding an TenXTemplate to signify the location of a runtime variable value. For example, for an TenXTemplate containing the following segment: 'ERROR: could not connect to host IP: $.$.$.$' The '$' entries designate that any TenXObject associated with this template will substitute the '$' slots with matching values from its individual [vars](https://doc.log10x.com/api/js/#TenXBaseObject+vars "An array of variable sequences extracted from the object's text.") array (e.g. 0,0,0,127). #### :material-menu-right-outline:**`timestampPrefix`** Template timestamp prefix. |Type|Default|Category| |---|---|---| |String|(|Parse| Specifies a prefix for an TenXTemplate's set of variable and symbol tokens describing a timestamp (e.g., ${DD/MM/YY HH-mm-ss}). #### :material-menu-right-outline:**`timestampPostfix`** Template timestamp postfix. |Type|Default|Category| |---|---|---| |String|)|Parse| Specifies a value appended to an TenXTemplate's and set of variable and symbol tokens describing a timestamp (e.g., ${DD/MM/YY HH-mm-ss}). #### :material-menu-right-outline:**`varMaxRecurIndexes`** Maximum number of template reusable variable value. |Type|Default|Category| |---|---|---| |Number|10|Parse| Controls the maximum number of variable values to reuse when encoding an TenXObject's [vars](https://doc.log10x.com/api/js/#TenXBaseObject+vars "An array of variable sequences extracted from the object's text.") array to stream. When generating a new TenXTemplate for an TenXObject instance whose variables contain repetitive values, the template may indicate that a value is a repetition of a previous variable value and does not need to be encoded again. For example, for an TenXObject whose [text](https://doc.log10x.com/api/js/#TenXBaseObject+text "The content of the event from which this instance was structured.") field contains the following segment: `userID=45dfgdfs4,parentID=45dfgdfs4`, the resulting TenXTemplate can be encoded as: `userID=$,parentID=$1` specifying that the value of the `parentID` variable can be found one slot behind in the TenXObject's [vars array](https://doc.log10x.com/api/js/#TenXBaseObject+vars "An array of variable sequences extracted from the object's text."), saving the need to encode this value multiple times. ### Advanced #### :material-menu-right-outline:**`templateCacheSize`** Maximum total byte size of in-memory cached templates. |Type|Default|Category| |---|---|---| |Number|20971520|Advanced| Controls the maximum total byte size of templates held in the in-memory cache. When the cache exceeds this limit, least recently used templates are pruned to a temporary file on disk. Pruned templates are automatically reloaded from disk when accessed again. Set to 0 to disable pruning and keep all templates in memory.
:material-github: This unit is defined in [template/unit.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/units/template/unit.yaml "template/unit.yaml"){target="\_blank"}.