--- title: "Executable Scanner" description: "launch a target process from which to read symbol values via its stdout" source: "https://github.com/log-10x/modules/tree/main/pipelines/compile/modules/scanner/executable/module.yaml" icon: "material/launch" --- Launches a subprocess to extract [symbol](https://doc.log10x.com/run/transform/structure/#symbols) values from source code or binary files. Supports OS utilities like [strings](https://linux.die.net/man/1/strings){target="\_blank"} for pre-compiled binaries, or custom code for reading from arbitrary files and remote sources (databases, web services). The subprocess stdout is parsed in one of two modes: ### Unstructured Splits stdout lines using [token delimiters](https://doc.log10x.com/compile/scan/#tokendelims "token delimiters") and assigns each value an [exec](https://doc.log10x.com/run/transform/symbol/#exec) context. See the [strings](#__tabbed_1_2) configuration example below. ### Structured Parses JSON objects from stdout containing hierarchical symbol trees with [context](https://doc.log10x.com/run/transform/symbol/#contexts) metadata (class, function, enum names). See [SymbolUnit.java](https://github.com/log-10x/pipeline-extensions/blob/main/cloud-extensions/src/main/java/com/log10x/ext/cloud/compile/symbol/SymbolUnit.java){target="\_blank"} for an example. ## :material-wrench-outline: Config Files To configure the Executable scanner module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. === "Strings-nix" Below is the default configuration from: [executable/strings-nix.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/executable/strings-nix.yaml "executable/strings-nix.yaml"){target="\_blank"}.
Edit Online
Edit strings-nix.yaml Locally
```yaml # # 🔟❎ 'compile' executable 'strings' configuration # This executable symbol scanner configuration uses OS-specific utilities to extract # constant string values from pre-compiled binaries (e.g. programs, shared libraries). # Any matching text values are added to the output symbol files. # The following configuration is added by default to the 10x 'compile' pipeline. # If another 'executable' scanner is defined whose 'osFilter' matches that of the # current target input file, it will take precedence over the scanners defined below. # To learn more about the 'executable' scanner, # see: https://doc.log10x.com/compile/scanner/executable # Set the 10x pipeline to 'compile' tenx: compile # =============================== Exec Options ================================ exec: # This scanner uses the Linux built-in 'strings' (https://linux.die.net/man/1/file) # command to extract symbol values from a target binary executable file - name: stringsNIX # This configuration targets NIX OSs osFilter: (nix|nux|lux|mac) # try for known extensions before invoking a 'selectorProcess' extensions: - .so - .dynlib # 'selector' processes provide a method for determining whether a file is an executable binary. # If a selector process is not provided the file's ELF/PE/OSX binary header is tested. # Since on NIX, executables do not have a known extension, # the 'selector' process uses the built-in 'file' command whose # output is used to verify the target file is a NIX executable file. selector: process: file args: - file # this macro value is replaced at run-time by the actual input file name timeout: 5s outputFilter: ".*?\\bexecutable\\b.*?" # select 'executable' files process: strings args: - -a # scan the whole file, not just initialized and loaded sections - file # This macro value is replaced at run-time by the actual text/binary input file timeout: 100s symbol: # 'minLength' specifies the min number of chars a value extracted via strings must be to # in order to qualify as a symbol. minLength: 4 # 'maxLength' specifies the max number of chars a value extracted via strings must be to # in order to qualify as a symbol. maxLength: 1024 # 'filters' defines regex patterns applied to each line read from the scanner process' stdout # that must NOT match for the line value to be considered a symbol. filters: - AWAV. # https://stackoverflow.com/questions/39322552/meaning-of-a-common-string-in-executables - "^(?=^.{0,7}$)(?=.*[^a-zA-Z]).*$" # Matches any string that is shorter than 8 characters (0 to 7 chars) and contains at least one non-alphabetic character. # 'setSelector' specifies regex patterns that determines whether to treat an input value as set of tokens setSelectors: - '^[0-9].*$' # Starts with digit - '^\*.*$' # Starts with * - '^(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}(?:/[^\s]*)?$' # Domain + optional path (e.g., github.com/user/repo) - '^type:\\..*$' # Go land metadata starts with "type:." (any suffix) ``` === "Strings-win" Below is the default configuration from: [executable/strings-win.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/executable/strings-win.yaml "executable/strings-win.yaml"){target="\_blank"}.
Edit Online
Edit strings-win.yaml Locally
```yaml # # 🔟❎ 'compile' executable 'strings' configuration # This executable symbol scanner configuration uses OS-specific utilities to extract # constant string values from pre-compiled binaries (e.g. programs, shared libraries). # Any matching text values are added to the output symbol files. # The following configuration is added by default to the 10x 'compile' pipeline. # If another 'executable' scanner is defined whose 'osFilter' matches that of the # current target input file, it will take precedence over the scanners defined below. # To learn more about the 'executable' scanner, # see: https://doc.log10x.com/compile/scanner/executable/ # Set the 10x pipeline to 'compile' tenx: compile # =============================== Exec Options ================================ exec: # This configuration provides a similar capability for scanning Windows executable files. # *IMPORTANT*: On Windows, the 'strings' command is not installed by default and # should be downloaded from: https://docs.microsoft.com/en-us/sysinternals/downloads/strings # and added to 'PATH' - name: stringsWin # target Win osFilter: win extensions: - .so - .dll process: strings args: - file # This macro value is replaced at run-time by the actual input file name timeout: 10s ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") multiple Executable scanner: |Name|Description|Category| |---|---|---| |[execName](#execname "a logical name for this exec scanner")|A logical name for this exec scanner|Scanner| |[execProcess](#execprocess "process to launch for extracting symbol values")|Process to launch for extracting symbol values|Scanner| |[execWorkDir](#execworkdir "the path of the working directory when running 'execProcess'")|The path of the working directory when running 'execProcess'|Scanner| |[execArgs](#execargs "list of arguments passed to 'execProcess' ('file' is replaced by scan target)")|List of arguments passed to 'execProcess' ('file' is replaced by scan target)|Scanner| |[execTimeout](#exectimeout "timeout period after which to terminate 'execProcess' process")|Timeout period after which to terminate 'execProcess' process|Scanner| |[execOsFilter](#execosfilter "a pattern that must match the name of the host OS to apply this scanner.")|A pattern that must match the name of the host OS to apply this scanner.|Filter| |[execExtensions](#execextensions "a list of file extensions a target file must match to apply this scanner")|A list of file extensions a target file must match to apply this scanner|Filter| |[execFileNameFilter](#execfilenamefilter "a pattern that the target input file must match to apply this scanner")|A pattern that the target input file must match to apply this scanner|Filter| |[execSelectorProcess](#execselectorprocess "name of a process to launch whose output whether to apply this input")|Name of a process to launch whose output whether to apply this input|Selector| |[execSelectorWorkDir](#execselectorworkdir "the path of the working directory when running 'execSelectorProcess'")|The path of the working directory when running 'execSelectorProcess'|Selector| |[execSelectorArgs](#execselectorargs "list of arguments passed to 'execSelectorProcess'")|List of arguments passed to 'execSelectorProcess'|Selector| |[execSelectorTimeout](#execselectortimeout "timeout period after which to terminate 'execSelectorProcess process")|Timeout period after which to terminate 'execSelectorProcess process|Selector| |[execSelectorOutputFilter](#execselectoroutputfilter "a filter pattern that must match the output of 'execSelectorProcess' to apply this scanner")|A filter pattern that must match the output of 'execSelectorProcess' to apply this scanner|Selector| |[execSymbolMinLength](#execsymbolminlength "minimal length of the string read from 'execProcess' to read as a symbol")|Minimal length of the string read from 'execProcess' to read as a symbol|Parse| |[execSymbolMaxLength](#execsymbolmaxlength "max length the string read from 'execProcess' to read as a symbol")|Max length the string read from 'execProcess' to read as a symbol|Parse| |[execSymbolFilters](#execsymbolfilters "patterns a line read from 'execProcess' must NOT match")|Patterns a line read from 'execProcess' must NOT match|Parse| |[execSymbolSetSelectors](#execsymbolsetselectors "regex patterns to determines whether to treat an input value as set of tokens")|Regex patterns to determines whether to treat an input value as set of tokens|Parse| |[execIsStructured](#execisstructured "sets whether output lines read from 'execProcess' are read as plain text or structured JSON")|Sets whether output lines read from 'execProcess' are read as plain text or structured JSON|Parse| |[execSymbolsPrefix](#execsymbolsprefix "a prefix lines read from 'execProcess' must match to parse as structured JSON symbols")|A prefix lines read from 'execProcess' must match to parse as structured JSON symbols|Parse| ### Scanner #### :material-menu-right-outline:**`execName`** A logical name for this exec scanner. |Type|Required|Category| |---|---|---| |String|✔|Scanner| A logical name for this exec scanner unique amongst all specified scanner option groups (e.g., 'stringsNIX'). #### :material-menu-right-outline:**`execProcess`** Process to launch for extracting symbol values. |Type|Default|Category| |---|---|---| |String|""|Scanner| Defines a process to launch and whose stdout pipe to parse for symbol values. #### :material-menu-right-outline:**`execWorkDir`** The path of the working directory when running 'execProcess'. |Type|Default|Category| |---|---|---| |String|""|Scanner| Defines the working directory when running [execProcess](https://doc.log10x.com/compile/scanner/executable/#execprocess "process to launch for extracting symbol values"). If omitted, uses the working directory. #### :material-menu-right-outline:**`execArgs`** List of arguments passed to 'execProcess' ('file' is replaced by scan target). |Type|Default|Category| |---|---|---| |List|\[\]|Scanner| Defines launch arguments [execProcess](https://doc.log10x.com/compile/scanner/executable/#execprocess "process to launch for extracting symbol values"). The `file` macro expands into the input file path. #### :material-menu-right-outline:**`execTimeout`** Timeout period after which to terminate 'execProcess' process. |Type|Default|Category| |---|---|---| |String|""|Scanner| Defines how long [execProcess](https://doc.log10x.com/compile/scanner/executable/#execprocess "process to launch for extracting symbol values") is allowed to run before termination. ### Filter #### :material-menu-right-outline:**`execOsFilter`** A pattern that must match the name of the host OS to apply this scanner. |Type|Required|Category| |---|---|---| |String|✔|Filter| Defines a pattern to match against the name of the host OS to determine whether to apply this scanner. #### :material-menu-right-outline:**`execExtensions`** A list of file extensions a target file must match to apply this scanner. |Type|Default|Category| |---|---|---| |List|\[\]|Filter| An array of file extensions to test a file candidate against. If the candidate matches any extensions, apply this scanner. #### :material-menu-right-outline:**`execFileNameFilter`** A pattern that the target input file must match to apply this scanner. |Type|Default|Category| |---|---|---| |String|""|Filter| Sets a regex pattern to test a file candidate name against. If the candidate matches any of the extensions, apply this scanner. ### Selector #### :material-menu-right-outline:**`execSelectorProcess`** Name of a process to launch whose output whether to apply this input. |Type|Default|Category| |---|---|---| |String|""|Selector| Define a process name to launch to obtain information about the input file. In some OSs (e.g., Linux/Unix), executable files do not necessarily have a known extension. For this, an OS utility such as NIX [file](https://man7.org/linux/man-pages/man1/file.1.html){target="\_blank"} can obtain information on the actual file contents. If a selector process is not provided the file's ELF/PE/OSX binary header is tested for a possible match. #### :material-menu-right-outline:**`execSelectorWorkDir`** The path of the working directory when running 'execSelectorProcess'. |Type|Default|Category| |---|---|---| |String|""|Selector| Defines the working directory when running [execSelectorProcess](https://doc.log10x.com/compile/scanner/executable/#execselectorprocess "name of a process to launch whose output whether to apply this input"). If omitted, will use the the working directory. #### :material-menu-right-outline:**`execSelectorArgs`** List of arguments passed to 'execSelectorProcess'. |Type|Default|Category| |---|---|---| |List|\[\]|Selector| Defines arguments for the selector process, if provided. The `file` macro takes the value of the target exec file to launch. #### :material-menu-right-outline:**`execSelectorTimeout`** Timeout period after which to terminate 'execSelectorProcess process. |Type|Default|Category| |---|---|---| |String|""|Selector| Sets how long to allow the selector process to run before termination (e.g. '10s'). #### :material-menu-right-outline:**`execSelectorOutputFilter`** A filter pattern that must match the output of 'execSelectorProcess' to apply this scanner. |Type|Default|Category| |---|---|---| |String|""|Selector| Defines a regex pattern to match against the contents of the stdout stream read from the [execSelectorProcess](https://doc.log10x.com/compile/scanner/executable/#execselectorprocess "name of a process to launch whose output whether to apply this input") process. If matched, apply this scanner to the input file. ### Parse #### :material-menu-right-outline:**`execSymbolMinLength`** Minimal length of the string read from 'execProcess' to read as a symbol. |Type|Default|Category| |---|---|---| |Number|0|Parse| Sets the minimal length a string read from the scanner process stdout must be to be considered a symbol. #### :material-menu-right-outline:**`execSymbolMaxLength`** Max length the string read from 'execProcess' to read as a symbol. |Type|Default|Category| |---|---|---| |Number|0|Parse| Sets the maximum length a string read from the scanner process stdout must be to be considered a symbol. #### :material-menu-right-outline:**`execSymbolFilters`** Patterns a line read from 'execProcess' must NOT match. |Type|Default|Category| |---|---|---| |List|\[\]|Parse| defines regex patterns applied to each line read from the scanner process' stdout that must NOT match for the line value to be captured as a symbol. For example, to avoid [AWAV binary](https://stackoverflow.com/questions/39322552/meaning-of-a-common-string-in-executables){target="\_blank"} when using the `strings` OS utility, specify: ```yaml execSymbolFilters: - ^AWAV ` ``` #### :material-menu-right-outline:**`execSymbolSetSelectors`** Regex patterns to determines whether to treat an input value as set of tokens. |Type|Default|Category| |---|---|---| |List|\[\]|Parse| defines regex patterns must ALL matc to treat an input value as a line (e.g., 'error connecting to {}'), or as a series of independent tokens (e.g., 'error', 'connecting', 'to') to reduce the cardinality and size of the output symbol file. For example, when processing files that contain values GO compile symbols such as: ``` github.com/open-telemetry/opentelemetry-collector-contrib/exporter/fileexporter.group[go.shape.struct ``` Specify the pattern below to break the string into separate symbol tokens (e.g., shape, struct,..): ```yaml setSelectors: - ^(?:github\.com|golang\.org|type:\.eq\.(?:github\.com|golang\.org))(/[a-zA-Z0-9_-]+)*$ - ^[a-zA-Z0-9_/:\.\[\]\*,-]*(?:\.[a-zA-Z0-9]+|\[[a-zA-Z0-9,\.\*/:_-]+\]|-fm)?$ ``` #### :material-menu-right-outline:**`execIsStructured`** Sets whether output lines read from 'execProcess' are read as plain text or structured JSON. |Type|Default|Category| |---|---|---| |String|""|Parse| Controls whether to read the [execProcess](https://doc.log10x.com/compile/scanner/executable/#execprocess "process to launch for extracting symbol values") stdout as plain text or as a JSON object containing source context information. If false (default), symbol values read from the process' output are marked as having an [exec](https://doc.log10x.com/run/transform/symbol/#exec) context. To learn more about producing structured symbol values, see [SymbolUnit.java](https://github.com/log-10x/pipeline-extensions/blob/main/cloud-extensions/src/main/java/com/log10x/ext/cloud/compile/symbol/SymbolUnit.java){target="\_blank"}. #### :material-menu-right-outline:**`execSymbolsPrefix`** A prefix lines read from 'execProcess' must match to parse as structured JSON symbols. |Type|Default|Category| |---|---|---| |Boolean|false|Parse| Sets a prefix that must precede lines read from the scan process' stdout to parse as structured JSON symbol trees. Only applies when [execIsStructured](https://doc.log10x.com/compile/scanner/executable/#execisstructured "sets whether output lines read from 'execProcess' are read as plain text or structured JSON") is true.
:material-github: This module is defined in [executable/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/compile/modules/scanner/executable/module.yaml "executable/module.yaml"){target="\_blank"}.