--- title: "Input Scanner" description: "capture symbol values from source code/binary files and link to an output\ \ symbol library" source: "https://github.com/log-10x/modules/tree/main/pipelines/compile/units/scan/unit.yaml" icon: "material/code-tags" --- Scan input source code/binary files to produces symbol files from a variety of programming languages, text and binary formats using configurable scanner [modules](https://doc.log10x.com/engine/module/). Scanner modules parse the contents of a target input file to generate an [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree){target="\_blank"} from which to capture [symbol](https://doc.log10x.com/run/transform/structure/#symbols) values and the [context](https://doc.log10x.com/run/transform/symbol/#contexts) in which they appear (e.g., class, function, printout). The run pipeline utilizes symbol files to [transform](https://doc.log10x.com/run/transform/ "Transform log and trace events into well-defined TenXObjects") input log/trace events into typed [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s)."). Once scanner modules complete, the compile pipeline [links](https://doc.log10x.com/compile/link/ "Link symbol unit files to an output symbol library file") symbol files into a single [symbol library](https://doc.log10x.com/run/symbol/#libraries) file for use at run time. ### :material-file-tree: Output Scanners emit information captured from each input into a respective symbol unit (.json) file. The path of each output symbol unit within [outputSymbolFolder](https://doc.log10x.com/compile/scan/#outputsymbolfolder "location of output symbol folder") matches its folder structure within [inputPaths](https://doc.log10x.com/compile/scan/#inputpaths "input folders to scan") For example, when launching a compile pipeline with the following [config](https://doc.log10x.com/config/yaml/): ```yaml tenx: compile inputPaths: - ~/dev outputSymbolFolder: ~/symbols ``` For a foo.js file located in: `~/dev/app/foo.js` The respective output symbol unit output file is: `~/symbols/app/foo.js.10x.json` ### :material-card-multiple-outline: Parallel Processing The compile pipeline supports running multiple instances of the [compiler app](https://doc.log10x.com/apps/compiler/) in parallel by using shared file locks to safely reuse existing symbol unit files residing in the [output path](#outputsymbolfolder "location of output symbol folder"). The underlying storage system must support [advisory locks](https://www.baeldung.com/linux/file-locking#1-advisory-locking){target="\_blank"} for parallel execution to be enabled. A separate scheduled task periodically can link symbol unit files into a single [symbol library](https://doc.log10x.com/compile/link/#symbol-library) artifact used by edge/cloud apps at runtime. ## :octicons-package-24: Modules
- :material-file-code:{ .lg .middle } __ANTLR Scanner__ --- Extract symbols from any programming language using the ANTLR framework. [:octicons-arrow-right-24: More info](/compile/scanner/antlr) - :material-language-java:{ .lg .middle } __JavaParser Scanner__ --- Scan Java source code for symbol values using the JavaParser library. [:octicons-arrow-right-24: More info](/compile/scanner/javaParser) - :material-language-python:{ .lg .middle } __CPython AST Scanner__ --- Scan Python source code for symbol values. [:octicons-arrow-right-24: More info](/compile/scanner/pythonAST) - :octicons-file-binary-24:{ .lg .middle } __JVM Bytecode Scanner__ --- Scan JVM .class files for symbol values. [:octicons-arrow-right-24: More info](/compile/scanner/bytecode) - :simple-scala:{ .lg .middle } __Scalameta Scanner__ --- Scan Java source code for symbol values using the Scalameta library. [:octicons-arrow-right-24: More info](/compile/scanner/scalameta) - :material-launch:{ .lg .middle } __Executable Scanners__ --- Launch a target process from which to read symbol values via its stdout. [:octicons-arrow-right-24: More info](/compile/scanner/executable) - :material-code-json:{ .lg .middle } __Text File Scanners__ --- Scan text/binary files for symbol values using the Jackson library. [:octicons-arrow-right-24: More info](/compile/scanner/text) - :material-regex:{ .lg .middle } __Regex Pattern Scanners__ --- Extract symbol values from a files using regular expressions. [:octicons-arrow-right-24: More info](/compile/scanner/pattern) - :material-zip-box-outline:{ .lg .middle } __Archive Scanner__ --- Scan compressed archives for enclosed files containing symbol values. [:octicons-arrow-right-24: More info](/compile/scanner/archive) - :material-library-outline:{ .lg .middle } __Symbol Unit Scanner__ --- Discover existing symbol unit files to reuse. [:octicons-arrow-right-24: More info](/compile/scanner/symbol) - :material-math-log:{ .lg .middle } __Log Method/stream Definitions__ --- Define logger method and output stream names. [:octicons-arrow-right-24: More info](/compile/scanner/logMethods)
## :material-wrench-outline: Config Files To configure the Input Scanner unit, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files: ??? tenx-scan "Scan" ### Scan {.hidden-heading} Configure the [Input Scanner](/compile/scan "Capture symbol values from source code/binary files and link to an output symbol library") to capture symbol values from source code/binary files and link to an output symbol library. Below is the default configuration from: [scanners/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/config.yaml "scanners/config.yaml"){target="\_blank"}.
Edit Online
Edit Input Scanner Config Locally
```yaml # 🔟❎ 'compile' symbol scanner config # Configure input and output options for the compile pipeline. # To learn more see https://doc.log10x.com/compile/scan tenx: compile # ========================== Configure I/O Paths ============================== # 'inputPaths' lists folders on disk containing source code and binary files to scan. # To learn more see https://doc.log10x.com/compile/scan/#inputpaths inputPaths: - $=path("data/compile/sources", "") # Default to none ("") if source folder not found # - # uncomment to specify additional input paths # 'outputSymbolFolder' specifies the folder for output symbol unit files. # To learn more see https://doc.log10x.com/compile/scan/#outputsymbolfolder outputSymbolFolder: $=TenXEnv.get("TENX_OUTPUT_SYMBOL_FOLDER", path("data/shared/symbols", "")) # Default to 10x temp dir ``` ??? tenx-advanced "Advanced" ### Advanced {.hidden-heading} Below is the default configuration from: [advanced/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/advanced/config.yaml "advanced/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ 'compile' symbol scanner advanced config # Configure input and output options for the compile pipeline. # To learn more see https://doc.log10x.com/compile/scan/ # Set the 10x pipeline to 'compile' tenx: compile inputPaths: [] # ============================ Advanced options =============================== # ----------------------------- Tokenizer options ----------------------------- # Tokenization options provide control over how the symbol scanners parse source and config file input. # To learn more https://doc.log10x.com/compile/scan/#tokenize # 'tokenDelims' defines which characters break a string of event characters into tokens to # classify as symbols or variables. tokenDelims: "<>|. ][:-+/*=\\,{}_();'\"$\t\n@" # 'fileExtStringFormats' controls which characters identify the start of a format specifier (https://en.wikipedia.org/wiki/Printf#Format_specifier). fileExtStringFormats: - .java:{ - .py:{ - .js:${ - .php:$ - .swift:\( - .kt:$ - .rb:#{ - .ex:#{ - .c:% - .cpp:% - .cs:{ - .go:% - .rs:{ - .ts:${ - .pl:$ - .sh:$ - .r:% - .m:% - .scala:$ maxSymbol: # 'unitsPerToken' controls the maximum number of symbol units to load from the # pipeline's symbol library when searching for the origin of a specific TenXTemplate symbol. unitsPerToken: 128 # ----------------------------- Subprocess options ---------------------------- # Subprocess options control whether scanners execute in the current 10x Engine instance # or via a sub-process to provide better isolation over parsing of complex source code ASTs. # To learn more https://doc.log10x.com/compile/scan/#subprocess_1 # 'unitsToScanPerBatch' sets the maximum number of input files to scan per subprocess instance. # When this threshold is exceeded the current subprocess terminated and a new one is spawned until scan is complete. unitsToScanPerBatch: 200 scan: # ----------------------------- Threading options ---------------------------- # 'threadPoolSize'controls the maximum number of threads to concurrently # process input source/binary files. If the value is between 0 and 1, the number calculates # a percentage of available cores (e.g., 0.5 = use up to 50% of available cores). # If the value is >= 1, the value sets a fixed number of threads (e.g., 10 = 10 threads). # If the value is 1, allocate a single thread to process input files. # To learn more see https://doc.log10x.com/compile/scan/#parallel threadPoolSize: "0.5" # 'scanUnitTimeout' sets the timeout interval when scanning a source/binary input # file before dropping it. unitTimeout: 30s # 'scanOperationTimeout' sets the timeout interval for the entire scanning operation # before terminating it. operationTimeout: 10m # -------------------------------- Debug Option ------------------------------- # The settings below provide control over what information is logged # to console/file when scanning input source code/text/binary files for symbol values. # These options are primarily used when adding support for a new programming language # syntax via the 'ANTLR' scanner, text file formats via the Jackson 'text' scanner # or an external command using the 'executable' scanner. debug: # 'origins' lists names of input source/binary files # (e.g. .class, .js, .yaml, .exe) for which debug information # is printed to the console via 'consoleOut' log4j logger. # Printed output is dependent on the scanner assigned to process the input file # and usually includes the AST of the input file from which tokens are selected. # This information can be used when debugging which symbols are extracted # from an input file in the context of an 'antlr', 'text' or 'executable' scanner. # Set to '*' to match all input files. origins: [ # '*' ] # 'symbols' lists the values of nodes within the AST of a # target input file (e.g. MyClass, MyEnum, myFunc, ..) for which # debug information is printed to the console via the 'consoleOut' logger. # This includes source context (e.g. class, enum) and path within the AST # in which the node was detected. Set to '*' to match all AST nodes. symbols: [ #'*' ] # 'inProcess' controls whether to scan source/binary input files from within the current # 10x process or launch a sequential series subprocesses for better isolation # inProcess: true # 'force' enables scanning of source/binary input files even if a matching # symbol unit was found in 'inputPaths'. This flag is useful when debugging # an Antlr/text/exec scanner using 'debugOrigins' and 'debugSymbols' # to ensure a scan takes place even if a matching symbol unit is found. # force: false ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the Input Scanner: |Name|Description|Category| |---|---|---| |[inputPaths](#inputpaths "input folders to scan")|Input folders to scan|Input| |[outputSymbolFolder](#outputsymbolfolder "location of output symbol folder")|Location of output symbol folder|Output| |[scanForce](#scanforce "control whether to scan input files even if a matching existing symbol file exists")|Control whether to scan input files even if a matching existing symbol file exists|Output| |[maxSymbolUnitSectionSize](#maxsymbolunitsectionsize "maximum number of symbol tokens to store in a single symbol unit file")|Maximum number of symbol tokens to store in a single symbol unit file|Output| |[scanInProcess](#scaninprocess "controls whether to scan source/binary input files using sub-process(es)")|Controls whether to scan source/binary input files using sub-process(es)|Subprocess| |[printOutput](#printoutput "controls whether to emit output scan operations progress to the console")|Controls whether to emit output scan operations progress to the console|Subprocess| |[unitsToScanPerBatch](#unitstoscanperbatch "maximum number of input files to scan per sub-process.")|Maximum number of input files to scan per sub-process.|Subprocess| |[scanThreadPoolSize](#scanthreadpoolsize "maximum number of threads to use concurrently process input source code /binary files")|Maximum number of threads to use concurrently process input source code /binary files|Parallel| |[scanUnitTimeout](#scanunittimeout "timeout interval for scanning a source code /binary input file")|Timeout interval for scanning a source code /binary input file|Parallel| |[scanOperationTimeout](#scanoperationtimeout "timeout interval for the entire scanning operation")|Timeout interval for the entire scanning operation|Parallel| |[tokenDelims](#tokendelims "token delimiters")|Token delimiters|Tokenize| |[fileExtStringFormats](#fileextstringformats "string format specifier characters")|String format specifier characters|Tokenize| |[maxSymbolUnitsPerToken](#maxsymbolunitspertoken "number of symbol units to retrieve when searching for the origin of an TenXTemplate symbol sequence")|Number of symbol units to retrieve when searching for the origin of an TenXTemplate symbol sequence|Tokenize| |[scanDebugOrigins](#scandebugorigins "list of input file names for which to log information")|List of input file names for which to log information|Debug| |[scanDebugSymbols](#scandebugsymbols "list of node types within a scanner AST tree for which to log information")|List of node types within a scanner AST tree for which to log information|Debug| |[scanDebugLoggerName](#scandebugloggername "debug logger name")|Debug logger name|Debug| ### Input #### :material-menu-right-outline:**`inputPaths`** Input folders to scan. |Type|Required|Category| |---|---|---| |List|✔|Input| Provides a list of folders that are traversed in search of source code (e.g. .js, .cpp), binary (e.g., .so, .jar, .zip) and text (e.g., .json, .xml) files to capture symbol values from and output to symbol unit files. Input folders may contain files and folders pulled from [GitHub](https://doc.log10x.com/compile/pull/github/ "Pull folders from GitHub repositories to scan for symbol content") or and [Artifactory](https://doc.log10x.com/compile/pull/artifactory "Pull files from Artifactory repositories to scan for symbol content"). ### Output #### :material-menu-right-outline:**`outputSymbolFolder`** Location of output symbol folder. |Type|Default|Category| |---|---|---| |File||Output| Specifies the root folder in which to write output symbol unit files. This compiler [scan phase](https://doc.log10x.com/compile/scan/ "Capture symbol values from source code/binary files and link to an output symbol library") mirrors the structure of the [inputPaths](https://doc.log10x.com/compile/scan/#inputpaths "input folders to scan") argument, placing each output symbol unit in the same relative path under [outputSymbolFolder](https://doc.log10x.com/compile/scan/#outputsymbolfolder "location of output symbol folder") as its source/binary input file under its respective `inputPaths` folder entry. For example, if [inputPaths](https://doc.log10x.com/compile/scan/#inputpaths "input folders to scan") is `~/dev` and it contains the file `~/dev/app/foo.js`, and this value is `~/symbols`, the respective output symbol unit file is: `~/symbols/app/foo.js.10x.json`. #### :material-menu-right-outline:**`scanForce`** Control whether to scan input files even if a matching existing symbol file exists. |Type|Default|Category| |---|---|---| |Boolean|false|Output| Controls whether to reuse matching symbol files within [inputPaths](https://doc.log10x.com/compile/scan/#inputpaths "input folders to scan"). This option is primarily useful when debugging a new scanner module configuration (e.g., ANTLR grammar) in conjunction with the [scanner debugging](https://doc.log10x.com/compile/scan "Capture symbol values from source code/binary files and link to an output symbol library") options. #### :material-menu-right-outline:**`maxSymbolUnitSectionSize`** Maximum number of symbol tokens to store in a single symbol unit file. |Type|Default|Category| |---|---|---| |Number|500|Output| Controls the maximum number of symbol tokens to store in a symbol unit before splitting it. This option supports large source code /binary input files that may have thousands of text symbols to avoid creating sizeable individual symbol unit files. Set to null to ignore. ### Subprocess #### :material-menu-right-outline:**`scanInProcess`** Controls whether to scan source/binary input files using sub-process(es). |Type|Default|Category| |---|---|---| |Boolean|false|Subprocess| The scanner can launch [subprocesses](https://doc.log10x.com/compile/scan/#subprocess_1) to capture [symbol](https://doc.log10x.com/run/transform/structure/#symbols) information from specific target source/binary input files. Generating input file [ASTs](https://en.wikipedia.org/wiki/Abstract_syntax_tree){target="\_blank"} to scan for symbols values for complex syntaxes can be a lengthy, resource-consuming operation, which in the case of [ANTLR syntaxes](https://tomassetti.me/improving-the-performance-of-an-antlr-parser/){target="\_blank"} may timeout or run out of memory. Process [isolation](https://en.wikipedia.org/wiki/Process_isolation){target="\_blank"} allows for terminating scan operations that timeout for specific files, while allowing others to proceed. #### :material-menu-right-outline:**`printOutput`** Controls whether to emit output scan operations progress to the console. |Type|Default|Category| |---|---|---| |Boolean|false|Subprocess| Controls whether information relating to symbol units scanning is emitted to the stdout stream. This value should be set to true when the current 10x process is spawned by a parent 10x 'compile' pipeline to report its progress. #### :material-menu-right-outline:**`unitsToScanPerBatch`** Maximum number of input files to scan per sub-process. |Type|Default|Category| |---|---|---| |Number|40|Subprocess| Sets the maximum number of source/binary input files to scan by a single 'compile' sub-process. Set to 0 to unlimited. ### Parallel #### :material-menu-right-outline:**`scanThreadPoolSize`** Maximum number of threads to use concurrently process input source code /binary files. |Type|Default|Category| |---|---|---| |String|0.5|Parallel| Controls the maximum number of threads to process input source code/binary files concurrently: - If the value is between 0 and 1, the number calculates a percentage of available cores (e.g., 0.5 = use up to 50% of available cores). - If the value is \>= 1, the value sets a fixed number of threads (e.g., 10 = 10 threads). - If the value is 1, allocate a single thread to process input files. #### :material-menu-right-outline:**`scanUnitTimeout`** Timeout interval for scanning a source code /binary input file. |Type|Default|Category| |---|---|---| |String|20s|Parallel| Sets the timeout interval when scanning a source code /binary input file before dropping it. Set to null to ignore. #### :material-menu-right-outline:**`scanOperationTimeout`** Timeout interval for the entire scanning operation. |Type|Default|Category| |---|---|---| |String|10m|Parallel| Sets the timeout interval for the entire scanning operation before terminating it. Set to null to ignore. ### Tokenize #### :material-menu-right-outline:**`tokenDelims`** Token delimiters. |Type|Default|Category| |---|---|---| |String|\<\>\|. \]\[:-+/\*=\\\\,{}\_();'\\"$\\t\\n@|Tokenize| Defines which characters break a string of event characters into tokens to classify as symbols or variables. #### :material-menu-right-outline:**`fileExtStringFormats`** String format specifier characters. |Type|Default|Category| |---|---|---| |List|\[\]|Tokenize| Controls which characters identify the start of a [format specifier](https://en.wikipedia.org/wiki/Printf#Format_specifier){target="\_blank"} for each programming language extension in the form of : (e.g., `.java:%`). It is a standard programming practice to use string formats to log messages in the form of: ```java log.error("could not connect to {} with status {}", host, status); ``` When an TenXTemplate tokenizer processing an event identifies a format prefix (e.g., `%`) it skips it and appends subsequent symbol tokens (e.g., `with status`) to the symbol sequence preceding them (e.g., `could not connect to`). #### :material-menu-right-outline:**`maxSymbolUnitsPerToken`** Number of symbol units to retrieve when searching for the origin of an TenXTemplate symbol sequence. |Type|Default|Category| |---|---|---| |Number|128|Tokenize| Controls the maximum number of symbol units to load from the pipeline's symbol library when searching for the origin of a specific [TenXTemplate](https://doc.log10x.com/run/template/ "Import joint JSON schemas files to expand events into well-defined TenXObjects.") symbol. The [symbolSequence](https://doc.log10x.com/api/js/#TenXObject+symbolSequence) function queries the source code/binary origin of symbol sequences in a target TenXTemplate to identify the logical [message](https://doc.log10x.com/run/initialize/message/ "Enrich TenXObjects with logical message symbol sequence and origin values") portion of a target app/infra events vs. variable and context information (e.g., user, host, severity). Since a [symbol](https://doc.log10x.com/run/transform/structure/#symbols) value may appear in hundreds or more locations across a code base, limiting the number of symbol units to load is necessary to reduce memory consumption. A second consideration is placing an upper limit on the number of units to load from the symbol library, which is the more frequent a symbol is, the lower the probability of selecting the correct origin. ### Debug #### :material-menu-right-outline:**`scanDebugOrigins`** List of input file names for which to log information. |Type|Default|Category| |---|---|---| |List|\[\]|Debug| Specifies a list of input file names (e.g., my.scala, foo.java) to log information relating to which symbols were captured/skipped. Specify '\*' for all files. #### :material-menu-right-outline:**`scanDebugSymbols`** List of node types within a scanner AST tree for which to log information. |Type|Default|Category| |---|---|---| |List|\[\]|Debug| List of nodes within an input AST tree, such as class and method names(e.g., 'MyClass', 'foo') for which to log information relating to the context in which they were captured/skipped as symbols. Specify '\*' for all nodes. #### :material-menu-right-outline:**`scanDebugLoggerName`** Debug logger name. |Type|Default|Category| |---|---|---| |String|[consoleOut](https://github.com/log-10x/config/blob/main/log4j2.yaml#L66 "consoleOut"){target="\_blank"}|Debug| Specifies the log4j logger for logging values for symbols/files matching 'debugOrigins' and 'debugSymbols'.
:material-github: This unit is defined in [scan/unit.yaml](https://github.com/log-10x/modules/tree/main/pipelines/compile/units/scan/unit.yaml "scan/unit.yaml"){target="\_blank"}.