--- title: "Compiler" description: "generate symbol libraries to enable structured event processing at runtime" source: "https://github.com/log-10x/modules/tree/main/pipelines/compiler/app.yaml" icon: "material/play-circle-outline" hidden: "true" --- Generate a [symbol library](https://doc.log10x.com/compile/link/#symbol-library) from source code and binary inputs to enable structured event processing at runtime. Commands on this page name `log10x/compiler-10x:1.1.69`. Substitute the version you want, and see [Image tags](https://doc.log10x.com/install/docker/#image-tags) for what `:latest` does and does not promise. ## :material-clipboard-play-outline: Setup Guide Follow the steps below. Steps that require customization link to the relevant [Config Files](#config-files) section where you can edit on github.dev or locally. ???+ tenx-bootstrap "Step 1: Install" Install the `Cloud` binary flavor: - :simple-linux: [Single line script](https://doc.log10x.com/install/singleline/ "curl https://raw.githubusercontent.com/log-10x/pipeline-releases/main/install.sh | bash"){data-copy="curl https://raw.githubusercontent.com/log-10x/pipeline-releases/main/install.sh | bash"} | [DEB](https://doc.log10x.com/install/linux/#ubuntu-debian) | [RPM](https://doc.log10x.com/install/linux/#red-hat-centos-7) - :simple-macos: [Homebrew](https://doc.log10x.com/install/macos/ "brew install --cask log-10x/tap/log10x-cloud"){data-copy="brew install --cask log-10x/tap/log10x-cloud"} - :material-microsoft-windows: [PowerShell script](https://doc.log10x.com/install/win/) - :simple-docker: [Docker image](https://doc.log10x.com/install/docker/) ??? tenx-config "Step 2: Set Environment Variables" Set these environment variables before running. See [path configuration](https://doc.log10x.com/install/paths/) for details. | Variable | Description | |----------|-------------| | `TENX_CONFIG` | Path to your [configuration directory](https://doc.log10x.com/install/paths/#config) | | `TENX_LICENSE_KEY` | Your Log10x license JWT ([licensing](https://doc.log10x.com/manage/license/)); unset, the engine runs on the built-in evaluation license | | `GH_TOKEN` | GitHub personal access token ([create one](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)) (optional) | | `DOCKER_USERNAME` | Docker registry username ([Docker Hub](https://docs.docker.com/security/for-developers/access-tokens/)) (optional) | | `DOCKER_TOKEN` | Docker registry token ([Docker Hub](https://docs.docker.com/security/for-developers/access-tokens/)) (optional) | | `ARTIFACTORY_TOKEN` | JFrog Artifactory token ([create one](https://jfrog.com/help/r/jfrog-platform-administration-documentation/access-tokens)) (optional) | ```bash export TENX_CONFIG=/path/to/your/config export TENX_LICENSE_KEY="$(cat license.jwt)" export GH_TOKEN=your-github-token ``` See [best practices](https://doc.log10x.com/engine/gitops/#best-practices) for managing secrets in production. ??? tenx-scan "Step 3: Set Up Input Sources" Specify source code or binary [inputPaths](#scan) to scan. === "Local Sources" Clone a repository to the default input path: ```bash git clone https://github.com/open-telemetry/opentelemetry-demo.git \ $TENX_CONFIG/data/compile/sources ``` === "GitHub Pull" Configure [githubPull](#github_2) to pull repos automatically: ```yaml githubPull: - token: $=TenXEnv.get("GH_TOKEN") repos: - open-telemetry/opentelemetry-demo ``` === "Helm Charts" Configure [helmChartNames](#helm_2) to pull from Helm charts: ```yaml helm: chartNames: - open-telemetry/opentelemetry-demo ``` The compiler uses [Docker CLI](https://docs.docker.com/get-started/get-docker/) and [Helm](https://helm.sh/docs/intro/install/). The Docker image includes these; otherwise ensure [helmCommand](https://doc.log10x.com/compile/pull/helm/#helmcommand) and [dockerCommand](https://doc.log10x.com/compile/pull/docker/#dockercommand) are valid. ??? tenx-transform "Step 4: Configure Output" 1. Set [outputSymbolFolder](#scan) for symbol unit files 2. Set [outputSymbolLibraryFile](#link) for the final library file ??? tenx-githubsync "Step 5: Push to GitHub (optional)" Push outputs to GitHub for [GitOps workflows](https://doc.log10x.com/engine/gitops/), enabling symbol reuse and delivery to edge/cloud apps: ```yaml githubPush: - token: $=TenXEnv.get("GH_TOKEN") repo: myUser/myRepo folder: symbols ``` ??? tenx-mainconfig "Step 6: Run" === ":material-laptop: Nix/Win/OSX" **Best for**: Local development and testing. ```bash tenx @apps/compiler ``` === ":material-docker: Docker (+ :material-laptop: local config)" **Best for**: Isolated environments with local configuration. ```bash docker run --rm \ -v $TENX_CONFIG:/etc/tenx/config/ \ -e TENX_CONFIG=/etc/tenx/config/ \ -e TENX_LICENSE_KEY=${TENX_LICENSE_KEY} \ -e GH_TOKEN=${GH_TOKEN} \ -e DOCKER_USERNAME=${DOCKER_USERNAME} \ -e DOCKER_TOKEN=${DOCKER_TOKEN} \ log10x/compiler-10x:1.1.69 \ @apps/compiler ``` === ":material-docker: Docker (+ :material-github: GitOps config)" **Best for**: CI/CD pipelines with version-controlled configuration. ```bash docker run --rm \ -e TENX_LICENSE_KEY=${TENX_LICENSE_KEY} \ -e GH_TOKEN=${GH_TOKEN} \ -e DOCKER_USERNAME=${DOCKER_USERNAME} \ -e DOCKER_TOKEN=${DOCKER_TOKEN} \ log10x/compiler-10x:1.1.69 \ '@github={"token": "${GH_TOKEN}", "repo": "my-user/my-repo"}' \ @apps/compiler ``` ??? tenx-checklist "Step 7: Verify" Verify no errors appear in the [log file](https://doc.log10x.com/manage/logging/#log-file-location). **Check output files:** Verify symbol files were generated in your configured [outputSymbolFolder](https://doc.log10x.com/compile/scan/#outputsymbolfolder) and [outputSymbolLibraryFile](https://doc.log10x.com/compile/link/#outputsymbollibraryfile) paths. ??? tenx-delete "Step 8: Teardown" Nothing runs in the background, uninstall removes only what was installed. === ":simple-macos: Homebrew" ```bash brew uninstall --cask log10x-cloud ``` === ":simple-linux: Linux" ```bash sudo rm -rf /opt/tenx-cloud /etc/tenx /etc/profile.d/tenx-cloud.sh ``` If installed via DEB: `sudo apt-get remove tenx-cloud` If installed via RPM: `sudo yum remove tenx-cloud` === ":material-microsoft-windows: Windows" Uninstall from **Settings > Apps > Installed apps**. === ":simple-docker: Docker" ```bash docker rmi log10x/compiler-10x:1.1.69 ``` ## :material-wrench-outline: Config Files To configure the Compiler app, [:material-cog: Edit](https://doc.log10x.com/config/app/#main-config "Learn how to edit app and module configurations") these files: ??? tenx-mainconfig "Main Config" ### Main Config {.hidden-heading} The main config file loads the Compiler app's required modules. Uncomment selected modules at: [compiler/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/apps/compiler/config.yaml "compiler/config.yaml"){target="\_blank"}.
Edit Online
Edit Compiler Config Locally
```yaml # # 🔟❎ Compiler app main config # The compile app produces symbol library files which enable 10x apps to # transform log/trace events into well-defined TenXObjects at runtime. # To learn more see https://doc.log10x.com/compile # =========================== Bootstrap Compiler ============================== # To learn more see https://doc.log10x.com/compile/bootstrap tenx: compile runtimeName: $=TenXEnv.get("TENX_RUNTIME_NAME", "myCompiler") # ============================ Load App Modules =============================== # Uncomment and edit selected config.yaml files in the folders below (e.g., compile/scanners/config.yaml) include: # ------------------------------ App settings --------------------------------- # Load general app settings: - compiler # Load gitops settings for pulling additional assets from Github. # Edit the gitops/config.yaml file with specific config # # For additional info see - https://doc.log10x.com/config/github - gitops # --------------------------- Process Inputs -------------------------------- # Process input source code and binary files on disk. - compile/scanners # https://doc.log10x.com/compile/scan # ---------------------------- Pull Inputs ---------------------------------- # This application can fetch source/binary files from remote inputs # Uncomment and edit config files (e.g. compile/pull/github/config.yaml) to remote pull # - compile/pull/github # https://doc.log10x.com/compile/pull/github - compile/pull/docker # https://doc.log10x.com/compile/pull/docker - compile/pull/helm # https://doc.log10x.com/compile/pull/helm # - compile/pull/artifactory # https://doc.log10x.com/compile/pull/artifactory # ---------------------------- Link Output ---------------------------------- # Link output files into a single symbol library artifact for runtime use - compile/link # https://doc.log10x.com/compile/link # ---------------------------- Push Output ---------------------------------- # Push output files to GitHub for use by subsequent compile and run instances # - compile/push # https://doc.log10x.com/compile/push ``` ??? tenx-pull "Pull" ### Pull {.hidden-heading} Activate a [Repo Fetcher](/compile/pull "Pull input source code/binary files from remote repositories") to pull input source code/binary files from remote repositories. === ":material-github: Github" #### github {.hidden-heading} Configure [GitHub repositories](/compile/pull/github "Pull folders from GitHub repositories to scan for symbol content") to pull folders from GitHub repositories to scan for symbol content. Below is the default configuration from: [github/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/pull/github/config.yaml "github/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit GitHub repository Config Locally
```yaml # 🔟❎ 'compile' GitHub pull sources configuration # Configures GitHub repositories folders to pull to disk to scan its contents. # To learn more see https://doc.log10x.com/compile/pull/github # For a reference list of industry-standard frameworks, see: # modules/pipelines/compile/modules/pull/github/library.yaml # Set the 10x pipeline to 'compile' tenx: compile # ============================== GitHub Options =============================== githubPull: # 'token' specifies the access token for this repo using a shell variable (recommended) or inline # See https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens - token: $=TenXEnv.get("GH_TOKEN") # (❗ EnvVar REQUIRED) # 'repo' specifies the repo names to pull from Github (e.g., apache/spark) repos: [ #owner/project ] # 'branch' specifies the branch to pull within 'repos'. If not set, pulls each repo's default branch branch: null # 'repoFolders' defines folders within 'repos' to pull. If not specified, pull the entire repo folders: [ # src/main ] ``` === ":material-docker: Docker" #### docker {.hidden-heading} Configure the [Docker image](/compile/pull/docker "Pull Docker image files to scan for symbol content") to pull Docker image files to scan for symbol content. Below is the default configuration from: [docker/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/pull/docker/config.yaml "docker/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit Docker image Config Locally
```yaml # 🔟❎ 'compile' Docker configuration # Pull docker images files using the Docker CLI client from container registries to # scan for symbol values. To learn more see https://doc.log10x.com/compile/pull/docker # Set the 10x pipeline to 'compile' tenx: compile # ============================= Docker Options ================================ docker: # 'username' specifies the user name for authenticating the docker client. # If not set, the docker client is assumed to be pre-authenticated. # To learn more see https://docs.docker.com/reference/cli/docker/login/#username username: $=TenXEnv.get("DOCKER_USERNAME") # (❗ EnvVar REQUIRED) # 'password' specifies the password argument for authenticating the docker client login command. # This value is written into the docker client's std input via the --password-stdin argument. # To learn more see https://docs.docker.com/reference/cli/docker/login/#password-stdin. password: $=TenXEnv.get("DOCKER_TOKEN") # (❗ EnvVar REQUIRED) # 'command' specifies the location of docker cli https://docs.docker.com/reference/cli/docker/ program #command: $=TenXString.includes(TenXEnv.get("os.name"), "Windows") ? "C:/Program Files/Docker/Docker/resources/bin/docker.exe":"/usr/local/bin/docker" # 'images' specify images to pull and export to matching .tar files to scan images: [ # docker.io/grafana/grafana:11.1.0 ] # 'remove' specifies whether to remove exported images once their contents are extracted. # To learn more see https://docs.docker.com/reference/cli/docker/image/rm remove: false # 'githubRepoToken' defines an access token for pulling Github repos referenced # by a target image's 'org.opencontainers.image.source' property. githubRepoToken: $=TenXEnv.get("GH_TOKEN") # (❗ EnvVar REQUIRED) ``` === ":simple-helm: Helm" #### helm {.hidden-heading} Configure the [Helm chart](/compile/pull/helm "Pull GitHub repositories and Docker images to scan for symbol content from Helm charts") to pull GitHub repositories and Docker images to scan for symbol content from Helm charts. Below is the default configuration from: [helm/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/pull/helm/config.yaml "helm/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit Helm chart Config Locally
```yaml # 🔟❎ 'compile' k8s Helm scanner configuration # Configures the k8s Helm chart symbol scanner which pulls referenced GitHub source repositories and # Docker container images to scan for symbol values. # To learn more see https://doc.log10x.com/compile/pull/helm # For a reference list of common infrastructure charts, see: # modules/pipelines/compile/modules/pull/helm/library.yaml # Set the 10x pipeline to 'compile' tenx: compile # =============================== Helm Options =============================== helm: # Charts can be specified as Helm chart names OR as local chart files. # 'chartNames' specifies a list of remote chart names to scan for values and template files. chartNames: [ # open-telemetry/opentelemetry-demo, # ingress-nginx/ingress-nginx, ] # 'chartFiles' specifies a list of local chart template and values.yaml files on disk. # File paths may be relative to the 'includePaths' list. If canonical, enclosing folders must added to 'includePaths'. chartFiles: [ # charts/opentelemetry-demo/values.yaml ] pull: # 'dockerImages' specifies whether to pull docker container images referenced by input charts # To learn more see https://doc.log10x.com/compile/pull/docker dockerImages: true github: # 'repos' specifies whether to pull GitHub repos referenced by 'org.opencontainers.image.source' annotations # To learn more see https://github.com/opencontainers/image-spec/blob/v1.0.1/annotations.md#pre-defined-annotation-keys repos: true # 'token' specifies the GitHub access token for pulling from referenced repos # To learn more see: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens token: $=TenXEnv.get("GH_TOKEN") # (❗ EnvVar REQUIRED) ``` === ":simple-jfrog: Artifactory" #### artifactory {.hidden-heading} Configure [Artifactory repositories](/compile/pull/artifactory "Pull files from Artifactory repositories to scan for symbol content") to pull files from Artifactory repositories to scan for symbol content. Below is the default configuration from: [artifactory/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/pull/artifactory/config.yaml "artifactory/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit Artifactory repository Config Locally
```yaml # 🔟❎ 'compile' Artifactory configuration # Pull artifacts (e.g., Java archives, .NET assemblies) from remote Artifactory instances to # scan for symbol values. To learn more see https://doc.log10x.com/compile/pull/artifactory # Set the 10x pipeline to 'compile' tenx: compile # ============================= Artifactory Options =========================== artifactory: # 'token' specifies an Artifactory API access token. # To learn more see https://jfrog.com/help/r/jfrog-platform-administration-documentation/access-tokens - token: $=TenXEnv.get("ARTIFACTORY_TOKEN") # (❗ REQUIRED) # 'instance' specifies the of location of the remote Artifactory instance (e.g., https://demo.jfrog.io/artifactory) instance: "" # (❗ REQUIRED) # 'repo' specifies the target Artifactory repository (e.g., pypi-local) repo: "" # (❗ REQUIRED) # 'files' specifies files to pull from the target Artifactory repository # (❗ REQUIRED) files: [ # dist/jfrog-pypi-example-1.0.0.tar.gz ] # 'folders' specifies folders paths within the target Artifactory instance repository to pull # (❗ REQUIRED) folders: [ ] # 'recursive' specifies whether to pull folders recursively (i.e., pull sub-folders as well). recursive: true ``` ??? tenx-scan "Scan" ### Scan {.hidden-heading} Configure the [Input Scanner](/compile/scan "Capture symbol values from source code/binary files and link to an output symbol library") to capture symbol values from source code/binary files and link to an output symbol library. Below is the default configuration from: [scanners/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/config.yaml "scanners/config.yaml"){target="\_blank"}.
Edit Online
Edit Input Scanner Config Locally
```yaml # 🔟❎ 'compile' symbol scanner config # Configure input and output options for the compile pipeline. # To learn more see https://doc.log10x.com/compile/scan tenx: compile # ========================== Configure I/O Paths ============================== # 'inputPaths' lists folders on disk containing source code and binary files to scan. # To learn more see https://doc.log10x.com/compile/scan/#inputpaths inputPaths: - $=path("data/compile/sources", "") # Default to none ("") if source folder not found # - # uncomment to specify additional input paths # 'outputSymbolFolder' specifies the folder for output symbol unit files. # To learn more see https://doc.log10x.com/compile/scan/#outputsymbolfolder outputSymbolFolder: $=TenXEnv.get("TENX_OUTPUT_SYMBOL_FOLDER", path("data/shared/symbols", "")) # Default to 10x temp dir ``` ??? tenx-link "Link" ### Link {.hidden-heading} Configure the [Symbol library linker](/compile/link "Link symbol unit files to an output symbol library file") to link symbol unit files to an output symbol library file. Below is the default configuration from: [link/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/link/config.yaml "link/config.yaml"){target="\_blank"}.
Edit Online
Edit Symbol library linker Config Locally
```yaml # 🔟❎ 'compile' merge configuration # Combine symbol units into a single output symbol library file. # To learn more see https://doc.log10x.com/compile/link/ # Set the 10x pipeline to 'compile' tenx: compile # ============================ Symbol Library Output ========================== # 'outputSymbolLibraryFile' specifies location of output symbol library file which # links the contents of 'outputSymbolFolder' to a single .json artifact and a # compressed .zip version for use at run time by 10x 'run' pipelines. # If a local output dir exists write to it, otherwise use the pipeline's temp folder. # To learn more see https://doc.log10x.com/compile/link outputSymbolLibraryFile: $=TenXEnv.get("TENX_OUTPUT_SYMBOL_LIBRARY_FILE", path("data/shared/symbols", "") + "/" + TenXEnv.get("runtimeName", "symbols") + ".10x.tar") # ============================ Symbol Filter options ========================== # The options below control which symbol files and values are merged into 'outputSymbolLibraryFile' symbol: # 'fileNameFilters', exclude test files from symbol library fileNameFilters: # ── Java ─────────────────────────────────────── - ^.*(Test|Tests|TestCase|TestSuite)\.java$ # *Test.java, *TestCase.java, *TestSuite.java - ^Test.*\.java$ # TestFoo.java (legacy) # ── JavaScript / TypeScript ──────────────────── - ^.*\.(test|spec)\.(js|jsx|ts|tsx|mjs|cjs)$ # *.test.js, *.spec.tsx # ── Python ───────────────────────────────────── - ^test_.+\.py$ # test_utils.py - ^.+_test\.py$ # utils_test.py - ^test(s|case|cases|suite|suites)?\.py$ # test.py, tests.py, testcase.py, testcases.py, testsuite.py # ── Scala ────────────────────────────────────── - ^.*(Spec|Test|Suite)\.scala$ # *Spec.scala, *Test.scala, *Suite.scala # ── C# ───────────────────────────────────────── - ^.*(Test|Tests|TestCase|TestSuite)\.cs$ # *Test.cs, *TestCase.cs, *TestSuite.cs # ── Go ───────────────────────────────────────── - ^.*_test\.go$ # *_test.go # Note: testdata.go is usually a data file, not test code → not filtered here # ── C++ ─────────────────────────────────────── - ^.*(_test|Test|Tests)s?\.(cpp|cc|cxx|hpp|h|hxx)$ # *_test.cpp, *Test.hpp, *Tests.cpp # 'folderFilters', patterns to exclude files by folder # Plain name (e.g., "test") → matches direct parent folder only # Regex (e.g., "[/\\]vendor[/\\]") → matches anywhere in full path folderFilters: - test # direct parent is "test" - tests # direct parent is "tests" - __tests__ # Jest convention - spec # RSpec/Jasmine convention - specs # RSpec/Jasmine convention - unit # unit test folder - integration # integration test folder - e2e # e2e tests (Cypress, Playwright) - end-to-end # alternate e2e naming - acceptance # acceptance test folder - testdata # Go test data folder - fixtures # test fixtures folder - mocks # mock implementations folder - "[/\\\\]vendor[/\\\\][a-z]+\\.[a-z]" # Go vendored deps (vendor/domain.tld pattern) # 'types' controls the types of symbols written into 'outputSymbolLibraryFile' # for reflective use at runtime. # Narrowing the this list reduces the size of the output 10x symbols file. Possible values: types: - class # source code class/interface name (e.g. 'AcmeLogic') - enum # source code enum/literal name (e.g. 'severity: {error,warn,debug}') - log # source code print format passed to a logging method/stream (e.g. 'err("could not connect to %s",..'); - exec # string value extracted from a binary executable input file (e.g. via 'strings' nix command) # - method # code method/function name (e.g. 'foo') # - const # string constant assigned into a variable or method arg (e.g. 'greeting = "hello world"') # - text # string value, usually read from a config file (e.g. JSON/XML/YAML field name) # - package # source code package/namespace/module name (e.g. 'com.acme.core') ``` ??? tenx-push "Push" ### Push {.hidden-heading} Configure [GitHub push committers](/compile/push "Push output symbol unit files to GitHub") to push output symbol unit files to GitHub. Below is the default configuration from: [push/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/push/config.yaml "push/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit GitHub push committer Config Locally
```yaml # 🔟❎ 'compile' GitHub configuration # Push output symbol unit files for reuse by subsequent compile cycles. # To learn more see https://doc.log10x.com/compile/push/ # Set the 10x pipeline to 'compile' tenx: compile # =============================== GitHub Options ============================== githubPush: # 'token' specifies the access token for pushing files to the target repo, read from env or set inline. # To learn more see: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens - token: $=TenXEnv.get("GH_TOKEN") # (❗ EnvVar REQUIRED) # 'repo' specifies the repo name to pull in Github # including the username of the owner (e.g., apache/spark) repo: null # (❗ REQUIRED) # 'branch' specifies the branch to use when pushing into this repo. # If not specified, push to the default repo branch branch: null # 'folder' specifies a folder within the target repo to push files, if not specified push to root folder: null # 'filter' specifies a a regex pattern a file on disk must match to be pushed to the target repo. filter: null ``` ## :material-tune: Advanced Config Files To configure advanced options (optional) for the Compiler app, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files: ??? tenx-bootstrap "Bootstrap" ### Bootstrap {.hidden-heading} Configure the [Pipeline Bootstrapper](/run/bootstrap "Authenticate the log10x account and launch a target pipeline") to authenticate the log10x account and launch a target pipeline. Below is the default configuration from: [bootstrap/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/bootstrap/config.yaml "bootstrap/config.yaml"){target="\_blank"}.
Edit Online
Edit Pipeline Bootstrapper Config Locally
```yaml # 🔟❎ compile bootstrap configuration # This config file specifies bootstrap options for the compile pipeline. # To learn more see https://doc.log10x.com/compile/bootstrap # To learn more see https://doc.log10x.com/config/ tenx: compile # =============================== Launch Settings ============================== # Inline license JWT. For production, prefer `licenseFile: /path/to/license.jwt`. licenseKey: $=TenXEnv.get("TENX_LICENSE_KEY") # 'includePaths' specifies folders on disk for resolving relative config file/folder references in addition to the working folder includePaths: [] # 'quiet' disables printing version information to the console. # quiet: true # 'jarFiles' specifies .jar files to dynamically load for use by compile, input and output API extensions. jarFiles: [] # 'metricEndpoint' specifies the Prometheus endpoint to report usage/health metrics (enterprise version only). # specifies a list of launch arguments that are disallowed from either command line or user config files. disabledArgs: [] ``` ??? tenx-githubsync "Github Sync" ### Github Sync {.hidden-heading} Below is the default configuration from: [gitops/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/gitops/config.yaml "gitops/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ GitOps configuration # This config file pulls symbol, config and lookup files from a remote repo to # enable centralized configuration management via GitHub. # To learn more see https://doc.log10x.com/config/github # The target 10x pipeline to run (e.g., run, compile) is specified in the config files pulled from the repo tenx: any include: - source: github options: # 'enabled' controls whether to enable this pull config enabled: false # (❗ REQUIRED) # 'token' specifies an access token for accessing the Github repo. # To learn more see https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens token: $=TenXEnv.get("GH_TOKEN") # 'repo' specifies the GitHub user/repo from which to pull repo: null # (❗ REQUIRED) # 'branch' on 'repo' (if null, uses default repo default branch) branch: main # 'paths' specifies Glob patterns within 'repo' to pull and extract to a temp folder. paths: - test # - config/*.yaml # https://doc.log10x.com/config/yaml # - symbols/*.10x.tar # https://doc.log10x.com/run/symbol # - lookups/*.csv # https://doc.log10x.com/api/js/#TenXLookup # - scripts/*.js # https://doc.log10x.com/api/js # 'syncInterval' specifies an interval by which check to check the remote branch for updates (null to disable) # This interval works in conjunction with: https://doc.log10x.com/run/reload syncInterval: 5min ``` ??? tenx-scan "Scan" ### Scan {.hidden-heading} Configure the [Input Scanner](/compile/scan "Capture symbol values from source code/binary files and link to an output symbol library") to capture symbol values from source code/binary files and link to an output symbol library. === ":material-code-tags: Advanced" #### advanced {.hidden-heading} Configure the [Input Scanner](/compile/scan "Capture symbol values from source code/binary files and link to an output symbol library") to capture symbol values from source code/binary files and link to an output symbol library. Below is the default configuration from: [advanced/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/advanced/config.yaml "advanced/config.yaml"){target="\_blank"}.
Edit Online
Edit Input Scanner Config Locally
```yaml # 🔟❎ 'compile' symbol scanner advanced config # Configure input and output options for the compile pipeline. # To learn more see https://doc.log10x.com/compile/scan/ # Set the 10x pipeline to 'compile' tenx: compile inputPaths: [] # ============================ Advanced options =============================== # ----------------------------- Tokenizer options ----------------------------- # Tokenization options provide control over how the symbol scanners parse source and config file input. # To learn more https://doc.log10x.com/compile/scan/#tokenize # 'tokenDelims' defines which characters break a string of event characters into tokens to # classify as symbols or variables. tokenDelims: "<>|. ][:-+/*=\\,{}_();'\"$\t\n@" # 'fileExtStringFormats' controls which characters identify the start of a format specifier (https://en.wikipedia.org/wiki/Printf#Format_specifier). fileExtStringFormats: - .java:% - .py:{ - .js:${ - .php:$ - .swift:\( - .kt:$ - .rb:#{ - .ex:#{ - .c:% - .cpp:% - .cs:{ - .go:% - .rs:{ - .ts:${ - .pl:$ - .sh:$ - .r:% - .m:% - .scala:$ maxSymbol: # 'unitsPerToken' controls the maximum number of symbol units to load from the # pipeline's symbol library when searching for the origin of a specific TenXTemplate symbol. unitsPerToken: 128 # ----------------------------- Subprocess options ---------------------------- # Subprocess options control whether scanners execute in the current 10x Engine instance # or via a sub-process to provide better isolation over parsing of complex source code ASTs. # To learn more https://doc.log10x.com/compile/scan/#subprocess_1 # 'unitsToScanPerBatch' sets the maximum number of input files to scan per subprocess instance. # When this threshold is exceeded the current subprocess terminated and a new one is spawned until scan is complete. unitsToScanPerBatch: 200 scan: # ----------------------------- Threading options ---------------------------- # 'threadPoolSize'controls the maximum number of threads to concurrently # process input source/binary files. If the value is between 0 and 1, the number calculates # a percentage of available cores (e.g., 0.5 = use up to 50% of available cores). # If the value is >= 1, the value sets a fixed number of threads (e.g., 10 = 10 threads). # If the value is 1, allocate a single thread to process input files. # To learn more see https://doc.log10x.com/compile/scan/#parallel threadPoolSize: "0.5" # 'scanUnitTimeout' sets the timeout interval when scanning a source/binary input # file before dropping it. unitTimeout: 30s # 'scanOperationTimeout' sets the timeout interval for the entire scanning operation # before terminating it. operationTimeout: 10m # -------------------------------- Debug Option ------------------------------- # The settings below provide control over what information is logged # to console/file when scanning input source code/text/binary files for symbol values. # These options are primarily used when adding support for a new programming language # syntax via the 'ANTLR' scanner, text file formats via the Jackson 'text' scanner # or an external command using the 'executable' scanner. debug: # 'origins' lists names of input source/binary files # (e.g. .class, .js, .yaml, .exe) for which debug information # is printed to the console via 'consoleOut' log4j logger. # Printed output is dependent on the scanner assigned to process the input file # and usually includes the AST of the input file from which tokens are selected. # This information can be used when debugging which symbols are extracted # from an input file in the context of an 'antlr', 'text' or 'executable' scanner. # Set to '*' to match all input files. origins: [ # '*' ] # 'symbols' lists the values of nodes within the AST of a # target input file (e.g. MyClass, MyEnum, myFunc, ..) for which # debug information is printed to the console via the 'consoleOut' logger. # This includes source context (e.g. class, enum) and path within the AST # in which the node was detected. Set to '*' to match all AST nodes. symbols: [ #'*' ] # 'inProcess' controls whether to scan source/binary input files from within the current # 10x process or launch a sequential series subprocesses for better isolation # inProcess: true # 'force' enables scanning of source/binary input files even if a matching # symbol unit was found in 'inputPaths'. This flag is useful when debugging # an Antlr/text/exec scanner using 'debugOrigins' and 'debugSymbols' # to ensure a scan takes place even if a matching symbol unit is found. # force: false ``` === ":material-regex: Pattern" #### pattern {.hidden-heading} Configure [Regex pattern scanners](/compile/scanner/pattern "Extract symbol values from a files using regular expressions") to extract symbol values from a files using regular expressions. Below is the default configuration from: [pattern/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/pattern/config.yaml "pattern/config.yaml"){target="\_blank"}.
Edit Online
Edit Regex pattern scanner Config Locally
```yaml # 🔟❎ 'compile' pattern quoted strings configuration # This pattern symbol scanner extracts single and double quoted string values # from target files. This pattern is used a default for capturing symbol values # from source files for which no ANTLR grammar or built-in scanner is available. # Set the 10x pipeline to 'compile' tenx: compile # ============================== Pattern Options ============================== pattern: # 'fileFilter' is set as a sink for prominent source code files. Available built-in and ANTLR # scanners will take precedence over this pattern scanner - fileFilter: \b\w+\.(java|cs|php|cpp|cxx|c|h|H|hxx|cc|hh|py|go|js|rs|bash|scala|sc|ts|rb|kt|lua|groovy|swift)\b extractQuotes: true # 'lineFilter' defines a regex pattern applied to each line within a target file. # Each matching region will be extracted as symbol value lineFilter: null # 'matchContext' sets the source context to assign any symbols collected using this this pattern selector. # For possible values, see https://doc.log10x.com/run/transform/symbol/#contexts matchContext: method_invoke # 'unescapeScheme' specifies an unescape algorithm to apply to matching sequences of patternLineFilter. # Supported values: [json,java,xml,html,javaScript]. unescapeScheme: java ``` === ":material-file-code: Cpp" #### Cpp {.hidden-heading} Configure the [ANTLR scanner](/compile/scanner/antlr "Extract symbols from any programming language using the ANTLR framework") to extract symbols from any programming language using the ANTLR framework. Below is the default configuration from: [antlr/cpp.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/cpp.yaml "antlr/cpp.yaml"){target="\_blank"}.
Edit Online
Edit ANTLR scanner Config Locally
```yaml # 🔟❎ 'compile' C++ ANTLR symbol scanner configuration # The 'cpp' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from c++ source files. # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR grammar .g4 file used to # to tokenize and construct an AST for the target cpp file lang: cpp fileExt: - .cpp - .cxx - .c++ - .c - .h - .H - .hpp - .hxx - .h++ - .cc - .hh grammarFile: grammar/CPP14Parser.g4 lexerFile: grammar/CPP14Lexer.g4 rootRule: translationUnit reserved: - '::' - '<<' - inline - namespace - < - class - '>' # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ - ^.*\bnew\(\);\b.*$ # The 'rule' list defines the nodes within a cpp AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: enumerator lang: cpp context: enum recursive: false capture: allSymbols - name: blockDeclaration lang: cpp context: method_invoke recursive: true capture: literalsOnly - name: enumbase lang: cpp context: enum recursive: true capture: literalsOnly - name: classHeadName lang: cpp context: class recursive: true capture: allSymbols - name: enumSpecifier lang: cpp context: class recursive: false capture: literalsOnly condition: ^enumHead$ tag: EnumHead - name: enumHead lang: cpp context: class recursive: false capture: allSymbols ifTag: EnumHead - name: functionDefinition lang: cpp context: method_decl recursive: false capture: allSymbols condition: ^functionDefinition$ tag: FunctionDefinition - name: declaratorid lang: cpp context: method_decl recursive: true capture: allSymbols ifTag: FunctionDefinition - name: jumpStatement lang: cpp context: var_assign recursive: true capture: literalsOnly - name: functionBody lang: cpp context: method_invoke recursive: false capture: literalsOnly # condition: ^functionBody$ # tag: FunctionBody - name: statement lang: cpp context: method_invoke recursive: true capture: literalsOnly - name: classSpecifier lang: cpp context: class recursive: false capture: literalsOnly subRule: memberSpecification - name: staticAssertDeclaration lang: cpp context: var_assign recursive: true capture: literalsOnly - name: parameterDeclarationList lang: cpp context: method_invoke recursive: false capture: literalsOnly # condition: ^parameterDeclarationList$ # tag: ParameterDeclarationList - name: enumeratorDefinition lang: cpp context: enum recursive: false capture: allSymbols - name: namespaceDefinition lang: cpp context: package recursive: false capture: allSymbols - name: constantExpression lang: cpp context: var_assign recursive: true capture: literalsOnly - name: templateArgumentList lang: cpp context: method_invoke recursive: false capture: literalsOnly # condition: ^templateArgumentList$ # tag: TemplateArgumentList ``` === ":material-file-code: Csharp" #### Csharp {.hidden-heading} Configure the [ANTLR scanner](/compile/scanner/antlr "Extract symbols from any programming language using the ANTLR framework") to extract symbols from any programming language using the ANTLR framework. Below is the default configuration from: [antlr/csharp.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/csharp.yaml "antlr/csharp.yaml"){target="\_blank"}.
Edit Online
Edit ANTLR scanner Config Locally
```yaml # 🔟❎ 'compile' C# ANTLR symbol scanner configuration # The 'csharp' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from c# source files. # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR-generated parser class used to # to tokenize and construct an AST for the target c# file. lang: csharp fileExt: - .csx - .cs parserClass: com.log10x.antlr.generated.csharp.CSharpParser lexerClass: com.log10x.antlr.generated.csharp.CSharpLexer # 'charStreamClass' is set to remove language elements introduced later than V6.0 which is the latest release supported by this grammar. # To learn more see https://github.com/antlr/grammars-v4/tree/master/csharp#readme charStreamClass: com.log10x.antlr.parsers.CSharpDowngrader rootRule: compilation_unit reserved: - namespace - interface - class - enum # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # Remove unsupported C# v10 File scoped namespaces - ^namespace.*;$ # Remove unsupported C# v9 Target-typed new operator # - "(? Edit Online
Edit CPython AST scanner Config Locally
```yaml # 🔟❎ 'compile' python symbol scanner settings # The Python AST symbol scanner leverages the Python native .py parsing engine. # to launch the 'astpretty' script from https://github.com/asottile/astpretty # to parse the contents of an input .py file. # If a python runtime is unavailable when executing the 'compile' pipeline, # the 'pythonASTPath' value can be set to null or empty string to default # to the ANTLR symbol scanner defined in 'python.yaml' # 'pythonPath' specifies the command for launching a Python run-time. # A test script verifies a Python run-time is available. If the test script is unsuccessful, # the scanner is disabled. pythonPath: python3 ``` === ":material-zip-box-outline: Archive" #### archive {.hidden-heading} Configure the [Archive scanner](/compile/scanner/archive "Scan compressed archives for enclosed files containing symbol values") to scan compressed archives for enclosed files containing symbol values. Below is the default configuration from: [archive/default.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/archive/default.yaml "archive/default.yaml"){target="\_blank"}.
Edit Online
Edit Archive scanner Config Locally
```yaml # 🔟❎ 'compile' archive symbol scanner config # The archive'scanner iterates through the contents of file archives # to seek source/binary files whose contents to scan into # the output symbol files. # Set the 10x pipeline to 'compile' tenx: compile archiveExtensions: - .zip - .jar - .war - .ear - .7s - .gz - .br - .lz4 - .lzma - .tar - .tar.br - .tar.gz - .tar.bz2 - .tar.lz4 - .tar.lzma ``` === ":material-math-log: LogMethods" #### logMethods {.hidden-heading} Configure the [Log method/stream definitions](/compile/scanner/logMethods "Define logger method and output stream names") to define logger method and output stream names. Below is the default configuration from: [logMethods/default.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/logMethods/default.yaml "logMethods/default.yaml"){target="\_blank"}.
Edit Online
Edit Log method/stream definitions Config Locally
```yaml # 🔟❎ 'compile' log methods configuration # Set the 10x pipeline to 'compile' tenx: compile # ============================ Log Options ============================= log: # 'methods' lists known method names collected from common logging # frameworks used to indicate source code logging events to stream at runtime. # Identifying those points in source code that perform logging operations # and marking them within an output symbol unit file (*.json) # allows the 'run' pipeline to correlate each TenXObject structured from the event # data back to its origin in the codebase. # To learn more about symbol contexts and their use, see https://doc.log10x.com/run/transform/symbol/#contexts methods: # Assertions are not enabled by default as they are less likely to appear in prod logs # - Assert # - assert # - assertEquals # - assertEqual - exception - Log - LOG - log - BOOST_LOG_TRIVIAL - trace - Trace - log_trace - logTrace - TRACE - LOG_TRACE - debug - Debug - logDebug - log_debug - DEBUG - LOG_DEBUG - Debugf - info - Info - log_info - logInfo - INFO - LOG_INFO - information - LogInformation - InfoF - warn - Warn - log_warn - logWarn - WARN - LOG_WARN - WarnF - warning - logWarning - LOG_WARNING - error - Error - logError - log_error - ERROR - LOG_ERROR - ErrorF - err - fatal - Fatal - log_fatal - logFatal - FATAL - LOG_FATAL - FatalF - critical - Panic # 'methodRegex' specifies an optional regex that will be applied # to a scanned source code method/function name. If matched, the method # will be considered to be a 'logging method' (see above). # For example: ^(log|Log)|(Log|log)$ methodRegex: null # 'streams' defines which source code variables within # this ANTLR syntax are treated as output streams (e.g. stdout, stderr, cerr). # Literal constants (e.g. "error connecting to {}") passed to a variable # listed below (e.g. 'cout') will be assigned a 'logger_method_invoke' # symbol source context. streams: # - assert - cout - cerr - Error - Debug - err - out - Console ``` === ":material-file-code: Go" #### Go {.hidden-heading} Configure the [ANTLR scanner](/compile/scanner/antlr "Extract symbols from any programming language using the ANTLR framework") to extract symbols from any programming language using the ANTLR framework. Below is the default configuration from: [antlr/go.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/go.yaml "antlr/go.yaml"){target="\_blank"}.
Edit Online
Edit ANTLR scanner Config Locally
```yaml # 🔟❎ 'compile' Go ANTLR symbol scanner configuration # The 'go' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from GO source files. # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR-generated parser class used to # to tokenize and construct an AST for the target go file. lang: go fileExt: - .go parserClass: com.log10x.antlr.generated.golang.GoParser lexerClass: com.log10x.antlr.generated.golang.GoLexer rootRule: sourceFile reserved: - package - func # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # The 'rule' list defines the nodes within a go AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: constSpec lang: go context: enum recursive: false capture: allSymbols subRule: identifierList - name: returnStmt lang: go context: var_assign recursive: true capture: literalsOnly - name: assignment lang: go context: var_assign recursive: true capture: literalsOnly - name: expressionStmt lang: go context: method_invoke recursive: true capture: literalsOnly - name: packageClause lang: go context: package recursive: false capture: allSymbols - name: arguments lang: go context: method_invoke recursive: true capture: literalsOnly - name: typeSpec lang: go context: class recursive: false capture: allSymbols - name: functionDecl lang: go context: method_decl recursive: false capture: allSymbols - name: methodDecl lang: go context: method_decl recursive: false capture: allSymbols ``` === ":material-code-json: Jackson" #### Jackson {.hidden-heading} Configure [Text file scanners](/compile/scanner/text "Scan text/binary files for symbol values using the Jackson library") to scan text/binary files for symbol values using the Jackson library. Below is the default configuration from: [text/jackson.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/text/jackson.yaml "text/jackson.yaml"){target="\_blank"}.
Edit Online
Edit Text file scanner Config Locally
```yaml # 🔟❎ 'compile' text symbol scanner configuration # The 'text' scanner parses text files for symbol values. It utilizes # the Jackson parser library to parse any text/binary format it supports. # This includes formats such json, yaml, xml, ini, protobuf and more. # The configuration below is added by default to the 10x 'compile' pipeline. # Even so, if another text scanner is defined via the 'textScanners' options group # whose 'fileNameFilter' matches that of the current target input file, # it will take precedence over the text scanners defined below. # To learn more about text scanner options below, see: # https://doc.log10x.com/compile/scanner/text # Set the 10x pipeline to 'compile' tenx: compile # =============================== Text Options ================================ text: - parserName: text fileNameFilter: '^.*\.(csv|txt|properties|csv|tsv|ini|conf|sh|log|out)$' maxLines: 500 lineOffset: 0 allowDigits: false minLength: 2 maxLength: 50 - parserName: json fileNameFilter: '^.*\.(json)$' parserFactoryClass: com.fasterxml.jackson.core.JsonFactory scanFieldValues: true - parserName: yaml fileNameFilter: '^.*\.(yml|yaml)$' parserFactoryClass: com.log10x.eng.scanner.text.TextYamlFactory scanFieldValues: true - parserName: xml fileNameFilter: '^.*\.(xml|xsd)$' parserFactoryClass: com.fasterxml.jackson.dataformat.xml.XmlFactory scanFieldValues: true ``` === ":material-file-code: Java" #### Java {.hidden-heading} Configure the [ANTLR scanner](/compile/scanner/antlr "Extract symbols from any programming language using the ANTLR framework") to extract symbols from any programming language using the ANTLR framework. Below is the default configuration from: [antlr/java.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/java.yaml "antlr/java.yaml"){target="\_blank"}.
Edit Online
Edit ANTLR scanner Config Locally
```yaml # 🔟❎ 'compile' Java ANTLR symbol scanner configuration # The 'java' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from Java source files. # IMPORTANT: while set to enabled by default, the dedicated java scanner utilizes # takes precedence over this scanner unless disabled. # To learn more see https://doc.log10x.com/compile/scanner/javaParser # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR-generated parser class used to # to tokenize and construct an AST for the target Java file. lang: java fileExt: - .java parserClass: com.log10x.antlr.generated.java.Java9Parser lexerClass: com.log10x.antlr.generated.java.Java9Lexer rootRule: compilationUnit reserved: - package - interface - class - enum # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # The 'rule' list defines the nodes within a Java AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: packageDeclaration lang: java context: package recursive: false capture: allSymbols - name: methodInvocation lang: java context: method_invoke recursive: true capture: literalsOnly - name: enumDeclaration lang: java context: class recursive: false capture: allSymbols - name: annotation lang: java context: annotation_invoke recursive: true capture: literalsOnly - name: normalClassDeclaration lang: java context: class recursive: false capture: allSymbols - name: variableInitializer lang: java context: var_assign recursive: true capture: literalsOnly - name: enumConstant lang: java context: enum recursive: false capture: allSymbols - name: returnStatement lang: java context: var_assign recursive: true capture: literalsOnly - name: methodDeclaration lang: java context: method_decl recursive: false capture: literalsOnly - name: argumentList lang: java context: method_invoke recursive: true capture: literalsOnly - name: methodInvocation_lfno_primary lang: java context: method_invoke recursive: true capture: literalsOnly - name: methodDeclarator lang: java context: method_decl recursive: false capture: allSymbols - name: methodInvocation_lf_primary lang: java context: method_invoke recursive: true capture: literalsOnly - name: normalInterfaceDeclaration lang: java context: class recursive: false capture: allSymbols ``` === ":material-file-code: Javascript" #### Javascript {.hidden-heading} Configure the [ANTLR scanner](/compile/scanner/antlr "Extract symbols from any programming language using the ANTLR framework") to extract symbols from any programming language using the ANTLR framework. Below is the default configuration from: [antlr/javascript.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/javascript.yaml "antlr/javascript.yaml"){target="\_blank"}.
Edit Online
Edit ANTLR scanner Config Locally
```yaml # 🔟❎ 'compile' JavaScript ANTLR symbol scanner configuration # The 'javascript' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from JavaScript source files. # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR-generated parser class used to # to tokenize and construct an AST for the target JavaScript file. lang: javascript fileExt: - .js parserClass: com.log10x.antlr.parsers.JavaScriptDecoratedParser lexerClass: com.log10x.antlr.generated.javascript.JavaScriptLexer rootRule: program2 reserved: - function - constructor - class maxLineLength: 4096 # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # The 'rule' list defines the nodes within a js AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: functionDeclaration lang: javascript context: method_decl recursive: false capture: allSymbols subRule: identifier - name: returnStatement lang: javascript context: var_assign recursive: true capture: literalsOnly - name: classDeclaration lang: javascript context: class recursive: false capture: allSymbols subRule: identifier - name: methodDefinition lang: javascript context: method_decl recursive: true capture: allSymbols subRule: identifier - name: formalParameterList lang: javascript context: method_decl recursive: false capture: literalsOnly - name: functionBody lang: javascript context: method_invoke recursive: false capture: literalsOnly - name: functionDecl lang: javascript context: method_decl recursive: false capture: allSymbols - name: anoymousFunctionDecl lang: javascript context: method_decl recursive: false capture: allSymbols - name: argumentsExpression lang: javascript context: method_invoke recursive: true capture: literalsOnly - name: variableDeclaration lang: javascript context: var_assign recursive: true capture: literalsOnly - name: assignmentExpression lang: javascript context: var_assign recursive: true capture: literalsOnly ``` === ":material-file-code: Python" #### Python {.hidden-heading} Configure the [ANTLR scanner](/compile/scanner/antlr "Extract symbols from any programming language using the ANTLR framework") to extract symbols from any programming language using the ANTLR framework. Below is the default configuration from: [antlr/python.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/python.yaml "antlr/python.yaml"){target="\_blank"}.
Edit Online
Edit ANTLR scanner Config Locally
```yaml # 🔟❎ 'compile' Python symbol scanner configuration # The 'python' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from Python source files. # NOTE: while set to enabled by default, the 'pythonAST' scanner which utilizes # the python run-time's built-in AST parser takes precedence over this scanner # unless a python run-time is not installed or the 'pythonAST' scanner is disabled. # To learn see https://doc.log10x.com/compile/scanner/pythonAST # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR-generated parser class used to # to tokenize and construct an AST for the target Python file. lang: python fileExt: - .py parserClass: com.log10x.antlr.generated.python.PythonParser lexerClass: com.log10x.antlr.generated.python.PythonLexer rootRule: root reserved: - def - __str__ - __init__ - class # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # The 'rule' list defines the nodes within a Python AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: classdef lang: python context: class recursive: false capture: allSymbols subRule: name condition: (^Enum$|^enum$) tag: enumLiteral - name: testlist_star_expr lang: python context: enum recursive: true capture: allSymbolsIfMatchCond ifTag: enumLiteral - name: trailer lang: python context: method_invoke recursive: true capture: literalsOnly subRule: name - name: return_stmt lang: python context: var_assign recursive: true capture: literalsOnly - name: decorator lang: python context: annotation_invoke recursive: true capture: literalsOnly subRule: arglist - name: arglist lang: python context: method_invoke recursive: true capture: literalsOnly - name: funcdef lang: python context: method_decl recursive: false capture: allSymbols subRule: name - name: assign_part lang: python context: var_assign recursive: true capture: literalsOnly - name: dictorsetmaker lang: python context: var_assign recursive: true capture: literalsOnly ``` === ":material-file-code: Scala" #### Scala {.hidden-heading} Configure the [ANTLR scanner](/compile/scanner/antlr "Extract symbols from any programming language using the ANTLR framework") to extract symbols from any programming language using the ANTLR framework. Below is the default configuration from: [antlr/scala.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/scala.yaml "antlr/scala.yaml"){target="\_blank"}.
Edit Online
Edit ANTLR scanner Config Locally
```yaml # 🔟❎ 'compile' Scala symbol scanner configuration # The 'scala' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from Scala source files. # IMPORTANT: while set to enabled by default, the dedicated scala scanner utilizes # takes precedence over this scanner unless disabled. # To learn more see https://doc.log10x.com/compile/scanner/scalameta # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR grammar .g4 file used to # to tokenize and construct an AST for the target .Scala file. lang: scala fileExt: - .scala - .sc grammarFile: grammar/scala.g4 rootRule: compilationUnit reserved: - interface - class - enum - this # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # The 'rule' list defines the nodes within a scala AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: classParamClauses lang: scala context: class recursive: false capture: literalsOnly - name: patVarDef lang: scala context: var_assign recursive: true capture: literalsOnly - name: argumentExprs lang: scala context: method_invoke recursive: true capture: literalsOnly - name: annotation lang: scala context: annotation_invoke recursive: true capture: literalsOnly - name: objectDef lang: scala context: class recursive: true capture: allSymbols - name: paramClauses lang: scala context: class recursive: false capture: literalsOnly - name: classTemplateOpt lang: scala context: class recursive: false capture: literalsOnly - name: funDcl lang: scala context: method_decl recursive: false capture: allSymbols - name: classDef lang: scala context: class recursive: true capture: allSymbols - name: qualId lang: scala context: package recursive: true capture: allSymbols - name: traitDef lang: scala context: class recursive: true capture: allSymbols - name: funDef lang: scala context: method_decl recursive: true capture: allSymbols subRule: funSig - name: type_ lang: scala context: method_decl recursive: false capture: literalsOnly - name: blockStat lang: scala context: var_assign recursive: true capture: literalsOnly - name: templateStat lang: scala context: method_invoke recursive: true capture: literalsOnly ``` === ":material-launch: Strings-nix" #### Strings-nix {.hidden-heading} Configure [Executable scanners](/compile/scanner/executable "Launch a target process from which to read symbol values via its stdout") to launch a target process from which to read symbol values via its stdout. Below is the default configuration from: [executable/strings-nix.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/executable/strings-nix.yaml "executable/strings-nix.yaml"){target="\_blank"}.
Edit Online
Edit Executable scanner Config Locally
```yaml # # 🔟❎ 'compile' executable 'strings' configuration # This executable symbol scanner configuration uses OS-specific utilities to extract # constant string values from pre-compiled binaries (e.g. programs, shared libraries). # Any matching text values are added to the output symbol files. # The following configuration is added by default to the 10x 'compile' pipeline. # If another 'executable' scanner is defined whose 'osFilter' matches that of the # current target input file, it will take precedence over the scanners defined below. # To learn more about the 'executable' scanner, # see: https://doc.log10x.com/compile/scanner/executable # Set the 10x pipeline to 'compile' tenx: compile # =============================== Exec Options ================================ exec: # This scanner uses the Linux built-in 'strings' (https://linux.die.net/man/1/file) # command to extract symbol values from a target binary executable file - name: stringsNIX # This configuration targets NIX OSs osFilter: (nix|nux|lux|mac) # try for known extensions before invoking a 'selectorProcess' extensions: - .so - .dynlib # 'selector' processes provide a method for determining whether a file is an executable binary. # If a selector process is not provided the file's ELF/PE/OSX binary header is tested. # Since on NIX, executables do not have a known extension, # the 'selector' process uses the built-in 'file' command whose # output is used to verify the target file is a NIX executable file. selector: process: file args: - file # this macro value is replaced at run-time by the actual input file name timeout: 5s outputFilter: ".*?\\bexecutable\\b.*?" # select 'executable' files process: strings args: - -a # scan the whole file, not just initialized and loaded sections - file # This macro value is replaced at run-time by the actual text/binary input file timeout: 100s symbol: # 'minLength' specifies the min number of chars a value extracted via strings must be to # in order to qualify as a symbol. minLength: 4 # 'maxLength' specifies the max number of chars a value extracted via strings must be to # in order to qualify as a symbol. maxLength: 1024 # 'filters' defines regex patterns applied to each line read from the scanner process' stdout # that must NOT match for the line value to be considered a symbol. filters: - AWAV. # https://stackoverflow.com/questions/39322552/meaning-of-a-common-string-in-executables - "^(?=^.{0,7}$)(?=.*[^a-zA-Z]).*$" # Matches any string that is shorter than 8 characters (0 to 7 chars) and contains at least one non-alphabetic character. # 'setSelector' specifies regex patterns that determines whether to treat an input value as set of tokens setSelectors: - '^[0-9].*$' # Starts with digit - '^\*.*$' # Starts with * - '^(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}(?:/[^\s]*)?$' # Domain + optional path (e.g., github.com/user/repo) - '^type:\\..*$' # Go land metadata starts with "type:." (any suffix) ``` === ":material-launch: Strings-win" #### Strings-win {.hidden-heading} Configure [Executable scanners](/compile/scanner/executable "Launch a target process from which to read symbol values via its stdout") to launch a target process from which to read symbol values via its stdout. Below is the default configuration from: [executable/strings-win.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/executable/strings-win.yaml "executable/strings-win.yaml"){target="\_blank"}.
Edit Online
Edit Executable scanner Config Locally
```yaml # # 🔟❎ 'compile' executable 'strings' configuration # This executable symbol scanner configuration uses OS-specific utilities to extract # constant string values from pre-compiled binaries (e.g. programs, shared libraries). # Any matching text values are added to the output symbol files. # The following configuration is added by default to the 10x 'compile' pipeline. # If another 'executable' scanner is defined whose 'osFilter' matches that of the # current target input file, it will take precedence over the scanners defined below. # To learn more about the 'executable' scanner, # see: https://doc.log10x.com/compile/scanner/executable/ # Set the 10x pipeline to 'compile' tenx: compile # =============================== Exec Options ================================ exec: # This configuration provides a similar capability for scanning Windows executable files. # *IMPORTANT*: On Windows, the 'strings' command is not installed by default and # should be downloaded from: https://docs.microsoft.com/en-us/sysinternals/downloads/strings # and added to 'PATH' - name: stringsWin # target Win osFilter: win extensions: - .so - .dll process: strings args: - file # This macro value is replaced at run-time by the actual input file name timeout: 10s ```
:material-github: This app is defined in [compiler/app.yaml](https://github.com/log-10x/modules/tree/main/pipelines/compiler/app.yaml "compiler/app.yaml"){target="\_blank"}.