--- title: "ANTLR Scanner" description: "extract symbols from any programming language using the ANTLR framework" source: "https://github.com/log-10x/modules/tree/main/pipelines/compile/modules/scanner/antlr/module.yaml" icon: "material/file-code" --- Extracts [symbols](https://doc.log10x.com/run/transform/structure/#symbols) from source files using [ANTLR](http://www.antlr.org "Extract symbols from any programming language using the ANTLR framework"){target="\_blank"} parsers. The [rules](https://doc.log10x.com/compile/scanner/antlr/rules/ "Rules for extracting symbols from a target ANTLR AST") for each [language](https://doc.log10x.com/compile/scanner/antlr/langs/ "Extract symbol values from a source code in a variety of programming language using ANTLR") control which symbols to extract from the AST and their [context](https://doc.log10x.com/run/transform/symbol/#contexts) within the source file (e.g. class name, constant, enum,..). This enables generating symbol units from source code in virtually any programming language for which a [grammar](https://github.com/antlr/grammars-v4){target="\_blank"} is defined. ### :material-toy-brick-outline: Extensibility Parsing source code files for a given language can be done by: - Compiling a [g4 grammar](https://github.com/antlr/grammars-v4){target="\_blank"} file using the ANTLR [tool](https://www.antlr.org/download.html){target="\_blank"} into a parser class file and specifying its fully qualified class name via the [antlrParserClass](https://doc.log10x.com/compile/scanner/antlr/langs/#antlrparserclass "Java parser class name") argument. - Loading a .g4 grammar file directly via the [antlrGrammarFile](https://doc.log10x.com/compile/scanner/antlr/langs/#antlrgrammarfile "ANTLR .g4 grammar file") argument, assuming the target grammar does not require any language-specific code extensions. The ANTLR scanner uses either pre-compiled or interpreted (.g4) parsers to read the contents of input files and iterate their [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree){target="\_blank"}. To learn more see the [ANTLR tutorial](https://tomassetti.me/antlr-mega-tutorial){target="\_blank"}. ## :octicons-package-24: Modules
- :material-code-tags:{ .lg .middle } __ANTLR Languages__ --- Extract symbol values from a source code in a variety of programming language using ANTLR. [:octicons-arrow-right-24: More info](/compile/scanner/antlr/langs) - :material-code-tags-check:{ .lg .middle } __ANTLR Language Ruleses__ --- Rules for extracting symbols from a target ANTLR AST. [:octicons-arrow-right-24: More info](/compile/scanner/antlr/rules)
## :material-wrench-outline: Config Files To configure the ANTLR scanner module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. === "Java" Below is the default configuration from: [antlr/java.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/java.yaml "antlr/java.yaml"){target="\_blank"}.
Edit Online
Edit java.yaml Locally
```yaml # 🔟❎ 'compile' Java ANTLR symbol scanner configuration # The 'java' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from Java source files. # IMPORTANT: while set to enabled by default, the dedicated java scanner utilizes # takes precedence over this scanner unless disabled. # To learn more see https://doc.log10x.com/compile/scanner/javaParser # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR-generated parser class used to # to tokenize and construct an AST for the target Java file. lang: java fileExt: - .java parserClass: com.log10x.antlr.generated.java.Java9Parser lexerClass: com.log10x.antlr.generated.java.Java9Lexer rootRule: compilationUnit reserved: - package - interface - class - enum # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # The 'rule' list defines the nodes within a Java AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: packageDeclaration lang: java context: package recursive: false capture: allSymbols - name: methodInvocation lang: java context: method_invoke recursive: true capture: literalsOnly - name: enumDeclaration lang: java context: class recursive: false capture: allSymbols - name: annotation lang: java context: annotation_invoke recursive: true capture: literalsOnly - name: normalClassDeclaration lang: java context: class recursive: false capture: allSymbols - name: variableInitializer lang: java context: var_assign recursive: true capture: literalsOnly - name: enumConstant lang: java context: enum recursive: false capture: allSymbols - name: returnStatement lang: java context: var_assign recursive: true capture: literalsOnly - name: methodDeclaration lang: java context: method_decl recursive: false capture: literalsOnly - name: argumentList lang: java context: method_invoke recursive: true capture: literalsOnly - name: methodInvocation_lfno_primary lang: java context: method_invoke recursive: true capture: literalsOnly - name: methodDeclarator lang: java context: method_decl recursive: false capture: allSymbols - name: methodInvocation_lf_primary lang: java context: method_invoke recursive: true capture: literalsOnly - name: normalInterfaceDeclaration lang: java context: class recursive: false capture: allSymbols ``` === "Cpp" Below is the default configuration from: [antlr/cpp.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/cpp.yaml "antlr/cpp.yaml"){target="\_blank"}.
Edit Online
Edit cpp.yaml Locally
```yaml # 🔟❎ 'compile' C++ ANTLR symbol scanner configuration # The 'cpp' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from c++ source files. # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR grammar .g4 file used to # to tokenize and construct an AST for the target cpp file lang: cpp fileExt: - .cpp - .cxx - .c++ - .c - .h - .H - .hpp - .hxx - .h++ - .cc - .hh grammarFile: grammar/CPP14Parser.g4 lexerFile: grammar/CPP14Lexer.g4 rootRule: translationUnit reserved: - '::' - '<<' - inline - namespace - < - class - '>' # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ - ^.*\bnew\(\);\b.*$ # The 'rule' list defines the nodes within a cpp AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: enumerator lang: cpp context: enum recursive: false capture: allSymbols - name: blockDeclaration lang: cpp context: method_invoke recursive: true capture: literalsOnly - name: enumbase lang: cpp context: enum recursive: true capture: literalsOnly - name: classHeadName lang: cpp context: class recursive: true capture: allSymbols - name: enumSpecifier lang: cpp context: class recursive: false capture: literalsOnly condition: ^enumHead$ tag: EnumHead - name: enumHead lang: cpp context: class recursive: false capture: allSymbols ifTag: EnumHead - name: functionDefinition lang: cpp context: method_decl recursive: false capture: allSymbols condition: ^functionDefinition$ tag: FunctionDefinition - name: declaratorid lang: cpp context: method_decl recursive: true capture: allSymbols ifTag: FunctionDefinition - name: jumpStatement lang: cpp context: var_assign recursive: true capture: literalsOnly - name: functionBody lang: cpp context: method_invoke recursive: false capture: literalsOnly # condition: ^functionBody$ # tag: FunctionBody - name: statement lang: cpp context: method_invoke recursive: true capture: literalsOnly - name: classSpecifier lang: cpp context: class recursive: false capture: literalsOnly subRule: memberSpecification - name: staticAssertDeclaration lang: cpp context: var_assign recursive: true capture: literalsOnly - name: parameterDeclarationList lang: cpp context: method_invoke recursive: false capture: literalsOnly # condition: ^parameterDeclarationList$ # tag: ParameterDeclarationList - name: enumeratorDefinition lang: cpp context: enum recursive: false capture: allSymbols - name: namespaceDefinition lang: cpp context: package recursive: false capture: allSymbols - name: constantExpression lang: cpp context: var_assign recursive: true capture: literalsOnly - name: templateArgumentList lang: cpp context: method_invoke recursive: false capture: literalsOnly # condition: ^templateArgumentList$ # tag: TemplateArgumentList ``` === "Scala" Below is the default configuration from: [antlr/scala.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/scala.yaml "antlr/scala.yaml"){target="\_blank"}.
Edit Online
Edit scala.yaml Locally
```yaml # 🔟❎ 'compile' Scala symbol scanner configuration # The 'scala' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from Scala source files. # IMPORTANT: while set to enabled by default, the dedicated scala scanner utilizes # takes precedence over this scanner unless disabled. # To learn more see https://doc.log10x.com/compile/scanner/scalameta # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR grammar .g4 file used to # to tokenize and construct an AST for the target .Scala file. lang: scala fileExt: - .scala - .sc grammarFile: grammar/scala.g4 rootRule: compilationUnit reserved: - interface - class - enum - this # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # The 'rule' list defines the nodes within a scala AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: classParamClauses lang: scala context: class recursive: false capture: literalsOnly - name: patVarDef lang: scala context: var_assign recursive: true capture: literalsOnly - name: argumentExprs lang: scala context: method_invoke recursive: true capture: literalsOnly - name: annotation lang: scala context: annotation_invoke recursive: true capture: literalsOnly - name: objectDef lang: scala context: class recursive: true capture: allSymbols - name: paramClauses lang: scala context: class recursive: false capture: literalsOnly - name: classTemplateOpt lang: scala context: class recursive: false capture: literalsOnly - name: funDcl lang: scala context: method_decl recursive: false capture: allSymbols - name: classDef lang: scala context: class recursive: true capture: allSymbols - name: qualId lang: scala context: package recursive: true capture: allSymbols - name: traitDef lang: scala context: class recursive: true capture: allSymbols - name: funDef lang: scala context: method_decl recursive: true capture: allSymbols subRule: funSig - name: type_ lang: scala context: method_decl recursive: false capture: literalsOnly - name: blockStat lang: scala context: var_assign recursive: true capture: literalsOnly - name: templateStat lang: scala context: method_invoke recursive: true capture: literalsOnly ``` === "Go" Below is the default configuration from: [antlr/go.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/go.yaml "antlr/go.yaml"){target="\_blank"}.
Edit Online
Edit go.yaml Locally
```yaml # 🔟❎ 'compile' Go ANTLR symbol scanner configuration # The 'go' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from GO source files. # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR-generated parser class used to # to tokenize and construct an AST for the target go file. lang: go fileExt: - .go parserClass: com.log10x.antlr.generated.golang.GoParser lexerClass: com.log10x.antlr.generated.golang.GoLexer rootRule: sourceFile reserved: - package - func # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # The 'rule' list defines the nodes within a go AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: constSpec lang: go context: enum recursive: false capture: allSymbols subRule: identifierList - name: returnStmt lang: go context: var_assign recursive: true capture: literalsOnly - name: assignment lang: go context: var_assign recursive: true capture: literalsOnly - name: expressionStmt lang: go context: method_invoke recursive: true capture: literalsOnly - name: packageClause lang: go context: package recursive: false capture: allSymbols - name: arguments lang: go context: method_invoke recursive: true capture: literalsOnly - name: typeSpec lang: go context: class recursive: false capture: allSymbols - name: functionDecl lang: go context: method_decl recursive: false capture: allSymbols - name: methodDecl lang: go context: method_decl recursive: false capture: allSymbols ``` === "Javascript" Below is the default configuration from: [antlr/javascript.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/javascript.yaml "antlr/javascript.yaml"){target="\_blank"}.
Edit Online
Edit javascript.yaml Locally
```yaml # 🔟❎ 'compile' JavaScript ANTLR symbol scanner configuration # The 'javascript' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from JavaScript source files. # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR-generated parser class used to # to tokenize and construct an AST for the target JavaScript file. lang: javascript fileExt: - .js parserClass: com.log10x.antlr.parsers.JavaScriptDecoratedParser lexerClass: com.log10x.antlr.generated.javascript.JavaScriptLexer rootRule: program2 reserved: - function - constructor - class maxLineLength: 4096 # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # The 'rule' list defines the nodes within a js AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: functionDeclaration lang: javascript context: method_decl recursive: false capture: allSymbols subRule: identifier - name: returnStatement lang: javascript context: var_assign recursive: true capture: literalsOnly - name: classDeclaration lang: javascript context: class recursive: false capture: allSymbols subRule: identifier - name: methodDefinition lang: javascript context: method_decl recursive: true capture: allSymbols subRule: identifier - name: formalParameterList lang: javascript context: method_decl recursive: false capture: literalsOnly - name: functionBody lang: javascript context: method_invoke recursive: false capture: literalsOnly - name: functionDecl lang: javascript context: method_decl recursive: false capture: allSymbols - name: anoymousFunctionDecl lang: javascript context: method_decl recursive: false capture: allSymbols - name: argumentsExpression lang: javascript context: method_invoke recursive: true capture: literalsOnly - name: variableDeclaration lang: javascript context: var_assign recursive: true capture: literalsOnly - name: assignmentExpression lang: javascript context: var_assign recursive: true capture: literalsOnly ``` === "Python" Below is the default configuration from: [antlr/python.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/python.yaml "antlr/python.yaml"){target="\_blank"}.
Edit Online
Edit python.yaml Locally
```yaml # 🔟❎ 'compile' Python symbol scanner configuration # The 'python' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from Python source files. # NOTE: while set to enabled by default, the 'pythonAST' scanner which utilizes # the python run-time's built-in AST parser takes precedence over this scanner # unless a python run-time is not installed or the 'pythonAST' scanner is disabled. # To learn see https://doc.log10x.com/compile/scanner/pythonAST # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR-generated parser class used to # to tokenize and construct an AST for the target Python file. lang: python fileExt: - .py parserClass: com.log10x.antlr.generated.python.PythonParser lexerClass: com.log10x.antlr.generated.python.PythonLexer rootRule: root reserved: - def - __str__ - __init__ - class # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # The 'rule' list defines the nodes within a Python AST tree that are captured # and added to the output symbol unit. # To learn more see: https://doc.log10x.com/compile/scanner/antlr/rules/ rule: - name: classdef lang: python context: class recursive: false capture: allSymbols subRule: name condition: (^Enum$|^enum$) tag: enumLiteral - name: testlist_star_expr lang: python context: enum recursive: true capture: allSymbolsIfMatchCond ifTag: enumLiteral - name: trailer lang: python context: method_invoke recursive: true capture: literalsOnly subRule: name - name: return_stmt lang: python context: var_assign recursive: true capture: literalsOnly - name: decorator lang: python context: annotation_invoke recursive: true capture: literalsOnly subRule: arglist - name: arglist lang: python context: method_invoke recursive: true capture: literalsOnly - name: funcdef lang: python context: method_decl recursive: false capture: allSymbols subRule: name - name: assign_part lang: python context: var_assign recursive: true capture: literalsOnly - name: dictorsetmaker lang: python context: var_assign recursive: true capture: literalsOnly ``` === "Csharp" Below is the default configuration from: [antlr/csharp.yaml](https://github.dev/log-10x/config/blob/main/pipelines/compile/scanners/antlr/csharp.yaml "antlr/csharp.yaml"){target="\_blank"}.
Edit Online
Edit csharp.yaml Locally
```yaml # 🔟❎ 'compile' C# ANTLR symbol scanner configuration # The 'csharp' configuration instructs the ANTLR scanner which symbol values (e.g. class/func names, code constants) # to extract from c# source files. # Set the 10x pipeline to 'compile' tenx: compile # ============================== ANTLR Options ================================ antlr: # The 'antlrLang' module is defined in: https://doc.log10x.com/compile/scanner/antlr/langs # These values define the ANTLR-generated parser class used to # to tokenize and construct an AST for the target c# file. lang: csharp fileExt: - .csx - .cs parserClass: com.log10x.antlr.generated.csharp.CSharpParser lexerClass: com.log10x.antlr.generated.csharp.CSharpLexer # 'charStreamClass' is set to remove language elements introduced later than V6.0 which is the latest release supported by this grammar. # To learn more see https://github.com/antlr/grammars-v4/tree/master/csharp#readme charStreamClass: com.log10x.antlr.parsers.CSharpDowngrader rootRule: compilation_unit reserved: - namespace - interface - class - enum # 'lineFilters' specifies a list of regex patterns for skipping input lines lineFilters: - ^\s*(//.*|/\*.*\*/)$ # Remove unsupported C# v10 File scoped namespaces - ^namespace.*;$ # Remove unsupported C# v9 Target-typed new operator # - "(?:material-github: This module is defined in [antlr/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/compile/modules/scanner/antlr/module.yaml "antlr/module.yaml"){target="\_blank"}.