<div align="center"><img src="https://github.com/bhsd-harry/wikiparser-node/raw/main/logo.png" width="200" alt="WikiParser-Node logo"></div><!-- markdownlint-disable-line line-length -->

# WikiParser-Node

[![npm version](https://badge.fury.io/js/wikiparser-node.svg)](https://www.npmjs.com/package/wikiparser-node)
[![CodeQL](https://github.com/bhsd-harry/wikiparser-node/actions/workflows/codeql.yml/badge.svg)](https://github.com/bhsd-harry/wikiparser-node/actions/workflows/codeql.yml)
[![CI](https://github.com/bhsd-harry/wikiparser-node/actions/workflows/node.js.yml/badge.svg)](https://github.com/bhsd-harry/wikiparser-node/actions/workflows/node.js.yml)
[![NPM downloads](https://img.shields.io/npm/dm/wikiparser-node)](https://www.npmjs.com/package/wikiparser-node)
[![jsDelivr hits (npm)](https://img.shields.io/jsdelivr/npm/hm/wikiparser-node)](https://www.npmjs.com/package/wikiparser-node)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/a2fbe7641031451baca2947ae6d7891f)](https://app.codacy.com/gh/bhsd-harry/wikiparser-node/dashboard)
![Coverage](./coverage/badge.svg)
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/13661/badge)](https://www.bestpractices.dev/projects/13661)

## Other Languages

- [简体中文](./README-%28ZH%29.md)

## Introduction

WikiParser-Node is an offline [Wikitext](https://www.mediawiki.org/wiki/Wikitext)
parser developed by [Bhsd](https://github.com/bhsd-harry) for the [Node.js](https://nodejs.org/)
environment. It can parse almost all [wiki syntax](https://www.mediawiki.org/wiki/Help:Advanced_editing)
and generate an [Abstract Syntax Tree (AST)](https://en.wikipedia.org/wiki/Abstract_syntax_tree)
([Try it online](https://bhsd-harry.github.io/wikiparser-node/#editor)). It also
allows for easy querying and modification of the AST, and returns the modified Wikitext.

Although WikiParser-Node is not primarily designed to convert Wikitext to HTML,
it provides pragmatic HTML rendering for many situations. [Here](https://bhsd-harry.github.io/wikiparser-website/)
is a list of example HTML pages from [MediaWiki.org](https://www.mediawiki.org/)
rendered using this package.

WikiParser-Node has been extensively tested against the official
[MediaWiki PHP parser tests](https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master/tests/parser/)
with ~3,000 test cases, covering various edge cases and peculiarities of
Wikitext. These tests are available [here](https://bhsd-harry.github.io/wikiparser-node/tests.html).

## Why WikiParser-Node

- **Round-trip editing for bots and automation**: parse Wikitext into an AST,
  query and modify nodes, then write back valid Wikitext.
- **LSP and linting ready for Node.js tooling**: powers [WikiLint](https://www.npmjs.com/package/wikilint)
  and [Wikitext LSP](https://www.npmjs.com/package/wikitext-lsp).
- **Browser/editor integration**: works with [CodeMirror](https://www.npmjs.com/package/@bhsd/codemirror-mediawiki),
  [Monaco](https://www.npmjs.com/package/monaco-wiki), and MediaWiki's official
  [CodeMirror extension](https://www.mediawiki.org/wiki/Extension:CodeMirror).
- **Large-scale usage evidence**:
  [full-dump parsing and linting on English Wikipedia scale](https://lint-wiki-dumps.toolforge.org/)
  is practical on consumer hardware.
- **Transparent quality signals**: [CI](https://github.com/bhsd-harry/wikiparser-node/actions/workflows/node.js.yml),
  [CodeQL](https://github.com/bhsd-harry/wikiparser-node/actions/workflows/codeql.yml),
  public [parser-test results](https://bhsd-harry.github.io/wikiparser-node/tests.html),
  and coverage are all visible in this repository.

## Used by
<!-- markdownlint-disable line-length -->
<div align="center">
<a href="https://helix-editor.com/"><img src="https://helix-editor.com/logo.svg" width="50" height="50" alt="Helix"></a>
<a href="http://www.qbittorrent.org/"><img src="https://raw.githubusercontent.com/qbittorrent/qBittorrent/master/src/icons/qbittorrent-tray.svg" width="50" height="50" alt="qBittorrent"></a>
<a href="https://www.mediawiki.org/"><img src="https://www.mediawiki.org/static/images/icons/mediawikiwiki.svg" width="50" height="50" alt="MediaWiki"></a>
<a href="https://euporie.readthedocs.io/en/latest/"><img src="https://raw.githubusercontent.com/joouha/euporie/dev/docs/_static/favicon.svg" width="50" height="50" alt="euporie"></a>
<a href="https://flow-control.dev/"><img src="https://github.com/neurocyte/flow-website/blob/master/assets/img/flow-control.png?raw=true" width="50" height="50" alt="Flow Control"></a>
<a href="https://lsp.sublimetext.io/"><img src="https://raw.githubusercontent.com/sublimelsp/LSP/main/icons/lsp_logo.svg" width="50" height="50" alt="LSP for Sublime Text"></a>
</div>
<!-- markdownlint-enable line-length -->

## Other Versions

### [WikiLint](https://www.npmjs.com/package/wikilint)

This version provides a [CLI](https://en.wikipedia.org/wiki/Command-line_interface),
but only retains the parsing and linting functionality. At the cost of losing
AST modification and HTML rendering capabilities, it executes [2.4x faster](#performance).
Projects can use WikiLint in their CI pipelines to automatically lint wiki
content (e.g., [qBittorrent](https://github.com/qbittorrent/wiki/blob/master/.pre-commit-config.yaml)).

It powers the [Wikitext LSP](https://www.npmjs.com/package/wikitext-lsp), which
provides multiple language services for editors such as [VS Code](https://marketplace.visualstudio.com/items?itemName=Bhsd.vscode-extension-wikiparser),
[Sublime Text](https://lsp.sublimetext.io/language_servers/#mediawiki), and [Helix](https://github.com/helix-editor/helix/wiki/Language-Server-Configurations#wikitext).

A list of available linting rules can be found [here](https://github.com/bhsd-harry/wikiparser-node/wiki/Rules).

**Note**: The major version number of WikiLint is always higher than that of
WikiParser-Node by 1.

### [WikiParser-Template](https://www.npmjs.com/package/wikiparser-template)

A lightweight version that only supports parsing and manipulation of templates.
This version is designed for use cases where only template processing is needed,
such as certain types of bots or web tools (e.g., [GANReviewTool](https://en.wikipedia.org/wiki/User:Novem_Linguae/Scripts/GANReviewTool))
that focus on template manipulation.

### [Browser-compatible](https://github.com/bhsd-harry/wikiparser-node/wiki/Browser-%28EN%29)

A browser-compatible version, which can be used for code highlighting or as a
[LSP](https://microsoft.github.io/language-server-protocol/) plugin in
conjunction with editors such as [CodeMirror](https://www.npmjs.com/package/@bhsd/codemirror-mediawiki)
and [Monaco](https://www.npmjs.com/package/monaco-wiki) ([Usage example](https://bhsd-harry.github.io/wikiparser-node)).
It has been integrated into the MediaWiki official [CodeMirror extension](https://www.mediawiki.org/wiki/Extension:CodeMirror)
since MediaWiki Release 1.45.

To ensure maximum responsiveness within browser environments, the parser is
optimized for speed and memory usage while AST manipulation and HTML rendering
are intentionally excluded to prevent main-thread blocking.

## Installation

### Node.js

Please install the corresponding version as needed (`WikiParser-Node` or
`WikiLint`), for example:

```sh
npm i wikiparser-node
```

or

```sh
npm i wikilint
```

#### Optional dependencies

You can install any of the following optional packages to enable advanced features:

| Package | Purpose |
| :-: | --- |
| `mathoid-texvcjs` | Diagnostics and HTML conversion support for `<math>` and `<chem>` extension tags |
| `stylelint-config-recommended` | Diagnostics for `style` attributes |
| `vscode-css-languageservice` | CSS language support for `style` attributes |
| `vscode-html-languageservice` | Language support for HTML tags |
| `vscode-json-languageservice` | JSON language support for `<templatedata>`, `<mapframe>` and `<maplink>` tags |
| `ajv` + `ajv-errors` | HTML conversion support for `<templatedata>` extension tags |
| `katex` | HTML conversion support for `<math>` and `<chem>` extension tags |
| `prismjs` + `prism-wiki` | Syntax highlighting of `<syntaxhighlight>` extension tags |

```sh
# Full LanguageService features for `Parser.createLanguageService()`
npm i mathoid-texvcjs stylelint-config-recommended vscode-css-languageservice vscode-html-languageservice vscode-json-languageservice

# Full HTML conversion support for `Parser.toHtml()`
npm i ajv ajv-errors katex mathoid-texvcjs prismjs prism-wiki
```

### Browser

You can download the code via CDN, for example:

```html
<script src="//cdn.jsdelivr.net/npm/wikiparser-node"></script>
```

or

```html
<script src="//unpkg.com/wikiparser-node/bundle/bundle-lsp.min.js"></script>
```

For more browser extensions, please refer to the corresponding [documentation](https://github.com/bhsd-harry/wikiparser-node/wiki/Browser-%28EN%29).

## Usage

### CLI usage

For MediaWiki sites with the [CodeMirror extension](https://mediawiki.org/wiki/Extension:CodeMirror)
installed, such as different language editions of Wikipedia and other
[Wikimedia Foundation-hosted sites](https://meta.wikimedia.org/wiki/Special:SiteMatrix),
you can use the following command to obtain the parser configuration:

```sh
npx getParserConfig <site> <script path> [user] [force]
# For example:
npx getParserConfig frwiki https://fr.wikipedia.org/w user@example.net
```

The generated configuration file will be saved in the [`config` directory](https://github.com/bhsd-harry/wikiparser-node/tree/main/config/).
You can then use the site name for [`Parser.config`](https://github.com/bhsd-harry/wikiparser-node/wiki/Parser-%28EN%29#config).

```javascript
// For example:
Parser.config = "frwiki";
```

### API usage

Please refer to the [Wiki](https://github.com/bhsd-harry/wikiparser-node/wiki/Home-%28EN%29).
In particular, there are some [usage examples](https://github.com/bhsd-harry/wikiparser-node/wiki/Home-%28EN%29#examples)
that demonstrate how to use this package to complete various tasks.

#### Round-trip editing quickstart (TypeScript)

```ts
import Parser from "wikiparser-node";
import type {TranscludeToken} from "wikiparser-node";

Parser.config = "enwiki";
const root = Parser.parse("{{Infobox|name=Old}}\nText"),
	template = root.querySelector<TranscludeToken>("template#Template:Infobox");
template?.setValue("name", "New");
const wikitext = String(root);
assert.strictEqual(wikitext, "{{Infobox|name=New}}\nText");
```

## Performance

WikiParser-Node prioritizes syntax accuracy and compliance over synthetic
benchmarks. It does not chase higher speed by cutting corners on complex edge
cases, but its real-world performance remains reliable for production use.

A full [scan](https://www.npmjs.com/package/lint-wiki-dumps) (parsing and
linting) of an English Wikipedia database dump (`*.xml.bz2`) with ~19 million
articles—from short stubs to massive, template-heavy pages—takes about 5 hours
(wall-clock) on a personal MacBook Air.

*Performance comparison between WikiParser-Node and [WikiLint](#wikilint)*

| Version | Average time per article | Max time per article |
| :-: | :-: | :-: |
| WikiParser-Node | 3.6ms | 2.8s |
| WikiLint | 1.5ms | 0.75s |

## Best fit

- MediaWiki bot workflows that require robust AST manipulation and
  round-trip-safe edits.
- Node.js pipelines for linting and refactoring Wikitext.
- LSP-based language tooling.
- Browser-side editing helpers, and gadgets/user scripts that require Wikitext parsing.

## Reliability and limitations

WikiParser-Node has been extensively validated against the official
[MediaWiki PHP parser tests](https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master/tests/parser/)
(~3,000 integration test cases), ensuring robust compliance with the Wikitext
quirks and edge cases. However, for complete transparency, the following
limitations are documented.

### Parser

1. Memory leaks may occur in rare cases.
1. Invalid page names with unicode characters are treated like valid ones ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Render%20invalid%20page%20names%20as%20plain%20text%20(T53090))).
1. Preformatted text with a leading space is only processed by [`Token#toHtml`](https://github.com/bhsd-harry/wikiparser-node/wiki/Token-%28EN%29#tohtml).
1. Link text containing `!!` in a table header ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Exclamation%20points%20in%20wikilink%20content%20in%20table%20precedence)).
1. Seemingly broken file link syntax in a table ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#File%20in%20table%20pipe%20precedence)).

### Linter

1. HTML tag pairs may not match when nested inside a list item ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#1.%20Lists%20and%20block-element%20interaction)),
   but the linter will not report it.

### HTML conversion

<details>
	<summary>Expand</summary>

#### Extension

1. Many [extensions](https://github.com/bhsd-harry/wikiparser-node/wiki/Extensions-%28EN%29#support)
   are not supported.
1. [Sub-referencing](https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Sub-referencing)
   is not supported.
1. Special cases involving templates inside the `link` parameter of a gallery
   image ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Link%20media%20option%20with%20template%20with%20arguments)).

#### Transclusion

1. Some parser functions are not supported, notably [`#formatdate`](https://bhsd-harry.github.io/wikiparser-node/tests.html#formatdate%20parser%20function),
   [`#time`](https://bhsd-harry.github.io/wikiparser-node/tests.html#Input%20times%20are%20UTC%2C%20not%20local%20time)
   and [`#invoke`](https://bhsd-harry.github.io/wikiparser-node/tests.html#Scribunto%3A%20hello%20world).
1. New lines in `{{localurl:}}` are not handled correctly ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Basic%20%7B%7Blocalurle%3A%7D%7D%20test)).

#### Heading

1. RTL support in the table of contents (TOC) ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#span%20tags%20with%20directionality%20in%20TOC)).

#### HTML tag

1. Style sanitization is sometimes different ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#CSS%20safety%20test%20(all%20browsers)%3A%20vertical%20tab%20(T57332%20%2F%20CVE-2013-4567))).

#### Table

1. `<caption>` elements are wrapped in `<tbody>` elements ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Trim%20whitespace%20in%20wikitext%20headings%2C%20list%20items%2C%20table%20captions%2C%20headings%2C%20and%20cells)).
1. Unclosed HTML tags in the table fostered content ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Fuzz%20testing%3A%20Parser24)).
1. `<tr>` elements should not be fostered ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Empty%20TR%20followed%20by%20a%20template-generated%20TR)).

#### Link

1. Link trail is not supported ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#1.%20Interaction%20of%20linktrail%20and%20template%20encapsulation)).
1. Block elements inside a link should break it into multiple links ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#%3Cpre%3E%20inside%20a%20link)).
1. Invalid or missing images (Examples [1](https://bhsd-harry.github.io/wikiparser-node/tests.html#Fuzz%20testing%3A%20image%20with%20bogus%20manual%20thumbnail),
   [2](https://bhsd-harry.github.io/wikiparser-node/tests.html#Gallery%20with%20line%20not%20in%20the%20file%20namespace)).
1. Link starting with `../` on a subpage ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Parent%20directory%20in%20redirect)).
1. Duplicate file namespace prefixes should be invalid ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Gallery%20with%20valid%20attributes)).
1. File redirects ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Gallery%20(with%20showfilename%20option))).
1. Figure caption should not be nested inside inline elements ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#File%20in%20formatting%20element%20violating%20content%20model%2C%20figcaption%20content)).

#### External link

1. No percent-encoding in displayed free external links ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Parsoid%3A%20pipe%20in%20transclusion%20parameter)).

#### Block element

1. Incomplete `<p>` wrapping when there are block elements (e.g., [`<pre>`](https://bhsd-harry.github.io/wikiparser-node/tests.html#%3Cpre%3E%20inside%20a%20link),
   [`<div>`](https://bhsd-harry.github.io/wikiparser-node/tests.html#Templates%3A%20Scopes%20should%20not%20be%20expanded%20unnecessarily)
   or even [closing tags](https://bhsd-harry.github.io/wikiparser-node/tests.html#Non-word%20characters%20don't%20terminate%20tag%20names%20(T19663%2C%20T42670%2C%20T54022))).
1. Mixed lists ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Mixed%20Lists%3A%20Test%204)).
1. `<p>` wrapping and preformatted text are not supported inside `<indicator>`.

#### Language conversion

1. Automatic language conversion is not supported.
1. Support for manual language conversion is minimal ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Explicit%20session-wise%20one-way%20language%20variant%20mapping%20(A%20flag%20and%20-%20flag))).
1. Language conversion should prevent `<p>` wrapping (Examples [1](https://bhsd-harry.github.io/wikiparser-node/tests.html#Code%20coverage%3A%20rules%20with%20no%20variants),
   [2](https://bhsd-harry.github.io/wikiparser-node/tests.html#Language%20converter%20markup%20with%20block%20content)).

#### Miscellaneous

1. Illegal HTML entities ([Example](https://bhsd-harry.github.io/wikiparser-node/tests.html#Illegal%20character%20references%20(T106578%2C%20T113194))).

</details>
