# html-data-parser 1.1.x
Parse and stream tabular data from HTML documents using Node.js and [isaacs/sax-js](https://github.com/isaacs/sax-js).
This readme explains how to use html-data-parser in your code or as a console program using the command line interface (CLI).
> Only supports HTML documents containing TABLE elements. Does not support parsing grid or other table like elements.
Related projects:
[html-link-parser](https://github.com/drewletcher/html-link-parser#readme) |
[text-data-parser](https://github.com/drewletcher/text-data-parser#readme) |
[pdf-data-parser](https://github.com/drewletcher/pdf-data-parser#readme) |
[xlsx-data-parser](https://github.com/drewletcher/xlsx-data-parser#readme)
## Installation
For use as module in a Node.js project. See Developers Guide below.
```bash
npm install html-data-parser
```
For use as command line utility. Requires Node.js 18+.
```bash
npm -g install html-data-parser
```
## Command Line Interface
Parse tabular data from an HTML document.
```bash
hdp --options=filename.json --heading=title --id=name --cells=# --headers=name1,name2,... --format=csv|json|rows
`filename|URL` - path name or URL of HTML file to process, required.
`output-file` - local path name for output of parsed data, default stdout.
`--options` - JSON or JSONC file containing JSON object with hdp options, default: hdp.options.json.
`--heading` - text of heading to find in document that precedes desired data table, default none.
`--id` - TABLE element id attribute to find in document.
`--cells` - number of cells in a data row, minimum or "min-max", default = "1-256".
`--headers` - comma separated list of column names for data, default none the first table row contains names.
`--format` - output data format CSV, JSON, or ROWS (JSON array of arrays), default JSON.
```
Note: If the `hdp` command conflicts with another program on your system use `htmldataparser` instead.
### Options File
The options file supports options for all html-data-parser modules. Parser will read plain JSON files or JSONC files with Javascript style comments. The default name of the options file is `hdp.options.json` located in the current working directory.
```javascript
{
/* HtmlDataParser options */
// url - local path name or URL of HTML file to process, required.
"url": "",
// output - local path name for output of parsed data, default stdout.
"output": "",
// format - output data format CSV, JSON or rows, default JSON, rows is JSON array of arrays (rows).
"format": "json",
// heading - text of heading to find in document that precedes desired data table, default none.
"heading": null,
// id - TABLE element id attribute to find in document.
"id": "",
// cells - number of cells for a data row, minimum or "min-max", default = "1-256".
"cells": "1-256",
// newlines - preserve new lines in cell data, default: false.
"newlines": false,
// trim whitespace from output values, default: true.
"trim": true,
/* RowAsObjectTransform options */
// hasHeaders - data has a header row, if true and headers set then headers overrides header row.
"RowAsObject.hasHeader": false
// headers - comma separated list of column names for data, default none. When not defined the first table row encountered will be treated as column names.
"RowAsObject.headers": []
/* RepeatCellTransform options */
// column - column index of cell to repeat, default 0.
"RepeatCell.column": 0
/* RepeatHeadingTransform options */
// hasHeaders - data has a header row, if true and headers set then headers overrides header row.
"RepeatHeading.hasHeader": true
// header - column name for the repeating heading field. Can optionally contain suffix :m:n with index for inserting into header and data rows.
"RepeatHeading.header": "subheading:0:0"
/* HTTP options */
// see HTTP Options below
}
```
Note: Transform property names can be shortened to `hasHeader`, `headers`, `column` and `header`.
### Examples
```bash
hdp ./test/data/html/helloworld.html --headers="Greeting" --format=csv
```
```bash
hdp ./test/data/html/helloworld.html --id="cosmic" --headers="BigBang"
```
```bash
hdp ./test/data/html/ansi.html --heading="Congressional Districts"
```
```bash
hdp https://www.sos.state.tx.us/elections/historical/jan2024.shtml ./test/output/hdp/tx_voter_reg.json
```
```bash
hdp --options="./test/RepeatCell.options.json"
RepeatCell.options.json:
{
"url": "./test/data/html/texas_jan2024.shtml",
"output": "./test/output/hdp/repeat_cell.json",
"format": "json",
"cells": 7,
"RepeatCell.column": 0
}
```
## Developer Guide
### Basic Usage
The parser processes the entire document then returns the row data as an array of arrays.
```javascript
import { HtmlDataParser } from "html-data-parser";
let parser = new HtmlDataParser({url: "filename.html"});
async function parseDocument() {
var rows = await parser.parse();
// process the rows
}
```
### Using Event Interface
Listen to parser events and process each row as it is parsed from the document.
```javascript
import { HtmlDataParser } from "html-data-parser";
let parser = new HtmlDataParser({url: "filename.html"});
parser.on('head', (head) => {
// triggered by /HEAD end tag
// zero or one event per document
// see head object below
})
parser.on('data', (row) => {
// process row, row is an array of cell values
});
parser.on('end', () => {
});
parser.on('error', (err) => {
// log error
})
```
```javascript
head = {
url, // starting url
title, // text from element
redirect, // HTTP status code of last redirect - 301, 302, 307 or 308
location // URL of last redirect
}
```
### Using Stream Interface
Use the NodeJS Stream interface to process rows as they are parsed from the document.
```javascript
import { HtmlDataReader } from "html-data-parser";
import { pipeline } from 'node:stream/promises';
let reader = new HtmlDataReader(options);
let writer = `<some writable that can handle Object Mode data>`
await pipeline(reader, writer);
```
## Class HtmlDataParser
`HtmlDataParser` given a HTML document will output an array of arrays (rows). Use the streaming classes `HtmlDataReader` and `RowAsObjectTransform` transform to convert the arrays to Javascript objects. With default settings `HtmlDataParser will output rows in __all__ TABLE found in the document. Using [HtmlDataParser Options](#html-data-parser-options) `heading` or `id` the parser can filter content to retrieve the desired data TABLE in the document.
`HtmlDataParser` only works on a certain subset of HTML documents specifically those that contain some TABLE elements and NOT other table like grid elements. The parser uses [isaacs/sax-js](https://github.com/isaacs/sax-js) library to transform HTML table elements into rows of cells.
Rows and Cells terminology is used instead of Rows and Columns because the content in a HTML document flows rather than being strict rows/columns like database query results. Some rows may have more cells than other rows. For example a heading or description paragraph will be a row (array) with one cell (string). See [Notes](#notes) below.
### HtmlDataParser Options
`HtmlDataParser` constructor takes an options object with the following fields. One of `url` or `data` arguments is required.
`{String|URL} url` - The local path or URL of the HTML document.
`{String|Uint8Array} data` - HTML document in a string.
`{Readable} rs` - Readable stream for the HTML document.
Common Options:
`{String|RegExp} heading` - Heading, H1-H6 element, in the document after which the parser will look for a TABLE; optional, default: none. The parser does a string comparison or regexp match looking for first occurrence of `heading` value in a heading element. If neither `heading` or `id` are specified then data output contains all rows from all tables found in the document.
`{String|RegExp} id` - TABLE element id attribute in the document to parse for tabular data; optional, default: none. The parser does a string comparison of the `id` value in TABLE elements ID attribute. If neither `heading` or `id` are specified then data output contains all rows from all tables found in the document.
`{Number} cells` - Minimum number of cells in tabular data; optional, default: 1. The parser will NOT output rows with less than `cells` number of cells.
`{Boolean} newlines` - Preserve new lines in cell data; optional, default: false. When false newlines will be replaced by spaces. Preserving newlines characters will keep the formatting of multiline text such as descriptions. Though, newlines are problematic for cells containing multi-word identifiers and keywords that might be wrapped in the cell text.
`{Boolean} trim` - trim whitespace from output values, default: true.
### HTTP Options
HTTP requests are mode using Node.js HTTP modules. See the source code file lib/httpRequest.js for more details.
`{Object} http` - options to pass thru to HTTP request
`{String} http.method` - HTTP method, default is "GET"
`{Object} http.params` - object containing URL querystring parameters.
`{Object} http.headers` - object containing HTTP headers
`{Array} http.cookies` - array of HTTP cookie strings
`{String} http.auth` - string for Basic Authentication (Authorization header), i.e. "user:password".
## Class HtmlDataReader
`HtmlDataReader` is a Node.js stream reader implemented with the Object mode option. It uses `HtmlDataParser` event interface to stream one data row (array) per chunk.
### HtmlDataReader Options
`HtmlDataReader` constructor options are the same as [HtmlDataParser Options](#html-data-parser-options).
## Class RowAsObjectTransform
`HtmlDataReader` operates in Object Mode. The reader outputs arrays (rows). To convert rows into Javascript objects use the `RowAsObjectTransform` transform. `RowAsObjectTransform` operates in Object mode where output is a Javascript Object of <name,value> pairs.
```javascript
import { HtmlDataReader, RowAsObjectTransform } from "html-data-parser";
import { pipeline } from 'node:stream/promises';
let reader = new HtmlDataReader(options);
let transform1 = new RowAsObjectTransform(options);
let writable = `<some writable that can handle Object Mode data>`
await pipeline(reader, transform1, writable);
```
### RowAsObjectTransform Options
`RowAsObjectTransform` constructor takes an options object with the following fields.
`{String[]} headers` - array of cell property names; optional, default: none. If a headers array is not specified then parser will assume the first row found contains cell property names.
`{Boolean} hasHeaders` - data has a header row, if true and headers options is set then provided headers override header row. Default is true.
If a row is encountered with more cells than in the headers array then extra cell property names will be the ordinal position. For example if the data contains five cells, but only three headers where specified. Specifying `options = { headers: [ 'name', 'type', 'info' ] }` then the Javascript objects in the stream will contain `{ "name": "value1", "type": "value2", "info": "value3", "4": "value4", "5": "value5" }`.
## Class RepeatCellTransform
`RepeatCellTransform` will normalize data the was probably generated by a report writer. The specified cell will be repeated in following rows that contain one less cell.
```javascript
import { HtmlDataReader, RepeatCellTransform } from "html-data-parser";
import { pipeline } from 'node:stream/promises';
let reader = new HtmlDataReader(options);
let transform1 = new RepeatCellTransform({ column: 0 });
let writable = <some writable that can handle Object Mode data>
await pipeline(reader, transform1, writable);
```
### RepeatCellTransform Options
`RepeatCellTransform` constructor takes an options object with the following fields.
`{Number} column` - column index of cell to repeat, default 0.
### Example
In this example "Dewitt" will be repeated in rows 2 and 3.
**HTML Document**
```
County Precincts Date/Period Total
Dewitt 44 JUL 2023 52,297
44 OCT 2023 52,017
44 JAN 2024 51,712
```
**Output**
```
[ "County", "Precincts", "Date/Period", "Total" ]
[ "Dewitt", "44", "JUL 2023", "52,297" ]
[ "Dewitt", "44", "OCT 2023", "52,017" ]
[ "Dewitt", "44", "JAN 2024", "51,712" ]
```
## Class RepeatHeadingTransform
`RepeatHeadingTransform` will normalize data the was probably generated by a report writer. Subheadings are rows containing a single cell interspersed in data rows. The header name is inserted in to the header row. The subheading value will be repeated in rows that follow until another subheading is encountered.
```javascript
import { HtmlDataReader, RepeatHeadingTransform } from "html-data-parser";
import { pipeline } from 'node:stream/promises';
let reader = new HtmlDataReader(options);
let transform1 = new RepeatHeadingTransform({header: "County:1:0"});
let writable = <some writable that can handle Object Mode data>
await pipeline(reader, transform1, writable);
```
### RepeatHeadingTransform Options
`RepeatHeadingTransform` constructor takes an options object with the following fields.
`{String} header` - column name for the repeating heading field. Can optionally contain an index of where to insert the header in the header row. Default "heading:0".
`{Boolean} hasHeaders` - data has a header row, if true and headers options is set then provided headers override header row. Default is true.
### Example
In this example `options = {header: "County:1:0"}`.
**HTML Document**
```
District Precincts Total
Congressional District 5
Maricopa 120 403,741
Pinal 30 102,512
Total: 150 506,253
```
**Output**
```
[ "District", "County", "Precincts", "Total" ]
[ "Congressional District 5", "Maricopa", "120", "403,741" ]
[ "Congressional District 5", "Pinal", "30", "102,512" ]
[ "Congressional District 5", "Total:", "150", "506,253" ]
```
## Class FormatCSV and FormatJSON
The `hdpdataparser` CLI program uses the `FormatCSV` and `FormatJSON` transforms to stringify Javascript objects that can be saved to a file.
```javascript
import { HtmlDataReader, RowAsObjectTransform, FormatCSV } from "html-data-parser";
import { pipeline } from 'node:stream/promises';
let reader = new HtmlDataReader(options);
let transform1 = new RowAsObjectTransform(options);
let transform2 = new FormatCSV();
await pipeline(reader, transform1, transform2, process.stdout);
```
## Examples
---
In the source code the `html-data-parser.js` program and the Javascript files in the `/test` folder are good examples of using the library modules.
### Hello World
[HelloWorld.html](./test/data/html/helloworld.html) is a single page HTML document with the string "Hello, world!" positioned on the page. The `HtmlDataParser` output is one row with one cell.
```json
[
["Hello, world!"]
]
```
To transform the row array into an object specify the headers option to `RowAsObjectTransform` transform.
```javascript
let transform = new RowAsObjectTransform({
headers: [ "Greeting" ]
})
```
Output as JSON objects:
```json
[
{ "Greeting": "Hello, world!" }
]
```
## Notes
---
* Only supports HTML files containing TABLE elements. Does not support other table like grid elements.
* Does not support identification of titles, headings, column headers, etc. by using style information for a cell.
* Vertical spanning cells are parsed with first row where the cell is encountered. Subsequent rows will not contain the cell and have one less cell. Currently, vertical spanning cells must be at the end of the row otherwise the ordinal position of cells in the following rows will be incorrect, i.e. missing values are not supported.