# _Streams_ Logger Streams is an intuitive and performant logger for Node.js and TypeScript applications. ## Introduction _Streams_ is an intuitive logger built on native Node.js streams. You can use the built-in logging components (e.g., the [Logger](#the-logger-class), [Formatter](#the-formatter-class), [Filter](#the-filter-class), [ConsoleHandler](#the-consolehandler-class), [RotatingFileHandler](#the-rotatingfilehandler-class), and [SocketHandler](#the-sockethandler-class)) for [common logging tasks](#usage) or implement your own logging [Node](#how-to-implement-a-custom-streams-data-transformation-node) to handle a wide range of logging scenarios. _Streams_ offers a graph-like API pattern for building sophisticated logging pipelines. ### Features - **A _zero-dependency_ logging framework built on native Node.js streams.** - A library of commonly used logging components: [Logger](#the-logger-class), [Formatter](#the-formatter-class), [Filter](#the-filter-class), [ConsoleHandler](#the-consolehandler-class), [RotatingFileHandler](#the-rotatingfilehandler-class), and [SocketHandler](#the-sockethandler-class). - A rich selection of [contextual data](#log-context-properties) (e.g., module name, function name, line number, etc.) for augmenting log messages. - Log your message in any format you choose including [JSON](#log-a-json-object) and [colorized](#log-a-colorized-hello-world-message) output formats. - A type-safe graph-like API pattern for constructing sophisticated [logging graphs](#graph-api-pattern). - Consume any native Node.js Readable, Writable, Duplex, or Transform stream and add it to your graph. - Error handling and selective detachment of inoperable graph components. - Use _Streams_ in your Node.js project, [without type safety](#use-streams-in-a-nodejs-project-without-type-safety), or take advantage of the TypeScript type definitions. ## Table of contents - [Installation](#installation) - [Concepts](#concepts) - [Usage](#usage) - [Examples](#examples) - [Formatting](#formatting) - [API](#api) - [Using a Socket Handler](#using-a-socket-handler) - [Hierarchical logging](#hierarchical-logging) - [How-Tos](#how-tos) - [Tuning](#tuning) - [Backpressure](#backpressure) - [Performance](#performance) - [Versioning](#versioning) - [Test](#test) - [Support](#support) ## Installation ```bash npm install streams-logger ``` ## Concepts Logging is essentially a data transformation task. When a string is logged to the console, for example, it typically undergoes a transformation step where relevant information (e.g., the timestamp, log level, process id, etc.) is added to the log message prior to it being printed. Likewise, when data is written to a file or the console, additional data transformations may take place e.g., serialization and representational transformation. _Streams_ facilitates these data transformation tasks by means of a network of [`Node`](#node) instances that is constructed using a [graph-like API pattern](#graph-api-pattern). ### Node Each data transformation step in a _Streams_ logging graph is realized through a `Node` implementation. Each `Node` manages and represents a native Node.js stream. A `Node` in a data transformation graph consumes an input, transforms or filters the data in some way, and optionally produces an output. Each component (e.g., Loggers, Formatters, Filters, Handlers, etc.) in a _Streams_ logging graph _is a_ `Node`. Each `Node` _has a_ native Node.js stream that it manages. ### Graph API pattern _Streams_ uses a graph-like API pattern for constructing a logging graph. Each graph consists of a network of `Node` instances that together comprise a graph logging pipeline. Please see the [Usage](#usage) or [Examples](#examples) for instructions on how to construct a _Streams_ data transformation graph. ## Usage In this hypothetical example you will log "Hello, World!" to the console and to a file. ### Instructions Import the Logger, Formatter, ConsoleHandler and RotatingFileHandler, and SyslogLevel enum. ```ts import { Logger, Formatter, ConsoleHandler, RotatingFileHandler, SyslogLevel } from "streams-logger"; ``` Create an instance of a Logger, Formatter, ConsoleHandler and RotatingFileHandler. - The `Logger` is set to log at level `SyslogLevel.DEBUG`. - The `Formatter` constructor is passed a `format` function that will serialize data contained in the `LogContext` to a string containing the ISO time, the log level, the function name, the line number, the column number, and the log message. - The `ConsoleHandler` will log the message to `process.stdout`. - The `RotatingFileHandler` will log the message to the file `./message.log`. ```ts const logger = new Logger({ level: SyslogLevel.DEBUG }); const formatter = new Formatter({ format: ({ isotime, message, name, level, func, line, col }) => { return `${isotime}:${level}:${func}:${line}:${col}:${message}\n`; }, }); const consoleHandler = new ConsoleHandler({ level: SyslogLevel.DEBUG }); const rotatingFileHandler = new RotatingFileHandler({ path: "./message.log", level: SyslogLevel.DEBUG, }); ``` Connect the Logger to the Formatter and connect the Formatter to the ConsoleHandler and RotatingFileHandler. _Streams_ uses a graph-like API pattern in order to construct a network of `Node`s for your logging task. Each component in a network, in this case the `Logger`, the `Formatter`, and the `ConsoleHandler` and `RotatingFileHandler`, _is a_ Node. ```ts const log = logger.connect(formatter.connect(consoleHandler, rotatingFileHandler)); ``` Log "Hello, World!" to the console and to the file `./message.log`. ```ts function sayHello() { log.info("Hello, World!"); } sayHello(); ``` Output ``` ⮶date-time function name⮷ column⮷ ⮶message 2024-06-12T00:10:15.894Z:INFO:sayHello:7:9:Hello, World! ⮴level ⮴line number ``` - If you prefer a colorized log message, please see the [_Log a Colorized "Hello, World!" Message_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/log_a_colorized_hello_world_message) example that demonstrates how to log a colorized message to the console using the external library [Chalk](https://www.npmjs.com/package/chalk). - If you prefer a JSON format, please see the [_Log a JSON Object_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/hello_world_json) example that demonstrates how to log a JSON object to the console. ## Examples ### _"Hello, World!"_ - TypeScript Please see the [Usage](#usage) section above or the [_Hello, World!_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/hello_world) example for a working implementation. ### _Use **Streams** in a Node.js project (without type safety)_ - Node.js Please see the [_Use **Streams** in a Node.js Project_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/use_streams_in_a_nodejs_project) example that demonstrates how to use _Streams_ in a Node.js project _without_ type checks. ### _Log a colorized "Hello, World!" message_ - TypeScript Please see the [_Log a Colorized "Hello, World!" Message_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/log_a_colorized_hello_world_message) example that demonstrates how to log to the console using the external library [Chalk](https://www.npmjs.com/package/chalk). ### _Log a JSON object_ - TypeScript Please see the [_Log a JSON Object_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/log_a_json_object) example that demonstrates how to log a JSON object to the console. ### _Log a JavaScript object_ - TypeScript Please see the [_Log a JavaScript Object_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/log_a_javascript_object) example that demonstrates how to log an object to the console. ### _Log to a file and the console_ - TypeScript Please see the [_Log to a File and the Console_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/log_to_a_file_and_the_console) example that demonstrates how to log to a file and the console using different `Formatters`. ### _A network connected streams logging graph_ - TypeScript Please see the [_A Network Connected **Streams** Logging Graph_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/a_network_connected_logging_graph) example that demonstrates how to connect _Streams_ logging graphs over the network. ### _Implement a transform_ - TypeScript Please see the [_Implement a Transform_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/implement_a_transform) example that demonstrates how to implement a custom _Streams_ `Node`. ### _Consume a socket_ - TypeScript Please see the [_Consume a Socket_](https://github.com/adamjpatterson/streams-logger/tree/main/examples/consume_a_socket) example that demonstrates how to wrap a `net.Socket` in a _Streams_ `Node`. ## Formatting You can format your log message using a `Formatter` Node. The `Logger` constructs a `LogContext` object on each logged message. The [properties](#log-context-properties) of each `LogContext` contain information about the context of the logged message (e.g., module name, function name, line number, etc.). You can define a serialization function and pass it to the constructor of a `Formatter`. The serialization function can construct a log message from the `LogContext` [properties](#log-context-properties). In the concise [example](#example-formatter) below this is accomplished by using a [template literal](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals). ### Log context properties _Streams_ provides a rich selection of contextual information with each logging call. This information is provided in a `LogContext` object that is passed as a single argument to the function assigned to the `format` property of the `FormatterOptions` object that is passed to the `Formatter` constructor. Please see the [example](#example-formatter) for instructions on how to incorporate contextual information into your logged message. | Property | Description | Config Prerequisite | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | | `col` | The column number of the logging call. | `captureStackTrace=true` | | `func` | The name of the function where the logging call took place. | `captureStackTrace=true` | | `hostname` | The [hostname](https://nodejs.org/api/os.html#oshostname). | | | `isotime` | The ISO 8601 [representation](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toISOString) of the time at which the logging call took place. | `captureISOTime=true` | | `label` | Optional user specified label. | | | `level` | The `SyslogLevel` of the logging call. | | | `line` | The line number of the logging call. | `captureStackTrace=true` | | `location` | The location of the logging call (i.e., path). | `captureStackTrace=true` | | `message` | The message of the logging call. | | | `metadata` | Optional user specified data. | | | `name` | The name of the logger. | | | `pid` | The process identifier. | | | `stack` | The complete stack trace. | `captureStackTrace=true` | | `threadid` | The thread identifier. | | > **NB** For high throughput logging applications, you can improve performance by preventing some contextual information from being generated; you can set `Config.captureStackTrace` and `Config.captureISOTime` to `false`. Please see [Tuning](#tuning) for instructions on how to disable contextual information. ### Example formatter In the following code excerpt, a formatter is implemented that serializes a `LogContext` to: 1. The time of the logging call in ISO format 2. The log level 3. The name of the function where the log event originated 4. The line number of the log event 5. The column number of the log event 6. The log message 7. A newline The `format` function is passed in a `FormatterOptions` object to the constructor of a `Formatter`. The `Logger` is connected to the `Formatter`. The `Formatter` is connected to the `ConsoleHandler`. ```ts import { Logger, Formatter, ConsoleHandler, SyslogLevel } from "streams-logger"; const logger = new Logger({ name: "main", level: SyslogLevel.DEBUG }); const formatter = new Formatter({ format: ({ isotime, message, name, level, func, line, col }) => { return `${isotime}:${level}:${func}:${line}:${col}:${message}\n`; }, }); const consoleHandler = new ConsoleHandler({ level: SyslogLevel.DEBUG }); const log = logger.connect(formatter.connect(consoleHandler)); function sayHello() { log.info("Hello, World!"); } sayHello(); ``` This is an example of what a logged message will look like using the `Formatter` defined above. ``` ⮶date-time function name⮷ column⮷ ⮶message 2024-06-12T00:10:15.894Z:INFO:sayHello:7:9:Hello, World! ⮴level ⮴line number ``` ## API The _Streams_ API provides commonly used logging facilities (i.e., the [Logger](#the-logger-class), [Formatter](#the-formatter-class), [Filter](#the-filter-class), [ConsoleHandler](#the-consolehandler-class), [RotatingFileHandler](#the-rotatingfilehandler-class), and [SocketHandler](#the-sockethandler-class)). However, you can [consume any Node.js stream](#how-to-consume-a-readable-writable-duplex-or-transform-nodejs-stream) and add it to your logging graph. ### The Logger class #### new streams-logger.Logger\(options, streamOptions) - `` The type of the logged message. **Default: ``** - options `` Optional options to be passed to the `Logger`. - level `` The syslog logger level. **Default: `SyslogLevel.WARN`** - name `` An optional name for the `Logger`. - parent `` An optional parent `Logger`. - queueSizeLimit `` Optionally specify a limit on the number of log messages that may queue while waiting for a stream to drain. See [Backpressure](#backpressure). - captureStackTrace `` Optionally specify if stack trace capturing is enabled. This setting will override the default. **Default: `Config.captureStackTrace`** - captureISOTime `` Optionally specify if capturing ISO time is enabled. This setting will override the default. **Default: `Config.captureISOTime`** - streamOptions `` Optional options to be passed to the stream. You can use `TransformOptions` to set a `highWaterMark` on the `Logger`. Use an instance of a Logger to propagate messages at the specified syslog level. _public_ **logger.level** - `` The configured log level (e.g., `SyslogLevel.DEBUG`). _public_ **logger.connect(...nodes)** - nodes `, unknown>>` Connect to an Array of `Nodes`. Returns: `, LogContext>` _public_ **logger.disconnect(...nodes)** - nodes `, unknown>>` Disconnect from an Array of `Nodes`. Returns: `, LogContext>` _public_ **logger.debug(message, label)** - message `` Write a DEBUG message to the `Logger`. - label: `` An optional label. Returns: `` _public_ **logger.info(message, label)** - message `` Write a INFO message to the `Logger`. - label: `` An optional label. Returns: `` _public_ **logger.notice(message, label)** - message `` Write a NOTICE message to the `Logger`. - label: `` An optional label. Returns: `` _public_ **logger.warn(message, label)** - message `` Write a WARN message to the `Logger`. - label: `` An optional label. Returns: `` _public_ **logger.error(message, label)** - message `` Write a ERROR message to the `Logger`. - label: `` An optional label. Returns: `` _public_ **logger.crit(message, label)** - message `` Write a CRIT message to the `Logger`. - label: `` An optional label. Returns: `` _public_ **logger.alert(message, label)** - message `` Write a ALERT message to the `Logger`. - label: `` An optional label. Returns: `` _public_ **logger.emerg(message, label)** - message `` Write a EMERG message to the `Logger`. - label: `` An optional label. Returns: `` _public_ **logger.setLevel(level)** - level `` A log level. Returns `` Set the log level. Must be one of `SyslogLevel`. ### The Formatter class #### new streams-logger.Formatter\(options, streamOptions) - `` The type of the logged message. This is the type of the `message` property of the `LogContext` that is passed to the `format` function. **Default: ``** - `` The type of the output message. This is the return type of the `format` function. **Default: ``** - options - format `(record: LogContext): Promise | MessageOutT` A function that will format and serialize the `LogContext`. Please see [Formatting](#formatting) for how to implement a format function. - streamOptions `` Optional options to be passed to the stream. You can use `TransformOptions` to set a `highWaterMark` on the `Formatter`. Use a `Formatter` in order to specify how your log message will be formatted prior to forwarding it to the Handler(s). A [`LogContext`](#the-logcontext-interface) object is created that contains information about the environment at the time of the logging call. The `LogContext` is passed as the single argument to `format` function. _public_ **formatter.connect(...nodes)** - nodes `, unknown>>` Connect to an Array of `Nodes`. Returns: `, LogContext>` _public_ **formatter.disconnect(...nodes)** - nodes `, unknown>>` Disconnect from an Array of `Nodes`. Returns: `, LogContext>` ### The Filter class #### new streams-logger.Filter\(options, streamOptions) - `` The type of the logged message. **Default: ``** - options - filter `(record: LogContext): Promise | boolean` A function that will filter the `LogContext`. Return `true` in order to permit the message through; otherwise, return `false`. - streamOptions `` Optional options to be passed to the stream. You can use `TransformOptions` to set a `highWaterMark` on the `Filter`. _public_ **filter.connect(...nodes)** - nodes `, unknown>>` Connect to an Array of `Nodes`. Returns: `, LogContext>` _public_ **filter.disconnect(...nodes)** - nodes `, unknown>>` Disconnect from an Array of `Nodes`. Returns: `, LogContext>` ### The ConsoleHandler class #### new streams-logger.ConsoleHandler\(options, streamOptions) - `` The type of the logged message. **Default: ``** - options `` Optional options to be passed to the `ConsoleHandler`. - level `` An optional log level. **Default: `SyslogLevel.WARN`** - streamOptions `` Optional options to be passed to the stream. You can use `WritableOptions` to set a `highWaterMark` on the `ConsoleHandler`. Use a `ConsoleHandler` in order to stream your messages to the console. _public_ **consoleHandler.setLevel(level)** - level `` A log level. Returns `` Set the log level. Must be one of `SyslogLevel`. ### The RotatingFileHandler class #### new streams-logger.RotatingFileHandler\(options, streamOptions) - `` The type of the logged message. **Default: ``** - options `` - path `` The path of the log file. - rotationLimit `` An optional number of log rotations. **Default: `0`** - maxSize `` The size of the log file in bytes that will initiate a rotation. **Default: `1e6`** - encoding `` An optional encoding. **Default: `utf-8`** - mode `` An optional mode. **Default: `0o666`** - level `` An optional log level. **Default: `SyslogLevel.WARN`** - flags `` An optional file system flag. **Default: `a`** - streamOptions `` Optional options to be passed to the stream. You can use `TransformOptions` to set a `highWaterMark` on the `RotatingFileHandler`. Use a `RotatingFileHandler` in order to write your log messages to a file. > **NB** For improved performance, the `RotatingFileHandler` maintains its own accounting of the log file size for purposes of file rotation; hence, it's important that out-of-band writes are not permitted on the same log file while it is operating on it. _public_ **rotatingFileHandler.setLevel(level)** - level `` A log level. Returns `` Set the log level. Must be one of `SyslogLevel`. ### The SocketHandler class #### new streams-logger.SocketHandler\(options, streamOptions) - `` The type of the logged message. **Default: ``** - options `` - socket `` A `net.Socket` that will serve as a communication channel between this `SocketHandler` and the remote `SocketHandler`. - reviver `<(this: unknown, key: string, value: unknown) => unknown>` An optional reviver for `JSON.parse`. - replacer `<(this: unknown, key: string, value: unknown) => unknown>` An optional replacer for `JSON.stringify`. - space `` An optional space specification for `JSON.stringify`. - level `` An optional log level. **Default: `SyslogLevel.WARN`** - payloadSizeLimit `` An optional limit on the size of a serialized log message in bytes. **Default: `1e6`** - ingressQueueThreshold `` An optional threshold for the `ingressQueue` in bytes; the socket will be paused if this threshold is exceeded. - streamOptions `` Optional options to be passed to the stream. You can use `DuplexOptions` to set a `highWaterMark` on the `SocketHandler`. Use a `SocketHandler` in order to connect _Streams_ graphs over the network. Please see the [_A Network Connected **Streams** Logging Graph_](#a-network-connected-streams-logging-graph) example for instructions on how to use a `SocketHandler` in order to connect _Streams_ logging graphs over the network. _public_ **socketHandler.connect(...nodes)** - nodes `, unknown>>` Connect to an Array of `Nodes`. Returns: `, LogContext>` _public_ **socketHandler.disconnect(...nodes)** - nodes `, unknown>>` Disconnect from an Array of `Nodes`. Returns: `, LogContext>` _public_ **socketHandler.setLevel(level)** - level `` A log level. Returns `` Set the log level. Must be one of `SyslogLevel`. ### The LogContext interface #### streams-logger.LogContext\ - `` The type of the logged message. **Default: ``** - `` The type of the Level enum. **Default: ``** A `LogContext` is a plain object created each time a message is logged at (or below) the level set on the `Logger`. It contains information about the process and environment at the time of the logging call. All _Streams_ `Node`s take a `LogContext` as an input and emit a `LogContext` as an output. The `LogContext` is passed as the single argument to the [format function](#formatting) of the `Formatter`; information about the environment can be extracted from the `LogContext` in order to format the logged message. The following properties will be available to the `format` function depending on the setting of `Config.captureStackTrace` and `Config.captureISOTime`. Please see the [Log context properties](#log-context-properties) table for details. _public_ **logContext.col** - `` The column of the logging call. Available if `Config.captureStackTrace` is set to `true`. _public_ **logContext.func** - `` The name of the function where the logging call took place. Available if `Config.captureStackTrace` is set to `true`. _public_ **logContext.hostname** - `` The hostname. _public_ **logContext.isotime** - `` The date and time in ISO format at the time of the logging call. Available if `Config.captureISOTime` is set to `true`. _public_ **logContext.label** - `` Optional user specified label. _public_ **logContext.level** - `` An uppercase string representation of the level. _public_ **logContext.line** - `` The line number of the logging call. Available if `Config.captureStackTrace` is set to `true`. _public_ **logContext.location** - `` The location of the logging call. Available if `Config.captureStackTrace` is set to `true`. _public_ **logContext.message** - `` The logged message. _public_ **logContext.metadata** - `` Optional user specified data. _public_ **logContext.name** - `` The name of the `Logger`. _public_ **logContext.pid** - `` The process identifier. _public_ **logContext.threadid** - `` The thread identifier. _public_ **logContext.stack** - `` The complete stack trace. Available if `Config.captureStackTrace` is set to `true`. ### The _Streams_ Config settings object The `Config` object is used to set default settings. It can be used for performance [tuning](#tuning). **Config.errorHandler** `<(err: Error, ...params: Array) => void>` Set an error handler. **Default: `console.error`** **Config.captureISOTime** `` Set this to `false` in order to disable capturing ISO time on each logging call. **Default: `true`** **Config.captureStackTrace** `` Set this to `false` in order to disable stack trace capture on each logging call. **Default: `true`** **Config.highWaterMark** `` Set the `highWaterMark` for streams in Buffer mode. **Default: `node:stream.getDefaultHighWaterMark(false)`** **Config.highWaterMarkObjectMode** `` Set the `highWaterMark` for streams in objectMode. **Default: `node:stream.getDefaultHighWaterMark(true)`** **Config.getDuplexOptions(writableObjectMode, readableObjectMode)** - writableObjectMode `` `true` for ObjectMode; `false` otherwise. - readableObjectMode `` `true` for ObjectMode; `false` otherwise. Returns: `` Use `Config.getDuplexOptions` when implementing a [custom _Streams_ data transformation Node](#how-to-implement-a-custom-streams-data-transformation-node). **Config.getReadableOptions(readableObjectMode)** - readableObjectMode `` `true` for ObjectMode; `false` otherwise. Returns: `` Use `Config.getReadableOptions` when implementing a [custom _Streams_ data transformation Node](#how-to-implement-a-custom-streams-data-transformation-node). **Config.getWritableOptions(writableObjectMode)** - writableObjectMode `` `true` for ObjectMode; `false` otherwise. Returns: `` Use `Config.getWritableOptions` when implementing a [custom _Streams_ data transformation Node](#how-to-implement-a-custom-streams-data-transformation-node). ### The SyslogLevel enum #### streams-logger.SyslogLevel\[Level\] - Level - EMERG = 0 - ALERT = 1 - CRIT = 2 - ERROR = 3 - WARN = 4 - NOTICE = 5 - INFO = 6 - DEBUG = 7 Use `SyslogLevel` to set the level in the options passed to `Logger`, `Filter`, and Handler constructors. ## Using a Socket Handler _Streams_ uses Node.js streams for message propagation. Node.js represents sockets as streams; hence, sockets are a natural extension of a _Streams_ logging graph. For example, you may choose to use a `ConsoleHandler` locally and log to a `RotatingFileHandler` on a remote server. Please see the [_A Network Connected **Streams** Logging Graph_](#a-network-connected-streams-logging-graph) example for a working implementation. ### Payload size limit _Streams_ enforces a size limit on the serialized message payload. You can specify a restrictive `payloadSizeLimit` in the `SocketHandlerOptions` passed to the constructor of the `SocketHandler`. The default is `1e6` bytes. ### Security The `SocketHandler` options take a socket instance as an argument. The `net.Server` that produces this socket may be configured however you choose. You can encrypt the data sent over TCP connections and authenticate clients by configuring your `net.Server` accordingly. #### Configure your server to use TLS encryption. TLS Encryption may be implemented using native Node.js [TLS Encryption](https://nodejs.org/docs/latest-v20.x/api/tls.html). #### Configure your client to use TLS client certificate authentication. TLS Client Certificate Authentication may be implemented using native Node.js [TLS Client Authentication](https://nodejs.org/docs/latest-v20.x/api/tls.html). ## Hierarchical logging The _Streams_ graph API naturally supports hierarchical logging. You may optionally specify an antecedent `Node` by assigning an instance of `Logger` to the `parent` property in the `LoggerOptions`. The antecedent `Node` will be attached to the `Logger` during its construction. Alternatively, _Streams_ also provides a single instance root `Logger` (`streams-logger.root`) that you may optionally integrate into your logging graphs. ## How-Tos ### How to implement a custom _Streams_ data transformation node _Streams_ is built on a type-safe graph API framework. Each component of a _Streams_ logging graph _is a_ `Node` instance. This means that any `Node` may be incorporated into your logging graph provided that it meets the contextual type requirements. In order to implement a _Streams_ data transformation `Node`, subclass the `Node` class, and provide the appropriate _Streams_ defaults to the stream constructor. For example, the somewhat contrived `LogContextToBuffer` implementation transforms the `message` contained in a `LogContext` to a `Buffer`; the graph pipeline streams the message to `process.stdout`. > **NB** In this example, `writableObjectMode` is set to `true` and `readableObjectMode` is set to `false`; hence, the Node.js stream implementation will handle the input as an `object` and the output as a `Buffer`. It's important that `writableObjectMode` and `readableObjectMode` accurately reflect the input and output types of your Node. #### Implement the `index.ts` module. ```ts import * as stream from "node:stream"; import { Logger, Node, Config, LogContext, SyslogLevelT } from "streams-logger"; export class LogContextToBuffer extends Node, Buffer> { public encoding: NodeJS.BufferEncoding = "utf-8"; constructor(streamOptions?: stream.TransformOptions) { super( new stream.Transform({ ...Config.getDuplexOptions(true, false), ...streamOptions, ...{ writableObjectMode: true, readableObjectMode: false, transform: ( chunk: LogContext, encoding: BufferEncoding, callback: stream.TransformCallback ) => { callback(null, Buffer.from(chunk.message, this.encoding)); }, }, }) ); } } const log = new Logger({ name: "main" }); const logContextToBuffer = new LogContextToBuffer(); const console = new Node(process.stdout); log.connect(logContextToBuffer.connect(console)); log.warn("Hello, World!"); ``` ##### Output ```bash Hello, World! ``` _Streams_ provides a few examples of [handlers](https://github.com/adamjpatterson/streams-logger/tree/main/src/handlers), which you can use for modeling your data transformation `Node`. ### How to consume a Readable, Writable, Duplex, or Transform Node.js stream You can incorporate any Readable, Writable, Duplex, or Transform stream into your logging graph, provided that it meets the contextual type requirements, by passing the stream to the `Node` constructor. In this hypothetical example a type-safe `Node` is constructed from a `net.Socket`. The type variables are specified as ``; the writable side of the stream consumes a `Buffer` and the readable side of the stream produces a `Buffer`. ```ts import * as net from "node:net"; import { once } from "node:events"; import { Node } from "streams-logger"; net.createServer((socket: net.Socket) => socket.pipe(socket)).listen(3000); const socket = net.createConnection({ port: 3000 }); await once(socket, "connect"); const socketHandler = new Node(socket); // Connect this `Node` to any `Node` that produces a Buffer. ``` ## Tuning **Depending on your requirements, the defaults may be fine.** However, for high throughput logging applications you may choose to adjust the `highWaterMark` and/or disable stack trace capturing. ### Tune the highWaterMark _Streams_ `Node` implementations use the native Node.js stream API for message propagation. You have the option of tuning the Node.js stream `highWaterMark` to your specific needs - keeping in mind memory constraints. You can set a `highWaterMark` using [`Config.highWaterMark`](#the-streams-config-settings-object) and [`Config.highWaterMarkObjectMode`](#the-streams-config-settings-object) that will apply to `Node`s in the _Streams_ library. Alternatively, the `highWaterMark` can be set in the constructor of each `Node`; please see the [API](#api) for instructions on how to do this. In this example, the `highWaterMark` of ObjectMode streams and Buffer mode streams is artificially set to `1e6` objects and `1e6` bytes. ```ts import * as streams from "streams-logger"; streams.Config.highWaterMark = 1e6; streams.Config.highWaterMarkObjectMode = 1e6; ``` > Please see the [API](#api) for more information on [`Config`](#the-streams-config-settings-object) object settings. ### Disable stack trace capture Another optional setting that you can take advantage of is to turn off stack trace capture. Stack trace capture can be disabled globally using the _Streams_ configuration settings object i.e., [`Config.captureStackTrace`](#the-streams-config-settings-object). Alternatively, you may disable stack trace capturing in a specific `Logger` by setting the `captureStackTrace` property of the `LoggerOptions` to `false`. Turning off stack trace capture will disable some of the information (e.g., function name and line number) that is normally contained in the `LogContext` object that is passed to the `format` function of a `Formatter`. You can turn off stack trace capturing for all `Logger` instances. ```ts import * as streams from "streams-logger"; streams.Config.captureStackTrace = false; ``` Alternatively, you can instantiate a `Logger` with stack trace capturing disabled. ```ts const logger = new Logger({ captureStackTrace: false }); ``` ### Putting it all together If you have a high throughput logging application, the following settings should get you to where you want to be while keeping Node.js stream buffers in check. ```ts import * as streams from 'streams-logger'; streams.Config.highWaterMark = 1e5; streams.Config.highWaterMarkObjectMode = 1e5; const logger = new Logger({ captureStackTrace: false }); ... // Create an instance of a `Formatter` and `ConsoleHandler`. const log = logger.connect( formatter.connect( consoleHandler ) ); ``` However, for typical error logging applications or debugging scenarios the defaults should work fine. ## Backpressure _Streams_ respects backpressure by queueing messages while the stream is draining. You can set a limit on how large the message queue may grow by specifying a `queueSizeLimit` in the Logger constructor options. If a `queueSizeLimit` is specified and if it is exceeded, the `Logger` will throw a `QueueSizeLimitExceededError`. **For typical logging applications setting a `queueSizeLimit` isn't necessary.** However, if an uncooperative stream peer reads data at a rate that is slower than the rate that data is written to the stream, data may buffer until memory is exhausted. By setting a `queueSizeLimit` you can effectively respond to subversive stream peers and disconnect offending `Node`s in your graph. If you have a _cooperating_ stream that is backpressuring, you can either set a default `highWaterMark` appropriate to your application or increase the `highWaterMark` on the specific stream in order to mitigate drain events. ## Performance _Streams_ is a highly customizable logger that performs well on a wide range of logging tasks. It is a good choice for both error logging and high throughput logging. It strictly adheres to the Node.js public API contract and common conventions. This approach comes with trade-offs; however, it ensures stability and portability while still delivering a performant logging experience. > Please see [Tuning](#tuning) for how to configure the logging graph for high throughput logging applications. ## Versioning The _Streams_ package adheres to semantic versioning. Breaking changes to the public API will result in a turn of the major. Minor and patch changes will always be backward compatible. Excerpted from [Semantic Versioning 2.0.0](https://semver.org/): > Given a version number MAJOR.MINOR.PATCH, increment the: > > 1. MAJOR version when you make incompatible API changes > 2. MINOR version when you add functionality in a backward compatible manner > 3. PATCH version when you make backward compatible bug fixes > > Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format. ## Test ### How to run the test #### Clone the repository. ```bash git clone https://github.com/adamjpatterson/streams-logger.git ``` #### Change directory into the root of the repository. ```bash cd streams-logger ``` #### Install dependencies. ```bash npm install ``` #### Run the tests. ```bash npm test verbose=false ``` ## Support For feature requests or issues, please open an [issue](https://github.com/adamjpatterson/streams-logger/issues) or contact one of the authors. - [Adam Patterson](https://github.com/adamjpatterson)