# by-request ## Simple Node HTTP/HTTPS client for use with promises and async/await The **by-request** package provides four ways to retrieve data via HTTP/HTTPS, all of which return `Promise`s, and all of which are suitable for use with `async`/`await`: * `requestBinary(`...`): Promise` * `requestFile(`...`): Promise` (_can also output to `Writable` streams instead of files_) * `requestJson(`...`): Promise` * `requestText(`...`): Promise` GET and POST requests are handled, and optional automatic file caching can be used. [![npm](https://img.shields.io/npm/v/by-request.svg)](https://www.npmjs.com/package/by-request/) [![Coverage Status](https://coveralls.io/repos/github/kshetline/by-request/badge.svg?branch=master)](https://coveralls.io/github/kshetline/by-request) [![npm downloads](https://img.shields.io/npm/dm/by-request.svg)](https://npmjs.org/package/by-request/) ![npm bundle size (scoped)](https://img.shields.io/bundlephobia/min/by-request) ![license](https://img.shields.io/badge/licence-mit-informational) ### Installation `npm install by-request` ### Examples ```typescript const audio = await requestBinary('https://musicalstuff.net/some_song.mp3'); ``` ```typescript const forecast = await requestText('https://forekast.info/forecast/local/', { cacheDir: 'weather-cache/', maxCacheAge: 900000, // 15 minutes params: { city: 'poughkeepsie', state: 'ny', country: 'us' } }); ``` ```typescript const options = {progress: (bytesRead, totalBytes) => { if (totalBytes) console.log((bytesRead / totalBytes * 100).toFixed(1) + '% downloaded'); else console.log((bytesRead / 1024).toFixed(1) + 'K downloaded'); }}; requestFile('https://tedious.org/2024/07/meeting_minutes.docx', options, 'documents/tldnr/').then(length => { console.log('Document retrieved, length in bytes: ' + length); }).catch(err => { console.error('Something went wrong: ' + err.message); }); ``` ```typescript let whereAmI = await requestJson('http://ip-api.com/json/'); ``` ```typescript let currentTemperature = parseFloat((await requestText('https://howhotisit.biz/forecast/?zip=02134')) .replace(/.*Current temperature:\s*([-.0-9]+).*/is, '$1')); ``` HTTP(S) redirects are automatically handled using [follow-redirects](https://github.com/follow-redirects/follow-redirects), and a wide variety of character encodings are supported using [iconv-lite](https://github.com/ashtuchkin/iconv-lite). HTTP(S) responses which are compressed using the `gzip`, `deflate` or `br` methods are automatically decompressed. (When using `requestBinary()` or `requestFile()`, this automatic decompression can be disabled.) URLs can be specified either as strings or via an `ExtendedRequestOptions` object. When a URL is specified as a string, additional options can still be specified via a subsequent `ExtendedRequestOptions` parameter. `ExtendedRequestOptions` is an extension of the standard Node [`RequestOptions`](https://nodejs.org/api/http.html#http_http_request_options_callback) object. Those options have been extended as follows: ```typescript interface ExtendedRequestOptions extends RequestOptions { autoDecompress?: boolean; body?: Buffer | string; // For POST requests cachePath?: string; dontDecompress?: boolean; dontEndStream?: boolean; followRedirects?: boolean; // follow-redirects forceEncoding?: boolean; ignoreBom?: boolean; json?: any; // For POST requests keepBom?: boolean; maxBodyLength?: number; // follow-redirects maxCacheAge?: number; maxRedirects?: number; // follow-redirects params?: string | Record; // For POST requests progress?: (bytesRead: number, totalBytes: number | undefined) => void; responseInfo?: (info: ResponseInfo) => void; trackRedirects?: boolean; // follow-redirects } ``` For the options marked "follow-redirects", see the [follow-redirects documentation](https://github.com/follow-redirects/follow-redirects/blob/master/README.md). The other options are available as follows: * `autoDecompress`: If a response has a `Content-Type` of gzip, gzipped, or gunzip, automatically decompress the content rather than returning the compressed data. * `body`: If a `body` for a request is provided, the request will be made using the POST method, and the `body` will be sent as part of the request. The `Content-Type` header will automatically be `text/plain; charset=UTF-8` unless you provide an overriding header value. * `cachePath`: This is either a file path for where cached data should be stored under a specific file name for the current request, or a directory (indicated using a `'/'` at the end of the path) where cached data will be stored using auto-generated file names. * `dontDecompress`: For use with `requestBinary()` and `requestFile()`, set to `true` to prevent automatic decompression of `gzip`, `deflate`, or `br` data. * `dontEndStream`: For use with `requestFile()`, this prevents `stream.end()` from automatically being called when `requestFile()` terminates. This option applies to either a stream passed in as an argument (in lieu of a file path), or to the internally created file output stream. If you choose not to end the internally created stream automatically, you should also use the `responseInfo` callback so that you can access that stream to end it later. Errors will always end any internally created stream. * `forceEncoding`: For use with `requestText()`, setting this to `true` causes the `encoding` argument passed into the function to override any encoding specified by the retrieved data itself. * `ignoreBom`: By default `requestJson()` and `requestText()` look to see if a UTF-7, -8, -16, or -32 BOM [(Byte Order Mark)](https://en.wikipedia.org/wiki/Byte_order_mark) is present as one way of determining content character encoding. Setting `ignoreBom` to `true` allows the BOM to be ignored. * `json`: Causes a POST request to be made, either using a `string` value treated as a literal JSON body, or any other type of value which will be stringified to create the POSTed `application/json; charset=UTF-8` body. * `keepBom`: If a BOM is detected by `requestJson()` or `requestText()`, it is normally deleted. Set `keepBom` to `true` to preserve the BOM. * `maxCacheAge`: When using the `cacheDir` option, this parameter allows you to ignore cached data older than `maxCacheAge` milliseconds. * `params`: Causes a POST request to be made, either using a `string` value treated as a literal pre-URL-encoded string of name and values, or a name/value pair object which will be URL-encoded to create the POSTed `application/x-www-form-urlencoded; charset=UTF-8` body. * `progress`: As seen in the `requestFile()` example at the beginning of this document, this is an optional callback that provides feedback during the retrieval of large resources, returning the number of bytes read at a particular point in time, and, if known (otherwise `undefined`), the total number of bytes expected. * `responseInfo`: This optional callback provides meta-information about the resource which has been retrieved and the retrieval process. The data provided looks like this: ```typescript interface ResponseInfo { bomDetected: boolean; bomRemoved: boolean; cachePath?: string; callback?: string; charset: string; contentEncoding: string; contentLength: number; contentType: string; fromCache?: boolean; stream: Writable; } ``` * `bomDetected`: Whether a BOM was detected. * `bomRemoved`: Whether a BOM was removed. * `callback`: If JSONP data has been retrieved, this is the name of the callback function. * `cachePath`: If the `cachePath` option is used in your request, this will contain the name of the file where data has been cached. If the cache was used rather than making a fresh HTTP/HTTPS request, this value and `fromCache` will be the only feedback received. * `charset`: The character encoding ultimately used to interpret text or JSON data. This will be `'binary'` for binary and file operations. * `contentEncoding`: The value of the HTTP `Content-Encoding` header. * `contentLength`: The total number of bytes read. For compressed data, this is the compressed length, not the expanded length. * `contentType`: The value of the HTTP `Content-Type` header. * `fromCache`: `true` if content was retrieved from cache. * `stream`: When using `requestFile()`, this is either the stream passed into the function, or the stream created for the file path. Note: If you provide your own `Content-Type` header, only the `body` option is valid for POSTed content (`json` and `params` options will be ignored), and you are responsible for correctly encoding the provided `body`. All requests without using the `body`, `json`, or `params` options use the GET method, rather than POST. ### Functions #### `requestBinary(`...`)` ```typescript async function requestBinary(urlOrOptions: string | ExtendedRequestOptions, options?: ExtendedRequestOptions): Promise ``` Gets binary data as a `Buffer`. Apart from decompression (which can be optionally disabled), data is retrieved as sent, with no character encoding transformations performed. #### `requestFile(`...`)` (or `wget(`...`)`) ```typescript async function requestFile(urlOrOptions: string | ExtendedRequestOptions, optionsOrPathOrStream?: ExtendedRequestOptions | string | Writable, pathOrStream?: string | Writable): Promise ``` Retrieves an HTTP(S) resource to save as a file, or to send to a `Writable` output stream. Apart from decompression (which can be optionally disabled), data is retrieved as sent, with no character encoding transformations performed. The function returns the length of the file. If a path string is specified that does not end in a slash (`/`), that path is used as the complete file path for saving the retrieved data. If it does end in a slash, it is treated as a directory, and the complete file path is formed by extracting a file name from the end of the URL for the resource. If no path (or `Writable` stream) is specified, only a URL, the file name is extracted from the end of the URL, and the file is created in the current working directory. #### `requestJson(`...`)` ```typescript async function requestJson(urlOrOptions: string | ExtendedRequestOptions, options?: ExtendedRequestOptions): Promise ``` Gets JSON or JSONP data, returning it as whatever data type is applicable. If JSONP data is detected, the `responseInfo` callback can be used to retrieve the name of the JSONP callback function. Since JSON is now almost universally encoded using UTF-8, no encoding parameter is provided for this function, and the retrieved data will be decoded by something other than UTF-8 only if the HTTP `Content-Type` header, or a BOM marker, indicate otherwise. Should the odd case arise where you need to force the use of a different encoding, use `requestText()` instead, along with `JSON.parse()`. #### `requestText(`...`)` ```typescript async function requestText(urlOrOptions: string | ExtendedRequestOptions, encoding?: string): Promise async function requestText(url: string, options: ExtendedRequestOptions, encoding?: string): Promise async function requestText(urlOrOptions: string | ExtendedRequestOptions, optionsOrEncoding?: ExtendedRequestOptions | string, encoding?: string): Promise ``` Gets text data and returns it as a string. The character encoding scheme used to decode the text is applied according to the following order of priority: 1. The `encoding` parameter (or `optionsOrEncoding` parameter, as a string value) when the `forceEncoding` option is `true`. This will override any declared encoding based on a BOM, HTTP headers, or text content. 2. The encoding expressed via a BOM, if present, and if not ignored via the `ignoreBom` option. 3. The `charset` value, if provided, in the HTTP `Content-Type` header. 4. The UTF-16 or UTF-32 encoding implied by the pattern of zero and non-zero values in the first four bytes of the HTTP(S) content. 5. Any content-specified encoding that can be seen by looking within the first 2K of data, interpreting that data as if it were ASCII, and finding declarations such as these: `` `` `` `@charset "iso-8859-1"` 6. The `encoding` parameter (or `optionsOrEncoding` parameter, as a string value) when the `forceEncoding` option is `false` (or not specified). 7. UTF-8. ### Error handling When using these functions via the returned `Promise`, the `Promise` `catch` method should be used to handle errors. When using `async`/`await`, errors should be caught using `try`/`catch`. Possible errors include bad HTTP status codes, timeouts and other I/O errors, corrupted or invalid data, unsupported character encodings, and invalid function parameters or options.