[![npm](https://img.shields.io/npm/v/scramjet-core)](https://www.npmjs.com/scramjet-core)
[![last commit](https://img.shields.io/github/last-commit/scramjetorg/scramjet-core)](https://github.com/scramjetorg/scramjet-core)
[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fscramjetorg%2Fscramjet-core.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fscramjetorg%2Fscramjet-core?ref=badge_shield)
[![Known Vulnerabilities](https://snyk.io/test/github/scramjetorg/scramjet-core/badge.svg)](https://snyk.io/test/github/scramjetorg/scramjet-core)
[![Discord](https://img.shields.io/discord/925384545342201896?label=discord)](https://discord.gg/52USU8q7AX)

Scramjet core
===============

This is the minimal, dependency free version of [`scramjet`](https://github.com/scramjetorg/scramjet) used as of Scramjet
version 3.0.0 as a base for `scramjet` and scramjet plugins.

Unless you are sure, you should be better off with using the main repo and module.

It is built upon the logic behind three well known javascript array operations - namingly map, filter and reduce. This
means that if you've ever performed operations on an Array in JavaScript - you already know Scramjet like the back of
your hand.

## Usage

Scramjet uses functional programming to run transformations on your data streams in a fashion very similar to the well
known event-stream node module. Most transformations are done by passing a transform function.

It's all about chaining, really - you develop your flow based on a chain of calls that return another method like this:

```javascript
    scramjet.from(someReadableStream)           // you can construct your stream any way you like
        .map(someMapper)                        // you can map the objects in the stream
        .map(someAsyncAPICall)                  // you can call an API for each item
        .filter(asynchronousFilterOperation)    // you can even filter by async function
        .catch(errorHandler)                    // there's built in error handling
        .until(doneCondition)                   // you can stop reading the stream whenever you're done
        .toArray();                             // you can accumulate
```

You can write your transforms in three ways:

1. Synchronous

 Example: a simple stream transform that outputs a stream of objects of the same id property and the length of the value string.

 ```javascript
    datastream.map(
        (item) => ({id: item.id, length: item.value.length})
    )
 ```

1. Asynchronous using ES2015 async await

Example: A simple stream that uses Fetch API to get all the contents of all entries in the stream

```javascript
    datastream.map(
        async (item) => fetch(item)
    )
```

1. Asynchronous using Promises

 Example: A simple stream that fetches an url mentioned in the incoming object

 ```javascript
    datastream.map(
        (item) => new Promise((resolve, reject) => {
            request(item.url, (err, res, data) => {
                if (err)
                    reject(err); // will emit an "error" event on the stream
                else
                    resolve(data);
            });
        })
    )
 ```

The actual logic of this transform function is as if you passed your function to the ```then``` method of a Promise
resolved with the data from the input stream.

## API Docs

Here's the list of the exposed classes and methods, please review the specific documentation for details:

* [`exports`](docs/index.md) - module exports explained
* [`scramjet.DataStream`](docs/data-stream.md) - the base class for all scramjet classes.
* [`scramjet.BufferStream`](docs/buffer-stream.md) - a DataStream of Buffers.
* [`scramjet.StringStream`](docs/string-stream.md) - a DataStream of Strings.
* [`scramjet.MultiStream`](docs/multi-stream.md) - a DataStream of Strings.
* [more on plugins](docs/plugins.md) - a description and link.

Note that:

* Most of the methods take a callback argument that operates on the stream items.
* The callback, unless it's stated otherwise, will receive an argument with the next chunk.
* You can use `async` functions or return `Promise`s wherever you like.
* Methods usually return the same class, so are chainable `↺` or are asynchronous `⇄`

The quick reference of the exposed classes:

{{#classes ~}}{{>stream-class}}{{/classes}}

## CLI

Check out the command line interface for simplified scramjet usage with [scramjet-cli](https://www.npmjs.com/package/scramjet-cli)

    $ sjr -i http://datasource.org/file.csv ./transform-module-1 ./transform-module-1 | gzip > logs.gz

## License and contributions

As of version 2.0 Scramjet is MIT Licensed.

[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fscramjetorg%2Fscramjet-core.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fscramjetorg%2Fscramjet-core?ref=badge_large)

## Help wanted

The project need's your help! There's lots of work to do - transforming and muxing, joining and splitting, browserifying, modularizing, documenting and issuing those issues.

If you want to help and be part of the Scramjet team, please reach out to me, scramjetorg on Github or email us: opensource@scramjet.org.