# The Littlefork guide for developers

Littlefork is written in JavaScript and runs on NodeJS. Packages can be
obtained from [NPM](https://npmjs.org). Littlefork is split in several
packages:

- [`littlefork`](https://npmjs.org/package/littlefork)

  The core of Littlefork. It provides the machinery to run plugins and an API
  to make the life of plugin developers easier.

- [`littlefork-cli`](https://npmjs.org/package/littlefork-cli)

  A command line interface for Littlefork. It wraps the core and provides the
  `littlefork` executable.

- [`littlefork-api`](https://npmjs.org/package/littlefork-api)

  A RESTful HTTP API, that wraps the core. It allows to control Littlefork
  using HTTP requests.

- [`littlefork-plugin-*`](https://www.npmjs.org/browse/keyword/littlefork-plugin)

  All the functionality that Littlefork provides is encapsulated in plugins.

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->


- [Plugins](#plugins)
- [The Envelope](#the-envelope)
  - [Data](#data)
  - [Queries](#queries)
- [Tutorial: Developing plugins](#tutorial-developing-plugins)
  - [Bootstraping a new plugin module](#bootstraping-a-new-plugin-module)
  - [A look at the boilerplate plugin](#a-look-at-the-boilerplate-plugin)
- [Npm scripts](#npm-scripts)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Plugins

A plugin is a function that takes an envelope as input, and returns an
envelope as output. The exact signature of a Littlefork plugin is the
following:

`plugin(Envelope, {log, cfg}) -> Envelope`

The envelope contains all data and queries and every plugin has to return data
with the same interface. We will look at the envelope in more detail
later.

The second argument is a bag of different objects. Currently it provides the
whole configuration object of the run and a logger object. In the future this
can be extended.

Plugins can return either an envelope or a promise that resolves to an
envelope. The Littlefork core accepts both return types. Internally if the
return type is a value, it will be turned into a resolved promise.

Plugins implement an interface for functors and applicatives and offers a few
combinators on top of those abstractions. See the [API guide](./api#plugins)
for details.

## The Envelope

Data and queries are wrapped in an envelope, which is just an object. This is
a minimal valid envelope.

```
{
  data: [],
  queries: []
}
```

Both `data` and `queries` are a list of objects.

The envelope as well as `data` and `queries` implement various interfaces such
as monoids and functors. See the API guide [here](./api#units)
and [here](./api#envelope) for details.

### Data

Plugins create units of data and they are free to extend the format of a data
unit with whatever field they want. Certain fields gold meaning across the
whole of Littlefork, and plugins should not change their meaning. Some fields
are understood only by a few plugins.

- `_lf_id_hash` - A unique identifier for this data unit. It is usually a
  SHA256 hash. If the `_lf_id_fields` field is set, the Littlefork core
  automatically calculates the `_lf_id_hash` for every unit. This is the
  recommended way. Otherwise the plugin has to provide the hash by itself.

- `_lf_id_fields` - A list of field names, that specify the significant fields
  to form an identifier for this unit. Littlefork will take the values for
  those fields to calculate a unique identifier.

  `_lf_id_fields: ['tweet_id']`

- `_lf_content_hash` - A SHA256 hash of the content of the data
  unit. Littlefork uses this field to see if a particular piece of data
  changed. If the `_lf_content_fields` field is et, the Littlefork core
  automatically calculates the `_lf_content_hash` for every unit. This is the
  recommended way. Otherwise the plugin has to provide the hash by
  itself.

- `_lf_content_fields` - A list of field names, that specify the significant
  fields to form the content for this unit. Littlefork will take the values
  for those fields to calculate content integrity.

  `_lf_content_fields: ['tweet']`

- `_lf_pubdates` - An object, that contains various dates around this
  unit. Possible date types are `fetch`, the date and time when the unit was
  fetched, and `source`, the date and time that the unit was created. In the
  future more types of dates can be added. The dates should be JavaScript
  `Date` objects.

  ```
  _lf_pubdates: {
    fetch: Date("2017-03-21T23:22:35.426Z"),
    source: Date("2016-08-14T12:00:35Z")
  }
  ```

- `_lf_links`
- `_lf_media`
- `_lf_relations`
- `_lf_downloads`
_ `_lf_markers`

### Queries

Queries are, the same as data, a list of objects. Each object must have two
fields:

- `type` - What kind of query it is. Plugins will pick all queries they wanna
  operate on based on the type.

- `term` - The actual query. The format of the term depends on the
  type. Queries of type `ddg_search` are simple strings, whereas queries of
  type `mongodb_query` are a MongoDB query description.

Plugins document which query types they expect.

## Tutorial: Developing plugins

We will develop an example plugin for Littlefork. The plugin will list files
in a directory (specified by a glob pattern) and turn each file into an unit
of data.

### Bootstraping a new plugin module

There is
a [boilerplate](https://gitlab.com/littlefork/littlefork-boilerplate-plugin)
repository, which is a good start. The commands below create a new plugin, and
inside the plugin there is a minimal Littlefork project, that can be used to
test the module itself.

```
git clone https://gitlab.com/littlefork/littlefork-boilerplate-plugin littlefork-plugin-fs
cd littlefork-plugin-fs
sed -i -e 's/boilerplate-plugin/plugin-fs/g' package.json
sed -i -e 's/\(^[ ]*"description": "\).*\(",$\)/\1Provide file system related plugins for Littlefork.\2/g' package.json
rm -rf .git && git init && git add -A && git commit -m "Initial commit."
npm link
cd project
npm link littlefork-plugin-fs
```

You might wanna edit the `package.json` and change other fields as well:

- `name`
- `description`
- `repository`
- `keywords`
- `author`
- `bugs`
- `homepage`

The boilerplate repository has the following structure:

- `lib` :: The location of your ES6 source code. The actual plugin.
- `_dist` :: The compiled ES5 version of `lib`.
- `test` :: All unit and property tests go here. Files ending with `spec.js`
  are part of the test suite.
- `project` :: A minimal Littlefork project, to easily test the module under
  development.

The ES6 sources have to be compiled to ES5 before they can be used. Run `npm
run build` to do that. It's handy to run `npm run watch` during
development. See below for a list of all available npm scripts.

### A look at the boilerplate plugin

The entry point of the module can be found in `lib/index.js`. In order to
export a Littlefork plugin, the module has to export the `plugins` attribute,
which is a map with the plugin name as key, and the plugin function as value.

We will write a little plugin, that reads all files in a directory, and turns
each file into a unit of data.

This is a minimal `lib/index.js`.

```
import passPlugin from './pass';

const plugins = {
  pass: passPlugin,
};

export {plugins};
export default {plugins};
```

This module exports one plugin, that is called `fs_list`. Next step is to
implement the actual plugin. A plugin is simply a function that receives an
`Envelope` and a context, and returns either an `Envelope`, or a `Promise` for
an `Envelope`.

A minimal `lib/pass.js` can look like this:

```
const plugin = envelope => envelope;

export default plugin;
```

## Npm scripts

The `package.json` defines several tasks, that can be executed with `npm run`.

- `npm run test` :: Run all mocha unit tests in `test/**/*.spec.js`.
- `npm run lint` :: Run eslint on your ES6 sources in `lib` and your tests in
  `test`. It also lints [JSDoc](http://usejsdoc.org/) in `lib`.
- `npm run docs` :: Build the API docs of the module. Places the result into
  the Readme.
- `npm run compile` :: Compile the ES6 sources to ES5
  using [Babel](https://babeljs.io/).
- `npm run build` :: Test, lint, document and compile. Runs the former `npm`
  scripts and fails on error.
- `npm run watch` :: Watch the sources in `lib` and `test` and build the
  module whenever a single file changes.
- `npm run publish` :: When publishing a module to
  [npmjs.org](https://www.npmjs.com/), as a initial step, it builds the whole
  package using `npm run build`. Only the readme and `_dist` are published.
- `npm run clean` :: Delete the compiled sources in `_dist`.