{
  "name": "aws-scatter-gather",
  "version": "1.1.0",
  "description": "An NPM package that facilitates the scatter gather design pattern using AWS SNS.",
  "main": "index.js",
  "scripts": {
    "test": "mocha tests/*.js"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/byu-oit-appdev/aws-scatter-gather.git"
  },
  "keywords": [
    "amazon",
    "AWS",
    "gather",
    "lambda",
    "scatter",
    "scather",
    "sns",
    "topic"
  ],
  "author": {
    "name": "James Speirs"
  },
  "license": "Apache-2.0",
  "bugs": {
    "url": "https://github.com/byu-oit-appdev/aws-scatter-gather/issues"
  },
  "homepage": "https://github.com/byu-oit-appdev/aws-scatter-gather#readme",
  "dependencies": {
    "aws-sdk": "^2.5.3",
    "chalk": "^1.1.3",
    "debug": "^2.3.2",
    "object-schemata": "^1.0.8",
    "uuid": "^2.0.2"
  },
  "devDependencies": {
    "chai": "^3.5.0",
    "mocha": "^2.5.3"
  },
  "gitHead": "0fdb3fa0906cfdcfecc17e2156e8739f8d8c940b",
  "readme": "# aws-scatter-gather\n\nThe purpose of this module is to facilitate distributed computing while using a scatter gather design pattern with AWS SNS Topics.\n\n\n# THIS IS CURRENTLY OUT OF DATE - WILL BE FIXED VERY SOON\n\n\n\n\n\n## Table of Contents\n\n- [Install](#install)\n- [Scatter-Gather Overview](#scatter-gather-overview)\n- [Code Examples](#code-examples)\n- [Getting Started](#getting-started)\n\n## Install\n\n```sh\n$ npm install aws-scatter-gather\n```\n\n## Scatter-Gather Overview\n\nThere are two types of processes involved in scatter-gather:\n\n1. Aggregators - Sends a request and gather's responses.\n\n2. Responders - Receives a request and sends a response.\n\nLook at the following diagram that has one aggregator and two responders:\n\n![Scatter-Gather Concept](./img/concept.png)\n\n1. The aggregator publishes out a single request to an AWS SNS Topic.\n\n2. The SNS Topic pushes the event to all subscribers (Responder A and Responder B).\n\n3. Eventually Responder A publishes a response to the SNS Topic that the SNS Topic will push to the aggregator.\n\n4. Eventually Responder B publishes a response to the SNS Topic that the SNS Topic will push to the aggregator.\n\nIn reality by having a single SNS Topic for both requests and responses we get a lot of superfluous network chatter, as can be seen in the following diagram:\n\n![Scatter-Gather Single Topic Problem](./img/reality.png)\n\nYou can see from this diagram that because the aggregator and responders are subscribed to the same topic that they publish to, that topic pushes the events to the caller as well.\n\nThe solution to this is to have two SNS Topics. The aggregator publishes to the topic that responders are subscribed to. Responders publish to the topic that the aggregator is subscribed to. See the following diagram:\n\n![Scatter-Gather Multiple Topics](./img/fix.png)\n\n## Code Examples\n\n##### Aggregator Example\n\nThe aggregator must exist on a computer process that is capable of both sending and receiving SNS Topic events. That pretty much rules out AWS Lambda functions.\n\nAlthough this example is to the point, a better example can be found in the [Getting Started](#getting-started) section.\n\n```js\nconst AWS               = require('aws-sdk');\nconst express           = require('express');\nconst Scather           = require('aws-scatter-gather');\n\n// create an express app and define the topics\nconst app = express();\nconst requestArn = 'arn:aws:sns:us-west-2:064824991063:TopicX'\nconst responseArn = 'arn:aws:sns:us-west-2:064824991063:TopicY';\n\n// your server will now process AWS SNS Notifications, Subscription Confirmations, etc.\napp.use(Scather.server.middleware());\n\n// start the server listening on port 3000\napp.listen(3000, function () {\n\n    // subscribe your server to a specific topic arn\n    Scather.server.subscribe(responseArn, 'http://my-server.com:3000');\n\n    // define the request configuration\n    const aggregator = Scather.aggregator({\n        expects: [ 'greet' ],\n        responseArn: responseArn\n        topicArn: requestArn\n    });\n\n    // make the request\n    aggregator('Bob', function(err, data) {\n        if (err) {\n            console.error(err.stack);\n        } else {\n            console.log(data.greet);    // 'Hello, Bob'\n        }\n    });\n});\n```\n\n##### Responder Example\n\nA responder runs best on an AWS Lambda that is subscribed to run on the response SNS Topic ARN.\n\nThe name of this particular lambda is `greet` which matches the item that the aggregator expects in the example above.\n\n```js\nconst AWS       = require('aws-sdk');\nconst Scather   = require('aws-scatter-gather');\n\nexports.handler = Scather.response(function(message, context, callback) {\n    callback(null, 'Hello, ' + message);\n});\n```\n\n\nUsing this package, aggregators must exist in an environment where it is able to receive SNS Topic events on running processes. Although it's probably possible to do this on an AWS lambda, it's not recommended. Responders on the other hand can exist within long running processes or within short lived processes (like an AWS Lambda).\n\n**If you plan on using Lambdas then the Lambdas must have a role that allows for publishing to AWS SNS Topics.**\n\n## Getting Started\n\nThis section of the documentation is a tutorial about getting your first scatter-gather application up and running. We'll talk about local development first and then briefly cover how to get that deployed and ready for production.\n\nThe objective of this tutorial is to create a web server that you can post requests to. Each request will use the aggregator to gather results and return them.\n\n1. Create your project directory.\n\n2. Create the responder **double** that will receive a number and publish it's value doubled.\n\n    1. Create a folder within your project directory named `double`.\n\n    2. Within that folder create a file named `index.js`. The name is important because we can plug it directly into an AWS Lambda later.\n\n    3. Inside of `double/index.js` put this code:\n\n    ```js\n    'use strict';\n    const AWS       = require('aws-sdk');\n    const Scather   = require('aws-scatter-gather');\n\n    exports.handler = Scather.response(function(message, context, callback) {\n        callback(null, message * 2);\n    });\n    ```\n\n    4. From the command line or a terminal:\n\n        1. Navigate to your project directory\n        2. Navigate into the `double` directory\n        3. Type the command `npm init` to create a package.json file.\n        4. Install the aws-scatter-gather dependency: `npm install --save aws-scatter-gather`\n\n    5. At this point your project directory should look like this:\n\n        - **my-project**\n                - **double**\n                    - **node_modules**\n                    - index.js\n                    - package.json\n\n3. Create a responder **increment** that will receive a number and publish it's value incremented by one.\n\n    1. Create a folder within your project directory named `increment`.\n\n    2. Within that folder create a file named `index.js`. The name is important because we can plug it directly into an AWS Lambda later.\n\n        3. Inside of `increment/index.js` put this code:\n\n        ```js\n        'use strict';\n        const AWS       = require('aws-sdk');\n        const Scather   = require('aws-scatter-gather');\n\n        exports.handler = Scather.response(function(message, context, callback) {\n            callback(null, message + 1);\n        });\n        ```\n\n        4. From the command line or a terminal:\n\n            1. Navigate to your project directory\n            2. Navigate into the `increment` directory\n            3. Type the command `npm init` to create a package.json file.\n            4. Install the aws-scatter-gather dependency: `npm install --save aws-scatter-gather`\n\n        5. At this point your project directory should look like this:\n\n            - **my-project**\n                    - **double**\n                        - **node_modules**\n                        - index.js\n                        - package.json\n                    - **increment**\n                        - **node_modules**\n                        - index.js\n                        - package.json\n\n\n### Local Development\n\n1. Structure your file system like this:\n\n    - **my-project**x\n        - **double**\n            - index.js\n            - package.json\n        - **increment**\n            - index.js\n            - package.json\n        - **server**\n            - aggregators.js\n            - index.js\n            - test.js\n            - package.json\n        - orchestra.js\n        - package.json\n\n    Be sure to include the `aws-sdk` and `aws-scatter-gather` packages as dependencies in each package.json file.\n\n2. If your computer is behind a NAT or a firewall and you want receive events from the remote AWS SNS then use [ngrok](https://ngrok.com/) or something similar to create a tunnel from the public internet to your local machine. Only do this if you want to receive events from AWS SNS because you'll already get local events.\n\n3. The orchestra.js should be something like this:\n\n    ```js\n    const AWS               = require('aws-sdk');\n    const Scather           = require('aws-scatter-gather');\n\n    // include lambda index files\n    const aggregator        = require('./aggregator/index');\n    const double            = require('./double/index');\n    const increment         = require('./increment/index');\n\n    // create a mock subscription for the responders\n    const requestArn = 'arn:aws:sns:us-west-2:064824991063:TopicY';\n    Scather.local.subscribe(requestArn, 'double', double.handler);\n    Scather.local.subscribe(requestArn, 'increment', increment.handler);\n\n    // execute the aggregator function\n    aggregator.handler({}, {}, function(err, data) {\n        console.log(err, data);\n    });\n    ```\n\n    The above example does not require you to have a network connection to AWS.\n\n4. Run your orchestra.js code in debug mode and use an inspector to troubleshoot errors.\n\n\n## Without AWS Lambda\n\nIt is possible to run both aggregators and responders either on or off AWS Lambdas. This section highlights how to do this off of an AWS Lambda.\n\nTo accomplish this:\n\n- You must be running your own NodeJS server that accepts connect middleware.\n- You will need to tell the server to use the Scather middleware.\n- You will need to make an SNS Topic subscription request using Scather's subscription interface.\n\nOnce done, any aggregators or responders that you define will automatically be called as AWS SNS Notifications come in on your server.\n\n##### Express Server Example\n\n[Express](http://expressjs.com/) is one of many NodeJS servers that accept connect middleware.\n\n```js\nconst AWS               = require('aws-sdk');\nconst express           = require('express');\nconst Scather           = require('aws-scatter-gather');\n\nconst app = express();\n\n// your server will now process AWS SNS Notifications, Subscription Confirmations, etc.\napp.use(Scather.server.middleware());\n\napp.listen(3000, function () {\n    console.log('Server listening on port 3000!');\n});\n\n// subscribe your server to a specific topic arn\nScather.server.subscribe('arn:aws:sns:us-west-2:064824991063:TopicX', 'http://myserver.com');\n```\n\nNote that although it is possible to use the AWS SDK to subscribe to AWS SNS directly, the middleware will not confirm any subscriptions that were not made through `Scather.server.subscribe()`.\n\n\n## Developing Locally\n\nIf you plan to do initial development on your local machine before moving them to AWS Lambdas then here are some tips on how to do that.\n\n1. Structure your file system like this:\n\n    - **my-project**\n        - **aggregator**\n            - index.js *(similar to [aggregator example](#aws-lambda-better-aggregator))*\n            - package.json\n        - **double**\n            - index.js *(similar to [double example](#aws-lambda-double-response))*\n            - package.json\n        - **increment**\n            - index.js *(similar to [increment example](#aws-lambda-increment-response))*\n            - package.json\n        - orchestra.js\n        - package.json\n\n    Be sure to include the `aws-sdk` and `aws-scatter-gather` packages as dependencies in each package.json file.\n\n2. If your computer is behind a NAT or a firewall and you want receive events from the remote AWS SNS then use [ngrok](https://ngrok.com/) or something similar to create a tunnel from the public internet to your local machine. Only do this if you want to receive events from AWS SNS because you'll already get local events.\n\n3. The orchestra.js should be something like this:\n\n    ```js\n    const AWS               = require('aws-sdk');\n    const Scather           = require('aws-scatter-gather');\n\n    // include lambda index files\n    const aggregator        = require('./aggregator/index');\n    const double            = require('./double/index');\n    const increment         = require('./increment/index');\n\n    // create a mock subscription for the responders\n    const requestArn = 'arn:aws:sns:us-west-2:064824991063:TopicY';\n    Scather.local.subscribe(requestArn, 'double', double.handler);\n    Scather.local.subscribe(requestArn, 'increment', increment.handler);\n\n    // execute the aggregator function\n    aggregator.handler({}, {}, function(err, data) {\n        console.log(err, data);\n    });\n    ```\n\n    The above example does not require you to have a network connection to AWS.\n\n4. Run your orchestra.js code in debug mode and use an inspector to troubleshoot errors.\n\n## API\n\n### aggregator ( configuration ) : Function\n\nCreate an aggregator function that can be called to start a scatter gather request.\n\nCreating an aggregator also creates a [local subscription](#local-subscriptions) to the `topicArn` or `responseArn` provided in the config parameter. To end that subscription use the `aggregator.unsubscribe()` function.\n\nParameters\n\n- *configuration* - The configuration to apply to the aggregator. Any time the aggregator is called it will use this same configuration. Properties for the configuration include:\n\n    - *expects* - An array of strings for function names that the aggregator expects to hear back from. Once all expected responses are received and the `minWait` time has been reached then the aggregator will complete. Defaults to `[]`.\n    - *functionName* - The name of the aggregator function. This value is not necessary, but it does help when looking at logs and some response functions might care about the name of the function that is making the request. Defaults to '-'.\n    - *maxWait* - The maximum number of milliseconds to wait before stopping response aggregation. Defaults to `2500`.\n    - *minWait* - The minimum number of milliseconds to wait before stopping response aggregation. Defaults to `0`.\n    - *responseArn* - The SNS Topic Arn that the aggregator will listen on for responses. Defaults to using the `topicArn` if not provided.\n    - *topicArn* - The SNS Topic Arn to publish the request on.\n\nReturns an aggregator function with unsubscribe property. **The aggregator function can either use a callback paradigm or a promise paradigm.**\n\n```js\nconst aggregator = Scather.Aggregator({ topicArn: 'arn:aws:sns:us-west-2:064824991063:TopicX' });\n\naggregator(5, function(err, data) {\n    console.log(results);\n});\n\naggregator(10)\n    .then(function(data) {\n        console.log(data);\n    });\n\n// check current subscription status\nconsole.log(aggregator.subscribed);     // true\n\n// no new aggregator calls after this and unsubscribe will wait\n// for running aggregators to finish\naggregator.unsubscribe();\n```\n\n### Event Interface\n\nThe aws-scatter-gather package is largely event driven. You can ignore the events if you want, or tap into them if you want to do something beyond what the aws-scatter-gather package provides.\n\n**Events**\n\n- **LOG** - This package outputs log information to the LOG event. You can listen to this event if you want to capture all logs that are derived by this package.\n- **NOTIFICATION** - This event is fired whenever an AWS SNS Topic event is produced. The events may come from the internal system or from AWS. You can determine if the event come from the aws-scatter-gather package by looking determining if the event property `Record[x].EventSource` is equal to `awssg:local`.\n- **PUBLISH** - This event occurs if an event is published by the aws-scatter-gather package.\n- **SNS** - This event occurs to report the success of an AWS-SDK call made by the aws-scatter-gather package.\n- **SUBSCRIBE** - This event occurs when a subscription is created.\n- **UNSUBSCRIBE** - This event occurs when a subscription is ended.\n\n**Example: Capturing all Notification Events**\n\nNotice in the example that the notifications are logged to the console by listening on `events.NOTIFICATION`.\n\n```js\nconst Scather = require('aws-scatter-gather');\nScather.events.on(Scather.events.NOTIFICATION, function(event) {\n    console.log(event);\n});\n```\n\n#### events.off ([ eventName, ] callback ) : undefined\n\nOnce an event has been added you can remove it via this method. The parameters you used to add the listener must be the same parameters to remove the listener.\n\nParameters:\n\n- *eventName* - Optional. The name of the event to remove the callback for.\n- *callback* - The event listener function to remove.\n\nReturns: undefined\n\n#### events.on ([ eventName, ] callback ) : undefined\n\nAdd an event listener function that will be called for each event specified.\n\nParameters\n\n- *eventName* - Optional. The name of the event to call the callback for. If omitted then the callback will be called for all events.\n- *callback* - The function to call with the event. The function will only receive one parameter.\n\nReturns: undefined\n\n#### events.once ([ eventName, ] callback ) : undefined\n\nAdd an event listener function that will be called just once for the event type specified. After the callback is called once then it will not be called again unless you put it back on the event interface.\n\nParameters\n\n- *eventName* - Optional. The name of the event to call the callback for. If omitted then the callback will be called for all events.\n- *callback* - The function to call with the event. The function will only receive one parameter.\n\nReturns: undefined\n\n## Local Subscriptions\n\nThe aws-scatter-gather package uses a two-tier subscription system for transmitting messages. The first tier is local and the second tier is an interface with the AWS SNS system. The entire system can work without using the second tier in what is essentially an offline mode. Here is how it works:\n\n1. An event occurs on the machine.\n2. The event propagates to listeners on the local machine.\n3. If the AWS has credentials, the Topic Arn is valid, and the server network is [enabled](server-enabled) then the local machine publishes the event the the AWS SNS.\n4. Any events from the AWS SNS are propagated to listeners on the local machine.\n\n#### local.subscribe ( topicArn, functionName, handler ) : undefined\n\nCreate a local subscription for the specified topic arn to the provided handler.\n\nParameters\n\n- *topicArn* - The topic to start listening on.\n- *functionName* - The name to assign to the lambda function.\n- *handler* - The function to remove that is like an AWS lambda handler.\n\nReturns: undefined\n\n#### local.unsubscribe ( topicArn, handler ) : undefined\n\nRemove the local subscription for the specified topic arn to the provided handler.\n\nParameters\n\n- *topicArn* - The topic to stop listening on.\n- *handler* - The function to remove that is like an AWS lambda handler.\n\nReturns: undefined\n\n### logger\n\n#### logger ( namespace [, silent ] ) : Object\n\nCreate an aws-scatter-gather logger instance. You can then log to `.info`, `.warn`, or `.error`. Logged events are pushed to the local event interface and handled there.\n\nParameters\n\n- *namespace* - The name to attach to all log events that describes the domain of events being logged.\n- *silent* - Optional. Whether the logging should be silent, being pushed to the local event interface but not logged to the console. Set to true to also log to the console. If not specified then the default silent setting for all logs will be used.\n\nReturns: An object with `.info`, `.warn`, and `.error` properties that are functions that can be called to log information.\n\n```js\n// create a logger instance\nconst Scather = require('aws-scatter-gather');\nconst myLog = Scather.logger('my-log');\nmyLog.info('My logger created');\n```\n\n#### logger.events\n\nA boolean property value that if set to true will log all events for the system to the console. Defaults to `false`.\n\n```js\n// log all events\nconst Scather = require('aws-scatter-gather');\nScather.logger.events = true;\n```\n\n#### logger.silent\n\nA boolean property value that sets the default logging to console mode. If set to true then the logs for the aws-scatter-gather package will be output to the console. Defaults to `true`.\n\n```js\n// log scather events to the console\nconst Scather = require('aws-scatter-gather');\nScather.logger.silent = false;\n```\n\n### mock\n\nThese functions are used for mocking and testing.\n\n#### mock.requestEvent ( topicArn, message, attributes ) : Object\n\nCreate an event that resembles the event that an SNS Topic would send to a lambda function when that event was produced by an aggregator. This is useful for testing Scather.response functions.\n\nParameters\n\n- *topicArn* - The SNS topic arn.\n- *message* - The message value send from the aggregator.\n- *attributes* - An object defining the attributes that accompany the message.\n\nReturns: An object that looks like an AWS SNS Event.\n\n### response ( [configuration, ] callback ) : Function\n\nCreate a function that is a Scather response generator. A response generator function will only be called when it receives an aggregator request event.\n\nParameters\n\n- *configuration* - An object that specifies how to run the response. Options include:\n\n    - *development* - If set to true and the response function encounters an error then it will publish the error to the SNS response arn. The aggregator will then capture the response, recognize that it received an answer (but not add it to the final result set), and it will log it.\n\n- *callback* - A function that will accept aggregator event data, process it, and respond to it. The callback has the signature `function ( data, attributes [, done ])` where:\n\n    - *value* - The value sent by the aggregator.\n\n    - *context* - The current context of the event.\n\n    - *done* - Optional. A function that is called when the response is finished. It takes two parameters, `error` and `data`. If there are no errors then set the `error` value to `null`. The value sent as `data` will be the value that the aggregator receives. If this parameter is omitted then the function will return a promise.\n\nReturns: Function. **This function can either use a callback paradigm or a promise paradigm.** When the function is called, if it is called with a third parameter (a callback function) then the callback will be called when completed. If the third parameter is omitted then the function will return a promise.\n\n##### Response Callback Paradigm Example\n\n```js\nconst AWS = require('aws-sdk');\nconst Scather = require('./index');\n\nexports.handler = Scather.response(function(message, context, callback) {\n    // only the data for aggregator events will cause this function to execute\n    callback(null, message * 2);\n});\n```\n##### Response Promise Paradigm Example\n\nThis example is not the greatest, but it gets the point across.\n\n```js\nconst AWS = require('aws-sdk');\nconst Scather = require('./index');\n\nconst respond = Scather.response(function(message, context) {\n    // only the data for aggregator events will cause this function to execute\n    return message * 2;\n});\n\nexports.handler = function(event, context, callback) {\n    respond(event, context)\n        .then(function(value) {\n            callback(null, value);\n        })\n        .catch(function(err) {\n            callback(err, null);\n        });\n};\n```\n\n### server\n\nHelper methods for those who are running their own server.\n\n#### server.enabled\n\nA property that if `true` will allow events to be published to the AWS SNS and notifications to be received by the local machine. Defaults to `true`.\n\n#### server.middleware ( configuration ) : Function\n\nReturns a middleware function that any NodeJS server that supports connect middleware can use to link the server with AWS SNS.\n\nParameters\n\n- *configuration* - An object representing the middleware configuration. It has the following properties:\n\n    - *passThrough* - Optional. If set to true then once the middleware has completed it will continue to run other server middleware and routes. Defaults to `false`.\n\nReturns: A middleware function.\n\n#### server.subscribe ( topicArn, endpoint ) : Promise\n\nCreate an AWS SNS subscription to the specified topic for the server.\n\nParameters\n\n- *topicArn* - The AWS SNS Topic arn to subscribe to.\n\n- *endpoint* - The public endpoint URL for this server.\n\nReturns: A promise that resolves on subscription confirmation or that rejects on error.\n\n#### server.unsubscribe ( topicArn, endpoint ) : Promise\n\nRemove an AWS SNS subscription to the specified topic for the server.\n\nParameters\n\n- *topicArn* - The AWS SNS Topic arn to subscribe to.\n\n- *endpoint* - The public endpoint URL for this server.\n\nReturns: A promise that resolves on unsubscribe request completion or that rejects on error.",
  "readmeFilename": "README.md",
  "_id": "aws-scatter-gather@1.1.0",
  "_shasum": "b6f6e6cf12e131a32805a94a6875feb3e3bb2ccb",
  "_from": "byu-oit-appdev/aws-scatter-gather",
  "_resolved": "git://github.com/byu-oit-appdev/aws-scatter-gather.git#0fdb3fa0906cfdcfecc17e2156e8739f8d8c940b"
}
