# Data

Kazan is using [CouchDB](http://couchdb.apache.org/) for data persistence.

All documents must follow these conventions:

- `"type"` property must be set
- `"createdAt"` and `"updatedAt"` must be set. All timestamps must be in the
   [ISO 8601](http://en.wikipedia.org/wiki/ISO_8601) format `yyyy-mm-ddThh:mm:ss.mmmZ`
- When removed, all other properties must remain in tact, with the additional
   `"_deleted": true` & `"removedAt"` properties

## TOC

1. [Databases](#databases)
   1. [raw-data](#raw-data)
   1. [integrated-data](#integrated-data)
   1. [issues](#issues)
   1. [entities](#entities)
   1. [_users](#_users)
1. [Data Lifecycle](#data-lifecycle)

## Databases

It creates four databases by default, the names [can be configured](../config.md).

1. [raw-data](rawdata)
1. [integrated-data](integrateddata)
1. [issues](issues)
1. [entities](entities)
1. [_users](users)

### raw-data

All user data submitted by users or imported from 3rd party services are stored
in the `raw-data` database. Every source data report is stored in a separate
CouchDB document. There is no altering of the source data, indicators are stored
in a separate `integrated-data` database after transformation,
see [Data Lifecycle](#datalifecycle).

Only [data providers](permissions.md#dataproviders) can write data. Only
[reviewers](permissions.md#reviewers) can read/write data belonging to
other users.

A `raw-data` document looks like this:

```js
{
   // _id & _rev are required by CouchDB
   // _id is made of {sourceId}_{scope.id}
   "_id": "bookkeeping_2016-1",
   "_rev": "2-b7d269ff331bc48dc89f2d2d83a4b0a4",
   // required: sourceId is used to group data by source.
   "sourceId": "bookkeeping",
   // scope is used to differentiate data reports from the same source.
   // Only source.id is required, but more meta information can be added
   "scope": {
       "id": "2016-1",
       "year": "2016",
       "month": "1"
   },
   // optional: human-friendly name of the report
   "name": "January 2016 expenses",
   // optional: description for the report
   "description": "",
   // submitted raw data, can be of any type (string, Array, Object, ...)
   // One of data and _attachments must be provide
   "data": "-1200 rent\n-210 utilities\n-890 household",
   // required: timestamps  & user reference
   // only createdBy.id is required, but more meta information can be added
   "createdAt": "2016-01-01T12:00:00.000Z",
   "updatedAt": "2016-01-01T12:00:00.000Z",
   "createdBy": {
       "createdAt": "2016-01-01T12:00:00.000Z",
       "fullname": "John Doe",
       "login": "john@example.com",
       "id": "abc4567",
       "roles": []
   },
   // when the has permissions based on roles, access can be limited by
   // listing group ids in the access property. If the user is assigned to
   // entities based on roles, all roles will be taken over to the report,
   // unless the `access` property has been passesd with a subset of these.
   "access": [
     "entity:district:123",
     "entity:region:abc"
   ],
   // transformedAt, transformationSummary, transformationError are added
   // by Kazana automatically after a the document has been transformed
   "transformedAt": "2016-01-01T12:00:00.000Z",
   "transformationSummary": "Transformed into 2 indicators."
}
```

### integrated-data

`integrated-data` is an internal read-only database. Kazana apps or plugins
should never change documents directly, all changes must go through the
`raw-data` database.

Whenever a new source data report gets stored in `raw-data` for which a
[transformation scrip](transformation.md) script exists gets transformed into
indicators and stored in the `integrated-data` database. If a source data
report gets removed from `raw-data`, all transformed indicators get removed
from `integrated-data` automatically. If a raw data report gets updated, all
existing indicators get removed and re-added.

Only [Analysts](permissions.md#dataproviders) can access indicators and indices
from `integrated-data`.

An `integrated-data` document looks like this:

```js
{
    // _id & _rev are required by CouchDB
    // _id is made of {dataReport.id}:{uuid}
    "_id": "bookkeeping_2013-1:dtZ7XR6pnd",
    "_rev": "1-7243f58bf660e2d32af035043c9ef5f6",
    // indicator data
    "data": {
        "amount": -1200.00,
        "category": "test-transform-grinder"
    },
    // dataReport meta data
    "dataReport": {
        "id": "bookkeeping_2013-1",
        "sourceId": "bookkeeping",
        "createdAt": "2016-01-01T12:00:00.000Z",
        "updatedAt": "2016-01-01T12:00:00.000Z",
        "createdBy": {
            "createdAt": "2016-01-01T12:00:00.000Z",
            "fullname": "John Doe",
            "login": "john@example.com",
            "id": "abc4567",
        }
    },
    // timestamps (removedAt is added when _deleted: true)
    "createdAt": "2016-01-02T12:00:00.000Z",
    "updatedAt": "2016-01-02T12:00:00.000Z"
}
```

### issues

_TO BE DONE_

Issues are created as part of [auditing](auditing.md). Issues never get
deleted, only marked as closed, so data quality can be analysed over time.

An `issue` document looks like this

```js
{
    "type": "movement",
    "name": "amount to high (> 1000)",
    "report": {
      "id": "bookkeeping_2016-01",
      "rev": "4-567revision",
      "sourceId": "bookkeeping",
      "scope": {
        "id": "2016-01",
        "year": 2016,
        "month": 1
      },
    },
    "createdAt": "2016-01-02T12:00:00.000Z",
    "updatedAt": "2016-01-02T12:00:00.000Z",
    // when issue gets closed
    "closedAt": "2016-01-02T12:00:00.000Z",
    "closedBy": "transform"
}
```

### entities

_WORK IN PROGRESS_

Entities is the place where meta information is stored, like raw data report
sources, indicators, indices, user groups, regions, and whatever an app might
need. Access can be [configured per entity type](entities.md#access). By default,
all entities can be accessed by users, and only be changed by admins. The
access control is implemented on the hapi level, the CouchDB database can
only be accessed by admins.

An `entities` document looks like this:

```js
{
    // _id & _rev are required by CouchDB
    // _id is made of {dataReport.id}:{uuid}
    "_id": "dataReport:abc4567",
    "_rev": "1-7243f58bf660e2d32af035043c9ef5f6",
    // type is required
    "type": "dataReport",
    // timestamps (removedAt is added when _deleted: true)
    "createdAt": "2016-01-02T12:00:00.000Z",
    "updatedAt": "2016-01-02T12:00:00.000Z",
    // optional
    "name": "Bookkeeping",
    "description": "Monthly expense report",
    // cron format, in this case on the 1st of every month
    "schedule": "0 0 1 * *",
    // more properties can be added as needed
}
```

For hierarchies where an entity belongs to another entity, the `"_id"` property
is prefixed by the parent entity ID. For example, if there is a region with
`"_id": "region:uuid123"`, a district within that region would have
`"_id": "region:uuid123:district:uuid567"`.

### \_users

_WORK IN PROGRESS_

Kazana is using [hoodie-server-account](https://github.com/hoodiehq/hoodie-server-account),
a generic hapi plugin that implements all kind of account-related functionality
backed by CouchDB.

A standard CouchDB user document looks like this:

```js
{
   // Properties required by CouchDB
   // _id MUST consist of "org.couchdb.user:" + username
   "_id": "org.couchdb.user:test",
   // _rev is required by CouchDB for every document
   "_rev": "1-c7eb42781549d144e6a42814376686e0",
   // name MUST be the username
   "name": "pat",
   // type MUST be set to "user"
   "type": "user",
   // iterations, password_scheme, derived_key & salt are automatically created
   // by CouchDB to hash the password. The original password does not get stored
   // and cannot be retrieved
   "iterations": 10,
   "password_scheme": "pbkdf2",
   "derived_key": "94266b18ecec62aa78cbe15cb27e98d7689ded5c",
   "salt": "ae995d9d359cb88105d120a0a8c498a2",
   // roles must be an array of strings. Roles can be used to give access
   // to databases. It must include an "id:..." role, see below
   "roles": ["id:abc4567"]
}
```

See https://github.com/hoodiehq/hoodie-server-account/blob/master/how-it-works.md


## Data Lifecycle

![kazana Data Lifecycle](https://raw.githubusercontent.com/eHealthAfrica/kazana/documentation/documentation/assets/kazana-data-lifecycle.png)

Data is submitted to Kazana as a report into the raw data store.
Before submission, it can optionally be validated against a JSON Schema.

References

- [Definition Data Warehouse](https://en.wikipedia.org/wiki/Data_warehouse)
- [Definition Extract, Transform, Load](https://en.wikipedia.org/wiki/Extract,_transform,_load#Real-life_ETL_cycle)