# Data Migration with `DataMigration`

<!-- TOC depthFrom:2 -->

- [Purpose](#purpose)
- [Limitations](#limitations)
- [Tips](#tips)
- [Use cases](#use-cases)
  - [Common example](#common-example)
  - [Add a field in a document](#add-a-field-in-a-document)
  - [Update a field](#update-a-field)
  - [Remove a field](#remove-a-field)
  - [Update the value of a field based on other field values](#update-the-value-of-a-field-based-on-other-field-values)

<!-- /TOC -->

## Purpose

The class `DataMigration` is helping you to apply safely massive data migration on
your MongoDb collections.

Under the hood, `DataMigration` is backing up migrated data from each altered
document to a temporary collection named with the following convention:

> `{collection_name}_mig_{migration_id}`

Documents present in this collection have all the necessary information required
to restore the documents to the state available before the migration.

After the stabilization phase (ie. 1 day, 1 week depending), you can clean the
migration up by invoking the `migration.clean` method. This will remove the 
collection as well as the fields added to each migrated document if applicable
(see the option `continueOnFailure`).

## Limitations

- A `undefined` value is restored to `null`
- If you are adding a nested object, the default rollback operation will not remove
  the top level field
- The migration workflow can not validate the external validation of the documents
  performed by libraries such as `Joi` or `ajv` for JSON Schema. **You must validate
  this explicitly on you own by reading the document with your model after the update operation!**

## Tips

Select only the part of the document that you will use to determine your data
migration with the `projection` parameter of the constructor.

Select only the part of the projection that you will restore after the migration
with the `options.filter` parameter.

> Because in most cases, it is not necessary to apply a migration based on fields
> not impacted by the migration, we chose to place this parameter in the `options`

You can define your own `rollback` function with the `options.rollback`. In most
cases, this is not required because `DataMigration` is able to restore the data
to their previous state. You might need to use this if you have added a nested
object to the document and need to remove the entire tree on rollback.

If your migration is updating a large volume of data, you might consider the 
parameter `options.delayBetweenBatches` and `options.maxBulkSize` to split the
bulk operations into smaller bulks and to apply a delay between each batch.

If your migration must update a large volume of data and you suppose that the
process might fail for any valid but ignored reason, you can set the option
`continueOnFailure` option to `true` to allow the migration process to be
relaunched and to restart to the last migrated document.

> Be careful, with the option `continueOnFailure`, the migration must add a 
> temporary flag to each document with the same name of the migration collection.
> This might lead to application side validation issues. **You must check this
> case on your side and during your tests**

## Use cases

- Add a field in a document
- Update a field from a document
- Remove a field from a document
- Update a field from a document based on other values present in the same document


### Common example

The following code is the base code to apply a migration. The function must have
access to a MongoDb connection, a DataMigration install and additional information
such as a `doIt` boolean determining if the migration must be effectively applied
and a `isRollback`, boolean as well, determining if the migration is an update or
a rollback of an already executed migration.

```js
'use strict';

/**
 * Performs the migration
 * @param {object} db Mongodb database connection
 * @param {object} migration The migration to apply
 * @param {boolean} doIt Do we execute the migration effectively
 * @param {boolean} [isRollback=false] Is it a data migration rollback
 * @returns {Promise<object>} The migration result
 */
function apply(db, migration, doIt, isRollback = false) {
  return migration[isRollback ? 'rollback' : 'update')(db, doIt);
}
```

### Add a field in a document

```js
'use strict';

const { DataMigration } = require('chpr-mongodb');

const migration = new DataMigration(
  'field_addition', // Migration ID
  'users', // Collection name
  {
    // Migration applied for all users
  },
  {
    _id: 1 // Projection - No other field required
  },
  () => ({ // Filter function to apply for restoration 
    $set: {
      added_field: true
    }
  })
);

// then invoke apply(db, migration, doIt, isRollback)
```

### Update a field

```js
'use strict';

const { DataMigration } = require('chpr-mongodb');

const migration = new DataMigration(
  'field_update', // Migration ID
  'users', // Collection name
  {
    created_at: {
      $gt: new Date(2018, 0, 1)
    }
  },
  {
    points: 1
  },
  user => ({ // Filter function to apply for restoration 
    $set: {
      points: points + 10 // Add 10 points
    }
  })
);

// then invoke apply(db, migration, doIt, isRollback)
```

### Remove a field

```js
'use strict';

const { DataMigration } = require('chpr-mongodb');

const migration = new DataMigration(
  'field_removal', // Migration ID
  'users', // Collection name
  {
    created_at: {
      $gt: new Date(2018, 0, 1)
    }
  },
  {
    points: 1
  },
  ()) => ({ // Filter function to apply for restoration 
    $unset: {
      points: 1
    }
  })
);

// then invoke apply(db, migration, doIt, isRollback)
```

### Update the value of a field based on other field values

```js
'use strict';

const { DataMigration } = require('chpr-mongodb');

const migration = new DataMigration(
  'field_update_condition', // Migration ID
  'users', // Collection name
  {
    created_at: {
      $gt: new Date(2018, 0, 1)
    }
  },
  {
    loyalty_status: 1,
    points: 1
  },
  user => ({ // Filter function to apply for restoration 
    $set: {
      points: points + loyalty_status * 10 // Add 10 points based on loyalty status
    }
  }),
  {
    filter: user => ({
      points: user.points // Only the points will be restored
    })
  }
);

// then invoke apply(db, migration, doIt, isRollback)
```