# Firestore Export Import

[![pipeline status](https://gitlab.com/endran/firestore-export-import/badges/develop/pipeline.svg)](https://gitlab.com/endran/firestore-export-import/commits/master)
[![licence](https://img.shields.io/npm/l/@endran/firestore-export-import.svg?style=flat)](https://opensource.org/licenses/MIT)
[![npm version](https://img.shields.io/npm/v/@endran/firestore-export-import.svg?style=flat)](https://badge.fury.io/js/%40endran%2Ffirestore-export-import)

An exporter and importer for Firestore, capable of handling huge data sets, since it uses streaming and batching. Can also be used with the Firebase Emulator suite.

## Installation

```bash
npm install @endran/firestore-export-import --save
```

or

```bash
yarn add @endran/firestore-export-import
```

## Usage

You can use _Firestore Export Import_ as a CLI tool, or as a library.

Use the **CLI** as below. `--export` has a higher priority then `--import`, which has a higher priority then `--clear`. Only one of the three commands will be executed.

-   `--export` will export any document or collection denoted by `ref` to the file denoted by `--file`. Use `--override` to allow overwriting any `file` that may already exist.
-   `--clear` erases all data denoted by `ref`, including all sub collection. use `__ROOT__` in any command to point at the root of Firestore.
-   `--import` writes the contents of an exported `--file` to firestore. Use `--batchSize` to limit the number of parallel promises fired at Firebase. If there is already existing data in Firestore, it will only override documents by id, and it will not erase documents in sub collections. Use `--clear` first if this is desirable.

```bash
fei --serviceAccount firebase-admin.json --ref "someCol" --file export.json --export --override
fei --serviceAccount firebase-admin.json --ref "__ROOT__" --clear
fei --serviceAccount firebase-admin.json --file export.json --import --batchSize 100
```

## File format:

_Firestore Export Import_ uses file streaming when exporting and importing. This assured that even the largest database can use _Firestore Export Import_. However, this also comes with a cost.

**Warning:** Do **NOT** use a formatted file for import, use the identical file syntax that is produced during export. Even though a `.json` file is exported, it is not treated like `.json` during import. The file is streamed line by line, so that it never has to be in memory completely. Formatting this file will corrupt it\'s structure and may lead to loss of data, or a corrupted database.

_Firestore Export Import_ exports all documents by path. The content of each document follows the same serialization principles as [node-firestore-import-export](https://www.npmjs.com/package/node-firestore-import-export), so there is support for complex `Timestamp`, `Geopoint` and `DocumentReference`. However, due to version issues it will serialize any object that matches the interface, instead of a `instanceof` check.

## Library use

If you need to export from code, use the following. To clear or import use `ClearService` or `ImportService`.

Both `ExportService` and `ImportService` also allow a `transformer` parameter, to modify data. This can be useful for example to encrypt personal information, per document, or to omit specific documents, by returning `null`.
`ExportService` also allows for `excludedCollections`, this array is treated as an array of regexes, and when matched the particular collection will be skipped during export.

```typescript
const exportService = new ExportService(firestore);
await exportService.export(ref, writeStream, timestamp);

// ---

export class ExportService {
    constructor(private firestore: FirebaseFirestore.Firestore) {}
    async export(
            path: string,
            writeStream: WriteStream,
            timestamp: string,
            transformer: (data: any, path: string) => Promise<any> = async data => data,
            excludedCollections: string[] = [],
        ): Promise<void>  {...}
}

export class ImportService {
    constructor(private firestore: FirebaseFirestore.Firestore) {}
    async import(
            readStream: fs.ReadStream,
            batchSize: number = Util.batchSize,
            transformer: (data: any, path: string) => Promise<any> = async data => data
        ): Promise<void> {...}
}

export class ClearService {
    constructor(private firestore: FirebaseFirestore.Firestore) {}
    async clear(path: string): Promise<void> {...}
}
```

Next to the content of the `ref` an additional `metadata` is added, which contains a `timestamp` and the `ref`.

## Help

```
Usage: index [options]

Options:
  -V, --version                output the version number
  -S, --serviceAccount <path>  Required. Location of service account JSON file.
  -E, --emulator <number>      Use the FireStore emulator on the provided port.
  -R, --ref <path>             Required for --export and --clear. Reference to collection or document. Use '__ROOT__' for the root of Firestore.
  -F, --file <path>            Required for --export and --import. Location of file on disk.
  --export                     Will download ref including sub-collections to file. Has priority over import and clear.
  --import                     Will upload contents of file, merging data if required. Has priority over clear.
  --clear                      Will delete ref, and all sub-collections.
  -O, --override               Optional. If set will override any existing file during export.
  -B, --batchSize <number>     Optional. Set the amount of concurrent promises fired at Firebase. Constrains memory load for constrained devices. Use 0 to disable batching. (default: 50)
  -h, --help                   output usage information

Warning:
  Do NOT use a formatted file for import, use the identical file syntax that is produced during export.
  Even though a `.json` file is exported, it is not treated like `.json` during import.
  The file is streamed line by line, so that it never has to be in memory completely.
  Formatting this file will corrupt it's structure and may lead to loss of data, or a corrupted database.
```

## Notes

**Service Account:**<br>
You can get the `firebase-admin.json` file from the [Firebase Console](https://console.firebase.google.com/). Select _<your project>_, click the _cog_, go to _User and Permission_, then _Service Accounts_, tick _Node.js_, and hit _Generate_.

**Empty documents:**<br>
Sadly _Firestore Export Import_ cannot handle orphaned sub-collections. Due to performance reasons `ref.listDocuments()` cannot be used, instead we must use a _cursor_ which does not return empty docs. `listDocuments()` can take upto 15 minutes to respond when your collection is over 100.000 documents, where as a cursor can handle each document really fast.

## Keywords

Firebase, Firestore, Export, Import, Data

## License

```text
The MIT License

Copyright (c) 2019 David Hardy
Copyright (c) 2019 codecentric nl

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```