@pluto-tv/service-dal-grpc-client
# Table of Contents
* [About the Project](#about-the-project)
* [Software Development Lifecycle](#software-development-lifecycle)
* [Changelog](#changelog)
* [NodeJS gRPC Client Publishing](#nodejs-grpc-client-publishing)
* [Understanding Generated Code](#understanding-generated-code)
* [Usage](#usage)
* [Understanding Helpers](#understanding-helpers)
* [convertResponseToDBStructure()](#convertresponsetodbstructure)
* [Mock Constructors](#mock-constructors)
* [Understanding Create API Data Cleanup](#understanding-create-api-data-cleanup)
* [Request Timeouts](#request-timeouts)
* [Setting the deadline to both client and server in general](#setting-the-deadline-to-both-client-and-server-in-general)
* [Setting the deadline per method invocation](#setting-the-deadline-per-method-invocation)
* [Only server side](#only-server-side)
* [Request Retry](#request-retry)
* [Contacts](#contacts)
# About The Project
Code located in the `services` directory is automatically generated based on protobuf files in `service-dal/services`.
See the `service-dal/package.json` for details on _how_ this is generated.
Google has [great resources](https://developers.google.com/protocol-buffers/docs/reference/javascript-generated) on using the generated code.
## Software Development Lifecycle
### Changelog
The changelog muse be maintained for this client separately from the DAL service. Some service updates don't affect the client and vice versa.
Each newly added feature, bugfix, and hotfix for the client should be added to [CHANGELOG.md](./CHANGELOG.md) under the # New Version heading.
### NodeJS gRPC Client Publishing
The NodeJS gRPC client will be published using a GitHub action workflow when a tag is created conforming to the following pattern. Tags are added by creating a GitHub [release](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository#creating-a-release) for the NodeJS Client.
A summary of the changelog in the release must be supplied even for prerelease versions. Non-prerelease versions must have a summary of all the changes since the last non-prerelease.
- `@plutotv/service-dal-grpc-client@{major}.{minor}.{patch}{prerelease?}`
## Understanding Generated Code
- [JS Generated Code](https://developers.google.com/protocol-buffers/docs/reference/javascript-generated)
- Case-conversion for JS generated code works as follows:
1) The first letter in the method name is always lower-case.
2) If the field name contains an underscore, the underscore is removed, and the following letter is capitalized.
> :warning: Note that the generated accessors always use camel-case naming, even if the field name in the .proto file uses lower-case with underscores ([as it should](https://developers.google.com/protocol-buffers/docs/style)).
The proto field `foo_bar_baz` has, for example, a `getFooBarBaz()` method.
- [Data Types and Default Values](https://developers.google.com/protocol-buffers/docs/proto3#default):
When a message is parsed, if the encoded message does not contain a particular singular element, the corresponding field in the parsed object is set to the default value for that field. These defaults are type-specific:
- For `strings`, the default value is the `empty string`.
- For `bytes`, the default value is `empty bytes`.
- For `bools`, the default value is `false`.
- For `numeric types`, the default value is `zero`.
- For `enums`, the default value is the `first defined enum value`, which must be 0.
- For `message` fields, the field is `not set`. Its exact value is language-dependent. See the [generated code guide](https://developers.google.com/protocol-buffers/docs/reference/overview) for details.
## Usage
A function called `promisifyGrpcClientFunction` exists to "promisify" the NodeJS gRPC Unary and Server Streaming functions.
This ensures a common interface whether the type of gRPC function.
```javascript
const dal = require('@pluto-tv/service-dal-grpc-client');
const client = createPromisifiedClient(
'0.0.0.0:50051',
dal.clips.services.ClipManagerClient,
dal.grpc.credentials.createInsecure(),
null,
{ metadata: { app_name: 'my_app_name', auth_token: 'CLIENT_AUTH_TOKEN' } },
);
// Unary example
const getClip = async (clipId) => {
const request = new dal.clips.messages.GetByIdRequest();
request.setId(clipId);
return promisifyGrpcClientFunction(client, 'getById', request, console);
};
// Server streaming example
const getAllClipIds = async () => {
const request = new dal.clips.messages.CFeedsSyncAllIdsRequest();
return promisifyGrpcClientFunction(client, 'cFeedsSyncAllIds', request);
};
```
## Understanding Helpers
### `convertResponseToDBStructure()`
> :warning: If using the generated methods to access an individual attribute value (ie, `gRPCResponse.getId()`, etc.), the use of this helper is **NOT** necessary.
Some attributes of collections in `pluto-main` are not properly formed `camelCase` (eg., `"ns_st_ce"`, `"screenshot16_9"`, `"SVOD"`, etc.).
All attributes are described in `protobuf` in `snake_case` following formatting standards for `protobufs`.
If you attempt to use the Response Object and transform it directly to a native JS object (`.toObject()`), you will need to manually recover these *irregular* attribute names.
All are restored to camelCase, however, the few that are irregular need to be restored manually.
The helper `convertResponseToDBStructure` has a defined set of keys that we match on and replace with their original, *irregular* values.
Use:
```
const convertedResponse = convertResponseToDBStructure(gPRCResponse);
```
Ideally this would be done automatically for the client in some sort of `interceptor` stage, but for now all responses **MUST** be passed through `convertResponseToDBStructure`.
### Mock Constructors
Unit tests in services/lambdas integrating with the DAL will need to mock DAL methods to retrieve 'fixture data'
where previously loaded to a Mongo docker image for tests. Tools are included with the DAL to assist in mocking methods
to return fixture data in DAL responses. These are located in `helpers/mock-constructors`. This contains functions to
create the data arrays needed by the DAL gRPC request messages.
## Understanding Create API Data Cleanup
Because protobuf files are strictly typed and there are no `null` defined in protobufs, object communicated over gRPC may have many attributes with values that are empty strings (`""`) or empty arrays (`[]`).
This data is processed through a [function](./../helpers/generic-manager-methods.js) (`cleanUpForWrite`) that strips out all attributes that are empty strings or empty lists.
## Request Timeouts
In gRPC, timeouts are referred to as `deadline`. By default, there is no deadline for a request to the server.
A maximum deadline can be set to serverside separately or to the server side and client side
### Setting the deadline to both client and server in general
To set a timeout for client in general:
```javascript
const options = {
'grpc.service_config': JSON.stringify({
methodConfig: [
{
name: [
{ service: 'channelpackage.ChannelManager', method: 'GetTimelinesForActiveContent' }
],
timeout: '1.0s'
}
]
})
};
client = createPromisifiedClient(
'0.0.0.0:50051',
clips.services.ClipManagerClient,
null,
options,
{ metadata: { app_name: 'my_app_name' } },
);
```
**methodConfig** - list of rules for timeouts for different methods
**timeout** - timeout applied to matching rule method, string as "1.000000001s" recognizes a number with up to 9 digits after the decimal point, followed by an "s", representing a number of seconds.
Or it can represent of object { seconds: 1, nanos: 1 } which is equal to "1.000000001s"
**name.service** - is the name of service to apply the rule and consists of package name and service name from proto file, "channelpackage.ChannelManager" e.g.:
```
syntax = "proto3";
package channelpackage;
service ChannelManager {
```
**name.method** - is the name of the service method to apply rule to, taken from the proto file "GetTimelinesForActiveContent", e.g.:
```
rpc GetTimelinesForActiveContent(GetTimelinesForActiveContentRequest) returns (GetTimelinesForActiveContentResponse) {}
```
If the method name is not provided then the rule is applied to the all methods in service.
### Setting the deadline per method invocation
The deadline can be set directly to the method call and it is provided as an absolute time im milliseconds when it needs to be timed out
In following example the timeout is set to 1 second
```javascript
channelClient.getTimelinesForActiveContent(request, null, { deadline: new Date(new Date().getTime() + 1000) }).then((response) => {
// ToDo
});
```
In the way above the client will also pass the server timeout in metadata (`{ 'grpc-timeout': '1000m' }`) automatically and will have a timeout set on the client side automatically.
### Only server side
Server can be communicated a timeout added to the request by passing a specific object (`{ 'grpc-timeout': '1000m' }`) as the second parameter to the client function.
`m` represents milliseconds and needs to be part of the value. `1000m` would be 1000 milliseconds or 1 second.
Below is an example of its usage in a unary request `getById`.
```javascript
const {
clips,
createPromisifiedClient,
convertResponseToDBStructure,
} = require('@pluto-tv/service-dal-grpc-client');
client = createPromisifiedClient(
'0.0.0.0:50051',
clips.services.ClipManagerClient,
null,
null,
{ metadata: { app_name: 'my_app_name', auth_token: 'CLIENT_AUTH_TOKEN' } },
);
const request = new clips.messages.GetByIdRequest();
request.setId('5b585b7b43bb69b706a06a30');
const gprcDoc = client.getById(request, { 'grpc-timeout': '1000m' })
.then((response) => convertResponseToDBStructure(response).document);
```
> :warning: `auth_token` matching the `auth_token` on the server is required for any method that write or delete.
There is also a way to set timeout for all the methods in the client:
```javascript
client = createPromisifiedClient(
'0.0.0.0:50051',
clips.services.ClipManagerClient,
null,
null,
{ metadata: { app_name: 'my_app_name', 'grpc-timeout': '1000m', auth_token: 'CLIENT_AUTH_TOKEN' } },
);
```
## Request Retry
In current implementation the grpc-js library is not yet having implemented the retry policy and in
the caller wrapper which promisifies the gRPC client 'async' npm module is used and the retry
functionality is used from that library.
There is a way to configure the retry count,
interval and a function to filter Errors to retry on or skip retrying.
**times** - retry count setting;
**interval** - interval in milliseconds between retries;
**errorFilter** - function to filter retries, return true to retry on all errors. At the same time this function is used to log any error on every retry;
```javascript
client = createPromisifiedClient(
'0.0.0.0:50051',
clips.services.ClipManagerClient,
null,
null,
{
options: {
retry: {
times: 7,
interval: 500,
errorFilter: (err) => {
// log your error here isth your logger, example : log.action('clipClientMethodInvocationError').error(err);
return true;
}
}
},
metadata: { app_name: 'my_app_name', auth_token: 'CLIENT_AUTH_TOKEN' }
},
);
```
# Contacts
- Pavlo Kalmykov - pkalmykov@pluto.tv
- Jonathan Jesperson - jonathan.jesperson@pluto.tv