# EIP Cloud Services Module ## Summary The EIP Cloud Services Module is a comprehensive Node.js package that provides seamless integration with various cloud services, including Redis, AWS S3, CDN, AWS Lambda, AWS SQS, and MySQL. This module is designed to simplify the complexities of interacting with these cloud services, offering a range of functionalities from data caching and storage to content delivery and serverless computing. The S3 service supports multipart browser upload orchestration through `createMultipartUpload`, `signMultipartUploadParts`, `completeMultipartUpload`, and `abortMultipartUpload`. Completion lists the uploaded parts from S3 so clients do not need access to response ETags. `s3.signDownload(key, bucket, { filename, expiresIn: 300 })` signs a private-object GET as an attachment without fetching the media. The caller must authorize access and resolve the immutable object key and bucket from trusted records. Filename must be a basename without path/control characters; Unicode names use the UTF-8 attachment parameter. Expiry is bounded to1–900 seconds and may end sooner when the signing credentials expire. The returned URL grants access to its bearer: return it only to the authorized caller and do not log or persist it. Signing does not confirm that the object exists. This local addition is not included in the previously frozen category-only1.8.0 release artifact; package consumption and live download acceptance remain pending. ## Installation and Import ### Revision-aware JSON work in progress `s3.readJsonRevision(key, bucket, { maxBytes })` returns `{ value, etag }` (or null for a missing key); `s3.writeJsonRevision(key, bucket, value, expectedETag)` conditionally replaces that object. The underlying `src/s3Revision.js` module pairs an unencrypted JSON object's bytes with its S3 ETag and performs a conditional JSON PUT using that ETag (or explicit null for create-only). It rejects missing conditions and verifies the serialized HTTP precondition before transport, protecting against older AWS serializers silently omitting it. Read bodies are bounded and closed; encoded/encrypted JSON is rejected rather than misread. Callers must inspect/read back uncertain acknowledgements and recompute changes after conflicts. The module does not coordinate secondary stores or ordinary legacy writers and does not migrate existing callers automatically. It is not included in the frozen1.8.0 category/download artifact; do not repack that release from current source without a new review. To use this module, first install it in your Node.js project. Then, import the required services as follows: ```javascript const { redis, cdn, mysql } = require('eip-cloud-services'); ``` ## Config Example Here is an example configuration for the EIP Cloud Services Module. Replace the placeholder values with your actual configuration details. ```javascript const packageJson = require('../package.json'); module.exports = { cdn: { ['cdnName']: { // e.g. "myCdn" ['envName1']: { //e.g. "production" type: 'google', urlMapName: 'YOUR_GCP_URL_MAP_NAME', projectId: 'YOUR_GCP_PROJECT_NAME', }, ['envName2']: { // e.g. "staging" type: 'amazon', distributionId: 'YOUR_DISTRIBUTION_ID', } }, }, redis: { // For regular redis connections port: 'YOUR_REDIS_PORT', host: 'YOUR_REDIS_HOST', prefix: 'YOUR_REDIS_PREFIX', }, redis: { // For redis cluster instances prefix: 'YOUR_REDIS_PREFIX', clusterEnabled: true, cluster: [ { port: 'YOUR_REDIS_PORT', host: 'YOUR_REDIS_HOST', }, { port: 'YOUR_REDIS_PORT_2', host: 'YOUR_REDIS_HOST_2', }, ... ] }, s3: { Bucket: 'YOUR_S3_BUCKET', logs: "verbose", // "verbose" or "outputs", any other value will not log. verbose will log all activity. outputs will only log when there is a file updated. logsFunction: ( message ) => { ... } // Optional, if nothing is provided console.log will be used. }, sqs: { region: 'eu-west-1', endpoint: 'http://localhost:4566' // Optional (e.g. localstack) }, mysql: { connectionLimit: 10, // Max connections host: 'my-database.domain.com', user: 'user', password: 'password', database: 'my-database', // defaults to undefined multipleStatements: false // defaults to true if not set } }; ``` ## Table of Contents 1. [Redis Module](#redis-module) 2. [AWS S3 Module](#aws-s3-module) 3. [CDN Module](#cdn-module) 4. [MySQL Module](#mysql-module) 5. [AWS Lambda Module](#aws-lambda-module) 6. [AWS SQS Module](#aws-sqs-module) 7. [AWS Secrets Manager Module](#aws-secrets-manager-module) 8. [OnDemand Module](#ondemand-module) # OnDemand Module ## Overview The OnDemand module provides dependency-free clients for Axel Springer's Content Ingest and Content Discovery APIs, plus streaming uploads to their Azure Blob Storage target. Staging endpoints are the defaults. Production Content Discovery/Ingest hosts are rejected unless a future caller explicitly supplies `allowProduction: true`. Secrets are caller-provided and are never stored by this package. Provider errors and returned metadata are sanitized so signed Azure query parameters are not included in thrown errors. ## Usage ```javascript const { ondemand } = require ( 'eip-cloud-services' ); const ingest = ondemand.createIngestClient ( { apiKey: process.env.ONDEMAND_INGEST_API_KEY } ); const discovery = ondemand.createDiscoveryClient ( { apiKey: process.env.ONDEMAND_DISCOVERY_API_KEY } ); const assets = await discovery.search ( { q: 'climate change', assetType: 'video', suppliers: [ 'https://cv.content-discovery.video/party/telegraph' ] } ); ``` Available Ingest operations are `createAsset`, `getAsset`, `updateAsset`, `startIngest`, `getIngest`, `createRendition`, and `updateRendition`. Discovery operations are `getVersion`, `getAsset`, `getAssets`, `search`, `searchByExpression`, and `getMasterDownload`. Use `uploadAzureBlob` with a clean blob URL and a separate SAS token. Pass the clean URL—not the signed upload URL—to `startIngest`: ```javascript await ondemand.uploadAzureBlob ( { blobUrl: cleanBlobUrl, sasToken: process.env.ONDEMAND_AZURE_SAS_TOKEN, body: videoStream, contentLength: videoSize, contentType: 'video/mp4' } ); await ingest.startIngest ( assetId, { url: cleanBlobUrl } ); ``` For large one-shot sources such as an S3 `GetObject` body, use the block uploader. It buffers only one block at a time, retries replayable blocks independently, and commits the blob only after the exact expected byte length has been received: ```javascript const source = await s3.getStream ( objectKey, bucket ); await ondemand.uploadAzureBlockBlob ( { blobUrl: cleanBlobUrl, sasToken: process.env.ONDEMAND_AZURE_SAS_TOKEN, body: source.body, contentLength: source.contentLength, contentType: 'video/mp4' } ); ``` # AWS Secrets Manager Module ## Overview This module provides helpers for reading and creating AWS Secrets Manager secrets. Secret reads use a local tmp-directory cache by default to avoid unnecessary AWS requests. Use `bypassCache` when the caller must force a fresh AWS lookup. ## Usage ```javascript const { secrets } = require('eip-cloud-services'); const appKey = await secrets.get('my-service/apple-news-app-key'); const configSecret = await secrets.get('my-service/config', { parseJson: true }); ``` ## Creating Secrets Existing secrets are never overwritten. ```javascript await secrets.set('my-service/apple-news-app-key', 'secret-value'); await secrets.set('my-service/config', { apiKey: 'secret-value' }); ``` CLI usage: ```bash npm run secrets:set -- --name=my-service/apple-news-app-key --value='secret-value' npm run secrets:set -- --name=my-service/config --json='{"apiKey":"secret-value"}' npm run secrets:set -- --name=my-service/apple-news-app-key --file=./secret.txt ``` If `--name` is omitted, the script uses the current `package.json` name. If no package name is available, it prompts for one. ## Cache Options By default `secrets.get()` checks a local tmp cache first and writes AWS results back to that cache. There is no TTL. To force AWS: ```javascript await secrets.get('my-service/apple-news-app-key', { bypassCache: true }); ``` Set `EIP_SECRETS_CACHE_DIR` or `secrets.cacheDir` to control cache location. # AWS Lambda Module ## Overview This module provides an interface for invoking AWS Lambda functions. It simplifies the process of triggering Lambda functions from your Node.js application, with support for both synchronous and asynchronous invocations. ## Installation Ensure this module is included in your Node.js project. Import the Lambda component as follows: ```javascript const lambda = require('eip-cloud-services/lambda'); ``` ## Usage ### Invoking a Lambda Function You can invoke a Lambda function by specifying the function name and the event payload. The module supports both waiting for the function execution to complete and fire-and-forget invocations. #### Synchronous Invocation (Wait for Execution to start) ```javascript const response = await lambda.invokeLambda('yourFunctionName', { key: 'value' }, true); console.log(response); ``` #### Asynchronous Invocation (Fire-and-Forget) ```javascript await lambda.invokeLambda('yourFunctionName', { key: 'value' }, false); ``` ### Parameters - `functionName`: The name of the Lambda function to invoke. - `eventPayload`: The payload to pass to the function. This can be any valid JSON object. - `waitForExecution` (optional): Set to `true` to wait for the execution to start and receive a response. - `context` (optional): The client context for the function execution. - `invocationType` (optional): Defaults to 'Event'. Can be set to 'RequestResponse' for synchronous execution. - `logType` (optional): The type of logging for the function. ## Configuration Configure the module with your AWS credentials and Lambda settings. ## Error Handling The module includes error handling to manage issues related to Lambda invocation, such as network errors or configuration problems. ## Logging Logging is provided to track the start and completion of Lambda invocations, aiding in debugging and monitoring. # AWS SQS Module ## Overview This module provides helpers for sending, receiving, and deleting messages in AWS SQS queues. ## Usage ```javascript const sqs = require('eip-cloud-services').sqs; const queueUrl = await sqs.getQueueUrl('my-queue'); await sqs.sendMessage(queueUrl, { type: 'example', payload: 'hello' }); const messages = await sqs.receiveMessages(queueUrl, { maxMessages: 5, waitTimeSeconds: 10 }); if (messages.length > 0) { await sqs.deleteMessages(queueUrl, messages); } ``` ## Configuration Set `sqs.region` and optionally `sqs.endpoint` in your config. The module defaults to `eu-west-1` and uses the AWS SDK credential chain. # MySQL Module ## Overview This MySQL module provides a simple and efficient interface for interacting with MySQL databases. It includes functionalities to manage database connections, execute queries, and handle connection pooling. ## Installation Ensure this module is included in your Node.js project. Import the MySQL component as follows: ```javascript const mysql = require('eip-cloud-services/mysql'); ``` ## Usage ### Executing Queries You can execute queries using the `query` method. This method automatically handles connections and releases them after query execution. ```javascript const results = await mysql.query('SELECT * FROM your_table'); console.log(results); ``` ### Manually Managing Database Connections It's not required for you to get connections manually, you can use the methods directly without having to setup connections. Getting a connection should only be used if you need to parse the connection to a third party module for example. #### Get a Connection Pool ```javascript const pool = mysql.getPool(); ``` #### Get a Single Connection ```javascript const connection = await mysql.getConnection(); ``` ### Closing the Connection Pool To gracefully close all connections in the pool: ```javascript await mysql.kill(); ``` ## Configuration Configure your MySQL connection settings (like host, user, password, etc.) in the `config` directory. ## Error Handling The module is designed to handle errors gracefully, providing clear error messages for database connection issues and query errors. ## Pooling - The module uses connection pooling for efficient database interaction. - The pool is automatically created and managed by the module. # CDN Module ## Overview This module provides a shared client for CDN invalidation requests. CDN invalidation requests are sent through the shared Tools API invalidate endpoint, so callers use one centralized route while the backend implementation remains independently updateable inside the invalidation service. ## Installation Ensure this module is included in your Node.js project. Import the CDN component as follows: ```javascript const cdn = require('eip-cloud-services/cdn'); ``` ## Usage ### Create a CDN Invalidation To invalidate cached content in a CDN, use the `createInvalidation` method. This sends the request to the shared Tools API invalidate endpoint, which executes the configured provider-specific invalidation in `eip-cdn-invalidator`. #### Invalidate in Amazon CloudFront ```javascript await cdn.createInvalidation('amazon', 'path/to/your/file.jpg', 'production'); ``` #### Invalidate in Google Cloud CDN ```javascript await cdn.createInvalidation('google', 'path/to/your/file.jpg', 'production'); ``` You can also invalidate multiple paths by passing an array of keys: ```javascript await cdn.createInvalidation('amazon', ['file1.jpg', 'file2.jpg'], 'staging'); ``` ## Configuration Configure the module with your CDN settings in the `config` directory. The configuration should include details like project ID, distribution ID, and URL map names for the respective CDNs. ## Error Handling The module is equipped to handle errors gracefully, including validation of input parameters and handling CDN-specific errors. ## Advanced Features - Validates the key argument to ensure proper formatting. - Constructs invalidation paths based on the provided keys. - Centralizes invalidation routing through the shared Tools API endpoint. # AWS S3 Module ## Overview This module provides an interface for interacting with AWS S3, offering functionalities like object retrieval, storage, deletion, and more. It supports advanced features such as encryption and cache control directives, making it a versatile tool for managing S3 operations efficiently. ## Installation Ensure this module is included in your Node.js project. Import the S3 component as follows: ```javascript const s3 = require('eip-cloud-services/s3'); ``` ## Usage ### Check if an Object Exists ```javascript const exists = await s3.exists('myObjectKey'); console.log(exists); // true if exists, false otherwise ``` ### Retrieve an Object ```javascript const object = await s3.get('myObjectKey'); console.log(object); ``` ### Store an Object ```javascript await s3.set('myObjectKey', 'myObjectData', { /* options */ }); ``` ### Delete an Object ```javascript await s3.del('myObjectKey'); ``` ### Copy an Object ```javascript await s3.copy('sourceObjectKey', 'destinationObjectKey'); ``` For objects that may exceed S3's 5 GiB atomic-copy limit, use the large-object copy helper. It automatically switches to multipart server-side copy and does not download the object through the caller: ```javascript await s3.copyLargeObject( 'temporary/video.mp4', 'permanent/video.mp4', 'source-bucket', 'destination-bucket', { contentType: 'video/mp4' } ); ``` ### Seekable Local Access to a Private Large Object Media tools such as FFmpeg often require byte-range seeking. `withLocalReadUrl` provides a callback-scoped loopback URL and translates HTTP range requests into authenticated S3 reads. The source is never written to local disk: ```javascript const metadata = await s3.withLocalReadUrl('temporary/video.mp4', 'source-bucket', async url => { return inspectVideo(url); }); ``` The URL is bound to `127.0.0.1`, contains an unguessable path, and is closed as soon as the callback settles. ### Move an Object ```javascript await s3.move('sourceObjectKey', 'destinationObjectKey'); ``` ### Managing Cache-Control and Encryption The module provides detailed control over cache behavior and supports encryption for stored objects, allowing you to optimize performance and security. ## Configuration Configure the module with your AWS credentials and preferred settings in the `config` directory. ## Error Handling The module includes comprehensive error handling, ensuring robust interaction with AWS S3 even in complex scenarios. # Redis Module ## Overview This Redis module provides a robust interface for interacting with Redis, supporting both standard Redis operations and advanced features like Redis Clusters. The module is designed to manage multiple Redis client instances efficiently, allowing seamless operation with different Redis configurations. ## Installation To use this Redis module, first ensure that it is included in your Node.js project. You can import the module as follows: ```javascript const redis = require('eip-cloud-services/redis'); ``` ## Usage ### Standard Redis Operations You can perform standard Redis operations like setting, getting, deleting keys, etc. Here are some examples: #### Set a Key ```javascript await redis.set('myKey', 'myValue'); ``` #### Get a Key ```javascript const value = await redis.get('myKey'); console.log(value); // Outputs: myValue ``` #### Delete a Key ```javascript await redis.del('myKey'); ``` ### Advanced Operations #### Using Redis Cluster If your Redis setup includes a cluster, the module automatically configures the client for cluster operations based on your configuration. #### Subscribing to a Channel ```javascript redis.subscribe('myChannel', (message) => { console.log(`Received: ${message}`); }); ``` #### Publishing to a Channel ```javascript await redis.publish('myChannel', 'Hello, World!'); ``` ### Managing Redis Clients The module internally manages multiple Redis clients for different purposes (like separate clients for pub/sub). However, this is abstracted away from the standard use of the module. ## Error Handling The module is designed to gracefully handle Redis errors, including connection issues and cluster errors. ## Editorial category lookup `discovery.searchCategories('bild' | 'welt', { q, limit })` uses the configured Discovery profile and API key. It returns IPTC AboutCvTerms from public `/cvterms/bild` or `/cvterms/welt` routes. `q` is optional (up to50 characters); limit defaults20 and is bounded1–50. The returned EditorialOne URIs identify vocabulary terms; callers need not connect to those private URIs. See https://docs.content-discovery.video/api-reference/categories. This additive operation requires a new package release before application deployment; the existing1.7.0 package does not include it. ### Conditional JSON revisions (1.8.0 candidate) `s3.readJsonRevision(key, bucket, { maxBytes })` returns the JSON value and ETag from the same S3 GET. `s3.writeJsonRevision(key, bucket, value, expectedETag)` requires that exact quoted ETag, or explicit null for create-only writes. The helper verifies that the SDK actually serialized the precondition before transport. Missing/uncertain acknowledgements require caller readback; there is no unconditional fallback. S3 client 3.715.0 is the tested minimum. These primitives do not update Redis, MySQL or search mirrors; callers own field merge/conflict policy and scheduled reconciliation.