import EventController from '../eventcontroller/EventController'; import Logger from '../logger/Logger'; import VideoFrameBuffer from '../videoframeprocessor/VideoFrameBuffer'; import VideoFrameProcessor from '../videoframeprocessor/VideoFrameProcessor'; import VideoFxConfig from './VideoFxConfig'; import VideoFxSpec from './VideoFxSpec'; /** * [[VideoFxProcessor]] Mechanism that drives the data transformation * of individual video frames to apply ML-based background blur and * background replacement effects on the video stream. */ export default class VideoFxProcessor implements VideoFrameProcessor { private logger; private static isSharedArrayBufferSupported; private fxLibScript; private effectConfig; private streamParameters; private engineWorker; private canvasOpsManager; private renderer; private sharedImageBuffer; private sharedImageData; private buildEnginePromise; private destroyedAssetsPromise; private segmentationRateManager; private segmentationMask; private segmentationRequestPromise; private outputCanvas; private canvasVideoFrameBuffer; private eventController?; private videoFxSpec; /** * Initializes a new instance of [[VideoFxProcessor]] with a default NoOp [[VideoFxConfig]]. * @param logger * @param processingBudgetPerFrame throttling constraint for processing * @param eventController EventController object to manage events */ constructor(logger: Logger, processingBudgetPerFrame?: number, eventController?: EventController); /** * Apply the [[VideoFxProcessor]]'s specialized effect onto the frame contained by * the buffer parameter. * @param buffers * @returns buffers */ process(buffers: VideoFrameBuffer[]): Promise; /** * Process an input stream and apply the visual effects specified in this effectConfig. * @param buffers the input stream * @returns an output stream of VideoFrameBuffer */ private fxProcess; /** * Perform a process call that just returns the input stream * @param buffers the input stream * @returns an output stream of VideoFrameBuffer */ private noOpProcess; /** * Check if the stream dimensions/parameters have changed. * @param stream frame from current video stream about to be processed * @returns boolean representing if the stream parameters changed */ private didStreamParametersChange; /** * Adjust the videoFxProcessor to handle new stream parameters. * @param stream frame from current video stream about to be processed */ private adjustProcessorForNewStreamParameters; /** * Clean up any excess memory that has been allocated by [[VideoFxProcessor]]. */ destroy(): Promise; /** * Manage the segmentation mask that is used to generate a final frame. This function * will calculate whether or not to re-generate a segmenation and also monitor * how computationally expensive the rate of segmentation are. * @param inputCanvas canvas to be used for segmentation */ private manageSegmentationMask; /** * Generate a segmentation mask from the input frame. * @param inferenceImageData * @returns image data representing segmenation mask */ private generateSegmentationMask; /** * Make a deep copy of the VideoFxConfig passed in. Needs to be updated if additional * fields are added to VideoFxConfig. * @param effectConfig * @returns newEffectConfig */ private cloneConfigFrom; setVideoFxSpec(videoFxSpec: VideoFxSpec): void; /** * Update the [[VideoFxProcessor]] to apply a new set of effects by updating the instance property * [[VideoFxConfig]]. If the effectConfig parameter fails validation, an error is thrown and there is * no update. * @param effectConfig updated [[VideoFxConfig]] with new video effects */ setEffectConfig(effectConfig: VideoFxConfig): Promise; /** * Confirm that the config consists of valid values. * @param config that will be validated */ private validateEffectConfig; /** * Confirm that the config consists of valid values for background replacement. * @param config that will be validated for background replacement */ private validateReplacementConfig; /** * Confirm that the processingBudgePerFrame constraint holds valid values. * @param constraint to be validated */ private validateProcessingBudgetPerFrame; /** * Loads all of the assets that are associated with the stored effectConfig. */ private loadAssets; /** * Determines the current set of specifications that define the current SDK version. * @returns an [[VideoFxAssetParams]] object defining the parameters of the * current SDK */ private getVideoFxAssetParams; /** * Generate a final path to an asset belonging to current version of SDK. * @param basePath Base of the path that will be used to generate a final path * @param videoFxAssetParams Parameters of the current SDK version * @returns A final path specific to an asset belonging to current SDK * version */ private getPathFromVideoFxAssetParams; /** * Fetch and then load the engine worker into a web-worker. */ private loadEngineWorker; /** * Build the videoFxEngine. * @param videoFxAssetParams [[VideoFxAssetParams]] defining current version of SDK */ private buildEngine; /** * Loads the Video Fx library to drive video post-processing, given the versioning * of the SDK worker from the VideoFxAssetParams interface passed as a parameter. * @param videoFxAssetParams */ private loadFxLib; /** * Getter for the current [[VideoFxConfig]] maintained as an instance property of * [[VideoFxProcessor]]. * @returns currentEffectConfig */ getEffectConfig(): VideoFxConfig; /** * Receives messages from the engine worker and then delegates * the proper response to a different function. * @param event notification to be received from the engine worker */ private engineWorkerReceiver; /** * Handle the message returned from the VideoFX Engine worker to * settle (resolve/reject) the promise associated with initializing the engine. * @param buildStatus status of the final result of building the engine */ private settleEngineBuildPromise; /** * Handle the received segmentation mask and then resolve/reject the promise to notify * the main processing function to prepare for the next frame (using ImageData transfer). * @param segmentationMask */ private settleSegmentationPromise; /** * Handle the received segmentation mask and then resolve/reject the promise to notify * the main processing function to prepare for the next frame (using shared array buffer). */ private settleSegmentationPromiseSAB; /** * Function that will convert the VideoFxProcessor into a NoOp version of itself. * Will only return input video stream and not use worker, engine, or segmentation * model. */ private setToNoOpProcess; /** * Detect client environment to determine if the [[VideoFxProcessor]] is supported. * @param logger to record/report events of checking processor support * @param attemptAssetLoad will also fetch and build all relevant components of the * processor to ensure end to end feature is supported * @returns a boolean promise that will resolve to true if supported and false if not */ static isSupported(logger?: Logger, attemptAssetLoad?: boolean): Promise; /** * Create a [[VideoFxProcessor]] that has loaded its necessary components, ready to instantly * process a video stream with the effects specified in the passed [[VideoFxConfig]]. * * ** NOTICE ** * * Amazon Chime background blur 2.0 and background replacement 2.0 * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * By installing or using this package, you agree to the AWS Customer Agreement, AWS Service Terms, and AWS Privacy Notice. * If you already have an AWS Customer Agreement, you agree that the terms of that agreement govern your download and use of this package. * This package is provided as AWS Content and subject to the AWS Customer agreement and any other agreement with AWS governing your use of * AWS services. */ static create(logger: Logger, effectConfig: VideoFxConfig, processingBudgetPerFrame?: number, spec?: VideoFxSpec): Promise; private publishVideoFxConfigEvent; private sameVideoFxConfig; }