# Data solutions framework on AWS Data solutions framework on AWS (DSF on AWS) is a framework for implementation and delivery of data solutions with built-in AWS best practices. DSF on AWS is an abstraction atop AWS services based on [AWS Cloud Development Kit](https://aws.amazon.com/cdk/) (CDK) L3 constructs, packaged as a library. ➡️ **More information on our [website](https://awslabs.github.io/data-solutions-framework-on-aws)** # API Reference ## Constructs ### AccessLogsBucket Amazon S3 Bucket configured with best-practices and smart defaults for storing S3 access logs. Default bucket name is `accesslogs---` > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Storage/access-logs-bucket](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Storage/access-logs-bucket) *Example* ```typescript const bucket = new dsf.storage.AccessLogsBucket(this, 'AccessLogsBucket') ``` #### Initializers ```typescript import { storage } from 'aws-dsf' new storage.AccessLogsBucket(scope: Construct, id: string, props?: BucketProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-cdk-lib.aws_s3.BucketProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Optional - *Type:* aws-cdk-lib.aws_s3.BucketProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | applyRemovalPolicy | Apply the given removal policy to this resource. | | addEventNotification | Adds a bucket notification event destination. | | addObjectCreatedNotification | Subscribes a destination to receive notifications when an object is created in the bucket. | | addObjectRemovedNotification | Subscribes a destination to receive notifications when an object is removed from the bucket. | | addToResourcePolicy | Adds a statement to the resource policy for a principal (i.e. account/role/service) to perform actions on this bucket and/or its contents. Use `bucketArn` and `arnForObjects(keys)` to obtain ARNs for this bucket or objects. | | arnForObjects | Returns an ARN that represents all objects within the bucket that match the key pattern specified. | | enableEventBridgeNotification | Enables event bridge notification, causing all events below to be sent to EventBridge:. | | grantDelete | Grants s3:DeleteObject* permission to an IAM principal for objects in this bucket. | | grantPublicAccess | Allows unrestricted access to objects from this bucket. | | grantPut | Grants s3:PutObject* and s3:Abort* permissions for this bucket to an IAM principal. | | grantPutAcl | Grant the given IAM identity permissions to modify the ACLs of objects in the given Bucket. | | grantRead | Grant read permissions for this bucket and it's contents to an IAM principal (Role/Group/User). | | grantReadWrite | Grants read/write permissions for this bucket and it's contents to an IAM principal (Role/Group/User). | | grantWrite | Grant write permissions to this bucket to an IAM principal. | | onCloudTrailEvent | Define a CloudWatch event that triggers when something happens to this repository. | | onCloudTrailPutObject | Defines an AWS CloudWatch event that triggers when an object is uploaded to the specified paths (keys) in this bucket using the PutObject API call. | | onCloudTrailWriteObject | Defines an AWS CloudWatch event that triggers when an object at the specified paths (keys) in this bucket are written to. | | s3UrlForObject | The S3 URL of an S3 object. For example:. | | transferAccelerationUrlForObject | The https Transfer Acceleration URL of an S3 object. | | urlForObject | The https URL of an S3 object. Specify `regional: false` at the options for non-regional URLs. For example:. | | virtualHostedUrlForObject | The virtual hosted-style URL of an S3 object. Specify `regional: false` at the options for non-regional URL. For example:. | | addCorsRule | Adds a cross-origin access configuration for objects in an Amazon S3 bucket. | | addInventory | Add an inventory configuration. | | addLifecycleRule | Add a lifecycle rule to the bucket. | | addMetric | Adds a metrics configuration for the CloudWatch request metrics from the bucket. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `applyRemovalPolicy` ```typescript public applyRemovalPolicy(policy: RemovalPolicy): void ``` Apply the given removal policy to this resource. The Removal Policy controls what happens to this resource when it stops being managed by CloudFormation, either because you've removed it from the CDK application or because you've made a change that requires the resource to be replaced. The resource can be deleted (`RemovalPolicy.DESTROY`), or left in your AWS account for data recovery and cleanup later (`RemovalPolicy.RETAIN`). ###### `policy`^Required - *Type:* aws-cdk-lib.RemovalPolicy --- ##### `addEventNotification` ```typescript public addEventNotification(event: EventType, dest: IBucketNotificationDestination, filters: NotificationKeyFilter): void ``` Adds a bucket notification event destination. > [https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html](https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) *Example* ```typescript declare const myLambda: lambda.Function; const bucket = new s3.Bucket(this, 'MyBucket'); bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(myLambda), {prefix: 'home/myusername/*'}); ``` ###### `event`^Required - *Type:* aws-cdk-lib.aws_s3.EventType The event to trigger the notification. --- ###### `dest`^Required - *Type:* aws-cdk-lib.aws_s3.IBucketNotificationDestination The notification destination (Lambda, SNS Topic or SQS Queue). --- ###### `filters`^Required - *Type:* aws-cdk-lib.aws_s3.NotificationKeyFilter S3 object key filter rules to determine which objects trigger this event. Each filter must include a `prefix` and/or `suffix` that will be matched against the s3 object key. Refer to the S3 Developer Guide for details about allowed filter rules. --- ##### `addObjectCreatedNotification` ```typescript public addObjectCreatedNotification(dest: IBucketNotificationDestination, filters: NotificationKeyFilter): void ``` Subscribes a destination to receive notifications when an object is created in the bucket. This is identical to calling `onEvent(EventType.OBJECT_CREATED)`. ###### `dest`^Required - *Type:* aws-cdk-lib.aws_s3.IBucketNotificationDestination The notification destination (see onEvent). --- ###### `filters`^Required - *Type:* aws-cdk-lib.aws_s3.NotificationKeyFilter Filters (see onEvent). --- ##### `addObjectRemovedNotification` ```typescript public addObjectRemovedNotification(dest: IBucketNotificationDestination, filters: NotificationKeyFilter): void ``` Subscribes a destination to receive notifications when an object is removed from the bucket. This is identical to calling `onEvent(EventType.OBJECT_REMOVED)`. ###### `dest`^Required - *Type:* aws-cdk-lib.aws_s3.IBucketNotificationDestination The notification destination (see onEvent). --- ###### `filters`^Required - *Type:* aws-cdk-lib.aws_s3.NotificationKeyFilter Filters (see onEvent). --- ##### `addToResourcePolicy` ```typescript public addToResourcePolicy(permission: PolicyStatement): AddToResourcePolicyResult ``` Adds a statement to the resource policy for a principal (i.e. account/role/service) to perform actions on this bucket and/or its contents. Use `bucketArn` and `arnForObjects(keys)` to obtain ARNs for this bucket or objects. Note that the policy statement may or may not be added to the policy. For example, when an `IBucket` is created from an existing bucket, it's not possible to tell whether the bucket already has a policy attached, let alone to re-use that policy to add more statements to it. So it's safest to do nothing in these cases. ###### `permission`^Required - *Type:* aws-cdk-lib.aws_iam.PolicyStatement the policy statement to be added to the bucket's policy. --- ##### `arnForObjects` ```typescript public arnForObjects(keyPattern: string): string ``` Returns an ARN that represents all objects within the bucket that match the key pattern specified. To represent all keys, specify ``"*"``. If you need to specify a keyPattern with multiple components, concatenate them into a single string, e.g.: arnForObjects(`home/${team}/${user}/*`) ###### `keyPattern`^Required - *Type:* string --- ##### `enableEventBridgeNotification` ```typescript public enableEventBridgeNotification(): void ``` Enables event bridge notification, causing all events below to be sent to EventBridge:. Object Deleted (DeleteObject) - Object Deleted (Lifecycle expiration) - Object Restore Initiated - Object Restore Completed - Object Restore Expired - Object Storage Class Changed - Object Access Tier Changed - Object ACL Updated - Object Tags Added - Object Tags Deleted ##### `grantDelete` ```typescript public grantDelete(identity: IGrantable, objectsKeyPattern?: any): Grant ``` Grants s3:DeleteObject* permission to an IAM principal for objects in this bucket. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable The principal. --- ###### `objectsKeyPattern`^Optional - *Type:* any Restrict the permission to a certain key pattern (default '*'). Parameter type is `any` but `string` should be passed in. --- ##### `grantPublicAccess` ```typescript public grantPublicAccess(allowedActions: string, keyPrefix?: string): Grant ``` Allows unrestricted access to objects from this bucket. IMPORTANT: This permission allows anyone to perform actions on S3 objects in this bucket, which is useful for when you configure your bucket as a website and want everyone to be able to read objects in the bucket without needing to authenticate. Without arguments, this method will grant read ("s3:GetObject") access to all objects ("*") in the bucket. The method returns the `iam.Grant` object, which can then be modified as needed. For example, you can add a condition that will restrict access only to an IPv4 range like this: const grant = bucket.grantPublicAccess(); grant.resourceStatement!.addCondition(‘IpAddress’, { “aws:SourceIp”: “54.240.143.0/24” }); Note that if this `IBucket` refers to an existing bucket, possibly not managed by CloudFormation, this method will have no effect, since it's impossible to modify the policy of an existing bucket. ###### `allowedActions`^Required - *Type:* string the set of S3 actions to allow. Default is "s3:GetObject". --- ###### `keyPrefix`^Optional - *Type:* string the prefix of S3 object keys (e.g. `home/*`). Default is "*". --- ##### `grantPut` ```typescript public grantPut(identity: IGrantable, objectsKeyPattern?: any): Grant ``` Grants s3:PutObject* and s3:Abort* permissions for this bucket to an IAM principal. If encryption is used, permission to use the key to encrypt the contents of written files will also be granted to the same principal. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable The principal. --- ###### `objectsKeyPattern`^Optional - *Type:* any Restrict the permission to a certain key pattern (default '*'). Parameter type is `any` but `string` should be passed in. --- ##### `grantPutAcl` ```typescript public grantPutAcl(identity: IGrantable, objectsKeyPattern?: string): Grant ``` Grant the given IAM identity permissions to modify the ACLs of objects in the given Bucket. If your application has the '@aws-cdk/aws-s3:grantWriteWithoutAcl' feature flag set, calling `grantWrite` or `grantReadWrite` no longer grants permissions to modify the ACLs of the objects; in this case, if you need to modify object ACLs, call this method explicitly. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable --- ###### `objectsKeyPattern`^Optional - *Type:* string --- ##### `grantRead` ```typescript public grantRead(identity: IGrantable, objectsKeyPattern?: any): Grant ``` Grant read permissions for this bucket and it's contents to an IAM principal (Role/Group/User). If encryption is used, permission to use the key to decrypt the contents of the bucket will also be granted to the same principal. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable The principal. --- ###### `objectsKeyPattern`^Optional - *Type:* any Restrict the permission to a certain key pattern (default '*'). Parameter type is `any` but `string` should be passed in. --- ##### `grantReadWrite` ```typescript public grantReadWrite(identity: IGrantable, objectsKeyPattern?: any): Grant ``` Grants read/write permissions for this bucket and it's contents to an IAM principal (Role/Group/User). If an encryption key is used, permission to use the key for encrypt/decrypt will also be granted. Before CDK version 1.85.0, this method granted the `s3:PutObject*` permission that included `s3:PutObjectAcl`, which could be used to grant read/write object access to IAM principals in other accounts. If you want to get rid of that behavior, update your CDK version to 1.85.0 or later, and make sure the `@aws-cdk/aws-s3:grantWriteWithoutAcl` feature flag is set to `true` in the `context` key of your cdk.json file. If you've already updated, but still need the principal to have permissions to modify the ACLs, use the `grantPutAcl` method. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable --- ###### `objectsKeyPattern`^Optional - *Type:* any --- ##### `grantWrite` ```typescript public grantWrite(identity: IGrantable, objectsKeyPattern?: any, allowedActionPatterns?: string[]): Grant ``` Grant write permissions to this bucket to an IAM principal. If encryption is used, permission to use the key to encrypt the contents of written files will also be granted to the same principal. Before CDK version 1.85.0, this method granted the `s3:PutObject*` permission that included `s3:PutObjectAcl`, which could be used to grant read/write object access to IAM principals in other accounts. If you want to get rid of that behavior, update your CDK version to 1.85.0 or later, and make sure the `@aws-cdk/aws-s3:grantWriteWithoutAcl` feature flag is set to `true` in the `context` key of your cdk.json file. If you've already updated, but still need the principal to have permissions to modify the ACLs, use the `grantPutAcl` method. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable --- ###### `objectsKeyPattern`^Optional - *Type:* any --- ###### `allowedActionPatterns`^Optional - *Type:* string[] --- ##### `onCloudTrailEvent` ```typescript public onCloudTrailEvent(id: string, options?: OnCloudTrailBucketEventOptions): Rule ``` Define a CloudWatch event that triggers when something happens to this repository. Requires that there exists at least one CloudTrail Trail in your account that captures the event. This method will not create the Trail. ###### `id`^Required - *Type:* string The id of the rule. --- ###### `options`^Optional - *Type:* aws-cdk-lib.aws_s3.OnCloudTrailBucketEventOptions Options for adding the rule. --- ##### `onCloudTrailPutObject` ```typescript public onCloudTrailPutObject(id: string, options?: OnCloudTrailBucketEventOptions): Rule ``` Defines an AWS CloudWatch event that triggers when an object is uploaded to the specified paths (keys) in this bucket using the PutObject API call. Note that some tools like `aws s3 cp` will automatically use either PutObject or the multipart upload API depending on the file size, so using `onCloudTrailWriteObject` may be preferable. Requires that there exists at least one CloudTrail Trail in your account that captures the event. This method will not create the Trail. ###### `id`^Required - *Type:* string The id of the rule. --- ###### `options`^Optional - *Type:* aws-cdk-lib.aws_s3.OnCloudTrailBucketEventOptions Options for adding the rule. --- ##### `onCloudTrailWriteObject` ```typescript public onCloudTrailWriteObject(id: string, options?: OnCloudTrailBucketEventOptions): Rule ``` Defines an AWS CloudWatch event that triggers when an object at the specified paths (keys) in this bucket are written to. This includes the events PutObject, CopyObject, and CompleteMultipartUpload. Note that some tools like `aws s3 cp` will automatically use either PutObject or the multipart upload API depending on the file size, so using this method may be preferable to `onCloudTrailPutObject`. Requires that there exists at least one CloudTrail Trail in your account that captures the event. This method will not create the Trail. ###### `id`^Required - *Type:* string The id of the rule. --- ###### `options`^Optional - *Type:* aws-cdk-lib.aws_s3.OnCloudTrailBucketEventOptions Options for adding the rule. --- ##### `s3UrlForObject` ```typescript public s3UrlForObject(key?: string): string ``` The S3 URL of an S3 object. For example:. `s3://onlybucket` - `s3://bucket/key` ###### `key`^Optional - *Type:* string The S3 key of the object. If not specified, the S3 URL of the bucket is returned. --- ##### `transferAccelerationUrlForObject` ```typescript public transferAccelerationUrlForObject(key?: string, options?: TransferAccelerationUrlOptions): string ``` The https Transfer Acceleration URL of an S3 object. Specify `dualStack: true` at the options for dual-stack endpoint (connect to the bucket over IPv6). For example: - `https://bucket.s3-accelerate.amazonaws.com` - `https://bucket.s3-accelerate.amazonaws.com/key` ###### `key`^Optional - *Type:* string The S3 key of the object. If not specified, the URL of the bucket is returned. --- ###### `options`^Optional - *Type:* aws-cdk-lib.aws_s3.TransferAccelerationUrlOptions Options for generating URL. --- ##### `urlForObject` ```typescript public urlForObject(key?: string): string ``` The https URL of an S3 object. Specify `regional: false` at the options for non-regional URLs. For example:. `https://s3.us-west-1.amazonaws.com/onlybucket` - `https://s3.us-west-1.amazonaws.com/bucket/key` - `https://s3.cn-north-1.amazonaws.com.cn/china-bucket/mykey` ###### `key`^Optional - *Type:* string The S3 key of the object. If not specified, the URL of the bucket is returned. --- ##### `virtualHostedUrlForObject` ```typescript public virtualHostedUrlForObject(key?: string, options?: VirtualHostedStyleUrlOptions): string ``` The virtual hosted-style URL of an S3 object. Specify `regional: false` at the options for non-regional URL. For example:. `https://only-bucket.s3.us-west-1.amazonaws.com` - `https://bucket.s3.us-west-1.amazonaws.com/key` - `https://bucket.s3.amazonaws.com/key` - `https://china-bucket.s3.cn-north-1.amazonaws.com.cn/mykey` ###### `key`^Optional - *Type:* string The S3 key of the object. If not specified, the URL of the bucket is returned. --- ###### `options`^Optional - *Type:* aws-cdk-lib.aws_s3.VirtualHostedStyleUrlOptions Options for generating URL. --- ##### `addCorsRule` ```typescript public addCorsRule(rule: CorsRule): void ``` Adds a cross-origin access configuration for objects in an Amazon S3 bucket. ###### `rule`^Required - *Type:* aws-cdk-lib.aws_s3.CorsRule The CORS configuration rule to add. --- ##### `addInventory` ```typescript public addInventory(inventory: Inventory): void ``` Add an inventory configuration. ###### `inventory`^Required - *Type:* aws-cdk-lib.aws_s3.Inventory configuration to add. --- ##### `addLifecycleRule` ```typescript public addLifecycleRule(rule: LifecycleRule): void ``` Add a lifecycle rule to the bucket. ###### `rule`^Required - *Type:* aws-cdk-lib.aws_s3.LifecycleRule The rule to add. --- ##### `addMetric` ```typescript public addMetric(metric: BucketMetrics): void ``` Adds a metrics configuration for the CloudWatch request metrics from the bucket. ###### `metric`^Required - *Type:* aws-cdk-lib.aws_s3.BucketMetrics The metric configuration to add. --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | | isOwnedResource | Returns true if the construct was created by CDK, and false otherwise. | | isResource | Check whether the given construct is a Resource. | | fromBucketArn | *No description.* | | fromBucketAttributes | Creates a Bucket construct that represents an external bucket. | | fromBucketName | *No description.* | | fromCfnBucket | Create a mutable `IBucket` based on a low-level `CfnBucket`. | | validateBucketName | Thrown an exception if the given bucket name is not valid. | --- ##### `isConstruct` ```typescript import { storage } from 'aws-dsf' storage.AccessLogsBucket.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- ##### `isOwnedResource` ```typescript import { storage } from 'aws-dsf' storage.AccessLogsBucket.isOwnedResource(construct: IConstruct) ``` Returns true if the construct was created by CDK, and false otherwise. ###### `construct`^Required - *Type:* constructs.IConstruct --- ##### `isResource` ```typescript import { storage } from 'aws-dsf' storage.AccessLogsBucket.isResource(construct: IConstruct) ``` Check whether the given construct is a Resource. ###### `construct`^Required - *Type:* constructs.IConstruct --- ##### `fromBucketArn` ```typescript import { storage } from 'aws-dsf' storage.AccessLogsBucket.fromBucketArn(scope: Construct, id: string, bucketArn: string) ``` ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `bucketArn`^Required - *Type:* string --- ##### `fromBucketAttributes` ```typescript import { storage } from 'aws-dsf' storage.AccessLogsBucket.fromBucketAttributes(scope: Construct, id: string, attrs: BucketAttributes) ``` Creates a Bucket construct that represents an external bucket. ###### `scope`^Required - *Type:* constructs.Construct The parent creating construct (usually `this`). --- ###### `id`^Required - *Type:* string The construct's name. --- ###### `attrs`^Required - *Type:* aws-cdk-lib.aws_s3.BucketAttributes A `BucketAttributes` object. Can be obtained from a call to `bucket.export()` or manually created. --- ##### `fromBucketName` ```typescript import { storage } from 'aws-dsf' storage.AccessLogsBucket.fromBucketName(scope: Construct, id: string, bucketName: string) ``` ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `bucketName`^Required - *Type:* string --- ##### `fromCfnBucket` ```typescript import { storage } from 'aws-dsf' storage.AccessLogsBucket.fromCfnBucket(cfnBucket: CfnBucket) ``` Create a mutable `IBucket` based on a low-level `CfnBucket`. ###### `cfnBucket`^Required - *Type:* aws-cdk-lib.aws_s3.CfnBucket --- ##### `validateBucketName` ```typescript import { storage } from 'aws-dsf' storage.AccessLogsBucket.validateBucketName(physicalName: string) ``` Thrown an exception if the given bucket name is not valid. ###### `physicalName`^Required - *Type:* string name of the bucket. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | env | aws-cdk-lib.ResourceEnvironment | The environment this resource belongs to. | | stack | aws-cdk-lib.Stack | The stack in which this resource is defined. | | bucketArn | string | The ARN of the bucket. | | bucketDomainName | string | The IPv4 DNS name of the specified bucket. | | bucketDualStackDomainName | string | The IPv6 DNS name of the specified bucket. | | bucketName | string | The name of the bucket. | | bucketRegionalDomainName | string | The regional domain name of the specified bucket. | | bucketWebsiteDomainName | string | The Domain name of the static website. | | bucketWebsiteUrl | string | The URL of the static website. | | encryptionKey | aws-cdk-lib.aws_kms.IKey | Optional KMS encryption key associated with this bucket. | | isWebsite | boolean | If this bucket has been configured for static website hosting. | | policy | aws-cdk-lib.aws_s3.BucketPolicy | The resource policy associated with this bucket. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `env`^Required ```typescript public readonly env: ResourceEnvironment; ``` - *Type:* aws-cdk-lib.ResourceEnvironment The environment this resource belongs to. For resources that are created and managed by the CDK (generally, those created by creating new class instances like Role, Bucket, etc.), this is always the same as the environment of the stack they belong to; however, for imported resources (those obtained from static methods like fromRoleArn, fromBucketName, etc.), that might be different than the stack they were imported into. --- ##### `stack`^Required ```typescript public readonly stack: Stack; ``` - *Type:* aws-cdk-lib.Stack The stack in which this resource is defined. --- ##### `bucketArn`^Required ```typescript public readonly bucketArn: string; ``` - *Type:* string The ARN of the bucket. --- ##### `bucketDomainName`^Required ```typescript public readonly bucketDomainName: string; ``` - *Type:* string The IPv4 DNS name of the specified bucket. --- ##### `bucketDualStackDomainName`^Required ```typescript public readonly bucketDualStackDomainName: string; ``` - *Type:* string The IPv6 DNS name of the specified bucket. --- ##### `bucketName`^Required ```typescript public readonly bucketName: string; ``` - *Type:* string The name of the bucket. --- ##### `bucketRegionalDomainName`^Required ```typescript public readonly bucketRegionalDomainName: string; ``` - *Type:* string The regional domain name of the specified bucket. --- ##### `bucketWebsiteDomainName`^Required ```typescript public readonly bucketWebsiteDomainName: string; ``` - *Type:* string The Domain name of the static website. --- ##### `bucketWebsiteUrl`^Required ```typescript public readonly bucketWebsiteUrl: string; ``` - *Type:* string The URL of the static website. --- ##### `encryptionKey`^Optional ```typescript public readonly encryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey Optional KMS encryption key associated with this bucket. --- ##### `isWebsite`^Optional ```typescript public readonly isWebsite: boolean; ``` - *Type:* boolean If this bucket has been configured for static website hosting. --- ##### `policy`^Optional ```typescript public readonly policy: BucketPolicy; ``` - *Type:* aws-cdk-lib.aws_s3.BucketPolicy The resource policy associated with this bucket. If `autoCreatePolicy` is true, a `BucketPolicy` will be created upon the first call to addToResourcePolicy(s). --- ### AnalyticsBucket Amazon S3 Bucket configured with best-practices and defaults for analytics. The default bucket name is `analytics---` > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Storage/analytics-bucket](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Storage/analytics-bucket) *Example* ```typescript import { Key } from 'aws-cdk-lib/aws-kms'; // Set context value for global data removal policy (or set in cdk.json). this.node.setContext('@data-solutions-framework-on-aws/removeDataOnDestroy', true); const encryptionKey = new Key(this, 'DataKey', { removalPolicy: cdk.RemovalPolicy.DESTROY, enableKeyRotation: true, }); new dsf.storage.AnalyticsBucket(this, 'MyAnalyticsBucket', { encryptionKey, removalPolicy: cdk.RemovalPolicy.DESTROY, }); ``` #### Initializers ```typescript import { storage } from 'aws-dsf' new storage.AnalyticsBucket(scope: Construct, id: string, props: AnalyticsBucketProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-dsf.storage.AnalyticsBucketProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.storage.AnalyticsBucketProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | applyRemovalPolicy | Apply the given removal policy to this resource. | | addEventNotification | Adds a bucket notification event destination. | | addObjectCreatedNotification | Subscribes a destination to receive notifications when an object is created in the bucket. | | addObjectRemovedNotification | Subscribes a destination to receive notifications when an object is removed from the bucket. | | addToResourcePolicy | Adds a statement to the resource policy for a principal (i.e. account/role/service) to perform actions on this bucket and/or its contents. Use `bucketArn` and `arnForObjects(keys)` to obtain ARNs for this bucket or objects. | | arnForObjects | Returns an ARN that represents all objects within the bucket that match the key pattern specified. | | enableEventBridgeNotification | Enables event bridge notification, causing all events below to be sent to EventBridge:. | | grantDelete | Grants s3:DeleteObject* permission to an IAM principal for objects in this bucket. | | grantPublicAccess | Allows unrestricted access to objects from this bucket. | | grantPut | Grants s3:PutObject* and s3:Abort* permissions for this bucket to an IAM principal. | | grantPutAcl | Grant the given IAM identity permissions to modify the ACLs of objects in the given Bucket. | | grantRead | Grant read permissions for this bucket and it's contents to an IAM principal (Role/Group/User). | | grantReadWrite | Grants read/write permissions for this bucket and it's contents to an IAM principal (Role/Group/User). | | grantWrite | Grant write permissions to this bucket to an IAM principal. | | onCloudTrailEvent | Define a CloudWatch event that triggers when something happens to this repository. | | onCloudTrailPutObject | Defines an AWS CloudWatch event that triggers when an object is uploaded to the specified paths (keys) in this bucket using the PutObject API call. | | onCloudTrailWriteObject | Defines an AWS CloudWatch event that triggers when an object at the specified paths (keys) in this bucket are written to. | | s3UrlForObject | The S3 URL of an S3 object. For example:. | | transferAccelerationUrlForObject | The https Transfer Acceleration URL of an S3 object. | | urlForObject | The https URL of an S3 object. Specify `regional: false` at the options for non-regional URLs. For example:. | | virtualHostedUrlForObject | The virtual hosted-style URL of an S3 object. Specify `regional: false` at the options for non-regional URL. For example:. | | addCorsRule | Adds a cross-origin access configuration for objects in an Amazon S3 bucket. | | addInventory | Add an inventory configuration. | | addLifecycleRule | Add a lifecycle rule to the bucket. | | addMetric | Adds a metrics configuration for the CloudWatch request metrics from the bucket. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `applyRemovalPolicy` ```typescript public applyRemovalPolicy(policy: RemovalPolicy): void ``` Apply the given removal policy to this resource. The Removal Policy controls what happens to this resource when it stops being managed by CloudFormation, either because you've removed it from the CDK application or because you've made a change that requires the resource to be replaced. The resource can be deleted (`RemovalPolicy.DESTROY`), or left in your AWS account for data recovery and cleanup later (`RemovalPolicy.RETAIN`). ###### `policy`^Required - *Type:* aws-cdk-lib.RemovalPolicy --- ##### `addEventNotification` ```typescript public addEventNotification(event: EventType, dest: IBucketNotificationDestination, filters: NotificationKeyFilter): void ``` Adds a bucket notification event destination. > [https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html](https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) *Example* ```typescript declare const myLambda: lambda.Function; const bucket = new s3.Bucket(this, 'MyBucket'); bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(myLambda), {prefix: 'home/myusername/*'}); ``` ###### `event`^Required - *Type:* aws-cdk-lib.aws_s3.EventType The event to trigger the notification. --- ###### `dest`^Required - *Type:* aws-cdk-lib.aws_s3.IBucketNotificationDestination The notification destination (Lambda, SNS Topic or SQS Queue). --- ###### `filters`^Required - *Type:* aws-cdk-lib.aws_s3.NotificationKeyFilter S3 object key filter rules to determine which objects trigger this event. Each filter must include a `prefix` and/or `suffix` that will be matched against the s3 object key. Refer to the S3 Developer Guide for details about allowed filter rules. --- ##### `addObjectCreatedNotification` ```typescript public addObjectCreatedNotification(dest: IBucketNotificationDestination, filters: NotificationKeyFilter): void ``` Subscribes a destination to receive notifications when an object is created in the bucket. This is identical to calling `onEvent(EventType.OBJECT_CREATED)`. ###### `dest`^Required - *Type:* aws-cdk-lib.aws_s3.IBucketNotificationDestination The notification destination (see onEvent). --- ###### `filters`^Required - *Type:* aws-cdk-lib.aws_s3.NotificationKeyFilter Filters (see onEvent). --- ##### `addObjectRemovedNotification` ```typescript public addObjectRemovedNotification(dest: IBucketNotificationDestination, filters: NotificationKeyFilter): void ``` Subscribes a destination to receive notifications when an object is removed from the bucket. This is identical to calling `onEvent(EventType.OBJECT_REMOVED)`. ###### `dest`^Required - *Type:* aws-cdk-lib.aws_s3.IBucketNotificationDestination The notification destination (see onEvent). --- ###### `filters`^Required - *Type:* aws-cdk-lib.aws_s3.NotificationKeyFilter Filters (see onEvent). --- ##### `addToResourcePolicy` ```typescript public addToResourcePolicy(permission: PolicyStatement): AddToResourcePolicyResult ``` Adds a statement to the resource policy for a principal (i.e. account/role/service) to perform actions on this bucket and/or its contents. Use `bucketArn` and `arnForObjects(keys)` to obtain ARNs for this bucket or objects. Note that the policy statement may or may not be added to the policy. For example, when an `IBucket` is created from an existing bucket, it's not possible to tell whether the bucket already has a policy attached, let alone to re-use that policy to add more statements to it. So it's safest to do nothing in these cases. ###### `permission`^Required - *Type:* aws-cdk-lib.aws_iam.PolicyStatement the policy statement to be added to the bucket's policy. --- ##### `arnForObjects` ```typescript public arnForObjects(keyPattern: string): string ``` Returns an ARN that represents all objects within the bucket that match the key pattern specified. To represent all keys, specify ``"*"``. If you need to specify a keyPattern with multiple components, concatenate them into a single string, e.g.: arnForObjects(`home/${team}/${user}/*`) ###### `keyPattern`^Required - *Type:* string --- ##### `enableEventBridgeNotification` ```typescript public enableEventBridgeNotification(): void ``` Enables event bridge notification, causing all events below to be sent to EventBridge:. Object Deleted (DeleteObject) - Object Deleted (Lifecycle expiration) - Object Restore Initiated - Object Restore Completed - Object Restore Expired - Object Storage Class Changed - Object Access Tier Changed - Object ACL Updated - Object Tags Added - Object Tags Deleted ##### `grantDelete` ```typescript public grantDelete(identity: IGrantable, objectsKeyPattern?: any): Grant ``` Grants s3:DeleteObject* permission to an IAM principal for objects in this bucket. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable The principal. --- ###### `objectsKeyPattern`^Optional - *Type:* any Restrict the permission to a certain key pattern (default '*'). Parameter type is `any` but `string` should be passed in. --- ##### `grantPublicAccess` ```typescript public grantPublicAccess(allowedActions: string, keyPrefix?: string): Grant ``` Allows unrestricted access to objects from this bucket. IMPORTANT: This permission allows anyone to perform actions on S3 objects in this bucket, which is useful for when you configure your bucket as a website and want everyone to be able to read objects in the bucket without needing to authenticate. Without arguments, this method will grant read ("s3:GetObject") access to all objects ("*") in the bucket. The method returns the `iam.Grant` object, which can then be modified as needed. For example, you can add a condition that will restrict access only to an IPv4 range like this: const grant = bucket.grantPublicAccess(); grant.resourceStatement!.addCondition(‘IpAddress’, { “aws:SourceIp”: “54.240.143.0/24” }); Note that if this `IBucket` refers to an existing bucket, possibly not managed by CloudFormation, this method will have no effect, since it's impossible to modify the policy of an existing bucket. ###### `allowedActions`^Required - *Type:* string the set of S3 actions to allow. Default is "s3:GetObject". --- ###### `keyPrefix`^Optional - *Type:* string the prefix of S3 object keys (e.g. `home/*`). Default is "*". --- ##### `grantPut` ```typescript public grantPut(identity: IGrantable, objectsKeyPattern?: any): Grant ``` Grants s3:PutObject* and s3:Abort* permissions for this bucket to an IAM principal. If encryption is used, permission to use the key to encrypt the contents of written files will also be granted to the same principal. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable The principal. --- ###### `objectsKeyPattern`^Optional - *Type:* any Restrict the permission to a certain key pattern (default '*'). Parameter type is `any` but `string` should be passed in. --- ##### `grantPutAcl` ```typescript public grantPutAcl(identity: IGrantable, objectsKeyPattern?: string): Grant ``` Grant the given IAM identity permissions to modify the ACLs of objects in the given Bucket. If your application has the '@aws-cdk/aws-s3:grantWriteWithoutAcl' feature flag set, calling `grantWrite` or `grantReadWrite` no longer grants permissions to modify the ACLs of the objects; in this case, if you need to modify object ACLs, call this method explicitly. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable --- ###### `objectsKeyPattern`^Optional - *Type:* string --- ##### `grantRead` ```typescript public grantRead(identity: IGrantable, objectsKeyPattern?: any): Grant ``` Grant read permissions for this bucket and it's contents to an IAM principal (Role/Group/User). If encryption is used, permission to use the key to decrypt the contents of the bucket will also be granted to the same principal. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable The principal. --- ###### `objectsKeyPattern`^Optional - *Type:* any Restrict the permission to a certain key pattern (default '*'). Parameter type is `any` but `string` should be passed in. --- ##### `grantReadWrite` ```typescript public grantReadWrite(identity: IGrantable, objectsKeyPattern?: any): Grant ``` Grants read/write permissions for this bucket and it's contents to an IAM principal (Role/Group/User). If an encryption key is used, permission to use the key for encrypt/decrypt will also be granted. Before CDK version 1.85.0, this method granted the `s3:PutObject*` permission that included `s3:PutObjectAcl`, which could be used to grant read/write object access to IAM principals in other accounts. If you want to get rid of that behavior, update your CDK version to 1.85.0 or later, and make sure the `@aws-cdk/aws-s3:grantWriteWithoutAcl` feature flag is set to `true` in the `context` key of your cdk.json file. If you've already updated, but still need the principal to have permissions to modify the ACLs, use the `grantPutAcl` method. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable --- ###### `objectsKeyPattern`^Optional - *Type:* any --- ##### `grantWrite` ```typescript public grantWrite(identity: IGrantable, objectsKeyPattern?: any, allowedActionPatterns?: string[]): Grant ``` Grant write permissions to this bucket to an IAM principal. If encryption is used, permission to use the key to encrypt the contents of written files will also be granted to the same principal. Before CDK version 1.85.0, this method granted the `s3:PutObject*` permission that included `s3:PutObjectAcl`, which could be used to grant read/write object access to IAM principals in other accounts. If you want to get rid of that behavior, update your CDK version to 1.85.0 or later, and make sure the `@aws-cdk/aws-s3:grantWriteWithoutAcl` feature flag is set to `true` in the `context` key of your cdk.json file. If you've already updated, but still need the principal to have permissions to modify the ACLs, use the `grantPutAcl` method. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable --- ###### `objectsKeyPattern`^Optional - *Type:* any --- ###### `allowedActionPatterns`^Optional - *Type:* string[] --- ##### `onCloudTrailEvent` ```typescript public onCloudTrailEvent(id: string, options?: OnCloudTrailBucketEventOptions): Rule ``` Define a CloudWatch event that triggers when something happens to this repository. Requires that there exists at least one CloudTrail Trail in your account that captures the event. This method will not create the Trail. ###### `id`^Required - *Type:* string The id of the rule. --- ###### `options`^Optional - *Type:* aws-cdk-lib.aws_s3.OnCloudTrailBucketEventOptions Options for adding the rule. --- ##### `onCloudTrailPutObject` ```typescript public onCloudTrailPutObject(id: string, options?: OnCloudTrailBucketEventOptions): Rule ``` Defines an AWS CloudWatch event that triggers when an object is uploaded to the specified paths (keys) in this bucket using the PutObject API call. Note that some tools like `aws s3 cp` will automatically use either PutObject or the multipart upload API depending on the file size, so using `onCloudTrailWriteObject` may be preferable. Requires that there exists at least one CloudTrail Trail in your account that captures the event. This method will not create the Trail. ###### `id`^Required - *Type:* string The id of the rule. --- ###### `options`^Optional - *Type:* aws-cdk-lib.aws_s3.OnCloudTrailBucketEventOptions Options for adding the rule. --- ##### `onCloudTrailWriteObject` ```typescript public onCloudTrailWriteObject(id: string, options?: OnCloudTrailBucketEventOptions): Rule ``` Defines an AWS CloudWatch event that triggers when an object at the specified paths (keys) in this bucket are written to. This includes the events PutObject, CopyObject, and CompleteMultipartUpload. Note that some tools like `aws s3 cp` will automatically use either PutObject or the multipart upload API depending on the file size, so using this method may be preferable to `onCloudTrailPutObject`. Requires that there exists at least one CloudTrail Trail in your account that captures the event. This method will not create the Trail. ###### `id`^Required - *Type:* string The id of the rule. --- ###### `options`^Optional - *Type:* aws-cdk-lib.aws_s3.OnCloudTrailBucketEventOptions Options for adding the rule. --- ##### `s3UrlForObject` ```typescript public s3UrlForObject(key?: string): string ``` The S3 URL of an S3 object. For example:. `s3://onlybucket` - `s3://bucket/key` ###### `key`^Optional - *Type:* string The S3 key of the object. If not specified, the S3 URL of the bucket is returned. --- ##### `transferAccelerationUrlForObject` ```typescript public transferAccelerationUrlForObject(key?: string, options?: TransferAccelerationUrlOptions): string ``` The https Transfer Acceleration URL of an S3 object. Specify `dualStack: true` at the options for dual-stack endpoint (connect to the bucket over IPv6). For example: - `https://bucket.s3-accelerate.amazonaws.com` - `https://bucket.s3-accelerate.amazonaws.com/key` ###### `key`^Optional - *Type:* string The S3 key of the object. If not specified, the URL of the bucket is returned. --- ###### `options`^Optional - *Type:* aws-cdk-lib.aws_s3.TransferAccelerationUrlOptions Options for generating URL. --- ##### `urlForObject` ```typescript public urlForObject(key?: string): string ``` The https URL of an S3 object. Specify `regional: false` at the options for non-regional URLs. For example:. `https://s3.us-west-1.amazonaws.com/onlybucket` - `https://s3.us-west-1.amazonaws.com/bucket/key` - `https://s3.cn-north-1.amazonaws.com.cn/china-bucket/mykey` ###### `key`^Optional - *Type:* string The S3 key of the object. If not specified, the URL of the bucket is returned. --- ##### `virtualHostedUrlForObject` ```typescript public virtualHostedUrlForObject(key?: string, options?: VirtualHostedStyleUrlOptions): string ``` The virtual hosted-style URL of an S3 object. Specify `regional: false` at the options for non-regional URL. For example:. `https://only-bucket.s3.us-west-1.amazonaws.com` - `https://bucket.s3.us-west-1.amazonaws.com/key` - `https://bucket.s3.amazonaws.com/key` - `https://china-bucket.s3.cn-north-1.amazonaws.com.cn/mykey` ###### `key`^Optional - *Type:* string The S3 key of the object. If not specified, the URL of the bucket is returned. --- ###### `options`^Optional - *Type:* aws-cdk-lib.aws_s3.VirtualHostedStyleUrlOptions Options for generating URL. --- ##### `addCorsRule` ```typescript public addCorsRule(rule: CorsRule): void ``` Adds a cross-origin access configuration for objects in an Amazon S3 bucket. ###### `rule`^Required - *Type:* aws-cdk-lib.aws_s3.CorsRule The CORS configuration rule to add. --- ##### `addInventory` ```typescript public addInventory(inventory: Inventory): void ``` Add an inventory configuration. ###### `inventory`^Required - *Type:* aws-cdk-lib.aws_s3.Inventory configuration to add. --- ##### `addLifecycleRule` ```typescript public addLifecycleRule(rule: LifecycleRule): void ``` Add a lifecycle rule to the bucket. ###### `rule`^Required - *Type:* aws-cdk-lib.aws_s3.LifecycleRule The rule to add. --- ##### `addMetric` ```typescript public addMetric(metric: BucketMetrics): void ``` Adds a metrics configuration for the CloudWatch request metrics from the bucket. ###### `metric`^Required - *Type:* aws-cdk-lib.aws_s3.BucketMetrics The metric configuration to add. --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | | isOwnedResource | Returns true if the construct was created by CDK, and false otherwise. | | isResource | Check whether the given construct is a Resource. | | fromBucketArn | *No description.* | | fromBucketAttributes | Creates a Bucket construct that represents an external bucket. | | fromBucketName | *No description.* | | fromCfnBucket | Create a mutable `IBucket` based on a low-level `CfnBucket`. | | validateBucketName | Thrown an exception if the given bucket name is not valid. | --- ##### `isConstruct` ```typescript import { storage } from 'aws-dsf' storage.AnalyticsBucket.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- ##### `isOwnedResource` ```typescript import { storage } from 'aws-dsf' storage.AnalyticsBucket.isOwnedResource(construct: IConstruct) ``` Returns true if the construct was created by CDK, and false otherwise. ###### `construct`^Required - *Type:* constructs.IConstruct --- ##### `isResource` ```typescript import { storage } from 'aws-dsf' storage.AnalyticsBucket.isResource(construct: IConstruct) ``` Check whether the given construct is a Resource. ###### `construct`^Required - *Type:* constructs.IConstruct --- ##### `fromBucketArn` ```typescript import { storage } from 'aws-dsf' storage.AnalyticsBucket.fromBucketArn(scope: Construct, id: string, bucketArn: string) ``` ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `bucketArn`^Required - *Type:* string --- ##### `fromBucketAttributes` ```typescript import { storage } from 'aws-dsf' storage.AnalyticsBucket.fromBucketAttributes(scope: Construct, id: string, attrs: BucketAttributes) ``` Creates a Bucket construct that represents an external bucket. ###### `scope`^Required - *Type:* constructs.Construct The parent creating construct (usually `this`). --- ###### `id`^Required - *Type:* string The construct's name. --- ###### `attrs`^Required - *Type:* aws-cdk-lib.aws_s3.BucketAttributes A `BucketAttributes` object. Can be obtained from a call to `bucket.export()` or manually created. --- ##### `fromBucketName` ```typescript import { storage } from 'aws-dsf' storage.AnalyticsBucket.fromBucketName(scope: Construct, id: string, bucketName: string) ``` ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `bucketName`^Required - *Type:* string --- ##### `fromCfnBucket` ```typescript import { storage } from 'aws-dsf' storage.AnalyticsBucket.fromCfnBucket(cfnBucket: CfnBucket) ``` Create a mutable `IBucket` based on a low-level `CfnBucket`. ###### `cfnBucket`^Required - *Type:* aws-cdk-lib.aws_s3.CfnBucket --- ##### `validateBucketName` ```typescript import { storage } from 'aws-dsf' storage.AnalyticsBucket.validateBucketName(physicalName: string) ``` Thrown an exception if the given bucket name is not valid. ###### `physicalName`^Required - *Type:* string name of the bucket. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | env | aws-cdk-lib.ResourceEnvironment | The environment this resource belongs to. | | stack | aws-cdk-lib.Stack | The stack in which this resource is defined. | | bucketArn | string | The ARN of the bucket. | | bucketDomainName | string | The IPv4 DNS name of the specified bucket. | | bucketDualStackDomainName | string | The IPv6 DNS name of the specified bucket. | | bucketName | string | The name of the bucket. | | bucketRegionalDomainName | string | The regional domain name of the specified bucket. | | bucketWebsiteDomainName | string | The Domain name of the static website. | | bucketWebsiteUrl | string | The URL of the static website. | | encryptionKey | aws-cdk-lib.aws_kms.IKey | Optional KMS encryption key associated with this bucket. | | isWebsite | boolean | If this bucket has been configured for static website hosting. | | policy | aws-cdk-lib.aws_s3.BucketPolicy | The resource policy associated with this bucket. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `env`^Required ```typescript public readonly env: ResourceEnvironment; ``` - *Type:* aws-cdk-lib.ResourceEnvironment The environment this resource belongs to. For resources that are created and managed by the CDK (generally, those created by creating new class instances like Role, Bucket, etc.), this is always the same as the environment of the stack they belong to; however, for imported resources (those obtained from static methods like fromRoleArn, fromBucketName, etc.), that might be different than the stack they were imported into. --- ##### `stack`^Required ```typescript public readonly stack: Stack; ``` - *Type:* aws-cdk-lib.Stack The stack in which this resource is defined. --- ##### `bucketArn`^Required ```typescript public readonly bucketArn: string; ``` - *Type:* string The ARN of the bucket. --- ##### `bucketDomainName`^Required ```typescript public readonly bucketDomainName: string; ``` - *Type:* string The IPv4 DNS name of the specified bucket. --- ##### `bucketDualStackDomainName`^Required ```typescript public readonly bucketDualStackDomainName: string; ``` - *Type:* string The IPv6 DNS name of the specified bucket. --- ##### `bucketName`^Required ```typescript public readonly bucketName: string; ``` - *Type:* string The name of the bucket. --- ##### `bucketRegionalDomainName`^Required ```typescript public readonly bucketRegionalDomainName: string; ``` - *Type:* string The regional domain name of the specified bucket. --- ##### `bucketWebsiteDomainName`^Required ```typescript public readonly bucketWebsiteDomainName: string; ``` - *Type:* string The Domain name of the static website. --- ##### `bucketWebsiteUrl`^Required ```typescript public readonly bucketWebsiteUrl: string; ``` - *Type:* string The URL of the static website. --- ##### `encryptionKey`^Optional ```typescript public readonly encryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey Optional KMS encryption key associated with this bucket. --- ##### `isWebsite`^Optional ```typescript public readonly isWebsite: boolean; ``` - *Type:* boolean If this bucket has been configured for static website hosting. --- ##### `policy`^Optional ```typescript public readonly policy: BucketPolicy; ``` - *Type:* aws-cdk-lib.aws_s3.BucketPolicy The resource policy associated with this bucket. If `autoCreatePolicy` is true, a `BucketPolicy` will be created upon the first call to addToResourcePolicy(s). --- ### ApplicationStage ApplicationStage class that creates a CDK Pipelines Stage from an ApplicationStackFactory. #### Initializers ```typescript import { utils } from 'aws-dsf' new utils.ApplicationStage(scope: Construct, id: string, props: ApplicationStageProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | the Scope of the CDK Construct. | | id | string | the ID of the CDK Construct. | | props | aws-dsf.utils.ApplicationStageProps | the ApplicationStage properties. | --- ##### `scope`^Required - *Type:* constructs.Construct the Scope of the CDK Construct. --- ##### `id`^Required - *Type:* string the ID of the CDK Construct. --- ##### `props`^Required - *Type:* aws-dsf.utils.ApplicationStageProps the ApplicationStage properties. --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | synth | Synthesize this stage into a cloud assembly. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `synth` ```typescript public synth(options?: StageSynthesisOptions): CloudAssembly ``` Synthesize this stage into a cloud assembly. Once an assembly has been synthesized, it cannot be modified. Subsequent calls will return the same assembly. ###### `options`^Optional - *Type:* aws-cdk-lib.StageSynthesisOptions --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | | isStage | Test whether the given construct is a stage. | | of | Return the stage this construct is contained with, if available. | --- ##### `isConstruct` ```typescript import { utils } from 'aws-dsf' utils.ApplicationStage.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- ##### `isStage` ```typescript import { utils } from 'aws-dsf' utils.ApplicationStage.isStage(x: any) ``` Test whether the given construct is a stage. ###### `x`^Required - *Type:* any --- ##### `of` ```typescript import { utils } from 'aws-dsf' utils.ApplicationStage.of(construct: IConstruct) ``` Return the stage this construct is contained with, if available. If called on a nested stage, returns its parent. ###### `construct`^Required - *Type:* constructs.IConstruct --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | artifactId | string | Artifact ID of the assembly if it is a nested stage. The root stage (app) will return an empty string. | | assetOutdir | string | The cloud assembly asset output directory. | | outdir | string | The cloud assembly output directory. | | policyValidationBeta1 | aws-cdk-lib.IPolicyValidationPluginBeta1[] | Validation plugins to run during synthesis. | | stageName | string | The name of the stage. | | account | string | The default account for all resources defined within this stage. | | parentStage | aws-cdk-lib.Stage | The parent stage or `undefined` if this is the app. | | region | string | The default region for all resources defined within this stage. | | stackOutputsEnv | {[ key: string ]: aws-cdk-lib.CfnOutput} | The list of CfnOutputs created by the CDK Stack. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `artifactId`^Required ```typescript public readonly artifactId: string; ``` - *Type:* string Artifact ID of the assembly if it is a nested stage. The root stage (app) will return an empty string. Derived from the construct path. --- ##### `assetOutdir`^Required ```typescript public readonly assetOutdir: string; ``` - *Type:* string The cloud assembly asset output directory. --- ##### `outdir`^Required ```typescript public readonly outdir: string; ``` - *Type:* string The cloud assembly output directory. --- ##### `policyValidationBeta1`^Required ```typescript public readonly policyValidationBeta1: IPolicyValidationPluginBeta1[]; ``` - *Type:* aws-cdk-lib.IPolicyValidationPluginBeta1[] - *Default:* no validation plugins are used Validation plugins to run during synthesis. If any plugin reports any violation, synthesis will be interrupted and the report displayed to the user. --- ##### `stageName`^Required ```typescript public readonly stageName: string; ``` - *Type:* string The name of the stage. Based on names of the parent stages separated by hypens. --- ##### `account`^Optional ```typescript public readonly account: string; ``` - *Type:* string The default account for all resources defined within this stage. --- ##### `parentStage`^Optional ```typescript public readonly parentStage: Stage; ``` - *Type:* aws-cdk-lib.Stage The parent stage or `undefined` if this is the app. * --- ##### `region`^Optional ```typescript public readonly region: string; ``` - *Type:* string The default region for all resources defined within this stage. --- ##### `stackOutputsEnv`^Optional ```typescript public readonly stackOutputsEnv: {[ key: string ]: CfnOutput}; ``` - *Type:* {[ key: string ]: aws-cdk-lib.CfnOutput} The list of CfnOutputs created by the CDK Stack. --- ### AthenaWorkGroup An Amazon Athena Workgroup configured with default result bucket. *Example* ```typescript new dsf.consumption.AthenaWorkGroup(this, 'AthenaWorkGroupDefault', { name: 'athena-default', resultLocationPrefix: 'athena-default-results/' }) ``` #### Initializers ```typescript import { consumption } from 'aws-dsf' new consumption.AthenaWorkGroup(scope: Construct, id: string, props: AthenaWorkgroupProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-dsf.consumption.AthenaWorkgroupProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.consumption.AthenaWorkgroupProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | grantRunQueries | Grants running queries access to Principal. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `grantRunQueries` ```typescript public grantRunQueries(principal: IPrincipal): void ``` Grants running queries access to Principal. ###### `principal`^Required - *Type:* aws-cdk-lib.aws_iam.IPrincipal Principal to attach query access to Athena Workgroup. --- ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { consumption } from 'aws-dsf' consumption.AthenaWorkGroup.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | resultBucket | aws-cdk-lib.aws_s3.IBucket | S3 Bucket used for query results. | | resultsEncryptionKey | aws-cdk-lib.aws_kms.IKey | KMS Key to encrypt the query results. | | workGroup | aws-cdk-lib.aws_athena.CfnWorkGroup | Athena Workgroup that is created. | | workGroupName | string | WorkGroup name with the randomized suffix. | | executionRole | aws-cdk-lib.aws_iam.IRole | Role used to access user resources in an Athena for Apache Spark session. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `resultBucket`^Required ```typescript public readonly resultBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket S3 Bucket used for query results. --- ##### `resultsEncryptionKey`^Required ```typescript public readonly resultsEncryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey KMS Key to encrypt the query results. --- ##### `workGroup`^Required ```typescript public readonly workGroup: CfnWorkGroup; ``` - *Type:* aws-cdk-lib.aws_athena.CfnWorkGroup Athena Workgroup that is created. --- ##### `workGroupName`^Required ```typescript public readonly workGroupName: string; ``` - *Type:* string WorkGroup name with the randomized suffix. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole Role used to access user resources in an Athena for Apache Spark session. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### DataCatalogDatabase An AWS Glue Data Catalog Database configured with the location and a crawler. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Governance/data-catalog-database](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Governance/data-catalog-database) *Example* ```typescript import { Bucket } from 'aws-cdk-lib/aws-s3'; new dsf.governance.DataCatalogDatabase(this, 'ExampleDatabase', { locationBucket: new Bucket(scope, 'LocationBucket'), locationPrefix: '/databasePath', name: 'example-db' }); ``` #### Initializers ```typescript import { governance } from 'aws-dsf' new governance.DataCatalogDatabase(scope: Construct, id: string, props: DataCatalogDatabaseProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-dsf.governance.DataCatalogDatabaseProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.governance.DataCatalogDatabaseProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | grantReadOnlyAccess | Grants read access via identity based policy to the principal. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `grantReadOnlyAccess` ```typescript public grantReadOnlyAccess(principal: IPrincipal): AddToPrincipalPolicyResult ``` Grants read access via identity based policy to the principal. This would attach an IAM Policy to the principal allowing read access to the Glue Database and all its Glue Tables. ###### `principal`^Required - *Type:* aws-cdk-lib.aws_iam.IPrincipal Principal to attach the Glue Database read access to. --- ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { governance } from 'aws-dsf' governance.DataCatalogDatabase.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | database | aws-cdk-lib.aws_glue.CfnDatabase | The Glue Database that's created. | | databaseName | string | The Glue Database name with the randomized suffix to prevent name collisions in the catalog. | | crawler | aws-cdk-lib.aws_glue.CfnCrawler | The Glue Crawler created when `autoCrawl` is set to `true` (default value). | | crawlerLogEncryptionKey | aws-cdk-lib.aws_kms.IKey | KMS encryption Key used by the Crawler. | | crawlerRole | aws-cdk-lib.aws_iam.IRole | The IAM Role used by the Glue crawler when created. | | crawlerSecurityConfiguration | aws-cdk-lib.aws_glue.CfnSecurityConfiguration | The Glue security configuration used by the Glue Crawler when created. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `database`^Required ```typescript public readonly database: CfnDatabase; ``` - *Type:* aws-cdk-lib.aws_glue.CfnDatabase The Glue Database that's created. --- ##### `databaseName`^Required ```typescript public readonly databaseName: string; ``` - *Type:* string The Glue Database name with the randomized suffix to prevent name collisions in the catalog. --- ##### `crawler`^Optional ```typescript public readonly crawler: CfnCrawler; ``` - *Type:* aws-cdk-lib.aws_glue.CfnCrawler The Glue Crawler created when `autoCrawl` is set to `true` (default value). This property can be undefined if `autoCrawl` is set to `false`. --- ##### `crawlerLogEncryptionKey`^Optional ```typescript public readonly crawlerLogEncryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey KMS encryption Key used by the Crawler. --- ##### `crawlerRole`^Optional ```typescript public readonly crawlerRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role used by the Glue crawler when created. --- ##### `crawlerSecurityConfiguration`^Optional ```typescript public readonly crawlerSecurityConfiguration: CfnSecurityConfiguration; ``` - *Type:* aws-cdk-lib.aws_glue.CfnSecurityConfiguration The Glue security configuration used by the Glue Crawler when created. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### DataLakeCatalog Creates a Data Lake Catalog on top of a `DataLakeStorage`. The Data Lake Catalog is composed of 3 `DataCatalogDatabase`, one for each storage layer. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Governance/data-lake-catalog](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Governance/data-lake-catalog) *Example* ```typescript import { Key } from 'aws-cdk-lib/aws-kms'; const logEncryptionKey = new Key(this, 'ExampleLogKey'); const storage = new dsf.storage.DataLakeStorage(this, "ExampleStorage"); const dataLakeCatalog = new dsf.governance.DataLakeCatalog(this, "ExampleDataLakeCatalog", { dataLakeStorage: storage, databaseName: "exampledb", crawlerLogEncryptionKey: logEncryptionKey }) ``` #### Initializers ```typescript import { governance } from 'aws-dsf' new governance.DataLakeCatalog(scope: Construct, id: string, props: DataLakeCatalogProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | the Scope of the CDK Construct. | | id | string | the ID of the CDK Construct. | | props | aws-dsf.governance.DataLakeCatalogProps | the DataLakeCatalog properties. | --- ##### `scope`^Required - *Type:* constructs.Construct the Scope of the CDK Construct. --- ##### `id`^Required - *Type:* string the ID of the CDK Construct. --- ##### `props`^Required - *Type:* aws-dsf.governance.DataLakeCatalogProps the DataLakeCatalog properties. --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { governance } from 'aws-dsf' governance.DataLakeCatalog.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | bronzeCatalogDatabase | aws-dsf.governance.DataCatalogDatabase | The Glue Database for the Bronze S3 Bucket. | | goldCatalogDatabase | aws-dsf.governance.DataCatalogDatabase | The Glue Database for the Gold S3 Bucket. | | silverCatalogDatabase | aws-dsf.governance.DataCatalogDatabase | The Glue Database for the Silver S3 Bucket. | | crawlerLogEncryptionKey | aws-cdk-lib.aws_kms.IKey | The KMS Key used to encrypt the Glue Crawler logs. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `bronzeCatalogDatabase`^Required ```typescript public readonly bronzeCatalogDatabase: DataCatalogDatabase; ``` - *Type:* aws-dsf.governance.DataCatalogDatabase The Glue Database for the Bronze S3 Bucket. --- ##### `goldCatalogDatabase`^Required ```typescript public readonly goldCatalogDatabase: DataCatalogDatabase; ``` - *Type:* aws-dsf.governance.DataCatalogDatabase The Glue Database for the Gold S3 Bucket. --- ##### `silverCatalogDatabase`^Required ```typescript public readonly silverCatalogDatabase: DataCatalogDatabase; ``` - *Type:* aws-dsf.governance.DataCatalogDatabase The Glue Database for the Silver S3 Bucket. --- ##### `crawlerLogEncryptionKey`^Optional ```typescript public readonly crawlerLogEncryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey The KMS Key used to encrypt the Glue Crawler logs. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### DataLakeStorage Creates the storage layer for a data lake, composed of 3 `AnalyticsBucket` for Bronze, Silver, and Gold data. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Storage/data-lake-storage](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Storage/data-lake-storage) *Example* ```typescript // Set the context value for global data removal policy this.node.setContext('@data-solutions-framework-on-aws/removeDataOnDestroy', true); new dsf.storage.DataLakeStorage(this, 'MyDataLakeStorage', { bronzeBucketName: 'my-bronze', bronzeBucketInfrequentAccessDelay: 90, bronzeBucketArchiveDelay: 180, silverBucketName: 'my-silver', silverBucketInfrequentAccessDelay: 180, silverBucketArchiveDelay: 360, goldBucketName: 'my-gold', goldBucketInfrequentAccessDelay: 180, goldBucketArchiveDelay: 360, removalPolicy: cdk.RemovalPolicy.DESTROY, }); ``` #### Initializers ```typescript import { storage } from 'aws-dsf' new storage.DataLakeStorage(scope: Construct, id: string, props?: DataLakeStorageProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | the Scope of the CDK Construct. | | id | string | the ID of the CDK Construct. | | props | aws-dsf.storage.DataLakeStorageProps | the DataLakeStorageProps properties. | --- ##### `scope`^Required - *Type:* constructs.Construct the Scope of the CDK Construct. --- ##### `id`^Required - *Type:* string the ID of the CDK Construct. --- ##### `props`^Optional - *Type:* aws-dsf.storage.DataLakeStorageProps the DataLakeStorageProps properties. --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { storage } from 'aws-dsf' storage.DataLakeStorage.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | accessLogsBucket | aws-dsf.storage.AccessLogsBucket | The S3 Bucket for access logs. | | bronzeBucket | aws-dsf.storage.AnalyticsBucket | The S3 Bucket for Bronze layer. | | dataLakeKey | aws-cdk-lib.aws_kms.IKey | The KMS Key used to encrypt all DataLakeStorage S3 buckets. | | goldBucket | aws-dsf.storage.AnalyticsBucket | The S3 Bucket for Gold layer. | | silverBucket | aws-dsf.storage.AnalyticsBucket | The S3 Bucket for Silver layer. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `accessLogsBucket`^Required ```typescript public readonly accessLogsBucket: AccessLogsBucket; ``` - *Type:* aws-dsf.storage.AccessLogsBucket The S3 Bucket for access logs. --- ##### `bronzeBucket`^Required ```typescript public readonly bronzeBucket: AnalyticsBucket; ``` - *Type:* aws-dsf.storage.AnalyticsBucket The S3 Bucket for Bronze layer. --- ##### `dataLakeKey`^Required ```typescript public readonly dataLakeKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey The KMS Key used to encrypt all DataLakeStorage S3 buckets. --- ##### `goldBucket`^Required ```typescript public readonly goldBucket: AnalyticsBucket; ``` - *Type:* aws-dsf.storage.AnalyticsBucket The S3 Bucket for Gold layer. --- ##### `silverBucket`^Required ```typescript public readonly silverBucket: AnalyticsBucket; ``` - *Type:* aws-dsf.storage.AnalyticsBucket The S3 Bucket for Silver layer. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### DataVpc Creates a VPC with best practices for securely deploying data solutions. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Utils/data-vpc](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Utils/data-vpc) *Example* ```typescript const vpc = new dsf.utils.DataVpc(this, 'DataVpc', { vpcCidr: '10.0.0.0/16', }); vpc.tagVpc('Name', 'My VPC'); ``` #### Initializers ```typescript import { utils } from 'aws-dsf' new utils.DataVpc(scope: Construct, id: string, props: DataVpcProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-dsf.utils.DataVpcProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.utils.DataVpcProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | tagVpc | Tag the VPC and the subnets. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `tagVpc` ```typescript public tagVpc(key: string, value: string): void ``` Tag the VPC and the subnets. ###### `key`^Required - *Type:* string the tag key. --- ###### `value`^Required - *Type:* string the tag value. --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { utils } from 'aws-dsf' utils.DataVpc.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | flowLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group created for the VPC flow logs. | | flowLogKey | aws-cdk-lib.aws_kms.IKey | The KMS Key used to encrypt VPC flow logs. | | flowLogRole | aws-cdk-lib.aws_iam.IRole | The IAM role used to publish VPC Flow Logs. | | s3VpcEndpoint | aws-cdk-lib.aws_ec2.IGatewayVpcEndpoint | The S3 VPC endpoint gateway. | | vpc | aws-cdk-lib.aws_ec2.IVpc | The amazon VPC created. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `flowLogGroup`^Required ```typescript public readonly flowLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group created for the VPC flow logs. --- ##### `flowLogKey`^Required ```typescript public readonly flowLogKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey The KMS Key used to encrypt VPC flow logs. --- ##### `flowLogRole`^Required ```typescript public readonly flowLogRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM role used to publish VPC Flow Logs. --- ##### `s3VpcEndpoint`^Required ```typescript public readonly s3VpcEndpoint: IGatewayVpcEndpoint; ``` - *Type:* aws-cdk-lib.aws_ec2.IGatewayVpcEndpoint The S3 VPC endpoint gateway. --- ##### `vpc`^Required ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc The amazon VPC created. --- ### PySparkApplicationPackage A construct that takes your PySpark application, packages its virtual environment and uploads it along its entrypoint to an Amazon S3 bucket This construct requires Docker daemon installed locally to run. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/pyspark-application-package](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/pyspark-application-package) *Example* ```typescript let pysparkPacker = new dsf.processing.PySparkApplicationPackage (this, 'pysparkPacker', { applicationName: 'my-pyspark', entrypointPath: '/Users/my-user/my-spark-job/app/app-pyspark.py', dependenciesFolder: '/Users/my-user/my-spark-job/app', removalPolicy: cdk.RemovalPolicy.DESTROY, }); ``` #### Initializers ```typescript import { processing } from 'aws-dsf' new processing.PySparkApplicationPackage(scope: Construct, id: string, props: PySparkApplicationPackageProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | the Scope of the CDK Construct. | | id | string | the ID of the CDK Construct. | | props | aws-dsf.processing.PySparkApplicationPackageProps | {@link PySparkApplicationPackageProps}. | --- ##### `scope`^Required - *Type:* constructs.Construct the Scope of the CDK Construct. --- ##### `id`^Required - *Type:* string the ID of the CDK Construct. --- ##### `props`^Required - *Type:* aws-dsf.processing.PySparkApplicationPackageProps {@link PySparkApplicationPackageProps}. --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { processing } from 'aws-dsf' processing.PySparkApplicationPackage.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | artifactsBucket | aws-cdk-lib.aws_s3.IBucket | The S3 Bucket for storing the artifacts (entrypoint and virtual environment archive). | | assetUploadManagedPolicy | aws-cdk-lib.aws_iam.IManagedPolicy | The IAM Managed Policy used by the custom resource for the assets deployment. | | assetUploadRole | aws-cdk-lib.aws_iam.IRole | The IAM Role used by the BucketDeployment to upload the artifacts to an s3 bucket. | | entrypointUri | string | The location (generally it's an S3 URI) where the entry point is saved. | | sparkVenvConf | string | The Spark Config containing the configuration of virtual environment archive with all dependencies. | | venvArchiveUri | string | The location (generally an S3 URI) where the archive of the Python virtual environment with all dependencies is stored. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `artifactsBucket`^Required ```typescript public readonly artifactsBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The S3 Bucket for storing the artifacts (entrypoint and virtual environment archive). --- ##### `assetUploadManagedPolicy`^Required ```typescript public readonly assetUploadManagedPolicy: IManagedPolicy; ``` - *Type:* aws-cdk-lib.aws_iam.IManagedPolicy The IAM Managed Policy used by the custom resource for the assets deployment. --- ##### `assetUploadRole`^Required ```typescript public readonly assetUploadRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role used by the BucketDeployment to upload the artifacts to an s3 bucket. In case you provide your own S3 Bucket for storing the artifacts (entrypoint and virtual environment archive), you must provide S3 write access to this role to upload the artifacts. --- ##### `entrypointUri`^Required ```typescript public readonly entrypointUri: string; ``` - *Type:* string The location (generally it's an S3 URI) where the entry point is saved. You can pass this location to your Spark job. --- ##### `sparkVenvConf`^Optional ```typescript public readonly sparkVenvConf: string; ``` - *Type:* string The Spark Config containing the configuration of virtual environment archive with all dependencies. --- ##### `venvArchiveUri`^Optional ```typescript public readonly venvArchiveUri: string; ``` - *Type:* string The location (generally an S3 URI) where the archive of the Python virtual environment with all dependencies is stored. You can pass this location to your Spark job. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | ARTIFACTS_PREFIX | string | The prefix used to store artifacts on the artifact bucket. | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `ARTIFACTS_PREFIX`^Required ```typescript public readonly ARTIFACTS_PREFIX: string; ``` - *Type:* string The prefix used to store artifacts on the artifact bucket. --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### RedshiftData Creates an asynchronous custom resource that handles the execution of SQL using Redshift's Data API. If `vpc` and `vpcSubnets` are passed, this construct would also create the Redshift Data Interface VPC endpoint and configure the custom resource in the same VPC subnet. *Example* ```typescript const namespace = new dsf.consumption.RedshiftServerlessNamespace(this, 'RedshiftNamespace', { name: "default", dbName: 'defaultdb', }); const workgroup = new dsf.consumption.RedshiftServerlessWorkgroup(this, "RedshiftWorkgroup", { name: "redshift-workgroup", namespace: namespace, }); const rsData = workgroup.accessData('DataApi'); rsData.createDbRole("EngineeringRole", "defaultdb", "engineering"); ``` #### Initializers ```typescript import { consumption } from 'aws-dsf' new consumption.RedshiftData(scope: Construct, id: string, props: RedshiftDataProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-dsf.consumption.RedshiftDataProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.consumption.RedshiftDataProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | assignDbRolesToIAMRole | Assigns Redshift DB roles to IAM role vs the `RedshiftDbRoles` tag. | | createDbRole | Creates a new DB role. | | grantDbAllPrivilegesToRole | Grants both read and write permissions on all the tables in the `schema` to the DB role. | | grantDbSchemaToRole | Grants access to the schema to the DB role. | | grantSchemaReadToRole | Grants read permission on all the tables in the `schema` to the DB role. | | ingestData | Ingest data from S3 into a Redshift table. | | mergeToTargetTable | Run the `MERGE` query using simplified mode. | | retrieveVersion | Retrieve DSF package.json version. | | runCustomSQL | Runs a custom SQL. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `assignDbRolesToIAMRole` ```typescript public assignDbRolesToIAMRole(dbRoles: string[], targetRole: IRole): void ``` Assigns Redshift DB roles to IAM role vs the `RedshiftDbRoles` tag. ###### `dbRoles`^Required - *Type:* string[] List of Redshift DB roles to assign to IAM role. --- ###### `targetRole`^Required - *Type:* aws-cdk-lib.aws_iam.IRole The IAM role to assign the Redshift DB roles to. --- ##### `createDbRole` ```typescript public createDbRole(id: string, databaseName: string, roleName: string): CustomResource ``` Creates a new DB role. ###### `id`^Required - *Type:* string The CDK Construct ID. --- ###### `databaseName`^Required - *Type:* string The name of the database to run this command. --- ###### `roleName`^Required - *Type:* string The name of the role to create. --- ##### `grantDbAllPrivilegesToRole` ```typescript public grantDbAllPrivilegesToRole(id: string, databaseName: string, schema: string, roleName: string): CustomResource ``` Grants both read and write permissions on all the tables in the `schema` to the DB role. ###### `id`^Required - *Type:* string The CDK Construct ID. --- ###### `databaseName`^Required - *Type:* string The name of the database to run this command. --- ###### `schema`^Required - *Type:* string The schema where the tables are located in. --- ###### `roleName`^Required - *Type:* string The DB role to grant the permissions to. --- ##### `grantDbSchemaToRole` ```typescript public grantDbSchemaToRole(id: string, databaseName: string, schema: string, roleName: string): CustomResource ``` Grants access to the schema to the DB role. ###### `id`^Required - *Type:* string The CDK Construct ID. --- ###### `databaseName`^Required - *Type:* string The name of the database to run this command. --- ###### `schema`^Required - *Type:* string The schema where the tables are located in. --- ###### `roleName`^Required - *Type:* string The DB role to grant the permissions to. --- ##### `grantSchemaReadToRole` ```typescript public grantSchemaReadToRole(id: string, databaseName: string, schema: string, roleName: string): CustomResource ``` Grants read permission on all the tables in the `schema` to the DB role. ###### `id`^Required - *Type:* string --- ###### `databaseName`^Required - *Type:* string The name of the database to run this command. --- ###### `schema`^Required - *Type:* string The schema where the tables are located in. --- ###### `roleName`^Required - *Type:* string The DB role to grant the permissions to. --- ##### `ingestData` ```typescript public ingestData(id: string, databaseName: string, targetTable: string, sourceBucket: IBucket, sourcePrefix: string, ingestAdditionalOptions?: string, role?: IRole): CustomResource ``` Ingest data from S3 into a Redshift table. ###### `id`^Required - *Type:* string The CDK Construct ID. --- ###### `databaseName`^Required - *Type:* string The name of the database to run this command. --- ###### `targetTable`^Required - *Type:* string The target table to load the data into. --- ###### `sourceBucket`^Required - *Type:* aws-cdk-lib.aws_s3.IBucket The bucket where the source data would be coming from. --- ###### `sourcePrefix`^Required - *Type:* string The location inside the bucket where the data would be ingested from. --- ###### `ingestAdditionalOptions`^Optional - *Type:* string Optional. Additional options to pass to the `COPY` command. For example, `delimiter '|'` or `ignoreheader 1` --- ###### `role`^Optional - *Type:* aws-cdk-lib.aws_iam.IRole Optional. The IAM Role to use to access the data in S3. If not provided, it would use the default IAM role configured in the Redshift Namespace --- ##### `mergeToTargetTable` ```typescript public mergeToTargetTable(id: string, databaseName: string, sourceTable: string, targetTable: string, sourceColumnId?: string, targetColumnId?: string): CustomResource ``` Run the `MERGE` query using simplified mode. This command would do an upsert into the target table. ###### `id`^Required - *Type:* string The CDK Construct ID. --- ###### `databaseName`^Required - *Type:* string The name of the database to run this command. --- ###### `sourceTable`^Required - *Type:* string The source table name. Schema can also be included using the following format: `schemaName.tableName` --- ###### `targetTable`^Required - *Type:* string The target table name. Schema can also be included using the following format: `schemaName.tableName` --- ###### `sourceColumnId`^Optional - *Type:* string The column in the source table that's used to determine whether the rows in the `sourceTable` can be matched with rows in the `targetTable`. Default is `id` --- ###### `targetColumnId`^Optional - *Type:* string The column in the target table that's used to determine whether the rows in the `sourceTable` can be matched with rows in the `targetTable`. Default is `id` --- ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. ##### `runCustomSQL` ```typescript public runCustomSQL(id: string, databaseName: string, sql: string, deleteSql?: string): CustomResource ``` Runs a custom SQL. Once the custom resource finishes execution, the attribute `Data` contains an attribute `execId` which contains the Redshift Data API execution ID. You can then use this to retrieve execution results via the `GetStatementResult` API. ###### `id`^Required - *Type:* string The CDK Construct ID. --- ###### `databaseName`^Required - *Type:* string The name of the database to run this command. --- ###### `sql`^Required - *Type:* string The sql to run. --- ###### `deleteSql`^Optional - *Type:* string Optional. The sql to run when this resource gets deleted --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { consumption } from 'aws-dsf' consumption.RedshiftData.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | executionRole | aws-cdk-lib.aws_iam.IRole | The IAM Role for the Redshift Data API execution. | | statusFunction | aws-cdk-lib.aws_lambda.IFunction | The Lambda Function for the Redshift Data API status checks. | | statusLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group for the Redshift Data API status checks. | | submitFunction | aws-cdk-lib.aws_lambda.IFunction | The Lambda Function for the Redshift Data submission. | | submitLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group for the Redshift Data API submission. | | taggingManagedPolicy | aws-cdk-lib.aws_iam.IManagedPolicy | The managed IAM policy allowing IAM Role to retrieve tag information. | | targetArn | string | The ARN of the target cluster or workgroup. | | targetId | string | The ID of the target cluster or workgroup. | | cleanUpFunction | aws-cdk-lib.aws_lambda.IFunction | The Lambda function for the S3 data copy cleaning up lambda. | | cleanUpLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group for the Redshift Data cleaning up lambda. | | cleanUpRole | aws-cdk-lib.aws_iam.IRole | The IAM Role for the the S3 data copy cleaning up lambda. | | customResourceSecurityGroup | aws-cdk-lib.aws_ec2.ISecurityGroup | The Security Group used by the Custom Resource when deployed in a VPC. | | vpcEndpoint | aws-cdk-lib.aws_ec2.InterfaceVpcEndpoint | The created Redshift Data API interface vpc endpoint when deployed in a VPC. | | vpcEndpointSecurityGroup | aws-cdk-lib.aws_ec2.ISecurityGroup | The Security Group used by the VPC Endpoint when deployed in a VPC. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `executionRole`^Required ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role for the Redshift Data API execution. --- ##### `statusFunction`^Required ```typescript public readonly statusFunction: IFunction; ``` - *Type:* aws-cdk-lib.aws_lambda.IFunction The Lambda Function for the Redshift Data API status checks. --- ##### `statusLogGroup`^Required ```typescript public readonly statusLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group for the Redshift Data API status checks. --- ##### `submitFunction`^Required ```typescript public readonly submitFunction: IFunction; ``` - *Type:* aws-cdk-lib.aws_lambda.IFunction The Lambda Function for the Redshift Data submission. --- ##### `submitLogGroup`^Required ```typescript public readonly submitLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group for the Redshift Data API submission. --- ##### `taggingManagedPolicy`^Required ```typescript public readonly taggingManagedPolicy: IManagedPolicy; ``` - *Type:* aws-cdk-lib.aws_iam.IManagedPolicy The managed IAM policy allowing IAM Role to retrieve tag information. --- ##### `targetArn`^Required ```typescript public readonly targetArn: string; ``` - *Type:* string The ARN of the target cluster or workgroup. --- ##### `targetId`^Required ```typescript public readonly targetId: string; ``` - *Type:* string The ID of the target cluster or workgroup. --- ##### `cleanUpFunction`^Optional ```typescript public readonly cleanUpFunction: IFunction; ``` - *Type:* aws-cdk-lib.aws_lambda.IFunction The Lambda function for the S3 data copy cleaning up lambda. --- ##### `cleanUpLogGroup`^Optional ```typescript public readonly cleanUpLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group for the Redshift Data cleaning up lambda. --- ##### `cleanUpRole`^Optional ```typescript public readonly cleanUpRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role for the the S3 data copy cleaning up lambda. --- ##### `customResourceSecurityGroup`^Optional ```typescript public readonly customResourceSecurityGroup: ISecurityGroup; ``` - *Type:* aws-cdk-lib.aws_ec2.ISecurityGroup The Security Group used by the Custom Resource when deployed in a VPC. --- ##### `vpcEndpoint`^Optional ```typescript public readonly vpcEndpoint: InterfaceVpcEndpoint; ``` - *Type:* aws-cdk-lib.aws_ec2.InterfaceVpcEndpoint The created Redshift Data API interface vpc endpoint when deployed in a VPC. --- ##### `vpcEndpointSecurityGroup`^Optional ```typescript public readonly vpcEndpointSecurityGroup: ISecurityGroup; ``` - *Type:* aws-cdk-lib.aws_ec2.ISecurityGroup The Security Group used by the VPC Endpoint when deployed in a VPC. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### RedshiftServerlessNamespace Create a Redshift Serverless Namespace with the admin credentials stored in Secrets Manager. *Example* ```typescript const namespace = new dsf.consumption.RedshiftServerlessNamespace(this, 'DefaultServerlessNamespace', { dbName: 'defaultdb', name: 'default' }); ``` #### Initializers ```typescript import { consumption } from 'aws-dsf' new consumption.RedshiftServerlessNamespace(scope: Construct, id: string, props: RedshiftServerlessNamespaceProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-dsf.consumption.RedshiftServerlessNamespaceProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.consumption.RedshiftServerlessNamespaceProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { consumption } from 'aws-dsf' consumption.RedshiftServerlessNamespace.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | adminSecret | aws-cdk-lib.aws_secretsmanager.ISecret | The created Secrets Manager secret containing the admin credentials. | | adminSecretKey | aws-cdk-lib.aws_kms.IKey | The KMS Key used to encrypt the admin credentials secret. | | createFunction | aws-cdk-lib.aws_lambda.IFunction | The Lambda Function for the Redshift Serverless creation. | | createLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Logs Log Group for the Redshift Serverless creation. | | createRole | aws-cdk-lib.aws_iam.IRole | The IAM Role for the Redshift Serverless creation. | | customResource | aws-cdk-lib.CustomResource | The custom resource that creates the Namespace. | | dataKey | aws-cdk-lib.aws_kms.Key | KMS key used by the namespace to encrypt the data. | | dbName | string | The name of the database. | | namespaceArn | string | The ARN of the created namespace. | | namespaceId | string | The ID of the created namespace. | | namespaceName | string | The name of the created namespace. | | roles | {[ key: string ]: aws-cdk-lib.aws_iam.IRole} | The roles attached to the namespace in the form of `{RoleArn: IRole}`. | | statusFunction | aws-cdk-lib.aws_lambda.IFunction | The Lambda Function for the creation status check. | | statusLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Logs Log Group for the creation status check. | | statusRole | aws-cdk-lib.aws_iam.IRole | The IAM Role for the creation status check. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `adminSecret`^Required ```typescript public readonly adminSecret: ISecret; ``` - *Type:* aws-cdk-lib.aws_secretsmanager.ISecret The created Secrets Manager secret containing the admin credentials. --- ##### `adminSecretKey`^Required ```typescript public readonly adminSecretKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey The KMS Key used to encrypt the admin credentials secret. --- ##### `createFunction`^Required ```typescript public readonly createFunction: IFunction; ``` - *Type:* aws-cdk-lib.aws_lambda.IFunction The Lambda Function for the Redshift Serverless creation. --- ##### `createLogGroup`^Required ```typescript public readonly createLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Logs Log Group for the Redshift Serverless creation. --- ##### `createRole`^Required ```typescript public readonly createRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role for the Redshift Serverless creation. --- ##### `customResource`^Required ```typescript public readonly customResource: CustomResource; ``` - *Type:* aws-cdk-lib.CustomResource The custom resource that creates the Namespace. --- ##### `dataKey`^Required ```typescript public readonly dataKey: Key; ``` - *Type:* aws-cdk-lib.aws_kms.Key KMS key used by the namespace to encrypt the data. --- ##### `dbName`^Required ```typescript public readonly dbName: string; ``` - *Type:* string The name of the database. --- ##### `namespaceArn`^Required ```typescript public readonly namespaceArn: string; ``` - *Type:* string The ARN of the created namespace. --- ##### `namespaceId`^Required ```typescript public readonly namespaceId: string; ``` - *Type:* string The ID of the created namespace. --- ##### `namespaceName`^Required ```typescript public readonly namespaceName: string; ``` - *Type:* string The name of the created namespace. --- ##### `roles`^Required ```typescript public readonly roles: {[ key: string ]: IRole}; ``` - *Type:* {[ key: string ]: aws-cdk-lib.aws_iam.IRole} The roles attached to the namespace in the form of `{RoleArn: IRole}`. These roles are used to access other AWS services for ingestion, federated query, and data catalog access. > [https://docs.aws.amazon.com/redshift/latest/mgmt/redshift-iam-authentication-access-control.html](https://docs.aws.amazon.com/redshift/latest/mgmt/redshift-iam-authentication-access-control.html) --- ##### `statusFunction`^Required ```typescript public readonly statusFunction: IFunction; ``` - *Type:* aws-cdk-lib.aws_lambda.IFunction The Lambda Function for the creation status check. --- ##### `statusLogGroup`^Required ```typescript public readonly statusLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Logs Log Group for the creation status check. --- ##### `statusRole`^Required ```typescript public readonly statusRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role for the creation status check. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### RedshiftServerlessWorkgroup - *Implements:* aws-cdk-lib.aws_ec2.IConnectable Create a Redshift Serverless Workgroup. A default namespace would be created if none is provided. *Example* ```typescript const workgroup = new dsf.consumption.RedshiftServerlessWorkgroup(this, "RedshiftWorkgroup", { name: "example-workgroup", namespace: new dsf.consumption.RedshiftServerlessNamespace(this, "RedshiftNamespace", { name: 'example-namespace', dbName: 'defaultdb', }) }); ``` #### Initializers ```typescript import { consumption } from 'aws-dsf' new consumption.RedshiftServerlessWorkgroup(scope: Construct, id: string, props: RedshiftServerlessWorkgroupProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-dsf.consumption.RedshiftServerlessWorkgroupProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.consumption.RedshiftServerlessWorkgroupProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | accessData | Creates an instance of `RedshiftData` to send custom SQLs to the workgroup. | | catalogTables | Creates a new Glue data catalog database with a crawler using JDBC target type to connect to the Redshift Workgroup. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `accessData` ```typescript public accessData(id: string, createVpcEndpoint?: boolean): RedshiftData ``` Creates an instance of `RedshiftData` to send custom SQLs to the workgroup. ###### `id`^Required - *Type:* string --- ###### `createVpcEndpoint`^Optional - *Type:* boolean if set to true, create interface VPC endpoint for Redshift Data API. --- ##### `catalogTables` ```typescript public catalogTables(id: string, catalogDbName: string, pathToCrawl?: string): DataCatalogDatabase ``` Creates a new Glue data catalog database with a crawler using JDBC target type to connect to the Redshift Workgroup. ###### `id`^Required - *Type:* string The CDK ID of the resource. --- ###### `catalogDbName`^Required - *Type:* string The name of the Glue Database to create. --- ###### `pathToCrawl`^Optional - *Type:* string The path of Redshift tables to crawl. --- ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { consumption } from 'aws-dsf' consumption.RedshiftServerlessWorkgroup.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | cfnResource | aws-cdk-lib.aws_redshiftserverless.CfnWorkgroup | The created Redshift Serverless Workgroup. | | connections | aws-cdk-lib.aws_ec2.Connections | Connections used by Workgroup security group. | | glueConnection | aws-cdk-lib.aws_glue.CfnConnection | The Glue Connection associated with the workgroup. | | namespace | aws-dsf.consumption.RedshiftServerlessNamespace | The associated Redshift Serverless Namespace. | | primarySecurityGroup | aws-cdk-lib.aws_ec2.ISecurityGroup | The primary EC2 Security Group associated with the Redshift Serverless Workgroup. | | selectedSubnets | aws-cdk-lib.aws_ec2.SelectedSubnets | The subnets where the Redshift Serverless Workgroup is deployed. | | vpc | aws-cdk-lib.aws_ec2.IVpc | The VPC where the Redshift Serverless Workgroup is deployed. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `cfnResource`^Required ```typescript public readonly cfnResource: CfnWorkgroup; ``` - *Type:* aws-cdk-lib.aws_redshiftserverless.CfnWorkgroup The created Redshift Serverless Workgroup. --- ##### `connections`^Required ```typescript public readonly connections: Connections; ``` - *Type:* aws-cdk-lib.aws_ec2.Connections Connections used by Workgroup security group. Used this to enable access from clients connecting to the workgroup --- ##### `glueConnection`^Required ```typescript public readonly glueConnection: CfnConnection; ``` - *Type:* aws-cdk-lib.aws_glue.CfnConnection The Glue Connection associated with the workgroup. This can be used by Glue ETL Jobs to read/write data from/to Redshift workgroup --- ##### `namespace`^Required ```typescript public readonly namespace: RedshiftServerlessNamespace; ``` - *Type:* aws-dsf.consumption.RedshiftServerlessNamespace The associated Redshift Serverless Namespace. --- ##### `primarySecurityGroup`^Required ```typescript public readonly primarySecurityGroup: ISecurityGroup; ``` - *Type:* aws-cdk-lib.aws_ec2.ISecurityGroup The primary EC2 Security Group associated with the Redshift Serverless Workgroup. --- ##### `selectedSubnets`^Required ```typescript public readonly selectedSubnets: SelectedSubnets; ``` - *Type:* aws-cdk-lib.aws_ec2.SelectedSubnets The subnets where the Redshift Serverless Workgroup is deployed. --- ##### `vpc`^Required ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc The VPC where the Redshift Serverless Workgroup is deployed. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### S3DataCopy Copy data from one S3 bucket to another. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Utils/s3-data-copy](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Utils/s3-data-copy) *Example* ```typescript import { Bucket } from 'aws-cdk-lib/aws-s3'; const sourceBucket = Bucket.fromBucketName(this, 'SourceBucket', 'nyc-tlc'); const bucketName = `test-${this.region}-${this.account}-${dsf.utils.Utils.generateUniqueHash(this, 'TargetBucket')}`; const targetBucket = new Bucket(this, 'TargetBucket'); new dsf.utils.S3DataCopy(this, 'S3DataCopy', { sourceBucket, sourceBucketPrefix: 'trip data/', sourceBucketRegion: 'us-east-1', targetBucket, }); ``` #### Initializers ```typescript import { utils } from 'aws-dsf' new utils.S3DataCopy(scope: Construct, id: string, props: S3DataCopyProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-dsf.utils.S3DataCopyProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.utils.S3DataCopyProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { utils } from 'aws-dsf' utils.S3DataCopy.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | copyFunction | aws-cdk-lib.aws_lambda.IFunction | The Lambda Function for the copy. | | copyLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group for the S3 data copy. | | copyRole | aws-cdk-lib.aws_iam.IRole | The IAM Role for the copy Lambba Function. | | cleanUpFunction | aws-cdk-lib.aws_lambda.IFunction | The Lambda function for the S3 data copy cleaning up lambda. | | cleanUpLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group for the S3 data copy cleaning up lambda. | | cleanUpRole | aws-cdk-lib.aws_iam.IRole | The IAM Role for the the S3 data copy cleaning up lambda. | | securityGroups | aws-cdk-lib.aws_ec2.ISecurityGroup[] | The list of EC2 Security Groups used by the Lambda Functions. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `copyFunction`^Required ```typescript public readonly copyFunction: IFunction; ``` - *Type:* aws-cdk-lib.aws_lambda.IFunction The Lambda Function for the copy. --- ##### `copyLogGroup`^Required ```typescript public readonly copyLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group for the S3 data copy. --- ##### `copyRole`^Required ```typescript public readonly copyRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role for the copy Lambba Function. --- ##### `cleanUpFunction`^Optional ```typescript public readonly cleanUpFunction: IFunction; ``` - *Type:* aws-cdk-lib.aws_lambda.IFunction The Lambda function for the S3 data copy cleaning up lambda. --- ##### `cleanUpLogGroup`^Optional ```typescript public readonly cleanUpLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group for the S3 data copy cleaning up lambda. --- ##### `cleanUpRole`^Optional ```typescript public readonly cleanUpRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role for the the S3 data copy cleaning up lambda. --- ##### `securityGroups`^Optional ```typescript public readonly securityGroups: ISecurityGroup[]; ``` - *Type:* aws-cdk-lib.aws_ec2.ISecurityGroup[] The list of EC2 Security Groups used by the Lambda Functions. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### SparkEmrCICDPipeline A CICD Pipeline to test and deploy a Spark application on Amazon EMR in cross-account environments using CDK Pipelines. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-cicd-pipeline](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-cicd-pipeline) *Example* ```typescript import { Bucket } from 'aws-cdk-lib/aws-s3'; interface MyApplicationStackProps extends cdk.StackProps { readonly stage: dsf.utils.CICDStage; } class MyApplicationStack extends cdk.Stack { constructor(scope: cdk.Stack, props?: MyApplicationStackProps) { super(scope, 'MyApplicationStack'); const bucket = new Bucket(this, 'TestBucket', { autoDeleteObjects: true, removalPolicy: cdk.RemovalPolicy.DESTROY, }); new cdk.CfnOutput(this, 'BucketName', { value: bucket.bucketName }); } } class MyStackFactory implements dsf.utils.ApplicationStackFactory { createStack(scope: cdk.Stack, stage: dsf.utils.CICDStage): cdk.Stack { return new MyApplicationStack(scope, { stage }); } } class MyCICDStack extends cdk.Stack { constructor(scope: Construct, id: string) { super(scope, id); new dsf.processing.SparkEmrCICDPipeline(this, 'TestConstruct', { sparkApplicationName: 'test', applicationStackFactory: new MyStackFactory(), cdkApplicationPath: 'cdk/', sparkApplicationPath: 'spark/', sparkImage: dsf.processing.SparkImage.EMR_6_12, integTestScript: 'cdk/integ-test.sh', integTestEnv: { TEST_BUCKET: 'BucketName', }, }); } } ``` #### Initializers ```typescript import { processing } from 'aws-dsf' new processing.SparkEmrCICDPipeline(scope: Construct, id: string, props: SparkEmrCICDPipelineProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | the Scope of the CDK Construct. | | id | string | the ID of the CDK Construct. | | props | aws-dsf.processing.SparkEmrCICDPipelineProps | the SparkCICDPipelineProps properties. | --- ##### `scope`^Required - *Type:* constructs.Construct the Scope of the CDK Construct. --- ##### `id`^Required - *Type:* string the ID of the CDK Construct. --- ##### `props`^Required - *Type:* aws-dsf.processing.SparkEmrCICDPipelineProps the SparkCICDPipelineProps properties. --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { processing } from 'aws-dsf' processing.SparkEmrCICDPipeline.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | artifactAccessLogsBucket | aws-cdk-lib.aws_s3.IBucket | The S3 Bucket for storing the access logs on the artifact S3 Bucket. | | artifactBucket | aws-cdk-lib.aws_s3.IBucket | The S3 Bucket for storing the artifacts. | | pipeline | aws-cdk-lib.pipelines.CodePipeline | The CodePipeline created as part of the Spark CICD Pipeline. | | pipelineLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group for storing the CodePipeline logs. | | repository | aws-cdk-lib.aws_codecommit.Repository | The CodeCommit Repository created as part of the Spark CICD Pipeline. | | integrationTestStage | aws-cdk-lib.pipelines.CodeBuildStep | The CodeBuild Step for the staging stage. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `artifactAccessLogsBucket`^Required ```typescript public readonly artifactAccessLogsBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The S3 Bucket for storing the access logs on the artifact S3 Bucket. --- ##### `artifactBucket`^Required ```typescript public readonly artifactBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The S3 Bucket for storing the artifacts. --- ##### `pipeline`^Required ```typescript public readonly pipeline: CodePipeline; ``` - *Type:* aws-cdk-lib.pipelines.CodePipeline The CodePipeline created as part of the Spark CICD Pipeline. --- ##### `pipelineLogGroup`^Required ```typescript public readonly pipelineLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group for storing the CodePipeline logs. --- ##### `repository`^Required ```typescript public readonly repository: Repository; ``` - *Type:* aws-cdk-lib.aws_codecommit.Repository The CodeCommit Repository created as part of the Spark CICD Pipeline. --- ##### `integrationTestStage`^Optional ```typescript public readonly integrationTestStage: CodeBuildStep; ``` - *Type:* aws-cdk-lib.pipelines.CodeBuildStep The CodeBuild Step for the staging stage. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### SparkEmrContainersJob A construct to run Spark Jobs using EMR Container runtime (EMR on EKS). It creates a Step Functions State Machine that orchestrates the Spark Job. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-emr-serverless-job](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-emr-serverless-job) *Example* ```typescript import { JsonPath } from 'aws-cdk-lib/aws-stepfunctions'; const job = new dsf.processing.SparkEmrContainersJob(this, 'SparkJob', { jobConfig:{ "Name": JsonPath.format('ge_profile-{}', JsonPath.uuid()), "VirtualClusterId": "virtualClusterId", "ExecutionRoleArn": "ROLE-ARN", "JobDriver": { "SparkSubmit": { "EntryPoint": "s3://S3-BUCKET/pi.py", "EntryPointArguments": [], "SparkSubmitParameters": "--conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.driver.memory=2G --conf spark.executor.cores=4" }, } } } as dsf.processing.SparkEmrContainersJobApiProps); new cdk.CfnOutput(this, 'SparkJobStateMachine', { value: job.stateMachine!.stateMachineArn, }); ``` #### Initializers ```typescript import { processing } from 'aws-dsf' new processing.SparkEmrContainersJob(scope: Construct, id: string, props: SparkEmrContainersJobProps | SparkEmrContainersJobApiProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-dsf.processing.SparkEmrContainersJobProps \| aws-dsf.processing.SparkEmrContainersJobApiProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.processing.SparkEmrContainersJobProps | aws-dsf.processing.SparkEmrContainersJobApiProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { processing } from 'aws-dsf' processing.SparkEmrContainersJob.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | stateMachine | aws-cdk-lib.aws_stepfunctions.StateMachine | The Step Functions State Machine created to orchestrate the Spark Job. | | stateMachineLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group used by the State Machine. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `stateMachine`^Optional ```typescript public readonly stateMachine: StateMachine; ``` - *Type:* aws-cdk-lib.aws_stepfunctions.StateMachine The Step Functions State Machine created to orchestrate the Spark Job. --- ##### `stateMachineLogGroup`^Optional ```typescript public readonly stateMachineLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group used by the State Machine. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### SparkEmrContainersRuntime A construct to create an EKS cluster, configure it and enable it with EMR on EKS. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-emr-containers-runtime](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-emr-containers-runtime) *Example* ```typescript import { ManagedPolicy, PolicyDocument, PolicyStatement } from 'aws-cdk-lib/aws-iam'; import { KubectlV27Layer } from '@aws-cdk/lambda-layer-kubectl-v27'; const kubectlLayer = new KubectlV27Layer(this, 'kubectlLayer'); const emrEksCluster = dsf.processing.SparkEmrContainersRuntime.getOrCreate(this, { publicAccessCIDRs: ['10.0.0.0/16'], kubectlLambdaLayer: kubectlLayer, }); const virtualCluster = emrEksCluster.addEmrVirtualCluster(this, { name: 'example', createNamespace: true, eksNamespace: 'example', }); const s3Read = new PolicyDocument({ statements: [new PolicyStatement({ actions: [ 's3:GetObject', ], resources: ['arn:aws:s3:::aws-data-analytics-workshop'], })], }); const s3ReadPolicy = new ManagedPolicy(this, 's3ReadPolicy', { document: s3Read, }); const execRole = emrEksCluster.createExecutionRole(this, 'ExecRole', s3ReadPolicy, 'example', 's3ReadExecRole'); ``` #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | addEmrVirtualCluster | Add a new Amazon EMR Virtual Cluster linked to Amazon EKS Cluster. | | addInteractiveEndpoint | Creates a new Amazon EMR managed endpoint to be used with Amazon EMR Virtual Cluster . | | addKarpenterNodePoolAndNodeClass | Apply the provided manifest and add the CDK dependency on EKS cluster. | | createExecutionRole | Create and configure a new Amazon IAM Role usable as an execution role. | | retrieveVersion | Retrieve DSF package.json version. | | uploadPodTemplate | Upload podTemplates to the Amazon S3 location used by the cluster. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `addEmrVirtualCluster` ```typescript public addEmrVirtualCluster(scope: Construct, options: EmrVirtualClusterProps): CfnVirtualCluster ``` Add a new Amazon EMR Virtual Cluster linked to Amazon EKS Cluster. ###### `scope`^Required - *Type:* constructs.Construct of the stack where virtual cluster is deployed. --- ###### `options`^Required - *Type:* aws-dsf.processing.EmrVirtualClusterProps the EmrVirtualClusterProps [properties]{@link EmrVirtualClusterProps}. --- ##### `addInteractiveEndpoint` ```typescript public addInteractiveEndpoint(scope: Construct, id: string, interactiveSessionOptions: SparkEmrContainersRuntimeInteractiveSessionProps): CustomResource ``` Creates a new Amazon EMR managed endpoint to be used with Amazon EMR Virtual Cluster . CfnOutput can be customized. ###### `scope`^Required - *Type:* constructs.Construct the scope of the stack where managed endpoint is deployed. --- ###### `id`^Required - *Type:* string the CDK id for endpoint. --- ###### `interactiveSessionOptions`^Required - *Type:* aws-dsf.processing.SparkEmrContainersRuntimeInteractiveSessionProps the EmrManagedEndpointOptions to configure the Amazon EMR managed endpoint. --- ##### `addKarpenterNodePoolAndNodeClass` ```typescript public addKarpenterNodePoolAndNodeClass(id: string, manifest: any): any ``` Apply the provided manifest and add the CDK dependency on EKS cluster. ###### `id`^Required - *Type:* string the unique ID of the CDK resource. --- ###### `manifest`^Required - *Type:* any The manifest to apply. You can use the Utils class that offers method to read yaml file and load it as a manifest --- ##### `createExecutionRole` ```typescript public createExecutionRole(scope: Construct, id: string, policy: IManagedPolicy, eksNamespace: string, name: string): Role ``` Create and configure a new Amazon IAM Role usable as an execution role. This method makes the created role assumed by the Amazon EKS cluster Open ID Connect provider. ###### `scope`^Required - *Type:* constructs.Construct of the IAM role. --- ###### `id`^Required - *Type:* string of the CDK resource to be created, it should be unique across the stack. --- ###### `policy`^Required - *Type:* aws-cdk-lib.aws_iam.IManagedPolicy the execution policy to attach to the role. --- ###### `eksNamespace`^Required - *Type:* string The namespace from which the role is going to be used. MUST be the same as the namespace of the Virtual Cluster from which the job is submitted --- ###### `name`^Required - *Type:* string Name to use for the role, required and is used to scope the iam role. --- ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. ##### `uploadPodTemplate` ```typescript public uploadPodTemplate(id: string, filePath: string): void ``` Upload podTemplates to the Amazon S3 location used by the cluster. ###### `id`^Required - *Type:* string the unique ID of the CDK resource. --- ###### `filePath`^Required - *Type:* string The local path of the yaml podTemplate files to upload. --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | | getOrCreate | Get an existing EmrEksCluster based on the cluster name property or create a new one only one EKS cluster can exist per stack. | | grantStartJobExecution | A static method granting the right to start and monitor a job to an IAM Role. | --- ##### `isConstruct` ```typescript import { processing } from 'aws-dsf' processing.SparkEmrContainersRuntime.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- ##### `getOrCreate` ```typescript import { processing } from 'aws-dsf' processing.SparkEmrContainersRuntime.getOrCreate(scope: Construct, props: SparkEmrContainersRuntimeProps) ``` Get an existing EmrEksCluster based on the cluster name property or create a new one only one EKS cluster can exist per stack. ###### `scope`^Required - *Type:* constructs.Construct the CDK scope used to search or create the cluster. --- ###### `props`^Required - *Type:* aws-dsf.processing.SparkEmrContainersRuntimeProps the EmrEksClusterProps [properties]{@link EmrEksClusterProps } if created. --- ##### `grantStartJobExecution` ```typescript import { processing } from 'aws-dsf' processing.SparkEmrContainersRuntime.grantStartJobExecution(startJobRole: IRole, executionRoleArn: string[], virtualClusterArn: string) ``` A static method granting the right to start and monitor a job to an IAM Role. The method will scope the following actions `DescribeJobRun`, `TagResource` and `ListJobRuns` to the provided virtual cluster. It will also scope `StartJobRun` as defined in the [EMR on EKS official documentation](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/iam-execution-role.html) ###### `startJobRole`^Required - *Type:* aws-cdk-lib.aws_iam.IRole the role that will call the start job api and which needs to have the iam:PassRole permission. --- ###### `executionRoleArn`^Required - *Type:* string[] the role used by EMR on EKS to access resources during the job execution. --- ###### `virtualClusterArn`^Required - *Type:* string the EMR Virtual Cluster ARN to which the job is submitted. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | ec2InstanceNodeGroupRole | aws-cdk-lib.aws_iam.IRole | The IAM role used by the tooling managed nodegroup hosting core Kubernetes controllers like EBS CSI driver, core dns. | | eksCluster | aws-cdk-lib.aws_eks.Cluster | The EKS cluster created by the construct if it is not provided. | | vpc | aws-cdk-lib.aws_ec2.IVpc | The VPC used by the EKS cluster. | | assetBucket | aws-cdk-lib.aws_s3.IBucket | The bucket holding podtemplates referenced in the configuration override for the job. | | assetUploadBucketRole | aws-cdk-lib.aws_iam.IRole | The IAM role used to upload assets (pod templates) on S3. | | awsNodeRole | aws-cdk-lib.aws_iam.IRole | The IAM Role used by IRSA for the aws-node daemonset. | | criticalDefaultConfig | string | The configuration override for the spark application to use with the default nodes for criticale jobs. | | csiDriverIrsaRole | aws-cdk-lib.aws_iam.IRole | The IAM Role created for the EBS CSI controller. | | eksSecretKmsKey | aws-cdk-lib.aws_kms.IKey | The KMS key used for storing EKS secrets. | | emrServiceRole | aws-cdk-lib.aws_iam.CfnServiceLinkedRole | The Service Linked role created for EMR. | | flowLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group for the VPC flow log when the VPC is created. | | flowLogKey | aws-cdk-lib.aws_kms.IKey | The KMS Key used for the VPC flow logs when the VPC is created. | | flowLogRole | aws-cdk-lib.aws_iam.IRole | The IAM Role used for the VPC flow logs when the VPC is created. | | karpenterEventRules | aws-cdk-lib.aws_events.IRule[] | The rules used by Karpenter to track node health, rules are defined in the cloudformation below https://raw.githubusercontent.com/aws/karpenter/"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml. | | karpenterIrsaRole | aws-cdk-lib.aws_iam.IRole | The IAM role created for the Karpenter controller. | | karpenterQueue | aws-cdk-lib.aws_sqs.IQueue | The SQS queue used by Karpenter to receive critical events from AWS services which may affect your nodes. | | karpenterSecurityGroup | aws-cdk-lib.aws_ec2.ISecurityGroup | The security group used by the EC2NodeClass of the default nodes. | | notebookDefaultConfig | any | The configuration override for the spark application to use with the default nodes dedicated for notebooks. | | podTemplateS3LocationCriticalDriver | string | The S3 location holding the driver pod tempalte for critical nodes. | | podTemplateS3LocationCriticalExecutor | string | The S3 location holding the executor pod tempalte for critical nodes. | | podTemplateS3LocationDriverShared | string | The S3 location holding the driver pod tempalte for shared nodes. | | podTemplateS3LocationExecutorShared | string | The S3 location holding the executor pod tempalte for shared nodes. | | podTemplateS3LocationNotebookDriver | string | The S3 location holding the driver pod tempalte for interactive sessions. | | podTemplateS3LocationNotebookExecutor | string | The S3 location holding the executor pod tempalte for interactive sessions. | | s3VpcEndpoint | aws-cdk-lib.aws_ec2.IGatewayVpcEndpoint | The S3 VPC endpoint attached to the private subnets of the VPC when VPC is created. | | sharedDefaultConfig | string | The configuration override for the spark application to use with the default nodes for none criticale jobs. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `ec2InstanceNodeGroupRole`^Required ```typescript public readonly ec2InstanceNodeGroupRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM role used by the tooling managed nodegroup hosting core Kubernetes controllers like EBS CSI driver, core dns. --- ##### `eksCluster`^Required ```typescript public readonly eksCluster: Cluster; ``` - *Type:* aws-cdk-lib.aws_eks.Cluster The EKS cluster created by the construct if it is not provided. --- ##### `vpc`^Required ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc The VPC used by the EKS cluster. --- ##### `assetBucket`^Optional ```typescript public readonly assetBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The bucket holding podtemplates referenced in the configuration override for the job. --- ##### `assetUploadBucketRole`^Optional ```typescript public readonly assetUploadBucketRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM role used to upload assets (pod templates) on S3. --- ##### `awsNodeRole`^Optional ```typescript public readonly awsNodeRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role used by IRSA for the aws-node daemonset. --- ##### `criticalDefaultConfig`^Optional ```typescript public readonly criticalDefaultConfig: string; ``` - *Type:* string The configuration override for the spark application to use with the default nodes for criticale jobs. --- ##### `csiDriverIrsaRole`^Optional ```typescript public readonly csiDriverIrsaRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role created for the EBS CSI controller. --- ##### `eksSecretKmsKey`^Optional ```typescript public readonly eksSecretKmsKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey The KMS key used for storing EKS secrets. --- ##### `emrServiceRole`^Optional ```typescript public readonly emrServiceRole: CfnServiceLinkedRole; ``` - *Type:* aws-cdk-lib.aws_iam.CfnServiceLinkedRole The Service Linked role created for EMR. --- ##### `flowLogGroup`^Optional ```typescript public readonly flowLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group for the VPC flow log when the VPC is created. --- ##### `flowLogKey`^Optional ```typescript public readonly flowLogKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey The KMS Key used for the VPC flow logs when the VPC is created. --- ##### `flowLogRole`^Optional ```typescript public readonly flowLogRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role used for the VPC flow logs when the VPC is created. --- ##### `karpenterEventRules`^Optional ```typescript public readonly karpenterEventRules: IRule[]; ``` - *Type:* aws-cdk-lib.aws_events.IRule[] The rules used by Karpenter to track node health, rules are defined in the cloudformation below https://raw.githubusercontent.com/aws/karpenter/"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml. --- ##### `karpenterIrsaRole`^Optional ```typescript public readonly karpenterIrsaRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM role created for the Karpenter controller. --- ##### `karpenterQueue`^Optional ```typescript public readonly karpenterQueue: IQueue; ``` - *Type:* aws-cdk-lib.aws_sqs.IQueue The SQS queue used by Karpenter to receive critical events from AWS services which may affect your nodes. --- ##### `karpenterSecurityGroup`^Optional ```typescript public readonly karpenterSecurityGroup: ISecurityGroup; ``` - *Type:* aws-cdk-lib.aws_ec2.ISecurityGroup The security group used by the EC2NodeClass of the default nodes. --- ##### `notebookDefaultConfig`^Optional ```typescript public readonly notebookDefaultConfig: any; ``` - *Type:* any The configuration override for the spark application to use with the default nodes dedicated for notebooks. --- ##### `podTemplateS3LocationCriticalDriver`^Optional ```typescript public readonly podTemplateS3LocationCriticalDriver: string; ``` - *Type:* string The S3 location holding the driver pod tempalte for critical nodes. --- ##### `podTemplateS3LocationCriticalExecutor`^Optional ```typescript public readonly podTemplateS3LocationCriticalExecutor: string; ``` - *Type:* string The S3 location holding the executor pod tempalte for critical nodes. --- ##### `podTemplateS3LocationDriverShared`^Optional ```typescript public readonly podTemplateS3LocationDriverShared: string; ``` - *Type:* string The S3 location holding the driver pod tempalte for shared nodes. --- ##### `podTemplateS3LocationExecutorShared`^Optional ```typescript public readonly podTemplateS3LocationExecutorShared: string; ``` - *Type:* string The S3 location holding the executor pod tempalte for shared nodes. --- ##### `podTemplateS3LocationNotebookDriver`^Optional ```typescript public readonly podTemplateS3LocationNotebookDriver: string; ``` - *Type:* string The S3 location holding the driver pod tempalte for interactive sessions. --- ##### `podTemplateS3LocationNotebookExecutor`^Optional ```typescript public readonly podTemplateS3LocationNotebookExecutor: string; ``` - *Type:* string The S3 location holding the executor pod tempalte for interactive sessions. --- ##### `s3VpcEndpoint`^Optional ```typescript public readonly s3VpcEndpoint: IGatewayVpcEndpoint; ``` - *Type:* aws-cdk-lib.aws_ec2.IGatewayVpcEndpoint The S3 VPC endpoint attached to the private subnets of the VPC when VPC is created. --- ##### `sharedDefaultConfig`^Optional ```typescript public readonly sharedDefaultConfig: string; ``` - *Type:* string The configuration override for the spark application to use with the default nodes for none criticale jobs. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DEFAULT_CLUSTER_NAME | string | The default name of the EKS cluster. | | DEFAULT_EKS_VERSION | aws-cdk-lib.aws_eks.KubernetesVersion | The default EKS version. | | DEFAULT_EMR_EKS_VERSION | aws-dsf.processing.EmrContainersRuntimeVersion | The default EMR on EKS version. | | DEFAULT_VPC_CIDR | string | The default CIDR when the VPC is created. | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DEFAULT_CLUSTER_NAME`^Required ```typescript public readonly DEFAULT_CLUSTER_NAME: string; ``` - *Type:* string The default name of the EKS cluster. --- ##### `DEFAULT_EKS_VERSION`^Required ```typescript public readonly DEFAULT_EKS_VERSION: KubernetesVersion; ``` - *Type:* aws-cdk-lib.aws_eks.KubernetesVersion The default EKS version. --- ##### `DEFAULT_EMR_EKS_VERSION`^Required ```typescript public readonly DEFAULT_EMR_EKS_VERSION: EmrContainersRuntimeVersion; ``` - *Type:* aws-dsf.processing.EmrContainersRuntimeVersion The default EMR on EKS version. --- ##### `DEFAULT_VPC_CIDR`^Required ```typescript public readonly DEFAULT_VPC_CIDR: string; ``` - *Type:* string The default CIDR when the VPC is created. --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### SparkEmrServerlessJob A construct to run Spark Jobs using EMR Serverless. Creates a State Machine that orchestrates the Spark Job. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-emr-serverless-job](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-emr-serverless-job) *Example* ```typescript import { PolicyDocument, PolicyStatement } from 'aws-cdk-lib/aws-iam'; import { JsonPath } from 'aws-cdk-lib/aws-stepfunctions'; const myExecutionRole = dsf.processing.SparkEmrServerlessRuntime.createExecutionRole(this, 'execRole1'); const job = new dsf.processing.SparkEmrServerlessJob(this, 'SparkJob', { jobConfig:{ "Name": JsonPath.format('ge_profile-{}', JsonPath.uuid()), "ApplicationId": "APPLICATION_ID", "ExecutionRoleArn": myExecutionRole.roleArn, "JobDriver": { "SparkSubmit": { "EntryPoint": "s3://S3-BUCKET/pi.py", "EntryPointArguments": [], "SparkSubmitParameters": "--conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.driver.memory=2G --conf spark.executor.cores=4" }, } } } as dsf.processing.SparkEmrServerlessJobApiProps); ``` #### Initializers ```typescript import { processing } from 'aws-dsf' new processing.SparkEmrServerlessJob(scope: Construct, id: string, props: SparkEmrServerlessJobProps | SparkEmrServerlessJobApiProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | aws-dsf.processing.SparkEmrServerlessJobProps \| aws-dsf.processing.SparkEmrServerlessJobApiProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.processing.SparkEmrServerlessJobProps | aws-dsf.processing.SparkEmrServerlessJobApiProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { processing } from 'aws-dsf' processing.SparkEmrServerlessJob.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | stateMachine | aws-cdk-lib.aws_stepfunctions.StateMachine | The Step Functions State Machine created to orchestrate the Spark Job. | | stateMachineLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group used by the State Machine. | | sparkJobExecutionRole | aws-cdk-lib.aws_iam.IRole | The Spark job execution role. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `stateMachine`^Optional ```typescript public readonly stateMachine: StateMachine; ``` - *Type:* aws-cdk-lib.aws_stepfunctions.StateMachine The Step Functions State Machine created to orchestrate the Spark Job. --- ##### `stateMachineLogGroup`^Optional ```typescript public readonly stateMachineLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group used by the State Machine. --- ##### `sparkJobExecutionRole`^Optional ```typescript public readonly sparkJobExecutionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The Spark job execution role. Use this property to add additional IAM permissions if necessary. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### SparkEmrServerlessRuntime A construct to create a Spark EMR Serverless Application, along with methods to create IAM roles having the least privilege. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-emr-serverless-runtime](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-emr-serverless-runtime) *Example* ```typescript import { Role, AccountRootPrincipal } from 'aws-cdk-lib/aws-iam'; const serverlessRuntime = new dsf.processing.SparkEmrServerlessRuntime(this, 'EmrApp', { name: 'SparkRuntimeServerless', }); const executionRole = dsf.processing.SparkEmrServerlessRuntime.createExecutionRole(this, 'ExecutionRole') const submitterRole = new Role (this, 'SubmitterRole', { assumedBy: new AccountRootPrincipal(), }); dsf.processing.SparkEmrServerlessRuntime.grantStartJobExecution(submitterRole, [executionRole.roleArn], ['EMR-serverless-app-ID']); ``` #### Initializers ```typescript import { processing } from 'aws-dsf' new processing.SparkEmrServerlessRuntime(scope: Construct, id: string, props: SparkEmrServerlessRuntimeProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | the Scope of the CDK Construct. | | id | string | the ID of the CDK Construct. | | props | aws-dsf.processing.SparkEmrServerlessRuntimeProps | {@link SparkEmrServerlessRuntimeProps}. | --- ##### `scope`^Required - *Type:* constructs.Construct the Scope of the CDK Construct. --- ##### `id`^Required - *Type:* string the ID of the CDK Construct. --- ##### `props`^Required - *Type:* aws-dsf.processing.SparkEmrServerlessRuntimeProps {@link SparkEmrServerlessRuntimeProps}. --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | grantStartExecution | A method which will grant an IAM Role the right to start and monitor a job. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `grantStartExecution` ```typescript public grantStartExecution(startJobRole: IRole, executionRoleArn: string): void ``` A method which will grant an IAM Role the right to start and monitor a job. The method will also attach an iam:PassRole permission to limited to the IAM Job Execution roles passed. The excution role will be able to submit job to the EMR Serverless application created by the construct. ###### `startJobRole`^Required - *Type:* aws-cdk-lib.aws_iam.IRole the role that will call the start job api and which need to have the iam:PassRole permission. --- ###### `executionRoleArn`^Required - *Type:* string the role use by EMR Serverless to access resources during the job execution. --- ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | | createExecutionRole | A static method creating an execution IAM role that can be assumed by EMR Serverless The method returns the role it creates. | | grantStartJobExecution | A static method granting the right to start and monitor a job to an IAM Role. | --- ##### `isConstruct` ```typescript import { processing } from 'aws-dsf' processing.SparkEmrServerlessRuntime.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- ##### `createExecutionRole` ```typescript import { processing } from 'aws-dsf' processing.SparkEmrServerlessRuntime.createExecutionRole(scope: Construct, id: string, executionRolePolicyDocument?: PolicyDocument, iamPolicyName?: string) ``` A static method creating an execution IAM role that can be assumed by EMR Serverless The method returns the role it creates. If no `executionRolePolicyDocument` or `iamPolicyName` The method will return a role with only a trust policy to EMR Servereless service principal. You can use this role then to grant access to any resources you control. ###### `scope`^Required - *Type:* constructs.Construct the scope in which to create the role. --- ###### `id`^Required - *Type:* string passed to the IAM Role construct object. --- ###### `executionRolePolicyDocument`^Optional - *Type:* aws-cdk-lib.aws_iam.PolicyDocument the inline policy document to attach to the role. These are IAM policies needed by the job. This parameter is mutually execlusive with iamPolicyName. --- ###### `iamPolicyName`^Optional - *Type:* string the IAM policy name to attach to the role, this is mutually execlusive with executionRolePolicyDocument. --- ##### `grantStartJobExecution` ```typescript import { processing } from 'aws-dsf' processing.SparkEmrServerlessRuntime.grantStartJobExecution(startJobRole: IRole, executionRoleArn: string[], applicationArns: string[]) ``` A static method granting the right to start and monitor a job to an IAM Role. The method will also attach an iam:PassRole permission limited to the IAM Job Execution roles passed ###### `startJobRole`^Required - *Type:* aws-cdk-lib.aws_iam.IRole the role that will call the start job api and which needs to have the iam:PassRole permission. --- ###### `executionRoleArn`^Required - *Type:* string[] the role used by EMR Serverless to access resources during the job execution. --- ###### `applicationArns`^Required - *Type:* string[] the EMR Serverless aplication ARN, this is used by the method to limit the EMR Serverless applications the role can submit job to. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | application | aws-cdk-lib.aws_emrserverless.CfnApplication | The EMR Serverless application. | | emrApplicationSecurityGroup | aws-cdk-lib.aws_ec2.ISecurityGroup | If no VPC is provided, one is created by default along with a security group attached to the EMR Serverless Application This attribute is used to expose the security group, if you provide your own security group through the {@link SparkEmrServerlessRuntimeProps} the attribute will be `undefined`. | | flowLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group for the VPC flow log when the VPC is created. | | flowLogKey | aws-cdk-lib.aws_kms.IKey | The KMS Key used for the VPC flow log when the VPC is created. | | flowLogRole | aws-cdk-lib.aws_iam.IRole | The IAM Role used for the VPC flow log when the VPC is created. | | s3VpcEndpoint | aws-cdk-lib.aws_ec2.IGatewayVpcEndpoint | *No description.* | | vpc | aws-cdk-lib.aws_ec2.IVpc | The VPC used by the EKS cluster. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `application`^Required ```typescript public readonly application: CfnApplication; ``` - *Type:* aws-cdk-lib.aws_emrserverless.CfnApplication The EMR Serverless application. --- ##### `emrApplicationSecurityGroup`^Optional ```typescript public readonly emrApplicationSecurityGroup: ISecurityGroup; ``` - *Type:* aws-cdk-lib.aws_ec2.ISecurityGroup If no VPC is provided, one is created by default along with a security group attached to the EMR Serverless Application This attribute is used to expose the security group, if you provide your own security group through the {@link SparkEmrServerlessRuntimeProps} the attribute will be `undefined`. --- ##### `flowLogGroup`^Optional ```typescript public readonly flowLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group for the VPC flow log when the VPC is created. --- ##### `flowLogKey`^Optional ```typescript public readonly flowLogKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey The KMS Key used for the VPC flow log when the VPC is created. --- ##### `flowLogRole`^Optional ```typescript public readonly flowLogRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM Role used for the VPC flow log when the VPC is created. --- ##### `s3VpcEndpoint`^Optional ```typescript public readonly s3VpcEndpoint: IGatewayVpcEndpoint; ``` - *Type:* aws-cdk-lib.aws_ec2.IGatewayVpcEndpoint --- ##### `vpc`^Optional ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc The VPC used by the EKS cluster. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ### SparkJob A base construct to run Spark Jobs. Creates an AWS Step Functions State Machine that orchestrates the Spark Job. > [https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-emr-serverless-job Available implementations: * {@link SparkEmrServerlessJob } for Emr Serverless implementation * {@link SparkEmrEksJob } for EMR On EKS implementation](https://awslabs.github.io/data-solutions-framework-on-aws/docs/constructs/library/Processing/spark-emr-serverless-job Available implementations: * {@link SparkEmrServerlessJob } for Emr Serverless implementation * {@link SparkEmrEksJob } for EMR On EKS implementation) #### Initializers ```typescript import { processing } from 'aws-dsf' new processing.SparkJob(scope: Construct, id: string, trackingTag: string, props: SparkJobProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | the Scope of the CDK Construct. | | id | string | the ID of the CDK Construct. | | trackingTag | string | *No description.* | | props | aws-dsf.processing.SparkJobProps | the SparkJobProps properties. | --- ##### `scope`^Required - *Type:* constructs.Construct the Scope of the CDK Construct. --- ##### `id`^Required - *Type:* string the ID of the CDK Construct. --- ##### `trackingTag`^Required - *Type:* string --- ##### `props`^Required - *Type:* aws-dsf.processing.SparkJobProps the SparkJobProps properties. --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | retrieveVersion | Retrieve DSF package.json version. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `retrieveVersion` ```typescript public retrieveVersion(): any ``` Retrieve DSF package.json version. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { processing } from 'aws-dsf' processing.SparkJob.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | stateMachine | aws-cdk-lib.aws_stepfunctions.StateMachine | The Step Functions State Machine created to orchestrate the Spark Job. | | stateMachineLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group used by the State Machine. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `stateMachine`^Optional ```typescript public readonly stateMachine: StateMachine; ``` - *Type:* aws-cdk-lib.aws_stepfunctions.StateMachine The Step Functions State Machine created to orchestrate the Spark Job. --- ##### `stateMachineLogGroup`^Optional ```typescript public readonly stateMachineLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup The CloudWatch Log Group used by the State Machine. --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | DSF_OWNED_TAG | string | *No description.* | | DSF_TRACKING_CODE | string | *No description.* | --- ##### `DSF_OWNED_TAG`^Required ```typescript public readonly DSF_OWNED_TAG: string; ``` - *Type:* string --- ##### `DSF_TRACKING_CODE`^Required ```typescript public readonly DSF_TRACKING_CODE: string; ``` - *Type:* string --- ## Structs ### AnalyticsBucketProps Properties for the `AnalyticsBucket` construct. #### Initializer ```typescript import { storage } from 'aws-dsf' const analyticsBucketProps: storage.AnalyticsBucketProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | encryptionKey | aws-cdk-lib.aws_kms.IKey | External KMS Key to use for the S3 Bucket encryption. | | accessControl | aws-cdk-lib.aws_s3.BucketAccessControl | Specifies a canned ACL that grants predefined permissions to the bucket. | | autoDeleteObjects | boolean | Whether all objects should be automatically deleted when the S3 Bucket is removed from the stack or when the stack is deleted. | | blockPublicAccess | aws-cdk-lib.aws_s3.BlockPublicAccess | The block public access configuration of this bucket. | | bucketKeyEnabled | boolean | Whether Amazon S3 should use its own intermediary key to generate data keys. Only relevant when using KMS for encryption. | | bucketName | string | The physical name of this S3 Bucket. | | cors | aws-cdk-lib.aws_s3.CorsRule[] | The CORS configuration of this bucket. | | enforceSSL | boolean | Enforces SSL for requests. | | eventBridgeEnabled | boolean | Whether this S3 Bucket should send notifications to Amazon EventBridge or not. | | intelligentTieringConfigurations | aws-cdk-lib.aws_s3.IntelligentTieringConfiguration[] | Intelligent Tiering Configurations. | | inventories | aws-cdk-lib.aws_s3.Inventory[] | The inventory configuration of the S3 Bucket. | | lifecycleRules | aws-cdk-lib.aws_s3.LifecycleRule[] | Rules that define how Amazon S3 manages objects during their lifetime. | | metrics | aws-cdk-lib.aws_s3.BucketMetrics[] | The metrics configuration of this bucket. | | notificationsHandlerRole | aws-cdk-lib.aws_iam.IRole | The IAM Role to be used by the notifications handler. | | objectLockDefaultRetention | aws-cdk-lib.aws_s3.ObjectLockRetention | The default retention mode and rules for S3 Object Lock. | | objectLockEnabled | boolean | Enable object lock on the S3 Bucket. | | objectOwnership | aws-cdk-lib.aws_s3.ObjectOwnership | The objectOwnership of the S3 Bucket. | | publicReadAccess | boolean | Grants public read access to all objects in the S3 Bucket. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | serverAccessLogsBucket | aws-cdk-lib.aws_s3.IBucket | S3 Bucket destination for the server access logs. | | serverAccessLogsPrefix | string | Optional log file prefix to use for the S3 Bucket's access logs. | | transferAcceleration | boolean | Whether this S3 Bucket should have transfer acceleration turned on or not. | | versioned | boolean | Whether this S3 Bucket should have versioning turned on or not. | --- ##### `encryptionKey`^Required ```typescript public readonly encryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey - *Default:* If `encryption` is set to `KMS` and this property is undefined, a new KMS key will be created and associated with this bucket. External KMS Key to use for the S3 Bucket encryption. The `encryption` property must be either not specified or set to `KMS` or `DSSE`. An error will be emitted if `encryption` is set to `UNENCRYPTED` or `S3_MANAGED`. --- ##### `accessControl`^Optional ```typescript public readonly accessControl: BucketAccessControl; ``` - *Type:* aws-cdk-lib.aws_s3.BucketAccessControl - *Default:* BucketAccessControl.PRIVATE Specifies a canned ACL that grants predefined permissions to the bucket. --- ##### `autoDeleteObjects`^Optional ```typescript public readonly autoDeleteObjects: boolean; ``` - *Type:* boolean - *Default:* False Whether all objects should be automatically deleted when the S3 Bucket is removed from the stack or when the stack is deleted. Requires the `removalPolicy` to be set to `RemovalPolicy.DESTROY`. --- ##### `blockPublicAccess`^Optional ```typescript public readonly blockPublicAccess: BlockPublicAccess; ``` - *Type:* aws-cdk-lib.aws_s3.BlockPublicAccess - *Default:* CloudFormation defaults will apply. New buckets and objects don't allow public access, but users can modify bucket policies or object permissions to allow public access The block public access configuration of this bucket. --- ##### `bucketKeyEnabled`^Optional ```typescript public readonly bucketKeyEnabled: boolean; ``` - *Type:* boolean - *Default:* False Whether Amazon S3 should use its own intermediary key to generate data keys. Only relevant when using KMS for encryption. If not enabled, every object GET and PUT will cause an API call to KMS (with the attendant cost implications of that). - If enabled, S3 will use its own time-limited key instead. Only relevant, when Encryption is set to `BucketEncryption.KMS` or `BucketEncryption.KMS_MANAGED`. --- ##### `bucketName`^Optional ```typescript public readonly bucketName: string; ``` - *Type:* string - *Default:* `analytics---` The physical name of this S3 Bucket. --- ##### `cors`^Optional ```typescript public readonly cors: CorsRule[]; ``` - *Type:* aws-cdk-lib.aws_s3.CorsRule[] - *Default:* No CORS configuration. The CORS configuration of this bucket. --- ##### `enforceSSL`^Optional ```typescript public readonly enforceSSL: boolean; ``` - *Type:* boolean - *Default:* False Enforces SSL for requests. S3.5 of the AWS Foundational Security Best Practices Regarding S3. --- ##### `eventBridgeEnabled`^Optional ```typescript public readonly eventBridgeEnabled: boolean; ``` - *Type:* boolean - *Default:* False Whether this S3 Bucket should send notifications to Amazon EventBridge or not. --- ##### `intelligentTieringConfigurations`^Optional ```typescript public readonly intelligentTieringConfigurations: IntelligentTieringConfiguration[]; ``` - *Type:* aws-cdk-lib.aws_s3.IntelligentTieringConfiguration[] - *Default:* No Intelligent Tiiering Configurations. Intelligent Tiering Configurations. --- ##### `inventories`^Optional ```typescript public readonly inventories: Inventory[]; ``` - *Type:* aws-cdk-lib.aws_s3.Inventory[] - *Default:* No inventory configuration The inventory configuration of the S3 Bucket. --- ##### `lifecycleRules`^Optional ```typescript public readonly lifecycleRules: LifecycleRule[]; ``` - *Type:* aws-cdk-lib.aws_s3.LifecycleRule[] - *Default:* No lifecycle rules. Rules that define how Amazon S3 manages objects during their lifetime. --- ##### `metrics`^Optional ```typescript public readonly metrics: BucketMetrics[]; ``` - *Type:* aws-cdk-lib.aws_s3.BucketMetrics[] - *Default:* No metrics configuration. The metrics configuration of this bucket. --- ##### `notificationsHandlerRole`^Optional ```typescript public readonly notificationsHandlerRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* A new IAM Role will be created. The IAM Role to be used by the notifications handler. --- ##### `objectLockDefaultRetention`^Optional ```typescript public readonly objectLockDefaultRetention: ObjectLockRetention; ``` - *Type:* aws-cdk-lib.aws_s3.ObjectLockRetention - *Default:* No default retention period The default retention mode and rules for S3 Object Lock. Default retention can be configured after a bucket is created if the bucket already has object lock enabled. Enabling object lock for existing buckets is not supported. --- ##### `objectLockEnabled`^Optional ```typescript public readonly objectLockEnabled: boolean; ``` - *Type:* boolean - *Default:* False, unless objectLockDefaultRetention is set (then, true) Enable object lock on the S3 Bucket. Enabling object lock for existing buckets is not supported. Object lock must be enabled when the bucket is created. --- ##### `objectOwnership`^Optional ```typescript public readonly objectOwnership: ObjectOwnership; ``` - *Type:* aws-cdk-lib.aws_s3.ObjectOwnership - *Default:* No ObjectOwnership configuration, uploading account will own the object. The objectOwnership of the S3 Bucket. --- ##### `publicReadAccess`^Optional ```typescript public readonly publicReadAccess: boolean; ``` - *Type:* boolean - *Default:* False Grants public read access to all objects in the S3 Bucket. Similar to calling `bucket.grantPublicAccess()` --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ##### `serverAccessLogsBucket`^Optional ```typescript public readonly serverAccessLogsBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket - *Default:* If "serverAccessLogsPrefix" undefined - access logs disabled, otherwise - log to current bucket. S3 Bucket destination for the server access logs. --- ##### `serverAccessLogsPrefix`^Optional ```typescript public readonly serverAccessLogsPrefix: string; ``` - *Type:* string - *Default:* No log file prefix Optional log file prefix to use for the S3 Bucket's access logs. If defined without "serverAccessLogsBucket", enables access logs to current S3 Bucket with this prefix. --- ##### `transferAcceleration`^Optional ```typescript public readonly transferAcceleration: boolean; ``` - *Type:* boolean - *Default:* False Whether this S3 Bucket should have transfer acceleration turned on or not. --- ##### `versioned`^Optional ```typescript public readonly versioned: boolean; ``` - *Type:* boolean - *Default:* False (unless object lock is enabled, then true) Whether this S3 Bucket should have versioning turned on or not. --- ### ApplicationStageProps Properties for the `ApplicationStage` class. #### Initializer ```typescript import { utils } from 'aws-dsf' const applicationStageProps: utils.ApplicationStageProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | env | aws-cdk-lib.Environment | Default AWS environment (account/region) for `Stack`s in this `Stage`. | | outdir | string | The output directory into which to emit synthesized artifacts. | | permissionsBoundary | aws-cdk-lib.PermissionsBoundary | Options for applying a permissions boundary to all IAM Roles and Users created within this Stage. | | policyValidationBeta1 | aws-cdk-lib.IPolicyValidationPluginBeta1[] | Validation plugins to run during synthesis. | | stageName | string | Name of this stage. | | applicationStackFactory | aws-dsf.utils.ApplicationStackFactory | The application CDK Stack Factory used to create application Stacks. | | stage | aws-dsf.utils.CICDStage | The Stage to deploy the application CDK Stack in. | | outputsEnv | {[ key: string ]: string} | The list of values to create CfnOutputs. | --- ##### `env`^Optional ```typescript public readonly env: Environment; ``` - *Type:* aws-cdk-lib.Environment - *Default:* The environments should be configured on the `Stack`s. Default AWS environment (account/region) for `Stack`s in this `Stage`. Stacks defined inside this `Stage` with either `region` or `account` missing from its env will use the corresponding field given here. If either `region` or `account`is is not configured for `Stack` (either on the `Stack` itself or on the containing `Stage`), the Stack will be *environment-agnostic*. Environment-agnostic stacks can be deployed to any environment, may not be able to take advantage of all features of the CDK. For example, they will not be able to use environmental context lookups, will not automatically translate Service Principals to the right format based on the environment's AWS partition, and other such enhancements. --- *Example* ```typescript // Use a concrete account and region to deploy this Stage to new Stage(app, 'Stage1', { env: { account: '123456789012', region: 'us-east-1' }, }); // Use the CLI's current credentials to determine the target environment new Stage(app, 'Stage2', { env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION }, }); ``` ##### `outdir`^Optional ```typescript public readonly outdir: string; ``` - *Type:* string - *Default:* for nested stages, outdir will be determined as a relative directory to the outdir of the app. For apps, if outdir is not specified, a temporary directory will be created. The output directory into which to emit synthesized artifacts. Can only be specified if this stage is the root stage (the app). If this is specified and this stage is nested within another stage, an error will be thrown. --- ##### `permissionsBoundary`^Optional ```typescript public readonly permissionsBoundary: PermissionsBoundary; ``` - *Type:* aws-cdk-lib.PermissionsBoundary - *Default:* no permissions boundary is applied Options for applying a permissions boundary to all IAM Roles and Users created within this Stage. --- ##### `policyValidationBeta1`^Optional ```typescript public readonly policyValidationBeta1: IPolicyValidationPluginBeta1[]; ``` - *Type:* aws-cdk-lib.IPolicyValidationPluginBeta1[] - *Default:* no validation plugins are used Validation plugins to run during synthesis. If any plugin reports any violation, synthesis will be interrupted and the report displayed to the user. --- ##### `stageName`^Optional ```typescript public readonly stageName: string; ``` - *Type:* string - *Default:* Derived from the id. Name of this stage. --- ##### `applicationStackFactory`^Required ```typescript public readonly applicationStackFactory: ApplicationStackFactory; ``` - *Type:* aws-dsf.utils.ApplicationStackFactory The application CDK Stack Factory used to create application Stacks. --- ##### `stage`^Required ```typescript public readonly stage: CICDStage; ``` - *Type:* aws-dsf.utils.CICDStage - *Default:* No stage is passed to the application stack The Stage to deploy the application CDK Stack in. --- ##### `outputsEnv`^Optional ```typescript public readonly outputsEnv: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* No CfnOutputs are created The list of values to create CfnOutputs. --- ### AthenaWorkgroupProps Properties for the AthenaWorkgroup Construct. #### Initializer ```typescript import { consumption } from 'aws-dsf' const athenaWorkgroupProps: consumption.AthenaWorkgroupProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | name | string | Name of the Workgroup. | | resultLocationPrefix | string | Specifies the location in Amazon S3 where query results are stored. | | bytesScannedCutoffPerQuery | number | Indicates the number of days after creation when objects are deleted from the Result bucket. | | enforceWorkGroupConfiguration | boolean | If set to "true", the settings for the workgroup override client-side settings. | | engineVersion | aws-dsf.consumption.EngineVersion | The engine version on which the query runs. | | executionRole | aws-cdk-lib.aws_iam.IRole | Role used to access user resources in an Athena for Apache Spark session. | | publishCloudWatchMetricsEnabled | boolean | Indicates that the Amazon CloudWatch metrics are enabled for the workgroup. | | recursiveDeleteOption | boolean | The option to delete a workgroup and its contents even if the workgroup contains any named queries. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | requesterPaysEnabled | boolean | Allows members assigned to a workgroup to reference Amazon S3 Requester Pays buckets in queries. | | resultBucket | aws-cdk-lib.aws_s3.IBucket | Amazon S3 Bucket where query results are stored. | | resultBucketName | string | Name for the S3 Bucket in case it should be created. | | resultsEncryptionKey | aws-cdk-lib.aws_kms.IKey | Encryption key used to encrypt query results. | | resultsRetentionPeriod | aws-cdk-lib.Duration | Indicates the number of days after creation when objects are deleted from the Result bucket. | | state | aws-dsf.consumption.State | The state of the Workgroup. | --- ##### `name`^Required ```typescript public readonly name: string; ``` - *Type:* string Name of the Workgroup. --- ##### `resultLocationPrefix`^Required ```typescript public readonly resultLocationPrefix: string; ``` - *Type:* string Specifies the location in Amazon S3 where query results are stored. --- ##### `bytesScannedCutoffPerQuery`^Optional ```typescript public readonly bytesScannedCutoffPerQuery: number; ``` - *Type:* number Indicates the number of days after creation when objects are deleted from the Result bucket. --- ##### `enforceWorkGroupConfiguration`^Optional ```typescript public readonly enforceWorkGroupConfiguration: boolean; ``` - *Type:* boolean - *Default:* True. If set to "true", the settings for the workgroup override client-side settings. --- ##### `engineVersion`^Optional ```typescript public readonly engineVersion: EngineVersion; ``` - *Type:* aws-dsf.consumption.EngineVersion - *Default:* AUTO. The engine version on which the query runs. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* The role is created if PySpark engine version is selected and no role is provided. Role used to access user resources in an Athena for Apache Spark session. --- ##### `publishCloudWatchMetricsEnabled`^Optional ```typescript public readonly publishCloudWatchMetricsEnabled: boolean; ``` - *Type:* boolean - *Default:* True. Indicates that the Amazon CloudWatch metrics are enabled for the workgroup. --- ##### `recursiveDeleteOption`^Optional ```typescript public readonly recursiveDeleteOption: boolean; ``` - *Type:* boolean - *Default:* Workgroup is retained. The option to delete a workgroup and its contents even if the workgroup contains any named queries. --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ##### `requesterPaysEnabled`^Optional ```typescript public readonly requesterPaysEnabled: boolean; ``` - *Type:* boolean - *Default:* False. Allows members assigned to a workgroup to reference Amazon S3 Requester Pays buckets in queries. --- ##### `resultBucket`^Optional ```typescript public readonly resultBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket - *Default:* Create a new bucket with SSE encryption using AnalyticsBucket if not provided. Amazon S3 Bucket where query results are stored. --- ##### `resultBucketName`^Optional ```typescript public readonly resultBucketName: string; ``` - *Type:* string - *Default:* Name will be provided. Name for the S3 Bucket in case it should be created. --- ##### `resultsEncryptionKey`^Optional ```typescript public readonly resultsEncryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey - *Default:* The key is created if Result Bucket is not provided. Encryption key used to encrypt query results. Has to be provided if Result bucket is provided. User needs to grant access to it for AthenaWorkGroup's executionRole (if Spark engine) or for principals that were granted to run queries using AthenaWorkGroup's grantRunQueries. --- ##### `resultsRetentionPeriod`^Optional ```typescript public readonly resultsRetentionPeriod: Duration; ``` - *Type:* aws-cdk-lib.Duration Indicates the number of days after creation when objects are deleted from the Result bucket. --- ##### `state`^Optional ```typescript public readonly state: State; ``` - *Type:* aws-dsf.consumption.State - *Default:* ENABLED. The state of the Workgroup. --- ### DataCatalogDatabaseProps Properties for the `DataCatalogDatabase` construct. #### Initializer ```typescript import { governance } from 'aws-dsf' const dataCatalogDatabaseProps: governance.DataCatalogDatabaseProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | name | string | Database name. | | autoCrawl | boolean | When enabled, this automatically creates a top level Glue Crawler that would run based on the defined schedule in the `autoCrawlSchedule` parameter. | | autoCrawlSchedule | aws-cdk-lib.aws_glue.CfnCrawler.ScheduleProperty | The schedule to run the Glue Crawler. | | crawlerLogEncryptionKey | aws-cdk-lib.aws_kms.IKey | KMS encryption Key used for the Glue Crawler logs. | | crawlerRole | aws-cdk-lib.aws_iam.IRole | The IAM Role used by the Glue Crawler when `autoCrawl` is set to `True`. | | crawlerTableLevelDepth | number | Directory depth where the table folders are located. | | glueConnectionName | string | The connection that would be used by the crawler. | | jdbcPath | string | The JDBC path that would be included by the crawler. | | jdbcSecret | aws-cdk-lib.aws_secretsmanager.ISecret | The secret associated with the JDBC connection. | | jdbcSecretKMSKey | aws-cdk-lib.aws_kms.IKey | The KMS key used by the JDBC secret. | | locationBucket | aws-cdk-lib.aws_s3.IBucket | S3 bucket where data is stored. | | locationPrefix | string | Top level location wwhere table data is stored. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | --- ##### `name`^Required ```typescript public readonly name: string; ``` - *Type:* string Database name. Construct would add a randomize suffix as part of the name to prevent name collisions. --- ##### `autoCrawl`^Optional ```typescript public readonly autoCrawl: boolean; ``` - *Type:* boolean - *Default:* True When enabled, this automatically creates a top level Glue Crawler that would run based on the defined schedule in the `autoCrawlSchedule` parameter. --- ##### `autoCrawlSchedule`^Optional ```typescript public readonly autoCrawlSchedule: ScheduleProperty; ``` - *Type:* aws-cdk-lib.aws_glue.CfnCrawler.ScheduleProperty - *Default:* `cron(1 0 * * ? *)` The schedule to run the Glue Crawler. Default is once a day at 00:01h. --- ##### `crawlerLogEncryptionKey`^Optional ```typescript public readonly crawlerLogEncryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey - *Default:* Create a new key if none is provided KMS encryption Key used for the Glue Crawler logs. --- ##### `crawlerRole`^Optional ```typescript public readonly crawlerRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* When `autoCrawl` is enabled, a new role is created with least privilege permissions to run the crawler The IAM Role used by the Glue Crawler when `autoCrawl` is set to `True`. Additional permissions are granted to this role such as S3 Bucket read only permissions and KMS encrypt/decrypt on the key used by the Glue Crawler logging to CloudWatch Logs. --- ##### `crawlerTableLevelDepth`^Optional ```typescript public readonly crawlerTableLevelDepth: number; ``` - *Type:* number - *Default:* calculated based on `locationPrefix` Directory depth where the table folders are located. This helps the Glue Crawler understand the layout of the folders in S3. --- ##### `glueConnectionName`^Optional ```typescript public readonly glueConnectionName: string; ``` - *Type:* string The connection that would be used by the crawler. --- ##### `jdbcPath`^Optional ```typescript public readonly jdbcPath: string; ``` - *Type:* string The JDBC path that would be included by the crawler. --- ##### `jdbcSecret`^Optional ```typescript public readonly jdbcSecret: ISecret; ``` - *Type:* aws-cdk-lib.aws_secretsmanager.ISecret The secret associated with the JDBC connection. --- ##### `jdbcSecretKMSKey`^Optional ```typescript public readonly jdbcSecretKMSKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey The KMS key used by the JDBC secret. --- ##### `locationBucket`^Optional ```typescript public readonly locationBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket S3 bucket where data is stored. --- ##### `locationPrefix`^Optional ```typescript public readonly locationPrefix: string; ``` - *Type:* string Top level location wwhere table data is stored. --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ### DataLakeCatalogProps Properties for the `DataLakeCatalog` Construct. #### Initializer ```typescript import { governance } from 'aws-dsf' const dataLakeCatalogProps: governance.DataLakeCatalogProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | dataLakeStorage | aws-dsf.storage.DataLakeStorage | The DataLakeStorage object to create the data catalog on. | | autoCrawl | boolean | When enabled, creates a top level Glue Crawler that would run based on the defined schedule in the `autoCrawlSchedule` parameter. | | autoCrawlSchedule | aws-cdk-lib.aws_glue.CfnCrawler.ScheduleProperty | The schedule when the Glue Crawler runs, if enabled. | | crawlerLogEncryptionKey | aws-cdk-lib.aws_kms.IKey | The KMS encryption Key used for the Glue Crawler logs. | | crawlerTableLevelDepth | number | Directory depth where the table folders are located. | | databaseName | string | The suffix of the Glue Data Catalog Database. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | --- ##### `dataLakeStorage`^Required ```typescript public readonly dataLakeStorage: DataLakeStorage; ``` - *Type:* aws-dsf.storage.DataLakeStorage The DataLakeStorage object to create the data catalog on. --- ##### `autoCrawl`^Optional ```typescript public readonly autoCrawl: boolean; ``` - *Type:* boolean - *Default:* True When enabled, creates a top level Glue Crawler that would run based on the defined schedule in the `autoCrawlSchedule` parameter. --- ##### `autoCrawlSchedule`^Optional ```typescript public readonly autoCrawlSchedule: ScheduleProperty; ``` - *Type:* aws-cdk-lib.aws_glue.CfnCrawler.ScheduleProperty - *Default:* `cron(1 0 * * ? *)` The schedule when the Glue Crawler runs, if enabled. Default is once a day at 00:01h. --- ##### `crawlerLogEncryptionKey`^Optional ```typescript public readonly crawlerLogEncryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey - *Default:* Create a new KMS Key if none is provided The KMS encryption Key used for the Glue Crawler logs. --- ##### `crawlerTableLevelDepth`^Optional ```typescript public readonly crawlerTableLevelDepth: number; ``` - *Type:* number - *Default:* calculated based on `locationPrefix` Directory depth where the table folders are located. This helps the Glue Crawler understand the layout of the folders in S3. --- ##### `databaseName`^Optional ```typescript public readonly databaseName: string; ``` - *Type:* string - *Default:* Use the bucket name as the database name and as the S3 location The suffix of the Glue Data Catalog Database. The name of the Glue Database is composed of the S3 Bucket name and this suffix. The suffix is also added to the S3 location inside the data lake S3 Buckets. --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ### DataLakeStorageProps Properties for the DataLakeStorage Construct. #### Initializer ```typescript import { storage } from 'aws-dsf' const dataLakeStorageProps: storage.DataLakeStorageProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | bronzeBucketArchiveDelay | number | Delay (in days) before archiving BRONZE data to frozen storage (Glacier storage class). | | bronzeBucketInfrequentAccessDelay | number | Delay (in days) before moving BRONZE data to cold storage (Infrequent Access storage class). | | bronzeBucketName | string | Name of the Bronze bucket. | | dataLakeKey | aws-cdk-lib.aws_kms.IKey | The KMS Key used to encrypt all DataLakeStorage S3 buckets. | | goldBucketArchiveDelay | number | Delay (in days) before archiving GOLD data to frozen storage (Glacier storage class). | | goldBucketInfrequentAccessDelay | number | Delay (in days) before moving GOLD data to cold storage (Infrequent Access storage class). | | goldBucketName | string | Name of the Gold bucket. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | silverBucketArchiveDelay | number | Delay (in days) before archiving SILVER data to frozen storage (Glacier storage class). | | silverBucketInfrequentAccessDelay | number | Delay (in days) before moving SILVER data to cold storage (Infrequent Access storage class). | | silverBucketName | string | Name of the Silver bucket. | --- ##### `bronzeBucketArchiveDelay`^Optional ```typescript public readonly bronzeBucketArchiveDelay: number; ``` - *Type:* number - *Default:* Move objects to Glacier after 90 days. Delay (in days) before archiving BRONZE data to frozen storage (Glacier storage class). --- ##### `bronzeBucketInfrequentAccessDelay`^Optional ```typescript public readonly bronzeBucketInfrequentAccessDelay: number; ``` - *Type:* number - *Default:* Move objects to Infrequent Access after 30 days. Delay (in days) before moving BRONZE data to cold storage (Infrequent Access storage class). --- ##### `bronzeBucketName`^Optional ```typescript public readonly bronzeBucketName: string; ``` - *Type:* string - *Default:* `bronze---` will be used. Name of the Bronze bucket. Use `BucketUtils.generateUniqueBucketName()` to generate a unique name (recommended). --- ##### `dataLakeKey`^Optional ```typescript public readonly dataLakeKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey - *Default:* A single KMS customer key is created. The KMS Key used to encrypt all DataLakeStorage S3 buckets. --- ##### `goldBucketArchiveDelay`^Optional ```typescript public readonly goldBucketArchiveDelay: number; ``` - *Type:* number - *Default:* Objects are not archived to Glacier. Delay (in days) before archiving GOLD data to frozen storage (Glacier storage class). --- ##### `goldBucketInfrequentAccessDelay`^Optional ```typescript public readonly goldBucketInfrequentAccessDelay: number; ``` - *Type:* number - *Default:* Move objects to Infrequent Access after 90 days. Delay (in days) before moving GOLD data to cold storage (Infrequent Access storage class). --- ##### `goldBucketName`^Optional ```typescript public readonly goldBucketName: string; ``` - *Type:* string - *Default:* `gold---` will be used. Name of the Gold bucket. Use `BucketUtils.generateUniqueBucketName()` to generate a unique name (recommended). --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise, the removalPolicy is reverted to RETAIN. --- ##### `silverBucketArchiveDelay`^Optional ```typescript public readonly silverBucketArchiveDelay: number; ``` - *Type:* number - *Default:* Objects are not archived to Glacier. Delay (in days) before archiving SILVER data to frozen storage (Glacier storage class). --- ##### `silverBucketInfrequentAccessDelay`^Optional ```typescript public readonly silverBucketInfrequentAccessDelay: number; ``` - *Type:* number - *Default:* Move objects to Infrequent Access after 90 days. Delay (in days) before moving SILVER data to cold storage (Infrequent Access storage class). --- ##### `silverBucketName`^Optional ```typescript public readonly silverBucketName: string; ``` - *Type:* string - *Default:* `silver---` will be used. Name of the Silver bucket. Use `BucketUtils.generateUniqueBucketName()` to generate a unique name (recommended). --- ### DataVpcProps The properties for the `DataVpc` construct. #### Initializer ```typescript import { utils } from 'aws-dsf' const dataVpcProps: utils.DataVpcProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | vpcCidr | string | The CIDR to use to create the subnets in the VPC. | | flowLogKey | aws-cdk-lib.aws_kms.IKey | The KMS key used to encrypt the VPC Flow Logs in the CloudWatch Log Group. | | flowLogRetention | aws-cdk-lib.aws_logs.RetentionDays | The retention period to apply to VPC Flow Logs. | | flowLogRole | aws-cdk-lib.aws_iam.IRole | The IAM Role used to send the VPC Flow Logs in CloudWatch. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | --- ##### `vpcCidr`^Required ```typescript public readonly vpcCidr: string; ``` - *Type:* string The CIDR to use to create the subnets in the VPC. --- ##### `flowLogKey`^Optional ```typescript public readonly flowLogKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey - *Default:* A new KMS key is created The KMS key used to encrypt the VPC Flow Logs in the CloudWatch Log Group. The resource policy of the key must be configured according to the AWS documentation. > [https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/encrypt-log-data-kms.html](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/encrypt-log-data-kms.html) --- ##### `flowLogRetention`^Optional ```typescript public readonly flowLogRetention: RetentionDays; ``` - *Type:* aws-cdk-lib.aws_logs.RetentionDays - *Default:* One week retention The retention period to apply to VPC Flow Logs. --- ##### `flowLogRole`^Optional ```typescript public readonly flowLogRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* A new IAM role is created The IAM Role used to send the VPC Flow Logs in CloudWatch. The role must be configured as described in the AWS VPC Flow Log documentation. > [https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html#flow-logs-iam-role](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html#flow-logs-iam-role) --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise, the removalPolicy is reverted to RETAIN. --- ### EmrVirtualClusterProps The properties for the `EmrVirtualCluster` Construct class. #### Initializer ```typescript import { processing } from 'aws-dsf' const emrVirtualClusterProps: processing.EmrVirtualClusterProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | name | string | The name of the Amazon EMR Virtual Cluster to be created. | | createNamespace | boolean | The flag to create EKS namespace. | | eksNamespace | string | The name of the EKS namespace to be linked to the EMR virtual cluster. | | tags | {[ key: string ]: string} | The tags assigned to the Virtual Cluster. | --- ##### `name`^Required ```typescript public readonly name: string; ``` - *Type:* string The name of the Amazon EMR Virtual Cluster to be created. --- ##### `createNamespace`^Optional ```typescript public readonly createNamespace: boolean; ``` - *Type:* boolean - *Default:* Do not create the namespace The flag to create EKS namespace. --- ##### `eksNamespace`^Optional ```typescript public readonly eksNamespace: string; ``` - *Type:* string - *Default:* Use the default namespace The name of the EKS namespace to be linked to the EMR virtual cluster. --- ##### `tags`^Optional ```typescript public readonly tags: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* none The tags assigned to the Virtual Cluster. --- ### PySparkApplicationPackageProps Properties for the `PySparkApplicationPackage` construct. #### Initializer ```typescript import { processing } from 'aws-dsf' const pySparkApplicationPackageProps: processing.PySparkApplicationPackageProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | applicationName | string | The name of the PySpark application. | | entrypointPath | string | The source path in the code base where the entrypoint is stored. | | artifactsBucket | aws-cdk-lib.aws_s3.IBucket | The S3 bucket where to upload the artifacts of the Spark Job This is where the entry point and archive of the virtual environment will be stored. | | assetUploadRole | aws-cdk-lib.aws_iam.IRole | The IAM Role used by the Lambda function. | | dependenciesFolder | string | The source directory where `requirements.txt` or `pyproject.toml` file is stored. These files are used to install external AND internal Python packages. If your PySpark application has more than one Python file, you need to [package your Python project](https://packaging.python.org/en/latest/tutorials/packaging-projects/). This location must also contain a `Dockerfile` that can [create a virtual environment and build an archive](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/using-python-libraries.html#building-python-virtual-env). | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | venvArchivePath | string | The path of the Python virtual environment archive generated in the Docker container. | --- ##### `applicationName`^Required ```typescript public readonly applicationName: string; ``` - *Type:* string The name of the PySpark application. This name is used as a parent directory in S3 to store the entrypoint and the optional virtual environment archive --- ##### `entrypointPath`^Required ```typescript public readonly entrypointPath: string; ``` - *Type:* string The source path in the code base where the entrypoint is stored. example `~/my-project/src/entrypoint.py` --- ##### `artifactsBucket`^Optional ```typescript public readonly artifactsBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket - *Default:* An S3 Bucket is created The S3 bucket where to upload the artifacts of the Spark Job This is where the entry point and archive of the virtual environment will be stored. --- ##### `assetUploadRole`^Optional ```typescript public readonly assetUploadRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* A new Role would be created with least privilege permissions The IAM Role used by the Lambda function. Additional permissions would be granted to this role such as S3 Bucket permissions. --- ##### `dependenciesFolder`^Optional ```typescript public readonly dependenciesFolder: string; ``` - *Type:* string - *Default:* No dependencies (internal or external) are packaged. Only the entrypoint is used in the Spark Job. The source directory where `requirements.txt` or `pyproject.toml` file is stored. These files are used to install external AND internal Python packages. If your PySpark application has more than one Python file, you need to [package your Python project](https://packaging.python.org/en/latest/tutorials/packaging-projects/). This location must also contain a `Dockerfile` that can [create a virtual environment and build an archive](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/using-python-libraries.html#building-python-virtual-env). --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. Resources like Amazon cloudwatch log or Amazon S3 bucket. If DESTROY is selected, the context value '@data-solutions-framework-on-aws/removeDataOnDestroy' in the 'cdk.json' or 'cdk.context.json' must be set to true. --- ##### `venvArchivePath`^Optional ```typescript public readonly venvArchivePath: string; ``` - *Type:* string - *Default:* No virtual environment archive is packaged. Only the entrypoint can be used in the Spark Job. It is required if the `dependenciesFolder` is provided. The path of the Python virtual environment archive generated in the Docker container. This is the output path used in the `venv-pack -o` command in your Dockerfile. --- ### RedshiftDataProps The properties for the `RedshiftData` construct. #### Initializer ```typescript import { consumption } from 'aws-dsf' const redshiftDataProps: consumption.RedshiftDataProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | secret | aws-cdk-lib.aws_secretsmanager.ISecret | The Secrets Manager Secret containing the admin credentials for the Redshift cluster / namespace. | | clusterId | string | The name of the Redshift provisioned to query. | | createInterfaceVpcEndpoint | boolean | If set to true, create the Redshift Data Interface VPC Endpoint in the configured VPC/Subnets. | | executionTimeout | aws-cdk-lib.Duration | The timeout for the query execution. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | secretKey | aws-cdk-lib.aws_kms.IKey | The KMS Key used to encrypt the admin credentials for the Redshift cluster / namespace. | | subnets | aws-cdk-lib.aws_ec2.SelectedSubnets | The subnets where the Custom Resource Lambda Function would be created in. | | vpc | aws-cdk-lib.aws_ec2.IVpc | The VPC where the Custom Resource Lambda Function would be created in. | | workgroupId | string | The `workgroupId` for the Redshift Serverless Workgroup to query. | --- ##### `secret`^Required ```typescript public readonly secret: ISecret; ``` - *Type:* aws-cdk-lib.aws_secretsmanager.ISecret The Secrets Manager Secret containing the admin credentials for the Redshift cluster / namespace. --- ##### `clusterId`^Optional ```typescript public readonly clusterId: string; ``` - *Type:* string - *Default:* The `workgroupId` is used The name of the Redshift provisioned to query. It must be configured if the `workgroupId` is not. --- ##### `createInterfaceVpcEndpoint`^Optional ```typescript public readonly createInterfaceVpcEndpoint: boolean; ``` - *Type:* boolean - *Default:* false If set to true, create the Redshift Data Interface VPC Endpoint in the configured VPC/Subnets. --- ##### `executionTimeout`^Optional ```typescript public readonly executionTimeout: Duration; ``` - *Type:* aws-cdk-lib.Duration - *Default:* 5mins The timeout for the query execution. --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise, the removalPolicy is reverted to RETAIN. --- ##### `secretKey`^Optional ```typescript public readonly secretKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey - *Default:* no secret key is used The KMS Key used to encrypt the admin credentials for the Redshift cluster / namespace. --- ##### `subnets`^Optional ```typescript public readonly subnets: SelectedSubnets; ``` - *Type:* aws-cdk-lib.aws_ec2.SelectedSubnets - *Default:* No subnets are used. The Custom Resource runs in the Redshift service team subnets. The subnets where the Custom Resource Lambda Function would be created in. A Redshift Data API Interface VPC Endpoint is created in the subnets. --- ##### `vpc`^Optional ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc - *Default:* No VPC is used. The Custom Resource runs in the Redshift service team VPC The VPC where the Custom Resource Lambda Function would be created in. A Redshift Data API Interface VPC Endpoint is created in the VPC. --- ##### `workgroupId`^Optional ```typescript public readonly workgroupId: string; ``` - *Type:* string - *Default:* The `clusterId` is used The `workgroupId` for the Redshift Serverless Workgroup to query. It must be configured if the `clusterId` is not. --- ### RedshiftServerlessNamespaceProps RedshiftServerlessNamespace properties. #### Initializer ```typescript import { consumption } from 'aws-dsf' const redshiftServerlessNamespaceProps: consumption.RedshiftServerlessNamespaceProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | dbName | string | The name of the primary database that would be created in the Redshift Serverless Namespace. | | name | string | The name of the Redshift Serverless Namespace. | | adminSecretKey | aws-cdk-lib.aws_kms.Key | The KMS Key used by the managed Secrets Manager Secret storing admin credentials. | | adminUsername | string | The admin username to be used. | | dataKey | aws-cdk-lib.aws_kms.Key | The KMS Key used to encrypt the data. | | defaultIAMRole | aws-cdk-lib.aws_iam.IRole | Default IAM Role associated to the Redshift Serverless Namespace. | | iamRoles | aws-cdk-lib.aws_iam.IRole[] | List of IAM Roles attached to the Redshift Serverless Namespace. | | logExports | aws-dsf.consumption.RedshiftServerlessNamespaceLogExport[] | The type of logs to be exported. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | --- ##### `dbName`^Required ```typescript public readonly dbName: string; ``` - *Type:* string The name of the primary database that would be created in the Redshift Serverless Namespace. --- ##### `name`^Required ```typescript public readonly name: string; ``` - *Type:* string The name of the Redshift Serverless Namespace. --- ##### `adminSecretKey`^Optional ```typescript public readonly adminSecretKey: Key; ``` - *Type:* aws-cdk-lib.aws_kms.Key - *Default:* A new KMS Key is created The KMS Key used by the managed Secrets Manager Secret storing admin credentials. --- ##### `adminUsername`^Optional ```typescript public readonly adminUsername: string; ``` - *Type:* string - *Default:* The default username is "admin" The admin username to be used. --- ##### `dataKey`^Optional ```typescript public readonly dataKey: Key; ``` - *Type:* aws-cdk-lib.aws_kms.Key - *Default:* A new KMS Key is created The KMS Key used to encrypt the data. --- ##### `defaultIAMRole`^Optional ```typescript public readonly defaultIAMRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* No default IAM Role is associated with the Redshift Serverless Namespace Default IAM Role associated to the Redshift Serverless Namespace. --- ##### `iamRoles`^Optional ```typescript public readonly iamRoles: IRole[]; ``` - *Type:* aws-cdk-lib.aws_iam.IRole[] - *Default:* No IAM roles are associated with the Redshift Serverless Namespace List of IAM Roles attached to the Redshift Serverless Namespace. This list of Roles must also contain the `defaultIamRole`. --- ##### `logExports`^Optional ```typescript public readonly logExports: RedshiftServerlessNamespaceLogExport[]; ``` - *Type:* aws-dsf.consumption.RedshiftServerlessNamespaceLogExport[] - *Default:* No logs are exported The type of logs to be exported. --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise, the removalPolicy is reverted to RETAIN. --- ### RedshiftServerlessWorkgroupProps RedshiftServerlessWorkgroup properties. #### Initializer ```typescript import { consumption } from 'aws-dsf' const redshiftServerlessWorkgroupProps: consumption.RedshiftServerlessWorkgroupProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | name | string | The name of the Redshift Serverless Workgroup. | | namespace | aws-dsf.consumption.RedshiftServerlessNamespace | The Redshift Serverless Namespace associated with the Workgroup. | | baseCapacity | number | The base capacity of the Redshift Serverless Workgroup in RPU. | | extraSecurityGroups | aws-cdk-lib.aws_ec2.SecurityGroup[] | The extra EC2 Security Groups to associate with the Redshift Serverless Workgroup (in addition to the primary Security Group). | | port | number | The custom port to use when connecting to workgroup. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | subnets | aws-cdk-lib.aws_ec2.SubnetSelection | The subnets where the Redshift Serverless Workgroup is deployed. | | vpc | aws-cdk-lib.aws_ec2.Vpc | The VPC where the Redshift Serverless Workgroup is deployed. | --- ##### `name`^Required ```typescript public readonly name: string; ``` - *Type:* string The name of the Redshift Serverless Workgroup. --- ##### `namespace`^Required ```typescript public readonly namespace: RedshiftServerlessNamespace; ``` - *Type:* aws-dsf.consumption.RedshiftServerlessNamespace The Redshift Serverless Namespace associated with the Workgroup. --- ##### `baseCapacity`^Optional ```typescript public readonly baseCapacity: number; ``` - *Type:* number - *Default:* 128 RPU The base capacity of the Redshift Serverless Workgroup in RPU. --- ##### `extraSecurityGroups`^Optional ```typescript public readonly extraSecurityGroups: SecurityGroup[]; ``` - *Type:* aws-cdk-lib.aws_ec2.SecurityGroup[] - *Default:* No extra security groups are used The extra EC2 Security Groups to associate with the Redshift Serverless Workgroup (in addition to the primary Security Group). --- ##### `port`^Optional ```typescript public readonly port: number; ``` - *Type:* number - *Default:* 5439 The custom port to use when connecting to workgroup. Valid port ranges are 5431-5455 and 8191-8215. --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise, the removalPolicy is reverted to RETAIN. --- ##### `subnets`^Optional ```typescript public readonly subnets: SubnetSelection; ``` - *Type:* aws-cdk-lib.aws_ec2.SubnetSelection - *Default:* Use the private subnets of the VPC The subnets where the Redshift Serverless Workgroup is deployed. --- ##### `vpc`^Optional ```typescript public readonly vpc: Vpc; ``` - *Type:* aws-cdk-lib.aws_ec2.Vpc - *Default:* A default VPC is created The VPC where the Redshift Serverless Workgroup is deployed. --- ### S3DataCopyProps Properties for S3DataCopy construct. #### Initializer ```typescript import { utils } from 'aws-dsf' const s3DataCopyProps: utils.S3DataCopyProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | sourceBucket | aws-cdk-lib.aws_s3.IBucket | The source S3 Bucket containing the data to copy. | | sourceBucketRegion | string | The source S3 Bucket region. | | targetBucket | aws-cdk-lib.aws_s3.IBucket | The target S3 Bucket. | | executionRole | aws-cdk-lib.aws_iam.IRole | The IAM Role to use in the custom resource for copying data. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | securityGroups | aws-cdk-lib.aws_ec2.ISecurityGroup[] | The list of security groups to attach to the custom resource. | | sourceBucketPrefix | string | The source bucket prefix with a slash at the end. | | subnets | aws-cdk-lib.aws_ec2.SubnetSelection | The subnets to deploy the custom resource in. | | targetBucketPrefix | string | The target S3 Bucket prefix with a slash at the end. | | vpc | aws-cdk-lib.aws_ec2.IVpc | The VPC to deploy the custom resource in. | --- ##### `sourceBucket`^Required ```typescript public readonly sourceBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The source S3 Bucket containing the data to copy. --- ##### `sourceBucketRegion`^Required ```typescript public readonly sourceBucketRegion: string; ``` - *Type:* string The source S3 Bucket region. --- ##### `targetBucket`^Required ```typescript public readonly targetBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The target S3 Bucket. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* A new role is created The IAM Role to use in the custom resource for copying data. --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise, the removalPolicy is reverted to RETAIN. --- ##### `securityGroups`^Optional ```typescript public readonly securityGroups: ISecurityGroup[]; ``` - *Type:* aws-cdk-lib.aws_ec2.ISecurityGroup[] - *Default:* If `vpc` is not supplied, no security groups are attached. Otherwise, a dedicated security group is created for each function. The list of security groups to attach to the custom resource. --- ##### `sourceBucketPrefix`^Optional ```typescript public readonly sourceBucketPrefix: string; ``` - *Type:* string - *Default:* No prefix is used The source bucket prefix with a slash at the end. --- ##### `subnets`^Optional ```typescript public readonly subnets: SubnetSelection; ``` - *Type:* aws-cdk-lib.aws_ec2.SubnetSelection - *Default:* The Custom Resource is executed in VPCs owned by AWS Lambda service. The subnets to deploy the custom resource in. --- ##### `targetBucketPrefix`^Optional ```typescript public readonly targetBucketPrefix: string; ``` - *Type:* string - *Default:* No prefix is used The target S3 Bucket prefix with a slash at the end. --- ##### `vpc`^Optional ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc - *Default:* The Custom Resource is executed in VPCs owned by AWS Lambda service. The VPC to deploy the custom resource in. --- ### SparkEmrCICDPipelineProps Properties for the `SparkEmrCICDPipeline` construct. #### Initializer ```typescript import { processing } from 'aws-dsf' const sparkEmrCICDPipelineProps: processing.SparkEmrCICDPipelineProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | applicationStackFactory | aws-dsf.utils.ApplicationStackFactory | The application CDK Stack to deploy in the different CDK Pipelines Stages. | | sparkApplicationName | string | The name of the Spark application to be deployed. | | cdkApplicationPath | string | The path to the folder that contains the CDK Application. | | integTestEnv | {[ key: string ]: string} | The environment variables to create from the Application CDK Stack outputs and to pass to the integration tests. | | integTestPermissions | aws-cdk-lib.aws_iam.PolicyStatement[] | The IAM Policy statements to add permissions for running the integration tests. | | integTestScript | string | The path to the Shell script that contains integration tests. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | sparkApplicationPath | string | The path to the folder that contains the Spark Application. | | sparkImage | aws-dsf.processing.SparkImage | The EMR Spark image to use to run the unit tests. | --- ##### `applicationStackFactory`^Required ```typescript public readonly applicationStackFactory: ApplicationStackFactory; ``` - *Type:* aws-dsf.utils.ApplicationStackFactory The application CDK Stack to deploy in the different CDK Pipelines Stages. --- ##### `sparkApplicationName`^Required ```typescript public readonly sparkApplicationName: string; ``` - *Type:* string The name of the Spark application to be deployed. --- ##### `cdkApplicationPath`^Optional ```typescript public readonly cdkApplicationPath: string; ``` - *Type:* string - *Default:* The root of the repository The path to the folder that contains the CDK Application. --- ##### `integTestEnv`^Optional ```typescript public readonly integTestEnv: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* No environment variables The environment variables to create from the Application CDK Stack outputs and to pass to the integration tests. This is used to interact with resources created by the Application CDK Stack from within the integration tests script. Key is the name of the environment variable to create. Value is generally a CfnOutput name from the Application CDK Stack. --- ##### `integTestPermissions`^Optional ```typescript public readonly integTestPermissions: PolicyStatement[]; ``` - *Type:* aws-cdk-lib.aws_iam.PolicyStatement[] - *Default:* No permissions The IAM Policy statements to add permissions for running the integration tests. --- ##### `integTestScript`^Optional ```typescript public readonly integTestScript: string; ``` - *Type:* string - *Default:* No integration tests are run The path to the Shell script that contains integration tests. --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ##### `sparkApplicationPath`^Optional ```typescript public readonly sparkApplicationPath: string; ``` - *Type:* string - *Default:* The root of the repository The path to the folder that contains the Spark Application. --- ##### `sparkImage`^Optional ```typescript public readonly sparkImage: SparkImage; ``` - *Type:* aws-dsf.processing.SparkImage - *Default:* [DEFAULT_SPARK_IMAGE](https://github.com/awslabs/data-solutions-framework-on-aws/blob/HEAD/framework/src/processing/lib/emr-releases.ts#L51) The EMR Spark image to use to run the unit tests. --- ### SparkEmrContainersJobApiProps Configuration for the EMR on EKS job. Use this interface when `SparkEmrContainerJobProps` doesn't give you access to the configuration parameters you need. > [[https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html]]([https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html]) #### Initializer ```typescript import { processing } from 'aws-dsf' const sparkEmrContainersJobApiProps: processing.SparkEmrContainersJobApiProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | schedule | aws-cdk-lib.aws_events.Schedule | The Schedule to run the Step Functions state machine. | | jobConfig | {[ key: string ]: any} | EMR on EKS StartJobRun API configuration. | | executionTimeout | aws-cdk-lib.Duration | Job execution timeout. | --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ##### `schedule`^Optional ```typescript public readonly schedule: Schedule; ``` - *Type:* aws-cdk-lib.aws_events.Schedule - *Default:* The Step Functions State Machine is not scheduled. The Schedule to run the Step Functions state machine. --- ##### `jobConfig`^Required ```typescript public readonly jobConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} EMR on EKS StartJobRun API configuration. > [[https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html]]([https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html]) --- ##### `executionTimeout`^Optional ```typescript public readonly executionTimeout: Duration; ``` - *Type:* aws-cdk-lib.Duration - *Default:* 30 minutes Job execution timeout. --- ### SparkEmrContainersJobProps Simplified configuration for the `SparkEmrEksJob` construct. #### Initializer ```typescript import { processing } from 'aws-dsf' const sparkEmrContainersJobProps: processing.SparkEmrContainersJobProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | schedule | aws-cdk-lib.aws_events.Schedule | The Schedule to run the Step Functions state machine. | | executionRole | aws-cdk-lib.aws_iam.IRole | The IAM execution Role ARN for the EMR on EKS job. | | name | string | The Spark job name. | | sparkSubmitEntryPoint | string | The entry point for the Spark submit job run. | | virtualClusterId | string | The EMR on EKS virtual cluster ID. | | applicationConfiguration | {[ key: string ]: any} | The application configuration override for the Spark submit job run. | | cloudWatchLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group name for log publishing. | | cloudWatchLogGroupStreamPrefix | string | The CloudWatch Log Group stream prefix for log publishing. | | executionTimeout | aws-cdk-lib.Duration | The execution timeout. | | maxRetries | number | The maximum number of retries. | | releaseLabel | aws-dsf.processing.EmrRuntimeVersion | The EMR release version associated with the application. | | s3LogBucket | aws-cdk-lib.aws_s3.IBucket | The S3 Bucket for log publishing. | | s3LogPrefix | string | The S3 Bucket prefix for log publishing. | | sparkSubmitEntryPointArguments | string[] | The arguments for the Spark submit job run. | | sparkSubmitParameters | string | The parameters for the Spark submit job run. | | tags | {[ key: string ]: any} | Tags to be added to the EMR Serverless job. | --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ##### `schedule`^Optional ```typescript public readonly schedule: Schedule; ``` - *Type:* aws-cdk-lib.aws_events.Schedule - *Default:* The Step Functions State Machine is not scheduled. The Schedule to run the Step Functions state machine. --- ##### `executionRole`^Required ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM execution Role ARN for the EMR on EKS job. --- ##### `name`^Required ```typescript public readonly name: string; ``` - *Type:* string The Spark job name. --- ##### `sparkSubmitEntryPoint`^Required ```typescript public readonly sparkSubmitEntryPoint: string; ``` - *Type:* string The entry point for the Spark submit job run. > [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html) --- ##### `virtualClusterId`^Required ```typescript public readonly virtualClusterId: string; ``` - *Type:* string The EMR on EKS virtual cluster ID. --- ##### `applicationConfiguration`^Optional ```typescript public readonly applicationConfiguration: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} - *Default:* No configuration is passed to the job. The application configuration override for the Spark submit job run. > [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html) --- ##### `cloudWatchLogGroup`^Optional ```typescript public readonly cloudWatchLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup - *Default:* CloudWatch is not used for logging The CloudWatch Log Group name for log publishing. --- ##### `cloudWatchLogGroupStreamPrefix`^Optional ```typescript public readonly cloudWatchLogGroupStreamPrefix: string; ``` - *Type:* string - *Default:* The application name is used as the prefix The CloudWatch Log Group stream prefix for log publishing. --- ##### `executionTimeout`^Optional ```typescript public readonly executionTimeout: Duration; ``` - *Type:* aws-cdk-lib.Duration - *Default:* 30 minutes The execution timeout. --- ##### `maxRetries`^Optional ```typescript public readonly maxRetries: number; ``` - *Type:* number - *Default:* No retry The maximum number of retries. --- ##### `releaseLabel`^Optional ```typescript public readonly releaseLabel: EmrRuntimeVersion; ``` - *Type:* aws-dsf.processing.EmrRuntimeVersion - *Default:* [EMR_DEFAULT_VERSION](https://github.com/awslabs/data-solutions-framework-on-aws/blob/HEAD/framework/src/processing/lib/emr-releases.ts#L46) The EMR release version associated with the application. The EMR release can be found in this [documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html) --- ##### `s3LogBucket`^Optional ```typescript public readonly s3LogBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket - *Default:* No logging to S3 The S3 Bucket for log publishing. --- ##### `s3LogPrefix`^Optional ```typescript public readonly s3LogPrefix: string; ``` - *Type:* string - *Default:* No logging to S3 The S3 Bucket prefix for log publishing. --- ##### `sparkSubmitEntryPointArguments`^Optional ```typescript public readonly sparkSubmitEntryPointArguments: string[]; ``` - *Type:* string[] - *Default:* No arguments are passed to the job. The arguments for the Spark submit job run. > [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html) --- ##### `sparkSubmitParameters`^Optional ```typescript public readonly sparkSubmitParameters: string; ``` - *Type:* string - *Default:* No parameters are passed to the job. The parameters for the Spark submit job run. > [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html) --- ##### `tags`^Optional ```typescript public readonly tags: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} - *Default:* No tags are added Tags to be added to the EMR Serverless job. --- ### SparkEmrContainersRuntimeInteractiveSessionProps The properties for the EMR Managed Endpoint to create. #### Initializer ```typescript import { processing } from 'aws-dsf' const sparkEmrContainersRuntimeInteractiveSessionProps: processing.SparkEmrContainersRuntimeInteractiveSessionProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | executionRole | aws-cdk-lib.aws_iam.IRole | The Amazon IAM role used as the execution role, this role must provide access to all the AWS resource a user will interact with These can be S3, DynamoDB, Glue Catalog. | | managedEndpointName | string | The name of the EMR managed endpoint. | | virtualClusterId | string | The Id of the Amazon EMR virtual cluster containing the managed endpoint. | | configurationOverrides | any | The JSON configuration overrides for Amazon EMR on EKS configuration attached to the managed endpoint. | | emrOnEksVersion | aws-dsf.processing.EmrContainersRuntimeVersion | The Amazon EMR version to use. | --- ##### `executionRole`^Required ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The Amazon IAM role used as the execution role, this role must provide access to all the AWS resource a user will interact with These can be S3, DynamoDB, Glue Catalog. --- ##### `managedEndpointName`^Required ```typescript public readonly managedEndpointName: string; ``` - *Type:* string The name of the EMR managed endpoint. --- ##### `virtualClusterId`^Required ```typescript public readonly virtualClusterId: string; ``` - *Type:* string The Id of the Amazon EMR virtual cluster containing the managed endpoint. --- ##### `configurationOverrides`^Optional ```typescript public readonly configurationOverrides: any; ``` - *Type:* any - *Default:* Configuration related to the [default nodegroup for notebook]{@link EmrEksNodegroup.NOTEBOOK_EXECUTOR } The JSON configuration overrides for Amazon EMR on EKS configuration attached to the managed endpoint. --- ##### `emrOnEksVersion`^Optional ```typescript public readonly emrOnEksVersion: EmrContainersRuntimeVersion; ``` - *Type:* aws-dsf.processing.EmrContainersRuntimeVersion - *Default:* The [default Amazon EMR version]{@link EmrEksCluster.DEFAULT_EMR_VERSION } The Amazon EMR version to use. --- ### SparkEmrContainersRuntimeProps The properties for the `SparkEmrContainerRuntime` Construct class. #### Initializer ```typescript import { processing } from 'aws-dsf' const sparkEmrContainersRuntimeProps: processing.SparkEmrContainersRuntimeProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | kubectlLambdaLayer | aws-cdk-lib.aws_lambda.ILayerVersion | The Lambda Layer with Kubectl to use for EKS Cluster setup. | | publicAccessCIDRs | string[] | The CIDR blocks that are allowed to access to your cluster’s public Kubernetes API server endpoint. | | createEmrOnEksServiceLinkedRole | boolean | Flag to create an IAM Service Linked Role for EMR on EKS. | | defaultNodes | boolean | The flag to create default Karpenter Node Provisioners for: * Critical jobs which use on-demand instances, high speed disks and workload isolation * Shared workloads which use EC2 Spot Instances and no isolation to optimize costs * Notebooks which leverage a cost optimized configuration for running EMR managed endpoints and spark drivers/executors. | | ec2InstanceRole | aws-cdk-lib.aws_iam.IRole | The IAM Role used for the cluster nodes instance profile. | | eksAdminRole | aws-cdk-lib.aws_iam.IRole | The IAM Role to configure in the EKS master roles. | | eksCluster | aws-cdk-lib.aws_eks.Cluster | The EKS Cluster to setup EMR on. | | eksClusterName | string | The name of the EKS cluster to create. | | eksVpc | aws-cdk-lib.aws_ec2.IVpc | The VPC to use when creating the EKS cluster. | | karpenterVersion | aws-dsf.processing.KarpenterVersion | The Karpenter version to use for autoscaling nodes in the EKS Cluster. | | kubernetesVersion | aws-cdk-lib.aws_eks.KubernetesVersion | The Kubernetes version used to create the EKS Cluster. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | tags | {[ key: string ]: string} | The tags assigned to the EKS cluster. | | vpcCidr | string | The CIDR of the VPC to use when creating the EKS cluster. | --- ##### `kubectlLambdaLayer`^Required ```typescript public readonly kubectlLambdaLayer: ILayerVersion; ``` - *Type:* aws-cdk-lib.aws_lambda.ILayerVersion The Lambda Layer with Kubectl to use for EKS Cluster setup. Starting k8s 1.22, CDK no longer bundle the Kubectl layer with the code due to breaking npm package size. A layer needs to be passed to the Construct. The CDK [documentation](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks.KubernetesVersion.html#static-v1_22) contains the libraries that you should add for the right Kubernetes version. --- ##### `publicAccessCIDRs`^Required ```typescript public readonly publicAccessCIDRs: string[]; ``` - *Type:* string[] The CIDR blocks that are allowed to access to your cluster’s public Kubernetes API server endpoint. --- ##### `createEmrOnEksServiceLinkedRole`^Optional ```typescript public readonly createEmrOnEksServiceLinkedRole: boolean; ``` - *Type:* boolean - *Default:* true Flag to create an IAM Service Linked Role for EMR on EKS. --- ##### `defaultNodes`^Optional ```typescript public readonly defaultNodes: boolean; ``` - *Type:* boolean - *Default:* true The flag to create default Karpenter Node Provisioners for: * Critical jobs which use on-demand instances, high speed disks and workload isolation * Shared workloads which use EC2 Spot Instances and no isolation to optimize costs * Notebooks which leverage a cost optimized configuration for running EMR managed endpoints and spark drivers/executors. --- ##### `ec2InstanceRole`^Optional ```typescript public readonly ec2InstanceRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* A role is created with AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, AmazonSSMManagedInstanceCore and AmazonEKS_CNI_Policy AWS managed policies. The IAM Role used for the cluster nodes instance profile. --- ##### `eksAdminRole`^Optional ```typescript public readonly eksAdminRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* An admin role must be passed if `eksCluster` property is not set. The IAM Role to configure in the EKS master roles. It will give access to kubernetes cluster from the AWS console. You will use this role to manage the EKS cluster and grant other access to it. --- ##### `eksCluster`^Optional ```typescript public readonly eksCluster: Cluster; ``` - *Type:* aws-cdk-lib.aws_eks.Cluster - *Default:* An EKS Cluster is created The EKS Cluster to setup EMR on. The cluster needs to be created in the same CDK Stack. If the EKS Cluster is provided, the cluster AddOns and all the controllers (ALB Ingress controller, Karpenter...) need to be configured. When providing an EKS cluster, the methods for adding nodegroups can still be used. They implement the best practices for running Spark on EKS. --- ##### `eksClusterName`^Optional ```typescript public readonly eksClusterName: string; ``` - *Type:* string - *Default:* The [DEFAULT_CLUSTER_NAME](https://github.com/awslabs/data-solutions-framework-on-aws/blob/HEAD/framework/src/processing/lib/spark-runtime/emr-containers/spark-emr-containers-runtime.ts#L65) The name of the EKS cluster to create. --- ##### `eksVpc`^Optional ```typescript public readonly eksVpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc - *Default:* A new VPC is created. The VPC to use when creating the EKS cluster. VPC should have at least two private and public subnets in different Availability Zones. All private subnets must have the following tags: * 'for-use-with-amazon-emr-managed-policies'='true' * 'kubernetes.io/role/internal-elb'='1' All public subnets must have the following tag: * 'kubernetes.io/role/elb'='1' Cannot be combined with `vpcCidr`. If combined, `vpcCidr` takes precedence and a new VPC is created. --- ##### `karpenterVersion`^Optional ```typescript public readonly karpenterVersion: KarpenterVersion; ``` - *Type:* aws-dsf.processing.KarpenterVersion - *Default:* [DEFAULT_KARPENTER_VERSION](https://github.com/awslabs/data-solutions-framework-on-aws/blob/HEAD/framework/src/processing/lib/karpenter-releases.ts#L11) The Karpenter version to use for autoscaling nodes in the EKS Cluster. --- ##### `kubernetesVersion`^Optional ```typescript public readonly kubernetesVersion: KubernetesVersion; ``` - *Type:* aws-cdk-lib.aws_eks.KubernetesVersion - *Default:* [DEFAULT_EKS_VERSION](https://github.com/awslabs/data-solutions-framework-on-aws/blob/HEAD/framework/src/processing/lib/spark-runtime/emr-containers/spark-emr-containers-runtime.ts#L61) The Kubernetes version used to create the EKS Cluster. --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ##### `tags`^Optional ```typescript public readonly tags: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* none The tags assigned to the EKS cluster. --- ##### `vpcCidr`^Optional ```typescript public readonly vpcCidr: string; ``` - *Type:* string - *Default:* The CIDR 10.0.0.0/16 is used The CIDR of the VPC to use when creating the EKS cluster. If provided, a VPC with three public subnets and three private subnets is created. The size of the private subnets is four time the one of the public subnet. --- ### SparkEmrServerlessJobApiProps Configuration for the EMR Serverless Job API. Use this interface when `SparkEmrServerlessJobProps` doesn't give you access to the configuration parameters you need. > [[https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartJobRun.html]]([https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartJobRun.html]) #### Initializer ```typescript import { processing } from 'aws-dsf' const sparkEmrServerlessJobApiProps: processing.SparkEmrServerlessJobApiProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | schedule | aws-cdk-lib.aws_events.Schedule | The Schedule to run the Step Functions state machine. | | jobConfig | {[ key: string ]: any} | EMR Serverless Job Configuration. | --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ##### `schedule`^Optional ```typescript public readonly schedule: Schedule; ``` - *Type:* aws-cdk-lib.aws_events.Schedule - *Default:* The Step Functions State Machine is not scheduled. The Schedule to run the Step Functions state machine. --- ##### `jobConfig`^Required ```typescript public readonly jobConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} EMR Serverless Job Configuration. > [[https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartJobRun.html]]([https://docs.aws.amazon.com/emr-serverless/latest/APIReference/API_StartJobRun.html]) --- ### SparkEmrServerlessJobProps #### Initializer ```typescript import { processing } from 'aws-dsf' const sparkEmrServerlessJobProps: processing.SparkEmrServerlessJobProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | schedule | aws-cdk-lib.aws_events.Schedule | The Schedule to run the Step Functions state machine. | | applicationId | string | The EMR Serverless Application to execute the Spark Job. | | executionRole | aws-cdk-lib.aws_iam.IRole | The IAM execution Role for the EMR Serverless job. | | name | string | The Spark Job name. | | sparkSubmitEntryPoint | string | The entry point for the Spark submit job run. | | applicationConfiguration | {[ key: string ]: any} | The application configuration override for the Spark submit job run. | | cloudWatchEncryptionKey | aws-cdk-lib.aws_kms.IKey | The KMS Key for encrypting logs on CloudWatch. | | cloudWatchLogGroup | aws-cdk-lib.aws_logs.ILogGroup | The CloudWatch Log Group name for log publishing. | | cloudWatchLogGroupStreamPrefix | string | The CloudWatch Log Group Stream prefix for log publishing. | | cloudWatchLogtypes | string | The types of logs to log in CloudWatch Log. | | executionTimeout | aws-cdk-lib.Duration | The execution timeout. | | persistentAppUi | boolean | Enable Spark persistent UI logs in EMR managed storage. | | persistentAppUIKey | aws-cdk-lib.aws_kms.IKey | The KMS Key ARN to encrypt Spark persistent UI logs in EMR managed storage. | | s3LogBucket | aws-cdk-lib.aws_s3.IBucket | The S3 Bucket for log publishing. | | s3LogEncryptionKey | aws-cdk-lib.aws_kms.IKey | The KMS Key for encrypting logs on S3. | | s3LogPrefix | string | The S3 Bucket prefix for log publishing. | | sparkSubmitEntryPointArguments | string[] | The arguments for the Spark submit job run. | | sparkSubmitParameters | string | The parameters for the Spark submit job run. | | tags | {[ key: string ]: any} | Tags to be added to the EMR Serverless job. | --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ##### `schedule`^Optional ```typescript public readonly schedule: Schedule; ``` - *Type:* aws-cdk-lib.aws_events.Schedule - *Default:* The Step Functions State Machine is not scheduled. The Schedule to run the Step Functions state machine. --- ##### `applicationId`^Required ```typescript public readonly applicationId: string; ``` - *Type:* string The EMR Serverless Application to execute the Spark Job. --- ##### `executionRole`^Required ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The IAM execution Role for the EMR Serverless job. --- ##### `name`^Required ```typescript public readonly name: string; ``` - *Type:* string The Spark Job name. --- ##### `sparkSubmitEntryPoint`^Required ```typescript public readonly sparkSubmitEntryPoint: string; ``` - *Type:* string The entry point for the Spark submit job run. > [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html) --- ##### `applicationConfiguration`^Optional ```typescript public readonly applicationConfiguration: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} - *Default:* No configuration is passed to the job. The application configuration override for the Spark submit job run. > [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html) --- ##### `cloudWatchEncryptionKey`^Optional ```typescript public readonly cloudWatchEncryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey - *Default:* No encryption The KMS Key for encrypting logs on CloudWatch. --- ##### `cloudWatchLogGroup`^Optional ```typescript public readonly cloudWatchLogGroup: ILogGroup; ``` - *Type:* aws-cdk-lib.aws_logs.ILogGroup - *Default:* No logging to CloudWatch The CloudWatch Log Group name for log publishing. --- ##### `cloudWatchLogGroupStreamPrefix`^Optional ```typescript public readonly cloudWatchLogGroupStreamPrefix: string; ``` - *Type:* string - *Default:* No prefix is used The CloudWatch Log Group Stream prefix for log publishing. --- ##### `cloudWatchLogtypes`^Optional ```typescript public readonly cloudWatchLogtypes: string; ``` - *Type:* string The types of logs to log in CloudWatch Log. --- ##### `executionTimeout`^Optional ```typescript public readonly executionTimeout: Duration; ``` - *Type:* aws-cdk-lib.Duration - *Default:* 30 minutes The execution timeout. --- ##### `persistentAppUi`^Optional ```typescript public readonly persistentAppUi: boolean; ``` - *Type:* boolean - *Default:* true Enable Spark persistent UI logs in EMR managed storage. --- ##### `persistentAppUIKey`^Optional ```typescript public readonly persistentAppUIKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey - *Default:* Use EMR managed Key The KMS Key ARN to encrypt Spark persistent UI logs in EMR managed storage. --- ##### `s3LogBucket`^Optional ```typescript public readonly s3LogBucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket - *Default:* No logging to S3 The S3 Bucket for log publishing. --- ##### `s3LogEncryptionKey`^Optional ```typescript public readonly s3LogEncryptionKey: IKey; ``` - *Type:* aws-cdk-lib.aws_kms.IKey - *Default:* No encryption The KMS Key for encrypting logs on S3. --- ##### `s3LogPrefix`^Optional ```typescript public readonly s3LogPrefix: string; ``` - *Type:* string - *Default:* No logging to S3 The S3 Bucket prefix for log publishing. --- ##### `sparkSubmitEntryPointArguments`^Optional ```typescript public readonly sparkSubmitEntryPointArguments: string[]; ``` - *Type:* string[] - *Default:* No arguments are passed to the job. The arguments for the Spark submit job run. > [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html) --- ##### `sparkSubmitParameters`^Optional ```typescript public readonly sparkSubmitParameters: string; ``` - *Type:* string - *Default:* No parameters are passed to the job. The parameters for the Spark submit job run. > [https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html](https://docs.aws.amazon.com/emr-on-eks/latest/APIReference/API_StartJobRun.html) --- ##### `tags`^Optional ```typescript public readonly tags: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} - *Default:* No tags are added Tags to be added to the EMR Serverless job. --- ### SparkEmrServerlessRuntimeProps Properties for the `SparkEmrServerlessRuntime` construct. #### Initializer ```typescript import { processing } from 'aws-dsf' const sparkEmrServerlessRuntimeProps: processing.SparkEmrServerlessRuntimeProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | name | string | The name of the application. | | architecture | aws-dsf.utils.Architecture | The CPU architecture type of the application. | | autoStartConfiguration | aws-cdk-lib.IResolvable \| aws-cdk-lib.aws_emrserverless.CfnApplication.AutoStartConfigurationProperty | The configuration for an application to automatically start on job submission. | | autoStopConfiguration | aws-cdk-lib.IResolvable \| aws-cdk-lib.aws_emrserverless.CfnApplication.AutoStopConfigurationProperty | The configuration for an application to automatically stop after a certain amount of time being idle. | | imageConfiguration | aws-cdk-lib.IResolvable \| aws-cdk-lib.aws_emrserverless.CfnApplication.ImageConfigurationInputProperty | The unique custom image configuration used for both the Spark Driver and the Spark Executor. | | initialCapacity |

aws-cdk-lib.IResolvable \| aws-cdk-lib.IResolvable \| aws-cdk-lib.aws_emrserverless.CfnApplication.InitialCapacityConfigKeyValuePairProperty[]

| The pre-initialized capacity of the application. | | maximumCapacity | aws-cdk-lib.IResolvable \| aws-cdk-lib.aws_emrserverless.CfnApplication.MaximumAllowedResourcesProperty | The maximum capacity of the application. | | networkConfiguration | aws-cdk-lib.IResolvable \| aws-cdk-lib.aws_emrserverless.CfnApplication.NetworkConfigurationProperty | The network configuration for customer VPC connectivity for the application. | | releaseLabel | aws-dsf.processing.EmrRuntimeVersion | The EMR release version associated with the application. | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | runtimeConfiguration | aws-cdk-lib.IResolvable \| aws-cdk-lib.aws_emrserverless.CfnApplication.ConfigurationObjectProperty[] | The runtime and monitoring configurations to used as defaults for all of the job runs of this application. | | workerTypeSpecifications |

aws-cdk-lib.IResolvable \| {[ key: string ]: aws-cdk-lib.IResolvable \| aws-cdk-lib.aws_emrserverless.CfnApplication.WorkerTypeSpecificationInputProperty}

| The different custom image configurations used for the Spark Driver and the Spark Executor. | --- ##### `name`^Required ```typescript public readonly name: string; ``` - *Type:* string The name of the application. The name must be less than 64 characters. *Pattern* : `^[A-Za-z0-9._\\/#-]+$` --- ##### `architecture`^Optional ```typescript public readonly architecture: Architecture; ``` - *Type:* aws-dsf.utils.Architecture - *Default:* x86_64 The CPU architecture type of the application. --- ##### `autoStartConfiguration`^Optional ```typescript public readonly autoStartConfiguration: IResolvable | AutoStartConfigurationProperty; ``` - *Type:* aws-cdk-lib.IResolvable | aws-cdk-lib.aws_emrserverless.CfnApplication.AutoStartConfigurationProperty - *Default:* True The configuration for an application to automatically start on job submission. --- ##### `autoStopConfiguration`^Optional ```typescript public readonly autoStopConfiguration: IResolvable | AutoStopConfigurationProperty; ``` - *Type:* aws-cdk-lib.IResolvable | aws-cdk-lib.aws_emrserverless.CfnApplication.AutoStopConfigurationProperty - *Default:* The application is stopped after 15 minutes of idle time The configuration for an application to automatically stop after a certain amount of time being idle. --- ##### `imageConfiguration`^Optional ```typescript public readonly imageConfiguration: IResolvable | ImageConfigurationInputProperty; ``` - *Type:* aws-cdk-lib.IResolvable | aws-cdk-lib.aws_emrserverless.CfnApplication.ImageConfigurationInputProperty - *Default:* EMR base image is used for both the Spark Driver and the Spark Executor The unique custom image configuration used for both the Spark Driver and the Spark Executor. > [https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/application-custom-image.html](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/application-custom-image.html) --- ##### `initialCapacity`^Optional ```typescript public readonly initialCapacity: IResolvable | IResolvable | InitialCapacityConfigKeyValuePairProperty[]; ``` - *Type:* aws-cdk-lib.IResolvable | aws-cdk-lib.IResolvable | aws-cdk-lib.aws_emrserverless.CfnApplication.InitialCapacityConfigKeyValuePairProperty[] - *Default:* No pre-initialized capacity is used The pre-initialized capacity of the application. > [https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/pre-init-capacity.html) --- ##### `maximumCapacity`^Optional ```typescript public readonly maximumCapacity: IResolvable | MaximumAllowedResourcesProperty; ``` - *Type:* aws-cdk-lib.IResolvable | aws-cdk-lib.aws_emrserverless.CfnApplication.MaximumAllowedResourcesProperty - *Default:* Depending on the EMR version The maximum capacity of the application. This is cumulative across all workers at any given point in time during the lifespan of the application is created. No new resources will be created once any one of the defined limits is hit. --- ##### `networkConfiguration`^Optional ```typescript public readonly networkConfiguration: IResolvable | NetworkConfigurationProperty; ``` - *Type:* aws-cdk-lib.IResolvable | aws-cdk-lib.aws_emrserverless.CfnApplication.NetworkConfigurationProperty - *Default:* a VPC and a security group are created, these are accessed as construct attribute. The network configuration for customer VPC connectivity for the application. If no configuration is created, the a VPC with 3 public subnets and 3 private subnets is created The 3 public subnets and 3 private subnets are each created in an Availability Zone (AZ) The VPC has one NAT Gateway per AZ and an S3 endpoint --- ##### `releaseLabel`^Optional ```typescript public readonly releaseLabel: EmrRuntimeVersion; ``` - *Type:* aws-dsf.processing.EmrRuntimeVersion - *Default:* [EMR_DEFAULT_VERSION](https://github.com/awslabs/data-solutions-framework-on-aws/blob/HEAD/framework/src/processing/lib/emr-releases.ts#L46) The EMR release version associated with the application. The EMR release can be found in this [documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html) --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ##### `runtimeConfiguration`^Optional ```typescript public readonly runtimeConfiguration: IResolvable | ConfigurationObjectProperty[]; ``` - *Type:* aws-cdk-lib.IResolvable | aws-cdk-lib.aws_emrserverless.CfnApplication.ConfigurationObjectProperty[] - *Default:* No custom configuration is used The runtime and monitoring configurations to used as defaults for all of the job runs of this application. > [https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/default-configs.html](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/default-configs.html) --- ##### `workerTypeSpecifications`^Optional ```typescript public readonly workerTypeSpecifications: IResolvable | {[ key: string ]: IResolvable | WorkerTypeSpecificationInputProperty}; ``` - *Type:* aws-cdk-lib.IResolvable | {[ key: string ]: aws-cdk-lib.IResolvable | aws-cdk-lib.aws_emrserverless.CfnApplication.WorkerTypeSpecificationInputProperty} - *Default:* EMR base image is used for both the Spark Driver and the Spark Executor The different custom image configurations used for the Spark Driver and the Spark Executor. > [https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/application-custom-image.html](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/application-custom-image.html) --- ### SparkJobProps The properties for the `SparkJob` construct. #### Initializer ```typescript import { processing } from 'aws-dsf' const sparkJobProps: processing.SparkJobProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | removalPolicy | aws-cdk-lib.RemovalPolicy | The removal policy when deleting the CDK resource. | | schedule | aws-cdk-lib.aws_events.Schedule | The Schedule to run the Step Functions state machine. | --- ##### `removalPolicy`^Optional ```typescript public readonly removalPolicy: RemovalPolicy; ``` - *Type:* aws-cdk-lib.RemovalPolicy - *Default:* The resources are not deleted (`RemovalPolicy.RETAIN`). The removal policy when deleting the CDK resource. If DESTROY is selected, context value `@data-solutions-framework-on-aws/removeDataOnDestroy` needs to be set to true. Otherwise the removalPolicy is reverted to RETAIN. --- ##### `schedule`^Optional ```typescript public readonly schedule: Schedule; ``` - *Type:* aws-cdk-lib.aws_events.Schedule - *Default:* The Step Functions State Machine is not scheduled. The Schedule to run the Step Functions state machine. --- ## Classes ### ApplicationStackFactory Abstract class that needs to be implemented to pass the application Stack to the CICD pipeline. *Example* ```typescript interface MyApplicationStackProps extends cdk.StackProps { readonly stage: dsf.utils.CICDStage; } class MyApplicationStack extends cdk.Stack { constructor(scope: Construct, id: string, props?: MyApplicationStackProps) { super(scope, id, props); // stack logic goes here... and can be customized using props.stage } } class MyApplicationStackFactory extends dsf.utils.ApplicationStackFactory { createStack(scope: Construct, stage: dsf.utils.CICDStage): cdk.Stack { return new MyApplicationStack(scope, 'MyApplication', { stage: stage } as MyApplicationStackProps); } } ``` #### Initializers ```typescript import { utils } from 'aws-dsf' new utils.ApplicationStackFactory() ``` | **Name** | **Type** | **Description** | | --- | --- | --- | --- #### Methods | **Name** | **Description** | | --- | --- | | createStack | Abstract method that needs to be implemented to return the application Stack. | --- ##### `createStack` ```typescript public createStack(scope: Construct, stage: CICDStage): Stack ``` Abstract method that needs to be implemented to return the application Stack. ###### `scope`^Required - *Type:* constructs.Construct The scope to create the stack in. --- ###### `stage`^Required - *Type:* aws-dsf.utils.CICDStage The stage of the pipeline. --- ### BucketUtils Utils for working with Amazon S3 Buckets. #### Initializers ```typescript import { utils } from 'aws-dsf' new utils.BucketUtils() ``` | **Name** | **Type** | **Description** | | --- | --- | --- | --- #### Static Functions | **Name** | **Description** | | --- | --- | | generateUniqueBucketName | Generate a unique Amazon S3 bucket name based on the provided name, CDK construct ID and CDK construct scope. | --- ##### `generateUniqueBucketName` ```typescript import { utils } from 'aws-dsf' utils.BucketUtils.generateUniqueBucketName(scope: Construct, id: string, name: string) ``` Generate a unique Amazon S3 bucket name based on the provided name, CDK construct ID and CDK construct scope. The bucket name is suffixed the AWS account ID, the AWS region and a unique 8 characters hash. The maximum length for name is 26 characters. ###### `scope`^Required - *Type:* constructs.Construct the current scope where the construct is created (generally `this`). --- ###### `id`^Required - *Type:* string the CDK ID of the construct. --- ###### `name`^Required - *Type:* string the name of the bucket. --- ### StepFunctionUtils Utils for working with AWS Step Functions. #### Initializers ```typescript import { utils } from 'aws-dsf' new utils.StepFunctionUtils() ``` | **Name** | **Type** | **Description** | | --- | --- | --- | --- #### Static Functions | **Name** | **Description** | | --- | --- | | camelToPascal | Convert camel case properties to pascal case as required by AWS Step Functions API. | --- ##### `camelToPascal` ```typescript import { utils } from 'aws-dsf' utils.StepFunctionUtils.camelToPascal(config: {[ key: string ]: any}) ``` Convert camel case properties to pascal case as required by AWS Step Functions API. ###### `config`^Required - *Type:* {[ key: string ]: any} --- ### Utils Utilities class used across the different resources. #### Initializers ```typescript import { utils } from 'aws-dsf' new utils.Utils() ``` | **Name** | **Type** | **Description** | | --- | --- | --- | --- #### Static Functions | **Name** | **Description** | | --- | --- | | generateHash | Generate an 8 character hash from a string based on HMAC algorithm. | | generateUniqueHash | Generate a unique hash of 8 characters from the CDK scope using its path and the stack name. | | loadYaml | Take a document stored as string and load it as YAML. | | randomize | Create a random string to be used as a seed for IAM User password. | | readYamlDocument | Read a YAML file from the path provided and return it. | | stringSanitizer | Sanitize a string by removing upper case and replacing special characters except underscore. | | toPascalCase | Convert a string to PascalCase. | --- ##### `generateHash` ```typescript import { utils } from 'aws-dsf' utils.Utils.generateHash(text: string) ``` Generate an 8 character hash from a string based on HMAC algorithm. ###### `text`^Required - *Type:* string the text to hash. --- ##### `generateUniqueHash` ```typescript import { utils } from 'aws-dsf' utils.Utils.generateUniqueHash(scope: Construct, id?: string) ``` Generate a unique hash of 8 characters from the CDK scope using its path and the stack name. ###### `scope`^Required - *Type:* constructs.Construct the CDK construct scope. --- ###### `id`^Optional - *Type:* string the CDK ID of the construct. --- ##### `loadYaml` ```typescript import { utils } from 'aws-dsf' utils.Utils.loadYaml(document: string) ``` Take a document stored as string and load it as YAML. ###### `document`^Required - *Type:* string the document stored as string. --- ##### `randomize` ```typescript import { utils } from 'aws-dsf' utils.Utils.randomize(name: string) ``` Create a random string to be used as a seed for IAM User password. ###### `name`^Required - *Type:* string the string to which to append a random string. --- ##### `readYamlDocument` ```typescript import { utils } from 'aws-dsf' utils.Utils.readYamlDocument(path: string) ``` Read a YAML file from the path provided and return it. ###### `path`^Required - *Type:* string the path to the file. --- ##### `stringSanitizer` ```typescript import { utils } from 'aws-dsf' utils.Utils.stringSanitizer(toSanitize: string) ``` Sanitize a string by removing upper case and replacing special characters except underscore. ###### `toSanitize`^Required - *Type:* string the string to sanitize. --- ##### `toPascalCase` ```typescript import { utils } from 'aws-dsf' utils.Utils.toPascalCase(text: string) ``` Convert a string to PascalCase. ###### `text`^Required - *Type:* string the string to convert to PascalCase. --- ## Enums ### Architecture List of supported CPU architecture, either X86_64 or ARM64. #### Members | **Name** | **Description** | | --- | --- | | X86_64 | *No description.* | | ARM64 | *No description.* | --- ##### `X86_64` --- ##### `ARM64` --- ### CICDStage The list of CICD Stages used in CICD Pipelines. #### Members | **Name** | **Description** | | --- | --- | | STAGING | *No description.* | | PROD | *No description.* | --- ##### `STAGING` --- ##### `PROD` --- ### EmrContainersRuntimeVersion #### Members | **Name** | **Description** | | --- | --- | | V7_0 | *No description.* | | V6_15 | *No description.* | | V6_14 | *No description.* | | V6_13 | *No description.* | | V6_12 | *No description.* | | V6_11_1 | *No description.* | | V6_11 | *No description.* | | V6_10_1 | *No description.* | | V6_10 | *No description.* | | V6_9 | *No description.* | | V6_8 | *No description.* | | V6_7 | *No description.* | | V6_6 | *No description.* | | V6_5 | *No description.* | | V6_4 | *No description.* | | V6_3 | *No description.* | | V6_2 | *No description.* | | V5_33 | *No description.* | | V5_32 | *No description.* | --- ##### `V7_0` --- ##### `V6_15` --- ##### `V6_14` --- ##### `V6_13` --- ##### `V6_12` --- ##### `V6_11_1` --- ##### `V6_11` --- ##### `V6_10_1` --- ##### `V6_10` --- ##### `V6_9` --- ##### `V6_8` --- ##### `V6_7` --- ##### `V6_6` --- ##### `V6_5` --- ##### `V6_4` --- ##### `V6_3` --- ##### `V6_2` --- ##### `V5_33` --- ##### `V5_32` --- ### EmrRuntimeVersion Enum defining the EMR version as defined in the [Amazon EMR documentation](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html). #### Members | **Name** | **Description** | | --- | --- | | V7_0 | *No description.* | | V6_15 | *No description.* | | V6_14 | *No description.* | | V6_13 | *No description.* | | V6_12 | *No description.* | | V6_11_1 | *No description.* | | V6_11 | *No description.* | | V6_10_1 | *No description.* | | V6_10 | *No description.* | | V6_9 | *No description.* | | V6_8 | *No description.* | | V6_7 | *No description.* | | V6_6 | *No description.* | | V6_5 | *No description.* | | V6_4 | *No description.* | | V6_3 | *No description.* | | V6_2 | *No description.* | | V5_33 | *No description.* | | V5_32 | *No description.* | --- ##### `V7_0` --- ##### `V6_15` --- ##### `V6_14` --- ##### `V6_13` --- ##### `V6_12` --- ##### `V6_11_1` --- ##### `V6_11` --- ##### `V6_10_1` --- ##### `V6_10` --- ##### `V6_9` --- ##### `V6_8` --- ##### `V6_7` --- ##### `V6_6` --- ##### `V6_5` --- ##### `V6_4` --- ##### `V6_3` --- ##### `V6_2` --- ##### `V5_33` --- ##### `V5_32` --- ### EngineVersion #### Members | **Name** | **Description** | | --- | --- | | AUTO | *No description.* | | ATHENA_V3 | *No description.* | | PYSPARK_V3 | *No description.* | --- ##### `AUTO` --- ##### `ATHENA_V3` --- ##### `PYSPARK_V3` --- ### KarpenterVersion The list of supported Karpenter versions as defined [here](https://github.com/aws/karpenter/releases) At this time only v0.32.1 is supported. #### Members | **Name** | **Description** | | --- | --- | | V0_32_1 | *No description.* | --- ##### `V0_32_1` --- ### RedshiftServerlessNamespaceLogExport Namespace log export types. #### Members | **Name** | **Description** | | --- | --- | | USER_LOG | *No description.* | | CONNECTION_LOG | *No description.* | | USER_ACTIVITY_LOG | *No description.* | --- ##### `USER_LOG` --- ##### `CONNECTION_LOG` --- ##### `USER_ACTIVITY_LOG` --- ### SparkImage The list of supported Spark images to use in the SparkCICDPipeline. #### Members | **Name** | **Description** | | --- | --- | | EMR_7_0 | *No description.* | | EMR_6_15 | *No description.* | | EMR_6_14 | *No description.* | | EMR_6_13 | *No description.* | | EMR_6_12 | *No description.* | | EMR_6_11 | *No description.* | | EMR_6_10 | *No description.* | | EMR_6_9 | *No description.* | --- ##### `EMR_7_0` --- ##### `EMR_6_15` --- ##### `EMR_6_14` --- ##### `EMR_6_13` --- ##### `EMR_6_12` --- ##### `EMR_6_11` --- ##### `EMR_6_10` --- ##### `EMR_6_9` --- ### State #### Members | **Name** | **Description** | | --- | --- | | ENABLED | *No description.* | | DISABLED | *No description.* | --- ##### `ENABLED` --- ##### `DISABLED` ---