# integ-runner --- ![cdk-constructs: Experimental](https://img.shields.io/badge/cdk--constructs-experimental-important.svg?style=for-the-badge) > The APIs of higher level constructs in this module are experimental and under active development. > They are subject to non-backward compatible changes or removal in any future version. These are > not subject to the [Semantic Versioning](https://semver.org/) model and breaking changes will be > announced in the release notes. This means that while you may use them, you may need to update > your source code when upgrading to a newer version of this package. --- ## Overview This tool has been created to be used initially by this repo (aws/aws-cdk). Long term the goal is for this tool to be a general tool that can be used for running CDK integration tests. We are publishing this tool so that it can be used by the community and we would love to receive feedback on use cases that the tool should support, or issues that prevent the tool from being used in your library. This tool is meant to be used with the [integ-tests](https://docs.aws.amazon.com/cdk/api/v2/docs/integ-tests-alpha-readme.html) library. ## Usage - Run all integration tests in `test` directory ```bash integ-runner [ARGS] [TEST...] ``` This will look for all files that match the naming convention of `/integ.*.(js|ts)$/` or `/integ_*.py`. Each of these files represents a self contained AWS CDK app, defining test cases and assertions using the [integ-tests](https://docs.aws.amazon.com/cdk/api/v2/docs/integ-tests-alpha-readme.html) library. By default, executing `integ-runner` will do the following for each file (app): 1. Synth each integration test to create a new snapshot in memory 2. Compare the snapshot to the previous version (i.e. files in `*.snapshot/**`), and fail on any differences.\ For new tests this will always fail. To accept a snapshot update, the integreation test has to be deployed and assertions have to pass.\ Execute the runner again, this time with `integ-runner --update-on-fail` and the following will happen for each file: 1. Deploy the previous snapshot 2. Deploy the new version as a Stack update and run assertions 3. If successful, update the snapshots All snapshot files (i.e. `*.snapshot/**`) must be checked-in to version control. If not, changes cannot be compared across systems and the [update workflow](#update-workflow) cannot be used. ### Options - `--update-on-failed` (default=`false`) Rerun integration tests if snapshot fails - `--clean` (default=`true`) Destroy stacks after deploy (use `--no-clean` for debugging) - `--verbose` (default=`false`) verbose logging, including integration test metrics (specify multiple times to increase verbosity) - `--parallel-regions` (default=`us-east-1`,`us-east-2`, `us-west-2`) List of regions to run tests in. If this is provided then all tests will be run in parallel across these regions - `--directory` (default=`test`) Search for integration tests recursively from this starting directory - `--force` (default=`false`) Rerun integration test even if the test passes - `--profiles` List of AWS Profiles to use when running tests in parallel - `--exclude` (default=`false`) If this is set to `true` then the list of tests provided will be excluded - `--strict` (default=`false`) Fail if any specified tests are not found. Cannot be used with `--exclude` - `--from-file` Read the list of tests from this file - `--disable-update-workflow` (default=`false`) If this is set to `true` then the [update workflow](#update-workflow) will be disabled - `--update-from-tags` Deploy snapshots from the given git tags in sequence before deploying the current code. Implies `--force`. See [Update From Tags](#update-from-tags) for details. - `--app` The custom CLI command that will be used to run the test files. You can include {filePath} to specify where in the command the test file path should be inserted. Example: --app="python3.8 {filePath}". Use together with `--test-regex` to fully customize how tests are run, or use with a single `--language` preset to change the command used for this language. - `--test-regex` Detect integration test files matching this JavaScript regex pattern. If used multiple times, all files matching any one of the patterns are detected. - `--watch` Run a single integration test in watch mode. In watch mode the integ-runner will not save any snapshots. Use together with `--app` to fully customize how tests are run, or use with a single `--language` preset to change which files are detected for this language. - `--language` The language presets to use. You can discover and run tests written in multiple languages by passing this flag multiple times (`--language typescript --language python`). Defaults to all supported languages. Currently supported language presets are: - `javascript`: - File RegExp: `^integ\..*\.js$` - App run command: `node {filePath}` - `typescript`:\ Note that for TypeScript files compiled to JavaScript, the JS tests will take precedence and the TS ones won't be evaluated. - File RegExp: `^integ\..*(? **Important:** The update workflow (both `--update-workflow` and `--update-from-tags`) only works > with **environment-agnostic** snapshots. A snapshot is environment-agnostic when its templates do > not embed specific account IDs or regions. Tests that use context lookups (e.g., VPC or AZ > providers) store dummy resolved values in their snapshots — these are fine for diffing but can > only be deployed if the snapshot doesn't hardcode real environment-specific values. If your test > declares a specific `env: { account, region }`, its snapshot is not portable and the update > workflow will not work reliably. The `integ-runner` will also attempt to warn you if you are making any destructive changes with a message like: ```bash !!! This test contains destructive changes !!! Stack: aws-cdk-lambda-1 - Resource: MyLambdaServiceRole4539ECB6 - Impact: WILL_DESTROY Stack: aws-cdk-lambda-1 - Resource: AliasAliasPermissionAF30F9E8 - Impact: WILL_REPLACE Stack: aws-cdk-lambda-1 - Resource: AliasFunctionUrlDC6EC566 - Impact: WILL_REPLACE Stack: aws-cdk-lambda-1 - Resource: Aliasinvokefunctionurl4CA9917B - Impact: WILL_REPLACE !!! If these destructive changes are necessary, please indicate this on the PR !!! ``` If the destructive changes are expected (and required) then please indicate this on your PR. ##### New tests If you are adding a new test which creates a new snapshot then you should run that specific test with `--disable-update-workflow`. For example, if you are working on a new test `integ.new-test.js` then you would run: ```bash integ-runner --update-on-failed --disable-update-workflow integ.new-test.js ``` This is because for a new test we do not need to test the update workflow (there is nothing to update). ##### Update From Tags Use `--update-from-tags` to verify that a stack can be safely updated through a sequence of specific versions. This is primarily useful for on-call engineers validating that a fix is safe for customers who are currently on a broken version. ```bash integ-runner --update-from-tags v2.150.0,v2.151.0 --update-on-failed integ.my-stack.js ``` This will: 1. Check out the snapshot at tag `v2.150.0` and deploy it (fresh stack creation) 2. Check out the snapshot at tag `v2.151.0` and deploy it (stack update) 3. Deploy the current code (final stack update) 4. If successful, update the snapshot The typical use case is regression verification: version N was good, N+1 introduced a regression, and you have a fix on the current branch. Running with `--update-from-tags vN,vN+1` proves that customers on N+1 can safely move to the fix. Notes: - `--update-from-tags` implies `--force` — all matched tests are deployed regardless of snapshot comparison results. - The runner fails fast if a snapshot does not exist at any of the specified tags. - `allowDestroy` checks apply at every transition, not just the final one. - Cannot be used together with `--no-update-workflow`. - When `--update-from-tags` is provided, the normal merge-base update workflow is skipped. ### watch It can be useful to run an integration test in watch mode when you are iterating on a specific test. ```console integ-runner integ.new-test.js --watch ``` In watch mode the integ test will run similar to `cdk deploy --watch` with the addition of also displaying the assertion results. By default the output will only show the assertion results. - To show the console output from watch run with `-v` - To also stream the CloudWatch logs (i.e. `cdk deploy --watch --logs`) run with `-vv` When running in watch mode most of the integ-runner functionality will be turned off. - Snapshots will not be created - Update workflow will not be run - Stacks will not be cleaned up (you must manually clean up the stacks) - Only a single test can be run Once you are done iterating using watch and want to create the snapshot you can run the integ test like normal to create the snapshot and clean up the test. #### cdk.context.json cdk watch depends on a `cdk.context.json` file existing with a `watch` key. The integ-runner will create a default `cdk.context.json` file if one does not exist. ```json { "watch": {} } ``` You can further edit this file after it is created and add additional `watch` fields. For example: ```json { "watch": { "include": ["**/*.js"] } } ``` ### integ.json schema See [@aws-cdk/cloud-assembly-schema/lib/integ-tests/schema.ts](../cloud-assembly-schema/lib/integ-tests/schema.ts) ### Defining an integration test See the `@aws-cdk/integ-tests` module for information on how to define integration tests for the runner to exercise. ### Config file All options can be configured via the `integ.config.json` configuration file in the current working directory. ```json { "maxWorkers": 10, "parallelRegions": [ "eu-west-1", "ap-southeast-2" ] } ``` Available options can be listed by running the following command: ```sh integ-runner --help ``` To use a different config file, provide the `--config` command-line option. ## Dealing with context and Feature Flags By default, `integ-runner` will use the most recent recommended set of feature flags. This has the consequence that when you upgrade `integ-runner`, the feature flag set may change and affect your generated snapshots, requiring a snapshot update. If your test directory has an `integ.context.json` file or `cdk.json` file upstream from the test, context values from those files will be used and completely replace the built-in flags. This makes sure that your context stays the same, even if you upgrade `integ-runner`. It also gives you the ability to change a context value for all tests at once. If there are multiple files they will not be merged: the closest file to the test will be used, with `integ.context.json` taking precedence over `cdk.json`. If `integ.context.json` is found, it is expected to be a one-level dictionary of context keys to values. If `cdk.json` is found, its `"context"` key will be used as a dictionary of context keys and values (same as the file that typically controls a CDK app's context and feature flags).