## Unit Tests

Unit tests are executed with `npm run build` and `npm run test`. GitHub Actions won't let PRs merge if the tests fail.

To run on Windows, use:

```
jest --testMatch */**/*.test.ts
```

## Integration Test

We have one set of [integration tests](test/default.integ.ts). It is a CDK app that creates a set of runners. Once deployed, the runners should be tested using the [self-hosted.yml](.github/workflows/self-hosted.yml) workflow. This should tell us GitHub integration and all runner provider types are working properly.

We keep a snapshot of the CloudFormation template generated by the integration test in [test/default.integ.snapshot](test/default.integ.snapshot). During build time and PR validation, we validate the integration test still results in the same CloudFormation template as the snapshot. GitHub Actions won't let PRs merge if the tests fail.

If the snapshot changes, you should commit it as part of your PR. You should ideally deploy and test it. The PR should mention whether the integration test was actually deployed and tested.

To assert the snapshot hasn't changed, use:

```
npm run integ:default:assert
```

To deploy the integration test, use:

```
npm run integ:default:deploy
```

To update the snapshot, use:

```
npm run integ:default:snapshot
```

## Manual Tests 

Integration tests check the happy paths. We should also test the unhappy paths manually. This is a list of scenarios we should manually test before releasing a new version:

* Setup page
  * GitHub app
  * Personal access token
  * GitHub Enterprise Server
* Idle reaper
  * Confirm idle runner is stopped automatically
  * Confirm runner doesn't stay registered in GitHub
  * Confirm runner is not retried
  * Step function result is aborted and not failed
* Retries
  * Confirm runner errors are retried
  * Confirm failed runner doesn't stay registered in GitHub
* Failed runner clean-up
  * Confirm `Delete Failed Runner` succeeds and `Rethrow Error` is the state that fails
  * Confirm the execution fails with the error that stopped the runner, and not a generic one
  * Confirm runner doesn't stay registered in GitHub
* Stolen runner detection
  * Confirm a new runner is created when an unknown job is assigned to one of our runners
* Other tests to be automated in the future
  * Go to [`default.integ.ts`](test/default.integ.ts) and enable the test flags at the top

The two retries scenarios can be tested with the following test cases: 

* Start step function without a job actually pending (e.g. by duplicating input from a previous job, or cancelling a job before a runner picks it up)
   * The step function should be aborted as an idle runner
   * No runner should be registered on GitHub at the end
* Let Lambda runner timeout by starting a job that lasts longer than 15 minutes
   * The runner should be retried and eventually the step function should be aborted as an idle runner
   * No runner should be registered on GitHub at the end

### Failed runner clean-up

Failed runner clean-up only happens when a runner provider actually fails. The quickest real failure is a Lambda runner
that times out before the idle reaper gets to it. The reaper's queue has a 10 minute delivery delay, so its first check
is at about 10 minutes no matter what `idleTimeout` says. Any runner timeout below that works:

```typescript
new GitHubRunners(this, 'runners', {
  providers: [
    new LambdaRunnerProvider(this, 'Lambda', {
      timeout: cdk.Duration.minutes(2), // dies well before the reaper's first check
    }),
  ],
  retryOptions: { retry: false }, // one clean pass instead of a day of retries
});
```

Start a step function without a job actually pending (e.g. by duplicating input from a previous job). After about two minutes:

* `Delete Failed Runner` should be green, with `runnerFound` and `runnerDeleted` both true in `$.delete`
* `Rethrow Error` should be the red state, and the execution should fail with the runner timeout and not with a clean-up error
* No runner should be registered on GitHub at the end

### Stolen runners

Stolen runners can be tested with the following steps:

1. Disable the webhook temporarily
2. Run the integration test
3. Confirm no runners were provisioned
4. Enable the webhook
5. Run integration tests again
6. Confirm that all jobs from both workflow runs succeeded