# Testing Security Code

The pattern that makes keychain code testable is a protocol between the app and the Security framework. Unit tests run against an in-memory implementation; tests that need the real keychain, the Secure Enclave or biometric hardware run on devices.

The keychain does not behave the same on Simulator, on a headless CI runner and on a phone. Tests written without that in mind fail intermittently, crash on runners, or pass while proving nothing.

Targets: Swift 5.9 and Swift 6, iOS 17 and 18, with notes for the iOS 26 post-quantum APIs. Background: TN3137 (macOS keychain APIs and implementations), WWDC19 session 413 "Testing in Xcode", the WWDC24 Swift Testing sessions (10179 and 10195), Apple Platform Security, and the OWASP MASTG.

Related: [keychain-fundamentals.md](keychain-fundamentals.md), [cryptokit-public-key.md](cryptokit-public-key.md), [migration-legacy-stores.md](migration-legacy-stores.md), [common-anti-patterns.md](common-anti-patterns.md).

## Contents

- [A protocol in front of the keychain](#a-protocol-in-front-of-the-keychain)
- [Seven mistakes in generated keychain tests](#seven-mistakes-in-generated-keychain-tests)
- [Simulator versus device](#simulator-versus-device)
- [Core test patterns](#core-test-patterns)
- [Secure Enclave tests](#secure-enclave-tests)
- [Biometric flows](#biometric-flows)
- [CI configuration](#ci-configuration)
- [Test plans](#test-plans)
- [Swift Testing](#swift-testing)
- [Further techniques](#further-techniques)
- [Takeaways](#takeaways)
- [Checklist](#checklist)

## A protocol in front of the keychain

```swift
import Foundation
import Security

enum VaultStoreError: Error, Equatable {
    case duplicateItem, itemNotFound, authFailed, userCanceled, interactionNotAllowed
    case unexpectedData
    case unhandled(status: OSStatus)

    init(status: OSStatus) {
        switch status {
        case errSecDuplicateItem: self = .duplicateItem
        case errSecItemNotFound: self = .itemNotFound
        case errSecAuthFailed: self = .authFailed
        case errSecUserCanceled: self = .userCanceled
        case errSecInteractionNotAllowed: self = .interactionNotAllowed
        default: self = .unhandled(status: status)
        }
    }
}

protocol SecretStore: Sendable {
    func put(_ secret: Data, for name: String) throws
    func value(for name: String) throws -> Data?
    func replace(_ secret: Data, for name: String) throws
    func remove(for name: String) throws
    func removeAll() throws
}
```

The production implementation, `KeychainSecretStore`, follows the rules in [keychain-fundamentals.md](keychain-fundamentals.md): `put` calls `SecItemAdd` and, on `errSecDuplicateItem`, `SecItemUpdate` with the new data; `remove` treats `errSecItemNotFound` as success; it is a final class whose stored properties are all `let`, marked `@unchecked Sendable`.

The test double keeps everything in a dictionary, counts calls and throws on request. It runs on Simulator, on CI and even on Linux, and needs no entitlements.

```swift
final class InMemorySecretStore: SecretStore, @unchecked Sendable {
    private let lock = NSLock()
    private var rows: [String: Data] = [:]
    private(set) var saveCount = 0
    private(set) var readCount = 0
    var failure: VaultStoreError?

    func put(_ secret: Data, for name: String) throws {
        try locked { saveCount += 1; rows[name] = secret }
    }
    func value(for name: String) throws -> Data? {
        try locked { readCount += 1; return rows[name] }
    }
    func replace(_ secret: Data, for name: String) throws {
        try locked {
            guard rows[name] != nil else { throw VaultStoreError.itemNotFound }
            rows[name] = secret
        }
    }
    func remove(for name: String) throws { try locked { rows[name] = nil } }
    func removeAll() throws { try locked { rows.removeAll() } }

    private func locked<T>(_ body: () throws -> T) throws -> T {
        lock.lock(); defer { lock.unlock() }
        if let failure { throw failure }
        return try body()
    }
}
```

Code under test depends only on the protocol:

```swift
struct SessionKeeper {
    let store: any SecretStore

    func remember(token: String) throws {
        try store.put(Data(token.utf8), for: "session.token")
    }
    func currentToken() throws -> String? {
        try store.value(for: "session.token").map { String(decoding: $0, as: UTF8.self) }
    }
}
```

## Seven mistakes in generated keychain tests

1. **No cleanup around real-keychain tests.** The first run passes; the second hits `errSecDuplicateItem` (-25299).
2. **Expecting a Secure Enclave on Simulator.** The Simulator has no Secure Enclave. Branch at compile time with `#if targetEnvironment(simulator)` and treat `SecureEnclave.isAvailable` as the runtime check on real hardware only; do not build Simulator behaviour on what the flag reports there. Creating a Secure Enclave key without that guard throws a `CryptoKitError` or fails the test run.
3. **Only testing the happy path.** Cover -25299, -25300 (`errSecItemNotFound`), -25293 (`errSecAuthFailed`), -128 (`errSecUserCanceled`) and -25308 (`errSecInteractionNotAllowed`).
4. **Expecting biometric hardware.** `canEvaluatePolicy(.deviceOwnerAuthenticationWithBiometrics, error:)` returns false on Simulator unless enrolment is simulated.
5. **No host app.** From Xcode 9 on, a logic test bundle on the iOS Simulator has no keychain access of its own. Give it a host application; otherwise expect `SecItemAdd` to fail with -25300 or -34018 (`errSecMissingEntitlement`).
6. **Unscoped items.** Tests must use their own `kSecAttrService` so they never touch app data or each other's rows.
7. **Mixing up keychain types.** The `security` command-line tool works on file-based keychains. iOS apps, and macOS apps using the data protection keychain, use a different store, so `security create-keychain` on a CI runner creates the wrong kind of keychain for testing `SecItemAdd`.

## Simulator versus device

| Capability | Simulator | Device |
| --- | --- | --- |
| Keychain add, read, update, delete | Works | Works |
| CryptoKit software algorithms | Works | Works, hardware accelerated |
| `kSecAttrAccessible` | Accepted, not enforced by hardware | Enforced |
| Secure Enclave | Not present; key creation fails | Present on A7 and later |
| Biometric-protected items | Returned without any prompt | Prompt shown |
| `canEvaluatePolicy` for biometrics | False, unless enrolment is simulated from the Simulator's Features menu | True when enrolled |
| ML-KEM and ML-DSA (iOS 26) | Works in software | Works |

The biometric row is the dangerous one: a Simulator test for a Face ID protected item passes without any authentication, which proves nothing about the real behaviour.

Guards:

```swift
import CryptoKit
import XCTest

func makeSigner() throws -> any SigningKeyProvider {
    #if targetEnvironment(simulator)
    return SoftwareSigningKey()
    #else
    return SecureEnclave.isAvailable ? try SecureEnclaveSigningKey() : SoftwareSigningKey()
    #endif
}

func requireHardware() throws {
    #if targetEnvironment(simulator)
    throw XCTSkip("Requires a physical device")
    #endif
}

let runningInSimulator = ProcessInfo.processInfo.environment["SIMULATOR_DEVICE_NAME"] != nil
let runningUnderTests = ProcessInfo.processInfo.environment["XCTestConfigurationFilePath"] != nil
```

## Core test patterns

Real-keychain integration test, with its own service and cleanup on both sides:

```swift
final class KeychainSecretStoreTests: XCTestCase {
    private var store: KeychainSecretStore!

    override func setUpWithError() throws {
        store = KeychainSecretStore(service: "tests.vault.\(name)")
        try store.removeAll()
    }

    override func tearDownWithError() throws {
        try store.removeAll()
    }

    func testSaveThenRead_ReturnsSameBytes() throws {
        let secret = Data("pin-4821".utf8)
        try store.put(secret, for: "pin")
        XCTAssertEqual(try store.value(for: "pin"), secret)
    }
}
```

What not to write: a test that calls `SecItemAdd` directly with no service attribute and no cleanup. It passes exactly once and then fails with -25299 forever.

Error paths through the mock:

```swift
func testSaveWhileLocked_SurfacesInteractionNotAllowed() {
    let mock = InMemorySecretStore()
    mock.failure = .interactionNotAllowed
    let keeper = SessionKeeper(store: mock)

    XCTAssertThrowsError(try keeper.remember(token: "abc")) { error in
        XCTAssertEqual(error as? VaultStoreError, .interactionNotAllowed)
    }
}
```

Repeat for `.duplicateItem`, `.authFailed` and `.userCanceled`. `interactionNotAllowed` deserves particular care: it is the failure CI runs hit most.

CryptoKit tests are Simulator-safe:

```swift
func testGCMRoundTripAndWrongKey() throws {
    let key = SymmetricKey(size: .bits256)
    let wire = try XCTUnwrap(try AES.GCM.seal(Data("ledger".utf8), using: key).combined)
    let parsed = try AES.GCM.SealedBox(combined: wire)
    XCTAssertEqual(try AES.GCM.open(parsed, using: key), Data("ledger".utf8))
    let stranger = SymmetricKey(size: .bits256)
    XCTAssertThrowsError(try AES.GCM.open(parsed, using: stranger))
}

func testP256SignatureRejectsTampering() throws {
    let signer = P256.Signing.PrivateKey()
    let signature = try signer.signature(for: Data("amount=10".utf8))
    XCTAssertTrue(signer.publicKey.isValidSignature(signature, for: Data("amount=10".utf8)))
    XCTAssertFalse(signer.publicKey.isValidSignature(signature, for: Data("amount=99".utf8)))
}

func testX25519PartiesDeriveMatchingKeys() throws {
    let alice = Curve25519.KeyAgreement.PrivateKey()
    let bob = Curve25519.KeyAgreement.PrivateKey()
    let salt = Data("pairing-v1".utf8)
    func derive(_ mine: Curve25519.KeyAgreement.PrivateKey,
                _ theirs: Curve25519.KeyAgreement.PublicKey) throws -> SymmetricKey {
        try mine.sharedSecretFromKeyAgreement(with: theirs)
            .hkdfDerivedSymmetricKey(using: SHA384.self, salt: salt,
                                     sharedInfo: Data("chat".utf8), outputByteCount: 32)
    }
    let aliceKey = try derive(alice, bob.publicKey)
    let bobKey = try derive(bob, alice.publicKey)
    let sealed = try ChaChaPoly.seal(Data("hello".utf8), using: aliceKey)
    XCTAssertEqual(try ChaChaPoly.open(sealed, using: bobKey), Data("hello".utf8))
}
```

Post-quantum tests follow the same round-trip shape and are gated on availability; the software implementations run on the iOS 26 Simulator runtime.

```swift
@available(iOS 26, *)
func verifyMLKEMRoundTrip() throws {
    let receiver = try MLKEM768.PrivateKey()
    let result = try receiver.publicKey.encapsulate()
    let recovered = try receiver.decapsulate(result.encapsulated)
    let bytes = { (key: SymmetricKey) in key.withUnsafeBytes { Data($0) } }
    XCTAssertEqual(bytes(recovered), bytes(result.sharedSecret))
}
```

## Secure Enclave tests

The enclave-backed signing key type and the software `P256.Signing.PrivateKey` share no common type, so a factory cannot return "one of them" directly. Put both behind a protocol.

```swift
protocol SigningKeyProvider: Sendable {
    func signature(over payload: Data) throws -> Data
    var publicKeyDER: Data { get }
}

struct SecureEnclaveSigningKey: SigningKeyProvider {
    private let enclaveKey: SecureEnclave.P256.Signing.PrivateKey

    init() throws {
        guard SecureEnclave.isAvailable else { throw VaultStoreError.unhandled(status: errSecUnimplemented) }
        guard let access = SecAccessControlCreateWithFlags(nil,
                kSecAttrAccessibleWhenUnlockedThisDeviceOnly, [.privateKeyUsage], nil) else {
            throw VaultStoreError.unhandled(status: errSecParam)
        }
        enclaveKey = try .init(accessControl: access)
    }
    func signature(over payload: Data) throws -> Data {
        try enclaveKey.signature(for: payload).derRepresentation
    }
    var publicKeyDER: Data { enclaveKey.publicKey.derRepresentation }
}

struct SoftwareSigningKey: SigningKeyProvider {
    private let softwareKey = P256.Signing.PrivateKey()
    func signature(over payload: Data) throws -> Data {
        try softwareKey.signature(for: payload).derRepresentation
    }
    var publicKeyDER: Data { softwareKey.publicKey.derRepresentation }
}

enum SignerFactory {
    static func make() -> any SigningKeyProvider {
        (try? makeSigner()) ?? SoftwareSigningKey()
    }
}
```

`.privateKeyUsage` is always part of a Secure Enclave signing key's access control flags; add `.biometryCurrentSet` or `.userPresence` next to it when the key should require authentication.

A test that creates a Secure Enclave key unconditionally crashes or fails on Simulator and CI. Skip it, or test through the protocol:

```swift
func testEnclaveSignature_VerifiesWithPublicKey() throws {
    try requireHardware()
    try XCTSkipUnless(SecureEnclave.isAvailable, "No Secure Enclave on this machine")
    let signer = try SecureEnclaveSigningKey()
    try assertVerifies(signer)
}

func testAnySigner_ProducesVerifiableSignatures() throws {
    try assertVerifies(SignerFactory.make())
}

private func assertVerifies(_ provider: any SigningKeyProvider) throws {
    let challenge = Data("challenge-17".utf8)
    let verifier = try P256.Signing.PublicKey(derRepresentation: provider.publicKeyDER)
    let der = try provider.signature(over: challenge)
    XCTAssertTrue(verifier.isValidSignature(try .init(derRepresentation: der), for: challenge))
}
```

## Biometric flows

Put `LAContext` behind a protocol so tests control the outcome. The quickest version copies `LAContext`'s own `canEvaluatePolicy(_:error:)` and `evaluatePolicy(_:localizedReason:reply:)` requirements into a `BiometricAuthContext` protocol and declares `extension LAContext: BiometricAuthContext {}`. A small adapter, shown here, keeps the protocol in your own vocabulary instead. Subclassing `LAContext` also works and is shorter, at the cost of coupling tests to the real class.

```swift
import LocalAuthentication

protocol BiometricAuthContext {
    func biometricsReady() -> Bool
    func authenticate(reason: String, reply: @escaping @Sendable (Bool, Error?) -> Void)
}

struct SystemBiometricContext: BiometricAuthContext {
    private let context = LAContext()
    func biometricsReady() -> Bool {
        context.canEvaluatePolicy(.deviceOwnerAuthenticationWithBiometrics, error: nil)
    }
    func authenticate(reason: String, reply: @escaping @Sendable (Bool, Error?) -> Void) {
        context.evaluatePolicy(.deviceOwnerAuthenticationWithBiometrics, localizedReason: reason, reply: reply)
    }
}
```

```swift
final class BiometricPrompt {
    private let context: BiometricAuthContext
    init(context: BiometricAuthContext) { self.context = context }

    func confirm(reason: String, then finish: @escaping @Sendable (Result<Void, Error>) -> Void) {
        guard context.biometricsReady() else {
            finish(.failure(LAError(.biometryNotAvailable)))
            return
        }
        context.authenticate(reason: reason) { ok, error in
            finish(ok ? .success(()) : .failure(error ?? LAError(.authenticationFailed)))
        }
    }
}

final class StubBiometricContext: BiometricAuthContext {
    var available = true
    var succeeds = true
    var error: Error?
    private(set) var evaluateCalled = false

    func biometricsReady() -> Bool { available }
    func authenticate(reason: String, reply: @escaping @Sendable (Bool, Error?) -> Void) {
        evaluateCalled = true
        reply(succeeds, self.error)
    }
}
```

This tests UI gating only. A secret behind the prompt still has to live in a keychain item bound with `SecAccessControl`, as [biometric-authentication.md](biometric-authentication.md) explains.

Scenarios to cover:

| Case | Error | Expected handling |
| --- | --- | --- |
| Success | none | Proceed |
| User cancels | `.userCancel` | Offer retry or abort |
| Too many failures | `.biometryLockout` | Fall back to the device passcode |
| Nothing enrolled | `.biometryNotEnrolled` | Explain how to enrol |

```swift
func testUnavailableBiometrics_NeverEvaluates() async {
    let stub = StubBiometricContext()
    stub.available = false
    let done = expectation(description: "completion")
    BiometricPrompt(context: stub).confirm(reason: "Approve transfer") { outcome in
        if case .success = outcome { XCTFail("Expected a failure") }
        done.fulfill()
    }
    await fulfillment(of: [done], timeout: 1)
    XCTAssertFalse(stub.evaluateCalled)
}
```

The completion checks the result with `if case` rather than
`XCTAssertThrowsError(try outcome.get())`: a `try` anywhere in a closure body
makes Swift infer the closure as throwing, and `finish` is non-throwing. The
test is `async` and awaits `fulfillment(of:)` because `waitForExpectations` is
main-actor isolated.

## CI configuration

`errSecInteractionNotAllowed` (-25308) is the error CI runs hit most: the runner's keychain is locked, or the operation wants a GUI that a headless session does not have.

Signing setup on a hosted macOS runner:

```sh
KC="${RUNNER_TEMP:-$TMPDIR}/ci-signing.keychain-db"
security create-keychain -p "$KC_PASSWORD" "$KC"
security set-keychain-settings -l -u -t 21600 "$KC"
security unlock-keychain -p "$KC_PASSWORD" "$KC"
security list-keychains -s "$KC" -d user
security import signing.p12 -k "$KC" -f pkcs12 -t cert -A -P "$P12_PASSWORD"
security set-key-partition-list -s -k "$KC_PASSWORD" -S apple-tool:,apple: "$KC"

xcodebuild test -scheme Vault -testPlan CITests \
  -destination "platform=iOS Simulator,name=$SIM_NAME"

# always-run cleanup step
security delete-keychain "$KC"
```

`set-key-partition-list` after the import is the step people miss. Without it `codesign` waits for a GUI confirmation that never comes and the job hangs. `-A` on the import grants every application access to the key, which is acceptable only for a throwaway CI keychain.

Xcode Cloud builds on ephemeral machines with managed signing, so there is no keychain to configure by hand. Enable Keychain Sharing on the test host, and expect -25308 when Swift Package Manager tries to save repository credentials.

fastlane's `setup_ci` creates a temporary keychain named `fastlane_tmp_keychain` and makes it the default, which can disturb the login keychain on a self-hosted runner.

```ruby
lane :ci_tests do
  setup_ci(timeout: 3600)
  sync_code_signing(type: "development", readonly: is_ci)
  run_tests(scheme: "Vault", testplan: "CITests", device: "iPhone 16")
end
```

| Status | Meaning on CI | Fix |
| --- | --- | --- |
| -25308 | Keychain locked or needs UI | Unlock it and set the partition list |
| -34018 | Missing entitlement | Give the test host the keychain entitlement |
| -25300 | Item not found, often no host app | Run tests inside a host app with the Keychain capability |
| -67585 | `errSecInternalComponent` | The import happened but no partition list was applied |
| -25307 | No default keychain | Create one and set it as default |

For the test host, add an almost empty iOS app target and turn on Keychain Sharing for it. Then set `TEST_HOST` and `BUNDLE_LOADER` on the test target so the bundle loads inside that app.

## Test plans

- `CITests.xctestplan`: mock keychain and Simulator-safe CryptoKit tests only.
- `DeviceTests.xctestplan`: real keychain, Secure Enclave, biometric hardware; runs on a physical device.

```sh
xcodebuild test -scheme Vault -testPlan CITests -destination "platform=iOS Simulator,name=$SIM_NAME"
xcodebuild test -scheme Vault -testPlan DeviceTests -destination 'platform=iOS,id=<UDID>'
```

Run the first on every push and the second nightly on a device farm.

## Swift Testing

```swift
import Testing

extension Tag {
    @Tag static var keychain: Self
    @Tag static var deviceOnly: Self
    @Tag static var ciSafe: Self
}

@Suite(.serialized, .tags(.keychain))
struct VaultTests {
    @Test(.tags(.ciSafe))
    func rememberedTokenIsReadable() throws {
        let keeper = SessionKeeper(store: InMemorySecretStore())
        try keeper.remember(token: "t-1")
        let token = try #require(try keeper.currentToken())
        #expect(token == "t-1")
    }

    static let onBuildServer = ProcessInfo.processInfo.environment["CI"] != nil

    @Test(.tags(.deviceOnly), .enabled(if: !onBuildServer))
    func realKeychainRoundTrip() throws {
        let store = KeychainSecretStore(service: "tests.vault.swift-testing")
        try store.removeAll()
        defer { try? store.removeAll() }
        try store.put(Data("x".utf8), for: "k")
        #expect(try store.value(for: "k") == Data("x".utf8))
    }

    @Test(arguments: [VaultStoreError.duplicateItem, .authFailed, .userCanceled, .interactionNotAllowed])
    func injectedFailuresPropagate(_ failure: VaultStoreError) {
        let mock = InMemorySecretStore()
        mock.failure = failure
        #expect(throws: VaultStoreError.self) { try SessionKeeper(store: mock).remember(token: "t") }
    }
}
```

`.serialized` keeps tests that share keychain state from running in parallel. Test plans can include or exclude by tag: `ciSafe` on CI, `deviceOnly` on the device farm.

## Further techniques

**Migration tests.** Inject both stores: an isolated `UserDefaults(suiteName:)` cleared with `removePersistentDomain(forName:)` and the mock keychain. Assert the value arrived, was read back, and is gone from the source.

```swift
func testMigrationMovesAndVerifies() throws {
    let suiteName = "tests.migration"
    let defaults = try XCTUnwrap(UserDefaults(suiteName: suiteName))
    defaults.removePersistentDomain(forName: suiteName)
    defaults.set("legacy-token", forKey: "token")
    let store = InMemorySecretStore()

    try LegacyTokenMigrator(defaults: defaults, store: store).run()

    XCTAssertEqual(try store.value(for: "token"), Data("legacy-token".utf8))
    XCTAssertNil(defaults.string(forKey: "token"))
}
```

`LegacyTokenMigrator` saves, reads back and compares, and removes the `UserDefaults` value only after the comparison succeeds, as in [migration-legacy-stores.md](migration-legacy-stores.md).

**Performance.**

```swift
func testSaveDeleteCost() {
    let runs = XCTMeasureOptions()
    runs.iterationCount = 20
    measure(metrics: [XCTCPUMetric(), XCTClockMetric()], options: runs) {
        try? store.put(Data(count: 64), for: "perf")
        try? store.remove(for: "perf")
    }
}
```

**Mutation testing.** Coverage says code ran, not that a test would notice it changing. Mutations such as flipping `==` and `!=`, deleting a `SecItemDelete` call or swapping `&&` and `||` expose the gap; a project can show 81% line coverage and a 16% mutation score. Muter, installed with Homebrew from the `muter-mutation-testing/muter` tap, provides the operators: `RelationalOperatorReplacement` finds inverted checks that would become authentication bypasses, and `RemoveSideEffects` finds logout paths whose deletes nobody verifies. Aim for a mutation score above 80%.

**MASTG-TEST-0052.** Sensitive data must be in the keychain, not in `NSUserDefaults` or property lists, and because the keychain survives uninstall the app needs a fresh-install detector: a `has_launched` marker that triggers `removeAll()` when missing, run after protected data is available.

## Takeaways

Aim for more than 95% of the suite running against the mock with no entitlements at all, and keep real-keychain tests in the device test plan. Three facts are easy to miss: on Simulator a Face ID or Touch ID protected item comes back with no prompt at all, `security create-keychain` makes a keychain of the wrong type for iOS-style tests, and mutation testing finds inverted conditions and removed side effects that coverage does not.

## Checklist

- [ ] All keychain access goes through the `SecretStore` protocol.
- [ ] The mock can inject duplicate, auth-failed, user-canceled, interaction-not-allowed and not-found failures.
- [ ] Real-keychain tests use their own service and clean up in `setUp` and `tearDown`.
- [ ] Secure Enclave tests are compiled out or skipped on Simulator, check `SecureEnclave.isAvailable` on hardware, or go through the protocol fallback.
- [ ] `LAContext` is behind a protocol; success, cancel, lockout and not-enrolled are covered.
- [ ] Two test plans: `CITests` and `DeviceTests`.
- [ ] CI sets the key partition list after importing certificates; the test host has Keychain Sharing.
- [ ] CryptoKit round trips for AES-GCM, ChaChaPoly, P256 and Curve25519, each with a wrong-key or tampered-input case.
- [ ] Every reachable `OSStatus` has an injected-failure test.
- [ ] Migration tests use an isolated suite and the mock, and check the source is cleared after verification.
- [ ] Muter reports at least 80% with `RelationalOperatorReplacement` and `RemoveSideEffects`.