# Testing Keychain, CryptoKit, and Biometric Code

> Scope: Unit, integration, and CI patterns for validating keychain, CryptoKit, and biometric security code across simulator, CI runners, and physical devices.

**Protocol-based abstraction is the single most important pattern for testable security code.** Wrapping Security framework calls behind a Swift protocol lets you inject an in-memory mock for unit tests while reserving real keychain integration tests for physical devices. The core challenge is that keychain behavior differs dramatically across three environments  -  Xcode simulator, CI runner, and physical device  -  and tests that ignore these differences produce flaky failures, crashes, or false confidence.

This reference covers mock design, CryptoKit round-trip tests, Secure Enclave guards, biometric mocking, CI/CD keychain creation, simulator limitations, Swift Testing framework patterns, mutation testing, and OWASP MASTG validation. All code targets Swift 5.9+/6.0, iOS 17-18+, with iOS 26 post-quantum notes where applicable.

Key sources: Apple TN3137 "On Mac keychain APIs and implementations," WWDC19-413 "Testing in Xcode," WWDC24-10179/10195 "Meet/Go further with Swift Testing," Apple Platform Security Guide, OWASP MASTG.

---

## Contents

- [Protocol-Based Keychain Abstraction](#protocol-based-keychain-abstraction)
  - [KeychainServiceProtocol with Real and Mock Implementations](#keychainserviceprotocol-with-real-and-mock-implementations)
- [Seven Mistakes AI Generators Make in Keychain Tests](#seven-mistakes-ai-generators-make-in-keychain-tests)
- [Simulator vs. Device Testing Matrix](#simulator-vs-device-testing-matrix)
  - [Conditional Compilation and Runtime Guards](#conditional-compilation-and-runtime-guards)
- [Essential Testing Patterns](#essential-testing-patterns)
  - [setUp/tearDown Cleanup for Real Keychain Tests](#setupteardown-cleanup-for-real-keychain-tests)
  - [No Cleanup  -  Flaky Across Runs](#no-cleanup-flaky-across-runs)
  - [Testing Error Paths with Injected Failures](#testing-error-paths-with-injected-failures)
  - [CryptoKit Round-Trip Tests (Simulator-Safe)](#cryptokit-round-trip-tests-simulator-safe)
- [Secure Enclave Test Strategy  -  Protocol Fallback](#secure-enclave-test-strategy-protocol-fallback)
  - [SigningKeyProvider Protocol with SE/Software Implementations](#signingkeyprovider-protocol-with-sesoftware-implementations)
  - [Testing Secure Enclave Code](#testing-secure-enclave-code)
- [Biometric Flow Testing  -  LAContext Mocking](#biometric-flow-testing-lacontext-mocking)
  - [Protocol-Based Approach (Preferred)](#protocol-based-approach-preferred)
  - [Biometric Scenarios to Cover](#biometric-scenarios-to-cover)
- [CI/CD Pipeline Configuration](#cicd-pipeline-configuration)
  - [GitHub Actions](#github-actions)
  - [Common CI Error Reference](#common-ci-error-reference)
  - [Test Host App Requirement](#test-host-app-requirement)
- [Xcode Test Plans  -  Separating Simulator from Device](#xcode-test-plans-separating-simulator-from-device)
- [Swift Testing Framework Patterns](#swift-testing-framework-patterns)
- [Advanced Patterns](#advanced-patterns)
  - [Migration Testing: UserDefaults to Keychain](#migration-testing-userdefaults-to-keychain)
  - [Performance Testing](#performance-testing)
  - [Mutation Testing](#mutation-testing)
  - [OWASP MASTG Keychain Validation](#owasp-mastg-keychain-validation)
- [Conclusion](#conclusion)
- [Summary Checklist](#summary-checklist)

## Protocol-Based Keychain Abstraction

The foundation of testable keychain code is a protocol abstracting the four Security framework operations. Every view model, service, or manager that touches the keychain depends on this protocol, never on the Security framework directly.

### KeychainServiceProtocol with Real and Mock Implementations

```swift
import Foundation
import Security

enum KeychainError: Error, Equatable {
    case duplicateItem
    case itemNotFound
    case authFailed
    case interactionNotAllowed
    case unexpectedData
    case unhandledError(status: OSStatus)

    init(status: OSStatus) {
        switch status {
        case errSecDuplicateItem:          self = .duplicateItem
        case errSecItemNotFound:           self = .itemNotFound
        case errSecAuthFailed:             self = .authFailed
        case errSecInteractionNotAllowed:  self = .interactionNotAllowed
        default:                           self = .unhandledError(status: status)
        }
    }
}

protocol KeychainServiceProtocol: Sendable {
    func save(_ data: Data, forKey key: String) throws
    func read(forKey key: String) throws -> Data?
    func update(_ data: Data, forKey key: String) throws
    func delete(forKey key: String) throws
    func deleteAll() throws
}
```

The real `KeychainService` implementation wraps `SecItem*` calls with the add-or-update pattern and proper `OSStatus` mapping (see `keychain-fundamentals.md` for the full implementation). Key points: `save` attempts update first to avoid `errSecDuplicateItem`; `delete` treats `errSecItemNotFound` as success; the class conforms to `@unchecked Sendable` with immutable stored properties.

The mock replaces Security framework with a dictionary. Runs everywhere  -  simulator, CI, even Linux  -  with zero entitlement requirements. Supports injectable errors and call counting:

```swift
final class MockKeychainService: KeychainServiceProtocol, @unchecked Sendable {
    var storage: [String: Data] = [:]
    var saveCallCount = 0
    var readCallCount = 0
    var deleteCallCount = 0
    var errorToThrow: KeychainError?

    func save(_ data: Data, forKey key: String) throws {
        if let error = errorToThrow { throw error }
        saveCallCount += 1
        storage[key] = data
    }

    func read(forKey key: String) throws -> Data? {
        if let error = errorToThrow { throw error }
        readCallCount += 1
        return storage[key]
    }

    func update(_ data: Data, forKey key: String) throws {
        if let error = errorToThrow { throw error }
        guard storage[key] != nil else { throw KeychainError.itemNotFound }
        storage[key] = data
    }

    func delete(forKey key: String) throws {
        if let error = errorToThrow { throw error }
        storage.removeValue(forKey: key)
        deleteCallCount += 1
    }

    func deleteAll() throws {
        if let error = errorToThrow { throw error }
        storage.removeAll()
    }
}
```

Business logic depends only on the protocol  -  never on `SecItem*` directly:

```swift
final class AuthenticationManager {
    private let keychain: KeychainServiceProtocol

    init(keychain: KeychainServiceProtocol) {
        self.keychain = keychain
    }

    func storeToken(_ token: String) throws {
        guard let data = token.data(using: .utf8) else {
            throw KeychainError.unexpectedData
        }
        try keychain.save(data, forKey: "auth_token")
    }

    func retrieveToken() throws -> String? {
        guard let data = try keychain.read(forKey: "auth_token") else { return nil }
        return String(data: data, encoding: .utf8)
    }
}
```

---

## Seven Mistakes AI Generators Make in Keychain Tests

Both research providers independently identified overlapping anti-patterns. This merged list covers the full set:

**1. Tests that use the real keychain without cleanup.** Tests calling `SecItemAdd` directly leave state across runs. Second run fails with `errSecDuplicateItem` (-25299). AI generators rarely include `setUp`/`tearDown` cleanup.

**2. Assuming Secure Enclave exists on simulator.** `SecureEnclave.isAvailable` returns `false` on every simulator. Tests calling `SecureEnclave.P256.Signing.PrivateKey()` directly throw `CryptoKitError` on simulator and crash CI.

**3. Not testing error paths.** Real keychain code must handle `errSecDuplicateItem` (-25299), `errSecItemNotFound` (-25300), `errSecAuthFailed` (-25293), and `errSecInteractionNotAllowed` (-25308). AI generators almost never test these failure modes.

**4. Assuming biometric hardware.** Tests instantiating a real `LAContext` and asserting `canEvaluatePolicy(.deviceOwnerAuthenticationWithBiometrics)` returns `true` fail on simulator where no biometric hardware exists.

**5. Missing test host app.** Since Xcode 9, test bundles on iOS simulator require a host app to access the keychain. Without one, `SecItemAdd` returns `-25300` or `-34018`. AI generators never mention this requirement.

**6. No service/account scoping.** Tests omitting `kSecAttrService` match items from other tests or even other apps. Every keychain operation in tests must use a unique, test-specific service identifier.

**7. Confusing data protection keychain with file-based keychain.** Per Apple TN3137, macOS has two keychain implementations. The `security` CLI works with the file-based keychain; iOS apps use the data protection keychain. CI scripts using `security create-keychain` create the wrong type for `SecItemAdd` targets.

---

## Simulator vs. Device Testing Matrix

Understanding exactly what works where prevents entire categories of test failures:

| Feature                                                       | Simulator                             | Physical Device              |
| ------------------------------------------------------------- | ------------------------------------- | ---------------------------- |
| Keychain CRUD (`SecItemAdd`, etc.)                            | ✅ Works                              | ✅ Works                     |
| CryptoKit software crypto (AES-GCM, ChaChaPoly, P256, SHA256) | ✅ Software                           | ✅ Hardware-accelerated      |
| `kSecAttrAccessible` values                                   | ✅ Accepted but not hardware-enforced | ✅ Hardware-enforced         |
| `SecureEnclave.isAvailable`                                   | Returns **false**                     | Returns **true** (A7+)       |
| `SecureEnclave.P256.Signing.PrivateKey()`                     | ❌ Throws                             | ✅ Works                     |
| Biometric prompt on protected items                           | ❌ Skipped  -  value returned silently  | ✅ Shows prompt              |
| `LAContext.canEvaluatePolicy(.biometrics)`                    | Returns **false**                     | Returns **true** if enrolled |
| Face ID simulation via Xcode menu                             | ✅ Manual only                        | N/A (real hardware)          |
| Post-quantum (ML-KEM, ML-DSA) iOS 26+                         | ✅ Software (iOS 26 runtime)          | ✅ Works                     |

**Critical subtlety:** On simulator, keychain items protected with `kSecAttrAccessControl` and biometric flags return their value without showing a biometric prompt. Simulator tests that store biometric-protected items and read them succeed silently, giving false confidence the biometric gate works.

### Conditional Compilation and Runtime Guards

```swift
// Compile-time: exclude SE code on simulator
#if targetEnvironment(simulator)
    let signingKey = SoftwareSigningKey()
#else
    let signingKey = SecureEnclave.isAvailable
        ? try SecureEnclaveSigningKey()
        : SoftwareSigningKey()
#endif

// Runtime skip in XCTest
func testDeviceOnlyFeature() throws {
    #if targetEnvironment(simulator)
    throw XCTSkip("Requires physical device")
    #endif
    // Device-only test code here
}

// Runtime detection via ProcessInfo
struct EnvironmentDetector {
    static var isSimulator: Bool {
        ProcessInfo.processInfo.environment["SIMULATOR_DEVICE_NAME"] != nil
    }
    static var isRunningTests: Bool {
        ProcessInfo.processInfo.environment["XCTestConfigurationFilePath"] != nil
    }
}
```

---

## Essential Testing Patterns

### setUp/tearDown Cleanup for Real Keychain Tests

```swift
final class KeychainIntegrationTests: XCTestCase {
    private let testService = "com.tests.keychain-integration"
    private var keychain: KeychainService!

    override func setUp() {
        super.setUp()
        keychain = KeychainService(service: testService)
        try? keychain.deleteAll()  // Clean slate
    }

    override func tearDown() {
        try? keychain.deleteAll()  // Leave no trace
        super.tearDown()
    }

    func testSaveAndRetrieveToken() throws {
        let token = "test-jwt-token-12345"
        try keychain.save(token.data(using: .utf8)!, forKey: "access_token")
        let retrieved = try keychain.read(forKey: "access_token")
        XCTAssertEqual(String(data: retrieved!, encoding: .utf8), token)
    }
}
```

### No Cleanup  -  Flaky Across Runs

```swift
// ❌ INCORRECT: No cleanup, no isolation
final class BadKeychainTests: XCTestCase {
    func testSaveToken() {
        let query: [String: Any] = [
            kSecClass as String: kSecClassGenericPassword,
            kSecAttrAccount as String: "token",
            kSecValueData as String: "secret".data(using: .utf8)!
        ]
        let status = SecItemAdd(query as CFDictionary, nil)
        XCTAssertEqual(status, errSecSuccess)
        // First run: passes ✅
        // Second run: FAILS with errSecDuplicateItem (-25299) ❌
    }
}
```

### Testing Error Paths with Injected Failures

```swift
final class KeychainErrorPathTests: XCTestCase {
    var mockKeychain: MockKeychainService!
    var authManager: AuthenticationManager!

    override func setUp() {
        mockKeychain = MockKeychainService()
        authManager = AuthenticationManager(keychain: mockKeychain)
    }

    func testStoreToken_whenDuplicateItem_throwsExpectedError() {
        mockKeychain.errorToThrow = .duplicateItem
        XCTAssertThrowsError(try authManager.storeToken("token")) { error in
            XCTAssertEqual(error as? KeychainError, .duplicateItem)
        }
    }

    func testRetrieveToken_whenAuthFailed_throwsError() {
        mockKeychain.errorToThrow = .authFailed
        XCTAssertThrowsError(try authManager.retrieveToken()) { error in
            XCTAssertEqual(error as? KeychainError, .authFailed)
        }
    }

    func testRetrieveToken_whenInteractionNotAllowed_throwsError() {
        // Simulates the most common CI failure scenario
        mockKeychain.errorToThrow = .interactionNotAllowed
        XCTAssertThrowsError(try authManager.retrieveToken()) { error in
            XCTAssertEqual(error as? KeychainError, .interactionNotAllowed)
        }
    }
}
```

### CryptoKit Round-Trip Tests (Simulator-Safe)

All CryptoKit software operations work on simulator. These tests run everywhere:

```swift
import XCTest
import CryptoKit

final class CryptoKitTests: XCTestCase {

    func testAESGCMRoundTrip() throws {
        let key = SymmetricKey(size: .bits256)
        let plaintext = "Sensitive credentials".data(using: .utf8)!
        let sealedBox = try AES.GCM.seal(plaintext, using: key)
        let ciphertext = sealedBox.combined!
        XCTAssertNotEqual(ciphertext, plaintext)

        let reopened = try AES.GCM.SealedBox(combined: ciphertext)
        let decrypted = try AES.GCM.open(reopened, using: key)
        XCTAssertEqual(decrypted, plaintext)
    }

    func testAESGCMWrongKeyFails() throws {
        let correctKey = SymmetricKey(size: .bits256)
        let wrongKey = SymmetricKey(size: .bits256)
        let sealed = try AES.GCM.seal("secret".data(using: .utf8)!, using: correctKey)
        XCTAssertThrowsError(try AES.GCM.open(sealed, using: wrongKey))
    }

    func testP256SignVerify() throws {
        let privateKey = P256.Signing.PrivateKey()
        let data = "Message to authenticate".data(using: .utf8)!
        let signature = try privateKey.signature(for: data)
        XCTAssertTrue(privateKey.publicKey.isValidSignature(signature, for: data))

        let tampered = "Tampered message".data(using: .utf8)!
        XCTAssertFalse(privateKey.publicKey.isValidSignature(signature, for: tampered))
    }

    func testCurve25519KeyAgreement() throws {
        let alice = Curve25519.KeyAgreement.PrivateKey()
        let bob = Curve25519.KeyAgreement.PrivateKey()
        let aliceShared = try alice.sharedSecretFromKeyAgreement(with: bob.publicKey)
        let bobShared = try bob.sharedSecretFromKeyAgreement(with: alice.publicKey)

        let aliceKey = aliceShared.hkdfDerivedSymmetricKey(
            using: SHA256.self, salt: Data(), sharedInfo: Data(), outputByteCount: 32)
        let bobKey = bobShared.hkdfDerivedSymmetricKey(
            using: SHA256.self, salt: Data(), sharedInfo: Data(), outputByteCount: 32)

        // Both parties can decrypt each other's messages
        let sealed = try AES.GCM.seal("test".data(using: .utf8)!, using: aliceKey)
        XCTAssertEqual(try AES.GCM.open(sealed, using: bobKey), "test".data(using: .utf8)!)
    }
}
```

**iOS 26 note:** Post-quantum cryptography (ML-KEM, ML-DSA) is available via CryptoKit starting iOS 26. Gate these tests with `@available(iOS 26, *)` and use the same round-trip pattern. Software-based PQC works on simulator (see `cryptokit-public-key.md`).

---

## Secure Enclave Test Strategy  -  Protocol Fallback

`SecureEnclave.P256.Signing.PrivateKey` and `P256.Signing.PrivateKey` are distinct types, so a single function should not promise to return one concrete type for both SE and software paths. Use a protocol-based abstraction:

### SigningKeyProvider Protocol with SE/Software Implementations

```swift
import CryptoKit

protocol SigningKeyProvider {
    func sign(_ data: Data) throws -> Data
    func publicKeyData() -> Data
}

final class SecureEnclaveSigningKey: SigningKeyProvider {
    private let key: SecureEnclave.P256.Signing.PrivateKey

    init() throws {
        guard SecureEnclave.isAvailable else {
            throw KeychainError.unhandledError(status: errSecUnimplemented)
        }
        self.key = try SecureEnclave.P256.Signing.PrivateKey()
    }

    func sign(_ data: Data) throws -> Data {
        try key.signature(for: data).derRepresentation
    }

    func publicKeyData() -> Data { key.publicKey.derRepresentation }
}

final class SoftwareSigningKey: SigningKeyProvider {
    private let key = P256.Signing.PrivateKey()

    func sign(_ data: Data) throws -> Data {
        try key.signature(for: data).derRepresentation
    }

    func publicKeyData() -> Data { key.publicKey.derRepresentation }
}

struct SigningKeyFactory {
    static func make() -> SigningKeyProvider {
        if SecureEnclave.isAvailable,
           let seKey = try? SecureEnclaveSigningKey() {
            return seKey
        }
        return SoftwareSigningKey()
    }
}
```

### Testing Secure Enclave Code

```swift
// ❌ INCORRECT: Crashes on simulator and CI
func testSecureEnclaveSigning_BROKEN() throws {
    let key = try SecureEnclave.P256.Signing.PrivateKey() // throws on simulator
    let sig = try key.signature(for: "data".data(using: .utf8)!)
    XCTAssertTrue(key.publicKey.isValidSignature(sig, for: "data".data(using: .utf8)!))
}

// ✅ CORRECT: Skip gracefully when SE unavailable
func testSecureEnclaveSigning_withGuard() throws {
    try XCTSkipUnless(SecureEnclave.isAvailable,
                      "Secure Enclave not available  -  skipping on simulator")
    let key = try SecureEnclave.P256.Signing.PrivateKey()
    let data = "authenticated payload".data(using: .utf8)!
    let sig = try key.signature(for: data)
    XCTAssertTrue(key.publicKey.isValidSignature(sig, for: data))
}

// ✅ CORRECT: Protocol-based test runs everywhere
func testSigningWithFallback() throws {
    let signer = SigningKeyFactory.make()
    let data = "payload".data(using: .utf8)!
    let sigBytes = try signer.sign(data)
    XCTAssertFalse(sigBytes.isEmpty)

    let publicKey = try P256.Signing.PublicKey(derRepresentation: signer.publicKeyData())
    let signature = try P256.Signing.ECDSASignature(derRepresentation: sigBytes)
    XCTAssertTrue(publicKey.isValidSignature(signature, for: data))
}
```

---

## Biometric Flow Testing  -  LAContext Mocking

Wrap `LAContext` behind a protocol for full control over biometric outcomes in tests. Alternatively, subclass `LAContext` directly (simpler but tighter coupling).

### Protocol-Based Approach (Preferred)

```swift
import LocalAuthentication

protocol BiometricAuthContext {
    func canEvaluatePolicy(_ policy: LAPolicy, error: NSErrorPointer) -> Bool
    func evaluatePolicy(_ policy: LAPolicy, localizedReason: String,
                        reply: @escaping (Bool, Error?) -> Void)
}

extension LAContext: BiometricAuthContext {}

final class BiometricAuthManager {
    private let context: BiometricAuthContext

    init(context: BiometricAuthContext = LAContext()) {
        self.context = context
    }

    var isBiometricsAvailable: Bool {
        context.canEvaluatePolicy(.deviceOwnerAuthenticationWithBiometrics, error: nil)
    }

    func authenticate(reason: String,
                      completion: @escaping (Result<Void, Error>) -> Void) {
        guard isBiometricsAvailable else {
            completion(.failure(LAError(.biometryNotAvailable)))
            return
        }
        context.evaluatePolicy(.deviceOwnerAuthenticationWithBiometrics,
                               localizedReason: reason) { success, error in
            completion(success ? .success(()) : .failure(error ?? LAError(.authenticationFailed)))
        }
    }
}

final class MockBiometricContext: BiometricAuthContext {
    var canEvaluateResult = true
    var evaluateResult = true
    var evaluateError: Error?
    var evaluateCalled = false

    func canEvaluatePolicy(_ policy: LAPolicy, error: NSErrorPointer) -> Bool {
        canEvaluateResult
    }

    func evaluatePolicy(_ policy: LAPolicy, localizedReason: String,
                        reply: @escaping (Bool, Error?) -> Void) {
        evaluateCalled = true
        reply(evaluateResult, evaluateError)
    }
}
```

### Biometric Scenarios to Cover

| Scenario     | canEvaluate | evaluatePolicy | Error                  | Expected App Behavior     |
| ------------ | ----------- | -------------- | ---------------------- | ------------------------- |
| Success      | true        | true           | nil                    | Proceed                   |
| User cancel  | true        | false          | `.userCancel`          | Retry or abort gracefully |
| Lockout      | true        | false          | `.biometryLockout`     | Fallback to passcode      |
| Not enrolled | false       | n/a            | `.biometryNotEnrolled` | Show enrollment guidance  |

```swift
func testBiometricAuthSuccess() {
    let mock = MockBiometricContext()
    mock.canEvaluateResult = true
    mock.evaluateResult = true
    let manager = BiometricAuthManager(context: mock)

    let exp = expectation(description: "auth")
    manager.authenticate(reason: "Test") { result in
        if case .failure = result { XCTFail("Expected success") }
        exp.fulfill()
    }
    waitForExpectations(timeout: 1)
    XCTAssertTrue(mock.evaluateCalled)
}

func testBiometricAuthUnavailable() {
    let mock = MockBiometricContext()
    mock.canEvaluateResult = false
    let manager = BiometricAuthManager(context: mock)

    let exp = expectation(description: "unavailable")
    manager.authenticate(reason: "Test") { result in
        if case .success = result { XCTFail("Expected failure") }
        exp.fulfill()
    }
    waitForExpectations(timeout: 1)
    XCTAssertFalse(mock.evaluateCalled)  // Should not attempt auth
}
```

---

## CI/CD Pipeline Configuration

Running keychain tests in CI is the most error-prone part. The `-25308` (`errSecInteractionNotAllowed`) error is the most common CI failure  -  keychain locked or requires GUI interaction in a headless environment.

### GitHub Actions

```yaml
name: iOS CI
on: [push, pull_request]
jobs:
  test:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v5
      - name: Create temporary keychain
        env:
          KEYCHAIN_PASSWORD: ${{ secrets.KEYCHAIN_PASSWORD }}
        run: |
          KEYCHAIN_PATH=$RUNNER_TEMP/app-signing.keychain-db
          security create-keychain -p "$KEYCHAIN_PASSWORD" $KEYCHAIN_PATH
          security set-keychain-settings -lut 21600 $KEYCHAIN_PATH
          security unlock-keychain -p "$KEYCHAIN_PASSWORD" $KEYCHAIN_PATH
          security list-keychain -d user -s $KEYCHAIN_PATH
          # Import cert + CRITICAL partition list step
          echo -n "$BUILD_CERTIFICATE_BASE64" | base64 --decode -o $RUNNER_TEMP/cert.p12
          security import $RUNNER_TEMP/cert.p12 -P "$P12_PASSWORD" \
            -A -t cert -f pkcs12 -k $KEYCHAIN_PATH
          security set-key-partition-list -S apple-tool:,apple: \
            -k "$KEYCHAIN_PASSWORD" $KEYCHAIN_PATH
      - name: Run simulator-safe tests
        run: |
          xcodebuild test -scheme MyApp \
            -destination 'platform=iOS Simulator,name=iPhone 16' \
            -testPlan CITests
      - name: Cleanup
        if: always()
        run: security delete-keychain $RUNNER_TEMP/app-signing.keychain-db
```

**`security set-key-partition-list` must be called after importing certificates**  -  this is the step most people miss. Without it, `codesign` hangs indefinitely waiting for a GUI prompt. The `-A` flag on import grants access to all applications, necessary in CI.

**Xcode Cloud:** Uses ephemeral environments  -  no manual `security create-keychain`. Apple manages signing automatically. Ensure Keychain Sharing capability is enabled. The `-25308` error is common when SPM tries to save credentials.

**Fastlane:** `setup_ci` creates a temporary `fastlane_tmp_keychain` and sets it as default. On self-hosted runners, this can interfere with the host machine's keychain.

```ruby
lane :ci_test do
  setup_ci(timeout: 3600)
  sync_code_signing(type: "development", readonly: is_ci)
  run_tests(scheme: "MyApp", testplan: "CITests", device: "iPhone 16")
end
```

### Common CI Error Reference

| Error                         | OSStatus | Cause                                 | Fix                                             |
| ----------------------------- | -------- | ------------------------------------- | ----------------------------------------------- |
| `errSecInteractionNotAllowed` | -25308   | Keychain locked / needs GUI           | Unlock keychain + `set-key-partition-list`      |
| `errSecMissingEntitlement`    | -34018   | No keychain-access-groups entitlement | Add entitlements to test host app               |
| `errSecItemNotFound`          | -25300   | No test host or missing entitlement   | Use test host app with keychain capability      |
| `errSecInternalComponent`     | -67585   | Partition list not set after import   | Call `set-key-partition-list` after cert import |
| Default keychain not found    | -25307   | No default keychain on CI runner      | Create and set default keychain                 |

### Test Host App Requirement

Since Xcode 9, test bundles on iOS simulator require a host app to access the keychain. Without one, `SecItemAdd` returns `-25300` or `-34018`. Create a minimal iOS app target, enable the Keychain Sharing capability, and set the test target's **Test Host** and **Bundle Loader** build settings to point at it.

---

## Xcode Test Plans  -  Separating Simulator from Device

Create two test plans for CI/device split:

- **CITests.xctestplan**: Only tests using `MockKeychainService` and simulator-safe CryptoKit. Skips integration, biometric, and SE tests.
- **DeviceTests.xctestplan**: Real keychain integration, Secure Enclave, and biometric hardware tests. Requires physical device.

```bash
# CI: simulator-safe tests on every push
xcodebuild test -scheme MyApp \
  -destination 'platform=iOS Simulator,name=iPhone 16' \
  -testPlan CITests

# Nightly: device farm runs everything
xcodebuild test -scheme MyApp \
  -destination 'platform=iOS,id=DEVICE_UDID' \
  -testPlan DeviceTests
```

---

## Swift Testing Framework Patterns

Swift Testing (WWDC24) introduces tags, traits, and parameterized tests that map well to security test organization:

```swift
import Testing
@testable import MyApp

extension Tag {
    @Tag static var keychain: Self
    @Tag static var deviceOnly: Self
    @Tag static var ciSafe: Self
}

@Suite(.serialized, .tags(.keychain))
struct KeychainTests {

    @Test("Save and retrieve round-trip", .tags(.ciSafe))
    func saveAndRetrieve() throws {
        let mock = MockKeychainService()
        let manager = AuthenticationManager(keychain: mock)
        try manager.storeToken("test-token")
        let result = try #require(try manager.retrieveToken())
        #expect(result == "test-token")
    }

    @Test("Device-only: real keychain integration",
          .enabled(if: ProcessInfo.processInfo.environment["CI"] == nil),
          .tags(.deviceOnly))
    func realKeychainIntegration() throws {
        let keychain = KeychainService(service: "com.test.swift-testing")
        try keychain.deleteAll()
        defer { try? keychain.deleteAll() }
        try keychain.save("token".data(using: .utf8)!, forKey: "key")
        let data = try #require(try keychain.read(forKey: "key"))
        #expect(String(data: data, encoding: .utf8) == "token")
    }

    @Test("Parameterized error paths",
          arguments: [
              KeychainError.duplicateItem,
              KeychainError.itemNotFound,
              KeychainError.authFailed,
              KeychainError.interactionNotAllowed
          ])
    func errorPathHandling(expectedError: KeychainError) {
        let mock = MockKeychainService()
        mock.errorToThrow = expectedError
        #expect(throws: KeychainError.self) {
            try mock.read(forKey: "any-key")
        }
    }
}
```

The `.serialized` trait ensures keychain tests modifying shared state run sequentially. Tags integrate with test plans for filtering  -  `.ciSafe` tests run in CI, `.deviceOnly` tests run on device farms.

---

## Advanced Patterns

### Migration Testing: UserDefaults to Keychain

Migration code is security-critical  -  silent failure leaves credentials in UserDefaults (see `migration-legacy-stores.md`). The class under test accepts injected dependencies for both stores:

```swift
final class StorageMigrationManager {
    private let defaults: UserDefaults
    private let keychain: KeychainServiceProtocol

    init(defaults: UserDefaults = .standard,
         keychain: KeychainServiceProtocol) {
        self.defaults = defaults
        self.keychain = keychain
    }

    func migrateIfNeeded() throws {
        let version = defaults.integer(forKey: "migration_version")
        if version < 1 {
            if let token = defaults.string(forKey: "auth_token"),
               let data = token.data(using: .utf8) {
                try keychain.save(data, forKey: "auth_token")
                defaults.removeObject(forKey: "auth_token")
            }
        }
        defaults.set(1, forKey: "migration_version")
    }
}
```

Test with isolated `UserDefaults(suiteName:)` and mock keychain:

```swift
func testMigrationMovesTokenToKeychain() throws {
    let defaults = UserDefaults(suiteName: "migration-test")!
    defaults.removePersistentDomain(forName: "migration-test")
    defaults.set("my-secret", forKey: "auth_token")
    defaults.set(0, forKey: "migration_version")

    let mock = MockKeychainService()
    let migrator = StorageMigrationManager(defaults: defaults, keychain: mock)
    try migrator.migrateIfNeeded()

    // Token moved to keychain, removed from UserDefaults
    XCTAssertEqual(String(data: mock.storage["auth_token"]!, encoding: .utf8), "my-secret")
    XCTAssertNil(defaults.string(forKey: "auth_token"))
}
```

### Performance Testing

```swift
func testKeychainWritePerformance() {
    let keychain = KeychainService(service: "com.test.perf")
    let options = XCTMeasureOptions()
    options.iterationCount = 20

    measure(metrics: [XCTClockMetric(), XCTCPUMetric()], options: options) {
        let data = UUID().uuidString.data(using: .utf8)!
        try? keychain.save(data, forKey: "perf-key")
        try? keychain.delete(forKey: "perf-key")
    }
}
```

### Mutation Testing

Mutation testing introduces deliberate bugs (flipping `==` to `!=`, removing `SecItemDelete` calls, swapping `&&` to `||`) and checks whether your tests catch them. A project can have 81% code coverage but only 16% mutation score  -  tests execute security code without validating it does the right thing.

**Muter** (`brew install muter-mutation-testing/muter/muter`) is the primary Swift mutation testing tool. Its `RelationalOperatorReplacement` operator catches authentication bypasses; `RemoveSideEffects` catches missing `SecItemDelete` calls in logout flows. For security code, target mutation score above **80%**.

### OWASP MASTG Keychain Validation

MASTG-TEST-0052 requires that sensitive data use the Keychain, not `NSUserDefaults` or `.plist` files. OWASP also documents that keychain data persists after app uninstallation  -  the app sandbox is wiped but keychain items remain. Standard mitigation is a fresh-install detector (see `common-anti-patterns.md`):

```swift
static func handleFreshInstall(keychain: KeychainServiceProtocol) {
    let hasLaunched = UserDefaults.standard.bool(forKey: "has_launched")
    if !hasLaunched {
        try? keychain.deleteAll()
        UserDefaults.standard.set(true, forKey: "has_launched")
    }
}
```

---

## Conclusion

Protocol-abstraction is non-negotiable for testable keychain code. Every `SecItem` call should be behind `KeychainServiceProtocol` so that 95%+ of your test suite runs against `MockKeychainService` with zero entitlement requirements and zero CI flakiness. Reserve real-keychain integration tests for a dedicated test plan on physical devices.

Three insights most guides miss: (1) the simulator silently returns biometric-protected items without prompting  -  tests appear to validate biometric gates but test nothing; (2) TN3137's distinction between file-based and data protection keychains means `security create-keychain` in CI creates the wrong keychain type; (3) mutation testing reveals that even high-coverage suites fail to catch inverted conditionals and removed side effects  -  the exact mutations that create real vulnerabilities.

---

## Summary Checklist

1. **Protocol abstraction**  -  All keychain access goes through `KeychainServiceProtocol`; no direct `SecItem*` calls in business logic
2. **Mock with injectable errors**  -  `MockKeychainService` supports `errorToThrow` for testing `errSecDuplicateItem`, `errSecAuthFailed`, `errSecInteractionNotAllowed`, and `errSecItemNotFound` paths
3. **setUp/tearDown cleanup**  -  Every integration test using real keychain has both pre-test and post-test cleanup with a test-specific `kSecAttrService`
4. **Secure Enclave guard**  -  All SE tests use `try XCTSkipUnless(SecureEnclave.isAvailable, ...)` or protocol-based fallback; never call `SecureEnclave.P256.*` unconditionally
5. **Biometric mock**  -  `LAContext` wrapped behind protocol or subclass mock; tests cover success, user cancel, lockout, and not-enrolled scenarios
6. **Simulator/device split**  -  Two Xcode test plans: `CITests` (mock-based, simulator-safe) and `DeviceTests` (real keychain, SE, biometrics on physical device)
7. **CI keychain setup**  -  GitHub Actions calls `security set-key-partition-list` after cert import; test target has host app with Keychain Sharing capability enabled
8. **CryptoKit round-trips**  -  Encrypt→decrypt and sign→verify tests for AES-GCM, ChaChaPoly, P256, Curve25519; wrong-key failure tests included
9. **Error path coverage**  -  Every `OSStatus` code the app can encounter has a corresponding test with injected mock failure
10. **Migration testing**  -  UserDefaults→Keychain migration tested with isolated `UserDefaults(suiteName:)` and mock keychain; verifies source cleared after migration
11. **Mutation testing baseline**  -  Muter mutation score ≥80% for security-critical code paths; `RelationalOperatorReplacement` and `RemoveSideEffects` operators enabled