# Core ML in Swift: extended patterns Companion to `../SKILL.md`. Baseline iOS 26 and Swift 6.3; lower floors are called out per API, and most of this works back to iOS 14 when the newer calls are left out. ## Contents - [An actor that owns models](#an-actor-that-owns-models) - [Generated classes](#generated-classes) - [A hand-written feature provider](#a-hand-written-feature-provider) - [Predictions in async code](#predictions-in-async-code) - [Batch inference](#batch-inference) - [Stateful sessions with MLState (iOS 18+)](#stateful-sessions-with-mlstate-ios-18) - [Pixel buffers](#pixel-buffers) - [MLMultiArray](#mlmultiarray) - [MLTensor (iOS 18+)](#mltensor-ios-18) - [Vision and Core ML together](#vision-and-core-ml-together) - [NaturalLanguage models](#naturallanguage-models) - [MLComputePlan in detail (iOS 17.4+)](#mlcomputeplan-in-detail-ios-174) - [Lifecycle: warm up and release](#lifecycle-warm-up-and-release) - [Errors](#errors) - [Tests with Swift Testing](#tests-with-swift-testing) - [Memory practices](#memory-practices) ## An actor that owns models An actor serializes loading, guarantees one instance per model, and keeps compiled models on disk between launches. ```swift import CoreML enum ModelStoreError: Error { case missing(String) case predictionFailed(String) } actor ModelStore { static let shared = ModelStore() private var loaded: [String: MLModel] = [:] private let compiledFolder: URL init() { compiledFolder = URL.applicationSupportDirectory.appending(path: "CompiledModels", directoryHint: .isDirectory) try? FileManager.default.createDirectory(at: compiledFolder, withIntermediateDirectories: true) } func prepare(_ name: String, settings: sending MLModelConfiguration = MLModelConfiguration()) async throws { _ = try await model(name, settings: settings) } func predict(_ name: String, from input: sending MLFeatureProvider) async throws -> sending MLFeatureProvider { nonisolated(unsafe) let engine = try await model(name) return try await engine.prediction(from: input) } func release(_ name: String) { loaded[name] = nil } func releaseEverything() { loaded.removeAll() } private func model(_ name: String, settings: sending MLModelConfiguration = MLModelConfiguration()) async throws -> MLModel { if let ready = loaded[name] { return ready } let location = try await compiledLocation(for: name) let fresh = try await MLModel.load(contentsOf: location, configuration: settings) loaded[name] = fresh return fresh } private func compiledLocation(for key: String) async throws -> URL { let files = FileManager.default let cached = compiledFolder.appending(path: key + ".mlmodelc") if files.fileExists(atPath: cached.path()) { return cached } if let shipped = Bundle.main.url(forResource: key, withExtension: "mlmodelc") { return shipped } guard let package = Bundle.main.url(forResource: key, withExtension: "mlpackage") else { throw ModelStoreError.missing(key) } let temporary = try await MLModel.compileModel(at: package) try? files.removeItem(at: cached) try files.moveItem(at: temporary, to: cached) return cached } } ``` Lookup order: a previously compiled copy in Application Support, then a `.mlmodelc` shipped in the bundle, then compiling a bundled `.mlpackage` and moving the result into the cache. Anything else is `missing`. `MLModel` is not `Sendable`, so the store never hands one out: returning it to a main-actor caller fails to compile under Swift 6. Callers warm a model with `prepare` and run it with `predict`. Inputs, outputs and the configuration are `sending`, so the caller passes a freshly built value and owns the result. The `nonisolated(unsafe)` copy in `predict` lets the cached model reach the async prediction call, which may run concurrently on one model (see "Predictions in async code" below). For downloaded models, put the version in the cache file name so a new version compiles once. ### Driving it from SwiftUI Most examples below feed one image feature, so they share a tiny helper: ```swift import CoreML func photoInput(_ frame: CVPixelBuffer) throws -> MLFeatureProvider { try MLDictionaryFeatureProvider(dictionary: ["photo": MLFeatureValue(pixelBuffer: frame)]) } ``` ```swift import Observation import CoreML @MainActor @Observable final class DishRecognizer { var dish = "" var certainty = 0.0 var working = false var failure: String? func recognize(_ picture: CGImage) async { working = true defer { working = false } do { let frame = try makeFrame(from: picture, width: 224, height: 224) let answer = try await ModelStore.shared.predict("DishClassifier", from: photoInput(frame)) dish = answer.featureValue(for: "classLabel")?.stringValue ?? "" certainty = answer.featureValue(for: "classLabelProbs")?.dictionaryValue[dish]?.doubleValue ?? 0 } catch { failure = error.localizedDescription } } } ``` `makeFrame(from:width:height:)` is defined in the pixel buffer section below. The frame is created inside `recognize`, which is what allows its feature provider to be sent to the store. The dictionary lookup reads the probability of the winning label out of the `classLabelProbs` feature. ## Generated classes For a model file named `SceneTagger`, Xcode generates `SceneTagger`, `SceneTaggerInput` and `SceneTaggerOutput`. ```swift import CoreML import Vision func bestDish(_ frame: CVPixelBuffer, using classifier: SceneTagger) throws -> (name: String, probability: Double) { let verdict = try classifier.prediction(input: SceneTaggerInput(image: frame)) return (verdict.classLabel, verdict.classLabelProbs[verdict.classLabel] ?? 0) } func legacyVisionModel(for classifier: SceneTagger) throws -> VNCoreMLModel { try VNCoreMLModel(for: classifier.model) } ``` The generated class exposes the underlying `MLModel` as `.model`, which is what `VNCoreMLModel(for:)` and `CoreMLModelContainer(model:)` expect. ## A hand-written feature provider Implement `MLFeatureProvider` when the input is assembled in code: ```swift import CoreML final class GlareCheckInput: MLFeatureProvider { let frame: CVPixelBuffer let sensitivity: Double init(frame: CVPixelBuffer, sensitivity: Double = 0.5) { self.frame = frame self.sensitivity = sensitivity } var featureNames: Set { ["photo", "sensitivity"] } func featureValue(for requested: String) -> MLFeatureValue? { if requested == "sensitivity" { return MLFeatureValue(double: sensitivity) } return requested == "photo" ? MLFeatureValue(pixelBuffer: frame) : nil } } func hasGlare(_ frame: CVPixelBuffer, model: MLModel) async throws -> Bool { let verdict = try await model.prediction(from: GlareCheckInput(frame: frame)) return verdict.featureValue(for: "glare")?.int64Value == 1 } ``` Unknown feature names return `nil`. ## Predictions in async code Keep loading, preprocessing and coordination off the main actor. - iOS 17 and later: `prediction(from:options:) async throws` is the natural call in async code and may run concurrently on one model. The compiler picks it inside `async` functions, so write `try await`. - The synchronous `prediction(from:)` is for synchronous contexts and for deployment targets below iOS 17. ```swift import CoreML actor InferenceService { nonisolated(unsafe) private let engine: MLModel init(engine: MLModel) { self.engine = engine } func run(_ features: sending MLFeatureProvider) async throws -> sending MLFeatureProvider { try await engine.prediction(from: features) } func runBlocking(_ features: MLFeatureProvider) throws -> MLFeatureProvider { try engine.prediction(from: features) } } ``` `runBlocking` is a synchronous actor method, so the synchronous overload applies there. Calling the async overload hands the non-`Sendable` model and input to a nonisolated method, which Swift 6 rejects for plain actor state ("sending 'self.engine' risks causing data races"). The model is therefore stored `nonisolated(unsafe)`, relying on the async API being safe to call concurrently, and `run` takes and returns `sending` values. ### Streaming frames ```swift import CoreML func liveLabels(from frames: AsyncStream, model: sending MLModel) -> AsyncThrowingStream { let (labels, sink) = AsyncThrowingStream.makeStream() let worker = Task { do { for await sample in frames { let verdict = try await model.prediction(from: photoInput(sample.buffer)) sink.yield(verdict.featureValue(for: "classLabel")?.stringValue ?? "") } sink.finish() } catch { sink.finish(throwing: error) } } sink.onTermination = { _ in worker.cancel() } return labels } struct CameraSample: @unchecked Sendable { let buffer: CVPixelBuffer } ``` `CVPixelBuffer` is not `Sendable`; the wrapper is acceptable only when nothing touches the buffer after it is handed over. The model parameter is `sending` for the same reason: the worker task takes it over, and the caller must not use that instance again. The stream comes from `makeStream()` rather than the closure initializer, because a `Task` started inside that closure cannot capture the model under Swift 6. ## Batch inference ```swift import CoreML func labelBatch(_ frames: [CVPixelBuffer], model: MLModel) throws -> [(String, Double)] { let rows = try frames.map(photoInput) let results = try model.predictions(fromBatch: MLArrayBatchProvider(array: rows)) var pairs: [(String, Double)] = [] for index in 0.. MLBatchProvider { let options = MLPredictionOptions() return try model.predictions(from: batch, options: options) } ``` The two correct spellings: `predictions(fromBatch:)` without options and `predictions(from:options:)` with them. `predictions(from:)` alone is not the options-free batch call. ## Stateful sessions with MLState (iOS 18+) Text generation, audio classification and time-series models carry context from one call to the next. `MLState` holds that context. ```swift import CoreML enum ListeningError: Error { case notListening } actor ChirpDetector { private let engine: MLModel private var memory: MLState? init(engine: MLModel) { self.engine = engine } func startListening() { memory = engine.makeState() } func detect(_ window: MLMultiArray) throws -> String? { guard let memory else { throw ListeningError.notListening } let input = try MLDictionaryFeatureProvider(dictionary: ["samples": MLFeatureValue(multiArray: window)]) let heard = try engine.prediction(from: input, using: memory) return heard.featureValue(for: "species")?.stringValue } func stopListening() { memory = nil } } ``` Rules: - One prediction at a time per state. Being `Sendable` means a state may be handed to another task or actor; overlapping calls on it are still unsafe, and its buffers stay untouched until the current call returns. - In plain serial code, such as the synchronous actor method above, call `prediction(from:using:)` without `await`. In async code, or when you need `MLPredictionOptions`, use `prediction(from:using:options:) async` (iOS 18); still one call in flight per state. - Concurrent sequences, such as several audio channels, get one `makeState()` each. - There is no reset call. Throw the state away and make a new one. - A state holds buffers specific to its model; drop it once the session is over. ## Pixel buffers A throwing version of the conversion helper: ```swift import CoreGraphics import CoreVideo enum FrameError: Error { case bufferAllocation case drawingContext } func makeFrame(from picture: CGImage, width: Int, height: Int) throws -> CVPixelBuffer { let attributes = [ kCVPixelBufferCGImageCompatibilityKey: true, kCVPixelBufferCGBitmapContextCompatibilityKey: true ] as CFDictionary var created: CVPixelBuffer? let format = kCVPixelFormatType_32ARGB guard CVPixelBufferCreate(nil, width, height, format, attributes, &created) == kCVReturnSuccess, let frame = created else { throw FrameError.bufferAllocation } CVPixelBufferLockBaseAddress(frame, []) defer { CVPixelBufferUnlockBaseAddress(frame, []) } let base = CVPixelBufferGetBaseAddress(frame) let stride = CVPixelBufferGetBytesPerRow(frame) let layout = CGImageAlphaInfo.noneSkipFirst.rawValue let rgb = CGColorSpaceCreateDeviceRGB() guard let canvas = CGContext(data: base, width: width, height: height, bitsPerComponent: 8, bytesPerRow: stride, space: rgb, bitmapInfo: layout) else { throw FrameError.drawingContext } canvas.draw(picture, in: CGRect(origin: .zero, size: CGSize(width: width, height: height))) return frame } ``` From a `CIImage`, create a `kCVPixelFormatType_32BGRA` buffer and render into it with `CIContext.render(_:to:)`: ```swift import CoreImage import CoreVideo func makeFrame(from image: CIImage, renderer: CIContext) -> CVPixelBuffer? { var created: CVPixelBuffer? CVPixelBufferCreate(kCFAllocatorDefault, Int(image.extent.width), Int(image.extent.height), kCVPixelFormatType_32BGRA, nil, &created) guard let frame = created else { return nil } renderer.render(image, to: frame) return frame } ``` Center-crop before scaling when the model expects a square and the photo is not: `image.cropped(to:)` with a square rect centered on `image.extent`, then scale to the model's side length. Per-channel normalization is shown in the tensor section. ## MLMultiArray ```swift import CoreML func planarImageTensor() throws -> MLMultiArray { let planes = try MLMultiArray(shape: [1, 3, 256, 256], dataType: .float32) for offset in 0.. MLFeatureValue { let array = try MLMultiArray(values) return MLFeatureValue(multiArray: array) } func pixelValue(_ planes: MLMultiArray, channel: Int, row: Int, column: Int) -> Float { planes[[0, channel, row, column] as [NSNumber]].floatValue } func floats(from array: MLMultiArray) -> [Float] { MLShapedArray(array).scalars } ``` Linear subscripts (`planes[offset]`) walk the whole buffer; index arrays address one element. Converting through `MLShapedArray` avoids raw pointer work; if you do use `dataPointer.assumingMemoryBound(to: Float.self)` with `UnsafeBufferPointer`, keep the array alive for the whole access. Tokenized text arrives as an `[Int32]` of token ids shaped `[1, sequence]`; audio features as `[1, frames, bins]`. Both follow the same create, fill, wrap-in-`MLFeatureValue(multiArray:)` pattern. ## MLTensor (iOS 18+) ```swift import CoreML func tensorTour() async -> (MLShapedArray, MLMultiArray) { let readings = MLTensor([1.5, 3.0, 4.5, 6.0, 7.5, 9.0] as [Float]) let blank = MLTensor(zeros: [2, 3], scalarType: Float.self) let unit = MLTensor(ones: [2, 3], scalarType: Float.self) let square = readings.reshaped(to: [2, 3]) let row = readings.expandingShape(at: 0) let summed = square + unit + blank let average = readings.mean() let winner = readings.argmax() let odds = row.softmax(alongAxis: -1) let joined = MLTensor(concatenating: [square, summed], alongAxis: 0) _ = (average, winner, joined) let materialized = await odds.shapedArray(of: Float.self) return (materialized, MLMultiArray(materialized)) } func imageNetNormalized(_ pixels: MLTensor) -> MLTensor { let mean = MLTensor([0.485, 0.456, 0.406] as [Float]).reshaped(to: [1, 3, 1, 1]) let spread = MLTensor([0.229, 0.224, 0.225] as [Float]).reshaped(to: [1, 3, 1, 1]) return (pixels - mean) / spread } func tensorFromArray(_ array: MLMultiArray) -> MLTensor { MLTensor(MLShapedArray(array)) } ``` `expandingShape(at: 0)` turns shape `[6]` into `[1, 6]`. Values only exist after `await shapedArray(of:)`. Keep to operations Apple documents: no `MLTensor(multiArray)`, no `std()` or `standardDeviation()`, no peeking into the lazy storage and no blocking read, unless Apple's documentation names the exact API and its availability. A standard deviation can be built from documented parts: ```swift import CoreML func spread(of values: MLTensor) -> MLTensor { let centered = values - values.mean() return (centered * centered).mean().squareRoot() } ``` ## Vision and Core ML together ### Swift-native Vision (iOS 18+) ```swift import CoreML import Observation import Vision @Observable @MainActor final class ShelfDetector { struct Finding: Identifiable { let id = UUID() let name: String let score: Float let box: NormalizedRect } var findings: [Finding] = [] private let detector: MLModel init(detector: MLModel) { self.detector = detector } func scan(_ photo: CGImage) async { do { let request = CoreMLRequest(model: try CoreMLModelContainer(model: detector)) let raw = try await request.perform(on: photo) findings = raw.compactMap { item in guard let object = item as? RecognizedObjectObservation, let best = object.labels.first else { return nil } return Finding(name: best.identifier, score: best.confidence, box: object.boundingBox) } } catch { findings = [] } } } func topClasses(in photo: CGImage, model: MLModel) async throws -> [(String, Float)] { let request = CoreMLRequest(model: try CoreMLModelContainer(model: model)) return try await request.perform(on: photo) .compactMap { $0 as? ClassificationObservation } .map { ($0.identifier, $0.confidence) } } ``` With a generated class, pass `generated.model` into `CoreMLModelContainer`. ### Legacy Vision (before iOS 18) ```swift import CoreML import Vision actor LegacyShelfDetector { private let visionModel: VNCoreMLModel init(model: MLModel) throws { visionModel = try VNCoreMLModel(for: model) } func objects(in photo: CGImage) throws -> [(label: String, confidence: Float, box: CGRect)] { let request = VNCoreMLRequest(model: visionModel) request.imageCropAndScaleOption = .scaleFill try VNImageRequestHandler(cgImage: photo).perform([request]) let found = request.results as? [VNRecognizedObjectObservation] ?? [] return found.compactMap { object in object.labels.first.map { ($0.identifier, $0.confidence, object.boundingBox) } } } } func legacyTopFive(in photo: CGImage, model: MLModel) throws -> [VNClassificationObservation] { let request = VNCoreMLRequest(model: try VNCoreMLModel(for: model)) request.imageCropAndScaleOption = .centerCrop try VNImageRequestHandler(cgImage: photo).perform([request]) let ranked = request.results as? [VNClassificationObservation] ?? [] return Array(ranked.prefix(5)) } ``` The legacy handler blocks while it works, so it runs inside an actor (or a `Task.detached` whose captures are all `Sendable`) rather than on the main actor. Vision observation classes are not `Sendable`; convert them to plain values before they leave the actor. `.scaleFill` stretches the whole image into the model's input; `.centerCrop` keeps the middle square, which suits most classifiers. ## NaturalLanguage models Text classifiers trained with Create ML load through `NLModel`: ```swift import CoreML import NaturalLanguage func ticketCategory(for message: String, modelURL: URL) throws -> (String, Double)? { let classifier = try NLModel(contentsOf: modelURL) guard let category = classifier.predictedLabel(for: message) else { return nil } let score = classifier.predictedLabelHypotheses(for: message, maximumCount: 1)[category] ?? 0 return (category, score) } ``` ## MLComputePlan in detail (iOS 17.4+) A logging pass over every operation of an ML Program, run with the compute units you intend to ship: ```swift import CoreML import OSLog func auditDispatch(of packageURL: URL, units: MLComputeUnits) async throws { let log = Logger(subsystem: "app.inference", category: "compute-plan") let settings = MLModelConfiguration() settings.computeUnits = units let plan = try await MLComputePlan.load(contentsOf: packageURL, configuration: settings) guard case .program(let program) = plan.modelStructure else { log.info("Per-operation dispatch is reported only for ML Program models") return } guard let entry = program.functions["main"] else { log.info("No main function in the program") return } for op in entry.block.operations { let device = plan.deviceUsage(for: op).map { "\($0.preferred)" } ?? "unknown" let weight = plan.estimatedCost(of: op)?.weight ?? 0 log.debug("\(op.operatorName, privacy: .public) on \(device, privacy: .public) weight \(weight)") } } ``` Reading the result: - **Neural Engine**: the fastest and most efficient place; nothing to do. - **GPU**: runs through Metal; good for large matrix work. - **CPU**: the fallback for operations the accelerators cannot run. A handful is normal; many, especially heavy ones, deserve investigation. If critical operations land on the CPU: try `.cpuAndGPU`, look up which operations the Neural Engine does not support, or convert again with a different minimum deployment target. ## Lifecycle: warm up and release ```swift import SwiftUI import Observation import CoreML @Observable @MainActor final class InferenceLifecycle { private(set) var ready = false func warmUp() async { ready = (try? await ModelStore.shared.prepare("DishClassifier")) != nil } func enteredBackground() async { await ModelStore.shared.releaseEverything() ready = false } } struct PantryApp: App { @State private var lifecycle = InferenceLifecycle() @Environment(\.scenePhase) private var phase var body: some Scene { WindowGroup { Text(lifecycle.ready ? "Ready" : "Loading") .environment(lifecycle) .task { await lifecycle.warmUp() } } .onChange(of: phase) { _, newPhase in Task { switch newPhase { case .background: await lifecycle.enteredBackground() case .active: await lifecycle.warmUp() default: break } } } } } ``` In a real app this type carries `@main`; it is omitted here so the snippet stays self-contained. ## Errors ```swift import CoreML func guardedPrediction(_ features: MLFeatureProvider, resource: String) async -> MLFeatureProvider? { guard let location = Bundle.main.url(forResource: resource, withExtension: "mlmodelc") else { return nil } do { let engine = try await MLModel.load(contentsOf: location) return try await engine.prediction(from: features) } catch { return nil } } ``` | Symptom | Usual cause | Fix | |---|---|---| | Model file not found | Wrong path, or the file is not a member of the target | Check target membership and the resource name | | Compilation fails | Corrupt `.mlpackage` or operations the OS does not support | Export again from coremltools | | Input shape mismatch | Image size or tensor shape differs from the model's description | Match `modelDescription.inputDescriptionsByName` | | Memory exhausted | The device cannot hold the model | Smaller or compressed model, or `.cpuOnly` as a last resort | | Unexpected CPU fallback | Operations unsupported on the chosen device | Stay on `.all` and inspect `MLComputePlan` | MLTensor code should use only operations confirmed in Apple's documentation (see the tensor section); invented helpers compile nowhere. ## Tests with Swift Testing ```swift import CoreML import Testing struct DishClassifierTests { private func model(units: MLComputeUnits) async throws -> MLModel { let url = try #require(Bundle.main.url(forResource: "DishClassifier", withExtension: "mlmodelc")) let settings = MLModelConfiguration() settings.computeUnits = units return try await MLModel.load(contentsOf: url, configuration: settings) } private func blankFrame() throws -> CVPixelBuffer { var created: CVPixelBuffer? CVPixelBufferCreate(kCFAllocatorDefault, 224, 224, kCVPixelFormatType_32ARGB, nil, &created) return try #require(created) } @Test func loadsWithInputs() async throws { let model = try await model(units: .cpuOnly) #expect(!model.modelDescription.inputDescriptionsByName.isEmpty) } @Test func producesProbabilities() async throws { let model = try await model(units: .cpuOnly) let verdict = try await model.prediction(from: photoInput(try blankFrame())) let winner = verdict.featureValue(for: "classLabel")?.stringValue ?? "" let probabilities = verdict.featureValue(for: "classLabelProbs")?.dictionaryValue.values.map { $0.doubleValue } ?? [] #expect(!winner.isEmpty) #expect(probabilities.allSatisfy { (0...1).contains($0) }) } @Test func staysUnderLatencyBudget() async throws { let model = try await model(units: .all) let probe = try photoInput(try blankFrame()) _ = try await model.prediction(from: probe) let clock = ContinuousClock() let rounds = 10 let elapsed = try await clock.measure { for _ in 0..