--- name: coreml description: "Swift-side Core ML: loading .mlmodel, .mlpackage and .mlmodelc, predictions via generated classes and MLFeatureProvider, compute units, async and stateful prediction with MLState, MLTensor, MLMultiArray, MLComputePlan, multi-model pipelines, deployment and profiling. Use when integrating, reviewing, deploying or profiling a Core ML model in an app. Not for Python conversion (apple-on-device-ai) or Vision requests (vision-framework)." metadata: source: multi-agent-pipeline --- # Core ML This skill covers the app side of Core ML: getting a model into memory, feeding it, reading its output, and shipping it. Baseline: iOS 26 and Swift 6.3, with each API's lower floor stated where it matters (much of Core ML goes back to iOS 14 or earlier). Out of scope here: converting or compressing models in Python (quantization, palettization, pruning) and choosing between on-device AI frameworks; those belong to `apple-on-device-ai`. Text recognition, barcodes and document scanning with Vision belong to `vision-framework`. Longer code (an actor that owns models, SwiftUI wiring, custom feature providers, stateful sessions, pixel buffers, tensors, Vision pipelines, NaturalLanguage, compute plans, lifecycle, errors, tests) is in `references/coreml-swift-integration.md`. ## Loading a model **Generated class.** Drop a `.mlmodel` or `.mlpackage` into a target and Xcode generates a Swift class with typed inputs and outputs. Use it whenever it exists. ```swift import CoreML func makeDenoiser() throws -> PhotoDenoiser { let settings = MLModelConfiguration() settings.computeUnits = .all return try PhotoDenoiser(configuration: settings) } ``` **From a URL.** For models that are downloaded or stored outside the bundle: ```swift import CoreML func bundledModel(named name: String) throws -> MLModel { guard let url = Bundle.main.url(forResource: name, withExtension: "mlmodelc") else { throw CocoaError(.fileNoSuchFile) } return try MLModel(contentsOf: url, configuration: MLModelConfiguration()) } ``` **Asynchronously.** `MLModel.load(contentsOf:configuration:)` (async, iOS 15+) keeps large models from stalling the main thread and is the preferred loader. **Compiling on device.** `MLModel.compileModel(at:)` (async form, iOS 16+) turns a `.mlmodel` or `.mlpackage` into a `.mlmodelc` on the device, which is how server-delivered models become loadable. Two rules travel together: 1. Compile once per model version, never on every launch; recompiling at each start is a bug. 2. The returned URL points into a temporary location. Move the `.mlmodelc` somewhere durable such as Application Support and reuse it. When reviewing runtime-loaded models, state both: the async compile API needs iOS 16, and its output must be cached. ## Configuration and compute units `MLModelConfiguration` chooses hardware, GPU precision and model parameters. | `computeUnits` | Hardware | When | |---|---|---| | `.all` (default) | CPU, GPU, Neural Engine; the system decides | Almost always | | `.cpuOnly` | CPU | Deterministic tests, CPU-only fallbacks, or when profiling shows accelerator policy, contention, thermal state or energy budget is the constraint | | `.cpuAndGPU` | CPU and GPU | The model needs the GPU but has operations the Neural Engine cannot run | | `.cpuAndNeuralEngine` (iOS 16+) | CPU and Neural Engine | Best energy efficiency for models the Neural Engine fully supports | ```swift import CoreML func efficientSettings(fallbackToCPU: Bool) -> MLModelConfiguration { let settings = MLModelConfiguration() settings.computeUnits = fallbackToCPU ? .cpuOnly : .cpuAndNeuralEngine settings.allowLowPrecisionAccumulationOnGPU = true return settings } ``` `allowLowPrecisionAccumulationOnGPU` buys speed for a small loss of precision. Choose `.cpuOnly` only after profiling says so. ## Predictions **Generated class:** ```swift import CoreML func tagScene(_ frame: CVPixelBuffer, with model: SceneTagger) throws -> (String, Double) { let result = try model.prediction(input: SceneTaggerInput(image: frame)) return (result.classLabel, result.classLabelProbs[result.classLabel] ?? 0) } ``` `classLabel` is the winning label and `classLabelProbs` maps every label to its probability. **Dictionary provider**, for inputs only known at run time: ```swift import CoreML func tagWithThreshold(_ frame: CVPixelBuffer, cutoff: Double, model: MLModel) async throws -> String? { let inputs = try MLDictionaryFeatureProvider(dictionary: [ "image": MLFeatureValue(pixelBuffer: frame), "threshold": MLFeatureValue(double: cutoff) ]) let outputs = try await model.prediction(from: inputs) return outputs.featureValue(for: "label")?.stringValue } ``` ### Sync or async prediction - `prediction(from:)` and `prediction(from:options:)` exist in a synchronous throwing form on every supported OS. - From iOS 17 there is also `prediction(from:options:) async throws`, which may be called concurrently on the same model. Inside an `async` function the compiler resolves to that overload, so the call needs `try await`; leaving out `await` there does not compile. - In synchronous code (a serial queue, a synchronous actor method) the synchronous form is used without `await`. - Loading stays async either way (`MLModel.load`). ### Batches Processing several inputs per call raises throughput: ```swift import CoreML func labels(for frames: [CVPixelBuffer], model: MLModel) throws -> [String] { let rows = try frames.map { try MLDictionaryFeatureProvider(dictionary: ["image": MLFeatureValue(pixelBuffer: $0)]) } let answers = try model.predictions(fromBatch: MLArrayBatchProvider(array: rows)) return (0.. [String] { let memory = model.makeState() return try windows.compactMap { window in let input = try MLDictionaryFeatureProvider(dictionary: ["samples": MLFeatureValue(multiArray: window)]) return try model.prediction(from: input, using: memory).featureValue(for: "label")?.stringValue } } ``` Rules for `MLState`: - It is `Sendable`, which lets it move between concurrency domains. It does not make concurrent predictions on one state safe: run them one at a time. - Do not read or write the state's buffers while a prediction is running. - Independent concurrent streams each get their own `model.makeState()`. - The async `prediction(from:using:options:)` (iOS 18) is there when you need options or are in async code; the same one-prediction-in-flight rule holds. The synchronous `prediction(from:using:)` is for synchronous loops. ## MLTensor (iOS 18+) `MLTensor` is a Swift multidimensional array for pre- and post-processing. Operations are lazy; results materialize only when you await `shapedArray(of:)`. ```swift import CoreML func normalizedScores(_ logits: [Float]) async -> MLMultiArray { let raw = MLTensor(logits).reshaped(to: [1, logits.count]) let centered = raw - raw.mean() let probabilities = centered.softmax(alongAxis: -1) let shaped = await probabilities.shapedArray(of: Float.self) return MLMultiArray(shaped) } func tensor(from array: MLMultiArray) -> MLTensor { MLTensor(MLShapedArray(array)) } ``` Stay with documented operations. There is no `MLTensor(multiArray)` initializer, no `std()` or `standardDeviation()`, no direct access to the lazy storage, and no synchronous extraction. Go through `MLShapedArray` for conversions, and build statistics from documented pieces (`mean()`, `squareRoot()`, arithmetic) or do them outside the tensor pipeline. `MLTensor(zeros:scalarType:)` and `MLTensor(ones:scalarType:)` create filled tensors. ## MLMultiArray `MLMultiArray` is the exchange type for non-image inputs and outputs; generated classes use it for array features. ```swift import CoreML func embeddingInput(tokens: Int, width: Int) throws -> MLMultiArray { let grid = try MLMultiArray(shape: [1, NSNumber(value: tokens), NSNumber(value: width)], dataType: .float32) grid[[0, 0, 0] as [NSNumber]] = 0.25 return grid } func firstValue(of grid: MLMultiArray) -> Float { grid[[0, 0, 0] as [NSNumber]].floatValue } func packed(_ values: [Float]) -> MLMultiArray { MLMultiArray(MLShapedArray(scalars: values, shape: [1, values.count])) } ``` The shape reads batch, sequence, features. Tokenized text and audio feature patterns are in the reference. ## Image input Image models take a `CVPixelBuffer`. When Vision drives the model (`CoreMLRequest` or `VNCoreMLRequest`), conversion happens automatically; the manual path is only for direct `MLModel` predictions. ```swift import CoreGraphics import CoreVideo func pixelBuffer(from picture: CGImage, side: Int) -> CVPixelBuffer? { let options = [ kCVPixelBufferCGImageCompatibilityKey: true, kCVPixelBufferCGBitmapContextCompatibilityKey: true ] as CFDictionary var buffer: CVPixelBuffer? guard CVPixelBufferCreate(kCFAllocatorDefault, side, side, kCVPixelFormatType_32ARGB, options, &buffer) == kCVReturnSuccess, let buffer else { return nil } CVPixelBufferLockBaseAddress(buffer, []) defer { CVPixelBufferUnlockBaseAddress(buffer, []) } let context = CGContext( data: CVPixelBufferGetBaseAddress(buffer), width: side, height: side, bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(buffer), space: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue ) context?.draw(picture, in: CGRect(x: 0, y: 0, width: side, height: side)) return buffer } ``` Normalization and center-crop helpers are in the reference. ## Chaining models When pre- or post-processing is its own model, call them in order: ```swift import CoreML func runChain(_ input: MLFeatureProvider, stages: [MLModel]) async throws -> MLFeatureProvider { var current = input for stage in stages { current = try await stage.prediction(from: current) } return current } ``` For chains that Xcode manages, build a pipeline model inside the `.mlpackage`; each sub-model then runs on the compute unit that suits it. ## Through Vision Vision takes care of resizing, cropping, normalization, color space and orientation. ```swift import CoreML import Vision func dominantLabel(in photo: CGImage, model: MLModel) async throws -> (String, Float)? { let request = CoreMLRequest(model: try CoreMLModelContainer(model: model)) let observations = try await request.perform(on: photo) guard let top = observations.first as? ClassificationObservation else { return nil } return (top.identifier, top.confidence) } ``` `CoreMLRequest` needs iOS 18. The legacy route for detection models: ```swift import CoreML import Vision func legacyDetections(in frame: CVPixelBuffer, model: MLModel) throws -> [(String, Float, CGRect)] { let request = VNCoreMLRequest(model: try VNCoreMLModel(for: model)) request.imageCropAndScaleOption = .scaleFill try VNImageRequestHandler(cvPixelBuffer: frame).perform([request]) let objects = request.results as? [VNRecognizedObjectObservation] ?? [] return objects.compactMap { object in guard let label = object.labels.first else { return nil } return (label.identifier, label.confidence, object.boundingBox) } } ``` `boundingBox` is normalized. Everything else about Vision (OCR, barcodes, documents) is in `vision-framework`. ## Profiling `MLComputePlan` (iOS 17.4+) tells you, before any prediction, which device each operation prefers and how expensive it is: ```swift import CoreML func opsOffAccelerators(modelURL: URL) async throws -> [String] { let plan = try await MLComputePlan.load(contentsOf: modelURL, configuration: MLModelConfiguration()) guard case .program(let program) = plan.modelStructure, let entry = program.functions["main"] else { return [] } return entry.block.operations.compactMap { op in guard let usage = plan.deviceUsage(for: op), case .cpu = usage.preferred else { return nil } let cost = plan.estimatedCost(of: op)?.weight ?? 0 return "\(op.operatorName) cost \(cost)" } } ``` The Instruments Core ML template shows load time, per-prediction latency with a per-operation breakdown, which device each operation ran on, and memory. Profile with Product > Profile, not under the debugger, or the numbers are not representative. ## Deployment | Strategy | Strength | Cost | |---|---|---| | In the app bundle | Available instantly and offline | Bigger download | | Background Assets | Preferred for large or updatable models | Asset pack setup | | On-demand resources | Smaller first download for apps already on ODR | Legacy; choose Background Assets for new work | | CloudKit or own server | Most flexible | Needs network, more plumbing | - Size ceilings change; check the current App Store Connect limits. For iOS/iPadOS 18 and later they list 4 GB for the thinned app bundle and 8 GB per thinned on-demand-resource asset pack. - Keep on-demand-resource guidance for projects that already use it. - Ship `.mlmodelc` (precompiled) so the device does not compile at first use. - For downloaded `.mlmodel`/`.mlpackage` files: compile once with `compileModel(at:)`, move the result out of Core ML's temporary directory, and key the cache by model version. - With Background Assets: make the asset pack locally available, resolve the model's URL inside it, then `MLModel.load(contentsOf:configuration:)`. - Test on physical devices, above all the lowest-memory one you support: load time, first prediction, repeated predictions, background and foreground transitions, and low-memory behavior. On-demand resources, for apps that already use them: ```swift import CoreML func loadTaggedModel(tag: String, name: String) async throws -> (MLModel, NSBundleResourceRequest) { let request = NSBundleResourceRequest(tags: [tag]) try await request.beginAccessingResources() guard let url = Bundle.main.url(forResource: name, withExtension: "mlmodelc") else { request.endAccessingResources() throw CocoaError(.fileNoSuchFile) } let model = try await MLModel.load(contentsOf: url, configuration: MLModelConfiguration()) return (model, request) } ``` Keep the returned request alive while the model is in use and call `endAccessingResources()` when you are done with it. ## Memory - Drop model references when the app goes to the background to release GPU and Neural Engine memory; load again on return to the foreground. - `.all` is the default. Use `.cpuOnly` only when profiling or policy points to contention, thermal limits, energy, deterministic tests or a real background-execution constraint. - Keep one `MLModel` per compiled model and share it through an actor. - Models above roughly 100 MB can trigger memory warnings; listen for `UIApplication.didReceiveMemoryWarningNotification` and release cached models. ## Common mistakes 1. **Loading on the main thread.** Large models take seconds; use `MLModel.load(contentsOf:configuration:)` or a background actor. 2. **Recompiling every launch.** Compilation is expensive; compile once and keep the `.mlmodelc` in Application Support. 3. **Hard-coding `.cpuOnly`.** `.all` lets Core ML use the Neural Engine and GPU, usually faster and more efficient. 4. **Declaring GPU or Neural Engine off-limits for anything near the background.** Access depends on system policy, execution mode, contention, thermal state and energy; profile on a device. `.cpuOnly` is a tradeoff, not a rule, and a backgrounded app may be suspended or throttled anyway. 5. **Feature type mismatches.** Images go in as `MLFeatureValue(pixelBuffer:)`, not raw bytes; mismatches crash obscurely or give silently wrong output. 6. **A new `MLModel` per prediction.** Loading costs memory and time; load once and reuse through an actor. 7. **No error handling.** Loading and prediction can fail on older or constrained devices; catch and fall back. 8. **Assuming everything runs on the Neural Engine.** Unsupported operations fall back to the CPU and can dominate latency; check with `MLComputePlan`. 9. **Hand-preprocessing images for Vision.** `CoreMLRequest` (iOS 18) and `VNCoreMLRequest` already handle orientation, scaling and pixel format. 10. **Dropping `await` on prediction in async code.** On iOS 17 and later the async overload is selected there; write `try await`. ## Review checklist - [ ] Model loaded asynchronously - [ ] `computeUnits` chosen deliberately - [ ] One model instance reused across predictions - [ ] Generated class used where Xcode provides one - [ ] Load and prediction failures handled - [ ] Runtime-compiled models cached in a durable location - [ ] Image inputs routed through Vision (`CoreMLRequest` on iOS 18+, otherwise `VNCoreMLRequest`) - [ ] Device dispatch checked with `MLComputePlan` (iOS 17.4+) - [ ] Batch API used for many inputs - [ ] Model size matches the delivery strategy (bundle, Background Assets, ODR) - [ ] Memory measured on target devices, especially older low-RAM ones - [ ] Latency measured outside the debugger ## References - `references/coreml-swift-integration.md` - `apple-on-device-ai` for conversion and compression in Python - https://developer.apple.com/documentation/coreml - https://developer.apple.com/documentation/coreml/mlmodel - https://developer.apple.com/documentation/coreml/mltensor - https://developer.apple.com/documentation/coreml/mlcomputeplan-1w21n - https://developer.apple.com/documentation/backgroundassets