{
  "skill_name": "coreml",
  "evals": [
    {
      "id": 1,
      "prompt": "Please audit a model wrapper from our iOS 26 app. An actor exposes an async method; on first call it builds the model via MLModel(contentsOf:), then calls `model.prediction(from: input)` with no await. Batches go through `model.predictions(from: batchProvider)`. A comment in the file claims Core ML has had no async prediction API until iOS 18. List the problems and the fixes.",
      "expected_output": "Points out that inside an async function on an iOS 17+ target the async prediction overload is selected, so the call must be `try await model.prediction(from:)`; that async prediction exists since iOS 17; recommends MLModel.load(contentsOf:configuration:) (iOS 15+) for loading; corrects the batch call; stays on Swift integration.",
      "files": [],
      "assertions": [
        "States that MLModel has an async prediction(from:options:) from iOS 17, so the comment claiming nothing async before iOS 18 is wrong",
        "States that inside an async function with an iOS 17+ deployment target the call must be written `try await model.prediction(from:)`, and that the synchronous form is for synchronous contexts or older targets",
        "Suggests `MLModel.load(contentsOf:configuration:)` as the loader and mentions its iOS 15 floor",
        "Replaces `predictions(from: batchProvider)` with `predictions(fromBatch:)` when no options are passed, or `predictions(from:options:)` when MLPredictionOptions are needed",
        "When compiling on device is discussed, gives iOS 16 as the floor for the async `MLModel.compileModel(at:)` and says the compiled result has to be kept",
        "Keeps the answer on the Swift integration; no detour into Python model conversion, quantization or choosing a framework"
      ]
    },
    {
      "id": 2,
      "prompt": "I have a stateful audio model on iOS 18+. I want to keep an MLState across 1-second windows, post-process the logits with MLTensor (softmax plus a standard deviation for a confidence score), and possibly process two microphones at once. Show me the Swift.",
      "expected_output": "Creates state with model.makeState() once per stream, passes it to every prediction, serializes predictions per state, uses one state per microphone, materializes tensors with await shapedArray(of:), and builds the spread from documented MLTensor operations.",
      "files": [],
      "assertions": [
        "Makes one state with `model.makeState()` and hands that same state to `prediction(from:using:)` for every window",
        "Calls synchronous `prediction(from:using:)` only from synchronous code and uses `try await` with `prediction(from:using:options:)` inside async code",
        "Runs calls on any one state strictly one after another and leaves its buffers alone during a call",
        "Gives each concurrent microphone stream its own state from `makeState()`",
        "Treats MLState as Sendable, while making clear that Sendable is no license for overlapping calls on a single state",
        "Reads tensor values with `await tensor.shapedArray(of:)`",
        "Does not use `MLTensor(multiArray)`, `std()` or `standardDeviation()`; builds the spread from documented operations such as mean, arithmetic and squareRoot, or computes it outside the tensor pipeline"
      ]
    },
    {
      "id": 3,
      "prompt": "Draft a brief note on how to ship a Core ML model of roughly 600 MB with our iOS app: weigh putting it in the app bundle, Background Assets, on-demand resources, and hosting it on our servers.",
      "expected_output": "Recommends Background Assets for a large, updatable model, treats on-demand resources as legacy, avoids invented size rules, points at current App Store Connect limits, recommends precompiled .mlmodelc and device validation, and keeps compute-unit guidance policy-based.",
      "files": [],
      "assertions": [
        "Picks Background Assets for a big model that will be updated, and presents on-demand resources as the older option kept for apps already on it",
        "Does not invent a blanket 200 MB cellular limit or a rule that anything over 50 MB must use on-demand resources",
        "Tells the reader to check current App Store Connect limits, or cites the iOS/iPadOS 18+ figures (4 GB thinned app bundle, 8 GB thinned on-demand-resource asset pack) with a caveat that they change",
        "Recommends shipping or caching a precompiled `.mlmodelc` and validating load and prediction on physical devices, including the lowest-memory one supported",
        "Frames `.cpuOnly` for background-adjacent work as a tradeoff driven by policy, contention, thermal state and energy, not as a rule"
      ]
    }
  ]
}