---
name: vision-framework
description: "On-device computer vision with Vision and VisionKit: OCR, face detection and landmarks, barcode and QR scanning, person and instance segmentation, object tracking, document reading and scanning, saliency, pose, Core ML inference, Swift-native (iOS 18+) and legacy VNRequest APIs, DataScannerViewController live scanning. Use when adding OCR, barcode scanning, face detection, segmentation, tracking or Core ML image inference through Vision. Not for Core ML model conversion."
metadata:
  source: multi-agent-pipeline
---

# Vision Framework

Vision finds text, faces, barcodes, objects, people and body poses in still
images and video frames. Everything runs on the device; no image leaves it.
VisionKit sits on top and supplies ready-made camera UI for live scanning and
document capture.

Baseline for this skill: iOS 26 and Swift 6.3. Where an API has a lower floor
it is stated next to the API, so older deployment targets can still use the
guidance.

Load a reference when the section below is not enough:

- `references/vision-requests.md`: longer request recipes (OCR pipeline, faces,
  barcodes, segmentation and blur, saliency, rectangles, horizon, batching,
  camera frames, tracking, coordinate helpers, performance).
- `references/visionkit-scanner.md`: camera permission, `DataScannerViewController`
  in UIKit and SwiftUI, delegate and async item streams, overlays, and
  `VNDocumentCameraViewController` with an OCR pass.

## Choosing an API generation

Vision ships two parallel APIs. New code should use the Swift-native one.

| Aspect | Swift-native API | Legacy API |
|---|---|---|
| Availability | iOS 18, macOS 15, tvOS 18, visionOS 2 | iOS 11 and later (individual requests vary, e.g. `VNRecognizeTextRequest` is iOS 13) |
| Types | Swift structs such as `RecognizeTextRequest`, `DetectFaceRectanglesRequest` | Objective-C classes such as `VNRecognizeTextRequest`, `VNDetectFaceRectanglesRequest` |
| Running | `try await request.perform(on: image)` | `VNImageRequestHandler` plus `perform([request])` and a completion closure |
| Results | Typed return value (`[RecognizedTextObservation]`, `[FaceObservation]`) | `request.results` cast from a generic observation array |
| Geometry | `NormalizedRect` / `NormalizedPoint` with conversion helpers | `CGRect` / `CGPoint` in normalized space |

Every Swift-native image request conforms to `ImageProcessingRequest`, whose
`perform(on:orientation:)` accepts a `CGImage`, `CIImage`, `CVPixelBuffer`,
`CMSampleBuffer`, image `Data` or a file `URL`. The orientation argument is
optional.

Most requests are value types. Requests that must remember earlier frames are
`final class` types conforming to `StatefulRequest`: `TrackObjectRequest`,
`TrackRectangleRequest`, `TrackOpticalFlowRequest`,
`TrackTranslationalImageRegistrationRequest`,
`TrackHomographicImageRegistrationRequest`, `DetectTrajectoriesRequest`,
`DetectHumanBodyPose3DRequest` and `GeneratePersonSegmentationRequest`. Keep one
instance alive for the whole sequence.

## The request pattern

Build the request, set options, await `perform(on:)`, read typed results.

```swift
import Vision

func menuLines(in photo: CGImage) async throws -> [String] {
    var request = RecognizeTextRequest()
    request.recognitionLevel = .accurate
    request.recognitionLanguages = [Locale.Language(identifier: "it-IT")]
    let observations = try await request.perform(on: photo)
    return observations.compactMap { $0.topCandidates(1).first?.string }
}
```

The same job with the legacy API, for deployment targets below iOS 18:

```swift
import Vision

func legacyMenuLines(in photo: CGImage) throws -> [String] {
    var lines: [String] = []
    let request = VNRecognizeTextRequest { finished, _ in
        let found = finished.results as? [VNRecognizedTextObservation] ?? []
        lines = found.compactMap { $0.topCandidates(1).first?.string }
    }
    request.recognitionLevel = .accurate
    try VNImageRequestHandler(cgImage: photo).perform([request])
    return lines
}
```

`VNImageRequestHandler.perform` is synchronous and calls the completion before
it returns, so run it on a background queue.

## Text recognition (OCR)

`RecognizeTextRequest` (iOS 18+) options:

- `recognitionLevel`: `.accurate` for stills and offline work, `.fast` for
  live video.
- `recognitionLanguages`: `[Locale.Language]`, ordered by priority.
- `usesLanguageCorrection`: applies language-model correction; turn it off for
  codes, serial numbers and similar strings.
- `customWords`: domain vocabulary the corrector should accept.

Each `RecognizedTextObservation` offers `topCandidates(_:)`; a candidate
(`RecognizedText`) has `string` and a `confidence` from 0.0 to 1.0. The
observation's `boundingBox` is a `NormalizedRect`.

`VNRecognizeTextRequest` has the same `recognitionLevel` and
`usesLanguageCorrection` switches but takes languages as identifier strings
such as `["de-DE", "en-GB"]`. That is the practical difference: `Locale.Language`
values in the new API, strings in the old one.

## Faces

- `DetectFaceRectanglesRequest` returns `[FaceObservation]`. Each has a
  `boundingBox` (`NormalizedRect`) and `roll`, `yaw` and `pitch` as
  `Measurement<UnitAngle>`.
- `DetectFaceLandmarksRequest` fills the optional `landmarks`. Regions such as
  `leftEye`, `rightEye`, `nose`, `outerLips` and `faceContour` expose `points`
  (`[NormalizedPoint]`) and `pointsInImageCoordinates(_:origin:)`. The legacy
  `VNFaceLandmarkRegion2D` calls the same data `normalizedPoints`.
- `DetectFaceCaptureQualityRequest` fills `captureQuality?.score`.

### Coordinates

Vision coordinates are normalized to 0...1 with the origin at the lower left.
UIKit and SwiftUI put the origin at the top left, so flip before drawing:

```swift
import Vision

func overlayFrame(for face: FaceObservation, imageSize: CGSize) -> CGRect {
    face.boundingBox.toImageCoordinates(imageSize, origin: .upperLeft)
}

func flippedRect(_ normalized: CGRect, imageSize: CGSize) -> CGRect {
    let w = normalized.width * imageSize.width
    let h = normalized.height * imageSize.height
    let x = normalized.minX * imageSize.width
    let y = imageSize.height - normalized.minY * imageSize.height - h
    return CGRect(x: x, y: y, width: w, height: h)
}
```

The manual version keeps x, width and height and computes
`y = imageHeight - originY - height`.

## Barcodes

`DetectBarcodesRequest` reads 1D and 2D codes, QR included. Set `symbologies`
to only what the product accepts, for example `[.qr, .pdf417, .ean13, .code128]`.
Each `BarcodeObservation` carries `payloadString` (decoded text, optional),
`symbology` and a normalized `boundingBox`. Frequently used symbologies:
`.qr`, `.aztec`, `.dataMatrix`, `.pdf417`, `.ean8`, `.ean13`, `.upce`,
`.code39`, `.code128`, `.itf14`. The full list is in
`references/vision-requests.md`.

## Structured documents (iOS 26+)

`RecognizeDocumentsRequest` goes beyond line-by-line OCR and understands layout.
It returns `[DocumentObservation]`; `observation.document` is a
`DocumentObservation.Container` with:

- `text`: the whole document as a text block (`.transcript` for the string)
- `paragraphs`: text blocks, each with `.transcript`
- `tables` (rows and cells), `lists` (items), `barcodes`
- `title`: optional text block

```swift
import Vision

func invoiceParagraphs(from scan: CGImage) async throws -> [String] {
    let request = RecognizeDocumentsRequest()
    guard let document = try await request.perform(on: scan).first?.document else { return [] }
    return document.paragraphs.map(\.transcript)
}
```

When the user should photograph paper with the camera, VisionKit's
`VNDocumentCameraViewController` is simpler: a full-screen camera with
automatic capture, perspective correction and multi-page scans. See
`references/visionkit-scanner.md`.

## Segmentation

`GeneratePersonSegmentationRequest` (iOS 18+, a stateful class) returns a
`PixelBufferObservation` mask. Read it as a `CGImage` with `try mask.cgImage`,
wrap it in `CIImage`, and composite with Core Image's
`CIFilter.blendWithMask()`.

`qualityLevel` trades speed for detail. Approximate figures, to be confirmed
on the target device:

| Level | Detail | Rough cost | Mask size |
|---|---|---|---|
| `.accurate` | best edges | about 1 s | full resolution |
| `.balanced` | good | about 100 ms | 960 x 540 |
| `.fast` | coarse | about 10 ms | 256 x 144, suits real time |

Legacy equivalent: `VNGeneratePersonSegmentationRequest` (iOS 15+) with
`qualityLevel`, `outputPixelFormat = kCVPixelFormatType_OneComponent8`, run
through `VNImageRequestHandler`; the mask is `request.results?.first?.pixelBuffer`.

Instance segmentation separates people from each other:

- `GeneratePersonInstanceMaskRequest` (iOS 18+) returns an optional
  `InstanceMaskObservation`. `allInstances` is an `IndexSet`;
  `generateMask(for: IndexSet(integer: n))` returns a `CVPixelBuffer` showing
  only person `n`.
- `VNGeneratePersonInstanceMaskRequest` (iOS 17+) returns
  `VNInstanceMaskObservation` with `allInstances`, `generateMask(forInstances:)`
  and `generateMaskedImage(ofInstances:from:croppedToInstancesExtent:)`, which
  takes the request handler and a crop flag.

Code for masks, background blur and instances is in `references/vision-requests.md`.

## Object tracking

`TrackObjectRequest` (iOS 18+) is a `final class` that conforms to
`ImageProcessingRequest` and `StatefulRequest` and keeps the tracking context
between frames itself. Seed it once, then call `perform(on:)` on each frame
with the same instance:

```swift
import Vision

final class BallTracker {
    private let request: TrackObjectRequest

    init(startBox: NormalizedRect) {
        request = TrackObjectRequest(detectedObject: DetectedObjectObservation(boundingBox: startBox))
    }

    func follow(_ frame: CVPixelBuffer) async throws -> (NormalizedRect, Float)? {
        guard let hit = try await request.perform(on: frame) else { return nil }
        return (hit.boundingBox, hit.confidence)
    }
}
```

The Swift-native `TrackObjectRequest` has no `trackingLevel`;
`TrackRectangleRequest` does. With the legacy API, create
`VNTrackObjectRequest(detectedObjectObservation:)`, set
`trackingLevel = .accurate`, reuse one `VNSequenceRequestHandler` for every
frame via `perform([request], on: pixelBuffer)`, and feed each result back
through `inputObservation`.

## Other requests

All of these are Swift-native request types available on iOS 18 and macOS 15.

| Request | What it gives you |
|---|---|
| `ClassifyImageRequest` | Scene labels (beach, food, dog, and so on) |
| `GenerateAttentionBasedSaliencyImageRequest` | Heat map of where a viewer's eye goes |
| `GenerateObjectnessBasedSaliencyImageRequest` | Heat map of regions that look like objects |
| `GenerateForegroundInstanceMaskRequest` | Masks for foreground subjects of any kind, not only people |
| `DetectRectanglesRequest` | Rectangular shapes: cards, receipts, screens |
| `DetectHorizonRequest` | Horizon angle for straightening a photo |
| `DetectHumanBodyPoseRequest` / `DetectHumanBodyPose3DRequest` | 2D body joints / 3D body pose |
| `DetectHumanHandPoseRequest` | Hand and finger joints |
| `DetectAnimalBodyPoseRequest` | Animal joints |
| `DetectFaceCaptureQualityRequest` | Face capture quality score 0...1, for choosing the best shot |
| `TrackRectangleRequest` | Follows a rectangle across frames |
| `TrackOpticalFlowRequest` | Motion field between consecutive frames |
| `DetectTrajectoriesRequest` | Paths of moving objects in video |

## Running Core ML models through Vision

Vision prepares the image for the model (scaling, cropping, pixel format,
color space, orientation), so hand it the original image.

```swift
import CoreML
import Vision

func plantSpecies(in leaf: CGImage, modelURL: URL) async throws -> (String, Float)? {
    let model = try await MLModel.load(contentsOf: modelURL)
    let request = CoreMLRequest(model: try CoreMLModelContainer(model: model))
    let results = try await request.perform(on: leaf)
    guard let best = results.first as? ClassificationObservation else { return nil }
    return (best.identifier, best.confidence)
}
```

Before iOS 18: wrap the model with `VNCoreMLModel(for:)`, create
`VNCoreMLRequest(model:completionHandler:)`, cast `results` to
`[VNClassificationObservation]`, and run it with `VNImageRequestHandler`.

Model loading, compute units, conversion and optimization belong to the
`coreml` skill.

## VisionKit live scanning: DataScannerViewController

`DataScannerViewController` (iOS 16+) is a full-screen live camera that finds
text and barcodes and highlights them.

```swift
import UIKit
import VisionKit

@MainActor
func presentTicketScanner(from host: UIViewController, delegate: DataScannerViewControllerDelegate) {
    guard DataScannerViewController.isSupported, DataScannerViewController.isAvailable else { return }
    let scanner = DataScannerViewController(
        recognizedDataTypes: [.text(languages: ["en"]), .barcode(symbologies: [.qr, .aztec])],
        qualityLevel: .balanced,
        recognizesMultipleItems: false,
        isHighFrameRateTrackingEnabled: true,
        isHighlightingEnabled: true
    )
    scanner.delegate = delegate
    host.present(scanner, animated: true) {
        try? scanner.startScanning()
    }
}
```

`isSupported` checks the hardware (A12 Bionic or later); `isAvailable`
additionally requires camera permission. In SwiftUI, wrap the controller in a
`UIViewControllerRepresentable`; see `references/visionkit-scanner.md`.

## Common mistakes

1. **Legacy handlers in new code.** A project targeting iOS 18 or later should
   use the struct requests and `perform(on:)`: typed results, async/await, and
   Swift errors instead of casts and callbacks.
2. **Drawing normalized coordinates directly.** Vision uses 0...1 with a
   lower-left origin; UIKit uses points from the top left. Convert with
   `toImageCoordinates(_:origin:)`, `VNImageRectForNormalizedRect(_:_:_:)` or a
   manual flip.
3. **Running requests on the main thread.** Recognition is heavy CPU/GPU work.
   Await the async API from a task, or dispatch legacy handlers to a
   background queue, never block the main actor.
4. **`.accurate` on a live camera feed.** It cannot keep up with 30 fps. Use
   `.fast` for video and `.accurate` for photos and offline batches.
5. **Ignoring confidence.** Low-confidence results are often wrong; drop
   anything under a threshold that fits the feature (0.5 is a common start).
6. **A new image handler per video frame when tracking.** Legacy tracking
   needs `VNSequenceRequestHandler`, which carries temporal state across frames.
7. **Asking for every symbology.** Each extra symbology costs time and invites
   false positives; list only the ones you accept, for example just `.qr`.
8. **Assuming the scanner exists.** Check `DataScannerViewController.isSupported`
   (hardware) and `isAvailable` (hardware plus camera authorization) first.

## Review checklist

- [ ] Swift-native API used unless the deployment target is below iOS 18
- [ ] No Vision work blocks the main thread
- [ ] Normalized coordinates converted before anything is drawn
- [ ] Results filtered by a confidence threshold
- [ ] Recognition level fits the input: `.fast` for video, `.accurate` for stills
- [ ] Recognition languages set when the input language is known
- [ ] Barcode symbologies restricted to the needed set
- [ ] `DataScannerViewController` support and availability checked before presenting
- [ ] `NSCameraUsageDescription` present in Info.plist for VisionKit cameras
- [ ] Person segmentation quality level matches the use case
- [ ] Legacy video tracking uses one `VNSequenceRequestHandler`, not a handler per frame
- [ ] Request failures and empty results handled

## References

- `references/vision-requests.md`
- `references/visionkit-scanner.md`
- https://developer.apple.com/documentation/vision
- https://developer.apple.com/documentation/visionkit
- https://developer.apple.com/documentation/vision/recognizetextrequest
- https://developer.apple.com/documentation/visionkit/datascannerviewcontroller