# Vision request recipes

Longer examples for the requests summarized in `../SKILL.md`. Swift-native
examples need iOS 18 or later unless marked otherwise; legacy examples are
labelled.

## Contents

- [Text recognition pipeline](#text-recognition-pipeline)
- [Faces with landmarks and quality](#faces-with-landmarks-and-quality)
- [Barcodes](#barcodes)
- [Person segmentation and background blur](#person-segmentation-and-background-blur)
- [Classification](#classification)
- [Saliency](#saliency)
- [Rectangles](#rectangles)
- [Horizon](#horizon)
- [Several requests on one image](#several-requests-on-one-image)
- [Camera frames from AVFoundation](#camera-frames-from-avfoundation)
- [Tracking across frames](#tracking-across-frames)
- [Coordinate helpers](#coordinate-helpers)
- [Performance](#performance)

## Text recognition pipeline

A reusable reader that turns a `UIImage` into lines with pixel-space frames.

```swift
import UIKit
import Vision

extension RecognizedTextObservation {
    var bestString: String? {
        topCandidates(1).first?.string
    }
}

struct ScannedLine: Sendable {
    let text: String
    let confidence: Float
    let frame: CGRect
}

enum ReceiptReaderError: Error {
    case noBitmap
}

struct ReceiptReader {
    func lines(in picture: UIImage) async throws -> [ScannedLine] {
        guard let bitmap = picture.cgImage else { throw ReceiptReaderError.noBitmap }
        var request = RecognizeTextRequest()
        request.recognitionLevel = .accurate
        request.recognitionLanguages = [Locale.Language(identifier: "en-CA")]
        request.usesLanguageCorrection = true
        let found = try await request.perform(on: bitmap)
        return found.compactMap { observation in
            guard let best = observation.topCandidates(1).first else {
                return nil
            }
            let frame = VNImageRectForNormalizedRect(
                observation.boundingBox.cgRect, bitmap.width, bitmap.height
            )
            return ScannedLine(text: best.string, confidence: best.confidence, frame: frame)
        }
    }
}
```

The `bestString` extension is reused by later examples. `boundingBox` is a
`NormalizedRect`; pass `.cgRect` to the C helper, or call
`boundingBox.toImageCoordinates(size, origin: .upperLeft)` to get a top-left
rectangle in one step.

### Several languages and domain words

```swift
import Vision

func multilingualRequest() -> RecognizeTextRequest {
    var request = RecognizeTextRequest()
    request.recognitionLanguages = [
        Locale.Language(identifier: "nl-NL"),
        Locale.Language(identifier: "en-US"),
        Locale.Language(identifier: "es-ES")
    ]
    request.customWords = ["visionOS", "TestFlight", "Xcode"]
    return request
}
```

List languages in priority order. `customWords` keeps the corrector from
"fixing" product names and jargon.

### Fast recognition on camera frames

```swift
import CoreMedia
import Vision

func quickLines(in frame: CMSampleBuffer) async throws -> [String] {
    var request = RecognizeTextRequest()
    request.recognitionLevel = .fast
    let found = try await request.perform(on: frame)
    return found.compactMap(\.bestString)
}
```

A `CMSampleBuffer` goes straight into `perform(on:)`; no conversion needed.

### Legacy version with a completion wrapper

```swift
import Vision

func legacyLines(in bitmap: CGImage) async throws -> [String] {
    try await withCheckedThrowingContinuation { continuation in
        DispatchQueue.global(qos: .userInitiated).async {
            let request = VNRecognizeTextRequest { finished, failure in
                if let failure {
                    continuation.resume(throwing: failure)
                    return
                }
                let found = finished.results as? [VNRecognizedTextObservation] ?? []
                continuation.resume(returning: found.compactMap { $0.topCandidates(1).first }.map(\.string))
            }
            request.recognitionLevel = .accurate
            request.recognitionLanguages = ["en-US", "pt-BR"]
            do {
                try VNImageRequestHandler(cgImage: bitmap).perform([request])
            } catch {
                continuation.resume(throwing: error)
            }
        }
    }
}
```

`VNRecognizeTextRequest` needs iOS 13. When `perform` throws, the completion
does not run, so the continuation is resumed exactly once either way.

## Faces with landmarks and quality

Run the three requests and join the results by position. Guard every index,
because the arrays are not guaranteed to line up in length.

```swift
import Vision

struct FaceReport: Sendable {
    let box: NormalizedRect
    let eyes: [NormalizedPoint]
    let nose: [NormalizedPoint]
    let mouth: [NormalizedPoint]
    let outline: [NormalizedPoint]
    let quality: FaceObservation.CaptureQuality?
}

func faceReports(in portrait: CGImage) async throws -> [FaceReport] {
    let boxes = try await DetectFaceRectanglesRequest().perform(on: portrait)
    let marks = try await DetectFaceLandmarksRequest().perform(on: portrait)
    let scores = try await DetectFaceCaptureQualityRequest().perform(on: portrait)

    return boxes.indices.map { i in
        let landmarks = i < marks.count ? marks[i].landmarks : nil
        return FaceReport(
            box: boxes[i].boundingBox,
            eyes: (landmarks?.leftEye.points ?? []) + (landmarks?.rightEye.points ?? []),
            nose: landmarks?.nose.points ?? [],
            mouth: landmarks?.outerLips.points ?? [],
            outline: landmarks?.faceContour.points ?? [],
            quality: i < scores.count ? scores[i].captureQuality : nil
        )
    }
}
```

`FaceObservation.CaptureQuality` exposes `score`. To match faces reliably
instead of by index, set `inputFaceObservations` on the landmark or quality
request to the rectangles you already found.

## Barcodes

```swift
import Vision

struct ScannedCode: Sendable {
    let payload: String?
    let kind: BarcodeSymbology
    let box: NormalizedRect
}

func codes(in label: CGImage,
           accepting kinds: [BarcodeSymbology] = [.qr, .ean13, .code128]) async throws -> [ScannedCode] {
    var request = DetectBarcodesRequest()
    request.symbologies = kinds
    return try await request.perform(on: label).map {
        ScannedCode(payload: $0.payloadString, kind: $0.symbology, box: $0.boundingBox)
    }
}

func firstLink(in poster: CGImage) async throws -> URL? {
    var request = DetectBarcodesRequest()
    request.symbologies = [.qr]
    let payload = try await request.perform(on: poster).first?.payloadString
    return payload.flatMap(URL.init(string:))
}
```

The Swift-native type is `BarcodeSymbology`; the legacy and VisionKit APIs use
`VNBarcodeSymbology`. Both carry the same names.

Two-dimensional: `.aztec`, `.dataMatrix`, `.microPDF417`, `.microQR`,
`.pdf417`, `.qr`.

One-dimensional: `.codabar`, `.code128`, `.code39`, `.code39Checksum`,
`.code39FullASCII`, `.code39FullASCIIChecksum`, `.code93`, `.code93i`,
`.ean13`, `.ean8`, `.gs1DataBar`, `.gs1DataBarExpanded`, `.gs1DataBarLimited`,
`.i2of5`, `.i2of5Checksum`, `.itf14`, `.msiPlessey`, `.upce`.

`supportedSymbologies` on the request lists what the current revision can read.

## Person segmentation and background blur

The mask comes back smaller than the photo, so scale it to the photo's extent
before blending.

```swift
import CoreImage
import CoreImage.CIFilterBuiltins
import Vision

func personMask(for photo: CGImage) async throws -> CIImage {
    let request = GeneratePersonSegmentationRequest()
    request.qualityLevel = .balanced
    let segmentation = try await request.perform(on: photo)
    let mask = CIImage(cgImage: try segmentation.cgImage)
    let sx = CGFloat(photo.width) / mask.extent.width
    let sy = CGFloat(photo.height) / mask.extent.height
    return mask.transformed(by: CGAffineTransform(scaleX: sx, y: sy))
}

func portraitBlur(_ photo: CGImage, blurRadius: Float = 18) async throws -> CIImage {
    let original = CIImage(cgImage: photo)
    let mask = try await personMask(for: photo)

    let blur = CIFilter.gaussianBlur()
    blur.inputImage = original
    blur.radius = blurRadius
    let softened = blur.outputImage?.cropped(to: original.extent) ?? original

    let blend = CIFilter.blendWithMask()
    blend.inputImage = original
    blend.backgroundImage = softened
    blend.maskImage = mask
    return blend.outputImage ?? original
}
```

The person stays sharp because the original is the input image and the blurred
copy is the background.

Legacy mask (iOS 15+):

```swift
import Vision

enum MaskError: Error {
    case empty
}

func legacyPersonMask(for photo: CGImage) throws -> CVPixelBuffer {
    let request = VNGeneratePersonSegmentationRequest()
    request.qualityLevel = .accurate
    request.outputPixelFormat = kCVPixelFormatType_OneComponent8
    try VNImageRequestHandler(cgImage: photo).perform([request])
    guard let first = request.results?.first else {
        throw MaskError.empty
    }
    return first.pixelBuffer
}
```

### One mask per person

```swift
import Vision

func individualMasks(in groupShot: CGImage) async throws -> [CVPixelBuffer] {
    guard let result = try await GeneratePersonInstanceMaskRequest().perform(on: groupShot) else { return [] }
    return try result.allInstances.map { try result.generateMask(for: IndexSet(integer: $0)) }
}

func legacyIndividualMasks(in groupShot: CGImage) throws -> [CVPixelBuffer] {
    let request = VNGeneratePersonInstanceMaskRequest()
    try VNImageRequestHandler(cgImage: groupShot).perform([request])
    guard let people = request.results?.first else {
        return []
    }
    return try people.allInstances.map { person in
        try people.generateMask(forInstances: [person])
    }
}
```

The legacy request needs iOS 17. The Swift-native observation also offers
`generateMaskedImage(for:imageFrom:croppedToInstancesExtent:)` when you want the
cut-out pixels rather than a mask.

## Classification

```swift
import Vision

func sceneTags(for picture: CGImage, limit: Int = 5) async throws -> [(label: String, score: Float)] {
    let labels = try await ClassifyImageRequest().perform(on: picture)
    return labels.prefix(limit).map { ($0.identifier, $0.confidence) }
}
```

## Saliency

Attention-based saliency predicts where people look. Objectness-based saliency
marks things that look like distinct objects.

```swift
import Vision

func focusRegions(in picture: CGImage) async throws -> [NormalizedRect] {
    let heat = try await GenerateAttentionBasedSaliencyImageRequest().perform(on: picture)
    return heat.salientObjects.map(\.boundingBox)
}

func objectRegions(in picture: CGImage) async throws -> [NormalizedRect] {
    let heat = try await GenerateObjectnessBasedSaliencyImageRequest().perform(on: picture)
    return heat.salientObjects.map(\.boundingBox)
}
```

`salientObjects` is a non-optional array in the Swift-native API; the legacy
`VNSaliencyImageObservation.salientObjects` is optional. The heat map itself is
`heat.heatMap`.

## Rectangles

```swift
import Vision

func cardOutlines(in picture: CGImage) async throws -> [NormalizedRect] {
    var request = DetectRectanglesRequest()
    request.minimumAspectRatio = 0.5
    request.maximumAspectRatio = 0.8
    request.minimumSize = 0.2
    request.maximumObservations = 3
    return try await request.perform(on: picture).map(\.boundingBox)
}
```

Aspect-ratio bounds filter shapes (a payment card is about 0.63), `minimumSize`
is a fraction of the image, and `maximumObservations` caps the result count.
Each `RectangleObservation` also exposes its four corners.

## Horizon

```swift
import Vision

func tiltCorrection(for picture: CGImage) async throws -> CGFloat? {
    guard let horizon = try await DetectHorizonRequest().perform(on: picture) else { return nil }
    return CGFloat(-horizon.angle.converted(to: .radians).value)
}
```

`angle` is a `Measurement<UnitAngle>`; convert it before using it as a
rotation. `horizon.transform(for:)` gives a ready-made affine transform.

## Several requests on one image

Swift-native requests are independent values, so run them concurrently:

```swift
import Vision

struct PhotoDigest: Sendable {
    let lines: [String]
    let faceCount: Int
    let links: [String]
}

func digest(of picture: CGImage) async throws -> PhotoDigest {
    async let text = RecognizeTextRequest().perform(on: picture)
    async let faces = DetectFaceRectanglesRequest().perform(on: picture)
    async let codes = DetectBarcodesRequest().perform(on: picture)

    return PhotoDigest(
        lines: try await text.compactMap(\.bestString),
        faceCount: try await faces.count,
        links: try await codes.compactMap(\.payloadString)
    )
}
```

`ImageRequestHandler(picture).perform(a, b, c)` is an alternative that shares
one decoded image across requests and returns a tuple of typed results.

With the legacy API, give one handler all requests at once:

```swift
import Vision

func legacyDigest(of picture: CGImage) throws -> (VNRecognizeTextRequest, VNDetectFaceRectanglesRequest, VNDetectBarcodesRequest) {
    let text = VNRecognizeTextRequest()
    let faces = VNDetectFaceRectanglesRequest()
    let codes = VNDetectBarcodesRequest()
    try VNImageRequestHandler(cgImage: picture).perform([text, faces, codes])
    return (text, faces, codes)
}
```

## Camera frames from AVFoundation

The capture delegate runs on its own serial queue and hands out non-Sendable
`CMSampleBuffer` values. Forward the newest frame into an `AsyncStream` and do
the Vision work in a consumer task; the stream's buffering policy drops frames
while one is being processed.

```swift
import AVFoundation
import Vision

struct CameraFrame: @unchecked Sendable {
    let sample: CMSampleBuffer
}

final class FramePump: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {
    let frames: AsyncStream<CameraFrame>
    private let sink: AsyncStream<CameraFrame>.Continuation
    private let output = AVCaptureVideoDataOutput()
    private let queue = DispatchQueue(label: "camera.frames", qos: .userInitiated)

    override init() {
        let pair = AsyncStream.makeStream(of: CameraFrame.self, bufferingPolicy: .bufferingNewest(1))
        frames = pair.stream
        sink = pair.continuation
        super.init()
    }

    func attach(to session: AVCaptureSession) {
        output.setSampleBufferDelegate(self, queue: queue)
        output.alwaysDiscardsLateVideoFrames = true
        if session.canAddOutput(output) {
            session.addOutput(output)
        }
    }

    func captureOutput(_ output: AVCaptureOutput,
                       didOutput sampleBuffer: CMSampleBuffer,
                       from connection: AVCaptureConnection) {
        sink.yield(CameraFrame(sample: sampleBuffer))
    }
}

func readLive(from pump: FramePump, show: @escaping @MainActor ([String]) -> Void) async {
    var request = RecognizeTextRequest()
    request.recognitionLevel = .fast
    for await frame in pump.frames {
        guard let found = try? await request.perform(on: frame.sample) else { continue }
        let lines = found.compactMap(\.bestString)
        await MainActor.run { show(lines) }
    }
}
```

The `@unchecked Sendable` wrapper is acceptable here because the frame is
never touched again after it is yielded.

## Tracking across frames

The Swift-native tracker keeps its own history, so there is no sequence
handler to manage. Keep the request object for as long as you track.

```swift
import Vision

final class SubjectFollower {
    private var request: TrackObjectRequest?
    private let minimumConfidence: Float = 0.3

    func begin(at box: NormalizedRect) {
        request = TrackObjectRequest(detectedObject: DetectedObjectObservation(boundingBox: box))
    }

    func next(_ frame: CVPixelBuffer) async throws -> NormalizedRect? {
        guard let request else { return nil }
        guard let hit = try await request.perform(on: frame), hit.confidence > minimumConfidence else {
            end()
            return nil
        }
        return hit.boundingBox
    }

    func end() {
        request = nil
    }
}
```

Legacy tracker with an explicit sequence handler:

```swift
import Vision

final class LegacySubjectFollower {
    private let sequence = VNSequenceRequestHandler()
    private var last: VNDetectedObjectObservation?

    func begin(at box: CGRect) {
        last = VNDetectedObjectObservation(boundingBox: box)
    }

    func next(_ frame: CVPixelBuffer) throws -> CGRect? {
        guard let seed = last else { return nil }
        let request = VNTrackObjectRequest(detectedObjectObservation: seed)
        request.trackingLevel = .accurate
        try sequence.perform([request], on: frame)
        guard let hit = request.results?.first as? VNDetectedObjectObservation, hit.confidence > 0.3 else {
            last = nil
            return nil
        }
        last = hit
        return hit.boundingBox
    }
}
```

## Coordinate helpers

Normalized values run from 0 to 1 with the origin at the lower left.

- `VNImageRectForNormalizedRect(rect, width, height)` scales a normalized
  rectangle to pixels.
- `VNImagePointForNormalizedPoint(point, width, height)` does the same for a
  point.
- Neither flips the y axis; UIKit drawing still needs a flip.
- Swift-native `NormalizedRect` and `NormalizedPoint` have
  `toImageCoordinates(_:origin:)` which scales and, with `.upperLeft`, flips.

```swift
import Vision

func viewRect(for normalized: CGRect, in viewSize: CGSize) -> CGRect {
    let pixelWidth = Int(viewSize.width)
    let pixelHeight = Int(viewSize.height)
    let scaled = VNImageRectForNormalizedRect(normalized, pixelWidth, pixelHeight)
    return CGRect(x: scaled.minX,
                  y: viewSize.height - scaled.minY - scaled.height,
                  width: scaled.width,
                  height: scaled.height)
}

func viewPoints(for normalized: [CGPoint], in viewSize: CGSize) -> [CGPoint] {
    normalized.map { p in
        CGPoint(x: p.x * viewSize.width, y: viewSize.height - p.y * viewSize.height)
    }
}
```

## Performance

Ballpark costs; measure on the oldest device you support.

| Scenario | Level | Typical cost |
|---|---|---|
| Live camera text | `.fast` | about 30 ms per frame |
| Photo library scan | `.accurate` | 200 to 500 ms per image |
| Batch document OCR | `.accurate` | 200 to 500 ms per page |
| Barcode scanning | `.fast` or `.balanced` | 15 to 50 ms per frame |

Memory:

- Reuse a single `VNSequenceRequestHandler` for a whole frame sequence.
- Work through large photo sets one image at a time so memory does not spike:

```swift
import Vision

func transcribeAll(_ pages: [CGImage]) async throws -> [[String]] {
    var output: [[String]] = []
    for page in pages {
        var request = RecognizeTextRequest()
        request.recognitionLevel = .accurate
        let found = try await request.perform(on: page)
        output.append(found.compactMap(\.bestString))
    }
    return output
}
```

- Let go of `CVPixelBuffer` references as soon as you are done; wrap tight
  synchronous loops over legacy handlers in `autoreleasepool`.

Threading:

- Swift-native `perform(on:)` is async and can be awaited from any context.
- Legacy `perform` is synchronous; call it from a background queue, never the
  main thread.
- `VNSequenceRequestHandler` has no internal locking; confine each instance to
  a single serial queue.

Reuse:

- Swift-native request structs are cheap values; build a fresh one per job
  rather than caching it. Stateful request classes are the exception: keep
  them for the sequence they track.
- A `VNImageRequestHandler` belongs to one image; create one per image. A
  `VNSequenceRequestHandler` is meant to be reused across a sequence.