# Core ML compression and profiling

Keep the work here on the Python tooling and on profiling. Swift prediction
code and runtime setup belong to the `coreml` skill unless the user asked for
app code; the Swift section below is only a quick orientation.

## Contents

- [Picking a technique](#picking-a-technique)
- [Data-free weight quantization](#data-free-weight-quantization)
- [Palettization](#palettization)
- [Pruning](#pruning)
- [Combining techniques](#combining-techniques)
- [Per-op settings](#per-op-settings)
- [Quantization-aware training](#quantization-aware-training)
- [Swift at a glance](#swift-at-a-glance)
- [MLTensor](#mltensor)
- [Neural Engine habits](#neural-engine-habits)
- [Faster loading](#faster-loading)
- [Profiling](#profiling)
- [Compression mistakes](#compression-mistakes)

## Picking a technique

| Technique | Minimum OS | Size cut | Accuracy risk | Best on |
|---|---|---|---|---|
| 50% pruning | iOS 16 | about 2x | low | CPU, Neural Engine |
| 75% pruning | iOS 16 | about 4x | medium | CPU, Neural Engine |
| 4-bit palettization | iOS 16 | about 8x | low to medium | Neural Engine |
| 2-bit palettization | iOS 16 | about 16x | medium to high | Neural Engine |
| INT8, per channel | iOS 16 | about 4x | low | CPU and GPU |
| W8A8 | iOS 17 | about 4x | low | Neural Engine (A17 Pro, M4 and later) |
| INT4, per block | iOS 18 | about 8x | medium | GPU |

## Data-free weight quantization

INT8 on a converted model:

```python
# Data-free INT8 on an already converted model.
import coremltools.optimize as cto

int8_op = cto.coreml.OpLinearQuantizerConfig(
    weight_threshold=512,          # skip tensors with this many elements or fewer
    mode="linear_symmetric",       # "linear" is the asymmetric variant with a zero point
)
int8 = cto.coreml.OptimizationConfig(global_config=int8_op)
small = cto.coreml.linear_quantize_weights(mlmodel, config=int8)
```

INT4 per-block, applied to the PyTorch model before conversion:

```python
import coremltools.optimize.torch.quantization as ptq

int4_settings = {"block_size": 128, "granularity": "per_block", "weight_dtype": "int4"}
cfg = ptq.PostTrainingQuantizerConfig.from_dict({"global_config": int4_settings})
compressed = ptq.PostTrainingQuantizer(torch_model, cfg).compress()
```

With calibration data, GPTQ usually holds accuracy better:

```python
import coremltools.optimize.torch.layerwise_compression as lwc

gptq = lwc.LayerwiseCompressorConfig.from_dict({
    "calibration_nsamples": 16,
    "global_config": {"block_size": 128, "granularity": "per_block",
                      "weight_dtype": "uint4", "algorithm": "gptq"},
})
compressed = lwc.LayerwiseCompressor(torch_model, gptq).compress(calibration_loader)
```

## Palettization

Palettization replaces weights with indices into a small table of values. The
Neural Engine handles it well, and a 4-bit palette tends to lose less accuracy
than 4-bit linear quantizing does.

```python
palette_op = cto.coreml.OpPalettizerConfig(
    nbits=4,                              # allowed: 8, 6, 4, 3, 2, 1
    mode="kmeans",                        # the other mode is "uniform"
    group_size=16,
    granularity="per_grouped_channel",    # iOS 18+
)
palette = cto.coreml.OptimizationConfig(global_config=palette_op)
paletted = cto.coreml.palettize_weights(mlmodel, config=palette)
```

| Bits | Size cut | Typical quality | Distinct values |
|---|---|---|---|
| 8 | about 2x | excellent | 256 |
| 6 | about 2.7x | very good | 64 |
| 4 | about 8x | good | 16 |
| 3 | about 10.7x | moderate | 8 |
| 2 | about 16x | fair | 4 |
| 1 | about 32x | poor | 2 |

## Pruning

Zero out small weights by magnitude:

```python
by_size = cto.coreml.OpMagnitudePrunerConfig(weight_threshold=2048, target_sparsity=0.75)
prune_cfg = cto.coreml.OptimizationConfig(global_config=by_size)
sparse = cto.coreml.prune_weights(mlmodel, config=prune_cfg)
```

Or by an absolute threshold:

```python
tiny = cto.coreml.OpThresholdPrunerConfig(minimum_sparsity_percentile=0.5, threshold=1e-12)
sparse = cto.coreml.prune_weights(
    mlmodel, config=cto.coreml.OptimizationConfig(global_config=tiny))
```

## Combining techniques

Palettize first, then prune with `joint_compression=True` so the sparsity is
applied on top of the palette:

```python
paletted = cto.coreml.palettize_weights(mlmodel, config=palette)
both = cto.coreml.prune_weights(paletted, config=prune_cfg, joint_compression=True)
```

## Per-op settings

`OptimizationConfig` can vary the treatment by op type or by op name. `None`
leaves that op uncompressed, which is useful for embeddings and other sensitive
layers.

```python
mixed = cto.coreml.OptimizationConfig(
    global_config=cto.coreml.OpPalettizerConfig(nbits=6),
    op_type_configs={
        "linear": cto.coreml.OpPalettizerConfig(nbits=4),
        "conv": cto.coreml.OpPalettizerConfig(nbits=6),
    },
    op_name_configs={"token_embedding": None},
)
```

## Quantization-aware training

QAT gives the best accuracy at low bit widths because the model learns around
the quantization during fine-tuning:

```python
import coremltools.optimize.torch.quantization as ptq

module_cfg = ptq.ModuleLinearQuantizerConfig(
    milestones=[0, 200, 400, 400],
    quantization_scheme="symmetric",
)
qat_cfg = ptq.LinearQuantizerConfig(global_config=module_cfg)
quantizer = ptq.LinearQuantizer(torch_model, qat_cfg)
torch_model = quantizer.prepare(example_inputs=(sample,), inplace=True)

for batch in train_loader:
    train_step(torch_model, batch)
    quantizer.step()

torch_model = quantizer.finalize(inplace=True)
```

## Swift at a glance

For orientation only; loading, configuration and prediction lifecycle in an app
belong to the `coreml` skill.

```swift
import CoreML

let runSettings = MLModelConfiguration()
runSettings.computeUnits = .all
runSettings.allowLowPrecisionAccumulationOnGPU = true
// runSettings.functionName = "legal"   // multifunction models, iOS 18+

let generated = try LeafClassifier(configuration: runSettings)   // Xcode-generated class
let compiledLocation = try await MLModel.compileModel(at: downloadedPackageURL)
let runtime = try MLModel(contentsOf: compiledLocation, configuration: runSettings)
```

- `model.prediction(input:)` is synchronous. From iOS 17, the async
  `try await model.prediction(input:)` is thread-safe and cancellable, and
  running several predictions concurrently this way can raise throughput for
  batch workloads; measure the gain on your model.
- `model.predictions(inputs:)` runs a batch.
- Without a generated class, build inputs with
  `MLDictionaryFeatureProvider(dictionary:)` using
  `MLFeatureValue(pixelBuffer:)` or `MLFeatureValue(double:)` and call
  `prediction(from:)`.
- Vision: wrap with `VNCoreMLModel(for:)`, run a `VNCoreMLRequest` through
  `VNImageRequestHandler(cgImage:).perform([request])`, and read
  `[VNClassificationObservation]` in the completion.
- Natural Language: `NLModel(mlModel:)` then `predictedLabel(for:)`.

## MLTensor

From iOS 18, `MLTensor` handles multidimensional math next to Core ML models:

```swift
let logits = MLTensor([2.0, 0.5, -1.0, 3.0] as [Float]).reshaped(to: [2, 2])
let probabilities = logits.softmax()
let identity = MLTensor(shape: [2, 2], scalars: [1, 0, 0, 1] as [Float])
let projected = logits.matmul(identity)
```

## Neural Engine habits

- Ops the Neural Engine cannot run push the graph to the CPU or GPU and add
  transfer cost. Find and replace them.
- Use 4-bit or 6-bit palettization for the best Neural Engine memory use and
  latency.
- On A17 Pro and M4 or later, W8A8 unlocks fast INT8 compute on the Neural
  Engine.

## Faster loading

- Ship or cache `.mlmodelc`. The first load of an uncompiled model runs a slow,
  device-specific compile; after `MLModel.compileModel(at:)`, move the result
  to a permanent location and reuse it.
- Split very large, slow-to-load models with `bisect_model()`.
- `MLComputePlan` (iOS 17.4+) reports per-op placement and cost from code.

## Profiling

- Xcode: open the `.mlpackage` and use the Performance tab for load time,
  prediction time and the compute unit each op runs on.
- Instruments: the Core ML instrument.
- Code: `MLComputePlan`.
- coremltools debugging helpers: `MLModelValidator`, `MLModelComparator`,
  `MLModelInspector`, `MLModelBenchmarker`.
- Tell Core ML that input shapes rarely change:

```python
hints = {"reshapeFrequency": ct.ReshapeFrequency.Infrequent}
tuned = ct.models.MLModel("Segmenter.mlpackage", optimization_hints=hints)
```

## Compression mistakes

| Mistake | Fix |
|---|---|
| Compressing without checking accuracy | Compare outputs with the original, for example with `MLModelComparator` |
| Ignoring `weight_threshold` | Leave tensors under about 512 elements unquantized; the overhead outweighs the saving |
| Calling sync prediction from async code | Use async prediction (iOS 17+) |
| Skipping precompilation | Compile once and cache `.mlmodelc` |
| Wrong compute units | `.all` in production, `.cpuOnly` only while debugging |
| Testing only in the Simulator | It has no Metal GPU or Neural Engine; use devices |
