# Model Serving

TensorFlow Serving, TorchServe, Triton Inference Server, model optimization, and production deployment patterns.

## Overview

Model serving provides infrastructure for deploying ML models to production with low latency, high throughput, and operational reliability.

## Core Concepts

### Serving Patterns
- **Online Inference**: Real-time, synchronous
- **Batch Inference**: Offline, bulk processing
- **Streaming Inference**: Continuous data streams
- **Edge Inference**: On-device deployment

### Key Metrics
- **Latency**: p50, p95, p99 response times
- **Throughput**: Requests per second
- **Availability**: Uptime percentage
- **Resource Utilization**: CPU/GPU/Memory

## TensorFlow Serving

### Model Export
```python
import tensorflow as tf

# Save model in SavedModel format
model = tf.keras.models.load_model("model.h5")

# Define serving signature
@tf.function(input_signature=[tf.TensorSpec(shape=[None, 224, 224, 3], dtype=tf.float32)])
def serve(images):
    return {"predictions": model(images)}

tf.saved_model.save(
    model,
    "models/image_classifier/1",
    signatures={"serving_default": serve}
)
```

### Docker Deployment
```dockerfile
FROM tensorflow/serving:latest

COPY models /models

ENV MODEL_NAME=image_classifier
ENV MODEL_BASE_PATH=/models

EXPOSE 8500 8501
```

### Client Request
```python
import requests
import numpy as np

# REST API
data = {"instances": images.tolist()}
response = requests.post(
    "http://localhost:8501/v1/models/image_classifier:predict",
    json=data
)
predictions = response.json()["predictions"]

# gRPC (more efficient)
import grpc
from tensorflow_serving.apis import predict_pb2, prediction_service_pb2_grpc

channel = grpc.insecure_channel("localhost:8500")
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

request = predict_pb2.PredictRequest()
request.model_spec.name = "image_classifier"
request.inputs["images"].CopyFrom(tf.make_tensor_proto(images))

response = stub.Predict(request, timeout=10.0)
```

## TorchServe

### Model Handler
```python
# custom_handler.py
from ts.torch_handler.base_handler import BaseHandler
import torch
import json

class ImageClassifierHandler(BaseHandler):
    def __init__(self):
        super().__init__()
        self.initialized = False

    def initialize(self, context):
        self.manifest = context.manifest
        model_dir = context.system_properties.get("model_dir")

        self.model = torch.jit.load(f"{model_dir}/model.pt")
        self.model.eval()
        self.initialized = True

    def preprocess(self, data):
        images = []
        for row in data:
            image = row.get("data") or row.get("body")
            # Transform image
            tensor = self.transform(image)
            images.append(tensor)
        return torch.stack(images)

    def inference(self, data):
        with torch.no_grad():
            return self.model(data)

    def postprocess(self, inference_output):
        return inference_output.tolist()
```

### Model Archive
```bash
# Create model archive
torch-model-archiver \
    --model-name image_classifier \
    --version 1.0 \
    --model-file model.py \
    --serialized-file model.pt \
    --handler custom_handler.py \
    --extra-files index_to_name.json

# Start TorchServe
torchserve --start \
    --model-store model_store \
    --models image_classifier=image_classifier.mar
```

## Triton Inference Server

### Model Repository
```
model_repository/
├── image_classifier/
│   ├── config.pbtxt
│   ├── 1/
│   │   └── model.onnx
│   └── 2/
│       └── model.onnx
└── text_classifier/
    ├── config.pbtxt
    └── 1/
        └── model.pt
```

### Configuration
```protobuf
# config.pbtxt
name: "image_classifier"
platform: "onnxruntime_onnx"
max_batch_size: 32

input [
  {
    name: "images"
    data_type: TYPE_FP32
    dims: [ 3, 224, 224 ]
  }
]

output [
  {
    name: "predictions"
    data_type: TYPE_FP32
    dims: [ 1000 ]
  }
]

instance_group [
  {
    count: 2
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]

dynamic_batching {
  preferred_batch_size: [ 8, 16, 32 ]
  max_queue_delay_microseconds: 100
}
```

### Ensemble Models
```protobuf
# ensemble_config.pbtxt
name: "ensemble_pipeline"
platform: "ensemble"
max_batch_size: 32

input [
  { name: "raw_image" data_type: TYPE_UINT8 dims: [ -1, -1, 3 ] }
]

output [
  { name: "classification" data_type: TYPE_FP32 dims: [ 1000 ] }
]

ensemble_scheduling {
  step [
    {
      model_name: "preprocessing"
      model_version: -1
      input_map { key: "raw_image" value: "raw_image" }
      output_map { key: "processed_image" value: "images" }
    },
    {
      model_name: "image_classifier"
      model_version: -1
      input_map { key: "images" value: "processed_image" }
      output_map { key: "predictions" value: "classification" }
    }
  ]
}
```

## Model Optimization

### Quantization
```python
import torch

# Dynamic quantization (CPU)
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear, torch.nn.LSTM},
    dtype=torch.qint8
)

# Static quantization
model.qconfig = torch.quantization.get_default_qconfig("fbgemm")
prepared_model = torch.quantization.prepare(model)
# Calibrate with representative data
for data in calibration_loader:
    prepared_model(data)
quantized_model = torch.quantization.convert(prepared_model)
```

### ONNX Export
```python
import torch.onnx

torch.onnx.export(
    model,
    dummy_input,
    "model.onnx",
    export_params=True,
    opset_version=13,
    do_constant_folding=True,
    input_names=["input"],
    output_names=["output"],
    dynamic_axes={
        "input": {0: "batch_size"},
        "output": {0: "batch_size"}
    }
)
```

### TensorRT Optimization
```python
import tensorrt as trt

logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, logger)

with open("model.onnx", "rb") as f:
    parser.parse(f.read())

config = builder.create_builder_config()
config.max_workspace_size = 1 << 30  # 1GB
config.set_flag(trt.BuilderFlag.FP16)  # Enable FP16

engine = builder.build_engine(network, config)

with open("model.trt", "wb") as f:
    f.write(engine.serialize())
```

## Kubernetes Deployment

### Serving Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-serving
spec:
  replicas: 3
  selector:
    matchLabels:
      app: model-serving
  template:
    metadata:
      labels:
        app: model-serving
    spec:
      containers:
      - name: triton
        image: nvcr.io/nvidia/tritonserver:23.10-py3
        args:
          - tritonserver
          - --model-repository=s3://models/repository
          - --strict-model-config=false
        ports:
        - containerPort: 8000
          name: http
        - containerPort: 8001
          name: grpc
        - containerPort: 8002
          name: metrics
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: 8Gi
          requests:
            memory: 4Gi
        livenessProbe:
          httpGet:
            path: /v2/health/live
            port: 8000
        readinessProbe:
          httpGet:
            path: /v2/health/ready
            port: 8000
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-serving-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: model-serving
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
```

## Best Practices

1. **Model Versioning**: Support multiple versions
2. **A/B Testing**: Traffic splitting capabilities
3. **Canary Deployments**: Gradual rollouts
4. **Health Checks**: Readiness and liveness probes
5. **Monitoring**: Latency, throughput, errors

## Anti-Patterns

- No model versioning
- Missing health checks
- Ignoring batch optimization
- No request validation
- Skipping load testing

## When to Use

- Real-time predictions needed
- Multiple models to serve
- Need for GPU acceleration
- High availability requirements
- Model version management

## When NOT to Use

- Simple batch processing
- One-off predictions
- No latency requirements
- Single model, low traffic