# @souscheflabs/ml-vision

[![npm version](https://img.shields.io/npm/v/@souscheflabs/ml-vision.svg)](https://www.npmjs.com/package/@souscheflabs/ml-vision)
[![license](https://img.shields.io/npm/l/@souscheflabs/ml-vision.svg)](https://github.com/SousChefLabs/ml-vision/blob/main/LICENSE)
[![TypeScript](https://img.shields.io/badge/TypeScript-5.0-blue.svg)](https://www.typescriptlang.org/)
[![React Native](https://img.shields.io/badge/React%20Native-0.73+-green.svg)](https://reactnative.dev/)

ML-powered product detection for React Native. Speed up kitchen inventory onboarding with multi-barcode scanning, receipt OCR, and visual product recognition.

## Features

- **Multi-Barcode Scanning** - Scan multiple barcodes from a single photo
- **Receipt OCR** - Photograph grocery receipts to auto-extract items
- **Visual Product Recognition** - Recognize fridge/pantry contents by appearance
- **Video Scanning** - Real-time product detection from camera feed
- **On-Device ML** - Fast inference using TFLite with GPU acceleration
- **Server Fallback** - Optional server-side processing for better accuracy

## Installation

```bash
npm install @souscheflabs/ml-vision
# or
yarn add @souscheflabs/ml-vision
```

### Peer Dependencies

This package requires the following peer dependencies:

```bash
npm install react-native-vision-camera react-native-mmkv
```

For on-device ML inference (Phase 3+):
```bash
npm install react-native-fast-tflite
```

### iOS Setup

```bash
cd ios && pod install
```

Add camera permissions to `Info.plist`:
```xml
<key>NSCameraUsageDescription</key>
<string>Camera is used to scan products and receipts</string>
```

### Android Setup

Add camera permission to `AndroidManifest.xml`:
```xml
<uses-permission android:name="android.permission.CAMERA" />
```

## Quick Start

### 1. Wrap your app with the provider

```typescript
import { MLVisionProvider } from '@souscheflabs/ml-vision';
import { MMKV } from 'react-native-mmkv';

// Create MMKV instance for caching
const storage = new MMKV({ id: 'ml-vision-cache' });

function App() {
  return (
    <MLVisionProvider
      config={{
        serverUrl: 'http://192.168.1.100:8000', // Optional: your ML server
        cacheEnabled: true,
      }}
      storage={storage}
    >
      <YourApp />
    </MLVisionProvider>
  );
}
```

### 2. Use the hooks in your components

```typescript
import { useMultiBarcodeScanner } from '@souscheflabs/ml-vision';
import { Camera } from 'react-native-vision-camera';

function BarcodeScanScreen() {
  const {
    results,
    isScanning,
    startScanning,
    stopScanning,
    frameProcessor,
  } = useMultiBarcodeScanner({
    formats: ['ean-13', 'upc-a', 'qr'],
    maxBarcodes: 20,
    onDetected: (barcodes) => {
      console.log('Found barcodes:', barcodes);
    },
  });

  return (
    <Camera
      device={device}
      isActive={isScanning}
      frameProcessor={frameProcessor}
    />
  );
}
```

## API Reference

### MLVisionProvider

The context provider that must wrap your app.

```typescript
<MLVisionProvider
  config={{
    serverUrl?: string;        // ML server URL for fallback
    serverTimeout?: number;    // Request timeout (default: 5000ms)
    cacheEnabled?: boolean;    // Enable result caching (default: true)
    cacheTTL?: number;         // Cache TTL in ms (default: 24 hours)
    enableGPUDelegate?: boolean; // Use GPU for inference (default: true)
  }}
  storage={mmkvInstance}       // MMKV instance for caching
  onInitialized={() => {}}     // Called when ready
  onError={(error) => {}}      // Called on init error
>
```

### useMultiBarcodeScanner

Scan multiple barcodes from camera frames or photos.

```typescript
const {
  // State
  isReady: boolean;
  isScanning: boolean;
  results: BarcodeDetectionResult[];
  error: Error | null;

  // Actions
  startScanning: () => void;
  stopScanning: () => void;
  scanPhoto: (uri: string) => Promise<BarcodeDetectionResult[]>;
  clearResults: () => void;

  // For VisionCamera
  frameProcessor: FrameProcessor;
} = useMultiBarcodeScanner(options);
```

### useProductDetector

Detect products in images using trained ML models.

```typescript
const {
  // State
  isModelLoaded: boolean;
  isDetecting: boolean;
  detections: ProductDetectionResult[];
  modelVersion: string;

  // Actions
  detectProducts: (uri: string) => Promise<ProductDetectionResult[]>;
  updateModel: () => Promise<void>;

  // For VisionCamera
  frameProcessor: FrameProcessor;
} = useProductDetector({
  model: 'fast' | 'accurate';
  minConfidence?: number;
  serverFallback?: boolean;
});
```

### useReceiptScanner

Extract items from receipt photos.

```typescript
const {
  // State
  isProcessing: boolean;
  items: ReceiptItem[];
  confidence: number;

  // Actions
  scanReceipt: (uri: string) => Promise<ReceiptScanResult>;
  confirmItem: (item: ReceiptItem) => void;
  rejectItem: (item: ReceiptItem) => void;
} = useReceiptScanner({
  serverFallback?: boolean;
  minConfidence?: number;
});
```

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                    Your React Native App                        │
├─────────────────────────────────────────────────────────────────┤
│  @souscheflabs/ml-vision                                            │
│  ├── Hooks (useMultiBarcodeScanner, useProductDetector, etc.)  │
│  ├── Frame Processors (barcode, OCR, TFLite detection)         │
│  └── CacheManager (MMKV-based caching)                         │
├─────────────────────────────────────────────────────────────────┤
│  On-Device Processing (fast, offline)                           │
│  ├── MLKit (barcodes, text recognition)                        │
│  ├── react-native-fast-tflite (custom models)                  │
│  └── CoreML/NNAPI delegates (GPU acceleration)                 │
├─────────────────────────────────────────────────────────────────┤
│  Server Fallback (accurate, requires network)                   │
│  ├── FastAPI inference server                                   │
│  ├── YOLOv8 full models                                        │
│  └── PaddleOCR for complex receipts                            │
└─────────────────────────────────────────────────────────────────┘
```

## Training Your Own Models

See the [Training Guide](../../docker/README.md) for instructions on:
- Setting up the Docker training environment
- Preparing your dataset
- Training YOLOv8 models
- Exporting to TFLite for mobile

## Project Structure

```
packages/ml-vision/
├── src/
│   ├── index.ts              # Main exports
│   ├── types/                # TypeScript definitions
│   ├── hooks/                # React hooks
│   ├── core/                 # Provider, cache, server client
│   ├── processors/           # Frame processors
│   └── utils/                # Utilities
├── models/                   # Bundled TFLite models
├── docs/                     # Documentation
└── dist/                     # Compiled output
```

## Development

```bash
# Build the package
npm run build

# Watch for changes
npm run watch

# Type check
npm run typecheck
```

## License

ISC

## Contributing

1. Fork the repository
2. Create your feature branch
3. Make your changes
4. Run tests and type checks
5. Submit a pull request

## Troubleshooting

### Model Loading Fails on Android

Ensure the model file is in `android/app/src/main/assets/`. Metro bundler handles this automatically for `.tflite` files in the `models/` directory.

### Camera Permission Denied

Add the camera permission to your app and request it at runtime:

```typescript
import { Camera } from 'react-native-vision-camera';

const permission = await Camera.requestCameraPermission();
if (permission === 'denied') {
  // Handle permission denied
}
```

### Low Detection Accuracy

1. Ensure good lighting conditions
2. Try lowering `minConfidence` threshold
3. Enable `serverFallback` for better accuracy on complex scenes

### Build Errors with TFLite

If you encounter build errors with `react-native-fast-tflite`:

```bash
# iOS: Clean and rebuild
cd ios && pod deintegrate && pod install && cd ..

# Android: Clean gradle cache
cd android && ./gradlew clean && cd ..
```

## Support

- [GitHub Issues](https://github.com/SousChefLabs/ml-vision/issues)
- [Documentation](./docs/)
- [Changelog](./CHANGELOG.md)