# @nikx/dory-util

Shared utilities for the Dory web scraping platform. This package contains common code used across `dory-core`, `dory-api`, and `dory-ui` to maintain a single source of truth and reduce code duplication.

## Features

- **Unique Key Generation**: Deterministic hash generation for duplicate detection
- **Merged Output Format**: Standardized output structure with metadata and content
- **Type Definitions**: Shared TypeScript types across all Dory projects

## Installation

```bash
npm install @nikx/dory-util
```

## Usage

### Generate Unique Keys

```typescript
import { generateUniqueKey } from '@nikx/dory-util';

// Generate key from entire dataset item
const key1 = generateUniqueKey(datasetItem);

// Generate key from specific fields
const key2 = generateUniqueKey(datasetItem, ['url', 'title']);

// Generate key using request metadata
const key3 = generateUniqueKey(datasetItem, [], requestData);
```

### Build Merged Output

```typescript
import { buildMergedOutput, buildSimpleMergedOutput } from '@nikx/dory-util';

// With request queue data
const mergedOutput = buildMergedOutput(
  datasets,
  requestQueueData,
  ['url'] // optional uniqueKeySelector
);

// Simple format (without request queue)
const simpleOutput = buildSimpleMergedOutput(
  datasets,
  ['url'] // optional uniqueKeySelector
);
```

### Types

```typescript
import type { MergedOutputItem, RequestQueueData } from '@dory/util';

const output: MergedOutputItem = {
  meta: {
    contentType: 'text/html',
    uniqueKey: 'abc123...',
    sourceUrl: 'https://example.com',
  },
  content: {
    title: 'Example',
    // ... scraped data
  }
};
```

## Development

```bash
# Install dependencies
npm install

# Build the package
npm run build

# Watch for changes
npm run watch

# Clean build artifacts
npm run clean
```

## Output Format

The merged output format provides a consistent structure:

```typescript
{
  meta: {
    contentType: string;        // MIME type of the source
    uniqueKey: string;           // Content-based unique identifier
    sourceUrl: string | null;    // Original URL
    requestMethod?: string;      // HTTP method used
    loadedUrl?: string;          // Final URL after redirects
    label?: string;              // Request label from Crawlee
    requestDepth?: number;       // Crawl depth
    retryCount?: number;         // Number of retries
  },
  content: {
    // ... actual scraped data
  }
}
```

## License

ISC