# Agent Specification: Acquisition Agent

## 1. Agent Overview

| Attribute | Value |
|-----------|-------|
| **Name** | Acquisition Agent |
| **ID** | research-acquisition-agent |
| **Purpose** | Download research papers, extract metadata, validate FAIR compliance, and assign persistent identifiers |
| **Lifecycle Stage** | Acquisition (Stage 2 of Research Framework) |
| **Model** | sonnet |
| **Version** | 1.0.0 |
| **Status** | Draft |

### Description

The Acquisition Agent transforms discovery results into a structured research corpus. It downloads PDFs from open access sources, extracts or retrieves metadata, assigns REF-XXX persistent identifiers, computes SHA-256 checksums for integrity verification, and validates FAIR (Findable, Accessible, Interoperable, Reusable) compliance. The agent handles paywalled papers through manual upload workflows and integrates with shared research corpora to avoid duplicate downloads.

## 2. Capabilities

### Primary Capabilities

| Capability | Description | NFR Reference |
|------------|-------------|---------------|
| PDF Download | Download papers from Semantic Scholar, arXiv, publisher sites | NFR-RF-A-01 |
| Metadata Extraction | Extract title, authors, year, DOI from PDF or API | NFR-RF-A-02 |
| FAIR Validation | Score papers on Findable, Accessible, Interoperable, Reusable criteria | NFR-RF-A-09 |
| Checksum Computation | Generate SHA-256 hashes for integrity verification | NFR-RF-A-05 |
| REF-XXX Assignment | Assign sequential persistent identifiers | BR-RF-A-001 |
| Bulk Acquisition | Process multiple papers in parallel with rate limiting | NFR-RF-A-03 |

### Secondary Capabilities

| Capability | Description |
|------------|-------------|
| Manual Upload Handling | Accept user-provided PDFs for paywalled papers |
| Shared Corpus Deduplication | Create symlinks to existing papers in shared repository |
| Format Validation | Verify PDF format via magic bytes check |
| Progress Reporting | Real-time progress updates for bulk operations |

## 3. Tools

### Required Tools

| Tool | Purpose | Permission |
|------|---------|------------|
| Bash | Execute downloads, compute checksums | Execute |
| Read | Access acquisition queue, existing metadata | Read |
| Write | Save PDFs, metadata JSON, reports | Write |
| Glob | Find existing papers for deduplication | Read |
| Grep | Search metadata for duplicates | Read |

### External APIs

| API | Endpoint | Purpose | Auth |
|-----|----------|---------|------|
| Semantic Scholar | `api.semanticscholar.org` | Paper metadata, open access URLs | None |
| arXiv | `arxiv.org/pdf/` | Direct PDF download | None |
| CrossRef | `api.crossref.org` | DOI resolution | None |
| Unpaywall | `api.unpaywall.org` | Open access detection | Email header |

### System Tools

| Tool | Purpose |
|------|---------|
| `curl` / `wget` | HTTP downloads with resume support |
| `sha256sum` | Checksum computation |
| `file` | MIME type validation |
| `pdftotext` | PDF text extraction |

## 4. Triggers

### Automatic Triggers

| Trigger | Condition | Action |
|---------|-----------|--------|
| Discovery Complete | Acquisition queue populated (UC-RF-001) | Start bulk acquisition |
| Workflow Stage | UC-RF-008 initiates Stage 2 | Process workflow queue |

### Manual Triggers

| Trigger | Command | Description |
|---------|---------|-------------|
| Single Acquisition | `aiwg research acquire REF-XXX` | Acquire specific paper |
| Bulk from Queue | `aiwg research acquire --from-queue` | Process entire queue |
| Manual Upload | `aiwg research acquire --upload /path/to/file.pdf` | Add local PDF |
| Retry Failed | `aiwg research acquire --retry-failed` | Retry failed downloads |

### Event Triggers

| Event | Source | Action |
|-------|--------|--------|
| Paper Selected | Discovery Agent | Add to acquisition queue |
| Quality Gate Failed | Workflow Agent | Acquire additional sources |

## 5. Inputs/Outputs

### Inputs

| Input | Format | Source | Validation |
|-------|--------|--------|------------|
| Acquisition Queue | JSON | `.aiwg/research/discovery/acquisition-queue.json` | Valid paper IDs |
| Paper IDs | Array of strings | Command arguments | Valid IDs in queue |
| Manual PDF Path | File path | User input | File exists, valid PDF |
| Manual Metadata | JSON object | User input (if extraction fails) | Required fields present |

### Outputs

| Output | Format | Location | Retention |
|--------|--------|----------|-----------|
| PDF Files | Binary PDF | `.aiwg/research/sources/pdfs/{REF-XXX}-{slug}.pdf` | Permanent |
| Metadata JSON | JSON | `.aiwg/research/sources/metadata/{REF-XXX}-metadata.json` | Permanent |
| Acquisition Report | Markdown | `.aiwg/research/sources/acquisition-report-{timestamp}.md` | Permanent |
| Checksums | Text | `.aiwg/research/sources/checksums.txt` | Permanent |

### Output Schema: Metadata JSON

```json
{
  "ref_id": "REF-025",
  "title": "OAuth 2.0 Security Best Practices",
  "title_slug": "oauth-2-security-best-practices",
  "authors": [
    {"name": "Smith, John", "affiliation": "Stanford University"},
    {"name": "Doe, Jane", "affiliation": "MIT"}
  ],
  "year": 2023,
  "venue": "ACM Conference on Computer and Communications Security (CCS)",
  "venue_tier": "A*",
  "doi": "10.1145/3576915.3623456",
  "abstract": "This paper presents security best practices for OAuth 2.0...",
  "license": "CC-BY-4.0",
  "url": "https://www.semanticscholar.org/paper/abc123def456",
  "pdf_url": "https://arxiv.org/pdf/2301.12345.pdf",
  "citations": 42,
  "acquisition_timestamp": "2026-01-25T14:30:00Z",
  "acquisition_source": "semantic-scholar-api",
  "fair_score": {
    "findable": 90,
    "accessible": 100,
    "interoperable": 95,
    "reusable": 90,
    "overall": 94
  },
  "checksum_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "file_size_bytes": 2457600,
  "provenance": {
    "discovery_query": "OAuth2 security best practices",
    "discovery_timestamp": "2026-01-25T10:00:00Z",
    "selected_by": "user-manual-selection"
  }
}
```

## 6. Dependencies

### Agent Dependencies

| Agent | Relationship | Interaction |
|-------|--------------|-------------|
| Discovery Agent | Upstream | Receives acquisition queue |
| Documentation Agent | Downstream | Provides PDFs and metadata for UC-RF-003 |
| Workflow Agent | Orchestrator | Receives task assignments, reports completion |
| Provenance Agent | Observer | Logs all acquisition operations |

### Service Dependencies

| Service | Purpose | Fallback |
|---------|---------|----------|
| HTTP Downloads | Paper acquisition | Manual upload |
| Semantic Scholar API | Metadata retrieval | PDF extraction |
| File System | PDF and metadata storage | Abort if unavailable |

### Data Dependencies

| Data | Location | Required |
|------|----------|----------|
| Acquisition Queue | `.aiwg/research/discovery/acquisition-queue.json` | Yes |
| REF Counter | `.aiwg/research/sources/ref-counter.txt` | Yes (created if missing) |
| Shared Corpus | `/tmp/research-papers/sources/` | Optional |

## 7. Configuration Options

### Agent Configuration

```yaml
# .aiwg/research/config/research-acquisition-agent.yaml
acquisition_agent:
  # Download Configuration
  download:
    timeout_seconds: 60
    retry_attempts: 3
    retry_backoff_ms: [5000, 10000, 20000]
    concurrent_downloads: 5
    user_agent: "AIWG Research Framework/1.0 (research tool)"

  # Storage Configuration
  storage:
    pdf_directory: ".aiwg/research/sources/pdfs"
    metadata_directory: ".aiwg/research/sources/metadata"
    max_file_size_mb: 100
    permissions: "644"

  # FAIR Scoring Weights
  fair_scoring:
    findable:
      doi_present: 40
      metadata_complete: 10  # per field: title, authors, year, venue, abstract
    accessible:
      persistent_url: 50
      clear_license: 50
    interoperable:
      json_format: 50
      schema_compliance: 50
    reusable:
      license_permits_reuse: 50
      provenance_documented: 50

  # Shared Corpus Integration
  shared_corpus:
    enabled: true
    path: "/tmp/research-papers/sources"
    symlink_enabled: true

  # REF-XXX Format
  ref_format:
    prefix: "REF"
    digits: 3  # REF-001, REF-002, etc.
```

### Environment Variables

| Variable | Purpose | Default |
|----------|---------|---------|
| `AIWG_RESEARCH_DOWNLOAD_TIMEOUT` | Download timeout in seconds | 60 |
| `AIWG_RESEARCH_CONCURRENT_DOWNLOADS` | Max parallel downloads | 5 |
| `AIWG_RESEARCH_SHARED_CORPUS` | Path to shared paper repository | None |

## 8. Error Handling

### Error Categories

| Error Type | Severity | Handling Strategy |
|------------|----------|-------------------|
| Download Timeout | Warning | Retry 3x with backoff |
| 404 Not Found | Warning | Log, skip, continue with next |
| Invalid PDF | Warning | Flag for manual upload |
| Disk Full | Error | Abort, cleanup, notify user |
| Metadata Extraction Failed | Warning | Prompt for manual input |
| Checksum Mismatch | Error | Re-download and verify |

### Error Response Template

```json
{
  "error_code": "ACQUISITION_DOWNLOAD_FAILED",
  "severity": "warning",
  "paper_id": "abc123def456",
  "message": "Failed to download PDF: Network timeout after 60 seconds",
  "retry_count": 3,
  "remediation": "Provide manual PDF upload or skip this paper",
  "next_action": "Continue with remaining papers"
}
```

### Recovery Procedures

| Scenario | Procedure |
|----------|-----------|
| Partial acquisition failure | Save successful downloads, log failures, allow retry |
| Corrupted download | Delete partial file, retry from scratch |
| Metadata extraction failed | Use API metadata, prompt user if unavailable |
| Shared corpus unavailable | Fall back to local-only storage |

## 9. Metrics/Observability

### Performance Metrics

| Metric | Target | Measurement |
|--------|--------|-------------|
| Download time per paper | <60 seconds median | Timer from request to save |
| Metadata extraction time | <10 seconds | Timer for extraction |
| Bulk throughput | 5 concurrent downloads | Active downloads |
| Success rate | >90% | Successful / total attempted |

### Logging

| Log Level | Events |
|-----------|--------|
| INFO | Acquisition start, paper acquired, completion |
| DEBUG | Download progress, metadata extraction steps |
| WARNING | Retry triggered, FAIR score low, paywalled paper |
| ERROR | Download failed, disk error, validation failure |

### Telemetry

```json
{
  "event": "acquisition_complete",
  "timestamp": "2026-01-25T14:30:00Z",
  "metrics": {
    "papers_attempted": 25,
    "papers_acquired": 23,
    "papers_failed": 2,
    "total_size_mb": 115.5,
    "average_download_time_ms": 8500,
    "fair_score_average": 87
  }
}
```

## 10. Example Usage

### Single Paper Acquisition

```bash
# Acquire a specific paper from queue
aiwg research acquire REF-025

# Output:
# Acquiring REF-025: "OAuth 2.0 Security Best Practices"
# Downloading from: https://arxiv.org/pdf/2301.12345.pdf
# Download complete: 2.4 MB
# Validating PDF format... OK
# Extracting metadata... OK
# Computing SHA-256 checksum... OK
# FAIR validation: 94/100 (High)
# Saved: .aiwg/research/sources/pdfs/REF-025-oauth-2-security-best-practices.pdf
# Metadata: .aiwg/research/sources/metadata/REF-025-metadata.json
```

### Bulk Acquisition from Queue

```bash
# Acquire all papers in queue
aiwg research acquire --from-queue

# Output:
# Acquisition queue: 25 papers
# Processing 5 concurrent downloads...
# [1/25] REF-001: Downloading... OK (1.2 MB)
# [2/25] REF-002: Downloading... OK (3.4 MB)
# [3/25] REF-003: PAYWALLED - Manual upload required
# ...
# [25/25] REF-025: Downloading... OK (2.4 MB)
#
# Acquisition Summary:
# - Acquired: 23/25 (92%)
# - Paywalled: 2 (manual upload required)
# - Total size: 115.5 MB
# - Average FAIR score: 87/100
#
# Report saved: .aiwg/research/sources/acquisition-report-2026-01-25T14-30-00.md
```

### Manual PDF Upload

```bash
# Upload paywalled paper manually
aiwg research acquire --upload /tmp/oauth-paper.pdf --ref REF-003

# Output:
# Validating PDF format... OK
# Extracting metadata from PDF...
# - Title: "OAuth 2.0 Authorization Framework"
# - Authors: [auto-extracted]
# - Year: 2023
# Confirm metadata? (y/n/edit): y
# Assigning identifier: REF-003
# Computing checksum... OK
# FAIR validation: 72/100 (Moderate - missing license info)
# Saved: .aiwg/research/sources/pdfs/REF-003-oauth-2-authorization-framework.pdf
```

### Shared Corpus Deduplication

```bash
# Paper already in shared corpus
aiwg research acquire REF-042

# Output:
# Checking shared corpus at /tmp/research-papers/sources/...
# Match found: Paper already acquired (DOI: 10.1145/example)
# Create symlink to shared corpus? (y/n): y
# Symlink created: .aiwg/research/sources/pdfs/REF-042.pdf -> /tmp/research-papers/sources/abc123.pdf
# Reusing existing metadata (no re-download)
```

## 11. Related Use Cases

| Use Case | Relationship | Description |
|----------|--------------|-------------|
| UC-RF-002 | Primary | Acquire Research Source with FAIR Validation |
| UC-RF-001 | Upstream | Discover Research Papers (provides queue) |
| UC-RF-003 | Downstream | Document Research Paper (receives PDFs) |
| UC-RF-008 | Orchestrated | Execute Research Workflow (Stage 2) |

## 12. Implementation Notes

### Architecture Considerations

1. **Parallel Download Management**: Use worker pool for concurrent downloads
2. **Transactional Acquisition**: All-or-nothing per paper (no partial saves)
3. **Idempotent Operations**: Re-acquiring same paper updates metadata, doesn't duplicate
4. **Storage Efficiency**: Symlinks for shared corpus, deduplication by DOI

### Performance Optimizations

1. **Streaming Downloads**: Stream large files to avoid memory issues
2. **Parallel Checksums**: Compute checksum while downloading (stream hash)
3. **Batch Metadata Retrieval**: Query API for multiple papers in one request
4. **Resume Support**: Resume interrupted downloads when supported

### Security Considerations

1. **URL Validation**: Only download from whitelisted domains
2. **File Type Verification**: Magic bytes check, not just extension
3. **Checksum Verification**: Detect corrupted or tampered files
4. **Copyright Compliance**: Respect publisher terms, prioritize open access

### Testing Strategy

| Test Type | Coverage Target | Focus Areas |
|-----------|-----------------|-------------|
| Unit Tests | 80% | Metadata extraction, FAIR scoring, REF assignment |
| Integration Tests | 70% | Download handling, file I/O, API interaction |
| E2E Tests | Key workflows | Full acquisition from queue to storage |

### Known Limitations

1. **Paywalled Papers**: Cannot auto-download; require manual upload
2. **Rate Limits**: Some publishers block rapid downloads
3. **PDF Quality**: Scanned PDFs may have poor metadata extraction
4. **Large Files**: Papers >100MB may timeout on slow connections

---

## References

- @$AIWG_ROOT/agentic/code/frameworks/research-complete/elaboration/use-cases/UC-RF-002-acquire-research-source.md
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/inception/vision-document.md - Section 7.1 (Acquisition Management)
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/inception/initial-risk-assessment.md - T-04 (Copyright Compliance)
- [FAIR Principles](https://www.go-fair.org/fair-principles/)

---

## Document Metadata

**Version:** 1.0 (Draft)
**Status:** DRAFT - Awaiting Review
**Created:** 2026-01-25
**Last Updated:** 2026-01-25
**Owner:** Agent Designer (Research Framework Team)