# Lost Keys

> **OSINT API key protection** - Prevent secret leaks before they cost you thousands

Lost Keys is a fast, intelligent CLI tool that prevents accidental commits of API keys, tokens, and other secrets by scanning your code **before** it reaches version control. Combining pattern-based detection, entropy analysis, and context-aware filtering, it catches secrets while minimizing false positives.

## ✨ Features

- 🔒 **22+ Secret Patterns** - Detects AWS keys, GitHub tokens, Stripe keys, database credentials, and more
- 🔒 **40+ Secret Patterns** - Detects AWS keys, GitHub tokens, Stripe keys, database credentials, and more
- 🎯 **Context-Aware Detection** - Reduces false positives by understanding comments, test files, and sample code
- ⚡ **Lightning Fast** - Scans 50 files in ~40ms with parallel processing
- 🎨 **Beautiful Error Messages** - Clear, actionable guidance with color-coded output
- 🔧 **Fully Configurable** - Customize confidence thresholds, whitelist patterns, and more
- 🎭 **Wildcard Whitelist** - Flexible pattern matching (`test-*`, `*-example`)
- 🪝 **Git Hook Integration** - Automatically blocks dangerous commits
- 📊 **Entropy Analysis** - Catches high-entropy secrets without known patterns
- 🚀 **Progress Indicator** - Visual feedback for large scans

## 📦 Installation

```bash
# Install globally
npm install -g lost-keys

# Or use with npx (no installation)
npx lost-keys init
```

**Requirements:** Node.js 18.0.0 or higher

## 🚀 Quick Start

Get up and running in 3 commands:

```bash
# 1. Initialize in your git repository
lost-keys init

# 2. Test it out
echo 'AWS_KEY=AKIAIOSFODNN7EXAMPLE' > test.txt
lost-keys scan --file test.txt

# 3. That's it! The pre-commit hook is now active
git add . && git commit -m "test"
```

## 📚 Usage

### Initialize Lost Keys

Set up Lost Keys in your repository (creates `.lost-keys.yml` and installs git hooks):

```bash
lost-keys init
```

This will:
- Create `.lost-keys.yml` configuration file
- Install pre-commit hook in `.git/hooks/`
- Display setup confirmation

### Scan Files

Scan files for secrets:

```bash
# Scan staged files (for pre-commit hook)
lost-keys scan --staged

# Scan specific file
lost-keys scan --file config.ts

# Scan and block on detection
lost-keys scan --staged --block
```

**Options:**
- `--staged` - Scan only git staged files (use in pre-commit hooks)
- `--block` - Exit with error code if high-confidence secrets found
- `--file <path>` - Scan a specific file

### Manage Whitelist

Handle false positives by whitelisting patterns:

```bash
# Add pattern (supports wildcards)
lost-keys whitelist add "test-key-*"
lost-keys whitelist add "AKIAIOSFODNN7EXAMPLE"

# List all whitelisted patterns
lost-keys whitelist list

# Remove pattern
lost-keys whitelist remove "test-key-*"

# Clear all patterns (requires confirmation)
lost-keys whitelist clear
lost-keys whitelist clear --yes  # Skip confirmation
```

**Wildcard Support:**
- `test-*` - Match anything starting with "test-"
- `*-example` - Match anything ending with "-example"
- `test-*-key` - Match with wildcard in the middle
- `test-?` - Match single character

## ⚙️ Configuration

Lost Keys is configured via `.lost-keys.yml` in your repository root:

```yaml
# Enable/disable scanning
enabled: true

# Block commits when high-confidence secrets are detected
block_on_detection: true

# Minimum confidence threshold to block (0-100)
# High: 90%+, Medium: 60-89%, Low: <60%
confidence_threshold: 90

# Whitelist patterns and files
whitelist:
  files:
    - .env.example
    - .env.sample
    - .env.template
    - '*.sample'
    - '*.example'
  patterns:
    - test-key-*
    - EXAMPLE_*

# Advanced settings
settings:
  # Skip files larger than this (in KB)
  max_file_size_kb: 1024

  # File extensions to skip
  excluded_extensions:
    - .png
    - .jpg
    - .pdf
    - .zip
```

### Configuration Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `enabled` | boolean | `true` | Enable/disable scanning |
| `block_on_detection` | boolean | `true` | Block commits on secret detection |
| `confidence_threshold` | number | `90` | Minimum confidence to block (0-100) |
| `whitelist.files` | string[] | See above | File patterns to skip |
| `whitelist.patterns` | string[] | `[]` | String patterns to ignore |
| `settings.max_file_size_kb` | number | `1024` | Max file size to scan |
| `settings.excluded_extensions` | string[] | See above | File extensions to skip |

## 🔍 Supported Secret Types

Lost Keys detects 22+ types of secrets across 9 categories:

### Cloud Providers
| Pattern | Example | Confidence |
|---------|---------|------------|
| AWS Access Key | `AKIAIOSFODNN7EXAMPLE` | 95% |
| AWS Secret Key | `wJalrXUtnFEMI/K7MDENG...` | 85% |
| GCP API Key | `AIzaSyD-9tSrke72PouQMnMX-a7...` | 90% |

### Version Control
| Pattern | Example | Confidence |
|---------|---------|------------|
| GitHub Personal Access Token (Classic) | `ghp_1234567890abcdef...` | 100% |
| GitHub Personal Access Token (Fine-grained) | `github_pat_11A...` | 100% |

### Payment Processing
| Pattern | Example | Confidence |
|---------|---------|------------|
| Stripe Secret Key | `sk_live_1234567890abcdef...` | 100% |
| Stripe Restricted Key | `rk_live_1234567890abcdef...` | 100% |

### Communication
| Pattern | Example | Confidence |
|---------|---------|------------|
| Slack Token | `xoxb-1234567890-...` | 95% |
| Slack Webhook | `https://hooks.slack.com/services/...` | 100% |
| SendGrid API Key | `SG.abcdefgh...` | 95% |
| Twilio Account SID | `ACabcdef1234567890...` | 90% |
| Twilio Auth Token | `abcdef1234567890...` | 85% |

### AI Services
| Pattern | Example | Confidence |
|---------|---------|------------|
| OpenAI API Key | `sk-1234567890abcdef...` | 95% |
| Anthropic API Key | `sk-ant-api...` | 95% |

### Databases
| Pattern | Example | Confidence |
|---------|---------|------------|
| PostgreSQL Connection String | `postgresql://user:pass@host...` | 90% |
| MySQL Connection String | `mysql://user:pass@host...` | 90% |
| MongoDB Connection String | `mongodb://user:pass@host...` | 90% |

### Authentication
| Pattern | Example | Confidence |
|---------|---------|------------|
| JWT Token | `eyJhbGciOiJIUzI1NiIsInR5cCI6...` | 85% |
| OAuth Client Secret | `oauth_secret=abc123...` | 80% |
| Bearer Token | `Authorization: Bearer eyJ...` | 85% |

### Cryptography
| Pattern | Example | Confidence |
|---------|---------|------------|
| SSH Private Key | `-----BEGIN RSA PRIVATE KEY-----` | 100% |

### Generic
| Pattern | Example | Confidence |
|---------|---------|------------|
| Generic API Key | `api_key=abcdef123...` | 70% |
| High-Entropy String | Any 40+ char string with high randomness | 75-85% |

## 🎨 Output Examples

### Successful Scan
```
ℹ Scanning 10 file(s)...
✓ No secrets detected
```

### Secret Detected
```
❌ Lost Keys: Secrets detected in staged files

src/config.ts:12
│
│ AWS Access Key detected
│ Found: AKIAIOSFO...
│ Confidence: HIGH (95%)
│
│ Remediation:
│ Rotate this key immediately in AWS IAM Console:
│ 1. Go to IAM → Users → [User] → Security Credentials
│ 2. Click "Make inactive" for the exposed key
│ 3. Create a new key and update your application
│
│ If this is a false positive:
│ Add to .lost-keys.yml:
│
│   whitelist:
│     patterns:
│       - "AKIAIOSFODNN7EXAMPL"  # AWS Access Key

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

❌ Found 1 secret(s) in 1 file(s)

🚫 COMMIT BLOCKED
```

### Progress Indicator
```
Scanning files... 25/50 (50%)
```

## 🔧 Troubleshooting

### False Positives

If Lost Keys detects a non-secret (like a test key or example), whitelist it:

```bash
# Whitelist specific pattern
lost-keys whitelist add "test-key-12345"

# Whitelist with wildcard
lost-keys whitelist add "test-*"
```

Or add to `.lost-keys.yml`:

```yaml
whitelist:
  patterns:
    - test-key-*
    - EXAMPLE_*
```

### Pre-commit Hook Not Running

1. **Check if hook is installed:**
   ```bash
   ls -la .git/hooks/pre-commit
   ```

2. **Reinstall hook:**
   ```bash
   lost-keys init
   ```

3. **Verify hook is executable:**
   ```bash
   chmod +x .git/hooks/pre-commit
   ```

### Slow Scans

Lost Keys is optimized for speed (50 files in ~40ms), but if scans are slow:

1. **Check file sizes** - Files >1MB are skipped by default
2. **Increase threshold:**
   ```yaml
   settings:
     max_file_size_kb: 2048
   ```
3. **Skip large directories** - Add to `.gitignore`

### High Memory Usage

If scanning very large repositories:

1. **Reduce concurrency** (modify scanner.ts)
2. **Increase file size limit** to skip more files
3. **Scan in batches** instead of all at once

## ❓ FAQ

### How does Lost Keys compare to other tools?

| Feature | Lost Keys | TruffleHog | GitGuardian |
|---------|-----------|------------|-------------|
| Speed | ⚡ 50 files in 40ms | 🐌 Slower | 🐌 Slower |
| False Positives | ✅ Context-aware | ❌ Many | ✅ Good |
| Whitelist | ✅ Wildcards | ❌ Limited | ✅ Yes |
| Local-only | ✅ Yes | ✅ Yes | ❌ Cloud-based |
| Price | ✅ Free (MIT) | ✅ Free | ❌ $$ |

### Does it send my code anywhere?

**No.** Lost Keys runs 100% locally on your machine. No code, secrets, or data is ever transmitted.

### What happens if I commit a secret?

The pre-commit hook will:
1. Scan staged files
2. Display detected secrets with guidance
3. Block the commit (return exit code 1)
4. Provide remediation steps

You can then:
- Remove the secret and commit again
- Whitelist it if it's a false positive
- Disable blocking temporarily (not recommended)

### Can I use this in CI/CD?

Yes! Run Lost Keys in your CI pipeline:

```yaml
# GitHub Actions example
- name: Scan for secrets
  run: |
    npm install -g lost-keys
    lost-keys scan --staged
```

### How accurate is the detection?

Lost Keys uses a multi-layered approach:
- **Pattern matching** (85-100% confidence)
- **Entropy analysis** (75-85% confidence)
- **Context awareness** (reduces false positives by 60%)

High-confidence detections (90%+) are very accurate. Medium/low confidence may need review.

### Can I customize the patterns?

Currently, patterns are built-in. Custom pattern support is planned for a future release. For now, use the whitelist to handle edge cases.

### Does it work offline?

Yes! Lost Keys is fully offline and doesn't require internet connectivity.

## 🏗️ Development

```bash
# Clone repository
git clone https://github.com/hilltopventuregroup/lost-keys.git
cd lost-keys

# Install dependencies
npm install

# Build
npm run build

# Run tests
npm test

# Run tests with coverage
npm test:coverage

# Lint
npm run lint

# Format code
npm run format
```

### Project Structure

```
lost-keys/
├── src/
│   ├── cli/              # CLI commands
│   ├── config/           # Configuration management
│   ├── detection/        # Pattern matching and scanning
│   ├── git/              # Git integration
│   ├── patterns/         # Secret detection patterns
│   └── utils/            # Utility functions
├── tests/                # Test suites
├── docs/                 # Documentation
└── .lost-keys.yml        # Configuration file
```

## 🤝 Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

**Areas we'd love help with:**
- Additional secret patterns
- Language-specific context detection
- Performance optimizations
- Documentation improvements

## 📄 License

MIT © [Hilltop Venture Group](https://github.com/hilltopventuregroup)

## 🔗 Links

- [GitHub Repository](https://github.com/hilltopventuregroup/lost-keys)
- [Issue Tracker](https://github.com/hilltopventuregroup/lost-keys/issues)
- [Changelog](CHANGELOG.md)

## 💡 Tips & Best Practices

1. **Run `lost-keys init` in every repository** - Set it up once, protected forever
2. **Review whitelist regularly** - Remove patterns you no longer need
3. **Use wildcards wisely** - `test-*` is better than listing every test key
4. **Keep confidence threshold at 90** - Balances security and false positives
5. **Commit `.lost-keys.yml`** - Share configuration with your team
6. **Don't whitelist real secrets** - Rotate them instead!

---

Made with ❤️ by [Hilltop Venture Group](https://github.com/hilltopventuregroup)

**Star us on GitHub** ⭐ if Lost Keys saves you from a secret leak!