# Contributing to DataQL

Thank you for your interest in contributing to DataQL! This document provides guidelines and best practices for contributing to the project.

## Development Workflow

### Prerequisites

- Go 1.21 or later
- Docker and Docker Compose (for e2e tests)
- Make (for build automation)

### Getting Started

1. Clone the repository:
   ```bash
   git clone https://github.com/adrianolaselva/dataql.git
   cd dataql
   ```

2. Install dependencies:
   ```bash
   make mod-download
   ```

3. Build the project:
   ```bash
   make build
   ```

### Running Tests

#### Unit Tests

Run unit tests with:
```bash
make test
```

For test coverage:
```bash
make coverage
```

#### Coverage ratchet (never regress)

Total coverage must never drop below the committed baseline in
`.coverage-baseline`. CI enforces this on every PR. The baseline only ever
moves up, on the path to the >= 90% target.

```bash
make coverage-check   # fail if coverage < baseline (what CI runs)
make coverage-bump    # raise the baseline after you add tests
```

If you add tests that raise coverage, run `make coverage-bump` and commit the
updated `.coverage-baseline` so the gain is locked in.

#### E2E Tests (Integration Tests)

E2E tests are **critical** for ensuring DataQL works correctly with all supported data sources. They must be run locally before submitting any PR.

**Starting the E2E Environment:**
```bash
# Start all infrastructure (PostgreSQL, MySQL, MongoDB, Kafka, etc.)
make e2e-up

# Wait for services to be healthy
make e2e-wait
```

**Running E2E Tests:**
```bash
# Run all shell-based e2e tests (recommended)
make e2e-test-scripts

# Or run Go-based integration tests
make e2e-test
```

**Running Specific Test Suites:**
```bash
# From project root
make e2e-test-postgres   # PostgreSQL tests (26 tests)
make e2e-test-mysql      # MySQL tests (26 tests)
make e2e-test-mongodb    # MongoDB tests (20+ tests)
make e2e-test-kafka      # Kafka tests (10+ tests)
make e2e-test-s3         # S3 tests (13 tests)
make e2e-test-sqs        # SQS tests (16 tests)

# Or from e2e directory
cd e2e
make test-postgres
make test-s3
make test-sqs
```

**Stopping the E2E Environment:**
```bash
make e2e-down
```

## E2E Testing Requirements

### When to Run E2E Tests

**E2E tests MUST be run locally:**

1. **Before submitting any Pull Request** - All tests must pass
2. **After implementing a new feature** - Add tests for the new functionality
3. **After fixing a bug** - Verify the fix doesn't break existing functionality
4. **After modifying query processing** - Ensure all data sources work correctly
5. **After updating dependencies** - Verify compatibility

### Adding New E2E Tests

When adding new functionality, create corresponding e2e tests:

1. Add test cases to the appropriate `e2e/tests/test-*.sh` file
2. Follow the existing test structure:
   ```bash
   test_your_new_feature() {
       log_info "Test: Description of what you're testing"
       result=$($DATAQL_BIN run -q "YOUR QUERY" -f "$URL" 2>&1)
       if echo "$result" | grep -q "expected_output"; then
           log_pass "Your test description"
       else
           log_fail "Your test description" "$result"
       fi
   }
   ```
3. Add the test function call in the appropriate section
4. Run the full test suite to ensure no regressions

### E2E Test Structure

```
e2e/
├── docker-compose.yaml    # Infrastructure configuration
├── .env                   # Environment variables
├── Makefile               # Test commands
├── README.md              # E2E documentation
├── scripts/               # Initialization scripts
│   ├── init-postgres.sql  # PostgreSQL schema and data
│   ├── init-mysql.sql     # MySQL schema and data
│   ├── init-mongodb.js    # MongoDB collections and data
│   └── init-localstack.sh # S3 buckets, SQS queues, DynamoDB tables
└── tests/                 # Test scripts
    ├── test-all.sh        # Run all test suites
    ├── test-postgres.sh   # PostgreSQL tests (26 tests)
    ├── test-mysql.sh      # MySQL tests (26 tests)
    ├── test-mongodb.sh    # MongoDB tests (20+ tests)
    ├── test-kafka.sh      # Kafka tests (10+ tests)
    ├── test-s3.sh         # S3 tests (13 tests) - CSV, JSON, JSONL from LocalStack
    └── test-sqs.sh        # SQS tests (16 tests) - Message queue reading from LocalStack
```

### Test Coverage by Data Source

| Source | Tests | Coverage |
|--------|-------|----------|
| PostgreSQL | 26 | SELECT, WHERE, ORDER BY, LIMIT, aggregates, exports |
| MySQL | 26 | SELECT, WHERE, ORDER BY, LIMIT, aggregates, exports |
| MongoDB | 20+ | Collections, queries, filters, exports |
| Kafka | 10+ | Peek mode, message parsing, exports |
| S3 | 13 | File reading (CSV, JSON, JSONL), queries, exports |
| SQS | 16 | Message reading, filtering, aggregation, exports |

## Adding a remote data source

Remote object-store / URL sources (S3, GCS, Azure Blob, HTTP) implement the
`source.Resolver` contract (`pkg/source`):

```go
type Resolver interface {
    // Replace the URIs this source owns with local file paths; pass everything
    // else through unchanged.
    ResolveFiles(files []string) ([]string, error)
    // Remove any temp files this resolver created.
    Cleanup() error
}
```

The core engine (`internal/dataql`) runs a slice of `source.Resolver`s in order,
so a new source is added by:

1. Creating a handler package that implements `ResolveFiles` + `Cleanup`
   (download the URI to a temp file, return its local path).
2. Registering it in the `remoteResolvers` slice in `internal/dataql/dataql.go`.
3. Adding unit tests (URL parsing, config/env, error paths — no live backend)
   and, where an emulator exists, an E2E script under `e2e/tests/`.

No other engine change is required. Support a local emulator via env (as GCS does
with `STORAGE_EMULATOR_HOST` and S3 with `AWS_ENDPOINT_URL`) so it can be
E2E-tested without real credentials.

## Pull Request Process

### Before Submitting

1. **Run linter:**
   ```bash
   make lint
   ```

2. **Run unit tests:**
   ```bash
   make test
   ```

3. **Run E2E tests:**
   ```bash
   make e2e-up
   make e2e-wait
   make e2e-test-scripts
   make e2e-down
   ```

4. **Verify all test suites pass:**
   - PostgreSQL (26 tests)
   - MySQL (26 tests)
   - MongoDB (20+ tests)
   - Kafka (10+ tests)
   - S3 (13 tests)
   - SQS (16 tests)

5. **Update documentation** if needed

### Quality gates (CI)

Every PR runs these gates. They are designed so quality only ever ratchets up:

| Gate | What it enforces | Run locally |
|------|------------------|-------------|
| Build | `go build ./...` compiles | `make build` |
| Tests | unit/integration tests pass (`-race`) | `make test` |
| Coverage ratchet | total coverage never drops below `.coverage-baseline` | `make coverage-check` |
| Lint | `golangci-lint` (govet, staticcheck SA*, errcheck, …) — blocking | `make lint` |
| Vulnerability scan | `govulncheck` finds no called-path vulnerabilities — blocking | `go run golang.org/x/vuln/cmd/govulncheck@latest ./...` |
| Offline self-sufficiency | binary runs with the network disabled (no runtime downloads) | `scripts/smoke-offline.sh` |
| Security (gosec) | blocking; inherent-rule exclusions documented in `scripts/gosec.sh` | `scripts/gosec.sh` |
| E2E | advisory; full suite against emulated backends | `make e2e` |

When you add tests that raise coverage, run `make coverage-bump` and commit the
updated `.coverage-baseline`.

### PR Requirements

- All CI checks must pass
- E2E tests must have been run locally (include confirmation in PR description)
- Code follows existing patterns and style
- New features have corresponding tests
- New features preserve the self-contained invariant (embedded, offline; see AGENTS.md)
- Commit messages are clear and descriptive (Conventional Commits)

### PR Description Template

```markdown
## Summary
Brief description of changes

## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update

## E2E Testing
- [ ] I have run `make e2e-test` locally
- [ ] All e2e tests pass
- [ ] New tests added for new functionality (if applicable)

## Test Results
<paste summary of test results here>
```

## Code Style

- Follow standard Go conventions
- Use `gofmt` for formatting
- Run `golangci-lint` before committing
- Keep functions focused and well-documented
- Write clear commit messages

## Reporting Issues

When reporting bugs, please include:

1. DataQL version (`dataql --version`)
2. Go version (`go version`)
3. Operating system
4. Steps to reproduce
5. Expected vs actual behavior
6. Relevant error messages

## Questions?

If you have questions about contributing, please open an issue with the "question" label.

---

Thank you for contributing to DataQL!