---
name: research-dev-environment
description: >-
  Development tooling for research projects including virtual environments, dependency management, GPU setup, and data
  access configuration
topics:
  - research
  - dev-environment
  - dependencies
  - virtual-env
  - gpu
  - data-access
  - tooling
volatility: evolving
last-reviewed: 2026-06-17
version-pin: null
sources:
  - url: https://docs.astral.sh/uv/
    hash: sha256:d0e9dc3efa09273d3490b7d73ccfb1fc243f2401ebaf27c9a4f286e180a19322
    retrieved: 2026-06-17
  - url: https://the-turing-way.netlify.app/reproducible-research/renv.html
    hash: sha256:b4a72a41bcbfd72fe708629f4f1982074f7c9c714311d8f01901d2b0f882a91a
    retrieved: 2026-06-17
---

Research dev environments have stricter reproducibility requirements than typical application development. A trading strategy that produces different results on a different machine is useless -- the environment itself is a variable that must be controlled. At the same time, research environments need flexibility for rapid iteration: installing new packages, switching between CPU and GPU, and accessing large datasets must be frictionless.

## Summary

Use `uv` (preferred) or `pip` with pinned dependencies in a virtual environment for reproducible Python dependency management. Lock the full dependency tree (not just direct dependencies). Configure GPU access and CUDA versions explicitly when applicable. Set up data access credentials via environment variables (never in code or config files). Use a Makefile with standard targets (`setup`, `run`, `test`) so that both humans and agents can operate the environment identically.

## Deep Guidance

### Python Environment Setup

**Recommended: `uv` for dependency management**

`uv` is the fastest Python package manager and provides deterministic resolution with a lockfile:

```bash
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a new project
uv init research-project
cd research-project

# Add dependencies
uv add numpy pandas scikit-learn optuna
uv add --dev pytest ruff mypy

# The lockfile (uv.lock) is auto-generated and pinned
# Commit both pyproject.toml and uv.lock
```

```toml
# pyproject.toml
[project]
name = "research-project"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
    "numpy>=1.26",
    "pandas>=2.2",
    "scikit-learn>=1.4",
    "optuna>=3.5",
    "pyyaml>=6.0",
    "structlog>=24.1",
]

[project.optional-dependencies]
gpu = ["torch>=2.2"]
tracking = ["mlflow>=2.11"]
notebooks = ["jupyter>=1.0", "papermill>=2.5"]

[tool.uv]
dev-dependencies = [
    "pytest>=8.0",
    "ruff>=0.3",
    "mypy>=1.9",
]
```

**Alternative: `pip` with `requirements.txt`**

If `uv` is not available, use `pip` with fully pinned requirements:

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

```
# requirements.txt — fully pinned (generated by pip freeze)
numpy==1.26.4
pandas==2.2.1
scikit-learn==1.4.1.post1
optuna==3.5.0
PyYAML==6.0.1
structlog==24.1.0
```

**Alternative: `conda` for complex native dependencies**

Use conda when the project requires system-level native libraries (CUDA toolkit, MKL, OpenBLAS) that pip cannot manage:

```yaml
# environment.yml
name: research
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.11
  - numpy=1.26
  - pandas=2.2
  - scikit-learn=1.4
  - cudatoolkit=12.1  # Native dependency
  - pip:
    - optuna==3.5.0   # pip packages within conda env
```

### GPU Configuration

For research projects that use GPU acceleration (ML model training, simulation, etc.):

```python
# src/gpu.py
import os
import logging

logger = logging.getLogger(__name__)

def configure_gpu(config: dict) -> str:
    """Configure GPU access. Returns device string."""
    if not config.get("gpu", {}).get("enabled", False):
        logger.info("GPU disabled by config, using CPU")
        return "cpu"

    # Restrict visible GPUs (useful for multi-GPU machines)
    gpu_ids = config.get("gpu", {}).get("device_ids", [0])
    os.environ["CUDA_VISIBLE_DEVICES"] = ",".join(str(i) for i in gpu_ids)

    try:
        import torch
        if torch.cuda.is_available():
            device = f"cuda:{gpu_ids[0]}"
            logger.info(
                "GPU configured: %s (%s, %.1f GB)",
                device,
                torch.cuda.get_device_name(0),
                torch.cuda.get_device_properties(0).total_mem / 1e9,
            )
            return device
        else:
            logger.warning("CUDA not available, falling back to CPU")
            return "cpu"
    except ImportError:
        logger.warning("PyTorch not installed, using CPU")
        return "cpu"
```

**GPU config in YAML**:
```yaml
gpu:
  enabled: true
  device_ids: [0]
  memory_fraction: 0.8  # Limit GPU memory usage
```

### Data Access Configuration

Data credentials are managed via environment variables, never committed to git:

```bash
# .env (gitignored)
DATA_SOURCE_PATH=/mnt/data/research
DATABASE_URL=postgresql://user:pass@host:5432/research
AWS_PROFILE=research-data
POLYGON_API_KEY=pk_xxx  # Market data API
```

```python
# src/data/credentials.py
import os
from dataclasses import dataclass

@dataclass
class DataCredentials:
    """Data access credentials loaded from environment."""
    data_path: str
    database_url: str | None = None
    api_key: str | None = None

    @classmethod
    def from_env(cls) -> "DataCredentials":
        data_path = os.environ.get("DATA_SOURCE_PATH", "data/raw")
        if not os.path.exists(data_path):
            raise EnvironmentError(
                f"DATA_SOURCE_PATH={data_path} does not exist. "
                "Set DATA_SOURCE_PATH to your data directory."
            )
        return cls(
            data_path=data_path,
            database_url=os.environ.get("DATABASE_URL"),
            api_key=os.environ.get("POLYGON_API_KEY"),
        )
```

### Makefile for Environment Management

```makefile
.PHONY: setup run test lint clean

PYTHON ?= python3
UV := $(shell command -v uv 2>/dev/null)

setup: ## Set up development environment
ifdef UV
	uv sync
	uv sync --group dev
else
	$(PYTHON) -m venv .venv
	.venv/bin/pip install -r requirements.txt
	.venv/bin/pip install -r requirements-dev.txt
endif
	@echo "Environment ready. Activate with: source .venv/bin/activate"

setup-gpu: setup ## Set up with GPU dependencies
ifdef UV
	uv sync --extra gpu
else
	.venv/bin/pip install -r requirements-gpu.txt
endif

run: ## Run experiment (usage: make run CONFIG=configs/exp-001.yml)
	$(PYTHON) -m src.runner.experiment_runner --config $(CONFIG)

test: ## Run test suite
	$(PYTHON) -m pytest tests/ -v --tb=short

lint: ## Lint and type-check
	ruff check src/ tests/
	mypy src/ --ignore-missing-imports

clean: ## Clean generated artifacts
	rm -rf .venv/ __pycache__/ .mypy_cache/ .pytest_cache/
	find . -name '*.pyc' -delete
```

### IDE Configuration

**VS Code** (`.vscode/settings.json`):
```json
{
    "python.defaultInterpreterPath": ".venv/bin/python",
    "python.analysis.typeCheckingMode": "basic",
    "editor.formatOnSave": true,
    "[python]": {
        "editor.defaultFormatter": "charliermarsh.ruff"
    },
    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": ["tests/"]
}
```

### Environment Verification Script

Run this at the start of every experiment to verify the environment:

```python
# scripts/verify_env.py
"""Verify that the research environment is correctly configured."""
import sys
import importlib

REQUIRED_PACKAGES = [
    "numpy", "pandas", "sklearn", "optuna", "yaml", "structlog",
]

def verify():
    errors = []

    # Python version
    if sys.version_info < (3, 11):
        errors.append(f"Python >= 3.11 required, got {sys.version}")

    # Required packages
    for pkg in REQUIRED_PACKAGES:
        try:
            importlib.import_module(pkg)
        except ImportError:
            errors.append(f"Missing required package: {pkg}")

    # Data access
    import os
    data_path = os.environ.get("DATA_SOURCE_PATH", "data/raw")
    if not os.path.exists(data_path):
        errors.append(f"Data path not found: {data_path}")

    if errors:
        print("Environment verification FAILED:")
        for e in errors:
            print(f"  - {e}")
        sys.exit(1)
    else:
        print("Environment verification passed.")

if __name__ == "__main__":
    verify()
```

### Dependency Update Strategy

Research projects should update dependencies cautiously:

1. **Lock everything**: Both direct and transitive dependencies are pinned.
2. **Update on a schedule**: Not every commit. Weekly or per-milestone.
3. **Test after update**: Run the full test suite after any dependency change.
4. **Document breaking changes**: If a dependency update changes experiment results, document which runs were affected.
5. **Separate experiment deps from infra deps**: Changing the plotting library should not affect experiment reproducibility. Use optional dependency groups.