---
title: Separate Processing and Data Access Layers
impact: HIGH
impactDescription: improves testability and maintainability of business logic
tags: architecture, data-access, layers, quality, python, pyspark
---

## Separate Processing and Data Access Layers

Mixing business logic with data storage code (SQL, API calls, file I/O) makes code hard to test and maintain. Separating these layers allows you to test business logic without a database or cluster.

**Incorrect (mixed logic and I/O):**

```python
def process_orders_and_save(spark, input_path, output_path):
    # Data Access + Logic mixed
    df = spark.read.parquet(input_path)
    
    # Business logic
    df = df.filter(col("amount") > 100)
    df = df.withColumn("tax", col("amount") * 0.1)
    
    # Data Access mixed
    df.write.mode("overwrite").parquet(output_path)
```

**Correct (separated layers):**

```python
# Processing Layer (Business Logic - Pure Spark transformations)
def calculate_tax(df):
    """Business logic that is easy to unit test with mock data"""
    return df.filter(col("amount") > 100) \
             .withColumn("tax", col("amount") * 0.1)

# Task/Service Layer (Orchestration)
def run_order_job(spark, config):
    # Data Access: Read
    raw_df = spark.read.parquet(config['input_path'])
    
    # Logic: Transform
    processed_df = calculate_tax(raw_df)
    
    # Data Access: Write
    processed_df.write.mode("overwrite").parquet(config['output_path'])
```

**Benefits:**
- **Unit Testability**: You can test `calculate_tax` with small, local DataFrames.
- **Flexibility**: You can change storage (Parquet to Delta) without touching business logic.
- **Readability**: The purpose of each function is clear.

**Tools:** Static analyzer, PR review