# Pipeline Debugging Checklist

A systematic approach to debugging GitLab CI/CD pipeline failures.

## Initial Assessment

- [ ] **Failure Type**: Identify if it's a job, stage, or pipeline failure
- [ ] **Failure Pattern**: Check if this is a recurring issue
- [ ] **Recent Changes**: Review commits since last successful pipeline
- [ ] **Environment**: Confirm which environment/branch is affected
- [ ] **Urgency Level**: Determine if this blocks production/releases

## Gather Information

- [ ] **Job Logs**: Download and review complete job logs
- [ ] **Pipeline Graph**: Check job dependencies and execution order
- [ ] **Artifacts**: Verify if required artifacts were produced
- [ ] **Previous Runs**: Compare with last successful pipeline
- [ ] **System Status**: Check GitLab runner status and availability

## Common Failure Points

### 🔧 Build Failures

- [ ] **Dependencies**: Package versions changed or unavailable
- [ ] **Build Tools**: Compiler/interpreter version mismatches
- [ ] **Environment Variables**: Missing or incorrect env vars
- [ ] **File Permissions**: Build scripts have execute permissions
- [ ] **Disk Space**: Sufficient space on runners

### 🧪 Test Failures

- [ ] **Test Data**: Database seeds or fixtures corrupted
- [ ] **External Services**: APIs or services unavailable
- [ ] **Timing Issues**: Race conditions or timeouts
- [ ] **Test Isolation**: Tests affecting each other
- [ ] **Environment Differences**: Dev vs CI environment disparities

### 🚀 Deployment Failures

- [ ] **Credentials**: Deploy keys or tokens expired
- [ ] **Target Environment**: Server accessible and healthy
- [ ] **Resource Limits**: Memory/CPU constraints
- [ ] **Network Issues**: Firewall or connectivity problems
- [ ] **Configuration**: Deploy scripts and configs valid

### 📦 Docker/Container Issues

- [ ] **Base Images**: Images available and not corrupted
- [ ] **Registry Access**: Docker registry accessible
- [ ] **Layer Caching**: Cache corruption or invalidation
- [ ] **Resource Limits**: Container memory/CPU limits
- [ ] **Security Scanning**: Image vulnerabilities blocking

## Investigation Tools

### GitLab CLI Commands

```bash
# View pipeline status
glab ci list --per-page 10

# Get specific pipeline details
glab ci view <pipeline-id>

# Download job logs
glab ci trace <job-id> > job.log

# Check runner status
glab ci status
```

### Debugging Techniques

- [ ] **Add Debug Output**: Insert echo/print statements
- [ ] **Enable Verbose Mode**: Add -v or --debug flags
- [ ] **Simplify Pipeline**: Comment out non-essential jobs
- [ ] **Local Reproduction**: Run failing commands locally
- [ ] **Binary Search**: Bisect commits to find breaking change

## Root Cause Analysis

- [ ] **Error Messages**: Parse and understand exact error
- [ ] **Stack Traces**: Identify failing code locations
- [ ] **Timing Analysis**: Check when failure started
- [ ] **Dependency Tree**: Trace upstream causes
- [ ] **Environmental Factors**: External service issues

## Common Solutions

### Quick Fixes

- [ ] **Retry Job**: For transient failures
- [ ] **Clear Cache**: When cache corruption suspected
- [ ] **Update Dependencies**: Lock or update versions
- [ ] **Increase Timeouts**: For slow operations
- [ ] **Fix Permissions**: Chmod scripts and files

### Systematic Fixes

- [ ] **Pin Versions**: Lock all dependency versions
- [ ] **Add Retries**: Implement retry logic for flaky operations
- [ ] **Improve Logging**: Add more diagnostic output
- [ ] **Mock Services**: Reduce external dependencies
- [ ] **Parallelize Jobs**: Reduce overall pipeline time

## Prevention Measures

- [ ] **Add Tests**: Cover the failure scenario
- [ ] **Monitor Metrics**: Track failure rates
- [ ] **Document Issue**: Update runbooks
- [ ] **Share Knowledge**: Inform team of resolution
- [ ] **Review Process**: Assess if process changes needed

## Advanced Debugging

### Performance Issues

- [ ] **Profile Jobs**: Identify slow operations
- [ ] **Optimize Caching**: Improve cache hit rates
- [ ] **Parallel Execution**: Split large jobs
- [ ] **Resource Allocation**: Adjust runner specs
- [ ] **Network Optimization**: Reduce data transfers

### Security Failures

- [ ] **Scan Results**: Review security scan outputs
- [ ] **Policy Violations**: Check compliance rules
- [ ] **Certificate Issues**: Verify SSL/TLS certs
- [ ] **Access Controls**: Confirm permissions
- [ ] **Audit Logs**: Review security events

## Documentation

- [ ] **Record Solution**: Document fix in team wiki
- [ ] **Update Runbook**: Add to troubleshooting guide
- [ ] **Create Issue**: Track systematic improvements
- [ ] **Post-Mortem**: For significant failures
- [ ] **Knowledge Sharing**: Present findings to team

---

**Debug Session Info**:

- Pipeline ID: \***\*\_\_\_\*\***
- Job ID: \***\*\_\_\_\*\***
- Failure Time: \***\*\_\_\_\*\***
- Debugger: \***\*\_\_\_\*\***
- Time to Resolution: \***\*\_\_\_\*\***

**Root Cause**:
_Describe the actual cause of the failure_

**Solution Applied**:
_Document the fix that resolved the issue_

**Follow-up Actions**:
_List any improvements or preventive measures to implement_