# fawkes

This project is used to centralize indexability logic for our Search
microservices and Sitemap Generator. The main purpose is to completely remove
discrepancies between pages indexed by Search and page URLs stored in Sitemap.

Link to documentation: https://studyportals.atlassian.net/wiki/x/BwAdvw

## Base structure

- organisations-seo (used by Organisation Search)
- scholarships-seo (used by Scholarship Search)
- sitemap-generator-seo (used by Sitemap Generator)
- src
  - common
  - organisations
  - scholarships
  - sitemap-generator

# Automatic Documentation System

## Overview

This system automatically generates and maintains documentation for Fawkes
indexing rules across multiple projects. It creates a hierarchical structure of
Confluence pages that document all indexing policies and rules, ensuring that
the documentation is always up-to-date with the actual codebase.

Link to documentation:
[Fawkes Indexing Rules Documentation](https://studyportals.atlassian.net/wiki/spaces/T/pages/3494969354/Fawkes+Indexing+Rules+Documentation)

## Features

- **Dynamic Documentation**: Automatically extracts indexing rules and policies
  directly from the codebase
- **Multi-Page Structure**: Creates a main index page with separate pages for
  each project
- **Hierarchical Organization**: Structures content as Projects → Policies →
  Rules
- **Automatic Page Creation**: Intelligently creates or updates pages as needed
- **Cross-Page Navigation**: Provides intuitive navigation between all pages

## Architecture

The system follows SOLID design principles to ensure maintainability and
extensibility:

- **Single Responsibility**: Each class has one specific responsibility
- **Open/Closed**: Easily extendable without modifying existing code
- **Liskov Substitution**: Interchangeable components through well-defined
  interfaces
- **Interface Segregation**: Clean, focused interfaces
- **Dependency Inversion**: High-level modules depend on abstractions

### Key Components

1. **Content Builder**: Extracts project, policy, and rule data from the Fawkes
   system
2. **Policy Fetchers**: Project-specific components that fetch and format policy
   data
3. **Document Builder**: Generates structured HTML content for Confluence pages
4. **Confluence Updater**: Handles creation and updating of Confluence pages
5. **Token Retriever**: Securely retrieves API token for Confluence access

## How It Works

1. The system starts by collecting all project data using the `ContentBuilder`
2. For each project, it extracts policies and rules using project-specific
   fetchers
3. The `DocumentBuilder` generates the content for the main index page and each
   project page
4. The `ConfluenceUpdater` creates or updates pages in Confluence as needed
5. Each project page links back to the main index, and the main index links to
   all project pages

## Setup and Configuration

### Environment Variables

The system requires the following environment variables:

```
CONFLUENCE_EMAIL=your-email@example.com
CONFLUENCE_API_TOKEN=your-api-token
CONFLUENCE_BASE_URL=https://your-instance.atlassian.net
CONFLUENCE_SPACE_KEY=YOUR_SPACE_KEY
CONFLUENCE_MAIN_PAGE_TITLE=Fawkes Indexing Rules Documentation
```

> **Important Note on Credentials**: Currently, the system uses personal
> credentials (email and API token) for Confluence access. This is a temporary
> solution and should be replaced with a team or service account in the future
> to avoid dependency on a specific individual's account.

### Deployment and Triggering

The system runs as an AWS Lambda function and is automatically triggered as part
of the CI/CD pipeline. This ensures documentation is always updated whenever new
code is deployed:

```yaml
# buildspec.yml (excerpt)
post_build:
  commands:
    - |
      if [ -z "$PR_ID" ]; then
        LAMBDA_NAME="FawkesConfluenceUpdater-$STAGE"
        echo "Invoking Lambda function $LAMBDA_NAME"
        echo '{"source":"cicd-pipeline","timestamp":"'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'"}' > payload.json
        aws lambda invoke --function-name $LAMBDA_NAME --invocation-type Event --payload fileb://payload.json response.json || echo "Lambda invocation failed, but continuing"
      fi
```

**Trigger Flow:**

1. When code is pushed to the `main` branch, the CI/CD pipeline is triggered
2. After successful deployment of a new Fawkes version, the post-build phase
   executes
3. The Lambda function is invoked with a simple payload containing the source
   and timestamp
4. The Lambda function authenticates with Confluence and executes the
   documentation generation process
5. This only happens for production deployments (not PR builds)

This automated approach ensures documentation stays in sync with the actual
codebase without manual intervention.

## Scalability and Extension

### Adding New Projects

To add documentation for a new project:

1. Create a new class that implements the `IPolicyFetcher` interface
2. Implement the `getProjectName()` and `fetchPolicies()` methods
3. Register the new fetcher in the `ContentBuilder` constructor

Example:

```typescript
export class NewProjectPolicyFetcher implements IPolicyFetcher {
  getProjectName(): string {
    return 'New Project Name';
  }

  fetchPolicies(): IPolicy[] {
    // Implement policy fetching logic
    return [...policies];
  }
}

// Then update ContentBuilder:
constructor(fetchers?: IPolicyFetcher[]) {
  this.policyFetchers = fetchers || [
    // Existing fetchers
    new NewProjectPolicyFetcher()
  ];
}
```

### Customizing Page Content

The `DocumentBuilder` class contains methods for generating page content. To
customize the format:

1. Modify the `buildMainDocument()` method for changes to the main index page
2. Modify the `buildProjectDocument()` method for changes to project pages
3. Modify the `buildPolicySection()` and `buildRulesTable()` methods for changes
   to policy and rule sections

## Conclusion

This documentation system ensures that Fawkes indexing rules are always
well-documented and accessible to the team. By automatically generating
documentation directly from the codebase, it eliminates the risk of
documentation becoming outdated or inconsistent with the actual implementation.