# gh-workspace-data

Keep project-related files outside the project repository without keeping them outside the working environment. The companion [Workspace Data extension](https://github.com/SorinGFS/codium-workspace-data) integrates this workflow with VSCodium and VS Code through their native Source Control and diff interfaces.

Projects often accumulate optional tests, benchmarks, fixtures, notes, research, experiments, extended documentation, and contributor tools that should not live in the package repository. `gh-workspace-data` stores that material in separate GitHub repositories and materializes only the current project's files under `#/`.

The materialized files are ordinary files. Editors, terminals, and project tools can use them normally, while the generated namespace remains excluded from the target project's Git and npm contents.

## At a glance

For a project whose Git `origin` identifies `github.com/acme/widget`, the default sources are:

| Visibility | Default repository | Intended use |
| --- | --- | --- |
| Public | `acme/public-data` | Collaborative data associated with the project |
| Private | `<authenticated-user>/private-data` | Personal or restricted data associated with the project |

Data is grouped first by an open-ended **concern** such as `tests`, `benchmarks`, `notes`, or `examples`:

```text
Repository storage                                      Project workspace
acme/public-data/tests/github.com/acme/widget/...       #/public/tests/...
alice/private-data/notes/github.com/acme/widget/...     #/private/notes/...
```

Similar public and private concerns never overlap in the workspace. GitHub repository visibility and access settings remain responsible for confidentiality.

The normal workflow is:

```sh
gh workspace-data init
gh workspace-data load
# Edit ordinary files below #/public or #/private.
gh workspace-data publish
```

`publish` opens or updates pull requests; it never pushes changes directly to a data repository's default branch.

## Install and get started

### Requirements

- GitHub CLI authenticated with `gh auth login`
- Git
- Node.js 20 or newer
- A target project hosted on GitHub

Install the extension for the current user:

```sh
gh extension install SorinGFS/gh-workspace-data
```

From any directory inside a target project, initialize and load its data:

```sh
gh workspace-data init
gh workspace-data load
```

`init` reserves the generated `#/` namespace, enforces its ignore policy, and installs `#/version-layers.js`. It does not add project-local wrappers or scripts.

`load` derives the project identity from Git `origin`, discovers every matching public and private concern, and uses sparse checkout to select the current project's directories from each available data repository. It then replaces each available visibility with its selected remote snapshot. **Loading discards unpublished local changes under `#/public` and `#/private`.** A missing public or private repository is skipped, so either visibility can be used independently.

Before publishing to a data repository, create it on GitHub with the intended visibility and at least one initial commit. Then add ordinary files beneath a visibility and concern:

```text
#/public/examples/example.json
#/private/notes/design.md
```

Publish all changed concerns:

```sh
gh workspace-data publish
```

## Loading and publication

1. Run `load` to replace public and private project data with the selected remote snapshots.
2. Add, edit, move, or delete files under `#/public` and `#/private`.
3. Run `publish` to create or update separate pull requests for changed public and private data.
4. Review and merge those pull requests, or use `publish --merge-owned` where appropriate.
5. Run `load` after a manual or deferred merge to refresh immediately, or let the next publication begin from the merged default branch.

Deleting loaded files or complete concerns and publishing intentionally deletes their corresponding data-repository content. Publication stops when workspace changes cannot be applied to the current destination without conflict. Loading does not reconcile local changes; it intentionally replaces them with remote data.

<details>
<summary><strong>Automatic merging for repositories you own</strong></summary>

For repositories owned by the authenticated GitHub user, publication can merge qualifying pull requests and reload successfully merged data in one command:

```sh
gh workspace-data publish --merge-owned
```

This option remains review-first for organization-owned repositories, repositories owned by another user, and fork-based contributions. Before merging, it verifies that the pull request is open and that its repository, branches, and head commit match the publication produced by the current invocation. It uses an ordinary squash merge without administrator bypass.

If checks, branch protection, a merge queue, or another GitHub condition prevents an immediate merge, the pull request remains open and reload is deferred. Successfully merged data is still reloaded when another publication remains deferred or open for review.

</details>

<details>
<summary><strong>Publishing again without an intervening load</strong></summary>

After merging manually, refreshing with `gh workspace-data load` is recommended but optional. If editing continues without loading, the next `publish` starts a new pull-request cycle from the latest default branch and applies only changes made after the previous publication. Conflicting default-branch edits still stop publication for reconciliation.

</details>

## Read-only inspection protocol

Editor integrations can inspect workspace changes against the exact loaded baseline without reproducing repository mapping, authentication, or Git logic.

<details>
<summary><strong>Inspection protocol version 1 commands, response format, and state</strong></summary>

The read-only capability and inspection commands are:

```sh
gh workspace-data capabilities --json
gh workspace-data status --json
gh workspace-data show --protocol 1 --visibility public --revision 0123456789abcdef0123456789abcdef01234567 --path tests/fixture.json
```

`capabilities --json` reports the installed package version, supported inspection protocol versions, and Load behavior without requiring a project, authentication, or network access:

```json
{
  "command": "gh workspace-data",
  "version": "0.8.0",
  "inspectionProtocolVersions": [1],
  "loadBehavior": "replace"
}
```

Integrations should determine compatibility from the reported protocol and behavior fields rather than from the informational package version. A command that is missing or does not report the required capabilities is incompatible with that integration.

`status --json` is local-only: it does not contact GitHub, alter the workspace, refresh remotes, or compare with a newer remote revision. It compares ordinary files under `#/public` and `#/private` with the exact baselines recorded by the last successful `load` or `publish`. Its single-line JSON result has this shape:

```json
{
  "protocolVersion": 1,
  "projectIdentity": "github.com/acme/widget",
  "state": "ready",
  "repositories": {
    "public": {
      "availability": "available",
      "baselineRevision": "0123456789abcdef0123456789abcdef01234567",
      "pullRequest": null
    },
    "private": {
      "availability": "missing",
      "baselineRevision": null,
      "pullRequest": null
    }
  },
  "changes": [
    {
      "id": "public:tests/fixture.json",
      "visibility": "public",
      "status": "modified",
      "path": "tests/fixture.json",
      "workspacePath": "#/public/tests/fixture.json",
      "baseline": { "available": true, "size": 128 }
    }
  ]
}
```

`state` is one of:

- `ready`: `changes` contains deterministic `added`, `modified`, and `deleted` entries.
- `notInitialized`: no synchronization state exists; `repositories` and `changes` are empty.
- `reloadRequired`: legacy state lacks the baseline inventory; run `gh workspace-data load` once to upgrade it.

For an added file, `baseline.available` is `false`; modified and deleted files have a loaded baseline. Changes are ordered by public/private visibility and then portable data path. Unknown protocol arguments, malformed state, unsafe filesystem objects, and other inspection failures exit with status 1 and write the diagnostic to standard error. Valid status states exit with status 0.

`show` writes the exact baseline bytes for one path to standard output. The visibility and concern-relative `--path` must identify an entry returned by the loaded baseline, and `--revision` must equal that visibility's reported `baselineRevision`; added files cannot be shown. The command retrieves the recorded Git blob through the authenticated GitHub CLI, verifies its size and SHA-256 digest against local state, and emits no bytes if the revision changed or verification fails. It never selects a newer commit. Consumers should treat the output as binary and include the reported baseline revision in any cached or virtual-document URI.

Synchronization state version 2 records repository identity, the repository containing each baseline object, source path, Git blob ID, SHA-256 digest, size, and file mode. This generated inventory is the authority for inspection and is refreshed atomically with materialized data. Version 1 remains accepted by load and publish, but inspection reports `reloadRequired` until a successful load creates version 2 state.

</details>

### VSCodium and VS Code integration

The companion [Workspace Data extension](https://github.com/SorinGFS/codium-workspace-data) consumes this protocol and presents public and private changes through the editor's native Source Control and diff interfaces. It checks `capabilities --json` on activation and offers an explicit install-or-upgrade action when this prerequisite is missing or incompatible. Before Load or publication, it checks the active GitHub CLI authentication and links to authentication setup when needed. Repository mapping, credentials, baseline retrieval, loading, and publication remain owned by GitHub CLI and `gh-workspace-data`; the editor extension does not create another Git repository under `#/` or compare with a newer remote revision.

Install the VSIX from the companion extension's [latest GitHub release](https://github.com/SorinGFS/codium-workspace-data/releases/latest), reload the editor window, and open the Source Control view. If the repository selector is hidden, reveal the **Repositories** view and select **Workspace Data**. The provider intentionally has no commit input because **Workspace Data: Publish** publishes through pull requests rather than Git commits. Its Load action checks current status and requests confirmation before overwriting unpublished changes.

See the companion extension's README for its complete installation and usage workflow.

## Repository selection

The defaults contain no hardcoded user identity:

```text
public:  <target-project-owner>/public-data
private: <authenticated-GitHub-user>/private-data
```

For a contributor working on another owner's project, public changes are proposed to the project owner's data repository through a branch or fork. Private data remains associated with the contributor's own repository.

A missing repository does not prevent the other visibility from working. Local files targeting a missing repository are preserved, but publication for that visibility is blocked until the repository exists and is accessible.

<details>
<summary><strong>Repository overrides</strong></summary>

Use environment variables when repository names do not follow the defaults:

```sh
WORKSPACE_DATA_PUBLIC_REPOSITORY=organization/shared-data gh workspace-data load
WORKSPACE_DATA_PRIVATE_REPOSITORY=user/personal-data gh workspace-data load
```

Use the same overrides for subsequent `load` and `publish` operations.

</details>

## Versioned data folders

Optionally, stored data can be organized into versioned layers. When a project is initialized, the extension provides the `#/version-layers.js` helper. Consumer scripts can use its output to select stored version layers.

A concern can keep all data at its root, divide it into semantic-version folders, or combine both:

```text
#/public/tests/
  index.js                 # Base data: always available
  index.json
  v15.1/                   # Version layer introduced at 15.1.0
    0/                     # Unnamed (numeric) test suite
    1/                     # Unnamed (numeric) test suite
  v16.0/                   # Version layer introduced at 16.0.0
    0/                     # Unnamed (numeric) test suite
  v17.0.2/                 # Complete version layer
    0/                     # Unnamed (numeric) test suite
    regression/            # Named test suite

#/private/benchmarks/
  index.js                 # Private concerns use the same layout
  v17/
    experimental-scenario/
```

The extension synchronizes these directories as ordinary data. It does not decide which version layers a test runner, benchmark, or other consumer should execute. Consumers opt into selection by importing the generated `#/version-layers.js` helper and calling `discoverVersionLayers`.

<details>
<summary><strong>Selection modes</strong></summary>

`discoverVersionLayers(root, packageVersion)` supports two policies:

| Policy | Invocation | Meaning |
| --- | --- | --- |
| Exact scope | `discoverVersionLayers(root, version)` | Base data, matching major and major/minor folders, plus eligible complete versions from the same major |
| Backwards compatible | `discoverVersionLayers(root, version, { backwardsCompatible: true })` | Base data plus every semantic-version layer whose introduction point is not newer than the package |

For package version `1.2.3`, exact-scope discovery includes:

```text
.          base layer
v1         matching major
v1.2       matching major/minor
v1.0.0     complete version in major 1 and not newer than 1.2.3
v1.1.3     complete version in major 1 and not newer than 1.2.3
v1.2.3     exact complete version
```

It excludes `v1.1` because partial minor layers must match the package minor, `v1.2.4` because it is newer, and every `v2` layer because it belongs to another major.

With `backwardsCompatible: true`, every valid layer at or before `1.2.3` is eligible across major boundaries. Partial folders are introduction points: `v1` means `1.0.0`, and `v1.2` means `1.2.0`. Layers run in ascending semantic order. Equal introduction points run from least to most specific, for example `v1`, `v1.0`, then `v1.0.0`.

Use cumulative discovery only when older expectations remain valid for the newer package. It supports delta-only folders, but it must not be used merely to make older data visible when that data describes an incompatible contract.

<details>
<summary><strong>Dispatcher configuration and mixed concerns</strong></summary>

A public test dispatcher can expose cumulative fixture selection in `#/public/tests/index.json`:

```json
{
  "callback": "isIdnHostname",
  "backwardsCompatible": true
}
```

This JSON file is a dispatcher convention; `gh-workspace-data` does not interpret it. The dispatcher decides how the setting applies. For example, a test dispatcher may accumulate numeric callback fixtures while retaining exact-scope selection for explicit conformance suites.

The same helper and policies are available to private concerns. Visibility changes where data is stored and materialized, not how semantic versions are compared.

Directory names are version selectors only when they match:

```text
v<major>
v<major>.<minor>
v<major>.<minor>.<patch>
```

Components use non-negative decimal integers without leading zeroes except `0`. Package prerelease and build versions use their numeric `major.minor.patch` core.

</details>

</details>

<details>
<summary><strong>Generated runtime support</strong></summary>

`#/version-layers.js` is extension-owned infrastructure shared by public and private consumers. From `#/public/<concern>/index.js` or `#/private/<concern>/index.js`, import it with:

```js
const {
  compareNames,
  compareNumericNames,
  discoverConcernEntryPoints,
  discoverNumberedJsonFixtures,
  discoverVersionLayers,
  discoverVersionLayerSets,
  readDirectories,
  selectVersionLayers,
  versionPattern
} = require('../../version-layers.js');
```

The comparators provide locale-independent lexical ordering and arbitrary-size numeric ordering. `readDirectories` returns direct ordinary child directories, and `versionPattern` identifies supported version-layer names. `discoverVersionLayers` retains the original exact-or-cumulative selection API.

</details>

<details>
<summary><strong>Traversal descriptors</strong></summary>

Consumers that apply exact and cumulative policies to different content can discover both sets once:

```js
const layerSets = discoverVersionLayerSets(root, packageVersion);
const fixtureLayers = selectVersionLayers(layerSets, {
  backwardsCompatible: configuration.backwardsCompatible ?? false
});
const suiteLayers = layerSets.exact;
```

For a package at `17.0.2` with `v15.1`, `v16.0`, `v17.0`, and `v18.0` directories, `layerSets` contains:

```js
{
  exact: ['.', 'v17.0'],
  cumulative: ['.', 'v15.1', 'v16.0', 'v17.0']
}
```

The actual arrays contain `{ name, root }` layer descriptors. The example shows their names for brevity.

`discoverNumberedJsonFixtures(fixtureLayers)` flattens numeric suite directories and numbered JSON files into deterministic descriptors:

```js
{
  layer: 'v15.1',
  suite: '0',
  file: '2.json',
  id: 'v15.1/0/2.json',
  path: '/project/#/public/tests/v15.1/0/2.json'
}
```

It preserves supplied layer order, then sorts suites and files numerically. Every discovered numeric suite must contain at least one numbered JSON fixture. The function does not read or interpret JSON content.

`discoverConcernEntryPoints(suiteLayers)` finds ordinary `index.js` files in explicit nonnumeric, nonversion concern directories:

```js
{
  layer: 'v17.0',
  concern: 'idna-test-v2',
  id: 'v17.0/idna-test-v2',
  root: '/project/#/public/tests/v17.0/idna-test-v2',
  entryPoint: '/project/#/public/tests/v17.0/idna-test-v2/index.js'
}
```

It preserves supplied layer order and sorts concerns lexically. It returns descriptors without loading entry points, leaving configuration, package API injection, assertions, and execution under consumer control.

The helper is never published to either data repository. The `init`, `load`, and `publish` commands restore its canonical installed bytes before continuing, so it should not be edited manually.

<details>
<summary><strong>Why the helper is outside public and private concerns</strong></summary>

Public and private dispatchers may need identical version-selection behavior. Keeping one generated helper at `#/version-layers.js` avoids copying runtime code into both data repositories and prevents those copies from drifting. CI that checks out data directly must materialize the helper separately from the matching `gh-workspace-data` version.

</details>

</details>

## Workspace safety

- `#/` is generated workspace state and is excluded from the target Git repository.
- Existing `.npmignore` files also exclude `/#/`; an absent `.npmignore` remains absent so npm continues using `.gitignore`.
- The extension materializes only ordinary files and directories and rejects filesystem links encountered in workspace data.
- `#/.data-state.json` tracks exact source repositories, revisions, baseline files, and open pull requests; do not edit it manually.
- Root-level generated runtime support is excluded from public and private publication.
- Losing synchronization state blocks publication rather than risking unintended remote deletion.

## Compatibility and verification

Automated tests cover Windows, macOS, and Linux on Node.js 20, 22, and 24. The matrix exercises the CLI entry point, overwrite-style loading, read-only status and baseline retrieval, baseline integrity checks, ignore-policy handling, generated runtime support, exact and backwards-compatible version ordering, owned-pull-request merge qualification, deferred reload behavior, publication history, workspace replacement, and rollback.

Run syntax validation and all 38 isolated tests with:

```sh
npm run check
npm test
```

The tests use temporary repositories and injected GitHub operations; they do not publish or merge live workspace data. Authenticated GitHub API behavior remains supplied by the installed GitHub CLI.

## Upgrade

```sh
gh extension upgrade workspace-data
```

## License

MIT
