# Benchling Webhook Integration for Quilt

The Benchling Webhook creates a seamless connection between [Benchling](https://www.benchling.com)'s Electronic Lab Notebook (ELN) and [Quilt](https://www.quilt.bio)'s Scientific Data Managements System (SDMS) for Amazon S3.
It not only allows you to view Benchling metadata and attachments inside Quilt packages, but also enables users to browse Quilt package descriptions from inside Benchling notebookes.

The webhook works through a [Benchling App](https://docs.benchling.com/docs/getting-started-benchling-apps) that must be installed in your Organization by a Benchling Administrator and configured to call your stack's unique webhook (see Installation, below).

## Availability

It is available in the Quilt Platform (1.65 or later) or as a standalone CDK stack via the `@quiltdata/benchling-webhook` [npm package](https://www.npmjs.com/package/@quiltdata/benchling-webhook).

## Functionality

### Auto-Packaging

![Packaged Notebook](imgs/benchling-package.png)

When scientists create notebook entries in Benchling, this webhook automatically:

- **Creates a dedicated Quilt package** for each notebook entry
- **Synchronizes metadata** from Benchling (experiment IDs, authors, etc.) into that package
- **Copies attachments** from that notebook into Amazon S3 as part of the package.
- **Enables orgnizational data discovery** by making contents available in ElasticSearch, and metadata available in Amazon Athena.

#### Package Naming

Each notebook package is named `{pkg_prefix}/{display_id}`, e.g. `benchling/EXP00000001`.
Set the prefix with the setup wizard's **Package S3 prefix** prompt (`packages.prefix` in the profile config).

The prefix can contain `{dotted.path}` placeholders, filled from the Benchling entry as saved in `entry_data.json`.
To put each scientist's notebook packages in the same namespace as their raw-data packages, use the entry creator's handle:

| `pkg_prefix` | Package name |
|---|---|
| `benchling` (default) | `benchling/EXP00000001` |
| `{creator.handle}` | `jdoe/EXP00000001` |
| `{creator.handle}`, creator `ernest.prabhakar` | `ernest-prabhakar/EXP00000001` |

- The S3 export folder follows the same name (`jdoe/EXP00000001/`).
- The name is computed each time, so a package moves only if its creator changes their Benchling handle.
- Quilt package names only allow letters, digits, `_` and `-`, so any other character in a filled-in value (such as the `.` in `ernest.prabhakar`) becomes `-`. Literal text in the prefix is used as is.
- If a placeholder has no value in the entry, that entry is not packaged. Its Canvas says "This entry can't be packaged" and names the missing value, and the webhook logs an error naming the placeholder and the entry. There is no fallback prefix.
- Existing packages are not moved when the prefix changes. An entry's Canvas looks for its package under the new name, which is created the next time the entry is packaged.
- If the bucket's Quilt workflow sets a `handle_pattern`, it must accept the new namespaces.

#### Turning Automatic Packaging Off

By default every supported entry event (`v2.entry.created`, `v2.entry.updated.fields`, `v2.entry.updated.reviewRecord`) and every new Canvas queues a packaging run.
To package entries only when a scientist asks for it, answer **No** to the setup wizard's "Package entries automatically on every entry event?" prompt.
That stores `packages.autoPackaging: false` in the profile config and syncs it to the webhook secret as `auto_packaging: "false"`; you can also edit either one directly (`npx @quiltdata/benchling-webhook sync-secrets` pushes a profile change to the secret).

With automatic packaging off:

- `v2.entry.created` and `v2.entry.updated.fields` events are acknowledged and skipped; nothing is queued.
- `v2.entry.updated.reviewRecord` events still run, because accepting a review seals the package and reopening the review unseals it (see [Accepted Reviews](#accepted-reviews)). Accepting a review therefore still packages the entry.
- A new Canvas shows the entry's current package if one exists, otherwise "Package Not Created", together with the linked packages, and its footer shows when it was drawn. It never waits on "Updating...".
- **Update Package** on the Canvas still queues a packaging run, so it is the only other way an entry is packaged.

The secret is re-read within about a minute, so the switch takes effect without a redeploy.
Packages already created stay where they are. Turning it back on packages an entry the next time Benchling sends an event for it.

#### Package Metadata

The package metadata is the package's `entry.json`. Besides the entry's IDs, name, creator, authors, and timestamps, it includes the entry's schema `fields` and `customFields`. Benchling keys these by display name, so the webhook re-keys them in snake_case: "Experiment Type" becomes `experiment_type` and "ELN-ID #" becomes `eln_id`. Each key maps to the field's `displayValue`, the text the scientist sees in Benchling: a dropdown option's or linked entity's name, not its ID. Custom fields have no `displayValue`, so they map to their `value`. Both keys are always present, as `{}` when the entry has none. The package's `entry_data.json` keeps the raw entry, with each field's display name, `value`, `type`, and so on.

```json
{
  "entry_id": "etr_abc123",
  "display_id": "EXP26000025",
  "name": "HPLC run 3",
  "fields": {
    "project": "VIR-0001",
    "experiment_type": "Assay"
  },
  "customFields": {
    "instrument": "HPLC-02"
  }
}
```

If two display names give the same key, the first keeps it and later ones get `_2`, `_3`, and so on. A name with no letters or digits (such as "###") is skipped. Both cases are logged as warnings.

A bucket's [Quilt workflow](https://docs.quilt.bio/advanced-features/workflows) can then validate these values. Keep `fields` and `customFields` optional in the schema, because packages made by webhook versions before 0.21.1 don't have them:

```json
{
  "type": "object",
  "properties": {
    "fields": {
      "type": "object",
      "properties": {
        "project": {"type": "string", "pattern": "^VIR-"}
      }
    }
  }
}
```

Editing a field on an entry re-packages it (the `v2.entry.updated.fields` event), so the package metadata follows the entry.

### Package Linking

![experiment_id](imgs/benchling-link.png)

In addition, Quilt users can 'tag' additional packages by setting the `experiment_id` (or a custom metadta key) to the display ID of a Benchling notebook, e.g., `EXP00001234`.

From inside the Quilt Catalog:

1. Navigate to the package of interest
2. Click 'Revise Package'
3. Go the metadata editor in the bottom left
4. In the bottom row, enter `experiment_id` as key and the display ID as the value.
5. Set the commit message and click 'Save'

#### Linking with an RO-Crate

A lab can also link data to a notebook entry without editing package metadata, by writing an [RO-Crate](https://www.researchobject.org/ro-crate/) that conforms to the [Quilt RO-Crate profile](https://w3id.org/quilt/ro-crate). In the crate's `ro-crate-metadata.json`, name the entry as an `ELNEntry` under the root dataset's `subjectOf`, with the entry's display ID as its `name` and its Benchling URL as its `@id`:

```json
{
  "@context": "https://w3id.org/ro/crate/1.2/context",
  "@graph": [
    {
      "@id": "ro-crate-metadata.json",
      "@type": "CreativeWork",
      "conformsTo": { "@id": "https://w3id.org/ro/crate/1.2" },
      "about": { "@id": "./" }
    },
    {
      "@id": "./",
      "@type": "Dataset",
      "name": "Plate reader run",
      "description": "Absorbance at 600 nm for notebook entry EXP00001234",
      "datePublished": "2026-09-29",
      "license": { "@id": "https://example.org/legal/internal-research-use/1.0" },
      "hasPart": [{ "@id": "absorbance.csv" }],
      "subjectOf": { "@id": "https://example.benchling.com/lab/f/lib_Xy12Ab34-assays/etr_AbC123-plate-reader-run/edit" }
    },
    { "@id": "absorbance.csv", "@type": "File" },
    {
      "@id": "https://example.benchling.com/lab/f/lib_Xy12Ab34-assays/etr_AbC123-plate-reader-run/edit",
      "@type": "CreativeWork",
      "additionalType": { "@id": "https://w3id.org/quilt/ro-crate#ELNEntry" },
      "name": "EXP00001234"
    }
  ]
}
```

When Quilt packages the crate, it records the entry in the package metadata as `"eln_entry": ["EXP00001234"]`. The entry's App Canvas then lists the package under Linked Packages, along with the creator, producer, and instrument the crate names. See the profile for how to describe those, and its [example crate](https://w3id.org/quilt/ro-crate/0.2/example1/ro-crate-metadata.json).

You can also set `eln_entry` yourself in the catalog's metadata editor, as either a list or a single display ID (`"eln_entry": "EXP00001234"`). Either form links the package. When `eln_entry` is a list, list-valued `creator`, `producer`, `instrument`, and `instrument_id` metadata also appear on the canvas, whether Quilt wrote them from a crate or you entered them by hand. A single display ID shows the link alone.

### Benchling App Canvas

![App Canvas - Home](imgs/benchling-canvas.png)

The webhook includes a Benchling App Canvas, which allows Benchling users to view, browse, sync, and link associated Quilt packages.

- Clicking the package name opens it in the Quilt Catalog
- Linked packages built from an RO-Crate also show the crate's creator, producer, and instrument
- The `sync` button opens the package or file in [QuiltSync](https://www.quilt.bio/quiltsync), if installed
- **Update Package** refreshes the entry package, as Benchling only notifies Quilt of changes when metadata fields are modified
- **Link Packages** lists the five newest packages in the configured package bucket under the entry package's prefix that are not linked to the entry and are not another entry's own package; Previous and Next page through the rest

From **Link Packages**, **Link _package-name_** creates a metadata-only revision: it sets the configured package link key (normally `experiment_id`) to this entry's display ID, or adds the display ID to `eln_entry` when the package already has an `eln_entry` or is linked to another entry. Every package entry, object hash, and physical key remains unchanged. The revision is validated before it is published, against the workflow the package was pushed with or else the bucket's default; a rejection is shown on the Canvas. Another entry's own package is never linked, since that entry's next export would drop the link, and a package sealed by an accepted review is never revised. The picker only offers packages from the webhook's configured writable bucket. Use **Link Packages** again to add more links after the Canvas refreshes. Related controls are unavailable until the entry package exists and disappear when an accepted review locks the Canvas.

The canvas also allows you to browse package contents:

![App Canvas - Browse](imgs/benchling-browse.png)

and view package metadata:

![App Canvas - Metadata](imgs/benchling-browse.png)

#### Inserting a Canvas

If the App Canvas is not already part of your standard notebook template, Benchling users can add it themselves:

1. Create a notebook entry
2. Select "Insert" → "Canvas"
3. Choose "Quilt Package"
4. After it is inserted, click the "Create" button

![App Canvas - Insert](imgs/benchling-insert.png)

#### Accepted Reviews

When an entry's review is accepted, the webhook seals the entry package: it records the linked packages' current revisions in `linked_packages.json`, stops updating the package, and the canvas, titled **🔒 Locked**, links to those revisions instead of `latest`. Re-opening the review unseals the package and updates resume; accepting it again reseals it at the linked packages' new revisions.

### Security Features

**Single Authentication Layer:**

- **FastAPI HMAC Verification** - All webhook requests verified against Benchling secret
- Signatures computed over raw request body
- Invalid signatures return 403 Forbidden

**Optional Network Filtering:**

- **Resource Policy IP Filtering** - Free alternative to AWS WAF ($7/month saved)
- Blocks unknown IPs at API Gateway edge (applies to all endpoints)
- **BREAKING CHANGE (v1.1.0+)**: Health endpoints NO LONGER exempt from IP filtering
- External monitoring services must be added to allowlist or IP filtering disabled
- NLB health checks unaffected (bypass API Gateway)
- IP filtering does NOT replace authentication (it's defense-in-depth)

**Infrastructure Security:**

- Private network (ECS in private subnets, no public IPs)
- VPC Link encrypted connection between API Gateway and NLB
- TLS 1.2+ encryption on all API Gateway endpoints
- CloudWatch audit trail for HMAC verification and resource policy decisions
- Least-privilege IAM roles

## Installation

### 1. Installing the Benchling App

This requires a Benchling admin to use `npx` from [NodeJS](https://nodejs.org) version 18 or later.

#### 1.1 Get the app manifest

Download `app-manifest.yaml` from the
[latest GitHub release](https://github.com/quiltdata/benchling-webhook/releases/latest),
or generate one locally:

```bash
npx @quiltdata/benchling-webhook@latest manifest
```

#### 1.2 Upload the manifest to Benchling

- Follow Benchling's [create](https://docs.benchling.com/docs/getting-started-benchling-apps#creating-an-app-from-a-manifest) and [install](https://docs.benchling.com/docs/getting-started-benchling-apps#installing-your-app) instructions.
- Save the **App Definition ID**, **Client ID**, and **Client Secret** for the next step.

### 2. Configuring the Benchling App

Your command-line environment must have AWS credentials for the account containing your Quilt stack.
All you need to do is use `npx` to run the package:

```bash
npx @quiltdata/benchling-webhook@latest
```

If you need to choose AWS credentials explicitly, prefer `--aws-profile`:

```bash
npx @quiltdata/benchling-webhook@latest --aws-profile myaws
```

You can also use `AWS_PROFILE`:

```bash
AWS_PROFILE=myaws npx @quiltdata/benchling-webhook@latest
```

`--profile` is different: it selects a local `benchling-webhook` config profile under `~/.config/benchling-webhook/`, not your AWS credential profile.

The wizard will guide you through:

1. **Catalog discovery** - Detect your Quilt catalog configuration
2. **Stack validation** - Extract settings from your CloudFormation stack
3. **Credential collection** - Enter Benchling app credentials
4. **Package settings** - Configure bucket, package prefix (see [Package Naming](#package-naming)), metadata key, optional Quilt workflow, and whether entries are packaged automatically (see [Turning Automatic Packaging Off](#turning-automatic-packaging-off))
5. **Deployment mode selection**:
   - **Integrated**: Uses your Quilt stack's built-in webhook, if any
   - **Standalone**: Deploys a separate webhook stack for testing

**Note**: Configuration is stored in `~/.config/benchling-webhook/` using the [XDG Base Directory](https://wiki.archlinux.org/title/XDG_Base_Directory) standard, supporting multiple profiles.

### 3. Configure Webhook URL

Add the webhook URL (displayed after setup) to your [Benchling app settings](https://docs.benchling.com/docs/getting-started-benchling-apps#installing-your-app). The URL is the API Gateway stage root, e.g. `https://{api-id}.execute-api.{region}.amazonaws.com/{stage}`.

**Important**: In the Benchling **Webhook Setup** dialog, set **Webhook Routing Setting** to **Suffixed**. Benchling then appends a path suffix based on the kind of event (`/lifecycle`, `/event`, or `/canvas`) to your configured webhook URL — which is what this webhook's routes expect. Do **not** select **Stable**: it posts every event to the bare URL with no suffix, and requests return `404 Endpoint not found`.

![Webhook Routing Setting](imgs/benchling-webhook-routing.png)

If your integration reads or writes within a specific Benchling project, share that project with the service account behind the Benchling App Client ID. This integration uses the app/service-account identity, not an end-user OAuth session. If project access appears broken, verify the service account can perform a simple read or list API call for the target project.

### 4. Test Integration

In Benchling:

1. Create a notebook entry
2. Insert Canvas → Select "Quilt Package"
3. Click "Create"

A Quilt package will be automatically created and linked to your notebook entry.
If you run into problems, contact [Quilt Support](support@quilt.bio)

## Multi-Stack Deployments (v0.9.8+)

Starting with version 0.9.8, you can deploy **multiple webhook stacks** in the same AWS account/region. This is useful for:

- **Multi-tenant deployments** - Separate stacks for each customer
- **Environment isolation** - Dev, staging, prod in same account
- **A/B testing** - Parallel stacks with different configurations

### Profile-Based Stack Names

Each profile automatically gets its own CloudFormation stack:

```bash
# Default profile uses legacy name (backwards compatible)
npx @quiltdata/benchling-webhook@latest deploy --profile default
# Creates: BenchlingWebhookStack

# Other profiles get unique names
npx @quiltdata/benchling-webhook@latest deploy --profile sales
# Creates: BenchlingWebhookStack-sales

npx @quiltdata/benchling-webhook@latest deploy --profile customer-acme
# Creates: BenchlingWebhookStack-customer-acme
```

### Custom Stack Names

You can also specify a custom stack name in your profile configuration:

```json
{
  "deployment": {
    "stackName": "MyCustomWebhookStack",
    ...
  }
}
```

### Managing Multiple Stacks

All commands support the `--profile` flag:

```bash
# Deploy a specific profile
npx @quiltdata/benchling-webhook@latest deploy --profile sales

# Check status
npx @quiltdata/benchling-webhook@latest status --profile sales

# View logs
npx @quiltdata/benchling-webhook@latest logs --profile sales

# Destroy stack
npx @quiltdata/benchling-webhook@latest destroy --profile sales
```
