# pi-langfuse

[![npm version](https://img.shields.io/npm/v/pi-langfuse)](https://www.npmjs.com/package/pi-langfuse)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

[**English**](./README.md) | [**简体中文**](./README_CN.md)

Langfuse observability extension for [Pi Coding Agent](https://github.com/earendil-works/pi-coding-agent). It sends complete Pi runs to [Langfuse](https://langfuse.com) so the prompt, agent workflow, LLM generations, tool calls, final response, usage, cost, and health scores appear in one trace.

## What This Adds to Pi

- One Langfuse trace per user prompt, grouped by Pi session.
- Root `agent`, per-request `generation`, and per-tool `tool` observations.
- Final assistant output capture, tool error visibility, and trace-level scores.
- Privacy controls for inputs, outputs, tool I/O, system prompt, and cwd.
- Secret redaction and local path hashing before upload.
- Capability-gated REST fallback for self-hosted Langfuse setups that expose the legacy trace API when OTel spans arrive but traces do not materialize. Langfuse v4 `events_only` deployments use OTel without legacy fallback ingestion.

## Prerequisites

- **Node.js** >= 22
- **Pi Coding Agent** installed and configured
- A **Langfuse** account ([cloud](https://cloud.langfuse.com) or self-hosted)

## Quick Start

1. Install the extension:

   ```bash
   pi install npm:pi-langfuse
   ```

2. Run Pi once. If no credentials are configured yet, Pi prompts for:
   - Langfuse public key, starting with `pk-lf-...`
   - Langfuse secret key, starting with `sk-lf-...`
   - Langfuse host, defaulting to `https://cloud.langfuse.com`

3. Run Pi normally:

   ```bash
   pi "Explain the architecture of Redis"
   ```

4. Open Langfuse and inspect the new trace.

## Configuration

Langfuse API keys are available in **Langfuse Cloud** -> **Settings** -> **API Keys**.

### Method 1: Interactive setup

Run any `pi` command with the extension loaded. On first run without configuration, Pi prompts in the CLI or TUI and saves the result to `~/.pi/agent/pi-langfuse/config.json`.

To run setup again:

```text
/langfuse-setup
```

To inspect the active configuration without exposing secrets:

```text
/langfuse-status
```

The status command reports the config source, host, masked public key, capture policy, active-run state, config path, and last runtime error.

### Method 2: Environment variables

Set these before starting Pi:

```bash
export LANGFUSE_PUBLIC_KEY="pk-lf-xxxx"
export LANGFUSE_SECRET_KEY="sk-lf-xxxx"
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"  # optional; LANGFUSE_HOST is also supported
```

Saved config takes precedence. Environment variables are only used when `~/.pi/agent/pi-langfuse/config.json` is missing or incomplete.

For short-lived SDK hosts, set the bounded final score-delivery attempt during shutdown:

```bash
export PI_LANGFUSE_SCORE_SHUTDOWN_TIMEOUT=2  # seconds; defaults to 2 seconds
```

The extension attempts queued trace-level scores before other shutdown telemetry work. This value cannot extend the overall shutdown deadline.

Privacy controls can also be set through environment variables:

```bash
export LANGFUSE_PRIVACY_PRESET="full-debug"
```

Available presets:

| Preset | Captures |
|--------|----------|
| `metadata-only` | Metadata only; omits inputs, outputs, tool I/O, system prompt, and cwd |
| `prompts-only` | Prompt/provider inputs plus metadata |
| `conversations` | Inputs and assistant outputs, but omits tool I/O, system prompt, and cwd |
| `full-debug` | Full trace detail; this is the default |

Fine-grained flags override presets:

```bash
export LANGFUSE_CAPTURE_INPUTS=true
export LANGFUSE_CAPTURE_OUTPUTS=true
export LANGFUSE_CAPTURE_TOOL_IO=false
export LANGFUSE_CAPTURE_SYSTEM_PROMPT=false
export LANGFUSE_CAPTURE_CWD=false
export LANGFUSE_CAPTURE_SOURCE_METADATA=false
export LANGFUSE_CAPTURE_PATHS=false
```

Source metadata remains off in every preset unless `LANGFUSE_CAPTURE_SOURCE_METADATA=true` is set explicitly.
The same holds for absolute paths and `LANGFUSE_CAPTURE_PATHS`.

All captured payloads are redacted before upload. The extension masks common API keys, bearer tokens, passwords, cookies, private keys, Langfuse keys, GitHub/npm/AWS-style tokens, and local absolute paths.

### Absolute paths

By default, local absolute paths (`/Users/...`, `/home/...`, `/tmp/...`, `C:\Users\...`) are
replaced everywhere with a stable `[PATH_HASH:<12 hex chars>]` digest, so usernames and repository
names never reach Langfuse. This applies to inputs, outputs, tool I/O, tool error messages, and the
`cwd` metadata field. Opt in to see real paths in traces:

```bash
export LANGFUSE_CAPTURE_PATHS=true
```

Or persist it in `config.json`:

```json
{ "capture": { "LANGFUSE_CAPTURE_PATHS": "true" } }
```

Like `LANGFUSE_CAPTURE_SOURCE_METADATA`, this stays off in every privacy preset until it is set
explicitly, and it only affects paths — secret masking (tokens, keys, cookies, passwords) is always
on regardless. Note that `LANGFUSE_CAPTURE_CWD=false` is a different control: it drops the `cwd`
metadata field entirely rather than changing how paths are rendered.

`/langfuse-status` reports the current setting under `Capture: absolute paths`.

### Payload limits

Before upload, payloads are shaped: strings are truncated and deeply nested or
very wide structures are trimmed. These caps keep traces small and protect the
Langfuse ingestion pipeline. Override any of them (no rebuild needed):

```bash
export PI_LANGFUSE_MAX_STRING_LENGTH=12000       # per-string chars (system prompt, inputs)
export PI_LANGFUSE_MAX_TOOL_PAYLOAD_LENGTH=24000 # per tool input/output chars
export PI_LANGFUSE_MAX_DEPTH=6                    # max nesting depth
export PI_LANGFUSE_MAX_ARRAY_ITEMS=50            # max array elements kept
export PI_LANGFUSE_MAX_OBJECT_KEYS=80            # max object keys kept
export PI_LANGFUSE_MAX_PAYLOAD_NODES=2000        # max total nodes per payload
```

Set any limit to `0`, `off`, `none`, or `unlimited` to disable that cap
entirely (captures the full value). Unset or invalid values fall back to the
defaults shown above. To capture a very large system prompt or big tool
payloads in full, raise or disable the relevant limit (e.g.
`PI_LANGFUSE_MAX_STRING_LENGTH=off`).

The REST fallback ingestion is chunked so each request body stays well below
the Langfuse gateway's payload limit (~4.5MB). These knobs control the chunk
budget and a hard ceiling for the whole fallback payload:

```bash
export PI_LANGFUSE_MAX_INGESTION_BATCH_BYTES=4194304  # per-request body budget, default 4MB

export PI_LANGFUSE_MAX_FALLBACK_TOTAL_BYTES=33554432  # whole-payload ceiling, default 32MB
```

When the accumulated fallback payload exceeds the 32MB ceiling, ingestion is
skipped with a warning instead of attempting an unrecoverably large upload.

### Reasoning tokens

Pi reports reasoning (thinking) tokens for Anthropic, OpenAI Codex, OpenRouter,
opencode-go and Qwen. Providers count them inside `output`, and by default the
extension reports `output` whole, so the reasoning share is not visible in
Langfuse. Opt in to report it as its own usage bucket:

```bash
export PI_LANGFUSE_SPLIT_REASONING_TOKENS=true
```

Or persist it in `config.json`:

```json
{ "capture": { "PI_LANGFUSE_SPLIT_REASONING_TOKENS": "true" } }
```

With the split on, a generation that used 37 output tokens of which 10 were
reasoning is reported as `output: 27` plus `output_reasoning_tokens: 10`. Both
keys contain `output`, so the Output row in Langfuse still shows 37; only the
breakdown beneath it gains the reasoning share. Reasoning is clamped to the
reported `output`, so the buckets always add up to the total.

> **Before enabling — check your model prices.** Langfuse matches prices to
> usage by exact key, and model definitions you created in your project take
> precedence over Langfuse's maintained defaults. A custom model priced on
> `input` and `output` only would cost `output_reasoning_tokens` at zero, so
> reasoning-heavy generations would look cheaper than they are. Add a price
> for `output_reasoning_tokens` to every custom reasoning model in
> **Settings → Models** first, then turn the split on. Langfuse's built-in
> prices for reasoning models already include it. Providers that report their
> own cost are unaffected: Langfuse uses the reported cost as-is and does not
> recompute it from usage.

The split is off by default, so upgrading changes nothing until you enable it.
Unset `PI_LANGFUSE_SPLIT_REASONING_TOKENS` (or set it to `false`) to go back;
traces already ingested keep their buckets.

### Method 3: Persistent `config.json`

Create or update `~/.pi/agent/pi-langfuse/config.json`:

```json
{
  "publicKey": "pk-lf-xxxx",
  "secretKey": "sk-lf-xxxx",
  "host": "https://cloud.langfuse.com",
  "privacyPreset": "conversations"
}
```

Fine-grained capture flags can also be persisted:

```json
{
  "publicKey": "pk-lf-xxxx",
  "secretKey": "sk-lf-xxxx",
  "host": "https://cloud.langfuse.com",
  "capture": {
    "LANGFUSE_PRIVACY_PRESET": "metadata-only",
    "LANGFUSE_CAPTURE_INPUTS": "true"
  }
}
```

> **Security**: Keep `~/.pi/agent/pi-langfuse/config.json` private. Never commit API keys to version control.
> When the extension writes this file itself, it creates the config directory with `0700` permissions and the file with `0600` permissions where the host filesystem supports POSIX modes.

## Verify the Extension

Check that Pi has loaded the package:

```bash
pi list
```

`pi-langfuse` should appear in the installed package list.

To verify the Langfuse host and API keys from inside Pi, run:

```text
/langfuse-test
```

This command makes a timeout-bounded authenticated request to Langfuse and, if it succeeds, sends a small test trace.

## What Appears in Langfuse

- Each Pi session gets its own Langfuse session ID.
- Each user prompt within that session becomes a separate trace.
- The trace contains the final assistant output shown in Pi.
- Tool runs appear as tool observations with arguments, results, and error state.
- LLM requests appear as generation observations, including usage and cost when the provider exposes them.
  Reasoning tokens are reported as their own usage bucket when `PI_LANGFUSE_SPLIT_REASONING_TOKENS` is enabled.
- Trace-level scores include tool counts, tool success rate, and whether the run had errors.

The package also includes a Langfuse CLI skill, so Langfuse data can be queried directly from Pi:

```text
/pi-langfuse-langfuse <your-query>
```

## Source Metadata

Repository source capture is independent of the privacy presets and disabled by default. Enable it only after deciding that commit identity is appropriate for the Langfuse project:

```bash
export LANGFUSE_CAPTURE_SOURCE_METADATA=true
```

For a Git worktree, the extension records only revision state:

```json
{
  "source_type": "git-repo",
  "vcs.ref.head.revision": "0123456789abcdef...",
  "git_detached": "false",
  "git_dirty": "false",
  "metadata_source": "git-detection"
}
```

The revision is the full `HEAD` commit. Dirty state includes tracked changes and untracked files, but never their paths or contents. Detached state is reported without a branch or tag name. Git remotes, URLs, credentials, usernames, branches, absolute paths, and repository names are never inspected or uploaded by this collector.

When capture is off, the collector does not invoke Git and reports `source_type: "disabled"`. Non-Git directories report `non-git`; a missing or unusable Git executable and incomplete Git state report `unavailable`.

Use native Langfuse and OpenTelemetry settings for deployment identity and explicit operator-owned overrides instead of repository files:

```bash
export LANGFUSE_RELEASE="1.2.3"
export LANGFUSE_TRACING_ENVIRONMENT="production"
export OTEL_SERVICE_NAME="pi-agent"
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.2.3,vcs.repository.name=public-repo"
```

`LANGFUSE_RELEASE` and `LANGFUSE_TRACING_ENVIRONMENT` keep their Langfuse semantics. `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES` are loaded as process-scoped OpenTelemetry resource attributes; values such as `vcs.repository.name` apply to every session sharing the runtime, may identify private source, and require a runtime restart to change.

### Compatibility and rollback

Earlier versions emitted `git_commit`, branch, remote, owner, repository, and repo-local `.pi-langfuse.metadata.json` values by default. New traces use `vcs.ref.head.revision` and stop reading that file. Update dashboards before enabling source capture; historical traces are unchanged.

Unset `LANGFUSE_CAPTURE_SOURCE_METADATA` to stop collection immediately. Pin `pi-langfuse@1.5.12` only if the old schema is required during migration; doing so also restores its broader default source disclosure.

## Troubleshooting

### No traces appearing?

- Verify the API keys and run `/langfuse-setup` again if needed.
- Run `/langfuse-status` to confirm the loaded host, config source, privacy mode, and last runtime error.
- Confirm the Langfuse project is active and accepts writes.
- Confirm the keys have write permission.
- Look for `📊 Langfuse:` log messages in Pi output.

### Extension not loading?

```bash
pi list
pi install npm:pi-langfuse
```

### "Missing config" on startup?

- Run `/langfuse-setup`.
- Or set `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` before starting Pi.

### Model or cost not showing?

- Some providers do not expose cost information.
- Inspect the raw observation data in Langfuse traces.
- The `model` field can come from provider events, finalized assistant messages, `model_select`, or `ctx.model`.

### API key errors?

- Public keys start with `pk-lf-`.
- Secret keys start with `sk-lf-`.
- For self-hosted deployments, verify the host URL.

## Development Docs

Development setup, source installation, runtime architecture, trace model, tracked fields, and validation steps are documented in [DEVELOPMENT.md](./DEVELOPMENT.md) and [DEVELOPMENT_CN.md](./DEVELOPMENT_CN.md).

## License

MIT