---
title: QMD Backend Algorithm
summary: Query matching and distribution, backend health checking, and load balancing for Quadromem Memory Database
---

# QMD Backend Algorithm

## Overview

The QMD (Quadromem Memory Database) backend provides external memory search capabilities via the `qmd` CLI tool and `mcporter` service. It handles large-scale memory indexing with support for multiple collections, custom indexing patterns, and advanced query matching.

**Location:** `src/memory/qmd-manager.ts`

### Key Features

- **External process management** - Spawns and manages `qmd` CLI subprocesses
- **Collection management** - Handles multiple memory collections (memory, custom, sessions)
- **Query parsing** - Structured query result parsing with snippet extraction
- **Health probing** - Backend availability checking with exponential backoff
- **Stream processing** - Line-by-line output parsing for large result sets

---

## Architecture

```
┌─────────────────────────────────────────────────────────┐
│  SophiaClaw Gateway                                     │
│                                                         │
│  ┌───────────────────────────────────────────────────┐ │
│  │  QmdMemoryManager                                 │ │
│  │                                                   │ │
│  │  ┌────────────────┐  ┌────────────────┐          │ │
│  │  │ qmd            │  │ mcporter       │          │ │
│  │  │ (CLI tool)     │  │ (port manager) │          │ │
│  │  └───────┬────────┘  └────────────────┘          │ │
│  │          │                                       │ │
│  │          │ spawn()                               │ │
│  │          ▼                                       │ │
│  │  ┌─────────────────────────────────────────┐    │ │
│  │  │  Child Process (stdout/stderr streams)  │    │ │
│  │  │                                         │    │ │
│  │  │  qmd query --collection memory          │    │ │
│  │  │      --query "embedding vector"         │    │ │
│  │  │      --limit 10                         │    │ │
│  │  │                                         │    │ │
│  │  │  Output:                                │    │ │
│  │  │  @@ -100,200                            │    │ │
│  │  │  { "score": 0.95, ... }                 │    │ │
│  │  │  @@ -300,150                            │    │ │
│  │  │  { "score": 0.87, ... }                 │    │ │
│  │  └─────────────────────────────────────────┘    │ │
│  └───────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
         │
         │ TCP/IPC
         ▼
┌─────────────────────────────────────────────────────────┐
│  QMD Backend Service                                    │
│                                                         │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Collection Store                               │   │
│  │  - memory/ (transcript snippets)                │   │
│  │  - custom/ (user-defined collections)           │   │
│  │  - sessions/ (session exports)                  │   │
│  └─────────────────────────────────────────────────┘   │
│                                                         │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Index Storage                                  │   │
│  │  - Vector indexes (HNSW/IVF)                   │   │
│  │  - BM25 indexes                                 │   │
│  │  - Metadata indexes                             │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘
```

---

## Manager Creation

### Factory Method

```typescript
export class QmdMemoryManager implements MemorySearchManager {
  static async create(params: {
    cfg: SophiaClawConfig;
    agentId: string;
    resolved: ResolvedMemoryBackendConfig;
    mode?: QmdManagerMode; // "full" | "status"
  }): Promise<QmdMemoryManager | null> {
    const resolved = params.resolved.qmd;
    if (!resolved) {
      return null;
    }

    // Resolve command paths
    const qmdCommand = resolveWindowsCommandShim(resolved.qmdCommand ?? "qmd");
    const mcporterCommand = resolveWindowsCommandShim(resolved.mcporter?.command ?? "mcporter");

    // Verify commands exist
    const qmdAvailable = await verifyCommand(qmdCommand);
    const mcporterAvailable = await verifyCommand(mcporterCommand);

    if (!qmdAvailable || !mcporterAvailable) {
      log.warn(`qmd backend unavailable: qmd=${qmdAvailable}, mcporter=${mcporterAvailable}`);
      return null;
    }

    // Resolve collections
    const collections = await resolveCollections(params.cfg, params.agentId, resolved);

    // Create manager instance
    return new QmdMemoryManager({
      cfg: params.cfg,
      agentId: params.agentId,
      qmdCommand,
      mcporterCommand,
      collections,
      mode: params.mode ?? "full",
    });
  }
}
```

### Collection Resolution

```typescript
async function resolveCollections(
  cfg: SophiaClawConfig,
  agentId: string,
  resolved: ResolvedQmdConfig,
): Promise<CollectionRoot[]> {
  const collections: CollectionRoot[] = [];

  // Add memory collection (default)
  if (resolved.memory?.enabled !== false) {
    const memoryPath = resolved.memory?.path ?? resolveAgentWorkspaceDir(cfg, agentId);
    collections.push({
      path: memoryPath,
      kind: "memory" as MemorySource,
    });
  }

  // Add custom collections
  if (resolved.collections) {
    for (const collection of resolved.collections) {
      const path = await resolveCollectionPath(cfg, agentId, collection);
      if (path) {
        collections.push({
          path,
          kind: collection.kind ?? ("custom" as MemorySource),
        });
      }
    }
  }

  // Add sessions collection
  if (resolved.sessions?.enabled !== false) {
    const sessionDir = resolveSessionDir(cfg, agentId);
    const sessionConfig: SessionExporterConfig = {
      dir: sessionDir,
      retentionMs: resolved.sessions?.retentionMs,
      collectionName: "sessions",
    };
    collections.push({
      path: sessionDir,
      kind: "sessions" as MemorySource,
    });
  }

  return collections;
}
```

---

## Query Processing

### Query Execution

```typescript
async function search(
  query: string,
  opts?: { maxResults?: number; minScore?: number; sessionKey?: string },
): Promise<MemorySearchResult[]> {
  // Parse query for keywords (for BM25 fallback)
  const keywords = extractKeywords(query);
  const normalizedQuery = normalizeHanBm25Query(query);

  // Build qmd command
  const args: string[] = [
    "query",
    "--collection",
    this.collectionName,
    "--query",
    normalizedQuery,
    "--limit",
    String(opts?.maxResults ?? 10),
  ];

  if (opts?.sessionKey) {
    args.push("--session", opts.sessionKey);
  }

  if (opts?.minScore) {
    args.push("--min-score", String(opts.minScore));
  }

  // Execute with timeout
  const result = await runWithTimeout(
    spawn(this.qmdCommand, args, {
      stdio: ["pipe", "pipe", "pipe"],
    }),
    QUERY_TIMEOUT_MS,
  );

  // Parse output
  return parseQmdOutput(result.stdout, result.stderr);
}
```

### Output Parsing

```typescript
function parseQmdOutput(stdout: string, stderr: string): MemorySearchResult[] {
  const results: MemorySearchResult[] = [];
  const lines = stdout.split("\n");

  let currentSnippet: QmdSnippet | null = null;

  for (const line of lines) {
    // Check for snippet header: @@ -start,length
    const headerMatch = line.match(SNIPPET_HEADER_RE);
    if (headerMatch) {
      // Finalize previous snippet
      if (currentSnippet) {
        results.push(finalizeSnippet(currentSnippet));
      }

      // Start new snippet
      currentSnippet = {
        start: parseInt(headerMatch[1], 10),
        length: parseInt(headerMatch[2], 10),
        content: "",
        metadata: null,
      };
      continue;
    }

    // Try to parse as JSON metadata
    try {
      const metadata = JSON.parse(line);
      if (currentSnippet) {
        currentSnippet.metadata = metadata;
      }
    } catch {
      // Not JSON, treat as content
      if (currentSnippet) {
        currentSnippet.content += line + "\n";
      }
    }
  }

  // Don't forget last snippet
  if (currentSnippet) {
    results.push(finalizeSnippet(currentSnippet));
  }

  return results;
}

function finalizeSnippet(snippet: QmdSnippet): MemorySearchResult {
  const metadata = snippet.metadata ?? {};
  return {
    id: metadata.id ?? randomId(),
    chunkId: metadata.chunkId ?? "",
    sessionId: metadata.sessionId ?? "",
    content: snippet.content.trim(),
    score: metadata.score ?? 0,
    snippets: [
      {
        start: snippet.start,
        length: snippet.length,
        content: snippet.content.trim(),
      },
    ],
  };
}
```

### Query Parser Integration

```typescript
// In src/memory/qmd-query-parser.ts
export function parseQmdQueryJson(output: string, query: string): QmdQueryResult {
  const lines = output.trim().split("\n");
  const results: Array<{
    score: number;
    snippet: string;
    file?: string;
    start?: number;
    length?: number;
  }> = [];

  for (const line of lines) {
    if (!line.trim()) continue;

    try {
      const parsed = JSON.parse(line);
      results.push({
        score: parsed.score ?? 0,
        snippet: parsed.snippet ?? "",
        file: parsed.file,
        start: parsed.start,
        length: parsed.length,
      });
    } catch (err) {
      // Skip unparseable lines
      log.warn(`Failed to parse QMD output line: ${err}`);
    }
  }

  return {
    query,
    results,
    total: results.length,
  };
}
```

---

## Process Management

### Spawn with Stream Processing

```typescript
async function runQmdCommand(args: string[]): Promise<{
  stdout: string;
  stderr: string;
  exitCode: number | null;
}> {
  return new Promise((resolve, reject) => {
    const child = spawn(this.qmdCommand, args, {
      stdio: ["pipe", "pipe", "pipe"],
    });

    let stdout = "";
    let stderr = "";

    // Stream processing with readline
    const rl = readline.createInterface({
      input: child.stdout,
      crlfDelay: Infinity,
    });

    rl.on("line", (line) => {
      // Process line immediately (don't wait for full output)
      if (line.startsWith("@@")) {
        // Snippet header - start new result
        onSnippetStart(line);
      } else {
        // Content or metadata
        onSnippetData(line);
      }
    });

    child.stderr.on("data", (data) => {
      stderr += data.toString();
    });

    child.on("close", (code) => {
      rl.close();
      resolve({ stdout, stderr, exitCode: code });
    });

    child.on("error", (err) => {
      rl.close();
      reject(err);
    });

    // Timeout handling
    setTimeout(() => {
      child.kill("SIGTERM");
      reject(new Error("QMD command timeout"));
    }, COMMAND_TIMEOUT_MS);
  });
}
```

### Concurrency Control

```typescript
// Serialize embedding operations to avoid overloading qmd
let qmdEmbedQueueTail: Promise<void> = Promise.resolve();

async function runWithQmdEmbedLock<T>(task: () => Promise<T>): Promise<T> {
  const previous = qmdEmbedQueueTail;
  let release: (() => void) | undefined;

  // Add task to queue
  qmdEmbedQueueTail = new Promise<void>((resolve) => {
    release = resolve;
  });

  // Wait for previous task
  await previous.catch(() => undefined);

  try {
    return await task();
  } finally {
    release?.();
  }
}

// Usage
async function embedAndIndex(content: string): Promise<void> {
  await runWithQmdEmbedLock(async () => {
    await runQmdCommand(["embed", "--content", content]);
  });
}
```

---

## Health Checking

### Probe Embedding Availability

```typescript
async function probeEmbeddingAvailability(): Promise<MemoryEmbeddingProbeResult> {
  try {
    // Test embedding generation
    const result = await runWithQmdEmbedLock(async () => {
      return await runQmdCommand(["probe", "--embedding", "--model", this.embeddingModel]);
    });

    if (result.exitCode === 0) {
      return { ok: true };
    } else {
      return {
        ok: false,
        error: `Probe failed: ${result.stderr}`,
      };
    }
  } catch (err) {
    return {
      ok: false,
      error: err instanceof Error ? err.message : String(err),
    };
  }
}
```

### Probe Vector Availability

```typescript
async function probeVectorAvailability(): Promise<boolean> {
  try {
    const result = await runQmdCommand(["probe", "--vector", "--collection", this.collectionName]);

    return result.exitCode === 0;
  } catch {
    return false;
  }
}
```

### Exponential Backoff

```typescript
// Track embedding failures with exponential backoff
let embedQueueFailureBackoffUntil: number | undefined;
let embedQueueFailureCount = 0;

async function runWithBackoff<T>(task: () => Promise<T>): Promise<T> {
  const now = Date.now();

  // Check if still in backoff period
  if (embedQueueFailureBackoffUntil && now < embedQueueFailureBackoffUntil) {
    const remaining = embedQueueFailureBackoffUntil - now;
    throw new Error(`Embedding backoff active, retry in ${remaining}ms`);
  }

  try {
    const result = await task();

    // Reset failure count on success
    embedQueueFailureCount = 0;
    embedQueueFailureBackoffUntil = undefined;

    return result;
  } catch (err) {
    // Exponential backoff: base * 2^count
    embedQueueFailureCount++;
    const backoffMs = Math.min(
      QMD_EMBED_BACKOFF_BASE_MS * Math.pow(2, embedQueueFailureCount),
      QMD_EMBED_BACKOFF_MAX_MS,
    );

    embedQueueFailureBackoffUntil = now + backoffMs;

    log.warn(
      `QMD embedding failed (count=${embedQueueFailureCount}), ` + `backing off for ${backoffMs}ms`,
    );

    throw err;
  }
}
```

---

## Scope Management

### Collection Scope

```typescript
// In src/memory/qmd-scope.ts
export function deriveQmdScopeChannel(sessionKey: string): string | undefined {
  // Parse session key format: channel:id:topic or channel:id
  const parts = sessionKey.split(":");
  if (parts.length >= 1) {
    return parts[0]; // e.g., "whatsapp", "telegram"
  }
  return undefined;
}

export function deriveQmdScopeChatType(
  channel: string,
): "private" | "group" | "broadcast" | undefined {
  // Infer chat type from channel + id patterns
  if (channel === "whatsapp") {
    return "private"; // Default for WhatsApp
  }
  if (channel === "telegram") {
    return "group"; // Default for Telegram
  }
  return undefined;
}

export function isQmdScopeAllowed(
  scope: { channel?: string; chatType?: string },
  config: QmdScopeConfig,
): boolean {
  // Check allowlist
  if (config.allowChannels && scope.channel) {
    if (!config.allowChannels.includes(scope.channel)) {
      return false;
    }
  }

  // Check denylist
  if (config.denyChannels && scope.channel) {
    if (config.denyChannels.includes(scope.channel)) {
      return false;
    }
  }

  // Check chat type restrictions
  if (config.denyChatTypes && scope.chatType) {
    if (config.denyChatTypes.includes(scope.chatType)) {
      return false;
    }
  }

  return true;
}
```

---

## Output Size Limits

```typescript
const MAX_QMD_OUTPUT_CHARS = 200_000;

function truncateOutputIfNeeded(output: string): string {
  if (output.length <= MAX_QMD_OUTPUT_CHARS) {
    return output;
  }

  // Truncate while preserving JSON structure
  const truncated = output.slice(0, MAX_QMD_OUTPUT_CHARS);

  // Try to end at line boundary
  const lastNewline = truncated.lastIndexOf("\n");
  if (lastNewline > 0) {
    return truncated.slice(0, lastNewline) + "\n...[truncated]";
  }

  return truncated + "...[truncated]";
}
```

---

## Han Script Optimization

```typescript
const HAN_SCRIPT_RE = /[\u3400-\u9fff]/u;
const QMD_BM25_HAN_KEYWORD_LIMIT = 12;

function hasHanScript(value: string): boolean {
  return HAN_SCRIPT_RE.test(value);
}

function normalizeHanBm25Query(query: string): string {
  const trimmed = query.trim();

  if (!trimmed || !hasHanScript(trimmed)) {
    return trimmed; // No optimization needed
  }

  // Extract keywords for BM25
  const keywords = extractKeywords(trimmed);
  const normalizedKeywords: string[] = [];
  const seen = new Set<string>();

  for (const keyword of keywords) {
    const token = keyword.trim();
    if (!token || seen.has(token)) {
      continue;
    }

    const includesHan = hasHanScript(token);

    // Skip Han unigrams (too broad for BM25)
    if (includesHan && Array.from(token).length < 2) {
      continue;
    }

    // Skip non-Han tokens < 2 chars
    if (!includesHan && token.length < 2) {
      continue;
    }

    seen.add(token);
    normalizedKeywords.push(token);

    // Limit keyword count
    if (normalizedKeywords.length >= QMD_BM25_HAN_KEYWORD_LIMIT) {
      break;
    }
  }

  // Return keywords or original query
  return normalizedKeywords.length > 0 ? normalizedKeywords.join(" ") : trimmed;
}
```

---

## Error Handling

### Command Execution Errors

```typescript
function handleQmdError(err: unknown): MemorySearchError {
  if (err instanceof Error) {
    if (err.message.includes("timeout")) {
      return {
        code: "TIMEOUT",
        message: "QMD command timed out",
        recoverable: true,
      };
    }

    if (err.message.includes("ENOENT")) {
      return {
        code: "NOT_FOUND",
        message: "QMD command not found",
        recoverable: false,
      };
    }

    if (err.message.includes("connection refused")) {
      return {
        code: "CONNECTION_REFUSED",
        message: "QMD backend unavailable",
        recoverable: true,
      };
    }
  }

  return {
    code: "UNKNOWN",
    message: err instanceof Error ? err.message : String(err),
    recoverable: true,
  };
}
```

### NUL Character Detection

```typescript
const NUL_MARKER_RE = /(?:\^@|\\0|\\x00|\\u0000|null\s*byte|nul\s*byte)/i;

function containsNulMarkers(output: string): boolean {
  return NUL_MARKER_RE.test(output);
}

function sanitizeOutput(output: string): string {
  // Remove NUL characters (can break JSON parsing)
  return output.replace(/\0/g, "");
}
```

---

## Testing Strategies

```typescript
describe("QmdMemoryManager", () => {
  it("creates manager when qmd is available", async () => {
    const manager = await QmdMemoryManager.create({
      cfg: {
        memory: {
          backend: "qmd",
          qmd: {
            /* config */
          },
        },
      },
      agentId: "test",
      resolved: {
        backend: "qmd",
        qmd: {
          /* resolved */
        },
      },
    });

    expect(manager).toBeInstanceOf(QmdMemoryManager);
  });

  it("returns null when qmd is unavailable", async () => {
    jest.spyOn(child_process, "spawn").mockImplementation(() => {
      throw new Error("ENOENT: qmd not found");
    });

    const manager = await QmdMemoryManager.create({
      cfg: {
        memory: {
          backend: "qmd",
          qmd: {
            /* config */
          },
        },
      },
      agentId: "test",
      resolved: {
        backend: "qmd",
        qmd: {
          /* resolved */
        },
      },
    });

    expect(manager).toBeNull();
  });

  it("parses QMD output correctly", async () => {
    const output = `@@ -100,200
{"id": "chunk1", "score": 0.95}
Content line 1
Content line 2
@@ -300,150
{"id": "chunk2", "score": 0.87}
More content`;

    const results = parseQmdOutput(output, "");

    expect(results).toHaveLength(2);
    expect(results[0].score).toBe(0.95);
    expect(results[0].snippets[0].start).toBe(100);
  });

  it("applies exponential backoff on failures", async () => {
    const manager = await QmdMemoryManager.create({
      /* ... */
    });

    // First failure
    await expect(manager.search("test")).rejects.toThrow();

    // Immediate retry should fail fast (backoff active)
    await expect(manager.search("test")).rejects.toThrow("backoff active");

    // Wait for backoff to expire
    await sleep(QMD_EMBED_BACKOFF_BASE_MS * 2);

    // Retry should attempt again
    // (mock would verify qmd spawn called)
  });
});
```

---

## Performance Considerations

### Stream Processing

- Process output line-by-line instead of buffering entire output
- Reduces memory footprint for large result sets
- Enables early termination if enough results found

### Concurrency Limits

- Serialize embedding operations to avoid overloading qmd
- Queue-based locking prevents concurrent spawns
- Backoff prevents cascade failures under load

### Output Truncation

- Limit output to 200K chars to prevent memory issues
- Preserve JSON structure when truncating
- Mark truncated output clearly

---

## Related Documentation

- [Fallback Manager](/algorithms/memory-search/fallback-manager)
- [Index Manager](/algorithms/memory-search/index-manager)
- [Memory Configuration](/memory/configuration)
- [QMD Scope](/memory/qmd-scope)