# JSFE — JavaScript Flow Engine

ESM TypeScript library for workflow + tool orchestration.

## Install
\`\`\`bash
npm i jsfe
\`\`\`

## 📖 Documentation

- **[JavaScript Flow Engine User Guide](user-guide.md)** 
- Comprehensive tutorials, examples, and best practices
- **[README.md](README.md)** - Technical API reference (this document)
- **[Flow author reference](docs/copilot-reference.md)** - the authoritative engine and JSON authoring specification for flow authors and AI flow generators (shipped in the package; the Flow Designer's CoPilot reads it from the published version, so change it with the engine)
- **[Development candidate acceptance](docs/development-release.md)** - reproducible package and cross-repository verification
- **[System Flows](docs/system-flows.md)** and **[Shopify Flows](docs/shopify-flows.md)** - the bundled flow and tool libraries in `flows/`, step by step

*For detailed tutorials, step-by-step examples, and comprehensive workflow patterns, see the **[User Guide](user-guide.md)**.*

## Usage

```javascript
import { WorkflowEngine } from "jsfe";

// 1. Create the engine
const engine = new WorkflowEngine(
  hostLogger,          // Your logging instance
  aiCallback,          // Your AI communication function
  flowsMenu,           // Array of flow definitions
  toolsRegistry,       // Array of tool definitions
  APPROVED_FUNCTIONS,  // Pre-approved local functions
  globalVariables,     // Optional: Session-wide variables
  validateOnInit,      // Optional: Enable pre-flight validation (default: true)
  language,            // Optional: User's preferred language
  aiTimeOut,           // Optional: AI timeout in milliseconds (default: 2000ms)
  messageRegistry,     // Optional: Custom message templates
  guidanceConfig       // Optional: User assistance configuration
);

// Optional: AI-Powered Intent Detection & Parameter Extraction
// You can manually invoke the engine's AI logic to detect flows and extract parameters
// without executing the flow immediately. This is useful for pre-flight checks or UI hints.
const detectionResult = await engine.detectFlowWithParameters(input);

if (detectionResult?.flow) {
  console.log(`Input triggers flow: ${detectionResult.flow.name}`);
  
  // If the flow definition includes parameters, the AI extracts them from the prompt
  // Example: "Send $50 to Bob" -> { amount: "50", payee: "Bob" }
  if (detectionResult.parameters) {
    console.log(`Extracted parameters:`, detectionResult.parameters);
  }
}

// 2. Initialize a session for each user
let sessionContext = engine.initSession(yourLogger, 'user-123', 'session-456');

// 3. Process user message and update session context
const userEntry = {
  role: 'user',
  content: 'I need help with my account',
};
const updatedSessionContext = await engine.updateActivity(userEntry, sessionContext);
sessionContext = updatedSessionContext; // CRITICAL: Always update your session reference

if (updatedSessionContext.response) {
  // Intent detected and handled - return the workflow response
  // If flow activated you should not proceed to your normal handling
  return updatedSessionContext.response;
}

// 4. Call your normal Generate reply process as usual
const reply = await yourConversationalReply(input);

// 5. Update the engine's context with the generated reply
const assistantEntry = {
  role: 'assistant',
  content: reply, // The reply generated by your process
};
sessionContext = await engine.updateActivity(assistantEntry, sessionContext);

// Return the generated reply to the user
return reply;
```

## Engine Initialization Parameters

The WorkflowEngine constructor accepts the following parameters in order, each serving a specific purpose in the engine's operation:

**1. hostLogger** (Logger | null)
- **Purpose**: Primary logging interface for the host application
- **Requirements**: Must support `.debug()`, `.info()`, `.warn()`, `.error()` methods
- **Usage**: Engine uses this for all operational logging and debugging output
- **Example**: Winston, or custom logger implementation
- **Nullable**: Can be `null` to disable host application logging

**2. aiCallback** (Function)
- **Purpose**: AI communication function for intent detection and response generation
- **Signature**: `async (systemInstruction: string, userMessage: string, jsonSchema?: string) => string` — the optional `jsonSchema`, when supplied by the engine, asks the host to request structured/JSON output conforming to that schema
- **Integration**: Engine calls this function when AI analysis is needed
- **Requirements**: Must return AI response as string, handle errors gracefully
- **Details**: See dedicated AI Callback Function section below

**3. flowsMenu** (FlowDefinition[])
- **Purpose**: Array of available workflow definitions
- **Content**: All workflows that the engine can detect and execute
- **Validation**: Engine validates flow structure during initialization
- **Requirements**: Each flow must have valid id, name, description, and steps
- **Primary Property**: Use `primary: true` to mark user-facing entry point flows vs. helper sub-flows

**4. toolsRegistry** (ToolDefinition[])
- **Purpose**: Array of external tool definitions for CALL-TOOL steps
- **Content**: HTTP APIs, local functions, and mock tools available to workflows
- **Validation**: Parameter schemas validated against OpenAI Function Calling Standard
- **Security**: Tools define their own security levels and authentication requirements

**5. APPROVED_FUNCTIONS** (Object)
- **Purpose**: Secure registry of pre-approved local JavaScript functions
- **Security**: Only functions in this object can be executed by local-type tools
- **Format**: Plain object where keys are function names and values are the actual functions
- **Validation**: Functions must match tool definitions in toolsRegistry

**6. globalVariables** (Record<string, unknown>, optional)
- **Purpose**: Session-wide variables accessible to all workflows
- **Scope**: Available to all flows in the session via variable interpolation
- **Security**: Safe sharing of host application data with workflows
- **Examples**: User ID, session ID, application configuration, environmental data
- **Nature**: **STATIC** - Set during engine initialization, same for all sessions

### Alternative Setting logger: engine.logger = logger
**hostLogger** (Logger)
- **Purpose**: Primary logging interface for the host application
- **Requirements**: Must support `.debug()`, `.info()`, `.warn()`, `.error()` methods
- **Usage**: Engine uses this for all operational logging and debugging output
- **Example**: Winston, or custom logger implementation

```javascript
const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'warn',  // Enable debug logging to trace validation
  format: winston.format.printf(({ level, message }) => {
    return `${level}: ${message}`;
  }),
  transports: [
    new winston.transports.Console()
  ]
});

engine.logger = logger;
```

### Dynamic Session Data: The `cargo` Property

Unlike `globalVariables` which are static and shared across all sessions, each `EngineSessionContext` has a `cargo` property for **dynamic, session-specific data sharing**:

```javascript

**hostLogger** (Logger | null)
- **Purpose**: Primary logging interface for the host application
- **Requirements**: Must support `.debug()`, `.info()`, `.warn()`, `.error()` methods
- **Usage**: Engine uses this for all operational logging and debugging output
- **Example**: Winston, or custom logger implementation
- **Nullable**: Can be `null` to disable host application logging

// After initSession, you can set dynamic session data
let sessionContext = engine.initSession('user-123', 'session-456');

// Set dynamic data that workflows can access
sessionContext.cargo.userPreferences = { theme: 'dark', language: 'en' };
sessionContext.cargo.currentOrder = { id: 'ORD-123', total: 99.99 };
sessionContext.cargo.temporaryData = 'Any dynamic content';

// Workflows can reference cargo data with variable interpolation
// Example in flow step: "Your order {{cargo.currentOrder.id}} total is ${{cargo.currentOrder.total}}"
```

**Key Differences: `globalVariables` vs `cargo`**
- **globalVariables**: Static, engine-wide, set at initialization, same for all sessions
- **cargo**: Dynamic, session-specific, can be modified anytime during conversation
- **Use Cases**: 
  - `globalVariables`: API keys, system config, app metadata
  - `cargo`: User data, conversation state, temporary values, personalization

**7. validateOnInit** (boolean, optional)
- **Purpose**: Enable comprehensive flow and tool validation during initialization
- **Default**: `true` - recommended for development and production
- **Performance**: Set to `false` only in high-performance scenarios with pre-validated flows
- **Output**: Detailed validation reports with errors, warnings, and success metrics
- **Primary Flow Validation**: Only flows marked with `primary: true` are validated as top-level workflows
  - Sub-flows are validated in context when called from primary flows
  - Prevents validation errors for helper flows that depend on parent flow variables
  - Ensures complete validation coverage through deep traversal of flow call graphs

**8. language** (string, optional)
- **Purpose**: User's preferred language for localized messages and prompts
- **Format**: ISO language code ('en', 'es', 'fr', 'de', etc.)
- **Default**: 'en' if not specified
- **Usage**: Engine selects appropriate prompt_xx properties from flow definitions

**9. aiTimeOut** (number, optional)
- **Purpose**: Timeout in milliseconds for AI callback function calls
- **Default**: 2000ms (2 second) if not specified
- **Usage**: Prevents AI calls from hanging indefinitely, providing better reliability
- **Range**: Recommended range 1000-30000ms depending on AI service response times
- **Special Value**: Set to `0` to disable timeout (no time limit on AI calls)
- **Error Handling**: Throws timeout error if AI call exceeds specified duration

```javascript
// Examples of different timeout configurations:

// Fast local AI (short timeout)
const fastEngine = new WorkflowEngine(logger, aiCallback, flows, tools, functions, 
  {}, true, 'en', 500); // 500ms timeout

// Fast cloud AI 
const standardEngine = new WorkflowEngine(logger, aiCallback, flows, tools, functions, 
  {}, true, 'en', 1000); // 1 second timeout

// No timeout limit (not recommended for production)
const noTimeoutEngine = new WorkflowEngine(logger, aiCallback, flows, tools, functions, 
  {}, true, 'en', 0); // 0 = no timeout
```

**10. messageRegistry** (MessageRegistry, optional)
- **Purpose**: Custom message templates for engine-generated user messages
- **Format**: Multi-language message registry with customizable system messages
- **Override**: Allows customization of built-in engine messages
- **Localization**: Supports multiple languages with fallback to default messages

**11. guidanceConfig** (GuidanceConfig, optional)
- **Purpose**: Configuration for user guidance and help messages
- **Features**: Controls how and when the engine provides user assistance
- **Modes**: Append, prepend, template, or none for guidance integration
- **Context**: Different guidance for general vs. payment/financial workflows

### AI Callback Function

The `aiCallback` parameter provides the engine access to your AI system for intent detection and workflow triggering. Here's a minimal implementation example:

```javascript
// Minimal AI callback implementation.
// The engine passes an optional third arg, jsonSchema; when present, request
// structured/JSON output from the model conforming to it.
async function aiCallback(systemInstruction, userMessage, jsonSchema) {
  try {
    const response = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Authorization": `Bearer ${process.env.OPENAI_API_KEY}`
      },
      body: JSON.stringify({
        model: "gpt-4o-mini",
        messages: [
          { role: "system", content: systemInstruction },
          { role: "user", content: userMessage }
        ],
        temperature: 0.1,
        max_tokens: 200
      })
    });

    if (!response.ok) {
      throw new Error(`AI API request failed: ${response.status} ${response.statusText}`);
    }

    const data = await response.json();
    return data.choices[0].message.content.trim();
    
  } catch (error) {
    throw new Error(`AI communication failed: ${error.message}`);
  }
}
```

**AI Callback Interface:**
- **Input**: `systemInstruction` (string), `userMessage` (string), optional `jsonSchema` (string), and
  optional `request` (object — intent detection only, see below)
- **Output**: AI response as a string
- **Purpose**: Analyzes user input to detect workflow intents and generate responses
- **Integration**: The engine calls this function when it needs AI analysis for intent detection

**How the Engine Generates Input Arguments:**
- **`systemInstruction`**: Dynamically generated by the engine based on:
  - Available flow definitions and their descriptions
  - Current session context and active flows
  - Current conversation state and collected variables
- **`userMessage`**: Intelligently composed by the engine, including:
  - The actual user input/prompt
  - Relevant contextual information from the conversation
  - Session state and variables needed for intent analysis

Both parameters are carefully engineered by the engine to work together for optimal intent detection. The engine automatically constructs comprehensive, context-aware prompts that provide the AI with all necessary information for accurate workflow selection and response generation. Your aiCallback implementation only needs to send these pre-constructed arguments to your AI service and return the response.

**The structured `request` argument (intent detection only):**
For intent detection — and only there — the engine passes a 4th argument: the same call in
structured form, so a host can answer it with a classifier instead of a text model.

```javascript
async function aiCallback(systemInstruction, userMessage, jsonSchema, request) {
  if (request?.task === 'detect_flow') {
    // request.input         — the user input, exactly as in <user-input>
    // request.conversation  — the last user/assistant turn: [{ role, content }], oldest first; [] when none
    // request.flows         — [{ id, name, description, parameters: [{ name, description, type?, enum? }] }]
    // Reply exactly as the text path does: a JSON string { flowName, parameters }, flowName being a
    // flow's name (or "None"). The engine validates it the same way, enums included.
  }
  // ... every other call, and the text path: unchanged
}
```

`systemInstruction` and `userMessage` are still the complete text prompt on that call, so a host can
run its text model alongside the classifier (for example, for free-text parameters). A callback that
ignores the 4th argument behaves exactly as before; every other call still receives three arguments.

**Host helpers — `jsfe/host`:**
Pure functions for the host side of `aiCallback`, so a production host and a local test harness
share one implementation. Loading `jsfe/host` never loads the engine.

```javascript
import { buildAiRequest, isDetectFlowRequest, toFlowRequest, fromFlowAnswers } from 'jsfe/host';

// An OpenAI-compatible chat-completions body from a model description. It picks the right
// response_format for each kind of jsonSchema argument, and never sends `response_format: null`.
const body = buildAiRequest(
  { name: 'gpt-4o-mini', reservedReplySize: 1024, supportsSchema: true },
  systemInstruction, userMessage, jsonSchema);
```

- `buildAiRequest(model, systemInstruction, userMessage, jsonSchema)`, `parseSchemaArg`, `schemaName`.
  `model` fields: `name`, `reservedReplySize`, and the optional `tokenParam`, `supportsSchema`,
  `fixedSampling` and `extraParams`.
- `isDetectFlowRequest`, `toFlowRequest` / `fromFlowAnswers` (which flow), `hasJevParameters` /
  `toParameterRequest` / `fromParameterAnswers` (its enum and boolean parameters),
  `freeTextParameters`, `mergeTextParameters`, `INTENT_YES_THRESHOLD`: these map the structured
  intent-detection request to TypeSafe System One and back. The host sends the requests.
  - Stage 1 asks one yes/no probability per flow and picks the highest if it reaches
    `INTENT_YES_THRESHOLD` (0.5), else `"None"`. Stage 2, for the chosen flow only, asks one per enum
    value and one per boolean parameter. Below the threshold a parameter is omitted (the flow asks for
    it), never set to `false`.
  - Any question that comes back unanswered makes the decoder **throw**, so the host can fall back
    to a text model rather than read a missing answer as "no".
  - `mergeTextParameters` adds a text model's free-text parameters only when the text model chose
    the same flow. The classifier's enum and boolean values always win.

**Two-stage intent detection (opt-in: `engine.intentDetection = 'two-stage'`):**
Stage 1 picks which PRIMARY flow, if any, the message starts, reading only each primary's `description`.
Stage 2 extracts the selected flow's parameters, reading each parameter's name, type, `enum` and `description`,
plus the flow's optional `parameterInstructions` (rules that span its parameters). A parameter marked
`internal: true` is the flow's own and is never asked. Each stage is ONE aiCallback call: the positional
arguments are the LLM instructions, and the 4th argument (`task: 'detect_flow'` with `stage: 1`, or
`task: 'extract_parameters'` with `stage: 2`) carries both executors' instructions, `llm` and `systemOne`.
The host runs either, or both as a hedge, and replies with the canonical JSON: `{ "flowName": … }` for
stage 1, `{ "parameters": { … } }` for stage 2. `jsfe/host` exports `buildFlowStage`, `buildParameterStage`
and the decoders (`decodeFlowLlm`, `decodeFlowSystemOne`, `decodeParametersLlm`, `decodeParametersSystemOne`,
`isNearMiss`), so a host never interprets an answer itself. Only primaries are ever activated by detection;
a stage-2 failure starts the flow without its parameters (it asks for them). The default, `'single-call'`,
is the original single call.

**Host start (`ContextEntry.start`):** `{ role: 'user', content, start: { flow, arguments } }` starts that
flow (any flow, by id or name) with those arguments on this turn, without asking the AI. Only the host sets it;
it is never derived from what the user typed.

**Flow helpers — `jsfe/helpers`:**
The functions the bundled system flows (`ask-choice`, `collect-input`, `authenticate-user`…) call from
expressions, so every host runs the same ones. Pure: no credentials, no network. The engine does not
register them by itself; a host lists them among its approved functions:

```javascript
import { STANDARD_FUNCTIONS } from 'jsfe/helpers';

const engine = new WorkflowEngine(logger, aiCallback, flows, tools,
  { ...STANDARD_FUNCTIONS, sendSMSOTP, sendEmail /* the host's own, credentialed functions */ });
```

- Reading answers: `matchesChoice`, `pickChoice`, `isAnswer`, `isAgentRequest`, `isOtherRequest`,
  `isSpelledFragment`, `isAmountAnswer`, `amountInAnswer`, `readEmail`, `readPhone`,
  `readInternationalPhone`, `readIdentifier`, `normalizeAndFindCapture`, and `matchInputs` (which of a
  form's inputs an answer fills; ties between inputs of the same form are returned for the flow to ask).
- Saying values: `amountToSpeech`, `dateToWords`, `last4`, `maskEmail`, `urlToSpeech`,
  `textWithUrlToSpeech`, `anythingElse`, `goodbye`.
- Checks: `validateDigits`, `validatePhone`, `validateEmail`, `validateCardNumber`,
  `validateExpiration`, and `validateOTP(code, container)`, which accepts a code whose SHA-256 matches
  `container.otpHash` within 10 minutes of `container.otpTimestamp` and leaves the reason for a refusal
  in `container.otpFailure` (`missing`, `expired` or `mismatch`). Sending the code is the host's job.
- Session-aware helpers read only the cargo a caller passes: `callerTextNumber`, `textDestination`,
  `callerIsInternational`.

`amountInAnswer` and `isAmountAnswer` retain their legacy parsing behavior. Explicit USD parsing uses
`readMoneyAmount(text, {currency:'USD',cardinals:true,intent:'payment'})` or `isMoneyAmount` with the
same options. `cardinals` defaults to false and `intent` is optional. FORM amount profiles opt in with
`{kind:'amount',money:{currency:'USD',cardinals:true,intent:'payment'}}`; ordinary amount profiles retain
legacy parsing. No currency conversion is inferred.

**Alternative AI Services:**
You can integrate any AI service (Claude, Gemini, local LLMs, etc.) by implementing this same interface. The engine only requires a function that takes system instructions and user input, then returns an AI response.

### Core Registries

The engine operates through four primary registries that define its capabilities:

#### 1. **Flows Registry** - Workflow Definitions

```javascript
const flowsMenu = [
  {
    id: "payment-workflow",
    name: "ProcessPayment", 
    prompt: "Process a payment",
    description: "Handle payment processing with validation",
    primary: true, // Optional: Marks this as a primary (user-facing) flow
    steps: [
      { type: "SAY", value: "Let's process your payment." },
      { type: "SAY-GET", variable: "amount", value: "Enter amount:" },
      { type: "CALL-TOOL", tool: "PaymentProcessor", args: {...}, variable: "payment_result", }
    ]
  }
];
```

##### Flow Definition Properties

- **`id`** (required): Unique identifier for the flow
- **`name`** (required): Human-readable flow name used in execution context
- **`prompt`** (required): User-facing description for AI intent detection
- **`description`** (required): Detailed description of the flow's purpose
- **`primary`** (optional): Boolean flag marking user-facing entry point flows
  - `true`: Primary flows are standalone workflows that users can directly trigger
  - `false` or omitted: Helper/sub-flows called by other flows, not directly accessible
  - **Validation Impact**: Only primary flows are validated as top-level workflows during initialization
  - **AI Detection**: Only primary flows are considered for intent detection and user interaction
- **`parameters`** (optional): Values intent detection may extract from the triggering input and
  inject as flow variables. Each is `{ name, description, type?, enum? }`:
  - `type` defaults to `"string"`.
  - `enum` (string parameters only) lists the allowed values. The prompt shows them
    (`ticket_type (string, one of: "Fraud" | "Cancellation")`), and the engine enforces them on the
    reply: a value matching ignoring case and surrounding whitespace is replaced by the declared
    spelling, and any other value is **dropped** with a warning — so the flow asks for it instead of
    running on a value it never declared. Validation rejects an empty enum, a non-string or padded
    value, a value repeated ignoring case, and an enum on a non-string type.
  - Declare an enum wherever the allowed values are a closed set. It also lets a host answer intent
    detection with a classifier instead of a text model (see the `request` argument of `aiCallback`).
  - Validation also requires `parameters` to be an array, every parameter to have a non-empty
    `name`, and names to be unique within the flow.
- **`scope`** (optional): `"isolated"` makes the flow a self-contained worker: a call (or replace) starts it with only its declared `variables` and the `parameters` actually passed — an omitted parameter is undefined, never the caller's or a previous call's value — and its variables never reach the caller (return results through `cargo`). Default `"shared"`: the caller's variables, by reference.
- **`steps`** (required): Array of workflow steps to execute
- **`variables`** (optional): Flow-specific variable definitions with types and descriptions
- **`version`** (optional): Flow version for compatibility tracking (defaults to "1.0")

Mutable object defaults and authored step data are copied for each activation; changing one session's
default data does not mutate a flow definition or another activation. This does not change the shared
variable-container behavior above. A shared `FLOW` step with `callType:'replace'` preserves that container
and does not initialize target defaults unless it declares `initializeDefaults:true`; that opt-in fills
only undefined defaults before explicit parameters. Isolated replacement initializes its own scope.
`SET` steps may declare `requireTrue:true` to throw `JSFERequiredSetFailed` unless their result is literal
true; absent or false keeps ordinary SET behavior.

##### Primary vs. Sub-Flow Architecture

The `primary` property enables a clean separation between:

**Primary Flows** (`primary: true`):
- User-facing workflows that can be directly triggered by user input
- Entry points for specific business processes (e.g., "StartPayment", "CreateTicket")
- Included in AI intent detection and flow selection
- Validated as standalone workflows during engine initialization

**Sub-Flows** (no `primary` property or `primary: false`):
- Helper workflows called by other flows (e.g., "RetryPayment", "ValidateInput")
- Internal logic flows for error handling, validation, or shared functionality
- Not directly accessible to users through intent detection
- Validated only when called from primary flows, ensuring proper variable context

This architecture prevents validation errors for sub-flows that depend on variables from their calling flows, while ensuring complete validation coverage during the engine's deep validation process.

##### Thin wrappers: `wraps`

A shared flow file (one set of flows loaded by many tenants) can be given a tenant-specific entry point without copying it.
A flow with `wraps: "<id or name>"` becomes the entry point in the wrapped flow's place:

- the wrapped flow is no longer primary, and stays callable as a worker (a FLOW step, or its exact name);
- the wrapper takes the wrapped flow's `description`, `parameters`, prompts and `primary` flag unless it declares its own,
  so intent detection reads one description, kept in one place;
- the wrapper's steps do the tenant's own work, then call the wrapped flow with arguments of their own.

```json
{ "id": "curacao-track-order", "name": "CuracaoTrackOrder", "version": "1.0.0", "wraps": "shopify-track-order",
  "steps": [
    { "id": "lookup", "type": "FLOW", "value": "caller-on-file", "callType": "call" },
    { "id": "track", "type": "FLOW", "value": "shopify-track-order", "callType": "replace",
      "parameters": { "caller_name": "{{caller_name}}", "order_number": "{{order_number}}" } }
  ] }
```

A `wraps` that names no flow in the menu is logged and ignored. Without `wraps`, nothing changes.
#### Referencing Tool Results in Later Steps
When a CALL-TOOL step finishes, it can store the returned data into a variable you name via the variable property. That variable will hold the entire return object from the tool.

```javascript
Using variable
{
  "type": "CALL-TOOL",
  "tool": "CreateSupportTicket",
  "variable": "ticket_result",
  "args": {
    "subject": "{{subject}}",
    "description": "{{description}}",
    "customer_email": "{{customer_email}}"
  }
},
{
  "type": "SAY",
  "value": "Ticket created: {{ticket_result.ticket.id}} — we'll email updates to {{ticket_result.ticket.customer_email}}."
}
```
##### Important:
Whatever your tool returns becomes the value of the variable you specify.
Because you get the raw return object, you do not need to use .result in your template paths—just reference the keys the tool returns (ticket_result.ticket.id, ticket_result.ok, etc.).
If you omit variable, you won’t be able to access the tool’s output later.

#### 2. **Tools Registry** - External Integrations

```javascript
const toolsRegistry = [
  {
    id: "PaymentProcessor",
    name: "Process Payment",
    description: "Processes financial transactions securely",
    parameters: { /* OpenAI Function Calling Standard Schema */ },
    implementation: {
      type: "local", // or "http" for REST APIs
      function: "processPayment",
      timeout: 10000
    },
    security: {
      requiresAuth: true,
      auditLevel: "critical",
      dataClassification: "financial"
    },
    // Optional output contract: the shape of what the CALL-TOOL `variable` receives (after
    // responseMapping, if any). Lets flow authors and AI co-pilots read result paths
    // (payment_result.transaction.id) from the definition instead of guessing them.
    returns: {
      type: "object",
      required: ["success"],
      properties: {
        success: { type: "boolean" },
        transaction: { type: "object", properties: { id: { type: "string" } } }
      }
    }
  }
];
```

**`returns` is optional and never changes behaviour.** A tool with or without it runs identically.
A host can opt in to a diagnostic, `engine.validateToolReturns = true` (default `false`), which checks
each result against `returns` and logs a warning on a mismatch or on a schema that does not compile.
The result is still delivered unchanged either way.

A second opt-in, `engine.logToolFailures = true` (default `false`), logs one warning when a tool *returns*
a failure (`success: false`, or a non-zero `error`): `Tool <id> answered a failure (<code>): <message>`, with digit
runs of four or more masked. Such a result is not an exception, so nothing else reports it, notably an HTTP
tool whose `acceptStatus` turns a 4xx into an answer. The result is delivered unchanged.

#### 3. **Approved Functions Registry** - Secure Local Functions

```javascript
const APPROVED_FUNCTIONS = {};

// Define secure local functions
async function processPayment(args) {
  // Secure payment processing logic
  return { transactionId: "...", status: "success" };
}

// Register approved functions
APPROVED_FUNCTIONS['processPayment'] = processPayment;
```

#### 4. **Global Variables** - Secure Sharing of Local Data
```javascript
const globalVariables = {
  caller_id: "(555) 123-4567",
  caller_name: "John Doe", 
  thread_id: "conversation-123"
};
```

### Session Management

- Each user requires a unique session context via `initSession(userId, sessionId)`
- **CRITICAL**: `updateActivity()` returns an updated `EngineSessionContext` 
- You must always update your session reference on every call
- The `EngineSessionContext` object should be persisted by your application
- Pass the same session context to `updateActivity` for conversation continuity
- **Session Isolation**: Each user/session must have its own context to prevent state contamination

#### Session Cargo for Dynamic Data

Each session context includes a `cargo` property for dynamic, session-specific data that workflows can access:

```javascript
// Initialize session
let sessionContext = engine.initSession('user-123', 'session-456');

// Set dynamic session data that workflows can reference
sessionContext.cargo.userProfile = {
  name: 'John Doe',
  tier: 'premium',
  preferences: { notifications: true }
};

// Workflows can access cargo data: "Welcome {{cargo.userProfile.name}}!"
sessionContext = await engine.updateActivity(userEntry, sessionContext);
```

#### Voice Sessions: `cargo.voice` and `cargo.digits`

Two cargo fields connect the engine to a voice channel:

- **`cargo.voice`** (set by the host): when truthy, the answer a `SAY-GET` collects into its
  `variable` is cleaned by `aiCallback` before it is stored. The cleaner receives the question just asked and the caller's
  words. It removes filler words, maps yes/no answers to `"yes"` / `"no"`, and writes spoken
  numbers, phone numbers and emails in their written form ("john dot doe at gmail dot com" →
  `john.doe@gmail.com`). It **never returns an email address or phone number that appears in the
  question**: when a prompt reads a failed value back ("I could not verify the email
  pinkas@example.com…"), the caller's answer is treated as a correction, and a near-miss such as
  `rpinkas@…` is kept as said. If the cleanup fails or returns nothing, the raw answer is used.
- **`cargo.digits`** (set by the engine): when a `SAY-GET` step declares `digits` (for example
  `{ "min": 6, "max": 6 }`), the engine copies it to `cargo.digits` while the answer is awaited, so
  the host can configure keypad (DTMF) collection. It is removed once the input is collected. A string value
  holding a `{{path}}` is filled from the flow's variables, as FLOW parameters are (a path, not an expression;
  an unknown path gives `''`), so a generic step can take its keypad settings from the calling flow
  (`"complete": "{{shapes}}"`); a step without templates is passed exactly as declared (jsfe ≥ 1.3.4). Keys the
  engine does not read are passed to the host unchanged.
  On voice, `digits` also shapes how the answer is stored: an answer whose digit count fits the range is
  stored as its digits, taken as heard and never sent to the voice cleaner ("es 41 34 18…" → `413418…`). On a
  one-key question (`max: 1`) only an answer that is exactly one key (a digit, `*` or `#`, punctuation aside)
  becomes that key; any other answer is stored as spoken and never sent to the cleaner, so a digit inside a
  sentence ("I want to pay 1") is never a key press (jsfe ≥ 1.3.1).
- **`asSpoken`** (a `SAY-GET` attribute, opt-in): the answer is stored exactly as given, never reduced to
  its digits and never sent to the voice cleaner. Use it where the words carry meaning the digits lose,
  such as an amount: without it, "$1" fitting `digits: { min: 1, max: 1 }` is stored as `1`. The value is
  `true`, or a condition on the flow's variables (`"ac_amount === true"`), so a worker flow can switch it
  with a parameter. Absent or false keeps the default behaviour.
- **`pii`** (a `SAY-GET` attribute, opt-in, jsfe ≥ 1.3.5): the answer is sensitive, such as a card's security
  code. The engine never writes it to its log, and while the question waits its frame carries
  `pendingVariablePii: true`, so the host can keep the next input out of its own logs and records (the host
  reads the active frame before the turn, as it reads `pendingVariable`). `true`, or a condition on the flow's
  variables, as `asSpoken`. It changes nothing about how the answer is stored or used.

#### Correct Session Update Pattern

```javascript
// Initialize session
let sessionContext = engine.initSession('user-123', 'session-456');

// For every updateActivity call, capture the returned context
sessionContext = await engine.updateActivity(userEntry, sessionContext);

// Check for workflow response
if (sessionContext.response) {
  return sessionContext.response;
}


// Continue with regular conversation...
sessionContext = await engine.updateActivity(assistantEntry, sessionContext);
```

#### ❌ Common Mistake - Not Updating Session Reference
```javascript
// WRONG - This will cause session corruption
const sessionContext = engine.initSession('user-123', 'session-456');
await engine.updateActivity(userEntry, sessionContext); // Session state lost!
```

#### ✅ Correct Pattern - Always Update Session Reference
```javascript
// CORRECT - Session state maintained
let sessionContext = engine.initSession('user-123', 'session-456');
sessionContext = await engine.updateActivity(userEntry, sessionContext); // Session updated
```

### Context Entry Types

- User input: `contextEntry.role = 'user'` - analyzed and may trigger flow execution
- Assistant response: `contextEntry.role = 'assistant'` - added to context for awareness

### ContextEntry Structure

```typescript
interface ContextEntry {
  role: 'user' | 'assistant' | 'system' | 'tool';  // Message role type
  content: string | Record<string, unknown>;       // Message content (text, object, etc.)
  timestamp: number;                               // Unix timestamp in milliseconds
  stepId?: string;                                 // Optional: Used by the system when a Step calls a Tool
  toolName?: string;                               // Optional: Used by the system to record Tool result into the chat context
  metadata?: Record<string, unknown>;              // Optional: Additional context data
}
```

## UpdateActivity Method

The `updateActivity` method is the primary interface for processing user inputs and assistant responses. It returns an updated `EngineSessionContext` that must be captured and used for subsequent calls.

### Method Signature

```typescript
async updateActivity(
  contextEntry: ContextEntry, 
  engineSessionContext: EngineSessionContext
): Promise<EngineSessionContext>
```

### Return Value

The method returns an updated `EngineSessionContext` containing:

- **Updated flow state**: Modified flow stacks, variables, and execution context
- **Response data**: If a workflow was triggered, `sessionContext.response` contains the result
- **Session metadata**: Updated timestamps, accumulated messages, and transaction data
- **`lastTurnToolCalls`**: every tool call this `updateActivity` attempted (see
  [Host Responsibilities](#host-responsibilities-tool-calls-cancellation-and-concurrency))
- **`lastFlowOutcome`**: the declared outcome of a flow that terminated during this call, if any
  (see [Declared flow outcomes](#declared-flow-outcomes-lastflowoutcome))
- **`lastTurnDispatch`**: `{ fromFlow, matchedFlow }` when a `DISPATCH` step re-routed this call's
  input (`matchedFlow` is `null` when no flow matched and the response is `null`); one-shot

### Integration Pattern

```javascript
// Process user input
let sessionContext = await engine.updateActivity({
  role: 'user',
  content: userMessage,
  timestamp: Date.now()
}, sessionContext);

// Check if a workflow was activated
if (sessionContext.response) {
  // Workflow handled the input - return the workflow response
  return sessionContext.response;
}

// No workflow triggered - proceed with normal conversation
const aiResponse = await yourAIFunction(userMessage);

// Update context with AI response
sessionContext = await engine.updateActivity({
  role: 'assistant', 
  content: aiResponse,
  timestamp: Date.now()
}, sessionContext);

return aiResponse;
```

### Critical Notes

- **Always capture the return value**: Failing to update your session reference will cause state corruption
- **Session isolation**: Each user/conversation needs its own session context
- **Response checking**: Check `sessionContext.response` to determine if a workflow handled the input
- **Persistence**: Your application should persist the updated session context between requests
- **`JSFEActiveFlowEmptyResponse`**: if a flow is still active but its last step produced no output
  and was not a `RETURN`, `updateActivity` resets the session's flow stacks (cargo and global
  variables are kept), clears pending messages, and throws an error with this `name`. The session
  is already repaired: the next turn starts with fresh intent detection. Log or record the error;
  no rollback is needed. A deliberate `RETURN ''` (which ends every flow) is not flagged.

Native FORM collection in 1.7.2 acknowledges an unreadable answer once, then yields the current
utterance while retaining its draft. Explicit recovery metadata distinguishes resumption, required
restart, unknown-action hold and no promised recovery; business completion remains independent.
See the [retry/yield/recovery contract](docs/copilot-reference.md#retry-yield-and-recovery).

### Host Responsibilities: Tool Calls, Cancellation and Concurrency

A flow can take actions in the outside world through `CALL-TOOL` — charge a card, send an SMS,
create a ticket. For ordinary `CALL-TOOL` flows, **the session returned by `updateActivity` records
which steps already ran**; the host owns its persistence. Native performers can additionally use an
explicitly attached durable action journal (the native default). An explicit
`engine.actionPersistence = 'session-only'` opt-out retains confirmation and saved session action guards
without journal storage. It cannot prevent a repeated charge after an effect succeeds and its updated
session save is lost. The reference host selects it with `jsfe: {actionPersistence:'session-only'}`;
`durableActions:true` conflicts with that setting. Keep each existing conversation on its original mode.
Neither policy removes the host responsibilities below.
Getting these rules wrong can repeat an action the user asked for once.

The reference host defaults `config.jsfe` to `{durableActions:false,sessionEncoding:'plain'}`. Durable
journaling and `sessionEncoding:'br-v1'` conversation writes are separate opt-ins. Encoded storage
requires a reader-first rollout and a lossless conversion before an older-reader rollback. Durable
recovery requires matching runtime/definition identities; a missing or different identity holds recovery
without replaying effects or acknowledging receipts. See [development release boundaries](docs/development-release.md)
for exact policy keys, storage limits and rollback requirements. Passing local tests does not approve deployment.

This is not hypothetical. In production a caller confirmed a payment and repeated "confirm" a
second later. The host cancelled the first turn while its charge was in flight and restored the
session it had snapshotted before the turn. The flow was back at "Say CONFIRM to pay", the
repeated word confirmed it again, and the card was charged twice.
`tests/host-tool-calls.test.mjs` reproduces it.

**1. Never restore a pre-turn session after the turn attempted a tool.**
`engineSessionContext.lastTurnToolCalls` lists every tool call the most recent `updateActivity`
attempted. A host may discard or roll back a turn's session only when that list is empty. When it
is not, persist exactly the session `updateActivity` returned, even if the turn's reply is never
delivered.

**2. A cancelled turn is still running — wait for it.** Cancelling on the host side (the user
barged in, a request was abandoned, a host timeout fired) does not stop the engine or a tool it
is calling. Always `await` the in-flight `updateActivity` before persisting or discarding the
session, and before passing that session to another `updateActivity` call.

**3. One `updateActivity` at a time per conversation, and per engine instance.** The engine keeps
a reference to the session it is processing. Two concurrent calls on the same conversation both
start from the pre-call state and both execute its next step, and concurrent calls on one engine
instance corrupt each other's session reference. Serialise per conversation (a lock, a queue),
and make a newer turn wait for an older turn's `updateActivity` to finish rather than proceed
after a fixed timeout.

**4. Make external side effects idempotent.** The rules above close every gap the engine and host
can see. They cannot close one: a host process that dies after a tool took effect but before the
session was persisted. For actions that must never repeat, such as payments, pass an idempotency
key to the external service (for example a transaction id the flow generates once and stores in a
variable) so a repeat is rejected at the source.

**5. Log what `updateActivity` throws — the engine does not.** An engine AI call that fails
(an `aiTimeOut`, a vendor error) reaches the host as `AI task processing failed: AI communication
failed: <cause>`, and every layer it passes through traces it at `debug` only, so one failure is
one line — the host's, at the level its outcome deserves (a fallback answer: warn; a
`JSFEExecutionError` the user hears: error). Since 0.9.94; before it, one timeout wrote six
warn/error lines inside the engine. `tests/ai-failure-logging.test.mjs` pins it.

#### `lastTurnToolCalls`

```typescript
interface ToolCallRecord {
  tool: string;              // tool name
  implementation: string;    // 'local' | 'http' | ...
  transactionId: string | null;
  at: number;                // epoch ms
}
```

- **One-shot.** Reset at the start of every `updateActivity` call, user or assistant role, so it
  describes that call only. Read it before making the next `updateActivity` call.
- **Attempts, not successes.** A record is written before the tool is invoked, including retries
  and calls that then throw or time out, because those may still have taken effect remotely.
- **Covers every `CALL-TOOL` step.** Local functions (`APPROVED_FUNCTIONS`) and HTTP tools pass
  through the same entry point.
- **Only calls that were attempted.** A call refused before invocation (argument validation,
  rate limiting) is not recorded, because nothing was sent.
- **Does NOT cover functions called from expressions.** `APPROVED_FUNCTIONS` are also callable
  inside templates, `SET` values, conditions and `customValidator`, and those calls are not
  recorded. This is deliberate: expressions call formatting and matching helpers on nearly every
  turn, and recording them would make every turn look like it acted, so a host could never discard
  a turn at all. **Anything with an external side effect must be invoked through `CALL-TOOL`**,
  never from an expression, or the host cannot see it.

```javascript
const before = structuredClone(session);
// Awaited even when the turn is being cancelled: the engine and its tools keep running.
const after = await engine.updateActivity({ role: 'user', content: input }, session);

if (turnWasCancelled && after.lastTurnToolCalls.length === 0) {
  session = before;      // nothing happened in the world: safe to discard an unheard prompt
} else {
  session = after;       // a tool may have run: keep the record of it
}
await persist(session);
```

### Declared flow outcomes: `lastFlowOutcome`

A flow can say how it ended, not just that it ended. An optional `outcome` (and `reason`) on a step
is remembered on the flow and applied when the flow terminates. Put it on the step that ends the
flow: a `RETURN`, an `END`, a `DISPATCH` or a final `SAY`. Both may use `{{…}}` interpolation with
the flow's variables; an `outcome` that interpolates to empty declares nothing:

```json
{ "id": "say_support_message", "type": "SAY", "value": "Sorry I couldn't help! ...",
  "outcome": "unresolved", "reason": "contact_support_fallback" }
```

- `"unresolved"`: the flow ended without resolving the user's need. Its transaction is marked
  **failed** (reason = `reason`, or `"unresolved"`), so audits count it as a failure, not a success.
- `"deflected"`: a decoy flow absorbed an intent-detection false positive by design. The
  transaction completes; do not treat it as a user struggle.
- `"resolved"` or any other value: informational; the transaction completes.
- A flow that declares nothing completes exactly as before.

The engine also stamps `engineSessionContext.lastFlowOutcome` for the host:

```typescript
interface FlowOutcome {
  flowName: string;
  outcome: string;
  reason?: string;
  endedBy: 'return' | 'completion' | 'dispatch'; // RETURN ended all flows; completion = ran out of steps (incl. after END); dispatch = a DISPATCH ended them and no flow matched
}
```

A `DISPATCH` that starts a flow swallows the ended flows' outcome; one that matches nothing keeps it.

It is **one-shot**: cleared at the start of every `updateActivity`, and set only when a flow that
declared an outcome terminated during that call. Hosts use it to run a recovery turn after an
`"unresolved"` ending. When a `RETURN` terminates several flows, only the flow that executed it
carries the RETURN's declaration; the others finalize with their own (usually none).

## Architecture Overview

### Stack-of-Stacks Design

The Flow Engine implements a sophisticated "stack-of-stacks" architecture that allows flows to be suspended
and resumed, enabling users to naturally interrupt one workflow to handle another task, then return to
their original workflow seamlessly.

#### Core Components

1. **Multiple Independent Flow Execution Stacks**
   - Active stack index tracks current execution context
   - Automatic stack switching for flow interruption/resumption
   - Proper isolation between different workflow contexts

2. **Flow Frame Structure (Runtime Execution Context)**
   Each flow execution maintains a complete context frame:
   ```typescript
   interface FlowFrame {
     flowName: string;                              // Human-readable flow name (from FlowDefinition.name)
     flowId: string;                                // Unique flow identifier (from FlowDefinition.id)
     flowVersion: string;                           // Flow version for compatibility (from FlowDefinition.version)
     flowStepsStack: FlowStep[];                    // Remaining steps (reversed for efficient pop)
     contextStack: ContextEntry[];                  // Complete interaction history with role info
     inputStack: unknown[];                         // Current input context for step execution
     variables: Record<string, unknown>;            // Unified variable storage (shared across sub-flows)
     transaction: TransactionObj;                   // Comprehensive transaction and audit tracking
     userId: string;                                // User identifier for this flow session
     startTime: number;                             // Flow start timestamp for timing analysis
     pendingVariable?: string;                      // Variable name awaiting user input (SAY-GET steps)
     lastSayMessage?: string;                       // Last SAY step output for context
     pendingInterruption?: Record<string, unknown>; // Interruption state management
     accumulatedMessages?: string[];                // Accumulated SAY messages for batching
     declaredOutcome?: { outcome: string; reason?: string }; // From a step's `outcome`, applied when the frame terminates
     parentTransaction?: string;                    // Parent transaction ID for sub-flow tracking
     justResumed?: boolean;                         // Flag indicating flow was just resumed
   }
   ```
   
   **Technical Implementation Details:**
   - **Flow Identity**: `flowName`, `flowId`, `flowVersion` are copied from the FlowDefinition
   - **Dynamic Properties**: Flow `prompt`, `description`, and localized prompts are accessed dynamically from FlowDefinition
   - **flowStepsStack**: Steps stored in reverse order for efficient pop operations
   - **contextStack**: Enhanced with role information for complete conversation context
   - **variables**: Flat storage shared across sub-flows for seamless data passing
   - **accumulatedMessages**: SAY steps are batched for efficient output
   - **justResumed**: Helps engine provide appropriate resumption messages
   ```

3. **Helper Function Architecture**
   All stack operations go through centralized helper functions:
   - `initializeFlowStacks(engine)`: Ensures proper structure
   - `getCurrentStack(engine)`: Gets currently active stack
   - `pushToCurrentStack(engine, frame)`: Adds flow to active stack
   - `popFromCurrentStack(engine)`: Removes flow from active stack

4. **Tool Definition Structure (Runtime Integration Context)**
   Tools provide external capabilities with comprehensive security and validation:
   ```typescript
   interface ToolDefinition {
     id: string;                                    // Unique tool identifier
     name: string;                                  // Human-readable tool name
     description: string;                           // Tool functionality description
     
     parameters?: {                                 // OpenAI Function Calling Standard schema
       type: string;                                // Parameter type structure
       properties?: Record<string, PropertySchema>; // Parameter validation rules
       required?: string[];                         // Required parameter names
       additionalProperties?: boolean;              // Additional parameter handling
     };
     
     implementation?: {                             // Execution configuration
       type: 'local' | 'http';                      // Implementation type
       function?: string;                           // Local function name (APPROVED_FUNCTIONS)
       url?: string;                                // HTTP endpoint with {param} placeholders
       method?: HttpMethod;                         // HTTP method
       pathParams?: string[];                       // URL parameter substitution
       queryParams?: string[];                      // Query string parameters
       responseMapping?: MappingConfig;             // Response transformation config
       timeout?: number;                            // Request timeout
       retries?: number;                            // Retry attempts
       acceptStatus?: number[];                     // Non-2xx statuses whose JSON body is the result (e.g. [400, 404])
     };
     
     security?: {                                   // Security controls
       rateLimit?: {                                // Rate limiting configuration
         requests: number;                          // Max requests per window
         window: number;                            // Time window in milliseconds
       };
     };
     apiKey?: string;                               // Authentication token
     riskLevel?: 'low' | 'medium' | 'high';         // Security classification
     returns?: object;                              // JSON Schema of the result the flow receives (optional)
   }
   ```
   
   **Technical Implementation Details:**
   - **Parameter Validation**: JSON Schema validation with ajv for type safety
   - **Local Function Execution**: Secure execution through APPROVED_FUNCTIONS registry
   - **HTTP Integration**: Full REST API support with authentication and retries
   - **Response Mapping**: Declarative transformation without code injection
   - **Security Controls**: Rate limiting, risk classification, audit logging
   - **Error Handling**: Automatic retry logic and graceful degradation

5. **Response Mapping Configuration (MappingConfig Interface)**
   The comprehensive response mapping system supports multiple transformation types:
   ```typescript
   export type MappingConfig =
     | JsonPathMappingConfig
     | ObjectMappingConfig  
     | ArrayMappingConfig
     | TemplateMappingConfig
     | ConditionalMappingConfig
     | PathConfig
     | string
     | Record<string, unknown>;

   // JSONPath-based field extraction and transformation
   export type JsonPathMappingConfig = {
     type: 'jsonPath';
     mappings: Record<string, {
       path: string;                                   // JSONPath expression for data extraction
       transform?: ValueTransformConfig;               // Optional value transformation
       fallback?: unknown;                             // Fallback value if path not found
     }>;
     strict?: boolean;                                 // Strict mode validation
   };

   // Object structure mapping with nested support
   export type ObjectMappingConfig = {
     type: 'object';
     mappings: Record<string, string | PathConfig | MappingConfig | object>;
     strict?: boolean;                                 // Strict mode validation
   };

   // Array processing with filtering, sorting, and pagination
   export type ArrayMappingConfig = {
     type: 'array';
     source?: string;                                  // Source array path
     filter?: ConditionConfig;                         // Filtering conditions
     itemMapping?: MappingConfig;                      // Per-item transformation
     sort?: { field: string; order?: 'asc' | 'desc' }; // Sorting configuration
     offset?: number;                                  // Pagination offset
     limit?: number;                                   // Pagination limit  
     fallback?: unknown[];                             // Fallback array if source not found
   };

   // Template-based string generation with variable substitution
   export type TemplateMappingConfig = {
     type: 'template';
     template: string;                                 // Template string with {{variable}} placeholders
     dataPath?: string;                                // Optional path to resolve template data from
   };

   // Conditional logic-based mapping selection
   export type ConditionalMappingConfig = {
     type: 'conditional';
     conditions: Array<{
       if: ConditionConfig;                            // Condition evaluation
       then: MappingConfig;                            // Mapping to apply if condition true
     }>;
     else?: MappingConfig;                             // Default mapping if no conditions match
   };

   // Path-based value extraction with transformation
   export type PathConfig = {
     path: string;                                     // Data path for extraction
     transform?: ValueTransformConfig;                 // Optional value transformation
     fallback?: unknown;                               // Fallback value if path not found
   };

   // Comprehensive value transformation system with 25+ transform types
   export interface ValueTransformConfig {
     type: 'parseInt' | 'parseFloat' | 'toLowerCase' | 'toUpperCase' | 'trim' | 
           'replace' | 'concat' | 'regex' | 'date' | 'default' | 'conditional' | 
           'substring' | 'split' | 'join' | 'abs' | 'round' | 'floor' | 'ceil' | 
           'template' | 'sum' | 'average' | 'count' | 'min' | 'max' | 'multiply' | 
           'divide' | 'percentage' | 'add' | 'subtract' | 'currentYear' | 
           'yearDifference' | 'handlebars' | 'custom';
     
     // Common transformation parameters
     fallback?: unknown;                               // Default value for failed transforms
     prefix?: string;                                  // String prefix for concat operations
     suffix?: string;                                  // String suffix for concat operations
     pattern?: string;                                 // Regex pattern for replace/match operations
     replacement?: string;                             // Replacement string for regex operations
     template?: string;                                // Template string for template transforms
     value?: unknown;                                  // Static value for default transforms
     
     // Mathematical operation parameters
     precision?: number;                               // Decimal precision for rounding operations
     divisor?: number;                                 // Divisor for division/percentage operations
     multiplier?: number;                              // Multiplier for multiplication operations
     addend?: number;                                  // Value to add for addition operations
     subtrahend?: number;                              // Value to subtract for subtraction operations
     
     // Array and aggregation parameters
     field?: string;                                   // Field name for array aggregations
     delimiter?: string;                               // Delimiter for join/split operations
     index?: number;                                   // Array index for element selection
     
     // Conditional and date parameters
     condition?: ConditionConfig;                      // Condition for conditional transforms
     fromYear?: number;                                // Start year for year difference calculations
     dataPath?: string;                                // Path for accessing context data
   }

   // Flexible condition evaluation system
   export interface ConditionConfig {
     field: string;                               // Field path for evaluation
     operator: 'equals' | 'eq' | 'notEquals' | 'ne' | 'contains' | 'exists' | 
               'notExists' | 'greaterThan' | 'gt' | 'lessThan' | 'lt' | 
               'greaterThanOrEqual' | 'gte' | 'lessThanOrEqual' | 'lte' | 
               'startsWith' | 'endsWith' | 'matches' | 'in' | 'hasLength' | 
               'isArray' | 'isObject' | 'isString' | 'isNumber';
     value?: unknown;                             // Comparison value for operators
   }
   ```
   
   **Response Mapping Technical Features:**
   - **Type Safety**: Full TypeScript interface definitions with validation
   - **Declarative Configuration**: No code injection - pure JSON configuration
   - **Comprehensive Transforms**: 25+ built-in transformation types
   - **Mathematical Operations**: Arithmetic, statistical, and precision control
   - **Date Processing**: Dynamic date calculations and formatting
   - **Template System**: Handlebars-style variable substitution with array iteration
   - **Conditional Logic**: Complex branching and filtering capabilities
   - **Array Processing**: Filtering, sorting, pagination, and aggregation
   - **Path Resolution**: JSONPath support with fallback handling
   - **Security**: Complete input sanitization and validation

## Features & Capabilities

### Flow Execution Modes
- ✅ **Linear Flow Execution** - Sequential step processing
- ✅ **Sub-Flow Calls** - Nested workflow execution
- ✅ **Flow Interruption** - Suspend current flow for new task
- ✅ **Flow Resumption** - Return to previously suspended flows
- ✅ **Flow Replacement** - Replace current flow with new flow
- ✅ **Flow Reboot** - Nuclear option: clear all flows and restart (SAY output accumulated before the reboot is kept and delivered with the new flow's output; `discardSay: true` drops it)

### Step Types
- ✅ **SAY** - Non-blocking output messages (accumulated)
- ✅ **SAY-GET** - Blocking output with user input request
- ✅ **SET** - Variable assignment with interpolation support
- ✅ **CALL-TOOL** - External tool execution with error handling
- ✅ **FLOW** - Sub-flow execution with multiple call types
- ✅ **CASE** - Conditional branching with expressions
- ✅ **SWITCH** - Conditional branching based on single value matching
- ✅ **RETURN** - Terminate all flows and return evaluated expression value (an EXIT/abort); `keepSay: true` delivers the SAY text queued before it ahead of the value
- ✅ **END** - Return from the current flow only, resuming the parent flow — a functional `return` (no value; variables are shared with the parent)
- ✅ **DISPATCH** - End all flows and route this turn's input through intent detection, as if no flow had been active (a matched flow starts in the same turn; no match → no response, the host answers). Resets everything; the ended flows' declared outcome is kept only when no flow matches. Takes `id`, `outcome`, `reason`. See the user guide's *DISPATCH Steps*

A terminal step (`RETURN`, `END`, `DISPATCH` or a final `SAY`) may also carry `outcome` / `reason`; see
[Declared flow outcomes](#declared-flow-outcomes-lastflowoutcome).

### Step Reliability: Validation & Retry
Any step can also declare:
- **`inputValidation`** — regex `patterns` (`field` / `pattern` / `message`) and/or a named `customValidator` from `APPROVED_FUNCTIONS`, applied to collected input before the step runs.
- **Automatic retry** — `maxRetries` (default 2), `retryDelay`, `retryStrategy` (`immediate` / `exponential` / `linear` / `manual`), and `retryOnConditions` (`errorPattern` → `action`), falling through to `onFail` once exhausted.

### Default Timeouts
- AI callback (`aiTimeOut`): **2000 ms**
- HTTP / REST tool calls: **20000 ms**
- Local function tool calls: **5000 ms**

Each tool may override its own `timeout`.

### Expression System

The JSFE engine features a **powerful, unified JavaScript expression evaluator** that provides consistent
syntax and behavior across all contexts. This system supports the full range of JavaScript expressions
while maintaining security through a trusted developer model and limiting user input usage as values.

#### Core Features
- ✅ **Full JavaScript Expression Support** - All standard operators, functions, and syntax
- ✅ **Type Preservation** - Numeric, boolean, and object types maintained correctly
- ✅ **Template Interpolation** - `{{expression}}` syntax for string templates
- ✅ **Direct Expression Evaluation** - Pure expressions return their native JavaScript types
- ✅ **Unified Security Model** - Same safety framework across all evaluation contexts

#### Expression Types

**Single Expressions** (preserve JavaScript types):
```javascript
// SET steps with pure expressions return native types
{ type: "SET", variable: "count", value: "{{attempt_count + 1}}" }        // Returns number
{ type: "SET", variable: "isValid", value: "{{age >= 18 && verified}}" }  // Returns boolean
{ type: "SET", variable: "user", value: "{{api_response.user}}" }         // Returns object
```

**Template Strings** (convert to strings for interpolation):
```javascript
// String templates with embedded expressions
{ type: "SAY", value: "Attempt {{attempt_count + 1}}/{{max_attempts}}" }  // String result
{ type: "SAY", value: "Welcome {{user.name}} ({{user.verified ? 'Verified' : 'Unverified'}})" }
```

#### Supported JavaScript Features
All JavaScript features are supported within a sandboxed environment with access limited to 
explicitly exported variables and functions.

#### Security Model
- **Developer Trust**: Workflows authored by developers, not end users
- **Parameter Filtering**: Only valid JavaScript identifiers allowed as parameters
- **No eval()**: Direct Function constructor with parameter validation
- **User Input Safety**: User inputs are treated as values, not code

### Internationalization and Localization

The JSFE engine supports multiple languages through localized properties in flow definitions
and steps. This allows you to create workflows that automatically display messages in the user's
preferred language.

#### Flow-Level Localization

Add localized prompts to your flow definitions using the `prompt_xx` pattern:

```javascript
const flowsMenu = [
  {
    id: "payment-flow",
    name: "ProcessPayment",
    prompt: "Process a payment",           // Default (English)
    prompt_es: "Procesar un pago",        // Spanish
    prompt_fr: "Traiter un paiement",     // French
    prompt_de: "Eine Zahlung bearbeiten", // German
    description: "Handle payment processing",
    steps: [ /* ... */ ]
  }
];
```

#### Step-Level Localization

Add localized messages to individual steps using the `value_xx` pattern:

```javascript
{
  type: "SAY-GET",
  variable: "amount",
  value: "Please enter the payment amount:",           // Default (English)
  value_es: "Por favor ingrese el monto del pago:",    // Spanish
  value_fr: "Veuillez saisir le montant du paiement:", // French
  value_de: "Bitte geben Sie den Zahlungsbetrag ein:"  // German
}
```

#### Language Selection

Set the user's preferred language when initializing the engine:

```javascript
const engine = new WorkflowEngine(
  logger,
  aiCallback,
  flowsMenu,
  toolsRegistry,
  APPROVED_FUNCTIONS,
  globalVariables,
  true,           // validateOnInit
  'es',           // language - Spanish
  1000,           // aiTimeOut - 1 second for faster service
  messageRegistry,
  guidanceConfig
);
```

#### Supported Language Codes

- `en` - English (default)
- `es` - Spanish (Español)
- `pt` - Portuguese (Português)
- `fr` - French (Français)
- `de` - German (Deutsch)
- Any ISO 639-1 language code

#### Language Fallback

If a localized property is not found for the specified language, the engine automatically falls
back to the default `prompt` or `value` property. This ensures your flows always work even if
some translations are missing.

```javascript
// Example: User has language='es' but only English is available
{
  type: "SAY",
  value: "Welcome to our service!"  // Will be used as fallback
  // value_es is missing, so English value is used
}
```

### Expression Template System

The engine supports safe JavaScript expressions within `{{}}` templates:

```javascript
// Simple variables
{{userName}}, {{account.balance}}

// Arithmetic
{{amount + fee}}, {{price * quantity}}

// Comparisons
{{age >= 18}}, {{status === 'active'}}

// Logical operations
{{isAdmin && hasAccess}}, {{retryCount < maxRetries}}

// Complex expressions
{{user.permissions.includes('admin') && creditScore > 700}}
```

### JavaScript Method and Function Calls

**String Methods:**
```javascript
// Case conversion
{{userInput.toLowerCase().trim()}}

// Email validation
{{email.includes('@') && email.length > 5}}

// Text processing
{{text.substring(0, 10).padEnd(15, '...')}}
```

**Array Methods:**
```javascript
// Length and content checks
{{items.length > 0 && items.includes('premium')}}

// Array manipulation
{{categories.slice(0, 3).join(', ')}}
```

**Math Methods:**
```javascript
// Tax calculation
{{Math.round(price * 1.08)}}

// Ensure non-negative
{{Math.max(balance, 0)}}
```

**Functions:**
```javascript
// Type conversion and validation
{{Number(input) > 0 && !isNaN(Number(input))}}
{{Boolean(user.isActive && user.hasAccess)}}

// URI encoding for API calls
{{encodeURIComponent(searchTerm)}}

// User-defined approved functions
{{currentTime()}} // If registered as approved function
{{extractCryptoFromInput(userMessage)}} // Custom business logic
```

## Demo/Test Mode: Flow Matching Without AI

For demos, tests, or developer convenience, you can set `aiCallback` to `null` when constructing
the engine. In this mode, intent detection will:

1. **Match by Flow Name or ID (case-insensitive):**
   - If the user input exactly matches a flow's `name` or `id`, that flow is activated.
2. **Partial Match Fallback:**
   - If no exact match is found, the engine will look for a flow whose `name` or `id` contains the input (case-insensitive).
3. **No Match:**
   - If no match is found, no flow is activated.

This makes it easy to run demos and tests without requiring a real AI intent detection function.
In production, always provide a real `aiCallback` for robust intent detection.

### Test Suite Patterns

When creating test suites, follow these patterns to ensure session isolation and prevent state contamination:

#### Proper Test Isolation Pattern

```javascript
// ❌ Dangerous - Shared session may cause contamination
const globalSession = engine.initSession('test-user', 'test-session');

for (const testCase of testCases) {
  // This will cause session corruption!
  await runTest(testCase, globalSession);
}

// ✅ Correct - Fresh session per test
for (let i = 0; i < testCases.length; i++) {
  const testCase = testCases[i];
  
  // Create fresh session for each test
  let sessionContext = engine.initSession('test-user', `test-session-${i+1}`);
  
  await runTest(testCase, sessionContext);
}
```

#### Test Function Pattern

```javascript
async function runTest(inputs, sessionContext) {
  for (const input of inputs) {
    // Always update session context
    sessionContext = await engine.updateActivity({
      role: 'user',
      content: input,
      timestamp: Date.now()
    }, sessionContext);
    
    // Check for workflow response
    if (sessionContext.response) {
      console.log('Workflow Response:', sessionContext.response);
    }
  }
}
```

This pattern ensures that each test runs in complete isolation, preventing the session corruption
that can occur when tests share state.

## Security & Compliance

### Expression Security
- ✅ **Direct JavaScript Evaluation** - Native JavaScript execution with security framework
- ✅ **Controlled Access** - Only exported variables and functions are accessable JavaScript identifiers allowed as parameters scope
- ✅ **User Input Safety** - User inputs treated as values, with protection against code injection

### Transaction Management
- ✅ **Comprehensive Audit Trail** - Every action logged
- ✅ **Transaction State Tracking** - Success/failure/pending states
- ✅ **Error Recovery Logging** - Detailed failure analysis
- ✅ **User Context Isolation** - Prevent cross-user data leaks

## Integration Capabilities

### REST API Support
- ✅ **HTTP Methods:** GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS
- ✅ **Content Types:** JSON, Form-data, URL-encoded, XML/SOAP, Plain text, Multipart
- ✅ **Authentication:** Bearer tokens, Basic auth, API keys, HMAC signatures
- ✅ **Parameter Handling:** Path params, Query params, Request body, Headers
- ✅ **Advanced Features:** Retries with exponential backoff, Timeouts, Rate limiting

### Declarative Response Mapping

The engine supports completely generic response transformation through declarative JSON configuration:

#### JSONPath Mapping
```javascript
{
  responseMapping: {
    type: "jsonPath",
    mappings: {
      "output_field": {
        path: "api.response.field[0].value",
        transform: { type: "parseInt", fallback: 0 },
        fallback: "$args.inputField"
      }
    }
  }
}
```

#### Object Mapping
```javascript
{
  responseMapping: {
    type: "object", 
    mappings: {
      "user_name": "name",
      "contact": {
        "email": "email",
        "phone": "phone"
      },
      "metadata": {
        "processed": true,
        "timestamp": "{{new Date().toISOString()}}"
      }
    }
  }
}
```

#### Template Mapping
```javascript
{
  responseMapping: {
    type: "template",
    template: "User {{name}} ({{email}}) from {{$args.source}}"
  }
}
```

## Core Features

- **OpenAI Function Calling Standard** schemas
- **JSON Schema validation** with ajv
- **Secure function registry**
- **Comprehensive error handling** & transaction management
- **Audit logging** for compliance
- **Rate limiting** and input validation

## Value Transformations

### Supported Transforms
- **Number conversion:** parseInt, parseFloat
- **String manipulation:** toLowerCase, toUpperCase, trim
- **Text processing:** replace (regex), concat (prefix/suffix)
- **Date handling:** Convert to ISO date string
- **Fallback values:** Default handling for missing data

### Security & Best Practices
- ✅ No code injection possible - all transformations are declarative
- ✅ Secure path traversal with validation
- ✅ Fallback handling for missing data
- ✅ Type coercion with validation
- ✅ Error handling with graceful degradation

## Example Tool Configurations

### Simple GET Request
```javascript
{
  implementation: {
    type: "http",
    url: "https://api.example.com/users",
    method: "GET"
  }
}
```

### Authenticated POST with JSON
```javascript
{
  implementation: {
    type: "http", 
    url: "https://api.example.com/users",
    method: "POST",
    contentType: "application/json"
  },
  apiKey: "your-bearer-token"
}
```

### Path Parameters with Response Mapping
```javascript
{
  implementation: {
    type: "http",
    url: "https://api.example.com/users/{userId}",
    method: "GET",
    pathParams: ["userId"],
    responseMapping: {
      type: "object",
      mappings: {
        "user_id": "id",
        "full_name": "name",
        "contact_email": "email"
      }
    }
  }
}
```

## Troubleshooting

### Common Session Management Issues

#### Problem: "engineSessionContext.flowStacks was invalid, initialized fresh flowStacks"

**Cause**: Session context corruption due to improper session management.

**Solutions**:
1. **Always update session reference**: 
   ```javascript
   // ❌ Wrong
   await engine.updateActivity(entry, sessionContext);
   
   // ✅ Correct  
   sessionContext = await engine.updateActivity(entry, sessionContext);
   ```

2. **Use unique sessions per user/conversation**:
   ```javascript
   // ❌ Wrong - sharing sessions
   const sharedSession = engine.initSession('shared', 'shared');
   
   // ✅ Correct - isolated sessions
   const userSession = engine.initSession(`user-${userId}`, `session-${sessionId}`);
   ```

3. **Initialize fresh sessions for testing**:
   ```javascript
   // For each test case
   let sessionContext = engine.initSession('test-user', `test-session-${testIndex}`);
   ```

#### Problem: Workflows not triggering or mock responses

**Cause**: Corrupted session state or improper context handling.

**Solution**: Ensure proper session lifecycle:
```javascript
// Initialize once per user/conversation
let sessionContext = engine.initSession(userId, sessionId);

// Update for every interaction
sessionContext = await engine.updateActivity(userInput, sessionContext);

// Check workflow response
if (sessionContext.response) {
  return sessionContext.response;
}

// Continue with regular conversation
sessionContext = await engine.updateActivity(assistantResponse, sessionContext);
```

#### Problem: State bleeding between users/sessions

**Cause**: Shared session contexts between different users or conversations.

**Solution**: Maintain strict session isolation:
```javascript
// Create session per user
const userSessions = new Map();

function getOrCreateSession(userId, sessionId) {
  const key = `${userId}-${sessionId}`;
  if (!userSessions.has(key)) {
    userSessions.set(key, engine.initSession(userId, sessionId));
  }
  return userSessions.get(key);
}

// Update and store session
let sessionContext = getOrCreateSession(userId, sessionId);
sessionContext = await engine.updateActivity(entry, sessionContext);
userSessions.set(`${userId}-${sessionId}`, sessionContext);
```