# Sesi Runtime Architecture

## Overview

**Sesi** is a concise and highly legible programming language. It uses a high-performance bytecode compilation model built in TypeScript. The architecture features a compiler that lowers the AST to a flat bytecode chunk, executed by a register-based virtual machine, with the original tree-walking interpreter retained as a fallback path. The architecture is optimized for simplicity and eliminating the need for `async/await` syntax or heavy module imports. By acting as a minimal orchestration layer, Sesi allows developers to write clean, straightforward logic that natively handles background processes and API or Reasoning calls without SDK overhead.

## Component Stack

```
┌─────────────────────────────────────────────┐
│         Sesi Program (.sesi file)           │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│      Lexer (src/lexer.ts)                   │
│  Converts source text → Tokens              │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│      Parser (src/parser.ts)                 │
│  Converts Tokens → Abstract Syntax Tree     │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│      Compiler (src/compiler.ts)             │
│  Lowers AST → Bytecode Chunk                │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│      VM (src/vm.ts)                         │
│  Bytecode virtual machine                   │
│  - Executes OpCode instructions             │
│  - Manages call frames & value stack        │
│  - Supports closures, try/catch, imports    │
└──────┬───────────────────────┬──────────────┘
       │                       │
       │                       │
┌──────▼──────────────┐  ┌────▼────────────────┐
│ Builtins            │  │ AI Runtime          │
│ (src/builtins.ts)   │  │ (src/ai-runtime.ts) │
│                     │  │                     │
│ - show()           │  │ - Gemini API calls  │
│ - len()             │  │ - Memory mgmt       │
│ - read_file()       │  │ - Structured output │
│ - write_file()      │  │ - Tool calling      │
│ - spawn()           │  │                     │
│ - exec()            │  │                     │
│ - sesi()            │  │                     │
│ - time()            │  │                     │
│ - random()          │  │                     │
│ - etc.              │  │                     │
└─────────────────────┘  └────┬────────────────┘
                              │
                    ┌─────────▼──────────┐
                    │  Gemini API        │
                    │  (@google/genai)   │
                    └────────────────────┘
```

## Execution Flow

### 1. Lexical Analysis (Lexer)

- **Input**: Source code string
- **Process**: Character-by-character tokenization
  - Identifier and keyword recognition
  - Number and string literal parsing
  - Operator and delimiter recognition
  - Comment stripping
- **Output**: Token stream

**Example**:

```
Source: let x = 10
Tokens: [LET, IDENTIFIER("x"), EQUAL, NUMBER(10), EOF]
```

### 2. Parsing (Parser)

- **Input**: Token stream
- **Process**: Recursive descent parsing
  - Statement parsing (declarations, control flow)
  - Expression parsing (operators, function calls, Reasoning constructs)
  - AST construction
  - Error recovery
- **Output**: Abstract Syntax Tree (AST)

**Example AST Node**:

```javascript
{
  type: 'LetStatement',
  name: 'x',
  value: {
    type: 'Literal',
    value: 10,
    rawType: 'number'
  }
}
```

### 3. Compilation & Execution (Compiler → VM)

- **Input**: AST
- **Compiler process**: Single-pass AST lowering
  - Emits flat OpCode instructions into a `Chunk`
  - Resolves local variable slots at compile time
  - Compiles closures, loops, conditionals, and try/catch
- **VM process**: Stack-based execution
  - Dispatches OpCode instructions in a tight loop
  - Maintains a value stack and call frame stack
  - Calls into builtins and AI runtime as needed
- **Output**: Program side effects (show, Reasoning calls, etc.)

> **Note**: The original tree-walking interpreter (`src/interpreter.ts`) is retained as a fallback execution path for any constructs not yet supported by the compiler.

## Scope and Environment Management

Sesi uses **lexical scoping** with an environment chain:

```
┌─────────────────────────┐
│  Global Environment     │
│  - Built-in functions   │
│  - Global variables     │
└────────┬────────────────┘
         │
    ┌────▼────────────────┐
    │ Function Environment│
    │ - Parameters        │
    │ - Local variables   │
    └────┬────────────────┘
         │
    ┌────▼────────────────┐
    │  Block Environment  │
    │  - Block variables  │
    └─────────────────────┘
```

### Environment Lookup

1. Check current environment
2. Walk up parent chain until found
3. If not found anywhere, error

### Variable Assignment

1. Try to update in current scope
2. If not in current scope, try parent scopes
3. Update in the scope where variable was found

## Type System

Sesi has a **dynamic type system** with runtime type checking:

```
RuntimeValue =
  | number
  | string
  | boolean
  | null
  | Array<RuntimeValue>
  | Object<string, RuntimeValue>
  | RuntimeFunction
  | RuntimeModule
```

### Type Coercion Rules

**String Concatenation** (operator `+`):

```
"Hello" + 5        → "Hello5"
"Age:" 30          → "Age: 30"
any + string       → str(any) + string
string + any       → string + str(any)
```

**Numeric Operations**:

```
10 + 20            → 30
"10" + 20          → "1020" (not numeric!)
10 20              → "10 20" (not numeric!)
```

**Truthiness**:

```
null   → false
false  → false
0      → false
""     → false
[]     → true
{}     → true
```

## Reasoning Integration

### Reasoning Runtime Lifecycle

1. **Initialization**:
   - Load @google/genai SDK
   - Validate GEMINI_API_KEY environment variable
2. **During Execution**:
   - Parse model calls from AST
   - Construct prompt from string concatenation
   - Make async API call to Gemini

- Wait for response (blocking in v1.x)
- Validate finish reason and non-empty text
- Return text response to program or throw

3. **Memory Management**:
   - Simple string buffers per memory ID
   - Append/update operations
   - Passed to next model call

### Model Call Flow

```
ModelCallExpression (AST)
    │
    ├─ Evaluate prompt expression
    ├─ Extract configuration (`thinkingLevel`, `max_tokens`, `cache`, etc.)
    ├─ Call AIRuntime.callModel()
    │   │
    │   ├─ Create Gemini interaction request
    │   ├─ Send to Gemini API
    │   ├─ Wait for response
    │   ├─ Validate finish reason == STOP
    │   └─ Extract text from response
    │
    └─ Return text to program or throw
```

## Error Handling (V1)

Sesi now has basic exception-style error handling in v1:

1. **Parse Errors**: Parser logs the error and synchronizes to continue parsing later statements.
2. **Runtime Errors**: Interpreter errors throw and can be caught with `try/catch`.
3. **Built-in I/O Errors**: `read_file()`, `write_file()`, and `list_dir()` throw on filesystem failure.
4. **Reasoning Errors**: `model()` throws when the SDK fails, when no text is returned. (Note: `MAX_TOKENS` finish reasons are now handled natively via an automatic async polling loop that prompts the model to continue where it left off, and therefore does not throw errors).
5. **Structured Output**: `structured_output()` attempts recovery, but currently logs parsing failures and returns `{}` if coercion still fails.

## Memory Model

### Stack

- Local variables
- Function parameters
- Scope frames

### Heap

- Arrays (dynamic)
- Objects (key-value maps)
- Strings (immutable)

### Reasoning Context

- Conversation memory per memory ID
- String buffers, not structured storage

## Performance Characteristics

| Operation       | Time  | Notes                     |
| --------------- | ----- | ------------------------- |
| Variable lookup | O(n)  | n = scope depth           |
| Array access    | O(1)  | Direct indexing           |
| Object access   | O(1)  | Map lookup                |
| Function call   | O(1)  | + body execution          |
| Model call      | ~2-5s | Depends on Gemini latency |
| Array iteration | O(n)  | Foreach loop              |

## Limitations (V1)

- **VM Single-threaded**: Each individual Sesi process is single-threaded.
- **Process-level Concurrency**: Sesi uses a multi-process model via `spawn()` for concurrent task execution.
- **Bytecode coverage**: The compiler covers expressions, loops, functions, closures, try/catch, and imports; edge-case constructs fall back to the tree-walking interpreter.
- **No JIT**: The VM executes bytecode directly; JIT compilation is a V4+ goal.
- **Simple type system**: Runtime checking only.
- **No macro system**: No compile-time code generation
- **No introspection**: Can't inspect function bodies
- **Limited error info**: Basic error messages

## Future Architecture (V2+)

### V2 Delivered ✅

- Bytecode compiler (`src/compiler.ts`) that lowers the AST to a flat `Chunk`
- Stack-based virtual machine (`src/vm.ts`) with OpCode dispatch
- Closures, loops, try/catch, and imports handled natively by the VM
- Legacy interpreter flag (`-t`) to enable the tree-walking execution path
- Streaming response support
- Advanced error handling with stack traces

### V3+ Vision

- Agent framework with state machines
- Knowledge base integration
- Advanced memory with embedding search
- Multi-agent orchestration
- Custom type definitions
- Macro system

## Code Organization

```
SKILLS.md                 # AI-agent workspace context and repo guardrails
eslint.config.mjs         # ESLint configuration
dist/                     # Compiled TypeScript output
example.js                # Helper script to run basic examples
example-ai.js             # Helper script to run Reasoning examples
examples.sesi             # Central execution suite for examples
package.json              # Dependencies & scripts
tsconfig.json             # TypeScript configuration
QUICKSTART.md             # Quick start guide
IMPLEMENTATION_SUMMARY.md # Progress and tracking
README.md                 # Project overview

src/
├── types.ts              # Type definitions and AST
├── lexer.ts              # Tokenization
├── parser.ts             # AST generation
├── compiler.ts           # AST → Bytecode compiler
├── chunk.ts              # Bytecode Chunk & OpCode definitions
├── vm.ts                 # Bytecode virtual machine
├── interpreter.ts        # Tree-walking interpreter (fallback)
├── ai-runtime.ts         # Gemini integration
├── builtins.ts           # Built-in functions
└── index.ts              # Main entry point

bin/
└── sesi.js               # CLI executable

main/                     # Main user scripts and debugging
├── build_website.sesi    # Sesi-generated landing page builder
└── tests/                # Additional syntax validation scripts

examples/
├── main/                        # Core language & systems feature examples
│   ├── 01_hello.sesi            # Hello World
│   ├── 02_variables.sesi        # Variables & operations
│   ├── 03_functions.sesi        # Functions with parameters
│   ├── 04_conditionals.sesi     # If/else control flow
│   ├── 05_loops.sesi            # While, for, for-in loops
│   ├── 06_arrays_objects.sesi   # Collections
│   ├── 07_prompts.sesi          # Prompt blocks composition
│   ├── 09_structured_output.sesi # Type-safe reasoning responses
│   ├── 11_memory_storage.sesi   # Multi-turn stateful reasoning
│   ├── 12_classification.sesi   # Systems classification loop
│   ├── 13_data_pipeline.sesi    # Complete systems pipeline with lazy summaries
│   ├── 16_modules.sesi          # Modules & standard library imports
│   ├── 17_http_client.sesi      # Native HTTP client (web_get/web_send)
│   ├── 18_parallel_requests.sesi # Parallel request concurrency with profiling
│   ├── 19_search_web.sesi       # Built-in search_web integration
│   ├── 21_custom_tools.sesi     # Declaring and using custom tools
│   ├── 23_file_conversion.sesi  # File and format conversions
│   ├── 24_http_handler.sesi     # Native HTTP server request routing
│   ├── 24_http_server.sesi      # Non-blocking native HTTP server
│   ├── 25_webpage_server.sesi   # Native webpage hosting
│   ├── 26_database.sesi         # Embedded document database operations
│   ├── 27_robust_web_db.sesi    # Dynamic database-backed web analytics
│   ├── 29_tool_piping.sesi      # Functional piping operator (|)
│   ├── 30_error_recovery.sesi   # Error retry, backoff, and timeout handling
│   ├── 31_synthesizer.sesi      # Sound/music synthesis & SVG drawing
│   ├── 32_browser_automation.sesi # Headless browser automation with Playwright
│   ├── 33_base64.sesi           # Base64 encoding and string encryption
│   ├── 34_sesi_api.sesi         # SwaggerUI API setup
│   ├── 35_speech_language.sesi  # Speech and language helpers
│   └── 36_regex_media.sesi     # Regex and media processing
│
└── optional/                    # Optional reasoning & advanced AI examples
    ├── 08_model_call.sesi       # Basic reasoning model calls
    ├── 10_code_generation.sesi  # Code generation logic
    ├── 14_folder_explainer.sesi # Workspace directory analyzer
    ├── 15_image_generation.sesi # Image generation API
    ├── 20_model_aliases.sesi    # Custom model naming aliases
    ├── 22_reasoning_plus_custom_tools.sesi # Reasoning composed with custom tools
    ├── 28_streaming.sesi        # Real-time response streaming
    └── 37_ai_video_generation.sesi # AI video generation workflow

docs/
├── SPECIFICATION.md      # Language spec
├── ARCHITECTURE.md       # This file
├── BUILTINS.md           # Built-in reference
├── CLI.md                # Comprehensive CLI & Parametric Eval guide
├── IMAGE_GENERATION.md   # Image generation guide
├── COMPARISON.md         # Language comparison showcase
├── REASONING.md          # Reasoning and simple logic guide
└── ROADMAP.md            # Future plans

tests/
├── basic.test.ts         # Core parsing & evaluation tests
├── cache.test.ts         # Execution caching tests
├── http.test.ts          # Web request builtins testing
├── module.test.ts        # Imports & module loading tests
├── parallel.test.ts      # Concurrent execution tests
├── security.test.ts      # Sandbox & guardrail tests
├── test-gemini.ts        # Base model integration test
├── test-gemini2.ts       # Extended model integration test
└── workflow.test.ts      # Complex sequence workflows tests
```

## Workspace Context File

The root-level `SKILLS.md` file is part of the practical repo architecture. It is not consumed by the Sesi runtime, but it is intended to guide AI-assisted development in this workspace.

## Testing Strategy

**Unit Tests** (per component):

```typescript
// Lexer: tokenization correctness
// Parser: AST structure correctness
// Interpreter: evaluation correctness
// AI Runtime: Gemini integration
```

**Integration Tests**:

```typescript
// Full program execution
// Complex Reasoning workflows
// Error handling
```

**Example Programs**:

- Basic arithmetic and control flow
- Functions and scoping
- Arrays and objects
- Reasoning model calls
- Structured output parsing
- Memory-based conversations

## Debugging

### Debug Output

Enable debug statement tracing:

```bash
SESI_DEBUG=1 sesi program.sesi
```

### AST Visualization

Print the pretty-printed AST tree:

```bash
sesi --ast program.sesi
```

### Token Stream

Print the scanned token stream table:

```bash
sesi --tokens program.sesi
```

## Conclusion

Sesi's architecture has evolved from a simple tree-walking interpreter to a dual-mode engine. The bytecode compiler + VM is now the **primary execution path**, delivering faster startup and lower per-instruction overhead. The tree-walking interpreter is retained as a well-understood fallback. Both paths share the same Lexer, Parser, builtins, and AI Runtime.

- High-throughput execution via bytecode VM
- Easy extensibility via the shared AST and builtin layer
- Smooth Reasoning integration regardless of execution path

As the language matures, JIT compilation and further VM optimizations can be added without changing the Sesi API.