import { TokenKind } from './types'; import type { Token } from './types'; // ─── Character helpers ──────────────────────────────────────────── function isAlpha(ch: string): boolean { return (ch >= 'a' && ch <= 'z') || (ch >= 'A' && ch <= 'Z') || ch === '_'; } function isDigit(ch: string): boolean { return ch >= '0' && ch <= '9'; } function isAlphaNum(ch: string): boolean { return isAlpha(ch) || isDigit(ch); } function isWhitespace(ch: string): boolean { return ch === ' ' || ch === '\t' || ch === '\n' || ch === '\r'; } // ─── Duration & Bytes regex helpers ─────────────────────────────── const DURATION_UNITS = ['ns', 'us', 'µs', 'ms', 's', 'm', 'h']; const BYTES_UNITS = ['B', 'kB', 'MB', 'GB', 'TB', 'KB', 'KiB', 'MiB', 'GiB', 'TiB']; // ─── Tokenizer ──────────────────────────────────────────────────── /** * Convert an offset (0-based) to Monaco line/column. * Monaco line/column are 1-based. */ export function offsetToPosition(input: string, offset: number): { lineNumber: number; column: number } { let line = 1; let col = 1; for (let i = 0; i < offset && i < input.length; i++) { if (input[i] === '\n') { line++; col = 1; } else { col++; } } return { lineNumber: line, column: col }; } export function tokenize(input: string): Token[] { const tokens: Token[] = []; let pos = 0; const len = input.length; function peek(offset = 0): string { const idx = pos + offset; return idx < len ? input[idx] : '\0'; } function advance(): string { return pos < len ? input[pos++] : '\0'; } function emit(kind: TokenKind, value: string, start?: number): void { const s = start ?? pos - value.length; tokens.push({ kind, value, start: s, end: s + value.length }); } function emitError(message: string, start: number, end: number): void { tokens.push({ kind: TokenKind.Error, value: message, start, end }); } while (pos < len) { const start = pos; const ch = advance(); // ── Whitespace ── if (isWhitespace(ch)) { while (pos < len && isWhitespace(peek())) advance(); emit(TokenKind.Whitespace, input.slice(start, pos)); continue; } // ── Comments ── if (ch === '#') { while (pos < len && peek() !== '\n') advance(); emit(TokenKind.Comment, input.slice(start, pos)); continue; } // ── Strings ── if (ch === '"' || ch === "'" || ch === '`') { const quote = ch; let escaped = false; while (pos < len) { const c = advance(); if (escaped) { escaped = false; continue; } if (c === '\\') { escaped = true; continue; } if (c === quote) { break; } } emit(TokenKind.String, input.slice(start, pos)); continue; } // ── Pipe operators ── if (ch === '|') { if (peek() === '=') { advance(); emit(TokenKind.PipeEquals, '|='); } else if (peek() === '~') { advance(); emit(TokenKind.PipeRegex, '|~'); } else { emit(TokenKind.Pipe, '|'); } continue; } // ── Operators starting with ! ── if (ch === '!') { if (peek() === '=') { advance(); emit(TokenKind.NotEquals, '!='); } else if (peek() === '~') { advance(); emit(TokenKind.RegexNotEquals, '!~'); } else { emit(TokenKind.Error, '!', start); } continue; } // ── Operators starting with = ── if (ch === '=') { if (peek() === '~') { advance(); emit(TokenKind.RegexEquals, '=~'); } else if (peek() === '=') { // == (label filter equality) advance(); emit(TokenKind.Equals, '=='); } else { emit(TokenKind.Equals, '='); } continue; } // ── Comparison operators ── if (ch === '>') { if (peek() === '=') { advance(); emit(TokenKind.GreaterEquals, '>='); } else { emit(TokenKind.GreaterThan, '>'); } continue; } if (ch === '<') { if (peek() === '=') { advance(); emit(TokenKind.LessEquals, '<='); } else { emit(TokenKind.LessThan, '<'); } continue; } // ── Brackets & punctuation ── if (ch === '{') { emit(TokenKind.LCurly, '{'); continue; } if (ch === '}') { emit(TokenKind.RCurly, '}'); continue; } if (ch === '[') { emit(TokenKind.LBracket, '['); continue; } if (ch === ']') { emit(TokenKind.RBracket, ']'); continue; } if (ch === '(') { emit(TokenKind.LParen, '('); continue; } if (ch === ')') { emit(TokenKind.RParen, ')'); continue; } if (ch === ',') { emit(TokenKind.Comma, ','); continue; } // ── Numbers, Duration, Bytes ── if (isDigit(ch)) { // Consume digits and optional decimal while (pos < len && (isDigit(peek()) || peek() === '.')) advance(); const numStr = input.slice(start, pos); // Check for Duration or Bytes suffix let suffix = ''; while (pos < len && isAlpha(peek())) { suffix += advance(); } // Try to classify the full token const fullStr = numStr + suffix; if (suffix && DURATION_UNITS.includes(suffix)) { emit(TokenKind.Duration, fullStr); } else if (suffix && BYTES_UNITS.includes(suffix)) { emit(TokenKind.Bytes, fullStr); } else { // It's just a number (suffix might be part of an adjacent identifier) // Put back any consumed alpha chars that don't form a unit // Actually we consumed them, better approach: check first // For simplicity, emit as number. If suffix exists but isn't a unit, // it'll be caught by the parser. emit(TokenKind.Number, numStr); // Re-emit the suffix as identifier if it's not empty and not a unit if (suffix) { pos = start + numStr.length; // backtrack to after the number } } continue; } // ── Identifiers / Keywords ── if (isAlpha(ch)) { while (pos < len && (isAlphaNum(peek()) || peek() === '_')) advance(); emit(TokenKind.Ident, input.slice(start, pos)); continue; } // ── Anything else → error ── emitError(`Unexpected character '${ch}'`, start, pos); } // Mark EOF tokens.push({ kind: TokenKind.EOF, value: '', start: pos, end: pos }); return tokens; } /** Return tokens with whitespace and comments filtered out. */ export function nonWhitespace(tokens: Token[]): Token[] { return tokens.filter((t) => t.kind !== TokenKind.Whitespace && t.kind !== TokenKind.Comment); }