import { readFileSync } from "node:fs"; import { FewShotPromptTemplate, PromptTemplate } from "@langchain/core/prompts"; import { type Example } from "@langchain/core/dist/prompts/base"; import { type OpenAI } from "openai"; import { MultiAttemptTemplate } from "../../prompt-templates/multi-attempt"; import { type Message, type SupportOfferResult, type SupportTicketContext, type MultiAttemptExample, } from "../../schemas"; import { type OfferSupportStrategy } from "./offer-detection-strategy"; /** * The MultiAttemptStrategy class is responsible for determining whether a support ticket should be offered based on the user's chat history. * It analyzes the chat interactions to identify repeated user questions and decides if the threshold for generating a support ticket is met. * * Usage: * - Initialize with a SupportTicketContext and an OpenAI instance. * - Use `shouldOfferSupport` to check if a support ticket should be generated based on the user's chat history. * - Uses few-shot and chain of throught prompting from an example data to help the LLM generate if support ticket should be offered. * - It utilizes OpenAI's language model to analyze chat history and make decisions. * * Methods: * - `getTemp`, `getModel`, `getTokenLimit`, `getStrategyName`, `getExamplePath`: Getters for class properties. * - `setContext`: Updates the chat history and environment context. * - `shouldOfferSupport`: Main method to determine if a support ticket should be offered. * * @class MultiAttemptStrategy */ export class MultiAttemptStrategy implements OfferSupportStrategy { chatHistory: Message[]; environmentContext: string; systemInstruction: PromptTemplate; temp = 0; model = "gpt-3.5-turbo"; tokenLimit = 1000; strategyName = "Multi-Attempt Detection"; examplePath = "./data/prompt/multi-attempt/examples.json"; constructor( supportTicketContext: SupportTicketContext, private readonly openAi: OpenAI ) { this.chatHistory = supportTicketContext.chatHistory; this.environmentContext = supportTicketContext.environmentContext; this.systemInstruction = new MultiAttemptTemplate().getInstruction; } public get getTemp(): number { return this.temp; } public get getModel(): string { return this.model; } public get getTokenLimit(): number { return this.tokenLimit; } public get getStrategyName(): string { return this.strategyName; } public get getExamplePath(): string { return this.examplePath; } public setContext(supportTicketContext: SupportTicketContext): boolean { this.chatHistory = supportTicketContext.chatHistory; this.environmentContext = supportTicketContext.environmentContext; return true; } public async shouldOfferSupport(): Promise { const completion = await this.getCompletion(); if (!completion) { throw new Error( "Failed to get a response from OpenAI, consider retrying." ); } const parsedResponse: Record = JSON.parse( completion ) as Record; return { name: this.getStrategyName, reoccurringIssue: parsedResponse.reoccurringIssue ?? null, numberTimesMentioned: parsedResponse.numberTimesMentioned ?? null, rationale: parsedResponse.rationale ?? null, offerSupport: parsedResponse.offerSupport ?? null, }; } readExamplesFromFile(): MultiAttemptExample[] { try { const fileContent = readFileSync(this.getExamplePath, "utf8"); const jsonData: MultiAttemptExample[] = JSON.parse( fileContent ) as MultiAttemptExample[]; return jsonData; } catch (error) { if (error instanceof Error) { throw new TypeError( `Unable to read file: ", ${this.getExamplePath}\nError Message: ${error.message}` ); } else { throw new TypeError(`Unable to read file: ", ${this.getExamplePath}`); } } } async getCompletion() { const examples: string = await this.getExamples(); const lastTenMessages = this.chatHistory.slice(-10); // We only care about the last 10 messages sent between the user and the assistant to detemine if support ticket should be offered. Increaseing may cause too much context and increase false positive const chatHistoryString = this.convertMessagesToString(lastTenMessages); const formattedSystemInstruction = await this.systemInstruction.format({ chatHistory: chatHistoryString, examples, }); try { const completion = await this.openAi.chat.completions.create({ messages: [{ role: "system", content: formattedSystemInstruction }], temperature: this.getTemp, model: this.getModel, /* eslint-disable @typescript-eslint/naming-convention */ response_format: { type: "json_object" }, max_tokens: this.getTokenLimit, /* eslint-enable @typescript-eslint/naming-convention */ }); return completion?.choices[0]?.message?.content ?? null; } catch (error_) { const error = error_ instanceof Error ? new Error(`Failed to call OpenAI Client: ${error_.message}`) : new Error("Failed to call OpenAI Client with an unknown error"); throw error; } } async getExamples(): Promise { const examples: Example[] = this.convertToRecordArray( this.readExamplesFromFile() ); const examplePrompt = PromptTemplate.fromTemplate(` Example No: {id}\n\n* Chat History: {chatHistory}\n\n* Reoccurring Issue: {reoccurringIssue}\n\n* Number of Times Mentioned: {numberTimesMentioned}\n\n* Rationale: {rationale}\n\n* Offer Support: {offerSupport}`); const multiAttemptExamples = new FewShotPromptTemplate({ examples, examplePrompt, prefix: "## {name}:\n\n", inputVariables: ["name"], }); return multiAttemptExamples.format({ name: "Examples" }); } convertToRecordArray(examples: MultiAttemptExample[]): Example[] { return examples.map((example) => { const result: Record = {}; for (const [key, value] of Object.entries(example)) { if (key === "chatHistory" && Array.isArray(value)) { result[key] = value .map((entry) => `${entry.role}: ${entry.content}`) .join("\n"); } else if (typeof value === "object" && value !== null) { result[key] = JSON.stringify(value); } else { result[key] = String(value); } } return result as Example; }); } convertMessagesToString(messages: Message[]): string { return messages .map((entry) => `${entry.role}: ${entry.content}`) .join("\n"); } }