# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [2.3.0] - 2025-01-XX

### Added

#### Structured Data Query Support
- **NEW**: `structuredData` parameter in query options for precise, contextual responses
- **NEW**: Enhanced query processing with structured JSON data
- **NEW**: Support for intent-based queries with entity extraction
- **NEW**: Constraint-based query filtering and requirements
- **NEW**: Context-aware response formatting
- **NEW**: Built-in query enhancement for better retrieval accuracy

#### Query Enhancement Features
- **NEW**: `intent` field for categorizing query types (product_info, troubleshooting, comparison, etc.)
- **NEW**: `entities` object for named entity values and metadata
- **NEW**: `constraints` array for query requirements and restrictions
- **NEW**: `context` object for additional contextual information
- **NEW**: `responseFormat` field for desired output format specification
- **NEW**: Enhanced prompt generation with structured data integration

#### Retrieval Improvements
- **NEW**: Query enhancement with structured data for better document matching
- **NEW**: Entity-based filtering for more precise retrieval
- **NEW**: Improved context building with structured information

### Enhanced
- **IMPROVED**: Query method now accepts structured data alongside natural language
- **IMPROVED**: Better prompt engineering with structured context
- **IMPROVED**: Enhanced document retrieval with query augmentation
- **IMPROVED**: More precise responses through structured input processing

## [2.2.0] - 2025-10-20

### 🚨 Breaking Changes

#### Dynamic Provider Architecture
- **BREAKING**: Removed hardcoded OpenAI configuration
- **BREAKING**: `config.openai` is no longer used - users must now provide embedding and LLM instances directly
- **BREAKING**: Changed from `config.openai.apiKey` to `config.embeddings` and `config.llm`
- **BREAKING**: Embedding dimensions must now be specified in `config.embeddingDimensions`

### Added

#### Multi-Provider Support
- **Dynamic Embeddings**: Users can now provide any LangChain-compatible embedding provider
  - OpenAI embeddings (`@langchain/openai`)
  - HuggingFace embeddings (`@langchain/community`)
  - Azure OpenAI embeddings (`@langchain/azure-openai`)
  - Google AI embeddings (`@langchain/google-genai`)
  - Custom embedding providers
  
- **Dynamic Language Models**: Users can now provide any LangChain-compatible LLM
  - OpenAI ChatGPT (`@langchain/openai`)
  - Anthropic Claude (`@langchain/anthropic`)
  - Azure OpenAI (`@langchain/azure-openai`)
  - Google Gemini (`@langchain/google-genai`)
  - Ollama for local models (`@langchain/community`)
  - Custom LLM providers

#### Enhanced RAGSystem Methods
- **`addDocuments()`**: Now properly saves processed chunks to the database (previously only processed but didn't save)
- **`addDocumentFromBuffer()`**: New method to process and save documents from memory buffers
- **`addDocumentFromUrl()`**: New method to process and save documents from URLs
- **Better error handling**: Each document in batch processing is tracked individually
- **Progress tracking**: Detailed logging for document processing and saving

### Fixed
- **Critical Bug**: `addDocuments()` now actually saves chunks to the database (was only processing before)
- **Chunk persistence**: All processed chunks are now properly stored in the vector database
- **Metadata preservation**: Document metadata is correctly merged and preserved through the entire pipeline

### Changed
- **DocumentProcessor**: No longer creates OpenAI embeddings internally - uses provided embeddings instance
- **DocumentStoreLangChain**: No longer creates OpenAI embeddings internally - uses provided embeddings instance
- **RAGWorkflow**: No longer creates OpenAI LLM internally - uses provided LLM instance
- **Removed dependencies**: OpenAI imports removed from core components (now injected by user)

### Documentation
- **DYNAMIC-PROVIDERS-MIGRATION.md**: Complete migration guide from v2.1.x to v2.2.0
- **example-dynamic-providers.js**: Comprehensive examples of different provider combinations
- **Updated README.md**: New quick start examples showing different providers
- **Updated package.json**: Added new example scripts for different providers

### Migration Path
See `DYNAMIC-PROVIDERS-MIGRATION.md` for detailed migration instructions from v2.1.x to v2.2.0.

**Quick Migration Example:**
```javascript
// Before (v2.1.x)
const rag = new RAGSystem({
  database: { /* config */ },
  openai: {
    apiKey: 'key',
    modelName: 'gpt-4',
    embeddingModel: 'text-embedding-ada-002'
  }
});

// After (v2.2.0)
import { OpenAIEmbeddings, ChatOpenAI } from '@langchain/openai';
const embeddings = new OpenAIEmbeddings({ openAIApiKey: 'key' });
const llm = new ChatOpenAI({ openAIApiKey: 'key', modelName: 'gpt-4' });
const rag = new RAGSystem({
  database: { /* config */ },
  embeddings: embeddings,
  llm: llm,
  embeddingDimensions: 1536
});
```

## [2.1.0] - 2025-09-30

### 🚨 BREAKING CHANGES

#### Dynamic Provider System
- **Provider Flexibility**: Users can now provide their own embedding and language model instances
- **No More Hardcoded OpenAI**: OpenAI is no longer the only supported provider
- **Configuration Changes**: `config.openai` replaced with `config.embeddings` and `config.llm`

### Added

#### Multi-Provider Support
- **OpenAI**: Traditional OpenAI embeddings and language models
- **Anthropic**: Claude models for language generation
- **Azure OpenAI**: Azure-hosted OpenAI models
- **Google AI**: Google's Gemini and embedding models
- **HuggingFace**: Local and cloud HuggingFace models
- **Ollama**: Local language models via Ollama
- **Custom Providers**: Any LangChain-compatible provider

#### New Configuration
- **Dynamic Embeddings**: Pass any LangChain embedding instance via `config.embeddings`
- **Dynamic LLM**: Pass any LangChain LLM instance via `config.llm`
- **Embedding Dimensions**: Configurable via `config.embeddingDimensions`
- **Provider Mixing**: Use different providers for embeddings vs. language models

#### Examples and Documentation
- **example-dynamic-providers.js**: Comprehensive examples for all supported providers
- **DYNAMIC-PROVIDERS-MIGRATION.md**: Complete migration guide for existing users
- **Updated package.json**: New npm scripts for testing different providers

### Enhanced

#### Core System
- **Provider Agnostic**: RAGSystem no longer depends on specific provider implementations
- **Better Error Messages**: Clear validation for required embedding and LLM instances
- **Flexible Architecture**: Easy to add support for new providers

#### Document Processing
- **Provider Independence**: DocumentProcessor works with any embedding provider
- **Consistent Interface**: Same API regardless of chosen provider

#### Vector Store
- **Universal Compatibility**: Works with any LangChain embedding provider
- **Dimension Flexibility**: Supports different embedding dimensions

### Migration Required

#### From v2.1.x to v2.2.0
- Replace `config.openai` with provider instances
- Install required provider packages
- Update embedding dimensions configuration
- See DYNAMIC-PROVIDERS-MIGRATION.md for detailed steps

### Removed
- **Hardcoded OpenAI imports**: No longer automatically imported
- **Fixed provider configuration**: `config.openai` configuration object

## [2.1.0] - 2025-09-30

### Added

#### Advanced Filtering System
- **User-based filtering**: Filter documents and queries by `userId` for multi-tenant applications
- **Knowledgebot filtering**: Filter by `knowledgebotId` to isolate different bot contexts
- **Custom metadata filtering**: Filter by any custom metadata fields (department, priority, category, etc.)
- **Multiple filter support**: Combine multiple filters in a single query

#### New Search Methods
- **`searchSimilarChunksByTextWithFilter()`**: Enhanced search with comprehensive filtering
- **`searchByUserId()`**: Direct search by user ID
- **`searchByKnowledgebotId()`**: Direct search by knowledgebot ID
- **`getDocumentsByUserId()`**: Retrieve all documents for a specific user
- **`getDocumentsByKnowledgebotId()`**: Retrieve all documents for a specific knowledgebot
- **`searchWithMultipleFilters()`**: Search with complex filter combinations

#### Enhanced RAGSystem Methods
- **`query()` with filtering**: Query method now accepts `userId`, `knowledgebotId`, and custom filters
- **`searchDocumentsByUserId()`**: Convenience method for user-specific searches
- **`searchDocumentsByKnowledgebotId()`**: Convenience method for bot-specific searches
- **Enhanced workflow filtering**: RAGWorkflow now supports dynamic metadata filtering

#### Metadata Preservation
- **Complete metadata inheritance**: All document-level metadata preserved in chunks
- **Filter-friendly storage**: Metadata stored in vector store for efficient filtering
- **Hierarchical metadata**: Support for both chunk-level and document-level metadata

### Enhanced

#### Document Processing
- **Enhanced chunk metadata**: Chunks now inherit all document metadata for filtering
- **Better metadata handling**: Improved preservation of custom metadata through processing pipeline

#### RAG Workflow
- **Dynamic filtering**: Workflow now accepts and applies filters from query metadata
- **Filter logging**: Enhanced logging shows applied filters for debugging
- **Flexible retrieval**: Configurable limits, thresholds, and filters per query

### Examples and Documentation
- **example-filtering.js**: Comprehensive example showing all filtering capabilities
- **Updated README.md**: Added filtering examples and use cases
- **Enhanced API documentation**: Complete documentation of all filtering methods

### Usage Examples

#### Basic User Filtering
```javascript
// Query for specific user
const results = await rag.query('What documents do I have?', {
  userId: 'user_123'
});

// Search user documents
const userDocs = await rag.searchDocumentsByUserId('technical info', 'user_123');
```

#### Knowledgebot Filtering
```javascript
// Query for specific bot
const botResults = await rag.query('Help with support', {
  knowledgebotId: 'support_bot'
});
```

#### Multiple Filters
```javascript
// Complex filtering
const filteredResults = await rag.query('Show important items', {
  userId: 'user_123',
  filter: {
    priority: 'high',
    department: 'engineering',
    category: 'technical'
  }
});
```

#### Document Storage with Metadata
```javascript
// Add document with filtering metadata
const documentData = await processor.processDocumentFromBuffer(buffer, 'doc.pdf', 'pdf', {
  userId: 'user_123',
  knowledgebotId: 'help_bot',
  department: 'sales',
  priority: 'high'
});

await rag.documentStore.saveDocument(documentData);
```

## [2.0.0] - 2025-09-30

### 🚨 BREAKING CHANGES

#### Removed Environment Variable Dependencies
- **Removed all `process.env` dependencies**: Package now requires explicit configuration
- **Removed automatic `.env` loading**: No more dotenv dependencies
- **Constructor changes**: RAGSystem and DocumentProcessor constructors now require config objects

#### Database Configuration Changes
- **Database pool now configurable**: Pool created from config instead of environment variables
- **Required database credentials**: Must provide host, port, database, username, password in config
- **Connection factory pattern**: Database connections created from configuration

#### OpenAI Configuration Changes
- **Required API key in config**: Must provide `config.openai.apiKey`
- **No fallback to process.env**: All OpenAI settings must be explicit in config
- **Enhanced configuration options**: Added embeddingDimensions and other advanced settings

### Added

#### Enhanced Configuration System
- **Comprehensive config validation**: Clear error messages for missing required configuration
- **Flexible database pool settings**: Configurable connection pool parameters
- **Advanced processing options**: Configurable chunk size, overlap, and other processing parameters
- **Better error handling**: Detailed error messages for configuration issues

#### Migration Support
- **Migration guide**: Complete guide for upgrading from v1.x to v2.0
- **Example without env vars**: New example showing configuration-only usage
- **Backward compatibility docs**: Clear documentation of breaking changes

### Changed

#### API Changes
- **RAGSystem constructor**: Now requires database and OpenAI configuration
- **DocumentProcessor constructor**: Now requires configuration object
- **Database setup**: Now accepts configuration object instead of using environment variables
- **Error messages**: More descriptive errors for missing configuration

#### Architecture Improvements
- **Dependency injection**: Components receive configuration instead of reading environment
- **Testability**: Easier to test with explicit configuration
- **Security**: No accidental environment variable exposure

### Migration Required

#### For v1.x Users
```javascript
// OLD (v1.x)
const rag = new RAGSystem(); // Used process.env

// NEW (v2.0)  
const rag = new RAGSystem({
  database: { host: 'localhost', port: 5432, database: 'db', username: 'user', password: 'pass' },
  openai: { apiKey: 'your-key' }
});
```

See [MIGRATION.md](./MIGRATION.md) for complete migration guide.

## [1.1.0] - 2025-09-30

### Added

#### Buffer Processing
- **New `processDocumentFromBuffer()` method**: Process documents directly from Buffer objects without file system I/O
- **New `extractTextFromBuffer()` method**: Extract raw text from buffers for supported file types
- **Support for buffer processing**: TXT, HTML, Markdown, and JSON file types from memory buffers
- **Enhanced metadata handling**: Additional metadata support for buffer-processed documents

#### URL Processing  
- **New `processDocumentFromUrl()` method**: Download and process documents directly from HTTP/HTTPS URLs
- **New `processDocumentsFromUrls()` method**: Batch process multiple URLs with concurrency control
- **Automatic file type detection**: Smart detection from URL extensions and HTTP Content-Type headers
- **Robust error handling**: Comprehensive error handling for network issues, timeouts, and HTTP errors
- **Parallel processing**: Configurable concurrency limits for efficient batch URL processing
- **Temporary file management**: Automatic cleanup of downloaded temporary files

#### Enhanced Document Processing
- **Improved DocumentProcessor class**: Standalone document processing capabilities
- **Better chunk generation**: Enhanced chunking with improved metadata preservation
- **Extended file type support**: Added JSON file processing support
- **Enhanced error handling**: More detailed error messages and validation

#### Database Integration
- **Fixed LangChain vector store configuration**: Resolved table naming conflicts
- **Improved vector store initialization**: More explicit configuration to prevent errors
- **Enhanced database schema**: Better separation between document metadata and vector embeddings

### Changed
- **Updated package description**: Now includes buffer and URL processing capabilities
- **Enhanced documentation**: Comprehensive API documentation for new methods
- **Improved test coverage**: Added comprehensive tests for buffer and URL processing
- **Better error messages**: More descriptive error handling throughout the system

### Fixed
- **Database table configuration**: Fixed LangChain PGVectorStore to use correct table (`document_chunks_vector`)
- **Vector store initialization**: Resolved issues with table name conflicts
- **Document metadata handling**: Improved metadata preservation during processing

### Technical Details

#### New Dependencies
- Enhanced HTTP client for URL processing
- Improved buffer handling utilities
- Extended file type detection

#### API Changes
- **Backward Compatible**: All existing APIs remain unchanged
- **New Exports**: Added `DocumentProcessor` to utils exports
- **Enhanced Configuration**: New options for buffer and URL processing

#### Performance Improvements
- **Parallel URL processing**: Configurable concurrency for batch operations
- **Memory efficiency**: Direct buffer processing without temporary files
- **Database optimization**: Better vector store configuration

### Examples

#### Buffer Processing Example
```javascript
import { DocumentProcessor } from 'rag-system-pgvector/utils';

const processor = new DocumentProcessor();
const buffer = Buffer.from('Document content', 'utf8');

const result = await processor.processDocumentFromBuffer(
    buffer, 
    'document.txt', 
    'txt',
    { source: 'api-upload' }
);
```

#### URL Processing Example
```javascript
const urls = [
    'https://example.com/doc1.pdf',
    'https://example.com/doc2.html'
];

const results = await processor.processDocumentsFromUrls(urls, {
    maxConcurrent: 3,
    metadata: { batch: 'web-import' }
});
```

#### Integration Example
```javascript
// Process buffer and add to RAG system
const processed = await processor.processDocumentFromBuffer(buffer, 'doc.txt', 'txt');
await rag.documentStore.saveDocument(processed);

// Process URL and add to RAG system  
const urlDoc = await processor.processDocumentFromUrl('https://example.com/doc.html');
await rag.documentStore.saveDocument(urlDoc);
```

## [1.0.0] - 2025-09-29

### Added
- Initial release of RAG System Package
- PostgreSQL with pgvector integration
- LangChain and LangGraph integration
- OpenAI embeddings support
- Document processing for PDF, DOCX, TXT, HTML, and Markdown
- Vector similarity search
- Production-ready error handling
- Configurable chunking and embedding
- Web interface support
- Comprehensive documentation

### Features
- Multi-format document processing
- High-performance vector search
- AI-powered question answering
- Production-ready architecture
- Easy npm package integration
- Flexible configuration system