## Efficient Lifelong Memory for LLM Agents โ Text & Multimodal
Store, compress, and retrieve long-term memories with semantic lossless compression. Now with multimodal support for text, image, audio & video.
Works with any AI platform that supports MCP (text memory) or Python integration (full multimodal)
๐ Atlas Cloud is a full-modal, OpenAI-compatible AI inference platform โ point SimpleMem's OPENAI_BASE_URL at it to power memory construction, retrieval, and judging with DeepSeek, Qwen, GLM, Kimi, MiniMax and more through a single API, no multi-vendor setup needed. Budget-friendly coding plan available.
|
**๐ Auto โ Text** (pure text input)
```python
from simplemem import SimpleMem
mem = SimpleMem() # auto mode
# add_dialogue() โ text backend auto-selected
mem.add_dialogue(
"Alice",
"Bob, let's meet at Starbucks tomorrow at 2pm",
"2025-11-15T14:30:00",
)
mem.add_dialogue(
"Bob",
"Sure, I'll bring the market analysis report",
"2025-11-15T14:31:00",
)
mem.finalize()
answer = mem.ask("When and where will Alice and Bob meet?")
# โ "16 November 2025 at 2:00 PM at Starbucks"
```
|
**๐ง Auto โ Omni** (multimodal input)
```python
from simplemem import SimpleMem
mem = SimpleMem() # auto mode
# add_image() โ omni backend auto-selected
mem.add_text(
"User loves hiking in the Rocky Mountains.",
tags=["session_id:D1"],
)
mem.add_image("photo.jpg", tags=["session_id:D1"])
mem.add_audio("voice_note.wav", tags=["session_id:D1"])
result = mem.query("What does the user enjoy?", top_k=5)
for item in result.items:
print(item["summary"])
mem.close()
```
|
> **๐ก Tip**: Auto mode picks the lightest backend that fits your data. You can still use `mode="text"` or `mode="omni"` explicitly if you prefer.
---
### ๐งฌ Advanced: Optimize Retrieval Config
Tune retrieval hyperparameters offline on your own dev set, then deploy the resulting `Config` for inference. This is a thin wrapper around EvolveMem's self-evolution loop:
```python
import simplemem
from simplemem import SimpleMem, load_config
# mem is a finalized SimpleMem instance with memories already built
dev_questions = [
("When is the meeting?", "2pm tomorrow at Starbucks"),
("What should Bob prepare?", "market analysis report"),
]
config = simplemem.optimize(mem, dev_questions, max_rounds=3)
config.save("my_config.json")
# Later, deploy with the optimized config
config = load_config("my_config.json")
mem = SimpleMem(config=config)
```
> EvolveMem runs an LLM-driven Evaluate โ Diagnose โ Propose โ Guard cycle over your dev questions, adjusting global retrieval flags (top_k, fusion mode, answer verification, reflection rounds, ...). For the full standalone version with benchmark adapters and per-category overrides, see [`EvolveMem/`](EvolveMem/).
---
### ๐ Advanced: Parallel Processing
For large-scale dialogue processing, enable parallel mode:
```python
from simplemem import create
mem = create(
mode="text",
clear_db=True,
enable_parallel_processing=True, # โก Parallel memory building
max_parallel_workers=8,
enable_parallel_retrieval=True, # ๐ Parallel query execution
max_retrieval_workers=4
)
```
> **๐ก Pro Tip**: Parallel processing significantly reduces latency for batch operations!
---
## ๐ Overview
**SimpleMem** is a unified memory stack for LLM agents, built on one principle: store *semantically lossless* memory at high information density, so an agent recalls more while spending far fewer tokens. The package brings together three works that share this principle but attack different parts of the problem.
### ๐ SimpleMem: the efficiency core (text)
Most memory systems force a bad trade-off. They either passively accumulate raw interaction history (redundant, token-hungry) or run expensive reasoning loops to filter noise (slow, costly). SimpleMem instead compresses interactions through a three-stage pipeline:
| Stage | What it does |
|:--|:--|
| **1. Semantic Structured Compression** | Distills unstructured interactions into compact memory units (self-contained facts with resolved coreferences and absolute timestamps), each indexed through multiple complementary views for flexible retrieval. |
| **2. Online Semantic Synthesis** | Merges related context within a session into unified abstract representations, removing redundancy as memory is built rather than at query time. |
| **3. Intent-Aware Retrieval Planning** | Infers the search intent behind a query to decide *what* to retrieve and assemble a precise, compact context. |
On the LoCoMo benchmark this delivers a 26.4% average F1 gain over prior systems while cutting inference-time token consumption by roughly 30x. Mechanism details (hybrid index layers, compression examples, retrieval planning): [**SimpleMem text memory โ**](docs/text-memory.md).
### ๐ง Omni-SimpleMem: multimodal memory (text, image, audio, video)
Omni-SimpleMem extends the compression-first philosophy to four modalities, built on three principles: **Selective Ingestion** (entropy-driven filtering per modality), **Progressive Retrieval** (hybrid FAISS + BM25 with pyramid token-budget expansion), and **Knowledge Graph Augmentation** (multi-hop cross-modal reasoning). Rather than being hand-designed, its architecture was *discovered* by an autonomous research pipeline that ran around 50 experiments across two benchmarks, diagnosing failure modes, proposing architectural changes, and even repairing data-pipeline bugs with no human in the inner loop. Tellingly, the bug fixes and architectural changes each contributed more than all hyperparameter tuning combined, taking the system from a naive baseline to state-of-the-art on both LoCoMo and Mem-Gallery. Full docs: [**Omni-SimpleMem โ**](OmniSimpleMem/).
### ๐งฌ EvolveMem: self-evolving retrieval
EvolveMem closes a blind spot shared by almost every memory system: the stored content evolves, but the *retrieval* machinery (scoring functions, fusion strategies, answer-generation policies) stays frozen after deployment. EvolveMem runs a closed-loop AutoResearch process (**Evaluate โ Diagnose โ Propose โ Guard โ Repeat**) in which an LLM diagnoses per-question failures and proposes configuration changes, guarded by automatic rollback on regression and exploration incentives during stagnation. It discovers new retrieval dimensions (query decomposition, entity-swap, answer verification) not in the original design, improves LoCoMo by 25.7% relative over the strongest baseline, and its evolved configurations transfer positively across benchmarks. Full docs: [**EvolveMem โ**](EvolveMem/).
### How they fit together
`from simplemem import SimpleMem` gives you the text core with automatic routing to the multimodal backend, and `simplemem.optimize(...)` taps EvolveMem to tune retrieval for your own data. One package, one mental model: compress losslessly, retrieve by intent, and let the system keep improving itself.
---
## ๐ฆ Installation
### ๐ Notes for First-Time Users
- Ensure you are using **Python 3.10+ in your active environment**, not just installed globally.
- An OpenAI-compatible API key must be configured **before running any memory construction or retrieval**, otherwise initialization may fail.
- When using non-OpenAI providers (e.g., Qwen or Azure OpenAI), verify both the model name and `OPENAI_BASE_URL` in `config.py`.
- For large dialogue datasets, enabling parallel processing can significantly reduce memory construction time.
### ๐ Requirements
- ๐ Python 3.10+
- ๐ OpenAI-compatible API (OpenAI, Qwen, Azure OpenAI, etc.)
### ๐ ๏ธ Setup
```bash
# ๐ฅ Clone repository
git clone https://github.com/aiming-lab/SimpleMem.git
cd SimpleMem
# ๐ฆ Install dependencies (pinned versions)
pip install -r requirements.txt
# โ OR โ install as an editable package
pip install -e . # default: text + multimodal + evolver
pip install -e ".[server]" # + MCP / HTTP server (mcp, fastapi, ...)
pip install -e ".[all]" # everything, including dev tools
# โ๏ธ Configure API settings
cp config.py.example config.py
# Edit config.py with your API key and preferences
```
### โ๏ธ Configuration Example
```python
# config.py
OPENAI_API_KEY = "your-api-key"
OPENAI_BASE_URL = None # or custom endpoint for Qwen/Azure
LLM_MODEL = "gpt-4.1-mini"
EMBEDDING_MODEL = "Qwen/Qwen3-Embedding-0.6B" # State-of-the-art retrieval
```
Because SimpleMem talks to any OpenAI-compatible endpoint, you can point it at a managed provider such as [Atlas Cloud](https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=SimpleMem) by setting `OPENAI_BASE_URL`:
```python
# config.py โ using Atlas Cloud as an OpenAI-compatible backend
OPENAI_API_KEY = "your-atlascloud-api-key"
OPENAI_BASE_URL = "https://api.atlascloud.ai/v1"
LLM_MODEL = "deepseek-ai/deepseek-v4-pro" # a reasoning model โ keep max tokens generous
EMBEDDING_MODEL = "Qwen/Qwen3-Embedding-0.6B"
```
> `deepseek-ai/deepseek-v4-pro` is a reasoning model, so leave enough output budget; if responses come back empty, raise the token limit (>= 512). Any other OpenAI-compatible chat model on Atlas Cloud (e.g. `Qwen/Qwen3-Next-80B-A3B-Instruct`, `zai-org/glm-5`, `moonshotai/kimi-k2.6`) works the same way.
---
## ๐ณ Run with Docker
The **MCP Server** can be run in Docker for a consistent, isolated environment. Data (LanceDB and user DB) is persisted in a host volume.
### Prerequisites
- [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/)
### Quick run
```bash
# From the repository root
docker compose up -d
```
- **Web UI:** http://localhost:8000/
- **REST API:** http://localhost:8000/api/
- **MCP (SSE):** http://localhost:8000/mcp/sse?token=<TOKEN>
Data is stored in `./data` on the host (created automatically).
### Custom configuration
1. Copy the environment template and edit it:
```bash
cp .env.example .env
# Edit .env: set JWT_SECRET_KEY, ENCRYPTION_KEY, LLM_PROVIDER, model URLs, etc.
```
2. Run with the env file:
```bash
docker compose --env-file .env up -d
```
### Using Ollama on the host
When `LLM_PROVIDER=ollama` and Ollama runs on your machine (not in Docker), set in `.env`:
```bash
LLM_PROVIDER=ollama
OLLAMA_BASE_URL=http://host.docker.internal:11434/v1
```
On Linux, `host.docker.internal` is enabled automatically via the Compose file.
### Useful commands
```bash
docker compose logs -f simplemem # Follow logs
docker compose down # Stop and remove containers
```
> ๐ For self-hosting the MCP server (Docker or bare metal), see [MCP Documentation](MCP/README.md).
---
## ๐ MCP Server *(text memory)*
SimpleMem is available as a **cloud-hosted memory service** via the Model Context Protocol (MCP), enabling seamless integration with AI assistants like Claude Desktop, Cursor, and other MCP-compatible clients.
**๐ Cloud Service**: [mcp.simplemem.cloud](https://mcp.simplemem.cloud) โ or self-host the MCP server locally using [Docker](#-run-with-docker).
### Key Features
| Feature | Description |
|---------|-------------|
| **Streamable HTTP** | MCP 2025-03-26 protocol with JSON-RPC 2.0 |
| **Multi-tenant Isolation** | Per-user data tables with token authentication |
| **Hybrid Retrieval** | Semantic search + keyword matching + metadata filtering |
| **Production Optimized** | Faster response times with OpenRouter integration |
### Quick Configuration
```json
{
"mcpServers": {
"simplemem": {
"url": "https://mcp.simplemem.cloud/mcp",
"headers": {
"Authorization": "Bearer YOUR_TOKEN"
}
}
}
}
```
> ๐ For detailed setup instructions and self-hosting guide, see [MCP Documentation](MCP/README.md)
---
## ๐ Reproduce Paper Results
Reproduce the LoCoMo / MemBench / Mem-Gallery numbers from the papers. Each pillar has its own benchmark runner in its own directory. Install the benchmark extras first: `pip install -e ".[benchmark]"`.
### ๐ SimpleMem (text) โ LoCoMo
Run from the repository root:
```bash
python test_locomo10.py # full LoCoMo benchmark
python test_locomo10.py --num-samples 5 # quick subset
python test_locomo10.py --result-file my_results.json
```
### ๐งฌ EvolveMem โ self-evolution + LoCoMo / MemBench
Run from the `EvolveMem/` directory (see [`EvolveMem/README.md`](EvolveMem/README.md)):
```bash
cd EvolveMem
python run_evolution.py --data data/locomo10.json --max-rounds 7
python run_benchmark.py locomo --sample 0 --initial weak --max-rounds 3
python run_benchmark.py membench --agent FirstAgent --max-rounds 3
```
### ๐ง Omni-SimpleMem โ LoCoMo / Mem-Gallery
Run from the `OmniSimpleMem/` directory (see [`OmniSimpleMem/README.md`](OmniSimpleMem/README.md)):
```bash
cd OmniSimpleMem
python benchmarks/locomo/run_locomo.py --data-path /path/to/locomo10.json --model gpt-4o
```
---
## ๐บ๏ธ Roadmap
Current capability by integration channel:
| Capability | Python (`pip install`) | MCP server (Claude Desktop, Cursor, ...) |
|:--|:--:|:--:|
| Text memory | โ
| โ
|
| Multimodal (image / audio / video) | โ
| โฌ planned |
| `optimize()` self-evolving retrieval | โ
| โฌ planned |
Planned work to close the gap (the MCP server is a standalone multi-tenant text service; these are real features, not doc fixes):
- [ ] **Multimodal over MCP.** Add `memory_add_image` / `memory_add_audio` / `memory_add_video` tools. Needs a file-upload path (base64 or URL, since MCP cannot pass local file paths), a multi-tenant adaptation of the Omni-SimpleMem storage backend, and server-side vision/audio model access.
- [ ] **EvolveMem over MCP.** Expose `optimize()` as an MCP tool. More tractable than multimodal (text in, JSON config out, no file transport), but the MCP retriever currently honors only `semantic_top_k` / `keyword_top_k` of the ~10 dimensions EvolveMem evolves. Requires extending the MCP retriever to support the remaining knobs (structured top_k, fusion mode/weights, entity swap, query decomposition, answer verification), an adapter to run the evolution loop over a tenant's stored memories, per-tenant config persistence, and async execution (the loop is LLM-heavy and would time out a synchronous request).
- [ ] **Docker** inherits both automatically once the MCP server supports them (add multimodal deps to the image and an Omni storage volume).
For full multimodal and self-evolving retrieval today, use the Python API (see [Quick Start](#-quick-start)).
---
## ๐ Citation
If you use SimpleMem in your research, please cite:
```bibtex
@article{simplemem2026,
title={SimpleMem: Efficient Lifelong Memory for LLM Agents},
author={Liu, Jiaqi and Su, Yaofeng and Xia, Peng and Zhou, Yiyang and Han, Siwei and Zheng, Zeyu and Xie, Cihang and Ding, Mingyu and Yao, Huaxiu},
journal={arXiv preprint arXiv:2601.02553},
year={2026},
url={https://arxiv.org/abs/2601.02553}
}
```
```bibtex
@article{evolvemem2026,
title={EvolveMem: Self-Evolving Memory Architecture via AutoResearch for LLM Agents},
author={Liu, Jiaqi and Ye, Xinyu and Xia, Peng and Zheng, Zeyu and Xie, Cihang and Ding, Mingyu and Yao, Huaxiu},
journal={arXiv preprint arXiv:2605.13941},
year={2026},
url={https://arxiv.org/abs/2605.13941}
}
```
```bibtex
@article{omnisimplemem2026,
title = {Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory},
author = {Liu, Jiaqi and Ling, Zipeng and Qiu, Shi and Liu, Yanqing and Han, Siwei and Xia, Peng and Tu, Haoqin and Zheng, Zeyu and Xie, Cihang and Fleming, Charles and Ding, Mingyu and Yao, Huaxiu},
journal = {arXiv preprint arXiv:2604.01007},
year = {2026},
}
```
---
## ๐ License
This project is licensed under the **MIT License** - see the [LICENSE](LICENSE) file for details.
---
## ๐ Acknowledgments
We would like to thank the following projects and teams:
- ๐ **Embedding Model**: [Qwen3-Embedding](https://github.com/QwenLM/Qwen) - State-of-the-art retrieval performance
- ๐๏ธ **Vector Database**: [LanceDB](https://lancedb.com/) - High-performance columnar storage
- ๐ **Benchmark**: [LoCoMo](https://github.com/snap-research/locomo) - Long-context memory evaluation framework
