🧠 MemContext: Next-Generation Multi-modal Agent Memory Framework

# 🧠 MemContext: Next-Generation Multi-modal Agent Memory Framework

<div align="center">

![License](https://img.shields.io/badge/license-Apache%202.0-blue)
![Python](https://img.shields.io/badge/python-3.10%2B-blue)
![Status](https://img.shields.io/badge/Status-In%20Development-orange)
![Multi-modal](https://img.shields.io/badge/Modality-Audio%20|%20Video%20|%20Image%20|%20Document%20|%20Text-green)

**The First Full-Modal Cross-Platform Memory Framework for AI Agents**

*Store Everything · Search Everything · Frame-Level Positioning · Multi-Platform Support*

[中文](README_zh.md) • [Core Features](#-core-features) • [Architecture](#-architecture) • [Use Cases](#-use-cases) • [Quick Start](#-quick-start)

</div>

---

## 📖 Introduction

**MemContext is dedicated to building a persistent, high-fidelity, evolvable, and cross-platform full-modal memory repository for AI Agents.**

Existing Agent Memory frameworks are mostly text-centric, often failing to preserve information from videos, audio, and documents in their original form. This leads to the loss of multi-modal content and context misalignment when recalling information across different timelines.

Designed with a **"Full-Modal Native"** principle, MemContext provides end-to-end capabilities for video, audio, and document streams—from **full-modal input and streaming storage to frame-level retrieval**. Whether it's a massive library of long-term footage or easily overlooked visual details, everything can be reliably written and precisely retrieved when needed.

Furthermore, MemContext is deeply adapted to mainstream ecosystems like **n8n, Coze, MCP, and Skills**. Through a plugin-based mechanism, it empowers existing Agent workflows with integrated cross-platform memory capabilities.

---

## ✨ Core Features

### 1. ♾️ Full-Modal Storage & Retrieval

**Beyond the limits of Text-to-Text.** MemContext unifies the processing paradigm for heterogeneous data.

* **Input:** Unified processing of video, audio, images, documents, and text.
* **Multi-modal Native Indexing:** Adopts a "Parallel Retrieval Workflow," establishing vector indexes directly for visual and auditory signals, rather than relying solely on their textual descriptions.
* **Cross-Modal Retrieval:** Supports "Search Video by Image" and "Search Text by Audio."

### 2. 🎞️ Infinite Stream Processing

**Forget the anxiety of limited context windows.**

* Designed specifically for **Long Context Data** and **24/7 Continuous Operation**.
* Supports the ingestion of infinite-length videos and continuous audio streams.
* **Dynamic Chunking:** Built-in continuity checks ensure that memory context does not break when processing massive meeting records, TV series, or surveillance streams.

### 3. 🎯 0.1s Spatiotemporal Precision

**Stop hallucinating timestamps.**

* Traditional RAG and Memory systems can only tell you "which file the answer is in"; MemContext tells you **"at which minute and second the answer occurs."**
* **SOTA Precision:** Capable of retrieving and positioning video/audio segments with **0.1-second accuracy**.
* **Value:** Instead of returning a bloated 1GB file, it returns the precise 0.1-second key segment, achieving true "needle in a haystack" retrieval.

### 4. 🤖 Seamless Multi-Platform Integration

**Out-of-the-box, deep compatibility with mainstream agent platforms.**

* **n8n Support:** Bring persistent, retrievable multi-modal memory to automation workflows.
* **Coze Support:** Empower Coze Agents with cross-session knowledge and context continuity.
* **MCP Support:** Facilitate enterprise-level agent management of multi-user and multi-session memory data.
* **Claude Skills Support:** Empower Claude assistants, significantly enhancing context depth and data invocation capabilities.
* **Unified API:** Regardless of the platform, integration is easy via standard RESTful APIs or SDKs.
* **Value:** Make your multi-modal memory capabilities truly "plug-and-play," adaptable across platforms, scenarios, and ecosystems without barriers.

---

## 🏗️ Architecture

MemContext draws inspiration from human cognitive processes and operating system storage structures, adopting a tiered storage architecture paired with a dual-retrieval engine.

<div align="center">
<img src="assets/MemContext_Workflow_v0.png" alt="MemContext Architecture Workflow" width="100%">
</div>

### 🧠 Memory Lifecycle

The journey of data through the memory pipeline:

1. **Multi-modal Input:** A unified processor extracts multi-dimensional features from heterogeneous data.
2. **STM (Short-Term Memory):** Handles immediate context streams, performing Embedding calculations and preliminary filtering.
3. **MTM (Medium-Term Memory):** A session-based buffer introducing a **Heat Calculation Algorithm**. The system dynamically judges data value based on Visit Frequency (), Interaction Depth (), and Recency ().
4. **LTM (Long-Term Memory):** High-heat information "crystallizes" into permanent storage (User Profiles & Kn

... (truncated)