MemHop

About MemHop

This large README is shown as a plain-text preview. Read the full README at the source

<p align="center">
  <h1 align="center">MemHop</h1>
  <p align="center">
    <strong>Long-term memory for AI agents — a six-layer cognitive memory database in a single embedded file. Pure Go, zero infrastructure.</strong>
  </p>
  <p align="center">
    <a href="README.zh.md">中文</a>
    &middot;
    <a href="https://qyiun666.github.io/meowagent.github.io/">Website</a>
    &middot;
    <a href="https://github.com/meowagent/meowagent">MeowAgent (coming soon)</a>
  </p>
</p>

<p align="center">
  <a href="https://github.com/qyiun666/MemHop/actions/workflows/workflow.yml"><img src="https://github.com/qyiun666/MemHop/actions/workflows/workflow.yml/badge.svg" alt="CI"></a>
  <a href="https://pkg.go.dev/github.com/qyiun666/MemHop"><img src="https://pkg.go.dev/badge/github.com/qyiun666/MemHop.svg" alt="Go Reference"></a>
  <img src="https://img.shields.io/badge/go-1.27+-00ADD8.svg" alt="go">
  <img src="https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue.svg" alt="license">
</p>

<p align="center">
  <strong>Current: v1.6.5</strong>
</p>

---

MemHop is an **embedded long-term memory database for AI agents and LLM applications**, written in pure Go. It is not a vector database — it is a memory system modeled after how the human brain organizes knowledge, with identity, episodic recall, semantic compression, a knowledge graph and archival storage. One agent, one `.meh` file, zero infrastructure.

MemHop is an **agent-dedicated** memory database: each agent binds to exactly one `.meh` file, and a file-level exclusive lock guarantees a single instance per file (a second `Open` fails fast). It runs on **Linux, macOS, and Windows** with no cgo and no external service beyond your LLM endpoint.

Built as the brain memory of [MeowAgent](https://github.com/meowagent/meowagent) (coming soon), MemHop works as an embedded organ rather than a standalone service. No server to run, no configuration to manage — just open a file and your agent has memory.

> **Our stance on agent memory.** Memory should not be an afterthought bolted on with a vector database plugin or a plain-text log dumped into a context window. An agent without internalised memory is just a stateless function pretending to be intelligent. MemHop exists because we believe memory must be *cognitive* — structured, compressed, consolidated, and forgotten the way a human brain does — and *embedded* — living inside the agent process itself, not behind a network call. One file, zero infrastructure, a mind that grows with every conversation.

## Features

- **Six-Layer Architecture** — L0 Profile → L1 Engram → L2 Context → L3 Knowledge → L4 Archive → L5 Plan, with Dream consolidation
- **Scene-is-the-session memory loop** — one L2 scene = one host session. `Search` reads that scene's depth-1 topic set straight from the in-memory cache (zero LLM, zero embedding, no scoring) *and opens the turn*, and from then on the library is what holds both ids — the write calls name no scene and no topic. The host records the turn itself: `AppendArchive` writes what happened while it ran (dialogue originals and operation events being L4 content that differs only by `Kind`), and `Update` closes the turn with how it opened and how it ended, distilling its utterances into that topic's keywords in exactly one LLM call. Everything a turn holds lives under that one id, and the task tree it opened is L5. The scene's `FusedKeywords` set *is* the context a host injects
- **V2 Storage** — `.meh` format (`FormatVersion=0x0012`) with A/B dual headers, per-record CRC32 + torn-write truncation recovery, mmap zero-copy, snapshot/checkpoint. Record frames carry an 8-byte `agent_id` (26-byte header) and the engine indexes every record by `(agent, idHash)` domain. The L3 knowledge graph lives in the file-wide reserved shared domain, which holds nothing else. **Only `0x0012` opens** — files at `0x0011` or older are rejected, with no migration path: an older file's profiles carry no `agent_type`, so every domain in it decodes as the primary agent — not one wrong value somewhere but the same wrong value in every domain at once, against a rule that a file has exactly one
- **Multi-Agent Domains** — `Open(path, llm, defaults, profile)` → a DB handle whose domains are reached as handles, never as ids: `Primary()` for the domain the file was opened on, `SubAgent(llm, profile)` for one created under it and addressed by its name. Many agents share one `.meh` file with fully isolated per-agent domains (caches, Dream pipelines, domain locks); same-agent operations serialize, different agents run in parallel; idle domains reclaim memory on access cadence (`Defaults.AgentIdleTTLMs`) while their records stay on disk. One exception (below): the L3 knowledge graph is file-wide shared
- **L1 Scene Hypergraph** — Dream creates co-occurrence hyperedges between scenes whose keyword sets overlap (Jaccard ≥ 0.15, a floor the engine fixes rather than a knob) and decays/prunes them over time; an edge is weighted by the similarity it was born from and fades from there — it strengthens only when an endpoint's turn list came out different — a turn lost counting as much as one gained — so re-measuring a pair whose lists are unchanged, even after the same turns are re-distilled into different keywords, never undoes the fade. L1 is maintained by Dream for explicit graph queries and future association — reads never score or spread activation
- **Dream Pipeline** — consolidation over L0–L2 plus retention on both content and plan: L2 compress → index rebuild → L1 nodes/hyperedges rebuild → L1 decay → L0 distill (emotion/MBTI), with `l4_prune` (drops a topic's content past the retention window — seven days by default, `Defaults.ContentRetentionMs`) and `l5_prune` (drops plan nodes past it, exempting a tree still in flight) on every pass; returns a per-stage `DreamReport`
- **L3 Knowledge Graph** — multiple independent hypergraphs with node import carrying positional source refs and relation edges (an edge is its members plus its kind, so one node pair can hold several relations), graph deletion, keyword/type/id lookup that ANDs together, and BFS subgraph queries. The graph pool is **file-wide**: every agent domain of the file shares one L3 pool (project knowledge is imported once, visible to all), and the pool's lifetime is the file's, not any one domain's
- **Single Instance by Design** — one `.meh` file has exactly one owner: a cross-platform exclusive lock (linux/darwin/windows) makes a second open fail fast, and the embedded path runs with no server process and no background daemon
- **Minimal & Embeddable** — 3 direct Go deps (xxhash, go-openai, golang.org/x/sys) — **the engine contacts no embedding / vector service at all**, and there is no dimension to declare in the config; `sync.RWMutex` + `atomic.Pointer`, zero infrastructure

## Quick Start

> Full integration guide (config, all layer APIs, turns and trajectories, pitfalls):
> [INTEGRATION_GUIDE.md](INTEGRATION_GUIDE.md) · 中文: [INTEGRATION_GUIDE.zh.md](INTEGRATION_GUIDE.zh.md)

```go
import (
    "context"
    "log"
    "os"
    "time"

    memhop "github.com/qyiun666/MemHop/api"
)

db, err := memhop.Open(
    "agent.meh", // the whole database; no server, no dimension to declare
    memhop.LlmConfig{ // required, validated before the path is touched
        APIURL: "https://api.openai.com/v1",
        APIKey: os.Getenv("OPENAI_API_KEY"),
        Model:  "gpt-4o-mini",
    },
    memhop.DefaultMemHopDefaults,
    // Required for a file that is not there yet: opening one has to know whose
    // memory it is. An existing file keeps the primary it already has.
    &memhop.ProfileInput{Name: "my-agent", Role: "assistant"},
)
if err != nil {
    log.Fatal(err)
}
defer db.Close()

// One .meh file carries isolated domains, and they come back as handles rather
// than as ids. Primary is the domain the file was opened on; SubAgent creates
// (or returns) one addressed by name, optionally on its own LLM endpoint.
sess, err := db.Primary()
if err != nil {
    log.Fatal(err)
}

// Read memory = read one scene (a scene IS a host session), which also
// opens the turn about to run. An empty SceneID continues the domain's current
// scene — after a reopen it restores the one whose turn counter ran furthest —
// so a host running one agent over one library names no scene at all;
// NewScene: true is the only way to start a fresh conversation. Naming a
// SceneID scopes this read to it and it must already exist, otherwise
// ErrNotFound (L3ID anchors only a read that creates a scene). The read costs
// no LLM, no embedding and no scoring; NewTopicID is the topic this turn
// lives in.
res, err := sess.Search(memhop.SearchQuery{})
if err != nil {
    log.Fatal(err)
}
for _, topic := range res.Topics { // this session's depth-1 set = the context
    _ = topic.FusedKeywords
}

// While the turn runs, the host records what happened into the turn Search
// opened — which turn that is the library remembers, so this call names no ids
// and with no turn open it is refused. Dialogue and events are the same kind of
// record: they differ only by Kind. Each call returns the slot its record took.
_, _ = sess.AppendArchive(memhop.ArchiveInput{
    Kind:      memhop.KindEvent,
    EventType: "tool_call",
    Content:   `{"tool":"grep"}`,
    CreatedAt: time.Now().UnixMilli(),
})

// End of turn: Input and Output land on the dialogue slots a reader looks for
// them on (Seq 1 and 2), so closing the same turn again rewrites those two
// lines instead of accumulating; Outcome is the host's own word for how the
// round ended, stored as one event per call. The turn's utterances are then
// distilled into its keyword track, returned with the topic.
topic, err := sess.Update(memhop.TurnEnd{
    Input:     "What did we discuss yesterday?",
    Output:    "Agent: ...",
    Outcome:   "resolved",
    CreatedAt: time.Now().UnixMilli(),
})
if err != nil {
    log.Fatal(err)
}
_ = topic.FusedKeywords

// Dream consolidation (L0-L2); an empty sceneID sweeps every scene of the
// domain. Update already schedules it in the background once a scene's
// topic count passes the threshold, so hosts rarely call it.
report, err := sess.Dream(context.Background(), "")
```


> **Concurrency contract.** Same-agent operations (Search / Update / Dream / write APIs) are serialized by the library's per-agent domain lock; different agents run in parallel on a `*DB`, so the host needs no external queue. `*memhop.Session` carries no cross-domain state beyond its bound domain — the scene and turn it is working are held by the domain, not by the caller. The file's exclusive lock still allows only one process per `.meh` file; `DB` exposes no locking API: the domain lock is the library's, and a host critical section needs its own. Nothing about the open turn is process-wide, so one more agent is one more `Open` — including one made between another round's start and its close, which leaves each round's keys and content apart.

Prerequisites: Go 1.27+ and an OpenAI-compatible LLM endpoint (configured through `api.LlmConfig`, which `api.Open` takes) — no embedding / vector service needed

### API Overview

| Group | Methods |
|-------|---------|
| Core loop | `Search(q) → opens the turn` · `PlanNodeAdd(parentSeq, title) → seq` (parentSeq 0 opens the turn's tree) / `PlanNodeUpdate(PlanStep{Seq, Status, …})` (the plan comes first, one step at a time) · `AppendArchive(ArchiveInput{...}) → seq` · `Update(TurnEnd{Input, Output, Outcome, CreatedAt}) → topic` · `Dream(ctx, sceneID)` — none of the turn writes names a scene or a topic id: `Search` opens the turn they all key on |
| L0 Profile | `GetL0` · `UpdateL0` |
| L1 Engram (read-only) | `ListL1() → []SceneNodeView` — every scene node of the domain in a stable order, ids as hex. Dream builds the nodes and the co-occurrence edges between them and is the only writer, so there is no L1 write call. `Importance` / `Valence` / `Arousal` are what consolidation computed, and `EmotionSet` says whether any pass ever stamped the two signals (0 is a legal reading, so the values alone cannot); `EdgeIDs` has no read of its own — two nodes sharing one are a pair Dream judged related |
| L2 Context | `ListScenes([l3ID])` · `UpdateScene(id, {Name, L3ID, Force})` · `RenameTopic(topicID, name)` · `SceneContext(sceneID, or "" for the domain's current scene)` · `MergeScenes` · `DeleteTopic` · `DeleteScene` |
| L3 Knowledge | `GetL3` · `ListL3` · `ImportL3` (reports every graph the batch resolved a domain into, including one it added nothing new to) · `UpdateL3` · `DeleteL3` · `QueryL3Nodes` · `QueryL3Subgraph` (its `edgeKinds` narrows the walk; an undefined kind is refused, not answered with an empty subgraph) |
| L4 Archive | `AppendArchive(ArchiveInput{Kind, Seq, Role, ContentType, EventType, NodeSeq, Content, CreatedAt}) → seq` is the only way content enters a topic, and the topic it enters is the turn `Search` opened: the call names no ids, so a mistyped or invented key is not something a host can pass, and with no turn open it is refused (`ErrInvalidQuery`) rather than landing content under a key no read ever lists, `Seq: 0` allocates and the slot taken comes back, and naming a held slot rewrites it. An event's `NodeSeq` has to name a step this turn created (`0` binds it to nothing). `SearchL4(q)` is the one read over both kinds: keyword (case-insensitive), time range, ids, topic, `Kind` (utterance / event), `NodeSeq` (one plan step **and every step under it**, which means nothing outside its turn; `0` leaves the condition unset) and content type are conditions, not modes — an unset `Kind` selects both; `Limit` keeps the tail of the read's order (slot order inside one topic, record-time order across topics) |
| Turn events (L4, kind `event`) | A turn's events are L4 content of kind `event` under the topic id Search issued for it (swept past the retention window — seven days by default, configurable; no delete API); read them with `SearchL4(L4Query{TopicID, Kind: &KindEvent})`, append them with `AppendArchive`. A topic's first event is Seq 3, because slots 1 and 2 belong to its dialogue |
| L5 Plan tree | `PlanNodeAdd(parentSeq, title) → seq` · `PlanNodeUpdate(PlanStep{Seq, Status, Title, Summary})` · `PlanState()` — the tree is what L5 itself records: one node per record, addressed by the turn **plus a step ordinal the library hands out** (`1, 2, 3 …` inside that turn — issued above everything that still names one: this turn's live steps **and** the highest ordinal its surviving events bound to a step, since a step and its events share one address while the two age separately. Not permanently unique — a sweep frees a swept step's ordinal once the events naming it go too — and a turn's numbering can therefore carry a gap, so a host reads no density into it and treats an ordinal it held before, on a turn older than the retention window, as a new step's address); the turn is the one `Search` opened, so these calls name no id, which makes `PlanState()` and `SearchL4{TopicID, Kind}` one key across two stores, and `SearchL4{TopicID, NodeSeq}` reads back the work of one step and every step under it. A node exists only because something created it: a `parentSeq` of 0 opens the turn's tree, one naming a step the tree does not hold is refused rather than answered by growing one, and `PlanNodeUpdate` restates a step that is already there — `Status` every time (`in_progress` / `done` / `failed`, there is no "leave it" spelling), `Title`/`Summary` kept when left blank. A created step starts `in_progress`, so the model has no "planned but not started" state. A plan write stores no content: a step's events are L4 records the host appends itself |
| DB handle | `Open(path, llm, defaults, profile)` · `Primary()` · `SubAgent(llm, profile)` · `Checkpoint` · `CompactTo(newPath)` (defragmented copy; the destination is the caller's to constrain) · `Stats` (file bytes and live record count — the numbers a compaction decision is made from) · `Close` · `IsClosed` |

## Architecture

```
Layer   Name             Human Parallel          Mechanism
─────   ──────────────   ───────────────────     ─────────────────────────────────────────────
 L5     Plan             Task tree               One node per step, keyed by the turn that opened it; expired trees are swept by Dream
 L4     Archive            Turn content            A turn's dialogue originals and operation events (Kind), addressed by (topic, Seq); retention window (seven days by default) — the keyword track is what outlives it
 L3     Knowledge        Semantic memory         Multi-source hypergraph knowledge base
 L2     Context          Working memory          Topics on a scene's surface, plus turns Dream folded under a fused group (one sink per topic)
 L1     Engram           Scene hypergraph        Scene nodes + keyword-overlap hyperedges; maintained by Dream for explicit graph queries
 L0     Profile          Identity                Agent personality, preferences & language habits
```

### Dream Pipeline

The Dream cycle is an automatic consolidation pass inspired by how sleep processes the day's experiences. It acts on **L0–L2 only** (L3 distillation is out of scope) plus retention pruning over L4 content and L5 plan nodes:

1. **L2 compression** — the LLM groups related topics per scene; each target scene runs in its own goroutine under a bounded fan-out (the whole pass holds the domain lock, so the scene count is not the concurrency), sinking merged topics under a new depth-1 fused node
2. **L1 rebuild** — scene nodes are synced from L2, the L2Meta topic cache is rebuilt in the same scan, and keyword-overlap hyperedges are created or refreshed
3. **L1 decay** — scene importance and edge weights decay over time, weak nodes are pruned
4. **L0 distill** — emotion/MBTI and a personality summary are distilled from the ranked L1 samples and merged into the stored profile: `Name`, `Role` and `Preferences` are left alone, and `Personality` is the one host-written field Dream also refines, so a host write that omits it clears the refined value until the next pass. The per-node emotions that same reply carries are backfilled onto the L1 nodes that have never been stamped — a node already carrying a reading keeps it, including a legal all-zero one; the stage is skipped when there are no samples to read

Trigger: once a scene's depth-1 topic count passes `Defaults.SceneDreamTopicThreshold` (default 24), `Update` schedules that scene's Dream in the background, one pass in flight per scene; hosts may also call it. `Dream(ctx, sceneID) (*DreamReport, error)` holds the domain lock for the whole cycle, sweeps every scene of the domain when `sceneID` is empty (scenes below `DreamCompressMinTopics` are skipped) and honours `ctx` cancellation between stages.

### Read & write path

**There is no scored retrieval.** A scene is a host session, so the engine never guesses which scene a message belongs to:

| Path | What it does | Cost |
|------|--------------|------|
| `Search(SearchQuery{SceneID, L3ID, NewScene})` | empty `SceneID` → continue the domain's current scene (after a reopen it restores from the records, the one whose turn counter ran furthest), creating one only when the domain holds none; `NewScene: true` → a fresh scene, the only way to start a second conversation over one domain; a named `SceneID` → that scene's depth-1 topics (user-timestamp order) plus the L0 profile — and `NewTopicID`, the topic this read opens for the coming turn | in-memory read (L2Meta), zero LLM / embedding / scoring; the only write is the scene record (its turn counter) |

Frequently Asked Questions