Agent Memory as a File Format

1. What an LLM‑based agent’s memory really is

One‑line note: Define the core data pieces (raw text, embeddings, metadata, retrieval scores) that constitute “memory” for an LLM agent.

An LLM‑based autonomous agent does not store a permanent “brain” inside the model itself. Instead, it writes observations, actions, and intermediate results to an external store that can be queried later. In practice each memory entry (sometimes called a record or node) contains the following components:

Together these fields form the agent memory that must be persisted, queried, and occasionally pruned.

---

2. File‑format showdown: JSON/NDJSON vs. Protobuf vs. SQLite/DuckDB

One‑line note: Side‑by‑side table summarizing serialization speed, size, schema evolution, and random‑access capability for each format.

How to verify the numbers for your own workload

1. Generate a representative sample of 100 k records.
2. Serialize each format using the same hardware and Python interpreter.
3. Measure wall‑clock time with time.perf_counter().
4. Compare file sizes with du -b.

For random‑access performance, benchmark a simple query such as “retrieve the 10 most recent entries with tag = ‘task’” on the same dataset.

---

3. Designing a forward‑compatible memory schema

One‑line note: Provide a concrete schema (a .proto file and a matching JSON Schema) that captures timestamps, source IDs, embeddings, relevance scores, and version tags.

The schema below treats timestamp_ms and embedding as required core fields, matching the JSON Schema definition. All other fields are optional, which allows newer agents to add data without breaking older readers.

Protocol Buffers definition (agentmemory.proto)

syntax = "proto3";

package agentmemory;

// Optional header that can be written once at the start of the file.
message MemoryFileHeader {
  string format_version = 1;   // e.g. "v1.0"
  uint64 created_unix_ms = 2;  // epoch milliseconds
}

// A single memory entry.
message MemoryEntry {
  // Unique identifier, preferably a UUID string.
  string id = 1;

  // When the entry was created (epoch milliseconds). Required.
  uint64 timestamp_ms = 2;

  // Raw textual payload. Required.
  string text = 3;

  // Optional list of tags for filtering.
  repeated string tags = 4;

  // Source identifier (e.g., tool name, API endpoint). Optional.
  string source = 5;

  // Embedding vector stored as raw bytes (float32 little‑endian). Required.
  bytes embedding = 6;

  // Retrieval relevance score from the most recent search. Optional.
  float relevance = 7;

  // Arbitrary user‑defined key/value pairs for future‑proofing. Optional.
  map<string, string> extra = 8;
}

JSON Schema equivalent (agent_memory_schema.json)

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "AgentMemoryEntry",
  "type": "object",
  "required": ["id", "timestamp_ms", "text", "embedding"],
  "properties": {
    "id": { "type": "string", "format": "uuid" },
    "timestamp_ms": { "type": "integer", "minimum": 0 },
    "text": { "type": "string" },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    },
    "source": { "type": "string" },
    "embedding": {
      "type": "string",
      "contentEncoding": "base64",
      "description": "Float32 little‑endian bytes, base64‑encoded."
    },
    "relevance": { "type": "number" },
    "extra": {
      "type": "object",
      "additionalProperties": { "type": "string" }
    }
  }
}

Why this design works for versioning

• Core identifiers (id, timestamp_ms, text, embedding) are required, so every entry contains the minimal information needed for retrieval and similarity search.
• All other fields are optional; adding a new field later does not break older parsers because they simply ignore unknown keys.
• The extra map offers a catch‑all location for experimental attributes without changing the schema.
• A single MemoryFileHeader written at the top of a Protobuf stream lets downstream tools reject files that use an incompatible format_version.

---

4. Hands‑on serialization & deserialization snippets

One‑line note: Minimal Python code showing how to write, read, append, and query the memory in the chosen format, including handling large vectors (base64 or binary blobs).

4.1 Using Protocol Buffers with an append‑only stream

import uuid
import time
import struct
from pathlib import Path
import agentmemory_pb2 as am   # Generated from agentmemory.proto
from google.protobuf.internal import encoder  # For varint encoding

MEMORY_PATH = Path("agent_memory.pb")

def _write_message(fp, msg: am.MemoryEntry) -> None:
    """Write a length‑delimited Protobuf message using varint encoding."""
    data = msg.SerializeToString()
    # Encode length as a varint (the same format used by protobuf's CodedOutputStream)
    size_prefix = encoder._VarintBytes(len(data))
    fp.write(size_prefix)
    fp.write(data)

def append_entry(
    text: str,
    embedding: bytes,
    tags: list[str] | None = None,
    source: str | None = None,
    relevance: float | None = None,
    extra: dict[str, str] | None = None,
) -> None:
    entry = am.MemoryEntry(
        id=str(uuid.uuid4()),
        timestamp_ms=int(time.time() * 1000),
        text=text,
        tags=tags or [],
        source=source or "",
        embedding=embedding,
        relevance=relevance if relevance is not None else 0.0,
        extra=extra or {}
    )
    with MEMORY_PATH.open("ab") as f:
        _write_message(f, entry)

def iter_entries():
    """Yield MemoryEntry objects from the stream without loading the whole file."""
    with MEMORY_PATH.open("rb") as f:
        while True:
            # Read varint‑encoded length
            size = 0
            shift = 0
            while True:
                b = f.read(1)
                if not b:
                    return  # EOF
                byte = b[0]
                size |= (byte &