LLM Memory Reimagined: Datalog Engine Boosts Agent Recall

A New Approach to LLM Memory

For months, security researcher Jordy Zomer watched LLM agents struggle with a fundamental problem during long vulnerability investigations: they kept losing track of established facts. The model would forget that an assumption was disproven, or confidently reason from outdated observations. Standard memory systems—storing conversations and retrieving relevant chunks—weren't enough. Zomer wanted the agent to maintain what it currently knows, not just recall what was said.

The solution, which he built almost by accident, is Lemmalog, an open-source Datalog engine that treats an LLM's knowledge as analysis state. The idea emerged from a realization: vulnerability research follows the same pattern as program analysis. You have facts, rules, and derived conclusions. When a fact changes, you update only the affected results—you don't rerun the entire analysis from scratch.

Why Datalog? The Power of Declarative Logic

Datalog is a declarative logic programming language. Instead of writing step-by-step instructions, you describe facts and rules, and the engine derives new facts. For example, if you know controls(attacker, object_a), points_to(object_a, object_b), and kernel_object(object_b), a rule can derive controls_kernel_object(attacker).

The magic happens when facts change. If points_to(object_a, object_b) turns out to be false, Lemmalog knows exactly which derived facts depended on it and can invalidate them automatically. No need to feed the entire conversation back to the LLM and hope it notices the contradiction. This is a fundamental shift from retrieval-based memory to incremental maintenance of knowledge.

How Lemmalog Works: LLM as Front-End, Datalog as Engine

Zomer splits the problem into two parts. The LLM handles the fuzzy, messy input: natural language notes, debugger output, source code. It extracts structured facts like freed(object_a) or reused_as(object_a, write_target). Lemmalog then handles the deterministic part: applying rules, tracking dependencies, and maintaining the current state.

This architecture offers a critical feature: provenance. You can ask why a conclusion is true, and Lemmalog shows the chain of facts and rules that support it. If an observation is later disproven, the system can trace exactly which conclusions are affected. This addresses a common failure mode where LLMs confidently assert things that were never established.

Handling Retractions and Temporal Facts

One of the trickiest problems was removing facts. If a conclusion has multiple supporting derivations, removing one fact shouldn't invalidate it entirely. Lemmalog tracks support counts and only removes a conclusion when all its derivations are gone. This mirrors real investigations where an exploit might be viable through multiple independent paths.

The engine also handles temporal validity. Facts can have validity intervals, so the system knows that viable(primitive_a) was true from 10:14 to 12:37, but not_viable(primitive_a) after that. This allows answering both "Is this currently viable?" and "Why did we think it was viable earlier?" without storing contradictory facts.

continue reading below...

Benchmark Results: Outperforming Full Context

Zomer tested Lemmalog on two benchmarks: LongMemEval and LoCoMo. On LongMemEval (102 questions), Lemmalog scored 0.463 F1, more than double the 0.197 F1 of GPT-4.1 with full conversation context. Crucially, it used only ~2,700 tokens per question versus ~104,000 for full context—a 38x reduction.

On LoCoMo (1,986 questions), Lemmalog scored 0.533 F1, coming third among dedicated memory systems behind PropMem (0.605) and OpenClaw (0.557), but ahead of full context (0.542). The most impressive results were on knowledge updates (0.579 vs 0.528 for PropMem) and adversarial questions (0.707 vs 0.509 for full context), where the structured approach excels at rejecting false premises.

The Importance of the Front-End

The biggest gains came from improving extraction and retrieval, not the Datalog engine itself. Entity resolution was critical—connecting "my car" to "Honda Civic" across sessions. Hybrid retrieval combining BM25, embeddings, and graph boosts solved the "kitchen gadget" vs "Instant Pot" problem. Zomer also discovered that a plural stemmer bug ("owns" not matching "own") was silently killing aggregation queries.

These fixes highlight a key insight: the hard part is building good IR from natural language, not computing the fixed point. The LLM remains essential for parsing messy reality, but once facts are structured, the database handles the logic.

Why This Matters for AI Agents

This research points to a hybrid future where symbolic systems and LLMs complement each other. Instead of relying on ever-larger context windows, agents can maintain a compact, structured state that grows with the investigation without exploding in size. The token efficiency is dramatic: after 500 turns, full context would require 1M tokens per query, while Lemmalog stays at ~2.5K.

For security researchers, this means fewer hallucinated dead ends and more reliable long-term reasoning. For the broader AI field, it suggests that memory isn't just about retrieval—it's about maintaining truth.

The Road Ahead

Lemmalog still has weaknesses, particularly in inferential reasoning (0.164 on LoCoMo vs 0.289 for PropMem). Flattening nuanced statements like "I prefer quiet restaurants except when traveling with friends" loses too much information. Zomer plans to address this by keeping conditional knowledge conditional and preserving original text for context.

The real test will be running a complex vulnerability investigation for hours and seeing if the agent stops resurrecting dead hypotheses. As Zomer puts it, "Perhaps we don't need a bigger context window every time an agent forgets something. Sometimes we can just maintain the state." The source code is available on GitHub.