Karpathy's LLM Knowledge Base Is Right. It Should Be a Database.

Karpathy recently shared how he is building an LLM-powered knowledge base.

The post went everywhere, for good reason. The core idea is huge: more of our token throughput is moving away from only manipulating code and toward manipulating knowledge.

That is exactly the right direction.

I just think the backend should be a database.

The important part

The thing Karpathy is pointing at is not just "use AI with notes."

It is a bigger shift in what we ask models to do.

Coding agents made it obvious that LLMs can manipulate a structured corpus: a codebase. They read files, infer architecture, make edits, run tests, and keep working inside the same environment.

The next step is using the same pattern for knowledge.

Your research, ideas, papers, transcripts, images, questions, and decisions should become a corpus the model can operate on. Not a random chat history. Not an endless pile of disconnected notes. A working knowledge base.

That is the right frame.

Externalizing and compounding your knowledge into useful context is probably the highest leverage thing you can do with these tools.

Karpathy's architecture

The system he described has three layers:

• raw sources: articles, papers, images, transcripts
• a wiki: LLM-generated markdown pages, summaries, entity pages
• a schema: instructions telling the model how to work with the knowledge base

Then there are workflows around ingestion, querying, and linting.

That is already quite close to what an agent-native knowledge system should look like.

Raw sources should be preserved. Generated synthesis should be separate. The agent needs instructions for how to operate. The system should be linted for contradictions and stale assumptions.

I agree with all of that.

But there is a line in the proposal that gives away the next step: once the wiki grows, you want proper search.

Exactly.

And once you want proper search, metadata, relationships, logs, provenance, entity pages, and update rules, you are building a database whether you call it that or not.

Just start with the database

Markdown is appealing because it is simple, portable, and model-friendly.

But the moment you ask multiple agents to read and write to the same knowledge base, the limitations show up quickly.

Where does metadata live? How do you avoid duplicate pages? How do you represent a relationship between a person, a paper, a podcast, and an insight? How do you search specific transcript chunks without loading entire files? How do you preserve source while also allowing synthesis to evolve?

You can solve all of this with files.

But you will keep bolting on database-shaped features.

RA-H takes the other route: use a local SQLite database as the backbone from the start.

Everything still remains local and inspectable. You are not giving up ownership. You are just giving the agent a cleaner substrate.

Nodes, edges, source, and search

The RA-H shape is deliberately simple.

Everything important becomes a node: a source, idea, person, project, article, podcast, decision, or conversation.

Each node has a title, explicit description, and source. Long source material can be chunked and embedded. The description tells the model what the thing is in plain language.

Connections live as edges. An edge does not merely say "these two things link." It explains why they connect.

That distinction matters.

If I ingest Karpathy's tweet, a No Priors podcast, and my own note about token throughput, I want the agent to understand the relationships between them. I want it to search the transcript for a specific claim, explain it simply, create a new insight node, and link that insight back to the original source.

That is exactly the kind of workflow a database handles cleanly.

Full-text search finds exact language. Vector search finds semantic matches. Ordinary SQL gives the agent a precise way to inspect and update the graph.

This is not anti-markdown

Markdown is still useful.

Instructions can be markdown. Skills can be markdown. Longform synthesis can be markdown. Human-readable exports can be markdown.

The issue is whether markdown files should be the primary data model.

For small personal systems, maybe. For agent-native systems with retrieval, writeback, provenance, and multiple tools interacting with the same corpus, I think SQLite is the boring correct answer.

The structure helps both sides.

It helps the human because it forces clearer thinking: what is this thing, why does it matter, and how does it connect?

It helps the agent because it can query the data instead of guessing how a folder of files should be interpreted.

The real skill is filling the context window

Karpathy has talked about token throughput. I think the adjacent skill is context throughput.

How efficiently can you get the right context into the model at the right time?

That is the art.

Not dumping everything into a giant prompt. Not trusting a provider's hidden memory. Not hoping backlinks will magically produce understanding.

The work is building a corpus that can be searched, retrieved, inspected, and updated.

That is why I like the database approach. It makes the context layer explicit.

Where this is heading

More agents will manipulate knowledge, not just code.

They will ingest source material, create summaries, identify entities, find contradictions, update project context, and write durable notes back into the user's corpus.

If that is the future, the substrate matters.

Karpathy's architecture is right about the direction: LLMs should work over a personal knowledge base.

My claim is narrower: the backbone should be a local database with explicit nodes, edges, source, and hybrid retrieval.

That is what RA-H is trying to make practical.

You can use the Mac app, or clone the open-source version. For more on the broader storage argument, read Don't Build a Second Brain. Build a Database..