Working context
What the agent needs for the task and conversation happening now.
Building a Second Brain · Agent memory · Personal infrastructure
A second brain is not another notes app. It is the custom memory layer that helps every future agent understand you, your work, and what came before.
This essay describes the personal-brain architecture I use today and the lessons I have drawn from building it. The operational ontology discussed near the end is future research, clearly separated from what is working now.
I think your second brain may be the last project you build alone.
Not because you stop building. The opposite happens. Once your agents can remember your decisions, preferences, projects, corrections, and ways of working, every new project starts with more context than the one before it. You no longer begin by explaining yourself from zero.
The second brain becomes the common memory behind the tools. One agent can help with research, another can write code, and another can review the result. They may have different interfaces and different strengths, but they do not each have to meet you as a stranger.
It is also the one project you never really finish. Every decision, failure, new relationship, and completed piece of work can improve the context available to the next agent. The system grows with its owner.
I work across several AI agents and coding harnesses. Each one has a different idea of memory. One reads repository instructions. Another can search previous conversations. Another builds a profile from past sessions. Some remember automatically; some only remember when asked.
These features are useful, but they fragment the person using them. A correction made in one harness may not reach another. A project decision can be buried in a conversation that the next agent cannot see. Switching tools means rebuilding context, even when the underlying work has not changed.
That is why I do not want any one harness to own my durable memory. The harness should be able to use the memory, but the memory should survive the harness.
I already accept this separation for passwords. I use Bitwarden because I do not want my credentials to belong to one browser. A browser can fill a password, but the vault remains independent. If I change browsers or devices, the durable record comes with me.
Agent memory should work in a similar way. A harness can keep temporary context for the task in front of it. It can remember its own interface preferences or recent sessions. But the durable record of who I am, what I have decided, and what I have learned should not disappear when I change tools.
What the agent needs for the task and conversation happening now.
Useful continuity that belongs to a particular agent or interface.
Durable knowledge that should remain available across authorised harnesses.
The point is not to eliminate harness memory. It is to decide which memories should be portable, correctable, attributable, and governed outside the tool that happened to create them.
A generic notes product can give everyone the same folders, search box, and embedding model. It cannot arrive knowing which parts of your life belong together—or which parts must remain apart.
My second brain has to reflect what matters to me: the projects I am building, the people connected to them, the sources I trust, the way I prefer to write, and the decisions that are still active. It also has to preserve corrections. Remembering the wrong thing confidently is worse than admitting that something is unknown.
The structure is personal because relevance is personal. Two people can read the same article and need different things from it. One wants a quotation for a paper. Another wants the idea connected to a product decision from six months ago. A useful brain has to know which relationship matters to its owner.
Independence from the harness does not mean putting everything into one vault. My personal memory and my company knowledge have different owners, subjects, permissions, and costs of being wrong. I separate them even when the same human is operating both systems.
That boundary matters more as agents become capable of writing. An agent reading a personal reflection should not silently turn it into company policy. An agent working inside company systems should not copy operational knowledge into a personal store. Memory needs routing, provenance, and access control—not merely better retrieval.
I explored that architecture in Two Brains, One Operator. The important idea here is simpler: memory should remain independent from the harness while staying accountable to its owner.
The personal system I use today exposes a small, consistent memory contract to
different harnesses. At the user-facing level, the two important operations are
remember and recall. A request is routed to the right
brain, and the selected brain handles its own search, provenance, and write rules.
This does not make every agent identical. Nor should it. Each harness still has its own context window, tools, instructions, and way of approaching a task. The shared brain supplies continuity without erasing those differences.
The result is practical. A new session can retrieve the writing voice I have already defined. It can find the history of this website, the research threads behind an article, and the publication boundaries I have set. I can correct the durable record once instead of hoping every harness learns the correction separately.
It still has flaws. Retrieval can surface irrelevant material. Old notes need maintenance. A memory can be well sourced and still be stale. Building the brain does not remove judgement; it gives judgement a durable place to accumulate.
The compounding effect is the reason I think of the second brain as infrastructure. A project should leave behind more than its final output. It should leave behind the decisions, sources, reusable patterns, failed approaches, and open questions that will make the next project easier.
The agent can then do more than retrieve an old file. It can recognise that a new question continues an existing line of research. It can connect a design choice to the reason it was made. It can tell me when new evidence conflicts with an old conclusion instead of silently stacking both versions together.
This is the shift: the second brain stops being a place where work is stored and becomes part of how new work begins.
The current system is strongest at durable knowledge: what I know, why I believe it, where it came from, and which brain owns it. That is not the same as modelling the live state of an organisation or giving an agent a safe way to change it.
My next research question is how this memory layer connects to an operational one: an ontology for the things that exist now; functions for reusable logic; documents for evidence; actions for governed change; automations for responding to events; and an AI layer that can discover and combine those building blocks.
I have prototypes and research material for parts of this model. I have not yet established the complete architecture as a finished public system. The next essays will separate what I have built and verified from what remains an experiment.
Memory gives an agent a past. The next challenge is helping it understand the present—and participate safely in what happens next.