We’ve Solved Online Learning

1. Supervised learning: “Please remember this”

In the beginning, I was the learning critic. We would work through a problem, find the right service or finally get a procedure working, and I would see something worth keeping. Then I would say:

“Please remember how to do this.”

Buffaly would turn the work we had just done into functions and entities. A successful sequence of steps could become an action, with inputs for the parts that would change next time. Discovering which repository produced an installation could become a stored relationship. We already had a system that could learn either one. What it needed from me was the decision that this particular result mattered.

Phase 1 / human-selected action learningEnlarge ↗

One learned procedure / different inputsEnlarge ↗

The value was immediate. We had already paid for the exploration: trying an approach, seeing why it failed and finding the version that worked. Why pay for that again? Retaining the procedure let the next call start there. Buffaly could supply new inputs instead of generating another implementation and potentially making the same mistake.

We learned thousands of things this way, and it worked. I could point to a useful result, ask Buffaly to remember it, and use the resulting entity or action on the next job. But the decision about what mattered was still mine. That was effective learning, not unsupervised learning.

The problem was that I had to be there to notice it. I was trying to get work done, not supervise every learning opportunity. The working agent had the same competing demands. We could solve something difficult, move on, and leave behind none of the structure that would make the next attempt easier.

I wanted Buffaly to supply that judgment itself. Our next attempt was the obvious one: give a critic the completed conversation and ask it to recover what we should have learned.

2. The offline critic: learning from the whole conversation

The first offline approach was brute force. After the work was done, a large model had to read an entire conversation, work out what had actually happened, and decide which parts were important enough to keep. The destination was the permanent ontology.

The permanent ontology was Buffaly's shared knowledge and code: the entities, relationships and actions maintained in its projects and skills. Learning meant adding to that shared collection. A new definition would remain available to future work, not just the conversation that produced it.

That made every learning decision a decision for the whole system. The critic had to understand the conversation, distinguish successful conclusions from failed attempts, check what was already known, and predict what would be useful in future work. Learning a fact and deciding that everybody should know it were effectively the same assignment.

Phase 2 / whole-conversation criticEnlarge ↗

It worked, but it did not scale. Consider a conversation containing a sensible plan, three failed attempts and a correction much later. Which version should the critic keep? What had actually been established? Was the same thing already represented under another name? Would anybody need it again? The model needed enough context and reasoning capacity to answer all of those questions together. That made this an expensive way to learn a small thing.

The permanent destination made mistakes expensive too. Once promoted, a definition stayed in the shared ontology until somebody corrected or removed it. There was no automatic expiry just because it turned out to be unnecessary. If a session used “production” to mean one particular installation, retaining that shorthand as a shared answer could send the next agent to the wrong system.

And an ontology can be cluttered with facts that are perfectly true. An obscure, one-off detail can turn up in searches for years, making agents inspect and reject it again and again. The namespace is the set of named definitions an agent works with. Filling the shared namespace with everything that happened makes it harder to find what matters. Every permanent addition carries an ongoing cost, even when it is correct.

The batch process also lacked the feedback loop we wanted. A critic swept through past work and proposed what the future might need. The working agent was no longer right there using the new definition and showing whether it helped. Every new conversation meant another expensive review and more proposed additions to fit into shared knowledge.

We had automated the review, but asked it to do too much at once. The next design broke that review into passes.

3. The multi-pass critic: better review, still supervised

The multi-pass critic separated noticing something worth learning from adding it to the ontology. The first critic kept a proposal ledger: a saved list of suggestions, the evidence behind them and the work already reviewed. A second critic worked out how to turn selected suggestions into actual definitions. We called it the materializer because its job was to turn a proposed idea into knowledge or code the system could use.

That separation mattered. A reviewer could inspect part of the conversation, record a useful suggestion and continue later. A checkpoint saved its place. It no longer had to rediscover every suggestion or produce the final ontology changes in one giant pass. The implementation kept separate files for the review, proposals, preview and proposed definitions rather than making each pass rewrite one growing document.

The proposal carried an argument for learning something. The reviewer had to tie it to what the user asked for, establish what that language referred to, and show how retaining it would reduce future confusion or wasted work. “This appeared in a conversation” was not enough. The prompt asked for a small set of changes the user could accept or deny.

The second critic could then concentrate on the proposed changes. It read the suggestions, searched the existing ontology, inspected the real definitions and prepared exact edits. Its instructions told it not to repeat the first review or reread the original conversation unless a proposal was ambiguous. The ledger was the handoff between two different jobs.

In our workflow, we put those proposals in front of a human. I could approve, reject or defer an item before it entered the permanent ontology. The second critic prepared the concrete change; a proposal was not permission to apply it. Its output ended with the explicit statement, “No ontology changes have been applied.”

Phase 3 / proposals and approved writesEnlarge ↗

This was a real improvement. The ledger let us review a little at a time, preserve the evidence and reject a bad idea before it entered shared knowledge. But the final judgment had not changed. We still needed a large model to decide what deserved a place in the permanent ontology, and in practice a human to approve it. We had made the batch process more manageable without removing its most expensive question.

In the first phase I had to say “remember this” while we were working. Now I could inspect a list afterward. That was more systematic, but I was still supervising what became shared knowledge. Until approved and turned into definitions, the suggestions remained just that—a list of proposals.

We had improved how we reviewed the work, but learning still required a decision about the permanent ontology. The next design needed to change that requirement.

4. Unsupervised learning: the online memory and action critics

The working agent already produces the evidence for learning: what was asked, what it tried, what worked and what it corrected. Our online critics review that work alongside the agent and decide which discoveries should become usable knowledge or code. They can make additions while the job continues, and later use supplies the next round of evidence.

We made that possible by separating five problems we had been asking one critic to solve at once: what kind of thing to learn, where to put it, how much judgment the decision requires, how other agents can find it, and how that knowledge should follow the organization of the work.

These are five complementary insights, not five more historical phases. Together, they let us learn continuously without asking every new definition to justify its place in the permanent ontology.

Insight 1: Separate entity learning from action learning

Learning which database a project uses is different from learning a reusable procedure for querying it. The first problem is about identity, properties and relationships. The second is about behavior: which steps belong together, what should become an input, what the function returns, and whether the procedure already exists.

So we separated the online memory critic, which learns about things and their relationships, from the online action critic, which learns procedures and improves existing actions. Functions are still entities in the same executable graph. We gave the critics different jobs; we did not split knowledge and code into unrelated systems.

Phase 4 · Insight 1 / separate the learning judgmentsEnlarge ↗

The System 2 companion checks whether an agent has done what the user asked. These companions look at what the work should teach Buffaly. Giving each one its own instructions, searches and model lets us improve its learning job without distracting the working agent from the original request.

The online memory critic searches what is already known about the things involved. It checks their structure—the properties they inherit, the values already filled in and the fields still empty—before deciding whether to create something new or update an existing entity. A useful discovery may require a new property or relationship. The critic can add it; it is not limited to filling out a fixed memory form.

The online action critic looks for a reusable procedure or a demonstrated improvement to an action learned in its session. Before writing, it searches using more than one phrasing and checks likely matches. We learned to make this distinction explicit: using a long script does not prove a new action is needed, and failing to find a tool does not mean it is missing. When a procedure is worth keeping, the critic can write code, retain instructions that call a model, or combine the two. A substantial PowerShell workflow can become a reusable action too, with inputs for the parts that change.

Splitting the assignments made each critic more focused. It did not yet solve the problem of where its output should live.

Insight 2: Create a local ontology that extends the permanent ontology

Until this final phase, learning had meant adding to the permanent ontology. To change that, we invented session-local ontologies: knowledge and actions learned within a particular working session. A critic could now keep something useful without first adding it to everybody else's shared knowledge.

A session-local ontology extends the permanent ontology. The agent can use all the shared permanent knowledge plus the entities, relationships and actions learned in its session. The local additions do not replace the original collection. A new entity can build on an existing type, and a new function can call an existing action, because both are in the same working graph.

Phase 4 · Insight 2 / extend the permanent ontology locallyEnlarge ↗

We store only the session's own additions, which we call its fragment. What the agent can use is larger: the permanent knowledge plus that fragment. Two independent sessions share the same starting knowledge, then each adds what it learns. The notation below makes that distinction precise; the fifth insight adds parent and child sessions.

These are real entities and actions the agent can use, not proposals waiting for a human to approve their existence. They can be saved, called and corrected as the work develops. Actions are stored in the session's SessionActions.pts file; entities are stored in SessionMemory.pts. The definitions become available to the working agent without changing the shared permanent ontology.

This changes the question from “Should every future agent know this?” to “Will retaining this help the work here?” That smaller question leads to the next insight.

Insight 3: Use a smaller model to extend the local ontology

Learning happens alongside the main job. It cannot cost another large-model conversation every time the working agent discovers something useful. We made the assignment narrow enough for smaller, less expensive models: learn one kind of thing, review a limited amount of completed work, and keep the result in that session.

Phase 4 · Insight 3 / smaller models for a smaller decisionEnlarge ↗

The two online critics independently review completed interactions: the request, the result and a limited view of the tools used to get there. The working agent can continue while they review. Neither critic has to reconstruct an entire conversation, invent a procedure for every possible future use and install it permanently, all in one step.

A smaller model still uses Buffaly's tools for writing and checking definitions. After saving an action, the critic reads it back, checks that search can find it, and runs it when safe. A saved review record keeps the original interaction, the reason for the decision and the checks performed. It also records when no change is needed—NoChange—so reviewing the same interaction again need not create another version.

Choosing the model for each learning job

Learning runs alongside the main work, so the critic needs to earn its cost. We gave each model the same set of completed interactions and examined the actions it learned. The comparison below asks one question: what share of those actions was worth keeping or improving?

GLM 5.2 came close to GPT-5.6 Sol: about seven in ten learned actions were useful, compared with about eight in ten. Some other configurations did much worse. Luna's medium setting produced useful actions about half the time; Terra's medium setting produced none worth keeping. The expensive model was no longer a prerequisite for useful learning.

How much of the learned code was useful?Enlarge ↗

The percentages describe the quality of what each model wrote. GPT-5.5's perfect score came from a single action across the entire test; the other models found useful procedures it missed. We needed a critic that noticed worthwhile opportunities as well as making good choices about what to keep.

GLM 5.2 was the practical choice for routine action learning. In a larger follow-up spanning nearly 600 interactions, more than four in five of its learned actions were worth keeping or improving. That fits the way Buffaly learns: useful local code can be exercised, corrected and generalized before it becomes shared infrastructure.

We tested memory learning separately. Recognizing useful objects and relationships calls for a different judgment from identifying reusable code. A model that extracted rich memory could still miss the procedures worth learning. Separating the critics let us choose a model for each job instead of paying for the same large model to do both.

Local learning could now be practical and frequent. But if it stayed invisible outside the source session, other agents would keep discovering the same things.

Insight 4: Allow non-local discovery, borrowing and promotion

The next insight was finding useful knowledge without importing everybody else's ontology. An agent needs to discover what other sessions have learned without loading all of it into its own working environment.

Search can find a relevant action or entity and identify the session that learned it. Finding it does not add it to the searching agent's ontology. The agent can inspect the match, then borrow the particular definition it needs. That makes another session's learning useful without bringing along everything else that session knows.

Phase 4 · Insight 4 / non-local discovery and selective reuseEnlarge ↗

An agent can borrow the definition it needs, use it and retain a local copy. Missing dependencies are reported explicitly, and invoking an action remains a separate operation. What that new use teaches us is the basis for deciding whether the definition should travel further.

Each session keeps its own fragment of the executable graph: the entities, relationships and actions it learned. Their origin remains clear. “The backup” can mean the backup we just created and verified, without becoming every future agent's default backup. Session-local does not mean temporary or disposable; the definitions can be saved and found from other sessions.

This also gives us a way to handle contradictions. A fact can be true for one product, installation or stage of a job and false for another. Keeping those identities separate prevents us from forcing different situations into one shared answer too early.

Find it elsewhere before adding it here

Search reaches beyond the definitions already available in the current session. It distinguishes shared and local matches from things learned by other sessions. Each outside match identifies the exact definition and where it came from; search does not load it. Seeing that another session has a useful action is different from adding that action to your own ontology.

That distinction keeps search from cluttering the agent's working knowledge. An agent can find another session's log analyzer without loading that session's traces, repositories, experimental actions or unrelated assumptions. In a search result, “unavailable” means the definition is not loaded here yet. It does not mean the capability does not exist.

The agent can inspect the match and borrow one selected action or entity. It gets the benefit of the other session’s work without importing the whole fragment or declaring its contents permanent.

Cross-session reuse and promotionEnlarge ↗

Borrow loads the selected definition unchanged into the requesting session. It makes an action available without running it. If the action depends on another definition that is missing, Buffaly reports that missing dependency rather than silently importing more knowledge. Keep saves the requesting session's own copy; later changes to the original do not automatically change that copy.

Knowledge improves as it travels

A session is a good place to learn something before we know how widely it matters. The exact path to one investigation's log can stay there. So can a useful description of a temporary deployment or a procedure tailored to the problem in front of us. Most of that detail never needs to become permanent knowledge. It remains in the leaves of the session tree, where it has meaning.

The useful parts can move further. Another session discovers a definition, borrows it and puts it to work in a different situation. That gives Buffaly an opportunity to improve the knowledge. A thin record can gain a useful property or a missing relationship. A function can acquire a parameter that replaces a hard-coded assumption. Two specific procedures can reveal a shared operation worth expressing once.

This also gives us a way to discover bugs. A first implementation may just about solve the original problem and then fail on a new input. The next use reveals the assumption that was wrong. Buffaly can fix it, retain the evidence and use the improved version. Repeated use turns a narrowly successful procedure into one that handles a wider range of work.

We found a small, concrete example in a learned database-backup action. It created the backup but wrote the timestamp variable literally into the filename. The critic corrected the code, and later runs produced the intended timestamped names. The code improved through use. The same feedback is valuable when deciding whether an action deserves a wider role.

Promotion lets us make that decision incrementally. A definition can become useful to related work at the parent level, then to work elsewhere, before we decide it belongs in the permanent ontology. At each step we can refine its meaning, generalize its inputs and test the behavior that other agents will rely on. The one-off details stay local; the definitions that earn broader use become better candidates for shared knowledge.

We can set the standard for that final step: useful across the intended range of work, clear about its inputs and effects, checked against existing capabilities, and tested in the situations it claims to support. The standard can be stricter for an action that changes a database than for one that reads a log. Promotion is where we decide that a definition is mature enough for the wider role.

We have already moved locally learned operations into permanent action families. Workbench's run-status lookup is one example: an operation learned during a session became a shared action with a run identifier as its input. What matters is the progression from a particular need to a capability other work can rely on.

This is how fragmented learning accumulates. Every discovery has somewhere to live, and useful knowledge has a route to improve. We can learn at the scale of individual jobs while allowing the permanent ontology to grow around definitions whose value and behavior are increasingly well understood.

Comparing and generalizing action definitions

This is where treating functions as entities becomes especially powerful. Buffaly can examine two actions, find the structure they share and identify the parts that differ. It can keep the shared procedure and turn the differences into inputs. The formal name for finding the closest common structure is least general generalization; making the changing parts into inputs is parameterization. In this simple example, the backup relationship stays the same while the repository changes:

Example A:  Verify(BackupOf(RepositoryA))
Example B:  Verify(BackupOf(RepositoryB))
Shared:     Verify(BackupOf(repository))
Variable:   repository

Buffaly has implemented these comparison and parameterization mechanisms for its definitions. The backup example shows why that matters: a procedure tied to one repository can become an operation that accepts a repository as its input. Further uses tell us which assumptions belong in the shared procedure and which should vary. Because the functions themselves are represented in the ontology, Buffaly can work on their structure as it improves them. The example above illustrates the mechanism; it is not a recorded automatic promotion.

The sequence matters: learn something concrete, use it, borrow it where it helps, and generalize when the examples justify it. We stopped requiring the routine learner to solve every one of those problems at once. It also gives promotion a stronger basis than the first impression of a retrospective reviewer.

Search connects knowledge from otherwise separate sessions. The final insight organizes sessions that already work together.

Insight 5: Let the ontology hierarchy mirror the session hierarchy

Working sessions already form a hierarchy. An agent can assign part of a job to a child session, which can assign a smaller part to another child. Their ontologies can follow that same structure.

It becomes a hierarchy of ontologies. A child session can use its parent's entire ontology, including the knowledge and actions the parent has learned. The child adds its own discoveries to that starting knowledge. Its children can do the same.

A child starts with everything its parent can use, then adds what it learns. A grandchild gets both sets of discoveries and adds its own. Sibling sessions share their parent's starting knowledge, but one sibling's private discoveries do not automatically become part of the other's ontology. The diagram shows these layers as P, A, B, C and D.

Phase 4 · Insight 5 / mirror the session hierarchyEnlarge ↗

The child does not need to relearn what its parent already knows, and its specialist discoveries need not immediately become everybody's knowledge. When a definition deserves a wider home, borrowing and promotion can give it one: another session, a parent session, or the permanent ontology.

How the five insights work together

Consider what happens during a startup investigation. The working agent identifies the installation, finds its logs and works out how to measure the time spent in database procedures. The memory critic can preserve the installation and the references needed to investigate it again. The action critic can retain the procedure as a function that accepts a log path. Each critic has a concrete result to learn from and a different decision to make.

Both additions enter that investigation's ontology. The agent can use them while the work continues. At this stage, the critics need to recognize that the knowledge is useful to this job; they have no need to predict every installation, log format or future investigation it might eventually serve. That smaller decision is what makes inexpensive, focused learning practical.

A child working on the same investigation starts with its parent's knowledge. A separate session investigating another problem can search for the parser and borrow it without importing the original investigation's logs, status updates and experimental dead ends. The hierarchy gives related work a common starting point; search lets useful discoveries travel beyond it.

The new use gives us another test of the procedure. It may work immediately. It may reveal that a path should be an input, that an assumption only held for the first installation, or that a parsing case was missed. Buffaly can retain the correction, make the action more general and test it again. Useful knowledge gains a wider role as the evidence for it improves. The run-specific details remain attached to the run.

That is how the five decisions fit together. Separate critics make focused learning choices; local ontologies let the results become useful immediately; smaller models keep routine learning economical; inheritance and discovery carry the useful definitions to new work. Promotion can then draw on actual use, corrections and broader applicability when choosing what becomes permanent.

We moved the hardest judgment to the point where we have evidence for it. The first critic no longer has to decide whether a discovery deserves to become permanent before anyone can benefit from learning it.

What earns a place in a session ontology?

Keeping knowledge within a session is not a reason to keep everything. We still need a practical definition of usefulness. Ours is simple: give the model less text to process, avoid repeated searches and tool calls, and reduce errors. Learning should make later work easier. It might settle which repository a name refers to, save a procedure that took several attempts to get right, or keep a correction attached to the function that needed it.

That objective went into the prompts. The offline critic was told to make “objects easier to locate, actions easier to replicate, and previous mistakes easier to avoid.” A proposal had to reflect what the user meant, have evidence behind it and save future work. We kept that objective, then narrowed the decisions the online critics would have to make.

For memory, the test is whether the agent can do something with the knowledge. “There is a backup” tells it very little. A backup record with a name, size, file hash—a fingerprint of its contents—and a link to its repository gives the agent something it can locate and verify. This graph shows a real learned example, with client names replaced:

Learned backup entity and verification actionEnlarge ↗

The retained backup definition includes these assignments:

// Excerpt from WebsiteProject_CloneBackup (client-neutral name)
SemanticEntityBase BackedUpRepository = new SemanticEntityBase();

ArchiveSizeBytes = "5916789";
BackupGitTag = "clone-backup-9d38bac";
BackedUpRepository = WebsiteProject_WebRepository;

Now the next request can find the backup, and code can follow BackedUpRepository to the repository it protects. That saves an investigation. The memory critic's instructions make this concrete: a learned object must contain something the agent can use—a path, an identifier, or a link to another entity that has one. These assignments come from the saved definition. We also checked whether the accumulated information survived being loaded again, an engineering lesson we return to below.

When should the action critic learn a procedure? It should solve a recognizable problem likely to come up again, make the work meaningfully easier or more reliable, and be something Buffaly can implement with its available services. An existing action should be reused unless the evidence shows it needs an improvement.

For an action, the investigation needs to establish what the procedure does and what inputs it needs. The log analyzer shows the payoff: the saved procedure runs without calling a model. A model may choose the action, supply the log path and interpret the result, while ordinary code does the parsing. If a procedure still needs judgment, its action can include a step that asks a model. We wanted models to work on decisions still open, rather than reproduce mechanics we had already worked out.

Exploration cost:  discover the procedure + try it + inspect the outcome
Later use:        select the operation + supply new inputs + execute

For a deterministic caller, even selection can be a direct function call.

We judge usefulness when selecting a candidate, when using it later, and when reviewing what to keep or improve. No single score replaces those checks. Writing a definition costs something. Maintaining it costs something. Searching through unnecessary definitions costs something. Sometimes the best learning decision is that the system already has what it needs. The critics can record “no change needed” and move on. In roughly four out of five reviews, the action critic did exactly that. We wanted a useful addition or correction, not a new definition after every interaction.

This is the assignment the online critics repeat: look at the work that just happened, identify something useful, check whether it already exists, and retain the justified addition or correction. The results let us test whether those decisions actually helped.