AgentOps Is Context Orchestration

Update (July 2026): AgentOps now owns one bounded Plan, Implement, and fresh Validate experiment. The separate knowledge loop uses Beads plus an LLM wiki. Devlog #6 records the negative result that ended my automatic-compounding claim.

I spent months calling this "agent orchestration."

Anthropic shipped agent teams. My workspace. Ralph. Multi-agent this, orchestration that. I've written thousands of words about it. I have a whole article called Multi-Agent Orchestration. I was naming the wrong layer.

After months of shipping code with these systems, the thing that actually matters is what's in the context window when an agent starts working.

The high-leverage problem is context: loading the right knowledge into the right window at the right moment. The routing, the message passing, the communication protocol are plumbing. What agents know when they sit down to work decides whether they ship or spiral. Call it context orchestration.

The Context Window IS the Intelligence

Your agent has a context window. That's it. Every time it starts, it's a blank slate with a text box. It doesn't know what you worked on yesterday. It doesn't know what broke last time. It doesn't know which of the three middleware stacks in your codebase is the right one. It has whatever you put into the window, and nothing else.

The context window is the agent's working memory, its reasoning space, its entire world. Tool calls and API integrations are scaffolding. The intelligence lives in the context.

So when your agents fail, look at what was actually in the window:

• Bad answers? Wrong context loaded.
• Contradictions? Context wasn't shared between agents.
• Hallucinations? Context was too sparse.
• Drifting off-task? Context signal-to-noise ratio collapsed.

Every one of those is a context engineering problem. The most elegant message bus in the world still ships agents that fail if the window is wrong.

The right question is "what does this agent need to know right now?"

Why This Matters at Scale

People figure this out in every other domain but haven't for agents: you can't individually brief every worker.

If you have a hundred agents, or even five running in parallel, you cannot hand-craft the context for each one. You need a system that ensures each agent gets exactly what it needs, when it needs it, without you being in the loop. That's the only way any of this works.

Real organizations operate this way. You don't walk up to every person and explain the mission, the constraints, the history, the current state. You build systems: onboarding, documentation, institutional knowledge, review processes. When someone shows up to work, they already have what they need to make good decisions. The organization's intelligence isn't in any single person; it lives in the system that loads the right context into the right person at the right moment.

That's what context orchestration is. It makes the inputs, boundaries, and handoffs inspectable. It does not make the output trustworthy; checks and a fresh verdict still have to earn that trust.

That's what I've been working toward. Most of the ecosystem is still building the other thing.

The Three Gaps

The tooling right now is focused on two things: what goes into the context window before the agent starts (specs, plans, context engineering), and how to route work between agents (message buses, state machines). Both important.

Validation, learning, the refinement loop after the agent pushes code: thin tooling. You write the spec. You run the plan. The agent pushes code. Then what? You eyeball it. You run it again when something breaks. You fix the same bug from last week because the agent doesn't remember solving it. That loop, the actual work, you're still doing by hand.

It's like molding clay on a wheel. Shape it, check it, reshape it, check it again. That is the process. Right now, most of the tooling stops after the first shape.

I spent months testing every framework, every approach, every tool I could find. I kept running into the same three gaps, all context problems:

Validation. Judgment validation, not "does it compile" validation. Your spec said "add middleware." Your codebase has three middleware stacks. Nobody asked "what could go wrong?" before the agent picked one and wrote 400 lines against it. After it built the thing, nobody checked whether the edge cases were handled. The happy path looked beautiful. Everything else was broken. The validation context was never loaded.

Learning. You solved a tricky auth bug on Monday. Wednesday the agent makes the exact same mistake. Groundhog Day, except instead of learning piano and ice sculpting, it's rediscovering the same dead-end solutions you already tried last week. Most memory plugins give you recall. Almost none give you learning. The agent doesn't retain judgment across sessions; it just carries notes. The right knowledge never made it into the window.

Here's the failure mode that burns evenings even when you do persist: last Thursday the flywheel cheerfully injected a middleware pattern from three weeks ago that a Monday dependency upgrade had deprecated. Broke the build. Ninety minutes rolling it back instead of shipping anything else that evening. Decay modeling helps. It isn't fully solved. Persistence without curation can make you slower.

Closing the loop. What did this session teach us? What should we work on next? That extraction step is where people hope compounding happens. Easy to skip. I measured whether the corpus beats a curated baseline and the first A/B said no (Devlog #6). The habit still costs ~2 minutes. The moat is unproven.

These are context lifecycle problems: everything that happens between "agent wrote code" and "the work is proven, with an explicit decision about what is worth preserving."

How AgentOps Was Born

I didn't sit down and design this system. I fell into it.

What follows is the origin story of the first stack, not the contract of current AgentOps. That older stack chained research, planning, orchestration, validation, and post-session extraction. The current core is deliberately smaller: one Plan, one bounded Implement pass, one fresh Validate pass, then report and stop.

I was doing the full cycle by hand. Research the codebase, write a plan, run a pre-mortem on the plan, decompose it into waves, crank through the implementation, validate the output, extract learnings. Every time. By hand. I called it "the pottery barn" because I was literally sitting at the wheel shaping clay all day. It was working. The cycle produced better code than anything else I tried.

After a few weeks of the pottery barn, I started getting annoyed. Why am I manually loading context that the system already produced? Why am I running the same validation steps every time? Why am I hand-crafting context scoping that follows the same pattern every time? (This is the part where I spent an entire Saturday doing the exact same six steps I'd done on Friday and thought "I am the bot now.")

So I started building skills around each step. /research explored the codebase. /plan decomposed work. /pre-mortem challenged the plan. /crank orchestrated workers. /vibe checked the output. /post-mortem proposed lessons.

I wired those pieces together with hooks because I was tired of remembering the sequence. That experiment taught me which responsibilities belonged together and which did not. Automatic orchestration and knowledge extraction no longer belong to the binding AgentOps core. Swarms are optional; learning is later, reviewed work.

Somewhere in the middle of all that, I realized what I'd actually built was a context orchestrator. Every skill, hook, and knowledge link shapes what an agent sees at a particular step. I can reconstruct what each agent was supposed to see; I still have to verify what it did.

What It Actually Does

The current AgentOps core is smaller than the system this essay originally described. It is a runtime-neutral operating loop with portable skills and an optional Go CLI. One invocation handles one experiment:

/rpi "add rate limiting to the API"

intent
  → Plan
  → one bounded Implement pass
  → fresh Validate against the same acceptance and evidence
  → durable PASS, FAIL, or NOT_PROVEN verdict
  → report and stop

Plan fixes the intent. The preferred source is a bead with observable acceptance and repository language. If there is no tracker item, AgentOps can use the caller's issue or chat text and snapshot the exact bytes.

Implement builds one candidate. The write scope, changed paths, checks, and evidence come from the actual work. The model does not transcribe a flattering candidate packet after the fact.

Validate runs fresh. A distinct context checks the unchanged intent, candidate identity, changed scope, and evidence for each acceptance check. Missing freshness or incomplete evidence becomes NOT_PROVEN, not a soft pass.

Then it stops. AgentOps reports the verdict. The calling repository decides whether to revise, land, release, or abandon the work.

The Knowledge Flywheel

The knowledge layer sits beside the core loop.

Beads record the work, decisions, dependencies, and evidence. An LLM wiki preserves the subset of knowledge that survives review, with links back to its sources. A future bead can link a relevant page before implementation. A stale page becomes repair work.

The full mechanics are in the deep dive. The important boundary is simple: the wiki can inform a later plan, but it never decides whether the current candidate passed. A larger corpus does not make the system smarter by itself.

What It Deliberately Does Not Own

AgentOps does not own retries, budgets, queues, work assignment, git operations, merges, releases, or delivery. It does not revise a failed candidate automatically. It does not turn every session into a wiki page.

Those boundaries keep one verdict meaningful. A caller may revise the intent and invoke another experiment, but the first result remains what it was. Optional orchestration can dispatch work. Optional learning can analyze verdict collections. Neither can rewrite the current outcome.

Other Context Tools

Beyond the core cycle, these handle specific context problems:

• /council: Optional independent perspectives for a contested judgment. It returns a report; fresh Validate still owns the verdict.

• /research: Context loading before work begins. Deep codebase exploration, knowledge base search, structured report.

• /implement: One bounded RED, GREEN, refactor experiment against the fixed intent.

• /swarm: Optional parallel execution when the work really has disjoint lanes.

• /quickstart: Walks you through your own codebase in under ten minutes.

There are more where these came from. Browse them after install.

What's Still Broken

This has real gaps.

I run this against public repositories and production-shaped work, but the boundaries still need pressure. A fresh context can share an author's blind spot. A passing command can test the wrong behavior. A complete file manifest can still sit under vague acceptance.

The knowledge layer has its own failure mode. An LLM wiki can preserve a wrong synthesis longer than chat would have. Provenance, review, and deletion matter more than page count.

Even with all the validation gates, stuff slips through. You still need good monitoring, good rollback, good hygiene. The cycle improves the odds without guaranteeing quality.

I'm sharing it because I want other people stress-testing it. The more people running this on real work, the faster we find the gaps.

Try It

npx skills@latest add boshu2/agentops --all -g

Run /quickstart. It walks you through your own codebase in under ten minutes. Fork it, break it, make it yours.

Don't want the full system? Keep the intent and acceptance in a durable issue, have a fresh context judge the candidate and evidence, and add a wiki page only when a reviewed lesson will change later work.

If you're running AI agents on real work, I want to hear what breaks.

Where This Came From

I spent months testing every approach I could find and writing about what works when you ship real code with AI agents.

• 12-Factor AgentOps: the full methodology
• The Knowledge Flywheel: Beads plus an LLM wiki
• Multi-Agent Orchestration: coordination patterns (and naming things wrong)
• Spec-Driven Development: why the planning step pays for itself
• The Validation Bottleneck: shift-left for AI agents
• My vibe coding devlog series: the failure log and how it shaped what this became