12-Factor AgentOps

Note (July 2026): this essay describes the factor set as first published. The methodology has since moved to v4: same twelve-factor shape, renamed factors, and a four-phase lifecycle (Prepare → Bound → Select → Govern). Current canon lives at 12factoragentops.com.

AI agents hallucinate. They lose context mid-session. They claim success on code that doesn't compile. Debugging their output takes longer than writing it yourself.

These are operational problems. Infrastructure had the same problems fifteen years ago, and we fixed them by building practices around unreliable components.

12-Factor AgentOps is DevOps for vibe-coding. Shift-left validation for coding agents. Catch it before you ship it.

12-Factor AgentOps applies that approach to AI workflows. For why this matters, see The REPL Is Dead.

The Core Insight

AI agents fail in the same ways infrastructure used to fail. And we already know how to fix it.

Servers crashed. We didn't make them crash-proof. We built Kubernetes and let them keep crashing. Deploys kept breaking production, so we built CI/CD to catch the break before it shipped. Config drift caused outages until GitOps made the repo the source of truth.

The agents are smart enough. What they're missing is context engineering.

I've deployed 50+ production AI applications across regulated, air-gapped environments. 100+ GPUs in air-gapped networks where "just restart it" isn't an option and "it works on my machine" gets you escorted out. Every failure pattern I've seen with coding agents (context overflow, silent degradation, confident wrong answers) has a direct parallel in infrastructure. And infrastructure already solved it.

Shift-Left Validation

Traditional workflow:

Write code → Ship → CI catches problems → Fix → Repeat

Current operating loop:

Intent → Plan → Implement → fresh Validate → durable verdict

The fresh verdict exists before the caller decides whether to land or release the change.

The Three Core Skills

1. /pre-mortem: Simulate failures before implementing

"What could go wrong with this plan?"

Run BEFORE implementing. Identifies risks, missing requirements, edge cases.

In practice: before my agents touch a K8s operator, /pre-mortem catches things like "this CRD change breaks backward compatibility" or "the reconciler will infinite-loop if the finalizer isn't idempotent." These aren't hypothetical; they're bugs I've shipped.

2. /vibe: Validate before you commit

"Does this code do what you intended?"

The semantic vibe check. Does the implementation match intent? Run BEFORE every commit.

This is the admission controller for your codebase. In K8s, you don't let a pod into the cluster without validation. Same principle: no commit enters the repo without a vibe check. Across thousands of commits in the months before I started this role, the ones that caused rework were almost always the ones I skipped this step on.

3. Learn later: Propose a reviewed repair

"What evidence would justify changing the next session?"

A later Learn step can inspect verdicts and propose a new bead or wiki edit. A person or repository gate still reviews the change.

This is closer to a postmortem action item than automatic memory. Not every session produces reusable knowledge, and a larger wiki does not make the next result better by itself. The current verdict stays fixed; any repair becomes explicit future work.

Coding Agent Specific

This framework targets coding agents, AI assistants that write, modify, and review code:

• Claude Code running in terminal/IDE
• AI pair programming sessions
• Code generation with validation workflows
• Agents using Read, Edit, Write, Bash for development

We are NOT building a framework for customer service chatbots, RAG Q&A, multi-modal agents, or general autonomous production agents.

For those, see 12-Factor Agents by Dex Horthy. We're coding-specific.

The 12 Factors

None of this is new. It's infrastructure patterns applied to a new domain.

Foundation (I-IV)

I. Automated Tracking: Track everything in git.

Infrastructure parallel: Infrastructure as Code.

Every agent action, decision, and output, versioned and traceable. Not in a chat log that disappears. In git, where it's searchable, diffable, and survives the session.

I track agent work through Beads, a git-backed issue tracker where every issue is a markdown file in the repo. When an agent closes an issue, the evidence lives in the commit. Across 31 repos, that turns "I think that agent did something" into "here's exactly what happened."

II. Context Loading: Stay under 40% context.

Infrastructure parallel: Memory management.

You don't run a server at 95% RAM. You don't pack a container until the OOM killer shows up. Same rule for agent context windows.

Load documentation just-in-time. Compress aggressively. Start fresh with each workflow phase. The 40% threshold is where I've consistently seen the quality cliff.

III. Focused Agents: One agent, one job.

Infrastructure parallel: Microservices.

A raid team doesn't send the healer to tank the boss. Specialization works. Same with agents.

In my multi-agent setup, each worker gets one issue. Not a feature. Not an epic. Write the JWT validation middleware. That's it. The worker doesn't know about the rest of the auth system. Focused agents produce better code because they stay within the context they can actually reason about.

IV. Continuous Validation: Check at every step.

Infrastructure parallel: CI/CD pipelines.

Don't trust the agent. Verify at every step. Not at the end. At every step.

The pipeline: /pre-mortem before work starts, /vibe before every commit, automated tests after every change. Three gates. If the agent claims it's done and any gate fails, it's not done. I've had agents produce code that compiles, passes tests, and is completely wrong. The vibe check catches semantic correctness that tests miss.

Operations (V-VIII)

V. Measure Everything: Observe agent behavior.

Infrastructure parallel: Prometheus/Grafana.

Track completion rates, rework rates, context utilization, cost per issue. Without metrics, you're vibing in the dark.

The AgentOps flywheel (ao) tracks session quality over time. Which prompts produce clean first-pass code? Which patterns cause rework? After hundreds of sessions, the data tells you things intuition misses.

VI. Resume Work: Save state, pick up later.

Infrastructure parallel: Persistent volumes.

Sessions crash. Context fills up. Machines restart. If your agent workflow can't survive an interruption, it's a toy.

Every piece of in-progress work lives in the issue tracker, not in the agent's head. When a session dies (and they do), the next session reads the hook, sees the state, and continues. The work survives the worker.

VII. Smart Routing: Send to right specialist.

Infrastructure parallel: Load balancing.

Not every task needs your most expensive model. Routing simple linting to Opus is like routing health checks through your WAF. Match the task to the capability.

Research spikes get the full context treatment. Mechanical refactors get lightweight agents with tight prompts. The routing decision is part of the dispatch, made at planning time.

VIII. Human Validation: Humans approve critical steps.

Infrastructure parallel: Change management.

Agents don't get to merge to main unsupervised. Period.

Automated gates catch the mechanical failures. Human review catches the "this works but it's the wrong approach" failures. In high-stakes regulated environments, every change gets human eyes.

Improvement (IX-XII)

IX. Mine Patterns: Extract learnings.

Infrastructure parallel: Postmortems.

Every session contains signal. The retro extracts it. The flywheel stores it. The next session loads it.

I have 35+ extracted learnings from agent sessions. Things like "workers write files, team lead commits (prevents git index corruption)" and "pre-identify shared files during planning (merge issues or assign same worker)." Each one is a bug I hit, extracted into a rule that stops the next team from hitting it.

X. Small Iterations: Continuous improvement.

Infrastructure parallel: Kaizen.

Big changes break things. Small changes are easier to measure and reverse. Ship a factor at a time. Measure. Adjust. Repeat.

Start with tracking (I), context management (II), and focused agents (III). Add validation (IV) when those feel natural. The framework is modular by design. You don't need all of them on day one.

XI. Fail-Safe Checks: Prevent repeat mistakes.

Infrastructure parallel: Admission controllers.

Once you know a failure mode, encode it. Make it impossible to repeat.

Example: agents used to overwrite shared files when running in parallel. Now the dispatch phase marks file ownership per worker. The check is automatic. The bug can't recur.

XII. Package Patterns: Bundle what works.

Infrastructure parallel: Helm charts.

When a workflow works, package it. Make it repeatable. Make it shareable.

In my orchestration setup, these are "formulas," workflow templates that encode proven approaches. Need to add a new API endpoint with tests? There's a formula for that. Need to refactor a module with backward compatibility? Formula. Once one person figures it out, the playbook is reusable.

The 40% Rule

Both humans and AI fall off a cliff when overloaded. Not gradual decline; a hard drop.

For AI agents, the threshold is around 40% of their context window. Beyond that, hallucinations spike and reasoning degrades. Chroma's research showed the degradation is measurable; 40% is where I set my line.

In practice:

• Never exceed 40% context utilization in a single workflow phase
• Load documentation just-in-time (JIT) rather than pre-loading everything
• Compress information aggressively before feeding it to agents
• Start fresh with each new workflow phase

I've watched agents produce perfect code at 30% context, then produce confident garbage at 60%. (I merged three of those before I learned to check.) The failure isn't obvious. The code still looks right. It compiles. It might even pass some tests. But the logic is wrong in ways that take longer to debug than rewriting from scratch. The 40% Rule exists because I've been on the wrong side of that cliff too many times.

The Origin

These factors came from building, not theorizing: thousands of commits across months of daily agent sessions, in the same air-gapped environments described at the top of this essay.

The infrastructure patterns (validation gates, context management, pattern extraction) were already in my muscle memory from years of K8s operations. The same operational philosophy Gene Kim described in The Phoenix Project (flow, feedback, continuous learning) applies directly to coding agents.

The current product boundary is narrower. AgentOps owns one bounded Plan, Implement, and fresh Validate experiment. Beads is the preferred intent source. An LLM wiki can preserve reviewed knowledge after the work, but it does not decide whether the current candidate passed. Orchestration remains an optional adapter outside the core loop.

The Takeaway

We didn't make infrastructure reliable by making servers better. We made it reliable by building operational practices around unreliable components.

Coding agents are a new indeterministic worker class. The practical response is familiar: bound intent, gate output, record the result, and maintain reusable knowledge.

The 12 factors came from production use, failed sessions, shipped work, and repeated review misses. Each one names a concrete operating obligation. The current canon and evidence live at 12factoragentops.com.

I built this discipline for engineers, and the same idea (give the agent durable context, gate its output, keep a human in control) is exactly what I now translate into plain operating habits for people who aren't software engineers.

Try It

# Clone the framework
git clone https://github.com/boshu2/12-factor-agentops

# Start with the quick-load summary (AI sessions)
cat docs/00-SUMMARY.md

# Or dive into specific factors
cat factors/01-context-is-everything.md  # Context discipline (the 40% rule)
cat factors/07-validate-externally.md  # Validation gates

Links: 12factoragentops.com · GitHub · Gene Kim's Vibe Coding · Original 12-Factor App · 12-Factor Agents (general agents)

Related

• The REPL Is Dead. Long Live the Factory.: The why: where coding agents are headed in 2026
• Building vibe-check: The toolchain for measuring AI collaboration
• Building This Website with Vibe-Coding: The 12 factors applied to a real Next.js build