ProjectsCase study
Agentic NetOps
Self-healing network operations on AWS. The LLM proposes, deterministic checks decide.
The problem
Cloud networks drift. Someone deletes a route or edits a firewall rule, and a service quietly loses its path. Finding and fixing that by hand is slow, and letting an AI fix it alone is a risk, because a model can be confidently wrong.
Agentic NetOps closed that loop on a lab network on AWS. It found the drift, proved it, repaired it and checked the repair, with a deterministic check owning every decision and the model only advising.
How it works
CloudTrail write events flowed through EventBridge and an SQS queue to a detector, which opened an incident in a Step Functions workflow. The workflow diffed a live snapshot of the network against its declared baseline to prove the drift, then measured the impact with VPC Reachability Analyzer and a probe inside the network.
An LLM on Bedrock wrote the diagnosis, but its guess at the fault was only scored against the deterministic classification, never trusted. The repair came from a planner that can only converge the network back to its baseline, so a free-form fix was impossible by construction, and versioned policy rules checked every plan.
A deterministic risk tier decided who had to sign off. A low-risk repair ran after a 60-second veto window unless someone canceled it, a medium one took one approval, and a high one took two different approvers. Whoever caused the drift was recused from every decision on their own change.
Repairs ran under per-incident STS credentials, limited three ways: by the role, by a tag-scoped permissions boundary, and by a session policy naming only the approved plan's resources. Then the network was snapshotted again. The diff had to come back empty and the checks green, and every decision went into a hash-chained ledger whose head was anchored out of band.
CloudTrail write events went through EventBridge and an SQS queue, debounced and with a dead-letter queue, to a detector Lambda. The detector started a Step Functions incident loop: prove the drift by diffing against the declared baseline; measure its impact with Reachability Analyzer, an in-VPC probe and a scored LLM diagnosis; then generate a converge-to-baseline repair and pass it through a policy gate. A deterministic risk tier routed the repair: low risk ran after a 60-second veto window, medium needed one approval, high needed two different approvers. The repair executed under per-incident STS credentials, was verified by an empty diff and green checks, and the ledger head was anchored out of band. Every stage wrote to a hash-chained decision ledger in DynamoDB.
The incident loop. Every stage wrote to the ledger.
Drawn from agentic-netops-src README, its mermaid diagram and The loop, stage by stage, at c1e78b0.
What was measured
- 191 s
- from a seeded fault to self-healed in a live run, with no human touch: the low-risk path, where the veto window opened and nobody stepped inSource: agentic-netops-src README, Live results, at c1e78b0 (2026-11-03)
- 129 s
- from a seeded fault to resolved on the one-approval path, in a live run: detected at 48 s, a plan through the policy gate and waiting for approval at 93 sSource: agentic-netops-src README, Live results, at c1e78b0 (2026-11-03)
- 9 of 9
- seeded fault classes remediated in a live AWS account: security groups, route tables, network ACLs, ENI attachments and DNSSource: agentic-netops-src README fault taxonomy and docs/evals.md, at c1e78b0
- 403
- the answer when one person tried to give a high-risk repair its second approval, in a live run. It took a second, different approver.Source: agentic-netops-src README, Live results (HIGH path), at c1e78b0
- $0
- idle cost: no NAT gateway, no EC2, no always-on compute, and the one metered check hard-capped in codeSource: agentic-netops-src README, Cost discipline, at c1e78b0
The production console, replaying the preserved live run. Forge one ledger entry and re-hash the whole chain, as an insider with table access could, and the ledger still verifies. The head anchored out of band when the incident closed is what no longer matches.
Source: agentic-netops-src scripts/replay.py at c1e78b0, captured 2026-11-09. Tap or click the figure to open it full size.
How it was checked
Tests that need no credentials
301 automated tests run with moto mocks and bogus static credentials, so any stray call to real AWS fails authentication. 11 of them drive the replay console in headless Chromium.
A standing red-team suite
On every run it re-checks tier forcing, plan tampering, replayed approvals, two-party bypass and ledger tampering.
Every forgery caught
A test forges every hashed field of every preserved ledger entry, 1,836 forgeries in all, and requires each one to be caught at its own sequence number.
The permissions boundary, proven
A script proves the remediation boundary action by action against the deployed policy, checks the resource tags live, and shows that nothing but the executor can assume the role, all without changing anything.
Checkable without AWS
Every decision was exported before teardown. The console replays it locally with every SHA-256 recomputed in the browser, and an offline verifier re-runs the API's chain check over all 70 incidents in the export: 34 verify, 36 were recorded before the hash chain existed, and 0 are broken.
Limits
It ran against one lab VPC with nine seeded fault classes, so it shows the loop and its controls, not scale. The stacks are torn down to a $0 bill, and the run lives on as an export that the console replays and the verifier re-checks.