Featured, then the body of work
The thesis, made into a benchmark — plus three more angles
The verification sandwich, safety by construction, and the networking bedrock it all
stands on.
⊨
A safety benchmark for AI network-operations agents — the thesis, made
measurable.
An agent proposes a fix; a deterministic scorer checks seven safety properties;
approval binds to a sha256 of the exact plan; and a gated harness executes that plan
in a Docker (FRR) lab so the lab's own validator proves recovery. Agents answer
clean-room, seeing only a redacted prompt.
13 implemented tasks span EVPN/VXLAN, BGP route-policy, SONiC
ACL/SAI, and a cross-domain ambiguity task. On the live-backed EVPN tasks, 18/18
combined remediations were live-verified in Docker FRR labs (suite_v2 + task-006); the
first BGP task adds a separate 3/6 clean-room live result. On hard-mode tasks
where the safe move is to refuse, both evaluated models abstained 24/24 with
0 unsafe-action attempts. 20 sha256 artifact locks; a Docker-free CI
re-verifies everything.
LIMITS — local Docker (FRR) labs only, not physical hardware and
not production networks; SONiC ACL/SAI tasks are fixture-replay only; Claude-family
models, three repeats (consistency, not statistics).
PythonFRR · Docker
BGP-EVPN/VXLANsha256 locks
⇄
Natural language → formally verified network configuration on SONiC.
+---------------+ +----------------+
| User terminal | | qwen2.5:7b |
+-------+-------+ | (Ollama,local) |
| +--------+-------+
| question + tools | tool calls
v |
+---------------+ <--tools--> |
| agent.py +--------------+
+-------+-------+
| read or write
+----+-----+------------+
v v v
+-------+ +----------+ +-----------+
| SONiC | | Batfish | | snapshot |
| VS | | (parser) | | builder |
+-------+ +----------+ +-----------+
A local LLM parses intent against a real SONiC virtual switch. Before anything applies,
Batfish verifies a candidate snapshot — new critical issues surface in the diff a human
reviews. After apply, the agent re-reads CONFIG_DB and checks each predicted change
actually materialized, with per-prediction verdicts.
The full propose → formally verify → human-approve → apply →
prove-it-happened loop runs against real switch software. Built in 7 phases, each gated
by a working demo.
LIMITS — Batfish verification is parser-level, not full dataplane
analysis; one virtual switch.
PythonOllama · qwen2.5
BatfishSONiC VS
⊘
A NetOps console where no execution path exists — and CI proves it.
collectors (show-only · allow-listed)
│ evidence
v
8 deterministic rules ───> incident
│
v
LangGraph workflow (6 nodes) ──> analysis
│
v
local-LLM RCA + RAG citations
│
v
remediation PLANNER — plans only
│ requires_approval=True (DB level)
v
[ authenticated admin approval gate ]
✗ no execution path exists in code
═ CI AST scan fails the build on any
remote-exec import in protected pkgs
Read-only collectors, 8 deterministic anomaly rules, a LangGraph incident workflow,
local-LLM RCA with RAG citations — and a planner that only ever drafts. Every plan is
requires_approval=True at the database level; an AST scan in CI fails the
build if any protected module imports a remote-execution library.
4 of 8 rules fire from real FRR lab data (the rest are labeled
simulator-only — provenance always tagged). The no-execution guarantee is enforced by
the build, not the README.
LIMITS — portfolio-scale demonstration; half the rule library fires
only on simulated signals.
FastAPILangGraph
FAISS RAGReact + TS
⬡
A leaf-spine EVPN/VXLAN fabric on a laptop, with a validator that proves
it's healthy.
192.168.10.10 192.168.10.20
hostA ── leaf1 leaf2 ── hostB
(AS 65001) (AS 65002)
lo 10.0.0.1 lo 10.0.0.2
\ /
10.0.12.0/24 10.0.23.0/24
\ /
spine1 (AS 65000)
lo 10.0.0.3
underlay: eBGP · overlay: EVPN type-3
IMET ──> VXLAN head-end replication
eBGP underlay, BGP-EVPN type-3 IMET control plane driving VXLAN head-end replication,
two tenant hosts stretched across one L2 segment — and a Python validator that asserts
the overlay end-to-end at every layer. No AI, deliberately: this is the networking
bedrock the AI projects stand on.
Host-to-host traffic crosses the VXLAN overlay, and the validator proves
why it works: EVPN AF up, IMET routes exchanged, HER FDB programmed.
LIMITS — type-2 MAC/IP learning intentionally disabled: the Docker
Desktop kernel breaks BUM split-horizon (documented, with the exact mechanism).
FRRBGP-EVPN
VXLANDocker Compose