Autonomous Monitoring That Can't Go Rogue.... agenticuptime.com
Joe Wee · 2026-04-01
When automation becomes the outage
On October 19, 2025, a race condition in AWS's automated DNS management system erased all DNS records for DynamoDB's US-EAST-1 endpoint. The cascading failure took down EC2, Lambda, Fargate, ECS, EKS, and NLB across 100+ services for ~15 hours. The monitoring system designed to catch this issue failed due to the same faulty automation logic [1]. Constellation Research noted: "Automation typically keeps AWS running seamlessly, but in this case it scaled problems because the underlying logic was wrong" [2].
One week later, an inadvertent configuration change to Azure Front Door cascaded into a global outage affecting Azure Portal, Microsoft 365, Xbox Live, and thousands of third-party services for ~14 hours [3].
Google's SRE book documents the "Diskerase" incident: automation misinterpreted an empty set as "everything," sending nearly all CDN machines to disk erasure, wiping production infrastructure within minutes [4]. The lesson: "automation is a force multiplier, not a panacea. Just multiplying force does not naturally change the accuracy of where that force is applied."
The OWASP Top 10 for Agentic Applications classifies this as ASI08 (Cascading Agent Failures): a single fault propagating across autonomous agents, compounding into system-wide harm [5]. Gartner reports that 57% of I&O leaders have experienced at least one AI project failure, with 1 in 5 AI projects in IT infrastructure collapsing entirely [6]. RAND's meta-analysis of 65 enterprise AI initiatives found ~80% fail to deliver promised business value [7].
The industry response: advisory mode
The major monitoring platforms are moving cautiously. PagerDuty's SRE Agent (GA October 2025) operates in advisory mode only - it proposes actions but does not mutate systems. Their engineering team wrote: "We're not crossing that line until those rails are in place and we've proven the agent's judgment on a much larger set of incidents" [8]. New Relic's SRE Agent (February 2026) explicitly does not make production changes: "accelerate understanding, not to take action on its own" [9]. Datadog's Bits AI (GA December 2025) runs investigations autonomously but requires human approval for code fix merging [10].
The pattern is clear: observe and recommend, but don't execute without human approval. The question is how to safely enable execution when you need it.
The product
AgenticUptime is a multi-tenant SaaS monitoring platform. Each tenant gets a Tyga Orchestrator with an architect agent coordinating 8 domain expert agents running on configurable intervals (default 60 seconds).
8 expert agents
Each agent extends a common ExpertAgent base class (540+ lines) and specialises in one infrastructure domain:
| Expert | Monitors | Key thresholds |
|---|---|---|
| Redis | Memory usage, evicted keys, connected clients, hit rate, slow log | Memory 75/90%, hit rate 80/60% |
| MongoDB | Connections, opcounters/sec, replication lag, lock %, WiredTiger cache | Replication lag 10/30s, lock 5/20% |
| MySQL | Connections, slow queries, replication lag, InnoDB buffer pool | Connections 100/200, buffer pool 95/90% |
| DNS | Query latency, failure rate, zone transfers | Domain-specific |
| Web | HTTP response codes, SSL cert validity, response time, error rate | Response 1000/5000ms, SSL 30/7 days |
| TURN/STUN | Allocation success, relay latency, ICE gathering | Domain-specific |
| GlusterFS | Peer status, volume health, brick status, heal count | SSH-based (AES-256-GCM credentials) |
| Overlay Network | Stale FDB entries, unhealthy manager nodes, VTEP fix rate | Stale FDB 50/200, unhealthy nodes 1/3 |
The Web expert batches monitoring to 20 servers per round to prevent timeouts on 50+ stacks. Each expert maintains persistent per-service-instance memory via Tyga Context with baseline maturity tracking (learning -> mature -> stale).
The architect
The architect agent receives all expert reports and performs cross-service correlation using a DEPENDENCY_MAP. If Redis is degraded and Web response times spike simultaneously, the architect identifies Redis as the root cause rather than treating them as independent incidents. This follows the finding from Liu's research on multi-agent incident response: homogeneous centralised team structures achieve the highest success rates, while decentralised specialist teams struggle to reach consensus without leadership [11].
The 4-tier safety model
This is where AgenticUptime goes beyond advisory mode. Instead of binary "advise only" vs "fully autonomous," the system uses a graduated safety model - similar to the risk classification recommended by the OWASP AI Agent Security Cheat Sheet [12]:
| Tier | Name | Examples | Execution |
|---|---|---|---|
| 0 | Observe | Logging, alerts, diagnostic data collection | Auto-execute |
| 1 | Safe Auto | Restart worker, clear cache, kill idle connections | Auto after safety validation |
| 2 | Approval Required | Compact collection, reindex, optimize table | Queued for human approval |
| 3 | Manual Only | Failover, data migration, replace brick | Escalated to human operator |
The approval workflow is configured per-tenant: autoApproveTiers: [0, 1], requireApprovalTiers: [2, 3], with a 1-hour escalation timeout. This mirrors the pattern StackAI recommends: "100% approval for high-risk actions, 5-20% sampling for low-risk" [13].
The A2A safety shield
Tier 1 actions pass through the A2A TrustGate firewall before execution. The a2aSafety.js service implements two layers:
Local Gate 1 (always enforced, works offline): regex denylist blocking flushall, drop.*database, kill.*-9, format.*disk, failover.*force, scale.*down.*0.
Remote Gate 1 + Gate 2 (via a2ainfrastructure.com/v1/evaluate): full remote gate evaluation with context (e.g., agenticuptime:redis:tier-1:auto).
Fallback: if the A2A endpoint is unreachable, tier 0-1 actions proceed (local gate passed), tier 2+ are blocked (fail-closed). The system degrades to observe-only - exactly the pattern Google SRE recommends: "automation should default to human operators if it runs into an unsafe condition" [4].
The Overlay Network example
Docker Swarm's overlay network uses VXLAN tunnels with FDB entries mapping container MACs to tunnel endpoints. A known class of bugs causes stale entries to accumulate [15], degrading network performance across the cluster.
Without safety tiers, a naive remediation agent would flush all FDB entries - including active ones. That's the Diskerase pattern: "empty set interpreted as everything."
With the safety model: FDB diagnostic collection is tier 0 (auto). Targeted stale-entry cleanup is tier 1 (auto after A2A validation - the safety shield checks for flush.*all patterns). Full VTEP reconfiguration is tier 2 (human approval). Network failover is tier 3 (manual only).
Multi-tenancy
Every component is tenant-scoped. Each tenant gets its own ConductorConfig in MongoDB - controlling which experts run, analysis intervals (10 seconds to 10 minutes), AI provider selection (your LLM endpoint, with fallback options), dependency maps, and approval workflows. Context is compressed using Tyga Encoder before each AI call.
References
- Robinson, D. (2025). "A single DNS race condition brought AWS to its knees." The Register
- Dignan, L. (2025). "AWS delivers outage post mortem: When automation bites back." Constellation Research
- ThousandEyes (2025). "Microsoft Azure Front Door Outage Analysis: October 29, 2025." ThousandEyes
- Murphy, N. et al. "The Evolution of Automation at Google." Google SRE Book, Chapter 7. sre.google
- OWASP Top 10 for Agentic Applications (2026). ASI08: Cascading Agent Failures. genai.owasp.org
- Gartner (2026). "AI Projects in I&O Stall Ahead of Meaningful ROI Returns." 57% failure rate, 1 in 5 collapse. Secondary source
- Ryseff, J. et al. (2024). "The Root Causes of Failure for AI Projects." RAND Corporation, RR-A2680-1. RAND
- Mayo, M. (2025). "Context Over Cleverness: Building PagerDuty's SRE Agent." PagerDuty Engineering
- Braun, D. (2026). "SRE Agent: Agentic AI Built for Operational Reality." New Relic
- Tai, K.X. (2025). "Bits AI SRE." Datadog
- Saunders, M. (2026). "Human-Centred AI for SRE: Multi-Agent Incident Response without Losing Control." InfoQ
- OWASP AI Agent Security Cheat Sheet. Risk classification for automated actions. OWASP
- StackAI (2026). "Human-in-the-Loop AI Agents: How to Design Approval Workflows." StackAI
- Baysal, H. (2026). "Asimov Safety Architecture." IETF Internet-Draft
- moby/moby Issue #29024: "Overlay networking VXLAN ARP and intermittent connectivity failures." GitHub
