The Problem
At 2am, a Kafka consumer falls behind. The delivery robots slow. No one is awake.
This post covers the architecture of an autonomous SRE system — not a product you can install, but a pattern you can adapt.
System Architecture
flowchart LR K[Kafka Consumer] —>|lag metric| D[Lag Detector] D —>|alert| A[AI Diagnosis Agent] A —>|action| R[K8s Operator] R —>|restart / scale| K
How the Diagnosis Agent Works
The agent receives the Kafka consumer group ID, lag delta, and recent error logs…
def diagnose(consumer_group: str, lag_delta: int, logs: list[str]) -> Action:
prompt = build_diagnosis_prompt(consumer_group, lag_delta, logs)
response = llm.complete(prompt)
return parse_action(response)
Results
After three months in production: 94% of lag incidents resolved without human intervention.