Why my stopped agent remained "UP" on the HUD

// CONTEXT

Testing the Lookout monitoring dashboard. I successfully connected a remote push agent and the tile appeared as "UP". I then ran `docker stop mon-agent-test` and `docker rm`. Expected the tile to instantly turn red or disappear, but it stubbornly stayed green.

// WHAT I THOUGHT FIRST (and got wrong)

- Suspected the React frontend was caching the WebSocket state. Force-refreshed the page — tile was still UP. - Suspected a zombie Docker container. Ran `docker ps` — nope, the agent was completely gone. - The tell: The backend wasn't throwing errors. It was perfectly happy because it simply hadn't received any bad news.

// THE ARCHITECTURE THAT DID IT

The agent pushes data (POST /metrics), but the backend only reacts to incoming traffic:

// ↳ INGEST ROUTE

@app.post("/metrics/push")
def ingest_metrics():
    service.state = "UP"
    service.last_seen = now()

// WHAT I FOUND

In a push-based architecture, the backend is passive. When the agent dies, it doesn't send a "I'm dying" request (unless catching SIGTERM). Therefore, the backend retains the last known state ("UP") indefinitely. The fix isn't on the frontend — it's adding a background "watchdog" task on the backend to sweep the database: "If now() - last_seen > timeout_threshold -> set CRITICAL".

// LIMIT

For a quick local demo, manually deleting the service via the API (`DELETE /api/v1/services/{id}`) clears the UI. But for production, a timeout watchdog is mandatory to handle sudden network partitions or power failures.

// TAKEAWAY

In distributed systems, the absence of data is data. But you have to actively write code to listen for the silence.