Incidents

An incident is one thing that went wrong. A site that's down for twenty minutes is one incident, not the forty failed checks, the page, its repeats and the recovery. Incidents in the web app is the list of them.

The list

One row per incident: pages, near-pages (loud messages that would have paged on a higher ceiling), and the floods that dedupe or storm control folded into one. The open ones and the last 7 days show by default.

Opening one shows what the phone got and what Mast knows about it:

what opened it: the failing checks with the cities that failed and the ones that still answered, or the silence that ran past a vital's grace,
who it paged, when each escalation step fired, who acknowledged it and from which device,
what closed it: two clean probe rounds, a vital's next beat, a resolve=1 from the sender, or a person.

Export 90 days as CSV downloads the table, for a postmortem or a monthly review.

How an incident closes

Probes and vitals close their own. A probe's outage closes after two clean rounds in a row, and a vital's after its first beat. The card on every phone it reached turns green, says how long it was open, and stops repeating.

Anything else closes when the sender says so, or when a person resolves it.

Closing one from the sender

A page repeats until somebody acknowledges it. When the thing it's about comes back, the sender can say so, and the card on the phone stops repeating and turns green:

# the outage
curl -X POST https://mast.tissue.dev/mk_1c9f… \
  -d priority=page -d key=db-01 \
  -d title="db-01 unreachable" -d body="no WAL segment in 11 minutes"

# the recovery, closing that incident
curl -X POST https://mast.tissue.dev/mk_1c9f… \
  -d resolve=1 -d key=db-01 \
  -d title="db-01 recovered" -d body="WAL shipping resumed"

resolve=1 closes the most recent open message on the channel carrying that key. The send doesn't become a second message: its title and body replace the outage text on the original card, the repeat and the escalation are cancelled, and every device the page reached gets a quiet update. The 202 names the message that was closed and how long it was open:

{ "ok": true, "id": "mm_7a3e5b91c04d2f68a1b3c5d7e9f02468", "state": "resolved", "open_for": 842 }

open_for is whole seconds from the moment Tissue Mast received the page to the moment it was closed.

A message id works in place of the key, for a sender that kept the id from its own 202:

curl -X POST https://mast.tissue.dev/mk_1c9f… \
  -d resolve=mm_7a3e5b91c04d2f68a1b3c5d7e9f02468 -d body="back up"

The title and body are optional on a resolve. Leave both out and the card keeps the wording it already has and just turns green. A resolve with no text that finds nothing open stores nothing, and answers "state": "nothing_open" with no id.

Three things to know before you wire this into a monitor:

A resolve that finds nothing open isn't an error. It's stored as an ordinary message at quiet, whatever priority was asked for, and it never repeats or asks for an acknowledgement. So a recovery that overtakes its own alert, or one for an incident a person already closed, still shows up in the feed and still wakes nobody. The catch is that a wrong key fails silently, with the page still ringing. Test the pair once on a test channel.
key alone isn't enough. A repeat of the same key folds into the open card only inside the channel's dedupe window, which is minutes; an incident isn't. Past the window, a recovery without resolve=1 is a separate card and the page keeps repeating.
A key can only close its own channel's incidents. A message id belonging to another channel is answered as though it didn't exist.

Flapping

A site that comes back and fails again inside ten minutes pages once more, as down again. After that, for half an hour, every further flip is noted on the open incident and pages nobody. A site that's still down past that window is paged for again. Probes has the rest of the rule.