Probes

A probe is a vital we run for you. Instead of your job checking in with Mast, Mast checks on your URL, port, certificate, DNS record, host or mail server on an interval, and pages you through a channel when it stops answering. Probes live in the web app under Probes, and in the API under /v1/mast/probes. They need a Tissue Mast account; the phone app alone has vitals, which cover the case where the thing being watched can make an outbound request.

Down is the only state that pages. Everything else a check can find is shown on the probe, on the overview and on a status page, and stays quiet.

Making one

New probe asks what to check, what a good answer looks like, and how often.

A URL. A request to an http:// or https:// address, following up to three redirects, with ten seconds for the whole check. It's a GET unless you pick HEAD, POST, PUT or PATCH. Optional conditions tighten what counts as up:

• the status the response must have, or several separated by commas, such as 200,301. With none set, any 2xx or 3xx is fine.
• a header the response must carry, written Name: text. The header's name matches in any case and its value has to contain text. Name: with nothing after it only asks that the header is there, which suits Strict-Transport-Security:.
• a string the body must contain.
• a string the body must not contain, for the page that answers 200 and says maintenance or error.
• a JSON path and the value expected there, for a health endpoint that reports on itself. The path is a JSON pointer such as /status or /checks/0/ok, or the dotted status.ok if you'd rather. The value is read as JSON, so "ok", true and 3 mean a string, a boolean and a number, and text that isn't JSON is taken as a string. Leave the value empty and the path only has to exist. A body that isn't JSON fails the check.

The body conditions can't go with HEAD, since a HEAD response has no body, and a probe that asks for both is refused. Only the first 256 KiB of a body is read. Ignore case makes both body strings match whatever the case of the page.

What Mast sends can be shaped too, for an endpoint that wants a key, a login or a request body:

• request headers, up to 10 of them, blank ones included, each written Name: value, such as Accept: application/json or X-Client: mast. A value is up to 1024 bytes and the lot up to 4 KiB. Headers are stored as Name: value with one space after the colon, blank entries are dropped, and they're shown to anyone who can read the probe, so don't put a key or a cookie in one; use the credentials below. Host, Content-Length, Transfer-Encoding, Connection and the other headers that describe the connection rather than the request are set by Mast and refused here. So are Accept-Encoding and Content-Encoding: Mast reads the body uncompressed, so it can't ask for a compressed one. Authorization and Content-Type have their own fields below.
• credentials: basic auth, a user name and password, or a bearer token. They're stored encrypted and never shown again. Reading the probe tells you which kind is set and, for basic auth, the user name. To change the password or token, enter a new one; a new user name needs its password too. Entering the same one again changes nothing and doesn't restart the probe. To remove the credentials, clear them. If the site echoes the secret back, as it was sent in the Authorization header, on its own, or escaped inside JSON, it's replaced with [credential] before Mast looks at the answer, so it never turns up in a check's detail. Basic auth's encoded header value, the base64 of user:password, is replaced with [credential] too, whatever the password's length. That also means a condition can't look for the secret itself: body contains Bearer [credential] matches an echoed token, and a condition or JSON path that holds the actual password or token, or the encoded basic value, is refused when you save the probe. With ignore_case on, that refusal ignores case as well, the same way the check does. A password or token shorter than four characters is only replaced where it appears in the Authorization header, because hunting for something that short would blank out ordinary text. Moving the probe to another host, port or scheme needs the secret entered again, or the credentials cleared: they were given for the old address, and Mast won't send them anywhere else.
• a request body, for POST, PUT and PATCH only, up to 16 KiB and sent exactly as written, with its content type. A body with no content type goes as application/json. Like the headers, the body is shown to anyone who can read the probe, so don't put a key or a password in it.

Request headers, credentials and the body go only to the address you gave. A redirect to a different host, port or scheme is followed without them, so a login redirect to another domain never sees your token or your login form. A 303 redirect is followed with a GET, and so is a 301 or 302 after a POST; the body isn't sent again. A 307 or 308 repeats the request with its method and body while it stays on the same host, port and scheme, and with its method alone once it leaves.

A check sends real requests on every round from every location, so a POST that creates something creates it each time. Point it at an endpoint made for health checks, or one that's harmless to repeat.

A port. host:port, for databases, MQTT and anything else that isn't covered below. With nothing else filled in, Mast opens a TCP connection to host:port and closes it, sending nothing. Two optional fields turn that into a short conversation:

• send: text to write once the connection is open, up to 1024 bytes. Write \r, \n and \t for carriage return, newline and tab, and \\ for a backslash, so a Redis check sends PING\r\n.
• reply contains: text the server's answer has to contain, such as +PONG. With nothing to send, Mast reads what the server says first, which is how you check a banner on an FTP or SSH port. Up to 4 KiB of the reply is read, within the check's deadline.

Ignore case applies to the reply match. A server that closes the connection without saying the text, or says something else, fails the check.

A certificate. host or host:port, port 443 when you leave it out. Mast connects, does a TLS handshake, checks that the chain is trusted and the certificate is for that name, and hangs up without sending an HTTP request. That makes it the probe for a TLS service that isn't a website (an IMAPS or LDAPS port, a mail server on 465) and for a site where you care about the certificate and not the page. An untrusted chain, a certificate for another name or one that has expired is down. One with less than 14 days left is class tls and stays up.

A DNS record. The name to look up, such as example.com, and one record type: A, AAAA, CNAME, MX, TXT or NS, A when you don't pick. An IP address isn't a name and is refused. Mast asks a public resolver and judges the answer. NXDOMAIN, SERVFAIL, a refusal, or an answer with no record of that type is down, and so is a resolver that doesn't answer within four seconds. Two optional conditions:

• a record is: one record has to be exactly this. Names compare without case or the trailing dot, an MX can be given as mx.example.com or with its preference as 10 mx.example.com, addresses compare as addresses, and a TXT record has to match character for character.
• a record contains: text one of the records must contain, such as v=spf1 in a TXT record.
• records are exactly: the whole answer, one value per line or separated by commas. The records have to be that set, no more and no fewer, in any order, with names compared without case or the trailing dot. It's the condition for an NS or MX set that should never change, where an extra record means someone else has been at your zone.

The resolver is the one to ask, an IP address with an optional port (53 otherwise). Leave it empty and Mast asks 9.9.9.9, and asks 8.8.8.8 when 9.9.9.9 answers SERVFAIL or doesn't answer within four seconds, so one public resolver's bad minute isn't counted against your domain. NXDOMAIN or an empty answer from 9.9.9.9 is believed and not asked again. When both fail, the check fails, and a SERVFAIL from either is the reason it gives. A resolver you name is asked alone. A resolver given by name is refused, and so is a private one.

A ping. A host name or an IP address, no port. Mast sends three pings, one after another, and waits two seconds for each. Any reply means the host is up. Losing two of the three makes the check slow, which shows on the probe and on a status page as degraded and pages nobody. No reply at all fails the check. A name with several addresses is pinged at the first one. Some of the places we check from aren't allowed to send pings at all; they sit out ping rounds instead of counting as failures, and a verdict is made on the places that can.

A mail server. host:port, port 25 when you leave it out. Mast connects and wants a 220 greeting. How far it goes after that is up to you:

• Banner only reads the greeting and says QUIT.
• EHLO also sends EHLO, which has to be answered 250.
• STARTTLS goes on to STARTTLS, which the server has to offer and accept, then does a TLS handshake and checks the certificate the way a certificate probe does.

A string the banner must contain is optional, and ignore case makes it match in any case. No mail is ever sent: the check never gets as far as MAIL FROM. Port 465 speaks TLS from the first byte, with no greeting in the clear, so check it with a certificate probe on host:465 instead.

Each location says its own name in EHLO, such as probe-us-w.tissue.systems, and each name resolves back to that location's address. The names are listed with the addresses at tissue.systems/mast/probes.

A refusal is not down. Mail servers turn away connections they don't like the look of, and the reasons have nothing to do with whether the server is up: an address on a blocklist, a reverse DNS check, a reputation score, or no reason given at all. A greeting that refuses us with a 5xx code, or a 4xx that names a reason about us, such as a blocklist or too many connections, is class blocked, shown on the probe as refused. It never pages and never closes an outage. A location deferred with a 4xx like that also leaves the server alone for 15 minutes. A 421 with no reason, or one that says the service isn't available, is the server saying it's unwell, and that fails the check like any other failure.

When every location that voted in a round was refused, the probe's channel gets one quiet notice, <name>: every location was refused, at normal with no acknowledgement and no repeat. It isn't sent again until a round gets through and then everyone is refused again. To stop the refusals, ask whoever runs the server to allow our addresses or EHLO names. Or check port 587 instead: servers there expect clients, not other mail servers, and rarely refuse by address.

Port 25 is checked at most every 5 minutes, whatever your plan allows. Mail servers throttle and blocklist addresses that connect to port 25 often, and a probe that knocked every 30 seconds would soon be refused everywhere. A shorter interval is refused when you save the probe. Port 587 and every other port take your plan's floor. Some of our locations can't reach port 25 at all, because their network blocks it; they sit out port-25 rounds instead of counting as failures.

Interval. Every 30 seconds, minute, 2, 5, 10, 15 or 30 minutes, or every hour. Free checks no more often than every 5 minutes; every paid plan goes down to 30 seconds. An interval below your plan's floor is greyed out in the form and refused by the API with the floor in the error. A mail server on port 25 has a floor of 5 minutes on every plan.

Channel. Where the page goes. A probe with no channel is checked and shown and pages nobody, and the probe's page says so in those words. A probe pages at the channel's default level, capped by the channel's ceiling as any send is, so a probe on a channel whose ceiling is loud can never wake you. Pick a channel made at page for the things that should.

The same fields over the API:

curl -X POST https://api.tissue.systems/v1/mast/probes \
  -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" \
  -d '{"name":"homepage","kind":"http","target":"https://example.com/",
       "expect_status":200,"body_contains":"<title>","max_ms":2000,
       "interval_secs":300,"channel_id":"mc_9f2a41c8"}'

kind is http, tcp, tls, dns, icmp or smtp. paused: true saves the probe without starting it. Each POST makes a new probe, so a create sent twice, or retried after a timeout, makes two; list the probes before retrying, and delete the extra one. Listing, reading, editing, deleting and pausing are GET, PATCH, DELETE and POST …/pause on the same path with the token's mast:read or mast:write scope.

Each kind reads its own fields, and a field another kind reads is refused rather than ignored:

Changing a probe's kind empties the fields the old kind read, apart from the ones the same request sets again. An expect_value on an http probe needs a json_path. A text field that's empty comes back as null.

A health endpoint behind a token, asked with a POST:

curl -X POST https://api.tissue.systems/v1/mast/probes \
  -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" \
  -d '{"name":"api health","kind":"http","target":"https://api.example.com/health",
       "method":"POST","request_body":"{\"deep\":true}",
       "request_headers":["X-Client: mast"],
       "auth":{"type":"bearer","token":"'"$HEALTH_TOKEN"'"},
       "json_path":"/status","expect_value":"ok",
       "interval_secs":60,"channel_id":"mc_9f2a41c8"}'

A DNS probe for a mail domain's MX record, asked of a resolver you name:

curl -X POST https://api.tissue.systems/v1/mast/probes \
  -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" \
  -d '{"name":"mail MX","kind":"dns","target":"example.com","method":"MX",
       "expect_value":"mx.example.com","resolver":"9.9.9.9",
       "interval_secs":300,"channel_id":"mc_9f2a41c8"}'

What a probe may check

A probe reaches public addresses only, whatever its kind. A target, a DNS probe's resolver and every address a target's name resolves to have to be public: loopback, private ranges, link-local, carrier-grade NAT and the other reserved spaces are refused, and so are localhost and names under .internal. A name that resolves to a public address and a private one is refused as a whole rather than half-used. The check runs when the probe is saved and again at every check, because a name that resolved to a public address on Monday can resolve to a private one on Tuesday. Every redirect hop is resolved and checked the same way. A probe whose name starts resolving into denied space shows not a public address on every check until you fix the name or the probe. A DNS probe's target is the name being asked about, not a host anyone connects to, so it isn't resolved here; the resolver it's sent to is the address that has to be public.

A kind that too few of our locations can run yet is refused when you save the probe, with an error that says how many can. A verdict needs at least three locations on two networks, and a probe that could never have that would sit inconclusive forever. The same goes for the newer conditions: body must not contain, JSON path, several statuses, a header and ignore case on a URL, send and reply contains on a port, records are exactly on a DNS record, and request headers, credentials, a body or a method other than GET and HEAD on a URL. A plain URL check and a plain port connect run everywhere and are never refused for this.

What a check can find

A location's check fails when the site refused the connection, timed out, failed TLS, served an expired certificate, redirected more than three times, answered the wrong status, answered without the string or with the one it must not have, or answered JSON without the value. For the other kinds it fails on a certificate nobody trusts or that names another host, a DNS answer that is NXDOMAIN, SERVFAIL, empty or missing the record you asked for, no reply to three pings, a port that didn't send the reply you asked for, and a mail server that never greets, isn't speaking SMTP, says it's unavailable, or fails the step you asked for. A mail server that refuses us is blocked, not a failure. The classes above are what the round of checks adds up to.

blocked is neither up nor down. For a mail server, it's the refusal described above. For a URL, the site answered, but a WAF or a CDN's bot filter refused the prober, so the site's own status is unknown. Paging on that would wake you for something your own CDN did on purpose, so it never pages, and it never closes an outage that was already confirmed either. The probe's page names the vendor when the fingerprint is recognisable and tells you what to allow: a rule for the probe's vantage in the service in front of your site. The addresses to allow, and the User-Agent every probe sends, are at tissue.systems/mast/probes, also as probes.txt and probes.json. A probe that stays blocked is checking nothing useful until you do. When every location in a round is turned away, the channel gets the same single quiet notice a refused mail server sends.

If Mast ever can't read a probe's saved credentials, no location can check it. That's a fault on our side, but it still leaves the site unwatched, so the probe goes back to pending (an outage already confirmed stays down), its last result says so, and the channel gets one quiet notice, <name>: Mast can't read its credentials. Enter the password or token again and checks pick up on the next round. A check you run by hand in the meantime says the same thing and doesn't count against the probe's uptime.

An expiring certificate is class tls and stays up. An expired one fails the TLS handshake and is down. That goes for every probe that sees a certificate: an https:// URL, a certificate probe, and a mail server probe that goes as far as STARTTLS.

Certificate notices

A probe that sees a certificate also tells you before it runs out. When the certificate has 30, 14 and 7 days left, a notice goes to the probe's channel at normal, and at one day left a loud one. These are notices, not pages: they never page, ask for an acknowledgement or escalate, and the channel's ceiling still applies to them. A probe with no channel sends none.

Each certificate gets each notice once. A renewed certificate has a new serial, so its notices start over; the old one's are not sent again. For a URL, the certificate is read on the first https hop, an http:// address that redirects to https:// included, and read again about every six hours. Once a certificate has expired there is no more notice; the probe is down, and that pages.

Domain expiry

On Solo and up, Mast also watches the registration of the domain behind each probe's target and tells you before it lapses. A probe on https://www.example.co.uk/ watches example.co.uk. We ask the domain's registry once a day over RDAP, the registries' public lookup service, and when the registration has 30 or 14 days left a notice goes to the probe's channel at normal. At 7 days it's loud. Like certificate notices, they never page, ask for an acknowledgement or escalate, and the channel's ceiling still applies.

Each mark is sent once for each registration. When you renew, the registry gives a date a year or more later and the marks start over. A date that moves by a few days without a renewal, because the registry corrected it, doesn't send a mark again. A channel gets one notice per mark for a domain however many of its probes watch that domain, and the title names the probe only when one probe does. Probes on the same domain with different channels each send to their own. A paused probe sends none, and neither does a probe with no channel.

Some registries don't run an RDAP server, and a few answer without an expiry date. For those domains, and whenever a lookup fails, the probe shows the domain expiry as unknown and sends nothing. An unknown expiry is never a reason to page you. If one lookup fails less than three days after a good one, we keep the date we already had.

A probe on an IP address, on a reverse-DNS name under .arpa, on a name like localhost, or on a bare public suffix like co.uk has no domain to watch. A site on a name someone else hands out, like me.github.io, watches the domain they registered, github.io in that case. To stop the notices for one probe, set domain_expiry to false on it, and set it back to true to turn them on again. In the API, GET /v1/mast/probes/{id} and each probe in GET /v1/mast/probes show domain_expiry with the domain, the expiry date, the days left, the registrar and when it was last looked up. A part day counts as a day, so a domain that runs out in 4 days and 20 hours shows 5. A domain with letters outside ASCII, like bücher.de, comes back in both spellings: domain is the registry's form, xn--bcher-kva.de, and domain_unicode is the same name in lowercase Unicode, bücher.de. BÜCHER.de, bücher.de and xn--bcher-kva.de are one domain, looked up once a day however many probes watch it. While the probe is paused its state is paused.

What down means

Every check is a round: eight locations on six networks fetch your target at the same moment, and the probe is judged on what came back. Down is more than half of them failing, on at least two different networks. One location failing is partial; a continent failing while the rest get through is regional. Both are shown and neither pages, because a page for either would mostly be about the internet between us and you, not about your site.

A location that cannot reach a control site of ours in the same round has lost its own connectivity, and nothing it says counts, its clean answers included. A location that answers after the deadline is not counted either. When fewer than three locations could vote, or fewer than half of them, the round is inconclusive: nothing changes and nobody is paged. A probe whose rounds keep coming out inconclusive is our problem, and we watch for it.

One failure is a suspicion

A down round is re-run 10 seconds later, and only the second down round in a row turns the probe down and pages. Until then the card keeps showing what it showed. Two clean rounds in a row close the outage, which resolves the page on every phone it reached, the same way a vital's first beat after a silence does.

The page says what was seen and where: the detail of a failing check (HTTP 502, expected 200, connect refused, timed out), what that detail usually means, which cities failed and which still answered, and when the failure started, was confirmed, and was last clean.

A site that comes back and fails again inside ten minutes pages once more, as down again. After that, for half an hour, every further flip is noted on the open incident and pages nobody. A site that stays down past that window is paged for again.

The path to your site

On Solo and up, a location whose check fails also traces the route to your target, the way traceroute does: it sends three probes to each hop, up to 30 hops, and keeps each hop's address, how many of the three came back, and the average and worst time. It traces to the address the check used, and a DNS probe traces to the resolver it asked. This happens only when a check fails because nothing answered: a timeout, or a connection that never opened. A wrong status code, a refused connection, a certificate or other TLS error, or a name that doesn't exist got an answer from somewhere, so the path isn't the problem and no trace is taken. Each location traces a probe at most once every ten minutes, so a site that keeps flapping doesn't get traced on every round. Hops that answer from a private address, like your own router or a carrier's internal network, show without an address.

When the probe is confirmed down, the page carries one line from that trace, saying where the path stopped answering: Path from Chicago: last answer from hop 7 (203.0.113.9, 31 ms), nothing past it toward 198.51.100.4. A hop that answered in under a millisecond shows as <1 ms. A path that reached your server while the check still failed points at the server; one that stops partway points at the network in between. Plenty of routers on the internet don't answer traces at all, so a silent hop on its own proves nothing. A site that comes back and goes down again within ten minutes may page without a path line, because the only trace that location has is from the earlier outage. Changing a probe's target, kind or resolver drops its traces, since they describe a different path, and a trace of the old target that was still running when you made the change isn't kept.

In the API, each check in recent on GET /v1/mast/probes/{id} has fired_at, when that location sent it, and each failed check carries trace: the trace that location took after it, or null if it took none. GET /v1/mast/probes/{id}/traces lists them newest first by at, the time each trace came in, 20 by default and up to 100 with ?limit=. On Free that call answers 402. Each trace has the hops, whether the target answered (reached), the one-line summary, and error, which says why a trace couldn't run, for instance when the location can't set the hop limit on its packets or has no IPv6 route to an IPv6 target. When error is set, summary reads trace did not run: and the same reason. A trace is kept as long as the check it followed is still in the probe's history, and never more than 7 days. It goes when the probe is deleted. A location that can't send traces from its network just doesn't add one, and nothing else about the check changes.

Where the checks run from

The locations and their networks are listed, with addresses, at tissue.systems/mast/probes. The probe shows which of them failed and which reached it on every check.

A site that refuses one location for half an hour while every other location reaches it, a geo block or a blocklist your provider subscribes to, stops counting that location for that probe. The probe's page lists it, with the time the refusals started; the first clean answer from that location counts it in again.

Reading a probe

Each probe has a row on the Probes screen: its state, how many of the recent checks passed, the interval, the vantage, the last result, and a strip of the last hour's checks coloured by class. Overview lists any probe that needs a look. Opening a probe shows the recent checks with their timings and details, the channel it pages, and three buttons:

• Pause and Resume. A paused probe keeps its history and checks nothing.
• Check now runs one check straight away, from one location, and shows the result. A hand-run check never pages, and its result doesn't count toward confirming or closing an outage. It does count as one check in that day's uptime, unless that location couldn't run it at all, for instance because Mast couldn't read the probe's credentials.
• Edit and Delete.

A probe's outage is an incident like any other: it appears under Incidents with the checks that opened and closed it, and on any status page that lists the probe as a component.

Caps, and probes that pause themselves

At the cap, New probe and the API both say so and name the plan that raises it. Nothing already checking is touched.

A probe pauses on its own if its target has been gone for 7 days (the name doesn't resolve, or there's no route to it) or if nobody has signed in to the account for 90 days. Both send an email first, both show the reason on the probe, and Resume starts it again. Limits has the exact rules.