Failure semantics and recovery.

FrogNet Specification — Draft 0.9, revision 2026-08-29. Cite a conformance claim against this revision, not against a section number alone.

1. Principle

Croakus: §18, §29, §30

Every condition in this section is a defined state, not an error. An implementation MUST NOT surface any of them as a fault to the application, and MUST NOT enter a degraded or suspended mode on any of them. The application is told that the ground moved; what that means is the application's, because only the application knows what its own state means.

This is a design position, not a convenience. Automatic recovery would require the platform to invent an answer about data it does not understand, and an invented answer that looks correct is the failure mode the whole system is built to avoid.

2. Failure classes

Liveness retry is specified in §6.5: FROGNET_ALIVE_9009_TRIES defaults to 3, only ambiguous negatives are re-asked, and a timeout MUST NOT be recorded as definitive absence. WINNER_HYSTERESIS = 0.50 (discovery/discovery.py:33) is not an election constant and MUST NOT be cited as one: it governs route next-hop flap. A challenger must beat the installed metric-22 winner's measured RTT by at least 50% to displace it, because replacing a winner is an ip route replace that yanks the next-hop on the /24 and interrupts flows in progress. Tunable via FROGNET_WINNER_HYSTERESIS; 1.0 flips on any improvement, 0.0 never flips.

2.1 What does not expire

Capability rows do not age out, and an implementation MUST NOT gate an election on their age [CAPABILITY_DOES_NOT_AGE_V1]. They are written unowned (§10.3), so they survive their writer and persist until a newer value overwrites them. The periodic reaper is garbage collection, not election aging.

An earlier design refused any ballot past a maximum age [BALLOT_ADMISSIBILITY_V1] and is superseded. It answered the right question with the wrong evidence. The fossil that won a database-host election for a full day was not a stale description — that machine still had its cores. It was an unreachable machine, and the question to ask is whether it is there, not how recently it wrote. Age only correlated with liveness and the correlation broke in both directions: a live node whose advertiser write timed out for half an hour was refused, while a dead node that wrote a minute before dying was admitted.

A missing envelope MUST NOT disqualify a candidate either: refusing a ballot for having no timestamp is an age judgement by another name, and against an older store it removed every node pond-wide.

Nor may liveness be re-measured locally. An implementation MUST NOT probe candidate reachability during a gather. There was such a probe and it was wrong: a local probe is a per-node input, so every node probes from its own vantage, obtains a different pool, and elects a different winner — measured 2026-08-08 as three simultaneous answers for the database host across six nodes.

Liveness is already a shared fact. The candidate pool is the capability rows in the store; every node reads the same rows from the same place, and a departed node's rows are removed by the reaper. A fact each node measures for itself is not a fact they can agree on, so the determination MUST be identical on every node by construction rather than by agreement. Candidates MUST be emitted in a deterministic order — sorted by address — so the lists themselves are identical everywhere.

One per-node measurement remains and is legitimate because it answers a different question: whether a candidate sits on this node's LAN, which selects the plane a LAN-scoped role is elected on rather than whether the candidate is alive. It is traced once per candidate per merge and memoised, because it runs inside the merge.

A read failure MUST NOT collapse into an empty candidate set [GATHER_NOT_SILENT_V1]. Silence turns “could not read” into “no candidates” with no trace, which is how a single malformed row once presented as a role with zero hosts. The failure MUST be logged and the accumulated partial list still returned, so one role cannot wedge the whole merge.

Where a host holds more than one capability row, every node MUST select the same one — the newest by timestamp per address [DBHOST_DETERMINISTIC_ROW_V1]. Selecting by store order lets two nodes pick different rows for one host and score them differently, splitting the winner even where the scoring function is deterministic. Here the timestamp is a tiebreak, never a gate.

3. Writes during a failure

Writes from a fragment that cannot reach its store are absent, not queued. An implementation MUST NOT buffer writes for later delivery, MUST NOT retry them against a different store, and MUST NOT report them as succeeded. On heal there is no merge, no conflict, and no replay (§2c, §6.4).

The consequence for readers is the one property most often got wrong: a value written on the far side of a partition remains observable until it ages past the freshness window, and then ceases to be observable rather than becoming wrong. A value outside the window MUST be treated as absent, not as stale-but-usable.

The store decides freshness, not the reader [ENVELOPE_TS_V1]. A freshness-bounded read passes its window to the store, which filters on the row's own update time, on the clock that stamped it. A reader MUST NOT compare its own clock against a timestamp inside the payload. That was the earlier behaviour and it failed exactly as skew predicts: a node whose clock ran two minutes slow disappeared from every roster while publishing correctly, and the only symptom was an empty membership list. One clock decides, and it is the one that wrote the row, so skew between nodes cannot matter.

Freshness is envelope information. A payload MAY carry a timestamp of its own and it MUST NOT be consulted for this purpose. Where a caller needs to report an age — the election logs a ballot's age — the age reported MUST be the store's number.

A row whose freshness cannot be established is not fresh. Where a freshness-bounded read returns a row carrying no update envelope, the reader MUST drop it and MUST say so. A store that does not implement the filter returns everything, and handing those rows back presents a full history as though it were current: departed peers in the roster, months-old calls offered as joinable.

4. Recovery

Recovery is not a separate code path. The method that introduces a node the first time reintroduces it after any of the conditions above: hostReset fires, every object that answers the verb re-states what it is, and the new store is populated by re-assertion rather than by transfer (§6.6). There is no disaster path because a split is not a disaster.

An implementation MUST NOT provide a distinct recovery routine for partition. Where one exists, it is a second implementation of advertise, and two implementations of one rule are two rules.

5. Convergence after failure

Convergence is defined as in §6.7: every member's derived topology stable against current observations. It is a statement about topology only. The store does not converge and MUST NOT be described as converging (§2c).

6. Non-goals

The following are not provided and MUST NOT be claimed: durable queueing of writes across a partition; exactly-once delivery of anything; a replay or write-ahead log; automatic conflict resolution; a quorum or consensus round; notification that a failure occurred. Events that compound when duplicated do not belong in the store (§9.7).

7. References

Specification §2, §6, §8, §9, §7 (Broker). Build manual: Croakus §18, §29, §30.