Specification §11
FNWP-1 and BLDC-1.
BLDC-1 decides what to send; FNWP-1 decides how it travels. The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as described in RFC 2119.
FrogNet Specification — Draft 0.9, revision 2026-08-29. Cite a conformance claim against this revision, not against a section number alone.
1. Division of labour
Croakus: §26, §27, §28, Appendix F
Two mechanisms are specified here and MUST NOT be conflated.
| Name | Stands for | Decides |
|---|---|---|
BLDC-1 | Bullfrog Long Distance Communications Protocol | What to send: a full body, a difference, or an assertion of identity |
FNWP-1 | FrogNet Wire Protocol | How it travels: framing, multiplexing, and reply matching |
BLDC-1 is the language; FNWP-1 is the envelope. A third protocol, SotF — the Song of the Frogs Adaptive Communications Protocol — is the Communicator's media plane and is specified in §15; its control plane rides BLDC-1 and FNWP-1 like any other exchange — a session envelope that reaches SAME against a learned template — while its media frames carry opaque payloads that never reach SAME and are sent in full every time (§15).
The verdict is reached at the handler on each side, by the codec, over canonical values — never on the wire. FNWP-1's involvement begins only after the verdict exists.
2. The pathway
THIS NODE
1 app --HTTP :80--> proxy --> handler (BLDC-1)
2 handler: compare canonical answer against reference
-> SAME | DIFF | FULL (+ request hash)
3 FNWP-1 frame --> established socket -->
PEER NODE
4 FNWP-1 reader --> handler (BLDC-1)
5 handler: rebuild full request from reference + diff
6 origin (Apache -> MySQL) produces the answer
7 handler: compare canonical answer against reference
-> SAME | DIFF | FULL
8 FNWP-1 frame --> back over the same socket
THIS NODE
9 handler: rebuild full reply from reference + diff
10 proxy --HTTP--> app
Every hop is one handler turning a full message into a verdict against a shared reference, and the handler on the far side turning that verdict back into the full message. Reconstruction is exact because both sides retain the reference. The application above the line makes ordinary HTTP calls and MUST require no modification.
3. Frame format
Every FNWP-1 message is a length-prefixed frame. A reader therefore always knows how many bytes to expect before the next frame begins, which is what permits many frames to share one stream unambiguously.
MAGIC b"FNW1" 4 bytes, literal, every frame opcode 1 byte REQ_FULL [MAGIC][0x01][req_hash:16][len:4][semantic_packet:len] REQ_REPEAT [MAGIC][0x02][req_hash:16] REQ_RAW [MAGIC][0x03][req_hash:16][len:4][http_request:len] REQ_DIFF [MAGIC][0x04][req_hash:16][len:4][semantic_diff:len] RESP_SAME [MAGIC][0x13][same_id:16] RESP_DIFF [MAGIC][0x11][same_id:16][len:4][sem_blob:len] RESP_RAW [MAGIC][0x14][same_id:16][len:4][status:2][hdrs_len:2][hdrs][body]
The magic value is the ASCII string FNW1 and MUST be emitted literally. It is a wire constant, not the protocol name. The protocol is FNWP-1; published material naming the protocol “FNW1” is using an obsolete name for it. Hash and identifier fields are 16 bytes (REQ_HASH_LEN, core/codec.py:34); SAME_ID_LEN is likewise 16 (core/semcache_wire.py:84).
Byte order differs between the two layers and MUST NOT be assumed uniform. FNWP-1 frame length and status prefixes are big-endian (network order): struct.pack("!I", ...) for payload lengths, "!HH" for status and header length. The semantic packet inside them is little-endian throughout: "<BBIH" for the header, "<HB" per field, "<q" and "<d" for scalars (core/codec.py). An implementation MUST encode the envelope big-endian and the contents little-endian.
The complete opcode set, verified in core/semcache_wire.py:
OP_LEGACY 0x00 pre-FNWP-1 framing OP_REQ_FULL 0x01 OP_RESP_DIFF 0x11 OP_REQ_REPEAT 0x02 OP_RESP_SAME 0x13 OP_REQ_RAW 0x03 OP_RESP_RAW 0x14 OP_REQ_DIFF 0x04 OP_REQ_MISS 0x21 the hash you sent means nothing to me OP_ERROR 0x30 [MAGIC][0x30][status:2][msg_len:2][msg] OP_SEQ_RESET 0x40 [MAGIC][0x40] — no payload OP_HELLO 0x50 [MAGIC][0x50][ip_len:1][ip_str] OP_RTT_PING 0x60 OP_RTT_PONG 0x61 OP_RTT_LOOP 0x62
REQ_MISS carries the request hash: [MAGIC][0x21][req_hash:16]. Hash truncation to REQ_HASH_LEN on the wire is a v4.1 change and saves 16 bytes per REQ_FULL, REQ_REPEAT, REQ_RAW and REQ_MISS; internally the hash is a full 32-byte SHA-256.
4. The semantic packet
The packet carried inside FULL and DIFF frames has its own header, <BBIH>: wire version, flags, opcode, field count. WIRE_VERSION is currently 5 (core/codec.py).
| Flag | Value | Meaning |
|---|---|---|
FLAG_COMPRESSED | 0x01 | LZ4 applied. Applied only where it actually shrinks the payload — a general-purpose compressor's own framing runs to a dozen bytes, which on a fifty-byte sensor reading is an expense, not a saving. |
FLAG_DIFF | 0x02 | Not all fields are present. |
Field types are one byte per field alongside a two-byte index ("<HB"):
TYPE_NULL 0 TYPE_RAW 1 TYPE_STR 2 TYPE_INT 3 TYPE_FLOAT 4 TYPE_BOOL 5 TYPE_JSON 7 (dict or list only)
WIRE_VERSION = 5 widened TYPE_INT from int32 to int64. A version mismatch is a hard failure, not a negotiation: v4 peers are rejected by _assert_version. An implementation MUST NOT attempt to interoperate across wire versions.
5. Engagement
Compression is not free and MUST NOT be unconditional. The decision is made per hop, about the next hop only, never about the final destination, and the default is the fast path everywhere. Semantic mode engages when the next hop measures slow, or when policy names it explicitly.
The measurement MUST be smoothed and the switch MUST be sticky: a running average rather than the last sample, distinct thresholds for entering and leaving semantic mode, and a minimum dwell time in a mode before it may change again. A link hovering near a single threshold would otherwise flap and spend more effort switching than either mode saves.
Verified in proxy/decision.py. Semantic-ness is evaluated at the next hop — the via if present, else the destination — never at the final destination:
ENTER_SEM_MS 250.0 enter semantic when EWMA RTT >= this EXIT_SEM_MS 200.0 leave when EWMA RTT <= this EWMA_ALPHA 0.25 smoothing; higher is more responsive HOLD_DOWN_SEC 30.0 minimum dwell after any mode switch PROBE_MIN_INTERVAL_SEC 2.0 maximum active probe rate per next hop PROBE_TIMEOUT_SEC 2.0 TCP probe timeout
EXIT_SEM_MS MUST be strictly below ENTER_SEM_MS; the gap is the hysteresis. A mode change MUST NOT be considered while inside the hold-down. Explicit policy — a next hop named in the semantic override address or network sets, or reachable as a via for any semantic subnet — overrides the RTT decision in both directions. The fast path is allowed across every interface; RTT governs semantic mode only.
6. Request identity
A template is useful only if the correct one is selected every time, so a request MUST be reduced to an identity. The URL is split in two: a small set of stable keys — the operation, the entity, the kind of thing asked for — which form the identity, and everything else, which is a value.
WORKED EXAMPLE — three URLs reduced to identity and values
/propogateNotification.php?event=abc
template: /propogateNotification.php
values: event
/api.php?entity=sensors&action=values&SensorName=X&parse=1
template: /api.php?entity=sensors&action=values
values: SensorName, parse
/frognet_echo.php
template: /frognet_echo.php
values: (none)
The identity is the shape of the answer, not the contents of it. Two consequences of that rule are normative because both were established by defect:
A doubled slash in a path MUST be collapsed before the URL is parsed. A standard parser reads a leading double slash as the start of a host name and returns an empty path; without the collapse, one endpoint reached two ways forks into two templates and compression halves with no visible cause.
Parameters that change how many rows return, or in what order, MUST be treated as values and MUST NOT enter the identity. They change the quantity and sequence of the answer, not its shape; keying on them produces a separate template for every value anyone ever asks for.
Where the body determines the shape, the body participates in the identity. A sensor upsert folds the sensor type and metric name into it, so every reading of that kind across the mesh shares one template regardless of which node emitted it. A previously unseen sensor therefore rides an existing template immediately: the template was learned for the shape, not for the sensor.
7. The first exchange
The first exchange of a given kind to a given peer has no template. It MUST be sent uncompressed, once, and both ends learn the shape from it. Learning the template and sending the exchange are the same act.
This path MUST NOT be fail-closed. An implementation that refuses a request for want of a template creates a trap with no exit: the first request fails, so no uncompressed request is ever sent, so no template is ever learned, so every subsequent request fails identically and permanently. One bootstrap costs the size of one message; refusing to pay it costs the endpoint forever. The answer to “I do not understand this yet” MUST be reduced compression, never a refusal.
8. The reference, and its three states
Each side keeps a reference per peer per exchange kind: the last set of values successfully exchanged. What goes on the wire is decided by comparing new values against that reference. The reference has three states, not two.
reference is UNSET never sent to this peer -> FULL reference is EMPTY sent before, no values, still none -> SAME reference has VALUES some of them changed -> DIFF
An implementation MUST distinguish UNSET from EMPTY. Collapsing them means every exchange carrying no varying data at all — the echo, the host list, the health probe — can never reach SAME, because “nothing to send” reads as “nothing sent yet.” Those are the highest-frequency exchanges on the mesh, and losing SAME on exactly them loses most of the benefit.
Values MUST be compared with a tolerance, so that a reading differing only in the low bits of a floating-point number is not a change.
Verified in core/codec.py: floats compare equal within 1e-9 absolute, applied uniformly — one tolerance, not one per field — and recursively through dicts and lists. Mixed int/float pairs compare as floats at the same tolerance; values of differing type are unequal without further comparison. LZ4 is applied only where the compressed output is smaller than the raw payload (_lz4_smart); LZ4 frame overhead is about 15 bytes, which on a fifty-byte reading is an expense rather than a saving.
The list-length threshold described in the build manual is not in force. DEFAULT_MAX_ITEMS = 64 and its accessor _max_items() exist in core/json_handler.py (overridable by FROGNET_JSON_MAX_ITEMS), but nothing in the tree calls the accessor. A collection of any length is therefore handled the same way. An implementation MUST NOT document a length-dependent behaviour that the code does not perform; the constant is dead until something calls it.
9. How SAME is proven
This is the mechanism most often misread as a cache. It is not one.
On a repeated request the far end MUST re-execute it against the real origin, obtain a fresh answer, and compare that answer to the one it sent last time. It MUST reply SAME only on genuine identity, and MUST NOT answer SAME from a stored copy. “Repeat” describes the request, never the work: it means the question is the same, not that the answer may be reused unexamined. A stale answer is therefore not possible, which is precisely what a cache cannot promise.
Comparison is over the canonical value, not the raw bytes. Two replies differing only in incidental formatting compare equal and cross as SAME.
The identifier returned with a SAME is an HMAC-SHA-256 token truncated to 16 bytes, keyed on a shared secret and computed over the request hash concatenated with the SHA-256 of the raw upstream response (core/semcache_id.py):
same_id = HMAC-SHA256( secret, req_hash || raw_hash )[:16]
secret /etc/frognet/sem_cache_secret, reloaded on mtime change
(rotatable without restarting the daemon or proxy)
req_hash 16 bytes on the wire, 32 internally
raw_hash SHA-256 of the raw upstream response
32 bytes, internal to the daemon, never on the wire
The token is verifiable (verify_same_id, constant-time compare) and is cryptographically bound to both the request and the response content. What that buys is that the two ends never disagree about which “last time” is meant, and that an identifier cannot outlive the content it names: change the data and the identifier changes with it.
An implementation MUST NOT operate without a secret in place. Where the file is absent the reference implementation falls back to a compiled-in default; that is a development convenience and MUST NOT be relied on, since with a known key the token is forgeable.
Two rules bound the mechanism. An error response MUST NOT be recorded as a reference: a failed response is not a legitimate thing to later call unchanged. And the reference MUST be advanced only after the far side has accepted the exchange — if the answering node failed, neither side moves and both still refer to the same prior state.
10. Twenty-one bytes
A repeat of an unchanged request for an unchanged resource is twenty-one bytes out and twenty-one bytes back: four of magic, one of opcode, sixteen of identifier, each way, in place of a full HTTP request and a full JSON reply. RESP_SAME is the smallest frame the protocol defines and a conforming implementation MUST NOT emit a shorter one.
It is worth being exact about why, because the obvious reading is wrong. Nothing is being squeezed. There is no coding scheme doing the work. The body is not sent because both ends already hold it and can prove cheaply that they hold the same one. Twenty-one bytes is not a small payload; it is the absence of one.
Compression ratio is therefore the wrong measure of this mechanism, and no ratio is claimed for the codec itself (§14). The right observation is that most traffic on most networks is repetition, and repetition can be named instead of transmitted.
11. Sequencing, fan-in and fan-out
A node does not open a socket per request. Concurrent exchanges toward one peer fan in to that peer's single established socket; the writer sends frames one at a time and MUST NOT wait for a reply before sending the next.
A request frame carries no sequence number, and the obvious mental model is wrong on this point. The sender assigns a number locally and files the waiting caller under it; the receiver assigns its own number from its own arrival counter and stamps that onto the reply. The two agree because they are counting the same events in the same order on the same socket, not because the number crossed the wire.
Because two independent counters can drift, SEQ_RESET exists to put them back in step. An implementation MUST provide it; the need for it is the direct consequence of the number not riding the request.
writer -> seq 41 REQ_FULL (dashboard poll)
writer -> seq 42 REQ_REPEAT (sensor read)
writer -> seq 43 REQ_FULL (file list)
pending = { 41:dashboard, 42:sensor, 43:filelist }
reader <- seq 42 RESP_SAME -> wake the sensor caller (first back)
reader <- seq 43 RESP_DIFF -> file-list caller
reader <- seq 41 RESP_DIFF -> dashboard caller
Replies arrive interleaved and MAY complete out of order; a small SAME may return before a larger body. An implementation MUST NOT hold a completed reply behind an outstanding one.
12. Collapsing duplicate work
Two distinct collapses operate, at different places, protecting different scarce resources. They are not the same mechanism and neither substitutes for the other.
| Side | Keyed on | Effect | Protects |
|---|---|---|---|
| Sending | request and peer | Several callers wanting the identical thing from the identical peer become one frame. The first caller owns the exchange; the others wait and receive the same bytes. | The link |
| Receiving | request alone, not the asker | Three nodes asking the same question in the same instant cost the answering node one origin execution among them, not three. | The origin |
Two rules keep the mechanism honest. Health probes MUST NOT be handed another caller's result: a probe given somebody else's answer has measured nothing. And a waiter that has waited too long MUST proceed to do the work itself — collapsing duplicate effort may cost duplicate effort, but it MUST NOT cost a request.
Probes MUST NOT be exempted from the collapse on the grounds of being cheap. This was learned by removing it: with probes exempt, an unreachable downstream node gave every probe its own timeout slot, the pending queue grew past four hundred, and one unreachable box became a fleet-wide failure. Exempting the highest-frequency traffic from the mechanism that bounds concurrency is how a local outage becomes a general one.
13. Change propagation
Topology change notification is carried as ordinary FNWP-1 traffic and MUST be bounded three ways. All three are required.
1 every wave carries an identity a node that has already seen it drops it, silently. this is what makes the epidemic terminate. 2 a node notifies only its DIRECT neighbours LAN next-hops and tunnel peers, not everything it knows. fan-out per node: O(neighbours), not O(fleet). re-propagation from each receiver still reaches everyone. 3 arrivals are debounced a burst collapses into one reconciliation, not one per message.
A node MUST skip all of its own addresses when notifying, not merely its primary one. A FrogNet host appears under several — tunnel endpoints, secondary aliases, transit addresses — and skipping only the obvious one leaves the rest eligible, so the node notifies itself repeatedly, each self-notification starting a process whose only work is to notice it has already seen the message.
A wave carries a change notification, never a value. Tuple values do not gossip; there is only one copy of them (§2a). Because notifications are ordinary requests they pass through the codec like everything else, and the only field that differs between successive waves is the wave identity — a value, not part of the template identity — so after the first notification to a peer, each subsequent one carries a single changed field.
14. What each side retains
Every node carries a local database. It serves two unrelated roles, and conflating them is the error that produces every “FrogNet replicates its memory” reading.
| Role | Present on | Holds |
|---|---|---|
| Semantic store | Every node, always | Templates and per-peer references. Nothing to do with FrogNet Memory. |
| FrogNet Memory backing store | The elected database host only, while it holds the role (§8) | The tuple space (§9). A node acquires this role at election and loses it at the next one, including after a merge. |
The same database instance serves both. An implementation MUST NOT infer from the presence of a database on every node that the tuple space is present on every node: it is not, and being elected is what adds it.
14.1 The semantic store has three layers, at three different scopes
The scope of each is normative, because it decides what has to agree with what.
| Layer | Keyed on | Scope |
|---|---|---|
Templatesfrognet_request_templates, frognet_response_templates | templateId alone | Machine-independent. A template describes the shape of an exchange, not a relationship with a peer. One template serves every peer, and a template learned against one peer is immediately usable with another (§11.6). |
ReferencesSemCacheProxyRefs | (peer address, opcode, req|resp) | Per peer, per direction. The last values successfully exchanged with that peer. This is what a DIFF is computed against and applied to. |
Response bodiesSemCacheProxy | SAME identifier | Local. What a SAME resolves to without touching disk on either side. |
References are the only layer that must agree across the wire, and they must agree pairwise. Two nodes exchanging traffic hold one reference each for that pairing, and the two rows must describe the same last exchange or a DIFF applied at the far end reconstructs the wrong value.
The agreement is maintained by the rule already stated in §11.9: a reference advances only after the far side has accepted the exchange. Neither side moves unilaterally, so a pair cannot drift by one. Where they have drifted anyway — a restart, a lost frame, a peer that never held the reference — the holder answers REQ_MISS (§11.3) and the exchange falls back to FULL, which re-establishes both sides from one message. An implementation MUST provide that path: without it a single divergence is permanent.
Templates need no such agreement. Being machine-independent, a template the far end lacks is learned on the next FULL, and two nodes holding different template sets are not in an error state.
[NEG_CACHE_TTL_V1] A negative lookup — “no template for this opcode yet” — MUST be cached only briefly, and the window MUST be shorter than one user-visible exchange. Otherwise a node that asked before the template was written keeps answering “no template” from its own negative cache after the template exists.
The store is local: it is reached on the loopback address, and the semantic layer creates its own tables rather than relying on an installer, because a plain proxy node never runs the database-host install and would otherwise have nowhere to learn a template.
That the meaning lives in this local store and not on the wire is why a tap on the link yields little: the wire carries references and differences, not messages (§12).
15. Non-goals
Not a cache (§11.9). Not a compression product: the saving is structural rather than statistical, and no ratio is claimed for the codec (§11.10, §14). Not encryption: the SAME identifier is an authentication tag, not a cipher. It proves that a party holding the secret asserted this response for this request; it conceals nothing. Confidentiality on the link is WireGuard's (§7, §12). Not content-negotiated: body handling is selected by inspection of the body, not by a declared Content-Type, into one of five modes (json, xml, html, text, raw); a body classified raw learns no template and every send of it is FULL (§17.7).
16. References
Specification §2, §9, §10, §12 (Security), §14 (Performance), §17. Source: core/semcache_wire.py, core/codec.py. Build manual: Croakus §26, §27, §28, Appendix F (Inside the Compression Engine), Appendix G (Inside a Handler). Measured results: the simulator and §14.