Case study · PyTorch

The data comes to the compute.

PyTorch distributed, converted to CroakOS in two steps. The first kept DDP’s arrangement and reached parity with gloo. The second rewrote the core algorithm as a multithreaded program on one computer, and trained the same model to the same loss 329 times faster.

batch 64, 40 steps
329×

Stock DDP with gloo took 1,018 s (ddp4430); meshfeed took 3.09 s (mf500). Held-out loss 0.4947 for both; mf500 matches a single-thread oracle to 6.0e-08.

batch 4096, 10 steps
5.5×

Stock DDP with gloo took 264 s (mf506); meshfeed took 48.2 s (mf503). Held-out loss 0.6916 for both; mf503 matches the oracle exactly.

Total time for the stated steps on the same five-node mesh (Seattle, New York, Wi-Fi, WireGuard and home Internet links), training the same model to the same held-out loss. The batch-64 DDP control ran in a separate session the same morning; the batch-4096 control ran in the same campaign. Report: PSYCHEDELIC-MESH-RESULTS-20261113.md, section 9.

Step one · September

The backend: the same contract, parity with gloo.

torch_backend.py and psychedelic_backend.py are a registered c10d backend with all fourteen collectives. DDP runs over it unchanged: every rank computes on its own data and the gradients are summed through shared memory instead of gloo’s ring.

campaignresult
four September campaigns, four ranks, Seattle–New York1.03× to 1.95× the speed of the same campaign’s gloo control
five-node mesh, 12–13 November, after Seattle5’s access point queue was changed to fq_codeltied with gloo: within 2% at batch 64 and 3% at batch 4096

Both sit near the floor the mesh’s links set for any gradient exchange: about 18 s a step to move 16.9 MB of gradient. Under DDP’s arrangement, no backend can go faster. Matched audit: the backend is 66% less code and 47% less complexity than gloo’s stack (ProcessGroupGloo, gloo and TCPStore; 3,621 against 10,658 NLOC). Audit

Step two · November

The rewrite: meshfeed.

The answer DDP computes needs only the sum of every node’s gradient, and a gradient can be computed wherever a node’s data can be read. A node’s minibatch is 4,352 bytes; its gradient is 16.9 MB.

Placement

Each node keeps its minibatches on its own disk and publishes them through the tuple space. The machine that measures fastest reads them and does the arithmetic with ordinary, unmodified PyTorch. The bits stay where they are until the compute needs them.

Compute sets

Splitting the batch-4096 compute over two or three machines (mf504, mf505) beat DDP by 2.6× and 2.2×, but lost to the single fastest machine: the gradient sum among them cost more than the compute it saved.

Size

meshfeed is 323 NLOC against 14,360 for DDP over gloo: 98% less code, 96% less complexity in the matched audit.

Reproduce: sh meshrun.sh $(/usr/local/bin/getEth0Address) <base> on each of the five nodes (report section 13). meshfeed.py · meshrun.sh

Why the rewrite is where the gains are

Parity first, then stop converting.

The conversion got PyTorch to a tie with gloo on links where any gradient exchange costs 18 s a step. That is as far as the old arrangement goes. Rewriting the core algorithm as a multithreaded program on one computer, with shared memory as RAM and held reads as the scheduler, changed the arrangement inside what the contract allows, and the time fell from minutes to seconds. The method, step 5e.

The code

Everything that produced these results is in the tree.

pathwhat it is
opt/frognet_semantic/agent_workload/tuplespacethe backend, the DDP hooks, the tensor plane, the c10d store on shared memory
usr/local/frognet-aimeshfeed, meshrun.sh, ddpbench.py and the campaign harness
opt/frognet_semantic/ribbit/examples/pytorch/README.mdhow to run a campaign, the correctness contract, the September tables
opt/frognet_semantic/ribbit/examples/pytorch/evidence/2026-11-15the matched audit: every file and function counted
The conversion assessment

Bring your system, its contract and its tests.

I apply the method on this site to your system and tell you what a conversion would take. You get a written assessment containing:

It is the architecture review on the rates table, applied to a conversion. The method behind it is on Converting a system; the two worked examples are PyTorch and lispers.net.