ProjectsCase study
HoldFast
Replication that keeps edge nodes in agreement while the network between them drops packets or cuts out.
The problem
Three computers at the edge of a network each receive position reports for the aircraft in their own area. Together they have to hold one shared picture of every aircraft, and keep holding it while the links between them drop packets, cut out, or a node restarts.
The usual answer is a central server that everyone reports to, and when that server is out of reach, nothing moves. HoldFast has no coordinator. Every node keeps working through a cut, and the nodes come back into agreement when the links return.
How it works
Each node keeps a map from aircraft to the latest report, stamped with a hybrid logical clock. A merge keeps, for each aircraft, the report that sorts highest by stamp, then origin, then contents. That merge is commutative, associative and idempotent, so the map is a CRDT: replicas that have seen the same reports end up identical, whatever order the messages arrive in and however often they are lost or repeated.
Nodes sync peer to peer by anti-entropy over plain HTTP: send a digest, get back what you are missing, send what the peer is missing. A lost exchange is never retried. The next round resends it, which is safe because every exchange can be repeated without changing the result.
The clock is the hard part. If one node's clock runs fast, a plain wall-clock map throws away fresh reports in favor of older ones stamped in the future. A hybrid logical clock stays close to real time, but a node's next write always sorts after everything it has already seen.
The cloud peer runs the node's own code in a Go Lambda behind API Gateway, with the picture in DynamoDB. Each request rebuilds a node from the table, runs its handler, and writes back with a conditional put that only lets a greater track in, so concurrent invocations and retried writes can only move an item up. Terraform deploys it with nothing provisioned, so it costs nothing while idle.
Reports from the ADS-B feed, either live from adsb.lol or the recorded hour, go into three edge nodes, a, b and c; a's area overlaps both others, and b's and c's don't overlap. Every pair of edges syncs by anti-entropy while tc netem drops packets or cuts links between them. The edges can also relay through an optional cloud peer on AWS, reached with a bearer token over https only: API Gateway, then a Go Lambda running the same node code, then DynamoDB with conditional writes. Off to the side, a harness polls every node's state every 100 ms, and Prometheus and Grafana watch the nodes; neither is in the data path.
How the pieces connect. The cloud peer is one more node, not a coordinator.
Drawn from holdfast README, How it works (its mermaid diagram) and Setup, at c9a083c.
What was measured
- 1.37 s
- p95 for a report to reach another node under 20% packet loss, replaying a recorded hour of real traffic with 444 aircraftSource: holdfast README, Results (results/real-hour/summary.csv, commit 4848211), read at c9a083c
- 60 of 60
- windows of that hour converged under 20% loss, and again after node c was cut off for 23.6 sSource: holdfast README, Results (results/real-hour/summary.csv, commit 4848211), read at c9a083c
- 1.4 s
- for every node to hold everything again after that cut, in all 60 windows. Timed in merge order, a measure added after the predictions.Source: holdfast README, Results, first bullet, read at c9a083c
- 0
- writes the hybrid logical clock lost with node c's clock 4 s fast, where a wall clock lost 485 over the same 8 windowsSource: holdfast README, Results, Clocks (results/real-base/summary.csv, commit 4848211), read at c9a083c
- 2.8x
- the p95 cost of relaying through a central hub instead of the mesh, with no loss, over 8 windowsSource: holdfast README, Results, central-server baseline (results/real-base/summary.csv, commit 4848211), read at c9a083c
- 2.1 s
- p95 through API Gateway, Lambda and DynamoDB in one measured 10-minute run on AWS, at night with about a dozen aircraft: 1,758 invocations, 0 errorsSource: holdfast README, Deployed on AWS (results/aws/summary.csv and aws-run.json), read at c9a083c
A partition on the synthetic trace. While node c is cut off, every node falls behind by a second per second. About 1.2 s after the heal, all three are caught up.
Source: holdfast docs/partition-timeline.png at c9a083c, converted to monochrome. Tap or click the figure to open it full size.
Clocks on the recorded hour. With node c's clock 4 s fast, a wall-clock map lost 485 writes over 8 windows; the hybrid logical clock lost none.
Source: holdfast docs/real-clock.png at c9a083c, converted to monochrome. Tap or click the figure to open it full size.
How it was checked
Property tests
The merge is checked to be commutative, associative and idempotent, and the order behind it to be a total order, over 2,000 generated cases in the gate. Replicas that exchange digests are checked to converge.
Seeded fault simulation
Three or four nodes run in one goroutine on fake clocks, and every fault is drawn from a seed: lost, delayed, duplicated and cut-off messages, partitions, clock jumps both ways, and restarts that wipe a node. The invariants are checked after every step, and a failure replays exactly from its seed. It found 2 bugs the measured runs missed: a restarted node reusing its old stamps, and expired tracks coming back after a clock stepped back.
Mutation testing
131 planted bugs, in the clock, the map, the node, the edge, the cloud peer, the feed reader and the harness's own arithmetic. The gate first checks that the unmutated tests pass, then fails if any planted bug survives.
Fuzzing and static checks
Both sync handlers and the ADS-B parser are fuzzed. The gate also runs the race detector, staticcheck, govulncheck, and a Trivy scan of both container images.
The cloud peer
It is tested against DynamoDB Local with concurrent writes, corrupt items, an instance whose clock runs behind another's, and a store failure partway through a request. The Terraform has plan tests of its own and a Trivy scan.
Results that can't drift
Predictions were committed to git before the runs that grade them. Raw polls are saved, and the recorded hour's summaries rebuild from them byte for byte, so fixing a metric means recomputing, never a quiet rerun. A test fails if the README's results tables stop matching the data.
One gate
97 Go tests and 2 fuzz targets, 51 harness tests and 6 Terraform plan tests, all run by one script locally and by CI on every push.
What I would do next
Everything here ran on one Docker host with three edges, under packet loss and cuts, but no added latency or bandwidth cap. Adding netem delay and rate limits is the next experiment. The edges don't authenticate each other yet, so node identity, with mutual TLS and short-lived certificates, comes after that. And last-writer-wins keeps the latest write, not the best estimate: replicating each sensor's own tracks and combining them when the picture is read would give every node the same estimate, while the replicated state stays a CRDT.