Summary. TierStore keeps configuration reads local for globally distributed search: persistent replicas beside each service, and one ordered write log per namespace.
A search request shouldn’t have to cross an ocean to find out how it should be handled.
That sounds obvious for the search index. It’s easier to overlook the configuration around it. A service can run close to the caller and still depend on a distant database to decide where to route a request or which configuration to use.
At SuggestAPI, we built TierStore for that part of globally distributed search: getting configuration and coordination data to the services that need it, without making each read a remote operation.
TierStore isn’t our search index. It doesn’t replace a customer’s catalog or search engine. It’s a narrow key-value store with an ordered write log for each namespace and full, persistent replicas beside the services that read them.
We distribute changes globally so reads can stay local.
The small data behind a search request
SuggestAPI sits in front of search infrastructure. We route requests to configured backends and return a consistent response, while the underlying search and catalog can stay where they are. Configuration is therefore part of the request path, even though it isn’t the content being searched. [1]
Consider an example: an index mapping changes from one backend to another. The mapping is small. It may be read repeatedly, while changes are relatively infrequent. The difficult part isn’t querying that mapping. It’s making a known version available wherever requests are handled.
This is the workload TierStore is designed for. Reads should stay local, changes need an order, and a restart shouldn’t discard the entire local state. Each replica also needs to know how much of the write stream it has applied.
For this workload, ordering within a namespace is enough. We don’t need independent, conflicting writes to the same namespace in every region, or a global transaction across unrelated namespaces. We need a clear relationship between a committed change and the replicas that eventually observe it.
We built the distribution layer, not a storage engine
TierStore uses MDBX, an embedded, memory-mapped key-value engine. The custom work sits around it: sequencing, replay, replica application, snapshots, and the interfaces our services use.
A database endpoint in each region reduces distance, but still makes configuration access a service call. An in-process replica removes that call. For a Rust service, a TierStore read is a local library operation. Other languages can use a sidecar in the same pod, paying local transport overhead without depending on a remote region.
Existing systems can implement parts of this design. We evaluated storage engines, sidecars, and replication strategies rather than assuming custom code would win. NATS is a credible messaging option; the architecture also identifies JetStream as an alternative to building out a custom sequencer.
What we wanted to own was the read and recovery model across our deployments. The same downstream replicas can work with a write log hosted on Kubernetes or, in the planned variant, at the Edge. Services reading configuration shouldn’t need a different architecture for each.
Cloudflare’s Quicksilver was the main design reference: distribute control data ahead of requests rather than fetching it remotely while handling them. Its evolution also shows how much consistency work appears when full replicas become partial caches. [2] [3]
Full replicas before clever caches
TierStore keeps the full dataset at each replica by default. There is no eviction in the normal read path.
That choice depends on a bounded configuration workload. In our synthetic one-million-key bootstrap case, with small values, a file snapshot was about 132 MB. That’s a different storage problem from replicating a large product catalog, and it doesn’t imply that copying everything everywhere will scale indefinitely.
A full replica also answers missing-key lookups locally. It doesn’t have to distinguish a key that doesn’t exist from one that was evicted, then ask another tier.
We explored partial caches in the benchmark harness. They introduced cache fills racing replicated changes, older deliveries arriving after eviction, and reads from replicas at different log positions. MDBX also struggled with heavy cache churn compared with its role as a full-copy replica.
For the current design, we prefer replicas that write when configuration changes arrive, not whenever a cache miss needs filling. If the dataset makes full copies impractical later, a caching tier becomes a separate decision with its own consistency rules.
One write order, two levels of fan-out
Each namespace has a home sequencer. It assigns consecutive sequence numbers to accepted operations and commits the current state and retained log together. In the implemented Kubernetes variant, that boundary is a durable local MDBX transaction.
Writes can be grouped for commit without losing client batch boundaries. Conditional writes are checked at the sequencer, and a batch with a failed version check changes nothing.
The log then flows through regional relays:
Namespace home sequencer
|
| ordered replication stream
v
Regional relay
|
+--> Service with an embedded replica
|
+--> Replica sidecar for another service
|
+--> Regional JSON-RPC API with a local replicaThe sequencer sends a change once per region. Each relay applies it to its own MDBX store and distributes it to the replicas in that region. Adding a reader doesn’t require another cross-region subscription directly to the home sequencer.
Different namespaces can have different home clusters. Unrelated configuration changes don’t have to share one global queue.
A replica commits a complete batch and its applied sequence number in the same transaction. That persisted sequence number is its watermark, the position from which it resumes after a restart. The replica cannot commit its progress independently of the data that progress describes.
A write acknowledgment isn’t global visibility
Replication is asynchronous. A successful write means the sequencer committed it, not that every region has applied it.
Normal reads return locally applied state. A region that falls behind can return an older configuration than one that has caught up. Locality doesn’t remove that trade-off.
For callers that need read-your-writes, the write response includes a sequence number. Passing it as minSeq to a read makes the target replica wait, with a timeout, until its watermark reaches that position. It then performs a fresh read.
The replication protocol carries an epoch as well as a sequence number. The epoch fences an obsolete namespace home after promotion. The current public minSeq token is sequence-only, so its read-your-writes contract applies within an unchanged epoch; failover-safe, epoch-qualified client tokens remain work to do.
Deletes are retained as tombstones. Duplicate entries are dropped, gaps trigger replay, and entries from an older epoch are rejected. These mechanisms make asynchronous replication’s order and progress explicit. They don’t turn TierStore into a globally linearizable database or a distributed lock service.
Local reads still needed careful implementation
Choosing an embedded engine didn’t finish the read path. Transaction management mattered.
In the saved eight-thread, 10,000-key microbenchmark, held MDBX read transactions reached a p99 of 0.21 microseconds. Opening a new transaction for every read performed much worse under concurrency.
| Local storage microbenchmark | Result | Scope |
|---|---|---|
| Held MDBX read transaction | 0.21 µs p99 | 10,000 keys, eight threads, in process |
| Held MDBX read throughput | 32.45–33.58M ops/s | Same microbenchmark; renewed every 1,024 reads |
The implementation reuses read transactions for up to 1,024 reads or 5 ms, and refreshes them when a newer local commit generation requires it. A minSeq read uses a fresh transaction after waiting. The goal is to avoid repeated setup without retaining an old snapshot after the local store advances.
The 0.21-microsecond number is an in-process storage measurement. It isn’t SuggestAPI search latency, sidecar latency, or a head-to-head comparison with a managed API.
The architectural result is more useful than a performance multiplier: the hot configuration read can avoid network I/O entirely.
Recovery belongs in the protocol
A replica with its local volume intact opens MDBX, reads its watermark, and asks for entries starting at the next sequence. If that history has been trimmed, the sender returns TooOld. The replica installs a snapshot, then resumes the log.
New replicas can bootstrap from published snapshots, with peer snapshots as a fallback. Regional relays retain logs and publish snapshots to object storage, so the sequencer doesn’t have to send the full dataset individually to every new reader.
We also validate inputs before they enter the log and again on replica application, including our 512-byte key and 1 MiB value limits. A replica that encounters an invalid entry halts and reports the error. Silently skipping it would let its watermark claim progress its data doesn’t represent.
Process health and freshness need separate treatment. Killing a healthy replica because it’s catching up can create a restart loop. We expose replication positions and connection state separately from liveness.
A healthy local replica can keep serving its last applied state while disconnected. Whether that state is acceptable is an application policy, not a promise made by a readiness response. Configuration that can tolerate lag is different from a decision requiring current authoritative state.
What the benchmarks showed
We started with the benchmark harness, then built the reusable store and services, followed by container images, Compose, Helm packaging, and push transport at the Edge. Measurements informed the design before it became deployable components.
The end-to-end harness includes namespace sequencers, regional relays, and MDBX replicas. Its most useful results are below.
| Scenario | All-replica visibility p99 | Modeled network floor | What completed |
|---|---|---|---|
| 300 simulated locations, six simulated regions, 50 writes/s | 129.9 ms | 128.2 ms | 1,000 measured writes; all completed the visibility check |
| 60 simulated locations, six simulated regions, 1,000 writes/s | 309.5 ms | 129.5 ms | 20,000 measured writes; all completed the visibility check |
The 50-writes/s case was roughly 1.6 ms above the modeled floor. The higher-load run took about 180.0 ms above it. Load mattered, and the smaller 60-location topology at 1,000 writes/s prevents us from treating the low-rate number as a universal guarantee.
The recovery results were also measured through the local harness:
| Recovery scenario | Data transferred | Time to ready | Validation |
|---|---|---|---|
| New replica, 1M keys, file snapshot plus log tail | 131.7 MB | 1.144 s | Zero key mismatches after drain |
| Existing replica, local data retained after 10 s down | 0.24 MB | 0.087 s | Zero key mismatches after drain |
Those runs used a modeled 1 Gbit/s link. They give us a recovery baseline, not a promise about real object storage, WAN paths, packet loss, or Kubernetes rollouts. Those still need their own measurements.
What we’ve built, and what remains
We now have runnable Rust services for the sequencer, relay, replica sidecar, and JSON-RPC API, plus shared store and protocol libraries. Local integration tests cover conditional writes, atomic batches, read-your-writes, watches, restart and replay, snapshot bootstrap, and TooOld recovery.
The Kubernetes sequencer is currently single-leader. Raft high availability is planned behind a LogCommitter interface, not implemented. Manual regional failover can lose writes acknowledged by the old home but not yet replicated. Local durability isn’t zero-loss regional durability.
At the Edge, we’ve built the relay’s push receiver and deployment configuration. The intended design sends each namespace’s log out to every region.
Building a distributed store means taking responsibility for ordering, recovery, and failure behavior. We chose to own that narrow problem because local configuration reads matter to our globally distributed search service. Getting a request to a nearby server shouldn’t be followed by a trip to a distant configuration service.