Evaluation of a Distributed Search's Design

Requirements compliance

Evaluate query serving, index freshness, and recovery separately. A system can keep answering queries while its index falls behind.

Availability

We use distributed storage to persist:

• Source documents collected by the crawler or ingestion process.

• Completed inverted-index generations produced by indexers.

Distributed storage replicates data across regions, which supports cross-region deployment and disaster recovery. Clusters of indexing and search nodes are deployed across multiple Availability Zones (AZs). If an Availability Zone fails, traffic is redirected to a healthy cluster. Within each cluster, redundant nodes provide failover capability.

Indexing is outside the query’s critical path in this design. Search can continue using the last published generation while indexing or replication is delayed. That protects query availability at the cost of freshness; it does not remove replication delay or guarantee that a recently changed document is visible.

Note: Define when a generation is ready to serve and whether a query may combine shards at different generations. Do not claim a globally current view unless the publication protocol provides one.