Pith. sign in

REVIEW 4 major objections 5 minor 25 references

Implementing Decentralized Per-Partition Automatic Failover in Azure Cosmos DB

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Azure Cosmos DB's new per-partition automatic failover restores write availability in under two minutes for every partition-set in a 4,300-partition regional power-outage test.

desk verdict An unusually candid systems paper on per-partition geo-failover in Cosmos DB; the design is plausible and the two-minute RTO is supported by real outage tests, but the shared acceptor-store dependency and missing artifacts keep it from being fully convincing. read the letter →

arxiv 2505.14900 v1 pith:KVOQRN46 submitted 2025-05-20 cs.DB

classification cs.DB
keywords per-partitionfailovergeo-failoverCASPaxosstatemachinereplicationAzureCosmosDBrecoverytimeobjectiveduelingproposersadaptivebackoff
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper describes a new mechanism in Azure Cosmos DB that lets each partition-set decide its own write region when the current one fails, instead of failing over an entire account through a central control plane. The authors claim this decentralized design, driven by a deterministic state machine called the Failover Manager and backed by a CAS Paxos store, restores write availability within two minutes of an outage. They report three thirty-minute power-outage tests on a 3-region account with 4,300+ write-region partitions in which every partition-set recovered in under two minutes. They also report that an adaptive scheduling scheme for CAS Paxos proposers cuts proposer failure rates to at most 0.0028%, even under high contention. If these results hold, the system meets its RTO target at scale while preserving customer-chosen consistency levels and recovery point objectives.

What carries the argument

The Failover Manager is a deterministic, formally specified (TLA+/TLC-verified) state machine, one per partition-set, that stores the partition-set's write region, read-lease set, and progress markers and admits compare-and-swap transitions proposed by any replica. State is persisted in a leaderless CAS Paxos replicated state machine without a write-ahead log, with acceptor state held in geographically distributed non-cross-partition-replicated Cosmos DB accounts. A heartbeat every 30 seconds triggers transitions; ungraceful failover waits for a quorum of regions to report, then picks the highest-priority region with the highest reported progress, while graceful failover quiesces writes, drains replication, and switches to the preferred region. An adaptive backoff and time-division multiplexing scheduler tunes CAS Paxos retry delays from measured Phase 2 durations to suppress dueling proposers.

What would settle it

A concrete test: take down the write region and, at the same time, take down a majority of the CAS Paxos acceptor-store regions; if partition-sets still restore write availability within two minutes, the acceptor dependency is not the binding constraint, and if they do not, the two-minute RTO claim is conditional on acceptor-store placement.

Watch

Extended reading notes

Core claim

The paper's central claim is that geo-failover in Cosmos DB can be made fine-grained and autonomous: each partition-set runs its own Failover Manager state machine, and replicas propose state transitions through CAS Paxos rounds. The state machine encodes the current write region, the set of read leases, and progress information, and transitions are triggered by partition heartbeats rather than by an operator or a control-plane workflow. The paper asserts that this removes the control plane as a scaling bottleneck and single point of failure, that the formally specified state machine guarantees availability is eventually restored and consistency levels are respected, and that the implementation achieves the stated RTO: availability restored in less than two minutes for every partition-set in the tested regional outage, with automatic failback to the preferred region after power returns.

Load-bearing premise

The failover decision's quorum lives in ordinary non-cross-partition-replicated Cosmos DB accounts, so if those acceptor stores fail or become unreachable at the same time as the data-plane region, the quorum needed to elect a new write region may be gone.

Editorial extensions

If this is right

  • Region-wide write availability recovers in under two minutes for every partition-set, so an outage of one data center no longer forces account-wide failover of healthy partitions.
  • Failover decisions scale with the number of backend partitions rather than with control-plane capacity, because each partition-set runs its own state machine on the machines that host it.
  • Consistency levels and recovery point objectives remain honored: ungraceful failover picks the region with the highest reported progress, and dynamic read-lease quorums let a two-region account continue with one region at a configured minimum durability.
  • Failed-over partition-sets automatically fail back to the user's preferred region once it recovers, using a progress table to discard only the 'false progress' written during the outage.
  • The adaptive CAS Paxos scheduler keeps proposer failure rates below 0.003% even with nine concurrent proposers, making leaderless consensus practical for the heartbeat workload.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence of the design is that failover RTO depends on the geographic placement of the acceptor stores; placing a majority of them in the same region as the write region could violate the two-minute target, so operators should treat acceptor-store placement as a first-class availability decision.
  • The same state-machine-plus-CAS-Paxos pattern could be applied to other geo-distributed systems that currently rely on a control plane for failover, provided they can tolerate the extra heartbeat load and the acceptor-store dependency.
  • The paper's distinction between graceful and ungraceful failover suggests a broader design principle: any failover system that can distinguish planned from unplanned transitions can use the planned path to bound data loss and the unplanned path to bound downtime.
  • Because the client SDK now uses a DNS TXT record and per-partition-set error caches instead of DNS updates, client behavior during failover becomes testable independently of DNS TTL misconfigurations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes the design and implementation of per-partition automatic failover in Azure Cosmos DB, replacing account-level geo-failover with a decentralized state machine per partition-set. The core components are a Failover Manager executing a deterministic state machine, a CAS Paxos store for persisting state machine data, heartbeat-based failure detection, graceful and ungraceful failover modes, and SDK changes to avoid dependence on DNS updates. The paper claims that write availability is restored within two minutes based on three power-outage experiments on a 3-region account with 4,300+ write-region partitions, and that an adaptive scheduling and time-division multiplexing approach reduces CAS Paxos proposer failure rates to at most 0.0028% in simulation.

Significance. If the claims are correct, this work addresses a genuine scalability and availability limitation of control-plane-coordinated geo-failover in a large cloud database, and it provides a detailed architectural blueprint for decentralized per-partition failover that could inform other geo-replicated systems. The paper's strengths include a concrete description of integration with existing Cosmos DB replication and control-plane workflows, a plausible solution to dueling proposers in CAS Paxos, and real outage tests at a non-trivial scale. However, the experimental evidence is narrow: the RTO claim rests on three runs on a single account, and the CAS Paxos evaluation is a simulation without confidence intervals. The paper also explicitly acknowledges a degenerate behavior that it says will be fixed only in the future, which limits the strength of the availability guarantee as presented.

major comments (4)
  1. [Section 6.1.3] The central RTO claim, stated as 'availability is restored within less than 2 minutes for every partition-set', is supported only by three power-outage runs on a single 3-region account with 4,300+ write-region partitions. The paper reports no raw recovery-time distributions, no confidence intervals, and no breakdown by partition-set. It also does not report acceptor-store latency, throttling, or CAS-conflict rates during the failover burst. Because the abstract claims the solution works 'at any scale', the experiment as presented does not establish that the RTO holds when many partition-sets across many accounts fail over concurrently and share the same acceptor-store capacity (Section 4.3.1). The authors should provide per-partition recovery-time statistics, acceptor-store load metrics under the tested outage, and either a larger-scale test or a validated model of fleet-wide failover load.
  2. [Section 6.2.3] The claimed reduction to a maximum 0.0028% proposer failure rate comes from a discrete-event simulation that is not sufficiently specified. No confidence intervals or variance measures are reported, the network latency distributions are described only as 'randomly assigned' and 'heterogeneous', and the simulation models 3 to 9 proposers per partition-set with seven acceptors but does not model contention from many partition-sets sharing the same acceptor stores during a regional outage. Since the RTO guarantee depends on CAS Paxos rounds completing within the 45-second lease-enforcer timeout, the simulation should include a model of shared acceptor-store load, or the authors should explicitly scope the claim to per-partition contention only.
  3. [Section 4.5] The paper acknowledges a degenerate behavior in the state machine: a graceful failover can succeed, then the destination region can fail, triggering an ungraceful failover, and the cycle can repeat, potentially causing a continuous outage. The text states 'We will amend our implementation to account for this', which indicates the fix is not part of the current implementation described in the paper. This is a load-bearing gap because the paper's central claim is that availability is restored within a bounded time. The authors should either present the fix as part of the implemented design and verify it, or explicitly state that the published system does not yet guarantee eventual availability in this scenario and that this is a known limitation.
  4. [Section 4.3.1] The design stores CAS Paxos acceptor state in a set of geographically distributed non-replicated Cosmos DB accounts that back 'all partitions globally'. The paper argues that the dependency on Cosmos DB is not circular, but the dependency remains a real availability risk: if these acceptor stores fail or throttle under load, quorum-based failover cannot proceed. The outage experiment exercises only one account's partitions and therefore does not test the shared acceptor-store dependency under the burst that would occur in a regional outage affecting many accounts. The paper should include an analysis or experiment that addresses the failure independence and load capacity of the acceptor stores, or it should qualify the RTO claim to scenarios where the acceptor stores are available and unthrottled.
minor comments (5)
  1. [Section 6.1] The subsection numbering is inconsistent: there are two subsections labeled 6.1.1 and two labeled 6.2.3. The sections should be renumbered for clarity.
  2. [Figures 7 and 8] Figure 7 and Figure 8 lack explicit axis labels and units. 'Time' should be labeled as seconds or minutes, and the y-axis should indicate the fraction or count of partition-sets.
  3. [Section 4.4] The TLA+ verification is described only through a short invariant snippet. No model-checking results, TLA+ specification files, or list of checked properties are included, making the verification claim difficult to assess. If the authors want the verification to be a credible contribution, they should provide more detail or artifacts.
  4. [Section 4.5] The phrase 'We will amend our implementation to account for this' should be changed to 'We amended' or explicitly moved to the Future Work section; as written, it reads as an admission that the current implementation does not handle the described scenario.
  5. [Section 6.2.2] The text says the simulator can 'compress years of system operation', but each experiment simulates one hour of operational time, with 10,000 simulations totaling about 1.14 years. This sentence should be reworded to avoid implying that a single simulation spans years.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the failover design is evaluated by external outage tests, an independent TLA+ specification, and simulations against a baseline; the acceptor-store dependency is explicitly discussed and is a design risk, not a circular argument.

full rationale

The paper's central claims are empirical and protocol-based, not derived from a fitted model. The RTO claim (Section 6.1.3) is supported by three power-outage tests on a 3-region account with 4,300+ write-region partitions, which are external observations rather than consequences of the paper's assumptions. The Failover Manager state machine is specified in TLA+ and model-checked with TLC (Section 4.4), an independent mechanical check of the stated safety/liveness properties; no property is assumed by defining it in terms of the result. The CAS Paxos implementation is transliterated from an external TLA+ spec (Section 4.3.1) with no self-citation chain. The adaptive scheduling improvement (Section 6.2) is evaluated by discrete-event simulation that compares the new backoff/TDM policy against an exponential-backoff baseline and reports failure rates; this is a comparative measurement, not a prediction forced by construction. The only apparent self-dependency is that CAS Paxos acceptor state is stored in non-replicated Cosmos DB accounts (Section 4.3.1). The paper explicitly addresses this: 'This might appear to create a circular dependency from Cosmos DB onto Cosmos DB. This is illusory...' That passage concerns an availability/dependency risk, not a logical derivation; the failover protocol's correctness does not assume the claim it is used to demonstrate. No load-bearing step reduces to its own inputs by definition, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 3 invented entities

The central design relies on CAS Paxos as a standard consensus protocol, on the fidelity of a TLA+ model that is not provided, and on the availability of Cosmos DB accounts that host the acceptor state. Key timing parameters (30s heartbeat, 45s lease) are chosen by hand and directly influence the 2-minute RTO claim. The paper introduces software components (Failover Manager, progress table, topology upsert intent) that are evaluated only within the paper.

free parameters (5)
  • Lease enforcement timeout = 45 seconds
    Chosen by hand; used in the CAS Paxos simulation to define the window after which a lease is lost. Directly bounds the liveness property but is not derived from a requirement.
  • Proposer update interval = 30 seconds
    Chosen by hand; each primary replica attempts a state update every 30 seconds. This cadence determines heartbeat frequency and conflict probability.
  • Base delay δ in initial exponential backoff = not specified
    The static backoff delay used in the initial implementation; the paper argues its choice is difficult across heterogeneous regions.
  • Number of acceptors = 7
    Simulation configuration 'mirroring production system characteristics'; the number of acceptor regions affects quorum size and availability.
  • Number of proposers = 3, 5, 7, 9
    Used to test contention levels; results differ by proposer count.
assumptions (4)
  • standard math CAS Paxos provides a replicated state machine without logs as described in [1].
    The paper builds its Failover Manager store on the CAS Paxos protocol, inheriting its safety and liveness properties.
  • domain assumption The TLA+ model accurately models the real Failover Manager state machine and Cosmos DB replication behavior.
    TLC model checking is used to verify invariants; the paper does not provide the TLA+ spec, so the fidelity cannot be checked.
  • domain assumption The acceptor state stores on non-replicated Cosmos DB accounts are independent of the data-plane and remain available during outages.
    The design stores acceptor state in Cosmos DB itself; if those stores are affected by the same outage, failover could be blocked.
  • domain assumption The discrete-event simulator's latency and failure models represent production network behavior.
    Simulation results depend on parameters 'representative of a production environment' but these are not fully specified.
invented entities (3)
  • Failover Manager
    purpose: A backend component executing a deterministic state machine for each partition-set to coordinate failover state transitions.
    Introduced in this paper; evaluated only through the paper's own outage tests and simulations, with no external artifact.
  • Progress table
    purpose: A new replication-protocol table tracking LSNs written in each epoch to undo 'false progress' during failback.
    Described as an extension to the replication protocol; no separate evidence outside the paper.
  • Topology upsert intent
    purpose: A control-plane concept expressing an intended topology change for the Failover Manager to execute as a CAS Paxos round.
    Used to reconcile control-plane and data-plane state during concurrent failovers; no external validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Implementing Decentralized Per-Partition Automatic Failover in Azure Cosmos DB." pith.science (2026). https://pith.science/paper/KVOQRN46

@misc{pith2026250514900,
  author       = {Pith},
  title        = {Pith review of: Implementing Decentralized Per-Partition Automatic Failover in Azure Cosmos DB},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KVOQRN46}},
  note         = {Machine review of arXiv:2505.14900}
}
read the original abstract

Azure Cosmos DB is a cloud-native distributed database, operating at a massive scale, powering Microsoft Cloud. Think 10s of millions of database partitions (replica-sets), 100+ PBs of data under management, 20M+ vCores. Failovers are an integral part of distributed databases to provide data availability during outages (partial or full regional outages). While failovers within a replica-set within a single region are well understood and commonly exercised, geo failovers in databases across regions are not as common and usually left as a disaster recovery scenario. An upcoming release of Azure Cosmos DB introduces a fine grained (partition-level) automatic failover solution for geo failovers that minimizes the Recovery Time Objective (RTO) and honors customer-chosen consistency level and Recovery Point Objective (RPO) at any scale. This is achieved thanks to a decentralized architecture which offers seamless horizontal scaling to allow us to handle outages ranging from node-level faults to full-scale regional outages. Our solution is designed to handle a broad spectrum of hardware and software faults, including node failures, crashes, power events and most network partitions, that span beyond the scope of a single fault domain or an availability zone.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 24 canonical work pages

  1. [1]

    CASPaxos: Replicated State Machines without logs

    D. Rystsov, "CASPaxos: Replicated State Machines without logs," 2018. [Online]. Available: https://arxiv.org/abs/1802.07000

  2. [2]

    Azure Cosmos DB,

    Microsoft, "Azure Cosmos DB," [Online]. Available: https://learn.microsoft.com/en-us/azure/cosmos-db/. [Accessed 2025]

  3. [3]

    Consistency levels in Azure Cosmos DB,

    Microsoft, "Consistency levels in Azure Cosmos DB," 2024. [Online]. Available: https://learn.microsoft.com/en-us/azure/cosmos-db/consistency-levels. [Accessed 2025]

  4. [4]

    Partitioning and horizontal scaling in Azure Cosmos DB,

    Microsoft, "Partitioning and horizontal scaling in Azure Cosmos DB," 11 2024. [Online]. Available: https://learn.microsoft.com/en -us/azure/cosmos- db/partitioning-overview. [Accessed 2025]

  5. [5]

    Commonly asked Service Fabric questions,

    Microsoft, "Commonly asked Service Fabric questions," 2024. [Online]. Available: https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric- common-questions#can-i-create-a-cluster-that-spans-multiple-azure-regions-or- my-own-datacenters. [Accessed 2025]

  6. [6]

    Windows Workflow Foundation Programming - Dynamic update,

    Microsoft, "Windows Workflow Foundation Programming - Dynamic update," [Online]. Available: https://learn.microsoft.com/en - us/dotnet/framework/windows-workflow-foundation/dynamic-update. [Accessed 2025]

  7. [7]

    The Paxos Algorithm,

    L. Lamport, "The Paxos Algorithm," October 2024. [Online]. Available: https://lamport.azurewebsites.net/tla/paxos-algorithm.html. [Accessed 2025]

  8. [8]

    CASPaxos -tla,

    T. Grieger, "CASPaxos -tla," [Online]. Available: https://github.com/tbg/caspaxos-tla. [Accessed 2025]

Show all 25 references
  1. [9]

    Transactions and optimistic concurrency control,

    Microsoft, "Transactions and optimistic concurrency control," [Online]. Available: https://learn.microsoft.com/en -us/azure/cosmos-db/nosql/database- transactions-optimistic-concurrency

  2. [10]

    Deploy cloud witness for a failover cluster,

    Microsoft, "Deploy cloud witness for a failover cluster," February 2025. [Online]. Available: https://learn.microsoft.com/en -us/windows-server/failover- clustering/deploy-cloud-witness. [Accessed 2025]

  3. [11]

    The TLA+ Home Page,

    L. Lamport, "The TLA+ Home Page," [Online]. Available: https://lamport.azurewebsites.net/tla/tla.html

  4. [12]

    Understanding cluster and pool quorum,

    Microsoft, "Understanding cluster and pool quorum," February 2025. [Online]. Available: https://learn.microsoft.com/en -us/windows-server/storage/storage- spaces/quorum. [Accessed 2025]

  5. [13]

    Serving Stale Data to Improve DNS Resiliency,

    D. Lawrence, W. Kumari and P. Sood, "Serving Stale Data to Improve DNS Resiliency," March 2020. [Online]. Available: https://www.rfc - editor.org/info/rfc8767. [Accessed 2025]

  6. [14]

    Paxos made simple.,

    L. Lamport, "Paxos made simple.," ACM SIGACT News (Distributed Computing Column), vol. 121, pp. 51-58, December 2001

  7. [15]

    Azure network round -trip latency statistics,

    Microsoft, "Azure network round -trip latency statistics," [Online]. Available: https://learn.microsoft.com/en-us/azure/networking/azure-network- latency?tabs=Americas%2CWestUS. [Accessed 2025]

  8. [16]

    Global distribution with Azure Cosmos DB - under the hood | Microsoft Learn

    Microsoft, “ Global distribution with Azure Cosmos DB - under the hood | Microsoft Learn” [Online]. Available Global distribution with Azure Cosmos DB- under the hood | Microsoft Learn . [Accessed 2025]. Conference Short Name:WOODSTOCK’18

  9. [17]

    Lamport, D

    L. Lamport, D. Malkhi, and L. Zhou. Vertical Paxos and primary -backup replication. Technical report, Microsoft Research, 2009

  10. [18]

    Lamport, D

    L. Lamport, D. Malkhi, and L. Zhou. Reconfiguring a state machine. SIGACT News, 41(1), Mar. 2010

  11. [19]

    Liskov and J

    B. Liskov and J. Cowling. Viewstamped replication revis ited. Technical Report MIT-CSAIL-TR-2012-021, MIT Computer

  12. [20]

    Andersen, and Michael Kaminsky

    Iulian Moraru, David G. Andersen, and Michael Kaminsky. 2013. There is more consensus in Egalitarian parliaments. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (SOSP '13). Association for Computing Machinery, New York, N Y, USA, 358 –372. ht...

  13. [21]

    Consistency level choices - Azure Cosmos DB | Microsoft Learn

    Microsoft, “Consistency level choices - Azure Cosmos DB | Microsoft Learn ” [Online]. Available: https://learn.microsoft.com/en-us/azure/cosmos- db/consistency-levels

  14. [22]

    Specifying Systems : the TLA+ Language and Tools for Hardware and Software Engineers

    Lamport, Leslie. Specifying Systems : the TLA+ Language and Tools for Hardware and Software Engineers. Boston :Addison-Wesley, 2003

  15. [23]

    Raft Consensus Algorithm: an Effective Substitute for Paxos in High Throughput P2P -based Systems

    Fazlali, Mohammad Reza et al. “Raft Consensus Algorithm: an Effective Substitute for Paxos in High Throughput P2P -based Systems.” ArXiv abs/1911.01231 (2019)

  16. [24]

    Azure Cosmos DB SQL SDK connectivity modes ,

    Microsoft, “Azure Cosmos DB SQL SDK connectivity modes ,” [Online]. Available: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/sdk- connection-modes

  17. [25]

    Leslie Lamport. 1998. The part-time parliament. ACM Trans. Comput. Syst. 16, 2 (May 1998), 133–169. https://doi.org/10.1145/279227.279229

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.