Reconcile What Happened While the System Was Down: A Recovery Cutover Ledger Guide
Author: OmniStack
Published at: 11/15/2026

On this page
- Start with the outage interval, not the queue
- Separate local state from external evidence
- Classify each item before replay
- Build the proposed recovery cutover ledger
- Replay only after idempotency and intent checks
- Handle the commit and event gap without promising exactly-once behaviour
- Define cutover acceptance and post-reopen monitoring
- FAQ
- What is a recovery cutover ledger?
- How should a team reconcile orders after an outage?
- Does idempotency make bulk replay safe?
- Does a transactional outbox provide exactly-once processing?
- Sources and limits
A hypothetical multi-site retailer brings checkout back after an outage. The database is reachable. The warehouse has an export file. The payment provider may have accepted some requests. Customer service has a list of caller timeouts. Nobody can yet say which orders are safe to ship.
That is the point at which recovery becomes a business control problem. Reconcile orders after an outage by defining the affected business window, separating local state from external evidence, classifying every item, and approving an action for each exception. Do not treat the queue as the source of truth. Do not turn every timeout into a retry.
This guide sets out a proposed recovery cutover ledger template for CTOs, CIOs, operations leaders, and engineering teams across APAC. Every scenario is hypothetical. The ledger is a proposed template, not an OmniStack deployment.
Start with the outage interval, not the queue
The outage interval is the business window in which an action may have reached one system but not another. It must be defined before reconciliation starts, with the affected workflows, system clocks, evidence sources, and the point at which normal processing may resume. A restored database proves database availability. It does not prove what happened in payment, warehouse, marketplace, or notification systems during the gap.
In the hypothetical retailer, checkout may have started timing out before the application recovered. The payment provider may have accepted a request after the caller stopped receiving responses. A warehouse export may have run while the order service had no final record. If the team begins with messages waiting in the broker, it has already discarded the most important fact: the same customer intent may exist in several evidence systems with different timestamps.
What breaks | Why the queue cannot resolve it | Evidence needed |
|---|---|---|
Payment response is missing | A timeout says the caller did not receive a response, not that the provider declined the request | Provider lookup, request reference, payload fingerprint, and local payment state |
Warehouse file exists without an order confirmation | An operational export may represent a downstream effect that the order service never committed | Export timestamp, line items, dispatch reference, and order identity match |
Order row exists without a published event | A local commit may have succeeded while the event publication failed | Transaction record, outbox record or broker log, and consumer state |
Customer received conflicting notifications | Notification delivery does not prove payment or fulfilment state | Notification log, template event, provider status, and underlying business object |
Marketplace state differs from the local order | External systems may accept or mutate an order while local processing is unavailable | Marketplace order record, acknowledgement, cancellation state, and timestamps |
The interval should use a named clock standard. A regional operator may have application logs in UTC, warehouse exports in local time, and payment reports aligned to the provider account timezone. Record the conversion rule in the ledger. During the outage is not evidence until the team can show which clock placed the object inside the window.
Define four boundaries in the proposed template:
- Start: the first reliable evidence that the workflow could no longer complete normally, such as confirmed dependency failure or workflow failure evidence.
- End: the point at which the affected workflow passed its recovery checks, not merely when infrastructure became reachable.
- Scope: the sites, channels, payment methods, warehouses, marketplaces, and notification paths included.
- Reopen gate: the named authority and evidence required before new traffic is allowed to create additional effects.
A recovery interval is not the same as a recovery objective. AWS Well-Architected guidance on disaster recovery testing supports testing failover and recovery objectives, but a tested recovery path proves the tested condition. It does not prove that every business effect during an outage has been reconciled.
For operators managing several locations, the scope must be granular enough to distinguish a store terminal failure from a payment-region failure. A single global interval can hide a partial outage. Record the affected site, channel, and workflow on each business object where that distinction changes the decision.
Separate local state from external evidence
Verdict: internal state is one evidence family, not the final answer. Reconciliation requires a comparison between the local committed record and independent evidence from providers, warehouses, marketplaces, exports, and message infrastructure.
A local order marked payment pending means the application has not recorded a completed payment. It does not mean the payment provider has no authorisation. A caller timeout means the caller lacks a response. It does not mean the request had no external effect. These distinctions determine whether the safe action is to update local state, query again, cancel, hold, or escalate.
The proposed ledger should preserve the original evidence rather than replacing it with a new status. Store the local record identifier, the provider request reference, the relevant external record, the time observed, and the source system. A screenshot may support an urgent decision, but a machine-readable provider report or API response is stronger closure evidence when it can be retained.
Evidence separation also protects the recovery team from a common false match. Two orders can share a customer, amount, and approximate timestamp. They are not the same intent. Match using the strongest available identity: business object ID, merchant reference, request identifier, provider transaction reference, warehouse dispatch reference, or marketplace order ID. Amount and time should support a match, not create one.
For payment workflows, the team should ask what the provider actually records: authorisation, capture, settlement, refund, reversal, or no request. Those states have different customer and finance consequences. A local paid flag without a provider reference is not equivalent to a provider capture record. A provider authorisation without a local order may require a hold and investigation rather than an automatic capture.
For fulfilment, compare the order service with warehouse exports and the warehouse management system. A pick file can show that work was released. It does not necessarily show that goods were picked, packed, handed to a carrier, or delivered. The approved action must match the actual downstream state, not the earliest event in the chain.
For marketplaces and external channels, retain the external order state even when the local system is later corrected. A local update that erases the original mismatch makes the next investigation harder. The ledger should point to the evidence location and record who accepted its interpretation.
Teams also need an operational evidence policy. The proposed template should identify which logs are authoritative for each workflow, how long the evidence remains available for the incident, and who can retrieve provider records. This is where data governance and management controls can support consistent ownership of identifiers, timestamps, and evidence definitions, although the control design remains specific to the organisation.

Classify each item before replay
Classification is the gate between evidence collection and action. Each item should be placed into an explicit class before any payment, dispatch, refund, cancellation, or event replay is approved. Unknown outcome is a controlled business state with an owner. It is not permission to charge, ship, refund, or replay automatically.
- Present and consistent: local and external records describe the same intent and effect. Close the item with the matching evidence.
- Missing locally but present externally: the provider or downstream system shows the effect, while the local application does not. Update local state or create a controlled repair record. Do not repeat the external action.
- Present locally but absent externally: the application claims an effect that the external system cannot confirm. Hold the item and investigate whether the local record is provisional, the external lookup is incomplete, or a later record is delayed.
- Duplicate local record: more than one local record appears to describe the same business intent. Preserve the evidence and identify which record, if any, maps to the external effect.
- Duplicate external effect: the same business action has produced more than one external effect, such as two captures or two dispatch instructions. Escalate to the owner of the affected business process.
- Unknown outcome: available evidence cannot establish whether the external effect occurred. Keep it open, assign an owner, define the next evidence check, and prevent an irreversible action until the decision rule is met.
The useful metric is not the number of messages replayed. It is the number of business objects with an approved action and closure evidence. High-volume replay can increase exposure if the underlying intent has not been matched.
Each class should have an action policy agreed before the incident. A missing locally but present externally payment may require a local reference update. A present locally but absent externally dispatch may require a warehouse confirmation. An unknown payment may require a provider lookup and a customer service hold. The policy belongs to the business owner because the consequences include finance, customer communication, and inventory.
The approach fails when the organisation has no stable business identity. If orders are identified only by mutable database sequence numbers, or if provider requests were sent without durable references, the team may not be able to prove equivalence. A recovery ledger can expose that weakness. It cannot manufacture evidence that was never recorded. In that case, manual review and conservative holds are safer than automated replay.
It also fails when the business treats every exception as an engineering defect. Some outcomes require finance, warehouse, or customer service decisions. Engineering can establish what the systems show. It should not silently decide whether a customer is charged twice or whether an order ships without confirmed payment.
Build the proposed recovery cutover ledger
A recovery cutover ledger is a proposed decision record for each affected business object. It connects the outage interval, original intent, local state, external evidence, classification, approved action, owner, and closure proof. The ledger is not a queue, a replacement database, or proof that any particular team has executed the process.
Ledger field | Decision it supports | Example value in a hypothetical outage |
|---|---|---|
Business object ID | Connects operational evidence to a customer-visible order, refund or dispatch | Hypothetical order identifier |
Workflow | Separates payment, fulfilment, marketplace, notification, and refund decisions | Checkout payment authorisation |
Original intent | Distinguishes the action requested from the effect observed later | Capture payment for an accepted order |
Outage interval status | Shows whether the object was active during the cutover window | Created after checkout degraded, before recovery sign-off |
Local committed state | Identifies what the primary application believes happened | Order accepted, payment pending |
External evidence source | Records independent evidence from provider or downstream system | Payment provider lookup shows authorisation succeeded |
Idempotency key or request reference | Connects retry decisions to the original business intent | Provider request reference retained from the original call |
Classification | Prevents blind replay by naming the exception type | Missing locally but present externally |
Approved action | Turns reconciliation into an accountable cutover decision | Update local payment reference, do not recapture |
Owner, time of decision, and closure evidence | Shows who accepted the action and what proves completion | Operations owner accepted after provider report was attached |
The ledger should be append-only for decisions. If the classification changes, record the reason, new evidence, decision owner, and timestamp. Editing a cell without preserving the prior decision creates a second incident: the organisation can no longer explain why an irreversible action was taken.
Use one row per business object and one linked record per external effect. A single row that combines an order, payment, refund, and dispatch can conceal partial success. If the workflow has separate irreversible steps, the ledger should either model those steps as child records or link to a detailed reconciliation record.
The minimum workflow is straightforward:
- Extract candidate objects from local logs, database records, provider reports, warehouse exports, marketplace records, and customer-facing events.
- Normalise identifiers and timestamps without overwriting the original values.
- Compare local and external evidence against the original intent.
- Assign a classification and record the reasoning.
- Approve one action, one owner, and one closure condition.
- Execute the action with a decision trail.
- Recheck delayed systems and close only when the evidence supports closure.
Do not make the ledger a spreadsheet that only one person understands. The fields should be accessible to the incident lead, operations owner, engineering owner, finance where payments are involved, and customer service where customer-visible action is possible. Access control still matters, particularly for payment and personal data. The ledger should reference sensitive evidence rather than duplicating it unnecessarily.
Related:data backup, recovery and resilience systems, useful when the recovery plan needs to connect restored infrastructure with evidence of business effects.

Replay only after idempotency and intent checks
Verdict: replay is safe only when the request identifier represents the same business intent and the receiving contract defines how duplicates are handled. A retry key is not a blanket replay licence.
The AWS Builders’ Library article Making retries safe with idempotent APIs describes the core problem: a caller may fail to receive a response even though the request eventually takes effect. A caller-supplied request identifier captures intent so the service can distinguish a retry of the same operation from a new operation. That distinction is critical during cutover.
For each candidate replay, compare:
- the original business object and requested action;
- the original request identifier or provider reference;
- the payload fingerprint or equivalent evidence;
- the provider documented idempotency window and equivalence rule;
- the observed external state and any late-arriving response.
If the same identifier is reused with a different amount, currency, recipient, line item set, or other material field, treat it as a defect or exception. Do not assume the provider will reject it safely. The application and provider may define same request differently.
A practical decision table belongs in the proposed runbook:
Observed condition | Decision | Reason |
|---|---|---|
Same intent, same idempotency key, external effect confirmed | Update or repair local state; do not replay | The effect already exists |
Same intent, same key, no external effect confirmed, provider contract supports retry for that condition | Retry with the original key and payload | The retry preserves intent |
Same key, different payload | Hold as an exception | Equivalence is not established |
No durable key and ambiguous external state | Manual review or provider-specific lookup | A new request could create a duplicate effect |
External provider unavailable during reconciliation | Keep unknown open and define a later check | Absence of evidence is not evidence of absence |
Late responses need their own handling. A provider may return an effect after the recovery team has classified the item as unknown. The ledger owner should be able to attach the late record, revise the classification, and cancel a planned replay before it runs. This is why the approved action needs a status such as proposed, approved, executing, blocked, or closed.
Idempotency also applies to consumers. A replayed event may be accepted twice by a downstream service unless that consumer stores a durable processing key or equivalent business identity. The recovery team should test the receiving side, not only the sender. A payment API may be idempotent while a notification consumer is not.
Where a recovery action crosses workflow tooling, service queues, or manual approvals, the proposed template should identify which step is system-driven and which step requires a named owner. workflow automation controls can be relevant for routing exceptions and preserving approval evidence, but the business decision still belongs to the accountable owner.
Handle the commit and event gap without promising exactly-once behaviour
A transactional outbox reduces the gap between a local state change and its event by writing both in one local transaction. It does not create exactly-once financial processing across every downstream system. Relays can publish duplicates, consumers can fail after processing, and external providers can apply effects outside the transaction boundary.
The failure is concrete. A hypothetical order service commits accepted and payment pending but crashes before publishing the order event. The order exists locally, yet the fulfilment service never sees it. In the opposite sequence, an event may be published while the local transaction is rolled back if the design does not bind the two operations safely.
A proposed outbox design writes the business state and an outbox record in the same local transaction. A relay reads unsent records and publishes them. The relay marks the record after publication, but a crash between publication and marking can produce a duplicate. Consumers therefore need idempotency based on the event ID, business object ID, or a carefully defined equivalent.
Failure point | What the ledger should capture | Recovery decision |
|---|---|---|
State committed, outbox record absent | Transaction evidence and expected event type | Repair the missing outbox record under controlled change |
Outbox record present, broker publication unknown | Relay attempt, broker acknowledgement, and consumer evidence | Republish only under consumer idempotency rules |
Event consumed, downstream effect unknown | Consumer processing key and external provider reference | Query the downstream system before replay |
Duplicate event delivered | Duplicate event IDs and consumer handling result | Close if the consumer suppressed the duplicate; otherwise classify the effect |
The AWS transactional outbox pattern is useful precisely because it states the boundary. The pattern addresses a dual-write problem within the service. It does not coordinate a payment provider, warehouse, marketplace, and notification platform in one atomic transaction. The recovery cutover ledger remains necessary for effects outside the local transaction.
Operationally, keep replay controls separate from normal event processing. A recovery replay should carry a reason, incident reference, operator identity, and original event identity. The consumer should be able to distinguish a legitimate retry from a new business action without changing the business meaning of the event.
Teams building or repairing these controls may also need a clear deployment and observability path. cloud and DevOps capability covering CI/CD, monitoring, backup, and security can be relevant when the recovery mechanism itself needs controlled releases and evidence capture. That is an engineering capability choice, not a substitute for business approval.
Define cutover acceptance and post-reopen monitoring
Verdict: reopen approval should be a named business decision backed by engineering evidence, with residual exceptions explicitly accepted and a second reconciliation pass scheduled for delayed effects.
The incident lead or recovery commander should coordinate the decision, but the authority to reopen depends on the workflow. Operations may approve warehouse processing. Finance may approve payment reconciliation. The product or technology owner may approve application traffic. No single technical green check should silently represent all of those decisions.
The acceptance record should state:
- which outage interval and systems were covered;
- which classifications must be closed before reopening;
- which exceptions may remain open under monitoring;
- who owns each residual exception;
- what customer, finance, warehouse, or provider action is blocked;
- when delayed external records will be checked again;
- what evidence supports the decision to resume normal processing.
Google SRE incident response guidance emphasises clear roles, communication, and handoff. Apply that discipline to reconciliation. The person who freezes replay should be named. The person who can approve a payment action should be named. The person who owns provider evidence should be named. Handover should include open ledger rows, not a verbal statement that the backlog is under control.
Post-reopen monitoring should watch for late effects and new divergence. A payment provider report may reveal an authorisation after the first pass. A warehouse may upload a delayed dispatch confirmation. A marketplace may send an order cancellation that was generated during the outage. These records should be matched against open and closed ledger items, with a rule for reopening a closed item when new evidence contradicts the prior decision.
Do not set a universal monitoring duration without understanding provider settlement, warehouse export, and marketplace timing. The correct interval is a property of the workflow and evidence source. Record the next check time and the reason for it. If the provider report is delayed, the ledger should say so rather than imply that the current absence of a record is final.
There is a point at which the ledger approach is the wrong tool. A catastrophic loss of identifiers, corrupted external records, or an outage spanning a period with no trustworthy timestamps may make object-level reconciliation impossible. The responsible decision may be to freeze the workflow, use a provider or warehouse statement, contact affected customers, and escalate the residual risk through the organisation incident process. Automating a false match would be worse than leaving an explicit exception.
OmniStack can be considered as a dedicated engineering team option when an organisation needs continuing developers, QA, UX, cloud, and data capability to build and maintain these controls. In-house is the better answer when recovery engineering is a permanent core competency, the organisation can retain the full team, and its leaders are prepared to own continuity through future incidents. The decision should follow accountability, not a temporary gap in the hiring pipeline.
Make the next decision concrete: name the recovery commander, choose one outage interval for a hypothetical workflow, and create the first ledger row from a local record plus one external evidence source. If the team cannot populate the business object ID, original intent, external reference, classification, and closure owner, stop the replay plan and fix the evidence path before reopening traffic.
FAQ
What is a recovery cutover ledger?
A recovery cutover ledger is a proposed decision record for each business object affected by an outage. It connects the outage interval, original intent, local committed state, external evidence, classification, approved action, owner, and closure evidence.
How should a team reconcile orders after an outage?
Define the affected business window, compare local records with payment, warehouse, marketplace, and message evidence, classify each order, and approve an action before replay. An unknown outcome should remain an owned exception until evidence establishes what happened.
Does idempotency make bulk replay safe?
No. Idempotency helps when the same request identifier represents the same business intent and the receiving service defines equivalent retries. A reused key with a different payload, an unknown provider state, or a non-idempotent consumer still requires an exception decision.
Does a transactional outbox provide exactly-once processing?
No. A transactional outbox binds a local state change to an outbox record, reducing a dual-write gap. Relays can still publish duplicates, and downstream consumers and external providers remain outside the local transaction boundary.
Sources and limits
The sources below support mechanisms, not blanket recovery coverage. They should not be read as proof that retries, traces, restored backups, or a ledger can reconstruct every business effect automatically.
- AWS Prescriptive Guidance: Transactional outbox pattern, for the local transaction and event publication boundary, including duplicate message considerations.
- AWS Builders’ Library: Making retries safe with idempotent APIs, for caller-supplied request identifiers, late-arriving requests, and intent-preserving retries.
- Google SRE Workbook: Incident response, for role clarity, communication, and handoff discipline during incidents.
- AWS Well-Architected Reliability Pillar: Plan for disaster recovery testing, for the value and boundary of tested recovery paths.

