When the Database Commits but the Event Does Not Leave

Tác Giả: OmniStack

Ngày đăng: 11/05/2026

When the Database Commits but the Event Does Not Leave

The failure is not that the database lost the order

In a hypothetical APAC retailer, checkout returns success for an order placed through a store channel. The order row is committed in the source database. The warehouse system never receives the order-created event, so fulfilment does not begin. Finance sees no matching downstream record, and the customer has a confirmation that operations cannot yet honour.

This is a money correctness problem, not a generic messaging problem. Both facts can be true: the database commit can be durable, while the event publication failed, timed out or became unknown. The release decision is therefore specific: accept the current design, require a transactional outbox, or fund replay and reconciliation before rollout.

For a multi-site operator, the exposure is larger than one missing message. A committed payment may remain absent from fulfilment. An inventory reservation may not reach the warehouse. A refund may be recorded locally while the payment provider receives no instruction. The business has accepted one state while another system continues to act on an older state.

The first diagnostic question should not be whether the API returned 200. It should be what durable evidence shows that the downstream intent was recorded and can be delivered, replayed or reconciled. An HTTP response proves what happened at that boundary. It does not prove that a separate broker, relay or consumer completed its work.

Why a successful write and a successful publish are different facts

A local database transaction controls the business row. A broker call controls a separate external effect. Unless both are covered by one reliable coordination mechanism, a process failure can leave the database committed and the event absent.

The dual-write gap appears when application code performs two actions in sequence: update the business table, then publish to a broker. The order of those actions matters, but neither order removes the failure window.

Sequence

What can break

Business result

Publish first, commit second

The process publishes, then the database transaction rolls back or times out

Downstream systems act on an order, payment or inventory change that the source of truth does not contain

Commit first, publish second

The process crashes after commit or the broker call fails

The source row exists, but downstream systems never receive the corresponding intent

Commit and publish in one request path

The broker response is delayed or unknown after the database commits

The caller cannot safely infer whether to retry without creating a duplicate

Publish succeeds, acknowledgement is lost

The producer retries because it cannot distinguish success from timeout

The broker or consumer may see the same event more than once

Adding another retry around the broker call may help in ordinary transient failures, but it does not create a durable record of intent when the application dies before the retry runs. It also does not resolve an unknown broker outcome. A retry policy is a recovery behaviour, not a transaction boundary.

For teams modernising a legacy order or payment service, the practical design question is narrower than whether to adopt event-driven architecture. Ask whether a committed business transition must produce a durable downstream instruction. If the answer is yes, the commit and the publication need separate evidence and a defined recovery path.

A committed database parcel is separated from an empty broker conveyor by a cracked bridge.

What the transactional outbox pattern changes

Verdict: use a transactional outbox when a committed business state must remain publishable after the application process fails. The AWS Prescriptive Guidance description of the transactional outbox pattern explains the core mechanism: write the business change and an outbox record in the same database transaction, then use a separate relay to publish from that durable record.

That is the important change. The application no longer has to keep an in-memory promise to publish after committing the order. The promise becomes a row committed in the same transaction as the order itself.

A hypothetical order transaction might write an order with status CONFIRMED and an outbox row containing an event identifier, event type, aggregate identifier, payload or payload reference, creation time and publication status. If the transaction commits, both records commit. If it rolls back, neither becomes visible as committed business state.

A relay reads unpublished outbox rows and sends events to the broker. It may mark a row as sent after publication, or maintain an attempt and status model that supports retry. The relay can be restarted independently from the checkout process. This is why the pattern changes the database commit event publish failure surface: an application crash after commit no longer erases the publication intent.

The limit is as important as the mechanism. AWS notes that duplicate messages can occur. A relay can publish an event and crash before recording that it has published it. On restart, the same outbox row may be sent again. The design does not create a single execution outcome across the database, broker and consumer.

The implementation needs an explicit state model. A row marked pending may be eligible for relay. A row marked in_flight needs lease expiry or equivalent recovery if the relay disappears. A row marked published still does not prove that every consumer completed its business effect. Publication and consumption are different checkpoints.

Teams also need to decide whether the event payload is stored in the outbox or reconstructed from the business tables. Reconstructing later can produce a different payload if the order changes after the original transition. For money, fulfilment and inventory workflows, the outbox should preserve the intent that was committed, or preserve enough immutable data to reproduce it without ambiguity.

A relay backlog is not merely an infrastructure metric. Its age tells operations how long downstream systems may have been acting on stale information. Its count tells engineering how much recovery work exists. Its business mapping tells the operations leader whether orders, refunds, stock movements or customer notifications are affected.

For teams building the relay and its deployment controls, DevOps and CI/CD engineering support is relevant only when it includes ownership of the operational evidence: deployment rollback, relay health, backlog visibility and recovery testing. A pipeline that deploys the relay without proving its failure modes has not solved the delivery problem.

Database vault commits a business record and outbox envelope together while a relay carries messages outward.

Build the proposed outbox fault-injection matrix

Proposed template, not an OmniStack deployment record. A transactional outbox fault-injection matrix turns an architectural claim into an acceptance test. Each row names the injected failure, the state that must survive, the recovery behaviour, the evidence required and the limit of what the test proves.

  • Run the test against the actual order, payment or fulfilment workflow in scope, not only a synthetic publisher.
  • Record correlation identifiers for the business transaction, outbox row, broker message and consumer intent record.
  • Capture durable state before recovery, during retry and after reconciliation.
  • Assign an owner to every unresolved outcome, especially unknown provider responses and business-priority decisions.

Fault injection

Expected committed state

Expected outbox or relay state

Expected consumer behaviour

Acceptance evidence

Limit

Crash after database commit, before publish

Order or payment state remains committed

Outbox row remains unpublished

No downstream side effect until relay resumes

Relay later publishes from the durable outbox record

Does not prove downstream business acceptance

Relay publishes, then crashes before marking sent

Committed state unchanged

Outbox row may be retried

Consumer handles duplicate message using an idempotency rule

Duplicate attempt produces one accepted business effect

Does not prove duplicates are impossible

Broker or API response times out

Original intent remains inspectable

Status is unknown until lookup, retry or reconciliation

Consumer does not infer decline or failure from missing response alone

Runbook classifies unknown outcome before retry

Provider contracts still matter

Consumer processes event, then fails before acknowledgement

Source of truth remains committed

Message may be redelivered

Consumer recognises the same event intent and avoids repeating the external effect

Idempotency key and stored result are checked

Semantic equivalence must be defined

Relay backlog accumulates during outage

Committed states continue to queue

Outbox age and count breach the workflow threshold

Operations decide whether to pause intake, catch up or reconcile

Backlog report links rows to business workflows

Matrix does not choose the commercial priority

The first row is the minimum test for the pattern. Stop the application after the database commit and before the relay can publish. Restart the relay and prove that it finds the durable row. A trace showing an attempted publish is not sufficient. Stronger evidence is the committed order, the surviving outbox record, the later broker publication and the consumer’s recorded handling.

The second and fourth rows test a harder property. A duplicate can arrive after a real business effect has already occurred. The consumer must recognise the same intent and return the previously recorded result or perform no second effect. A successful second HTTP response does not prove this. The test must inspect the business state that would have been duplicated: one reservation, one refund instruction, one fulfilment command or one ledger transition.

The timeout row deserves separate treatment. A provider may have accepted a payment or fulfilment instruction even though the caller received no response. Retrying immediately can create a second instruction. The runbook needs a lookup, status query or reconciliation step before deciding that the first attempt failed.

The backlog row connects engineering health to operations. A relay may be technically healthy while the oldest outbox record is too old for the business workflow. The threshold belongs to the workflow owner. A stock event and a customer notification may not have the same tolerance for delay, but the matrix should make that decision visible rather than burying it in a generic queue alert.

Related:workflow automation engineering, useful when replay, exception routing and reconciliation need to become explicit operational workflows.

Define idempotency by intent, not by retry count

Idempotency means that repeating the same business intent produces one accepted effect, not that the system permits an arbitrary number of retries. The AWS Builders Library article on making retries safe with idempotent APIs is useful here because it centres on caller intent, client request identifiers, late-arriving requests and the problem of the same identifier being reused for a different request.

Apply that discipline carefully to events. An event identifier is useful, but it is not the entire business rule. A consumer needs to determine what the message means, which aggregate transition it represents and whether a late message is still valid.

Question

Weak rule

Stronger consumer contract

What identifies a repeated event?

Compare a transport message identifier only

Use an event or intent identifier linked to the business aggregate and transition

What if the same identifier has a different payload?

Ignore the second message silently

Reject, quarantine or escalate because the identifier has conflicting intent

What if the message arrives late?

Process it because it is syntactically valid

Check version, state transition and business validity before applying it

Where is the result recorded?

Rely on logs or broker acknowledgement

Store processed intent and the resulting business outcome in durable consumer state

In a hypothetical fulfilment consumer, an order-created event may arrive twice. The consumer should not create two warehouse jobs. It may store the event identifier, order identifier, expected transition and the created job reference. If the same identifier arrives with a different warehouse location or amount, silent deduplication is unsafe. The consumer should move the message to an exception path and preserve the conflict for investigation.

Payment workflows make the distinction sharper. A retry to authorise the same order for the same amount may be the same intent if the identifier, order and amount match. A retry using the same identifier for a changed amount is not the same intent. The consumer or provider integration must reject or investigate it. A retry counter cannot make that semantic decision.

Versioning is another part of the contract. A late event may be a valid duplicate, an obsolete transition or an out-of-order message. Consumers should compare the event’s aggregate version or transition context with their current state. The acceptable action may be apply, ignore as already applied, quarantine or reconcile. Those outcomes should be visible in metrics and operational records.

Idempotency storage also needs a retention decision. Retaining a processed record for too short a period can allow a late duplicate to repeat an effect. Retaining it indefinitely may create operational and data-management obligations. The correct period depends on the maximum replay and late-arrival window for the workflow, and that window must be documented rather than assumed.

What evidence a CTO should ask for before accepting the release

A release should not be accepted because the outbox table exists. Acceptance requires evidence that a committed intent survives publisher failure, that duplicate delivery produces one accepted business effect, and that operations can see and resolve an unknown or delayed outcome.

Ask for the outputs of the proposed matrix, with the injected failure and recovery times recorded. The evidence should show durable records, not only traces. Distributed tracing can explain where a relay stopped or which attempt was made, but it cannot prove that a warehouse job was created once or that a refund was reconciled.

  • Crash-after-commit output: the committed business row, surviving outbox row and later publication from that row.
  • Duplicate-delivery output: two delivery attempts linked to one intent, with one accepted business effect and a stored consumer result.
  • Unknown-outcome runbook: the lookup, retry, quarantine and reconciliation steps for a broker or provider timeout.
  • Backlog visibility: count, oldest age, workflow type and business impact, with an owner who can decide whether to pause intake or prioritise recovery.
  • Replay procedure: selection criteria, authorisation, dry-run or preview behaviour, audit record and post-replay verification.
  • Known-gap register: unresolved edge cases, affected workflows, named owners and current decision status.

There is a useful distinction between observability and correctness. A dashboard can report that the relay is running while a malformed event is rejected by every consumer. A trace can show a publish acknowledgement while the consumer later applies the same refund twice. The acceptance pack should connect technical records to the business effect.

Use this decision rule:

  • Accept when the in-scope matrix passes, duplicate handling is proven, backlog and replay are visible, and remaining gaps have named owners.
  • Pilot when the outbox mechanism works but reconciliation or duplicate handling remains unproven for a high-value workflow. Limit the scope and keep manual review active.
  • Defer when a committed state can still disappear from downstream operations without replay, alerting or a defined manual reconciliation path.

Teams modernising the surrounding application should also keep the transactional boundary narrow. Putting a remote API call inside the database transaction does not make that remote call atomic. It lengthens lock duration and still leaves the external provider’s outcome subject to timeout. Record intent locally, publish through a recoverable relay, and handle external uncertainty explicitly.

Where the transactional outbox is the wrong answer

The outbox pattern is not a universal repair. It is the wrong first move when the business event is disposable, when the source database is not the authority for the transition, or when the consumer cannot define what duplicate handling means.

A telemetry signal that can be regenerated may not need an outbox row for every observation. A search index update may be rebuilt from the source database if the rebuild process is tested and the temporary inconsistency is acceptable. A workflow that requires a provider’s synchronous decision before committing may need a different saga or reservation design, because an outbox cannot make the provider response atomic with the local transaction.

The pattern also fails as an organisational solution when nobody owns reconciliation. A durable outbox can preserve many failed intentions while the business remains unable to decide which should be replayed, cancelled or manually corrected. More storage does not create a policy.

There is a further failure mode in legacy modernisation. Teams sometimes add an outbox to a service but leave hidden side effects inside the transaction path: email delivery, stock calls, payment attempts or warehouse API requests. The outbox then gives a false sense of safety while the application still performs non-transactional work before the commit. Map every side effect and decide whether it becomes an event, a command, a lookup or a compensating action.

The approach is also a poor fit when the team cannot operate the relay. The relay needs deployment controls, retry limits, poison-message handling, schema compatibility, backlog alerts and recovery drills. If those capabilities are absent, a queue may simply move the failure into a less visible place.

In-house ownership wins when the existing team already has system knowledge, operational ownership and release authority, and can reserve capacity for the matrix, relay hardening and reconciliation work. A dedicated engineering team is appropriate only when the internal team understands the business but is consumed by daily operations and cannot give the failure path a named delivery owner.

Where a dedicated engineering team fits

The practical work is usually distributed across application engineering, data, QA, DevOps and operations: define the outbox schema, harden the relay, instrument the consumer, inject failures, verify duplicate effects and write the reconciliation runbook. A dedicated engineering team option can carry that work against the client roadmap when internal capacity is the constraint, provided the scope is the release evidence and recovery behaviour rather than open-ended staffing.

The boundary must be explicit. The client remains accountable for business policy, risk acceptance and the decision to pause, replay or manually reconcile a workflow. The delivery team owns the implementation evidence: fault-injection runs, consumer idempotency tests, backlog visibility, deployment controls and the handover materials that let the client operate the system.

For organisations extending a legacy platform, backend engineering support for service boundaries and event workflows may fit where the work includes durable state design and operational testing, not merely endpoint construction. For process-heavy environments, process automation engineering can help connect exception queues and reconciliation steps to the people who must act on them.

The decision should be made against the release gap. If the internal team can run the proposed matrix and own the remaining defects, keep the work in-house. If the matrix is repeatedly deferred because roadmap delivery consumes all available capacity, assign one delivery-owning team and make the acceptance evidence part of the release scope.

Sources and limits

  • AWS Prescriptive Guidance: Transactional outbox pattern. Used for the mechanism of writing business state and outbox state in one database transaction, and for the limit that duplicate messages can occur.
  • AWS Builders Library: Making retries safe with idempotent APIs. Used for idempotency concepts: caller intent, client request identifiers, late-arriving requests and identifier reuse with different intent.
  • These sources support engineering mechanisms. They do not prove that any particular implementation is correct, accepted by a regulator, or complete for a specific APAC operating model.

Make the next release decision explicit

Start with one workflow where a committed state has material operational consequences: order creation, payment capture, refund, inventory reservation or fulfilment instruction. Draw the database transaction, outbox row, relay attempt, broker boundary and consumer effect on one page. Mark every point where the outcome can be unknown.

Run the proposed transactional outbox fault-injection matrix against that workflow. Do not accept screenshots of logs as proof. Require the durable row, the replay, the consumer’s idempotency record and the resulting business state. Inject the failure after commit before publish, after publish before marking sent, during an unknown broker response, and after consumer processing before acknowledgement.

At the review meeting, force three decisions into the record: what can be replayed automatically, what requires reconciliation, and who has authority to stop or prioritise the workflow during backlog recovery. If any answer is to investigate only after the failure occurs, the release still has an unowned failure surface.

Accept the release if the matrix passes for the workflows in scope and the remaining gaps have named owners. Pilot if the mechanism works but reconciliation or duplicate handling is unproven for high-value paths. Defer if a committed state can still disappear from downstream operations without replay, alerting or manual reconciliation.