When the Database Commits but the Event Does Not Leave
Author: OmniStack
Published at: 11/05/2026

On this page
- The failure is not that the database lost the order
- Why a successful write and a successful publish are different facts
- What the transactional outbox pattern changes
- Build the proposed outbox fault-injection matrix
- Define idempotency by intent, not by retry count
- What evidence a CTO should ask for before accepting the release
- Where the transactional outbox is the wrong answer
- Where a dedicated engineering team fits
- Sources and limits
- Make the next release decision explicit
The failure is not that the database lost the order
In a hypothetical APAC retailer, checkout returns success for an order placed through a store channel. The order row is committed in the source database. The warehouse system never receives the order-created event, so fulfilment does not begin. Finance sees no matching downstream record, and the customer has a confirmation that operations cannot yet honour.
This is a money correctness problem, not a generic messaging problem. Both facts can be true: the database commit can be durable, while the event publication failed, timed out or became unknown. The release decision is therefore specific: accept the current design, require a transactional outbox, or fund replay and reconciliation before rollout.
For a multi-site operator, the exposure is larger than one missing message. A committed payment may remain absent from fulfilment. An inventory reservation may not reach the warehouse. A refund may be recorded locally while the payment provider receives no instruction. The business has accepted one state while another system continues to act on an older state.
The first diagnostic question should not be whether the API returned 200. It should be what durable evidence shows that the downstream intent was recorded and can be delivered, replayed or reconciled. An HTTP response proves what happened at that boundary. It does not prove that a separate broker, relay or consumer completed its work.
Why a successful write and a successful publish are different facts
A local database transaction controls the business row. A broker call controls a separate external effect. Unless both are covered by one reliable coordination mechanism, a process failure can leave the database committed and the event absent.
The dual-write gap appears when application code performs two actions in sequence: update the business table, then publish to a broker. The order of those actions matters, but neither order removes the failure window.
Sequence | What can break | Business result |
|---|---|---|
Publish first, commit second | The process publishes, then the database transaction rolls back or times out | Downstream systems act on an order, payment or inventory change that the source of truth does not contain |
Commit first, publish second | The process crashes after commit or the broker call fails | The source row exists, but downstream systems never receive the corresponding intent |
Commit and publish in one request path | The broker response is delayed or unknown after the database commits | The caller cannot safely infer whether to retry without creating a duplicate |
Publish succeeds, acknowledgement is lost | The producer retries because it cannot distinguish success from timeout | The broker or consumer may see the same event more than once |
Adding another retry around the broker call may help in ordinary transient failures, but it does not create a durable record of intent when the application dies before the retry runs. It also does not resolve an unknown broker outcome. A retry policy is a recovery behaviour, not a transaction boundary.
For teams modernising a legacy order or payment service, the practical design question is narrower than whether to adopt event-driven architecture. Ask whether a committed business transition must produce a durable downstream instruction. If the answer is yes, the commit and the publication need separate evidence and a defined recovery path.

What the transactional outbox pattern changes
Verdict: use a transactional outbox when a committed business state must remain publishable after the application process fails. The AWS Prescriptive Guidance description of the transactional outbox pattern explains the core mechanism: write the business change and an outbox record in the same database transaction, then use a separate relay to publish from that durable record.
That is the important change. The application no longer has to keep an in-memory promise to publish after committing the order. The promise becomes a row committed in the same transaction as the order itself.
A hypothetical order transaction might write an order with status CONFIRMED and an outbox row containing an event identifier, event type, aggregate identifier, payload or payload reference, creation time and publication status. If the transaction commits, both records commit. If it rolls back, neither becomes visible as committed business state.
A relay reads unpublished outbox rows and sends events to the broker. It may mark a row as sent after publication, or maintain an attempt and status model that supports retry. The relay can be restarted independently from the checkout process. This is why the pattern changes the database commit event publish failure surface: an application crash after commit no longer erases the publication intent.
The limit is as important as the mechanism. AWS notes that duplicate messages can occur. A relay can publish an event and crash before recording that it has published it. On restart, the same outbox row may be sent again. The design does not create a single execution outcome across the database, broker and consumer.
The implementation needs an explicit state model. A row marked pending may be eligible for relay. A row marked in_flight needs lease expiry or equivalent recovery if the relay disappears. A row marked published still does not prove that every consumer completed its business effect. Publication and consumption are different checkpoints.
Teams also need to decide whether the event payload is stored in the outbox or reconstructed from the business tables. Reconstructing later can produce a different payload if the order changes after the original transition. For money, fulfilment and inventory workflows, the outbox should preserve the intent that was committed, or preserve enough immutable data to reproduce it without ambiguity.
A relay backlog is not merely an infrastructure metric. Its age tells operations how long downstream systems may have been acting on stale information. Its count tells engineering how much recovery work exists. Its business mapping tells the operations leader whether orders, refunds, stock movements or customer notifications are affected.
For teams building the relay and its deployment controls, DevOps and CI/CD engineering support is relevant only when it includes ownership of the operational evidence: deployment rollback, relay health, backlog visibility and recovery testing. A pipeline that deploys the relay without proving its failure modes has not solved the delivery problem.

Build the proposed outbox fault-injection matrix
Proposed template, not an OmniStack deployment record. A transactional outbox fault-injection matrix turns an architectural claim into an acceptance test. Each row names the injected failure, the state that must survive, the recovery behaviour, the evidence required and the limit of what the test proves.
- Run the test against the actual order, payment or fulfilment workflow in scope, not only a synthetic publisher.
- Record correlation identifiers for the business transaction, outbox row, broker message and consumer intent record.
- Capture durable state before recovery, during retry and after reconciliation.
- Assign an owner to every unresolved outcome, especially unknown provider responses and business-priority decisions.
Fault injection | Expected committed state | Expected outbox or relay state | Expected consumer behaviour | Acceptance evidence | Limit |
|---|---|---|---|---|---|
Crash after database commit, before publish | Order or payment state remains committed | Outbox row remains unpublished | No downstream side effect until relay resumes | Relay later publishes from the durable outbox record | Does not prove downstream business acceptance |
Relay publishes, then crashes before marking sent | Committed state unchanged | Outbox row may be retried | Consumer handles duplicate message using an idempotency rule | Duplicate attempt produces one accepted business effect | Does not prove duplicates are impossible |
Broker or API response times out | Original intent remains inspectable | Status is unknown until lookup, retry or reconciliation | Consumer does not infer decline or failure from missing response alone | Runbook classifies unknown outcome before retry | Provider contracts still matter |
Consumer processes event, then fails before acknowledgement | Source of truth remains committed | Message may be redelivered | Consumer recognises the same event intent and avoids repeating the external effect | Idempotency key and stored result are checked | Semantic equivalence must be defined |
Relay backlog accumulates during outage | Committed states continue to queue | Outbox age and count breach the workflow threshold | Operations decide whether to pause intake, catch up or reconcile | Backlog report links rows to business workflows | Matrix does not choose the commercial priority |
The first row is the minimum test for the pattern. Stop the application after the database commit and before the relay can publish. Restart the relay and prove that it finds the durable row. A trace showing an attempted publish is not sufficient. Stronger evidence is the committed order, the surviving outbox record, the later broker publication and the consumer’s recorded handling.
The second and fourth rows test a harder property. A duplicate can arrive after a real business effect has already occurred. The consumer must recognise the same intent and return the previously recorded result or perform no second effect. A successful second HTTP response does not prove this. The test must inspect the business state that would have been duplicated: one reservation, one refund instruction, one fulfilment command or one ledger transition.
The timeout row deserves separate treatment. A provider may have accepted a payment or fulfilment instruction even though the caller received no response. Retrying immediately can create a second instruction. The runbook needs a lookup, status query or reconciliation step before deciding that the first attempt failed.
The backlog row connects engineering health to operations. A relay may be technically healthy while the oldest outbox record is too old for the business workflow. The threshold belongs to the workflow owner. A stock event and a customer notification may not have the same tolerance for delay, but the matrix should make that decision visible rather than burying it in a generic queue alert.
Related:workflow automation engineering, useful when replay, exception routing and reconciliation need to become explicit operational workflows.
Define idempotency by intent, not by retry count
Idempotency means that repeating the same business intent produces one accepted effect, not that the system permits an arbitrary number of retries. The AWS Builders Library article on making retries safe with idempotent APIs is useful here because it centres on caller intent, client request identifiers, late-arriving requests and the problem of the same identifier being reused for a different request.
Apply that discipline carefully to events. An event identifier is useful, but it is not the entire business rule. A consumer needs to determine what the message means, which aggregate transition it represents and whether a late message is still valid.
Question | Weak rule | Stronger consumer contract |
|---|---|---|
What identifies a repeated event? | Compare a transport message identifier only | Use an event or intent identifier linked to the business aggregate and transition |
What if the same identifier has a different payload? | Ignore the second message silently | Reject, quarantine or escalate because the identifier has conflicting intent |
What if the message arrives late? | Process it because it is syntactically valid | Check version, state transition and business validity before applying it |
Where is the result recorded? | Rely on logs or broker acknowledgement | Store processed intent and the resulting business outcome in durable consumer state |
In a hypothetical fulfilment consumer, an order-created event may arrive twice. The consumer should not create two warehouse jobs. It may store the event identifier, order identifier, expected transition and the created job reference. If the same identifier arrives with a different warehouse location or amount, silent deduplication is unsafe. The consumer should move the message to an exception path and preserve the conflict for investigation.
Payment workflows make the distinction sharper. A retry to authorise the same order for the same amount may be the same intent if the identifier, order and amount match. A retry using the same identifier for a changed amount is not the same intent. The consumer or provider integration must reject or investigate it. A retry counter cannot make that semantic decision.
Versioning is another part of the contract. A late event may be a valid duplicate, an obsolete transition or an out-of-order message. Consumers should compare the event’s aggregate version or transition context with their current state. The acceptable action may be apply, ignore as already applied, quarantine or reconcile. Those outcomes should be visible in metrics and operational records.
Idempotency storage also needs a retention decision. Retaining a processed record for too short a period can allow a late duplicate to repeat an effect. Retaining it indefinitely may create operational and data-management obligations. The correct period depends on the maximum replay and late-arrival window for the workflow, and that window must be documented rather than assumed.
What evidence a CTO should ask for before accepting the release
A release should not be accepted because the outbox table exists. Acceptance requires evidence that a committed intent survives publisher failure, that duplicate delivery produces one accepted business effect, and that operations can see and resolve an unknown or delayed outcome.
Ask for the outputs of the proposed matrix, with the injected failure and recovery times recorded. The evidence should show durable records, not only traces. Distributed tracing can explain where a relay stopped or which attempt was made, but it cannot prove that a warehouse job was created once or that a refund was reconciled.
- Crash-after-commit output: the committed business row, surviving outbox row and later publication from that row.
- Duplicate-delivery output: two delivery attempts linked to one intent, with one accepted business effect and a stored consumer result.
- Unknown-outcome runbook: the lookup, retry, quarantine and reconciliation steps for a broker or provider timeout.
- Backlog visibility: count, oldest age, workflow type and business impact, with an owner who can decide whether to pause intake or prioritise recovery.
- Replay procedure: selection criteria, authorisation, dry-run or preview behaviour, audit record and post-replay verification.
- Known-gap register: unresolved edge cases, affected workflows, named owners and current decision status.
There is a useful distinction between observability and correctness. A dashboard can report that the relay is running while a malformed event is rejected by every consumer. A trace can show a publish acknowledgement while the consumer later applies the same refund twice. The acceptance pack should connect technical records to the business effect.
Use this decision rule:
- Accept when the in-scope matrix passes, duplicate handling is proven, backlog and replay are visible, and remaining gaps have named owners.
- Pilot when the outbox mechanism works but reconciliation or duplicate handling remains unproven for a high-value workflow. Limit the scope and keep manual review active.
- Defer when a committed state can still disappear from downstream operations without replay, alerting or a defined manual reconciliation path.
Teams modernising the surrounding application should also keep the transactional boundary narrow. Putting a remote API call inside the database transaction does not make that remote call atomic. It lengthens lock duration and still leaves the external provider’s outcome subject to timeout. Record intent locally, publish through a recoverable relay, and handle external uncertainty explicitly.
Where the transactional outbox is the wrong answer
The outbox pattern is not a universal repair. It is the wrong first move when the business event is disposable, when the source database is not the authority for the transition, or when the consumer cannot define what duplicate handling means.
A telemetry signal that can be regenerated may not need an outbox row for every observation. A search index update may be rebuilt from the source database if the rebuild process is tested and the temporary inconsistency is acceptable. A workflow that requires a provider’s synchronous decision before committing may need a different saga or reservation design, because an outbox cannot make the provider response atomic with the local transaction.
The pattern also fails as an organisational solution when nobody owns reconciliation. A durable outbox can preserve many failed intentions while the business remains unable to decide which should be replayed, cancelled or manually corrected. More storage does not create a policy.
There is a further failure mode in legacy modernisation. Teams sometimes add an outbox to a service but leave hidden side effects inside the transaction path: email delivery, stock calls, payment attempts or warehouse API requests. The outbox then gives a false sense of safety while the application still performs non-transactional work before the commit. Map every side effect and decide whether it becomes an event, a command, a lookup or a compensating action.
The approach is also a poor fit when the team cannot operate the relay. The relay needs deployment controls, retry limits, poison-message handling, schema compatibility, backlog alerts and recovery drills. If those capabilities are absent, a queue may simply move the failure into a less visible place.
In-house ownership wins when the existing team already has system knowledge, operational ownership and release authority, and can reserve capacity for the matrix, relay hardening and reconciliation work. A dedicated engineering team is appropriate only when the internal team understands the business but is consumed by daily operations and cannot give the failure path a named delivery owner.
Where a dedicated engineering team fits
The practical work is usually distributed across application engineering, data, QA, DevOps and operations: define the outbox schema, harden the relay, instrument the consumer, inject failures, verify duplicate effects and write the reconciliation runbook. A dedicated engineering team option can carry that work against the client roadmap when internal capacity is the constraint, provided the scope is the release evidence and recovery behaviour rather than open-ended staffing.
The boundary must be explicit. The client remains accountable for business policy, risk acceptance and the decision to pause, replay or manually reconcile a workflow. The delivery team owns the implementation evidence: fault-injection runs, consumer idempotency tests, backlog visibility, deployment controls and the handover materials that let the client operate the system.
For organisations extending a legacy platform, backend engineering support for service boundaries and event workflows may fit where the work includes durable state design and operational testing, not merely endpoint construction. For process-heavy environments, process automation engineering can help connect exception queues and reconciliation steps to the people who must act on them.
The decision should be made against the release gap. If the internal team can run the proposed matrix and own the remaining defects, keep the work in-house. If the matrix is repeatedly deferred because roadmap delivery consumes all available capacity, assign one delivery-owning team and make the acceptance evidence part of the release scope.
Sources and limits
- AWS Prescriptive Guidance: Transactional outbox pattern. Used for the mechanism of writing business state and outbox state in one database transaction, and for the limit that duplicate messages can occur.
- AWS Builders Library: Making retries safe with idempotent APIs. Used for idempotency concepts: caller intent, client request identifiers, late-arriving requests and identifier reuse with different intent.
- These sources support engineering mechanisms. They do not prove that any particular implementation is correct, accepted by a regulator, or complete for a specific APAC operating model.
Make the next release decision explicit
Start with one workflow where a committed state has material operational consequences: order creation, payment capture, refund, inventory reservation or fulfilment instruction. Draw the database transaction, outbox row, relay attempt, broker boundary and consumer effect on one page. Mark every point where the outcome can be unknown.
Run the proposed transactional outbox fault-injection matrix against that workflow. Do not accept screenshots of logs as proof. Require the durable row, the replay, the consumer’s idempotency record and the resulting business state. Inject the failure after commit before publish, after publish before marking sent, during an unknown broker response, and after consumer processing before acknowledgement.
At the review meeting, force three decisions into the record: what can be replayed automatically, what requires reconciliation, and who has authority to stop or prioritise the workflow during backlog recovery. If any answer is to investigate only after the failure occurs, the release still has an unowned failure surface.
Accept the release if the matrix passes for the workflows in scope and the remaining gaps have named owners. Pilot if the mechanism works but reconciliation or duplicate handling is unproven for high-value paths. Defer if a committed state can still disappear from downstream operations without replay, alerting or manual reconciliation.

