Why Your Bank App Says "Successful" Before the SMS Arrives
When you drop your clothes at the dry cleaner, nobody asks you to wait at the counter while they wash, dry and iron everything.
They hand you a ticket and say, "We'll call you when it's ready."
You go about your day. The work still happens. It just doesn't happen while you stand there waiting.
That simple idea is behind one of the most important patterns in modern software. It's the reason your banking app can say "Transfer successful" in about a second, while the SMS alert, the receipt and the rewards points arrive a few moments later.
Engineers call it asynchronous processing, and the tool that makes it work is the message queue.
In this article, I'll explain what a message queue is, why payment systems depend on it, and the rules experienced engineers follow so that nothing gets lost in between.
What must happen now, and what can happen next
When you tap "Send" on a transfer, a lot happens. But not all of it needs to happen before you see a result.
Some things must happen immediately:
- Check that you have enough money
- Debit your account
- Record the transaction in the ledger
- Confirm the result to you
Other things can safely happen a moment later:
- Send the SMS or push notification
- Generate the receipt
- Update rewards or loyalty points
- Feed fraud analytics and reporting
If a payment service did all of this in one long chain, every transfer would be as slow as its slowest step. And if the SMS provider had a bad day, your transfer would fail even though the money part worked perfectly.
So instead, the payment service does the critical work, then drops a message into a queue that says, in effect, "A payment just completed. Whoever needs to know, here are the details." Then it moves on.
Other services pick up that message and do their part in the background.
How a message queue works
There are three main roles:
- Producer: the service that publishes a message. Here, the payment service.
- Broker: the system that stores messages safely until they're processed. Apache Kafka and RabbitMQ are the most common.
- Consumer: a service that reads messages and acts on them. The SMS service, the receipt service, the rewards service.
[Payment Service] --publishes--> [ payment-completed queue ] --> [SMS Service]
(must happen NOW) --> [Receipt Service]
--> [Rewards Service]
--> [Fraud Analytics]
(can happen NEXT)
The beauty of this design is that the producer doesn't need to know who is listening. Tomorrow, the team can add a new consumer, say a monthly spending insights service, without touching the payment service at all.
That's called loose coupling, and it's one of the reasons large systems can keep growing without collapsing under their own weight.
The rules experienced engineers follow
Queues make systems faster and more resilient. They also introduce new ways for things to go wrong. Here are the rules I hold teams to when building this way.
1. Make every consumer idempotent
Most message brokers guarantee that a message will be delivered at least once. Not exactly once. At least once.
That means the same message can arrive twice: after a network hiccup, a consumer restart, or a retry. If your consumer isn't prepared for that, one payment can turn into two SMS alerts. Annoying. Or one credit can turn into two credits. Expensive.
An operation is idempotent if doing it once or ten times gives the same result. The common way to achieve this is to record every message ID you've processed, protected by a unique constraint:
@KafkaListener(topics = "payment-completed", groupId = "sms-service")
public void onPaymentCompleted(PaymentCompletedEvent event) {
// Insert the event ID into a table with a UNIQUE constraint.
// Returns false if this event was already processed.
boolean firstTime = processedEvents.markProcessed(event.eventId());
if (!firstTime) {
log.info("Duplicate event {} ignored", event.eventId());
return;
}
smsService.sendPaymentAlert(event.customerPhone(), event.amount());
}
For anything that moves money, this isn't optional. One credit must never become two.
2. Retry with backoff, then park it in a dead letter queue
Sometimes a message fails because of a temporary problem: the SMS provider is slow, the database is busy. Retrying makes sense.
But retrying instantly and forever just hammers a system that's already struggling. So we retry with exponential backoff: wait one second, then two, then four, then eight.
If it still fails after a few attempts, we don't let it block every message behind it. We move it to a dead letter queue (DLQ), where it can be inspected and replayed later.
In Spring Kafka, that looks like this:
@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
var recoverer = new DeadLetterPublishingRecoverer(template);
var backOff = new ExponentialBackOffWithMaxRetries(4);
backOff.setInitialInterval(1_000L); // 1 second
backOff.setMultiplier(2.0); // 1s, 2s, 4s, 8s
backOff.setMaxInterval(10_000L);
return new DefaultErrorHandler(recoverer, backOff);
}
Failed messages land on a topic like payment-completed.DLT, and nothing behind them gets stuck.
3. Choose your keys carefully when order matters
Kafka splits a topic into partitions so it can process messages in parallel. Order is only guaranteed within a partition.
If events for the same account must be processed in order (a debit before a reversal, for example), give them the same key. Kafka sends every message with the same key to the same partition.
kafkaTemplate.send("account-events", event.accountNumber(), event);
// ^ key: same account, same partition, same order
Pick the wrong key, or no key, and two events for the same account can be processed out of order.
4. Watch the backlog
The danger with background work is that it can fall behind quietly. Customers won't complain about a slow queue. They'll complain that their alert never came.
So monitor consumer lag: the gap between the latest message published and the latest message processed. If lag keeps growing, consumers can't keep up, and you want to know before your customers do.
5. Don't lose the message between the database and the queue
Here's a subtle trap. The payment service writes to the ledger, then publishes a message. What if the database write succeeds and the publish fails? The money moved, but no SMS, no receipt, no fraud check.
The transactional outbox pattern solves this. Instead of publishing directly, the service writes the ledger entry and an "outbox" row in the same database transaction. A separate relay reads the outbox and publishes to the queue.
BEGIN;
INSERT INTO ledger_entries (...) VALUES (...);
INSERT INTO outbox (event_id, topic, payload) VALUES (...);
COMMIT;
-- A relay process publishes outbox rows to Kafka, then marks them sent.
Either both are saved, or neither is. Nothing gets lost in between.
When not to use a queue
Asynchronous processing isn't free. It adds moving parts, makes debugging harder, and means data is briefly out of sync.
Don't make something asynchronous if the user genuinely needs the result before continuing. Checking the balance and debiting the account must happen synchronously. A customer can wait a few seconds for an SMS. They can't wait a few seconds to find out whether their money moved.
The skill is in drawing that line well.
What the customer feels
Customers never see a queue. What they feel is:
- "Transfer successful" in about a second
- The alert and receipt arriving shortly after
- An app that keeps working even when one part of the system is having a bad day
And when it's done well, nothing gets lost along the way.
Key takeaways
- Split work into what must happen now and what can happen next.
- Message queues let services communicate without waiting on each other.
- Delivery is at least once, so consumers must be idempotent.
- Retry with exponential backoff, then move failures to a dead letter queue.
- Use message keys to keep related events in order.
- Monitor consumer lag.
- Use the outbox pattern so a database write and a published event can't drift apart.
Good engineering isn't doing everything at once. It's knowing what must happen now, what can happen next, and making sure nothing gets lost in between.
Just like a good dry cleaner never loses the ticket.
Matthew Odumosu is a Senior Software Engineer with 8+ years of experience building payment, banking and insurance systems. He writes about engineering principles in plain language, for engineers at every level.
Connect on LinkedIn: linkedin.com/in/matthew-odumosu
Tags: Software Engineering, System Design, Kafka, Microservices, Fintech
Top comments (0)