Reserve what the purchase will cost, not what the merchant asked for

#The situation

My rule for a card balance is simple. A wrong decline is tolerable: the cardholder tries again or pays another way. An overspend is not: every cent below zero is money the company has paid out and may never get back. The real-time decision is the only moment where the issuer can say no. Everything after it is bookkeeping.

Two merchant shapes break this rule routinely. At a restaurant the terminal authorizes the bill and settles the bill plus the tip. At an automated fuel dispenser the pump sends a 1.00 "status check" to confirm that the card works. The issuer approves 1.00, and the pump then dispenses fuel up to a limit set by the card network. The real amount reaches the issuer only afterwards, in an authorization advice. Under the card network's published rule, the issuer carries the liability for the dispensed amount up to a published limit far above the 1.00. The issuer can dispute only the part above that limit. A card with a small balance is exactly what that rule exposes.

#How it was found

Nobody had seen the problem. There was no alert, no report and no manual process for a purchase that settled above its reserve. That is my account of the time before the work.

I noticed the gap while building test coverage for the money paths. My rule for those tests is that every message from the card processor is a real one. It is captured from real traffic, never written by hand. A captured restaurant lifecycle settled above its authorization. The test that replayed it showed the account below zero with nothing reported. I wrote failing tests on captured messages for each shape of the gap before I designed anything.

Then I measured, on replayed real traffic over twelve weeks. About 2.6 % of all settled purchases settled above their authorization. At tip merchants the share was far higher. It was about 8.6 % at restaurants and 3.1 % at fast food. At taxis it was 22.1 %, at bars 25.1 %, and at barbers and beauty salons 20.0 %. Measured on the same traffic, even a 30 % reserve leaves a tail. A reserve lowers how often a balance goes below zero; it cannot remove it. So a log line for every remaining case is needed whatever the reserve is.

The fuel finding was different. The application had no fuel rule at all. In the replayed traffic, more than half of the fuel-dispenser attempts were exactly 1.00 status checks.

An older system confirmed the shape with real money. Its report for the sponsor bank lists every posting that took an account below zero. When I asked the business for evidence of the loss, that report was the answer. Over ten months it listed many card postings that took an account below zero. Restaurants came first and ATM withdrawals second, then fast food, taxis and service stations. Half of the crossings were under 3.00 and nine of ten were under 25.00. Nothing in that report came from the platform I worked on. It showed that the shape I had measured on replayed traffic costs real money with live cardholders.

#What a settlement above the authorization looks like

Two examples, with round amounts, shaped like the captured messages.

A restaurant authorizes 10.00 on an account that holds 10.00. The decision approves and reserves 10.00. The settlement arrives for 13.00. The system books 13.00, because it has no choice, and the balance reads −3.00. Nothing else happens.

A fuel dispenser sends a 1.00 status check on an account that holds 1.00. The decision approves and reserves 1.00. An authorization advice arrives for 60.00. The balance reads −59.00. Nothing else happens.

#Constraints

I solely own every technical decision on this work. AI sessions were my tool for research, code and tests. I chose between the approaches the sessions proposed and rejected several. A separate AI session that did not write the code reviewed it.

Every end-to-end test replays a captured processor message, byte for byte, with only identifiers scrubbed. The fuel status-check request did not exist anywhere in our records. So I first logged the real-time request and captured the real one from it. I turned the capture into a test message, and only then designed the rule. Unit tests and property tests cover the arithmetic separately.

The ledger lives in PostgreSQL. Money writes must be safe under concurrency by their SQL shape, never by discipline. A balance is changed by a relative update. A decision is made only from values read after the row lock. Schema changes must never block the running service. Replayed traffic cannot exercise the real-time decision, so only a card run against the running service can exercise the full fix.

One processor fact decided where to act. The processor declines a purchase on its own if the real-time decision does not answer within a few seconds. So every approved purchase has passed through our decision. The decision is the one place that can reserve more.

#Options and decisions

How much to reserve at tip merchants. The card networks publish a tip tolerance above the authorization. I considered one global rate of 25 %, a rate of 30 %, and a rate per merchant category. I chose one global 25 % at five categories: restaurants, fast food, taxis, bars, and barbers and beauty. It is a deliberate average, changeable in one place. Per-category rates would have been tuning before measuring. The captured restaurant tip was 30 %, so it exceeds the reserve by 0.50 on a 10.00 bill. The report catches that case, which is the point of having one.

What to do about fuel. The options were a fixed reserve on every status check, a reserve capped at the account balance, or declining every status check. Declining all fuel blocks a legitimate purchase for every cardholder. Capping at the balance approves the 1.00 and then lets the pump run to whatever the balance is. That is the same liability shape with a different number. I chose a fixed buffer on top of the 1.00, released as soon as the authorization advice with the real amount is stored. The consequence is accepted: a card whose balance or daily limit cannot cover the fixed buffer cannot start fuel at a dispenser. I set both numbers as an interim choice, changeable in one place, and the business later confirmed them.

Decide once, store it on the purchase. The reserve is decided when the purchase is first seen, at the decision or at the first transaction message, and stored on the purchase record. Later messages read the stored value and never compute a new one. Two facts forced this. First, the processor's authorization advice can carry a different merchant category than the original authorization, so the category cannot be read again later. Second, an experiment showed a pending purchase being given a new buffer when the rate changed between the decision and the message. That would move money for a purchase that was already approved under the old rule.

def buffer_for(first_message, account):
    if first_message.merchant_category in TIP_MERCHANT_CATEGORIES:
        return Buffer(rate=Decimal("0.25"), fixed=0)
    if looks_like_fuel_status_check(first_message):   # category, authorization, hold == 1.00
        return Buffer(rate=Decimal("0"), fixed=FUEL_BUFFER)
    return Buffer(rate=Decimal("0"), fixed=0)

def on_first_seen(purchase, message, account):
    purchase.buffer = buffer_for(message, account)     # stored once
    reserve(account, message.hold + purchase.buffer.applied_to(message.hold))

def on_later_message(purchase, message, account):
    buffer = purchase.buffer                            # read, never recomputed
    if message.has_authorization_advice:
        buffer = buffer.without_fixed()                 # the real amount is known now
    re_reserve(account, message.hold + buffer.applied_to(message.hold))

Where "advice seen" comes from also mattered. The question "has the real amount arrived?" is answered from the stored event history, never from the current message alone. The current message's events are stored first. Every message then decides from the same history, whatever it carries.

Where the rule lives. One policy module decides a new purchase's buffer, and both the real-time decision and the posting engine call it. Before, the rate was computed in four places that could drift apart.

Log every below-zero result. I considered logging only at settlement and logging after every message. I chose every message: pending, advice and settlement. A fuel advice above the reserve is known well before the settlement. The below-zero result is written as a structured log line with the balance before and after. The line is written only when this message lowered the available balance and left it below zero. So one debt is not logged again by every later message, but a deeper debt is. The check reads the balance before and after through one shared formula, with one column read each. Balance writes are SQL additions, and an in-memory object does not hold the number until the flush. The line is written after the commit. A log line or an alert is an external call. An external call never runs while a row lock is held, and never before a commit that may still roll back. A failing write never fails the posting.

def after_commit(result):
    before, after = result.available_before, result.available_after
    if after < 0 and after < before:        # this message lowered the balance below zero
        log_below_zero(result)              # a failure here never reaches the posting

Processor-side rules rejected. The processor can also adjust holds by its own rules. I kept the whole buffer rule in the application. One place to read, one place to test, no second system that can disagree with the ledger.

#The approach

The order of work mattered more than the amount of code.

1. Log the real-time request so the fuel status check can be captured. Capture it and six other request shapes as test messages.
2. Make the real-time decision check, report and reserve one amount, hold plus buffer plus fee, with the rate stored per purchase. The buffer counts toward the daily limit.
3. Add the below-zero log line on every path that books a message, including the recovery of missed messages.
4. Add the fuel buffer and move the whole buffer policy into one module. Add the two database checks that the buffer is never negative, and the index the advice lookup needed.
5. Drive a card through the full fuel lifecycle against the running service.
6. Validate the two database checks in the next release.

#Hard parts

#The same purchase looks different on the two paths

The card processor describes one purchase in different words on the real-time request and on the transaction message. On the decision path a fuel status check is "a real-time request at a fuel category with a hold of exactly 1.00". On the message path it is a pending purchase at a fuel category with a hold of exactly 1.00. Its first recorded step must be the authorization, with no advice yet. I matched each path on its own fields and never mixed the two vocabularies.

#Adding a CHECK constraint without locking the table

The purchase table in PostgreSQL was large enough that a full scan under a write lock would stop the running service. Adding a CHECK constraint normally scans every row under exactly that lock. I added both constraints as NOT VALID in one release. That takes the lock only for a catalog change and enforces the rule on new rows at once. I validated them in the next release. Validation reads the old rows under a weaker lock that lets reads and writes continue. The migration connection got a five-second cap on lock waits, so a migration that cannot get its lock fails fast instead of queueing behind traffic. The result: the database now guarantees that buffer is never negative, validated on every row.

-- release N: enforce for new rows, do not scan the table
ALTER TABLE card_purchases
  ADD CONSTRAINT card_purchases_buffer_non_negative CHECK (buffer_fixed >= 0) NOT VALID;

-- release N+1: scan once, under a lock that lets reads and writes continue
ALTER TABLE card_purchases VALIDATE CONSTRAINT card_purchases_buffer_non_negative;

#A lookup under a lock

The "has the advice arrived?" question read the purchase's stored history. That table was large and had no index on the purchase identifier. The lookup was a sequential scan, roughly 130 to 510 ms measured twice against the running service. It ran while the account row was locked, so every other operation on that account waited for it. An index on the purchase identifier fixed it. On a copy of the table of the same size the same lookup went from 28 ms to well under a millisecond.

#Before and after

#Result

The work is merged, released through the delivery pipeline, and verified end to end against the running service. In that run a 1.00 status check at a fuel category reserved the 1.00 plus the fixed buffer. A 60.00 advice released the buffer and held 60.00. The settlement booked 60.00 and released the rest. An advice above the balance produced exactly one below-zero line. An advice that left the balance at exactly zero produced none. A status check on an account already below zero was declined. Three concurrent status checks on one account were approved because the balance covered three reserves. Of two more on a smaller balance, exactly one was approved. As recorded during the run, the balances matched the journal before and after.

For a cardholder this means that at a restaurant or in a taxi the approval reserves a little more than the bill. At a fuel dispenser it reserves a fixed amount until the pump reports what was dispensed. A purchase the balance cannot cover is declined at the approval. On the platform I worked on, the money at risk was measured on replayed traffic. On the older system, the report for the sponsor bank counted the real losses.

#Lessons

• Pin the gap with a failing test on a captured message before designing the fix. The test fixes the facts; the design argues against the facts, not against an opinion.
• A money rule is decided once and stored on the row it governs. Anything re-read later from a value that can change will move money for a decision already made.
• Add a constraint in one release and validate it in the next. The split costs one more migration and saves the running service from a table scan under a write lock.
• A log line or an alert is an external call. It runs after the commit and never under a row lock, and its failure never fails the posting.
• Measure on data you can measure again. The first measurement I made used a denominator I could not reproduce later. The numbers in this article are the ones that reproduced.

#What this shows

• Measuring a money leak on real traffic before designing the fix, and letting the measurement choose the scope.
• Deciding a money rule once, storing it with the row, and making the database guarantee it without blocking the service.
• Proving the fix end to end on captured messages and a real card run against the running service.