The subscriber paid, and the router still has them suspended

In plain terms: a payment is applied once, against a transaction id. That same id updates the subscription the access server reads. A provisioning job pushes the router to match. If any step is retried, it must recognise that it already happened. The database is the source of truth. The cache can be rebuilt. The router cannot be the place you look up who has paid.

The callback came twice

Mobile-money providers retry. A timeout on your side is not a timeout on theirs. The second notification will create a second payment if the handler is “insert a row and acknowledge.” The subscriber’s balance jumps. Finance reverses it by hand. The next retry does it again.

The handler verifies the notification is really from the provider, then looks up the transaction id. If that id is already applied, it acknowledges and stops. Acknowledgement before verification is how a forged callback opens service. Verification after you have already inserted the payment is how you double-charge and then try to clean up.

Paid in billing, rejected on the access server

The invoice is paid. The access server still rejects the username, or the opposite: the session is up and the invoice is unpaid, because someone enabled the user in a panel that does not post a payment. Those screens have to read and write the same subscription state.

Disconnect has to be a real request to the access server, not a flag in the billing database. A flag the router learns about at the next interim update leaves the customer online for the rest of the evening and then support argues about a session the billing screen says is closed.

Provisioning that runs halfway

Creating a secret, a queue, or a firewall rule on the router is several calls. The job dies after the first. The queue retries and creates a second secret. The customer has two credentials, one of which nobody wrote down, and a later “suspend” only removes the one the database remembers.

Each change is one job with one id. Claiming the job is atomic. Running it twice converges on the same router state. If it cannot finish, it goes to a dead-letter queue that a person can see, not a silent retry loop. Alerts fire on that queue, on failed payment callbacks, and on access-server latency, with one request id carried from the API through the queue, the callback, and the router call. Without that id, the three logs cannot be lined up and the incident becomes a story.

Whose subscribers are these

An ISP platform that hosts more than one operator has to refuse a user of operator A opening operator B’s customers. That check sits on every subscriber query, not on the menu. An administrator who is allowed to cross that line is a named role, not a missing filter.

Payments, logins, failed logins, and provisioning writes land in an append-only audit. The audit is not the operational record you edit. It is the record you cannot edit. Passwords, router credentials, and provider secrets stay out of the repository and out of the chat where the deploy was planned.

What we test before it takes a payment

• The same callback, delivered twice, one payment.
• A callback that fails verification, no payment, no service change.
• A paid invoice that the access server will accept, and an unpaid one it will reject.
• A disconnect that drops the live session, not only the row in the database.
• A provisioning job killed in the middle and run again, one set of credentials on the router.
• A user of one operator requesting another operator’s subscriber, refused.

Backups matter the day the database is the source of truth: a daily copy, a way to restore to a point in time, and a drill that has actually been restored. A cache that rebuilds from that database is a convenience. A cache that is the only copy of who has paid is an outage.