A specialist publisher: subscription fulfilment moved from weekly batch to live
A specialist academic publisher in Oxford was processing subscription changes in a Thursday evening batch. New subscribers waited up to a week to receive what they had paid for.
The reading
A specialist publisher of academic journals and reference works, based in Oxford, in continuous operation since the 1950s. Twenty-eight staff, four journal titles, around 11,000 institutional and individual subscribers worldwide. Fulfilment was handled by a long-serving operations team of four.
Subscription changes - new subscribers, cancellations, address changes, access upgrades - were processed in a Thursday evening batch run that had, we were told, existed in roughly its current form since 1987. The batch read from a submissions mailbox and a web form, updated the subscriber database, and triggered the access and dispatch processes.
A subscriber who paid on a Thursday morning received access on Thursday night. A subscriber who paid on a Thursday afternoon received access the following Thursday. The gap was invisible until a university librarian complained about it in a very polite email.
The Thursday batch
The batch had worked for thirty-seven years. It had worked because the operations team was careful, the Thursday evening was predictable, and the subscriber base had grown slowly enough that the batch could handle the week's volume. The reasons the batch had existed in the first place (overnight compute cost, mainframe scheduling) were long obsolete.
The operations team was aware the batch was anachronistic. They had raised it at the quarterly meeting on three occasions. The directors had, each time, agreed and taken no action, because the batch worked.
- 11,000 subscribers, weekly Thursday batch
- Average 3.5-day lag from payment to access
- Worst-case 7-day lag for Thursday-afternoon subscribers
- Legacy batch script dating from 1987, in active use
- Operations team spending Friday morning on exception handling from Thursday's batch
What we built
We replaced the batch with an event-driven fulfilment process. A payment triggered an immediate update to the subscriber record, which triggered access provisioning, which triggered the dispatch instruction for the next scheduled print run. The operations team saw each event in a live queue instead of a Friday morning exception report.
The technical choice we weighed longest: whether to keep the Thursday batch as a safety net. We did, for the first six months. Each Thursday the batch ran, found that everything was already up to date, and logged a zero-row result. The operations team found this reassuring; it allowed them to trust the new system before relying on it. We retired the batch in month seven at their suggestion.
Stack: a Node service with a message queue (RabbitMQ), Postgres for the subscriber database, event hooks into the existing print and access management systems.
The numbers at ninety days
| What | Before | After |
|---|---|---|
| Payment-to-access lag | 3.5 days avg, 7 days worst | <2 min |
| Friday morning exception handling | 4 person-hours | 20 min |
| Librarian complaints about access delay | 6/quarter | 0/quarter |
| New-subscriber first-session rate | 61% in week 1 | 89% in week 1 |
| Operations team confidence in system | "The batch is reliable" | "The queue is live" |
The anachronism retires
The Thursday batch had been correct in 1987 and had become incorrect, by degrees, over thirty-seven years. The firm had never had a specific reason to replace it, which is often how long-running processes outlive their rationale.
The system-first point is that the batch had not failed; it had merely stopped being right. One of the questions we ask of long-standing firms is which of their processes were correct when they were introduced and are no longer correct now. The one-week training test applies: a new member of the operations team joins and sees a live queue, which is intuitive. They would have needed a week just to understand the batch.