One of my first consulting projects was working on the call center software for a large health insurer. The client was implementing a massive core system replacement and various satellite applications, lie the call center software, needed to retrofitted to call data from both. As this was the early 2000s, the buzzword de-jour was "service oriented architecture (SOA)" a series of web services on an "Enterprise Service Bus (ESB)" that would of the provide data to the affiliated applications from either the old core or the new.
Large Enterprises Forget Large Things
The centerpiece service on the ESB was the router. It's job to to find members and identify which system their data resided in as of which date and provided keys that could be used in subsequent ESB service calls. The router was in many ways amazing. Built with full Oracle Java stack running on the latest IBM blade servers. It easily could churn through a million transactions without issue. There was one problem. . .
The router was invoked with a date and a member ID which it used to identify whether members records were in the new system or the old system. But members don't always call the call center with their ID, about 2/3 of call center calls use name and date of birth in a search to find the member. As designed without an ID (either from the old system or the new system), the router couldn't identify which system the member would be in and couldn't provide the correct directions to other services. This issue was first identified when I couldn't complete a design a document mapping call center data fields to XSLT templates for the ESB web services. The router was in testing but not yet in production. This problem should have been caught in requirements but at the end of day it hadn't escaped to production or even to user testing.
Never Assume Your Use Cases are Known
How was this missed? Well, the most critical transactions for a health insurer are the automated claims submissions from various hospitals arriving by the thousands per hour. These largely have identifiers and are specific dates of service. The router processed these transactions quickly and efficiently and many of the engineers working on the project had extensive past knowledge working in these processing data pipelines. The most obvious customer service use case just wasn't considered.
I tell new people starting on my teams never to assume that I am aware of even the most obvious problems and try to always come with a solution. My design document did propose a solution one I had to present to a risk management review. The call center software could not use the router and instead issue two calls to a related search service with dummy dates and values that would ensure that either got two response (one from the old system and one from the new) and would allow us to decide how to move forward. My proposed solution had a few problems in it and wasn't adopted but thats fine.
Software projects are complicated and chaotic but no problem is truly unsolvable. There are bad solutions that hardcode core system logic in code, or create massive technical debt like mine but there are always solutions. It's just key to be the one participating and moving the discussion forward to the solution not just pointing out problems waiting for them to be fixed.
This is one of those posts where I write about behavior I strive to achieve but absolutely fall short of. I’ve worked on projects in miserable situations where honestly you can’t help but laugh when some major issue causes a derailment. But, this is a bad cycle to fall into. Even on the most hopeless projects, there is always a solution a way to break out and put things on solid footing.