Summa

Sometimes I get asked, “So, what do you do for a living?”.  This is a pretty normal question and even a bit flattering that the questioner is interested in me, although I know it will be a short conversation.  Here’s how it might continue.

“I develop software”.  “Oh really?  What kind? Games?”.  “Business software, enterprise level, usually in Java”.  This is usually the end as the person was just trying to get to know me a little, and it is apparent that this isn’t going anywhere familiar or comfortable.  

But every now and then the person will persist. “I mean, so what’s your day consist of?  Like, what do you do when you get to work.  Do you talk to people or write something or what?”.

This is when I know the conversation is about to end.  In order to explain what I do to develop software, I have to become very, very technical.  There are no metaphors in every day life that can really help.  Even the vocabulary will be unfamiliar.  The best solution is to spill my beer on myself, or fake a heart attack.

The odds are very good that the person asking is a User of software.  They are good at something, but it is not software development.  They could be bankers, doctors, logistics managers or advertising mavens.  They are smart people, but there is no way that they can understand what is involved to develop the software tools to help with their every day tasks.  This makes every conversation between us awkward as we try to develop software tools that will make them more efficient and competitive.

Users, particularly those buying software, do understand budgets and schedules.  

If they’re shopping for software that has an off-the-shelf solution, then they are on comfortable ground.  They have feature lists, price tags, user reviews, etc.  A buyer can make an informed decision about the cost-benefit of each product as applied to the need.  There is no need for the buyer to concisely specify their need, only to find an affordable product that does at least as much as needed.

The Gap

But when there are no off the shelf products and custom software must be developed, they find themselves in a swamp of ill-formed questions and unintelligible answers.  “How much will it cost and how long will it take to solve this problem?”.  And they never like the only possible honest answer that starts with, “Well, it depends…”.

So how do we manage this communication chasm?  We all agree that having some software tools is good for the business.  We all agree that there is a cost to develop custom software.  But as we go deeper, we don’t understand much.  The developers don’t understand the business domain and the business folks don’t understand the software domain.  

Devil in the Details

One way to look at this is by observing the growth in the number of details over the course of development.  The details can be categorized as Business Domain or Solution Domain.  Another cut is to think about who understands the detail; the Business User or the Solution Developer or both.  The number of details is grossly different between the Business and Solution Domains.  I think it looks something like this:

Project Phase vs Number of Details vs Understanding

As the project advances from business summary to requirements to design to implementation, the number of details that must be understood increases geometrically.  Notice that the Details axis is logarithmic.

However, the number of details that are understood by both the business users and the technologists flattens out just below the requirements.

We all expect the requirements to be the solid foundation for the expansion of details that follows (design and implementation), but that usually runs into some problems:

• Users are not in the habit of concisely describing what they want or what they do (this is the Business Domain).  MacDonald’s® tries to solve this with a whopping big procedures manual, but most non-franchised businesses do not.
• The developers do not understand many requirements in the way the users mean them because there are tacit implications in the Business Domain where developers have no basis of understanding.
• Because developers assume their understanding of the requirements is good, say 90%, they do not realize that many of the strategies/tactics for design and coding are based on incorrect understanding of the requirements.
• It is well understood that errors are much more costly to correct at later stages of the project.  The 25-50% of the requirements that are not correctly understood carry a significant budget and schedule cost. The cost is roughly proportional to the number of details at the point of correction less the number of details when the error occurred.

This is a hypothetical, so the particular numbers aren’t important. But experience tells me that the trends are correct.  The developers will understand 50-75% of the detailed requirements and that may grow a little as the project proceeds.

The business people understand all the details at the requirements stage, but don’t realize that every requirement expands into 10-100 details in the implementation.

Command and Control Approach

A common approach is based on the flawed assumption that all development can be perfectly planned like building a house.  It assumes past projects failed (or ran over budget/schedule), because the users were not faithful when they created the requirements.  This project management approach will insist the requirements team works harder and make the users sign the requirement document.  The idea is that users will take this more seriously if they have to sign it.  

This fails, but you won’t know it until you put the finished product in front of them and they say, “this isn’t what I expected!”.  This happens even if the requirements are very detailed because the developers must make assumptions about the 9,900 details that the users never knew existed.

I’m not suggesting that the users should learn to understand the 9900 details.  If they did they would be software engineers and would not have time to be business domain experts.

The Agile Approach

A better approach is found in agile development methodologies where the Business Users and Solution Developers work together closely throughout the project.  These techniques address the detail understanding gap in several ways:

• Deliver in iterations so the user can see if the developed fraction of product meets their needs.  This provides feedback without spending the whole budget.
• Keep the user involved day-to-day to answer questions so that developers’ assumptions are based on the users’ needs rather than the developers’ assumption of the users’ needs.  This also overcomes the issue that users are not concise; the developer can ask very specific questions that elicit information from the user that they would not have thought of without prompting.
• Design and build only what is necessary to satisfy the current iteration.  If developers try to plan too far ahead, they will over-build.  This is wasteful because some of that effort will not be needed and because changing direction, even slightly, will be impeded by resistance to scrap that development effort.
• Deliver tests with each iteration of code.  Having good tests will make changing directions easier because it makes refactoring the code easier.  Also, the tests are a restatement of the requirements, as understood by the developers.  It causes the developers to explore the requirements more thoroughly and prove the delivered code satisfies the requirements.