Agile or Waterfall — Aren’t We Just Struggling With Requirements Understanding?

Who is this article for?

All software development professionals will find interest in this article, but those managing, crafting or analyzing requirements may find the most interest.

Introduction

Why is so much ceremony needed around software development?  Regardless of the methodology there is a great deal of meeting, clarifying, verifying, demonstrating, and checking.   If you came from another planet and watched how we develop software you’d come to the conclusion that we don’t know how to communicate complex abstract ideas into concrete reality.  And we don’t. To be sure, we’ve tried:  detailed specifications, UML diagrams, use cases, user stories, and just plain discussion.  All of these are incremental changes to a better requirements landscape, but fleshing out our software needs still takes considerable time and effort.

Process and Methodology as Requirements Risk Mitigation

BDUF ( Big Design Up Front ) was the waterfall method to define requirements.   It was a way of saying….”Let’s get everything straight BEFORE we start the expensive process of coding and testing all this.”    It’s logical, but makes the broad assumption that people will know what they want and deviation from the initial plan will be minimal.   There are some projects where this is a very safe assumption, but not always.

Agile capitalized on this fault and said “You can’t know everything up front, so just get started and course correct along the way.”   Does this mean that agile projects always succeed? No.  It means that they allow adaptation to change.  Projects where innovation is important or where the product owner is unfamiliar with developing software naturally make good candidates with Agile.  But its pricey and if you don’t need all the involvement…..then going back to waterfall might be a good option.

With both methodologies you see an attempt to address the real problem:  understanding and materializing requirements so that we get exactly what we want.   The abstract nature of software means we’re collectively imagining what it could and should be.  But we haven’t found a silver bullet for this type of collective dreaming.

Communicating and interpreting our desires is not as easy as it seems.  Some of us are good at drawing pictures…..but most of us aren’t.  To mitigate the possibility that we’re not going in the right direction we’ve invented these elaborate software processes ( Waterfall & Agile …..and the pragmatic others ) to keep our collective minds on the same path to vision creation.

Are  There Better Ways?

Even with these processes we still end up in projects where we hear these things said:

“Oh…that’s not what I meant.  It should do this…”

“No, no,no….you didn’t understand what I was saying. “

“That wasn’t the requirement.  Read it again….I built what was asked for.”

“Yes. I know she said that….but what she really meant was…”

So, software requirements are still an error prone bottle neck in producing the product.  If we hope to make software development better;  then we must get better at this.  In paraphrasing Eliyahu Goldratt in The Goal: You can only run as fast as your slowest player.  That player for software is requirements gathering, analysis, understanding and verification.

Looking outside our field, other industries, who’ve been around longer and develop complex systems too, have similar challenges.  What do they do?

Design –  My wife is a talented contract graphic designer for a local St. Louis firm called Kuhlmann-Leavitt.  They’ve won many awards and have very satisfied customers.  All this despite the nebulous, subjective nature of graphic and three dimensional design.   How do they do it?  The design field is interesting.  They don’t estimate anything.  They simply ask that you hire them for your artistic work and set a budget.  They’ll do the best design work they can for that money.  No more, no less.   So right away the budget issue is taken out of the picture.  Lastly, they don’t ask for detailed requirements,  just some idea of what you’re looking for.  You’re not hiring Kuhlmann-Leavitt to build.  You’re hiring them to create.  My bolding hints at the nuance.  Considerable authority and autonomy is given to the designers.  This is just expected.  Software development relies much more heavily on ‘domain experts’.

Manufacturing – In the manufacturing world BDUF could be rewritten to be MDUF ( Massive Design Up Front ).  Not only is the product designed way up front and in detail, but the process and mechanisms to build the product, which aren’t always the same, are designed up front too.  Before the real product even undergoes one production run…many small demo runs are carried out.  The results are stunning.  Once the product starts rolling off the assembly line its produced to exacting tolerances.  It mostly fits the requirements of marketing and can be replicated thousands of times over.

Construction – We’re talking about unique, one of kind construction here.  Not production line assembly of a subdivision.  This one-off building is, to my mind, the most similar to software.  Some requirements come from regulations and building codes.  Others come from the purchaser of the structure.  There are multiple stakeholders.  Requirements are digested, and then artifacts are created.  A series of mock-ups are assembled and based on this a cost estimate is created.  Even with all the up-front planning there are always “gotchas” and changes that are thrown into the project.

In all three of these examples the end product is visual and intuitive which helps with requirements understanding.  A picture really is worth a thousand words.  Software, as workflow, data and creative design is more abstract.  We’re trying to craft ideas and concepts into something that is somewhat tangible.  Certainly the user interface can be mocked in a similar way and there is great value that comes from the discussions around this, but a sufficiently detailed workflow and data set always serve to vex our attempts to provide clarity to requirements.

What’s next?  What can we do?

Here’s part of our problem:

1. We discuss the first iteration of requirements with our product owners.  The person leading that discussion is gifted in facilitating this.

2. From this meeting, the requirements are crafted into user stories.

See it?  We just lost all the context, emotion, body language, nuance, and reactions by everyone else by drafting something into writing.  I don’t care how good a writer you are…you can’t replicate the requirements conversation.  So why do we try?

The next problem comes when you ask each person to interpret the written user stories.  Hopefully each team member was present for the conversation so they could somewhat remember what transpired.  But, even with attendance, the human memory is frail and human perceptions, bias and concentration vary. Each person may see and read the requirements slightly different.  In short there are too many lenses.

Rather than try to abstract requirement conversations into another form ( writing ) or create even more abstract artifacts ( UML, Use Cases ) we should seek to record them via video, photographs of whiteboard drawings, and publishing all this to some kind of requirements wiki or blog.  Doing this for each iteration review, iteration planning, or project meeting would surely capture a much more complete document of the project’s scope, changes, stakeholder motives and historical background.  Further, it would exist in perpetuity for future developers to watch……just like a movie or project documentary.

Summary

I do realize that the suggestion to move towards requirements documentaries may be fraught with organizational, legal, and other challenges.  I would never suggest that change on this scale is easy.   But I do believe there is productivity savings to be found here.  As I hinted in Decline of Written Requirements  I think we’re using 20th century tools and techniques to gather and analyze requirements in a 21st century world.  To lessen the need for process mitigation of requirements risk we may need to refocus and leverage our new tools: cell phone cameras,  high quality digital video, wikis, and blogs.

The Pragmatic Manifesto

This is a manifesto for pragmatic software development.  We are uncovering sensible, common sense practices for creating better software.  We value:

1. TEAMWORK: Working as a team with our customers, our team members, and our leaders to accomplish the product, project and company goals.

2. FLEXIBILITY: Flexibly adjusting our practices and efforts to meet the needs of the product, project and team.

3.  HUMILITY: Accepting that we don’t know everything and we should always strive to learn more and adjust our efforts as needed.

4. DISCERNMENT: Not get caught up in the latest buzz, but think for ourselves, our team, our product and our customers.

5. RISK MANAGEMENT: Realize that “risk” and the management of it, is the greatest determinant of software development success.  This doesn’t necessarily mean we avoid risk.

6. TALENT: Recognize that teams are composed of people with varying strengths and weaknesses.  There are no supermen. Sometimes we shine…sometimes we’re learning.

7. VISION:  Seeing the future and present and crafting solutions that will address both.

If you believe in this manifesto…then do it.  No need to sign up, attend classes, write books or pontificate. Just, live the principles.

Christopher R. Goldsbury

The Bad Attitudes of Agile

Who is this article for?

All software development professionals will find interest in this article, but managers, CIOs and software architects will find the greatest interest.  The topic may be controversial to many, but I offer this article as insight into what seems to be a growing problem in the Agile movement.

“Why are you here?  Agile doesn’t need managers.”

Ever hear this one before? Imagine how shocking it is to hear that the developers think your position shouldn’t exist……as if you as the manager had some contribution in creating that position.   It’s most commonly directed at project managers as they first meet the development team they’ll be working with.  To be sure the original Agile Manifesto makes absolutely no mention of project management and subsequent agile theorists go further and suggest adjusting the project manager role to be more of a coach or support role.

However, this view ignores reality.

Small non-integration dependent development projects, to be sure, probably require very little supervision of any kind as long as you have a competent, experienced and capable team.   However, the larger the project, the more integration dependent the project, and the less development centered the project…….the more a project manager is needed to coordinate, communicate and lead the overall effort.  A project in which the development portion is only 10% of the overall budget can allow a scrum master to lead the development while reporting to the project manager.

Furthermore the development team is almost never aware or good at managing a budget.  The amount of time required to develop software requires that little time be spent on anything else.  This creates a bit of a blind spot for some developers as they begin to believe that everything they are doing *IS* the project and that anyone else is just a peripheral annoyance.

The bad attitude here is the inability to recognize other roles and professions as having value and strictly adhering to a philosophical interpretation without recognizing the need for flexibility given the winds of reality.   If taken too far the attitude can come across as almost unionist or neo-communist in its presentation by extending the view to all management in all situations.  Surely the individual who adopts such an all-encompassing wholesale reduction of the corporate culture and organizational structure into one flat level is a radical.   His views are on the periphery, but if he’s the right person ( a leader ) his views can gain traction and worsen the relationship between the development team and management; turning the goal of project completion into a class warfare between management and workers.

“The team runs the project, not the managers…….we’ll decide what gets done.”

This view is often an outgrowth of the notion that management roles are no longer needed.  It flies in the face of the truth; that many decisions require collaboration among many elements of the company; not just the development team……..including the software design and architecture.

In other instances developers positing this notion are just unaware that there are other aspects to a project.  Or even worse a developer has been burned terribly by a bad experience and feels the need to “take control” of the project before some perceived breakdown occurs.

Regardless agile becomes the pre-text,  a foundation, for an attitude which suggests that most, if not all, the management structure above the development team has no contribution and should be summarily removed from contribution to the effort.  Letting this attitude take hold, in my experience, usually results in endless re-architecture sessions, severe budget-overruns,  no real end date, and a fractured emotional team that becomes disillusioned with its own mission.

“There are no due dates or schedules in Agile.”

Those of us with deep insight into capital budgets and corporate finance know how silly this is.  However, if you read Ken Schwaber’s Scrum book it does talk of abandoning the Gantt chart for a burn down chart.  In truth the burn down chart is a neat and well thought out innovation, but there are those who take this to mean that there is no schedule for delivery……i.e. the money never runs out.

This was a painful experience for myself.  I watched as a team led by a strong, charismatic technical leader ,whom we all reported to, abandoned any time based goals in favor of just producing a “working product” for the customer.  Without any time boundaries the team careened every which way.  Work ethics declined or were non-existent.  Those who wanted the product to succeed lost any motivation and drive.  The customers became bewildered as to why so much emphasis was being placed on various technical architectures while features and product change requests became lost.  The burndown charts further confused them.  All they really wanted to know was; when will the product be complete?  The team would only respond with; “We’re not on a schedule. We keep developing until we’re done.”

 When attempts were made to set realistic goals by anyone; they would immediately get knocked down as “anti-agile”.   When the team was informed that their project was hopelessly over budget; their eyes looked clouded, and confused.  The connection between what they were doing, time and, ultimately, money had become lost in the abstract design patterns etched on their whiteboards.

Realistically…….there is always a due date and a schedule for delivery; explicit or implicit.  No one puts up money for a development project with the view that it will never complete.   Even more realistically, I’ve found that Gantt charts are still very useful for coordinating deep integration or non development aspects between non-agile teams and agile teams.

The no schedule attitude arises mostly because agile techniques put forth the notion that the project should continue to add new features until the money runs out.  This is ideal and ignores what happens when a development team hasn’t even completed the bare minimum of functionality within the budget; rendering the application useless.  The bad part of this attitude is taking a new technique for tracking team progress and accountability and twisting it into a reason for not being accountable for delivery.

“Agile code is self documenting.  There’s no need for requirements, architecture diagrams or technical specifications.”

If you are a software architect or technical manager this attitude is usually targeting you between the eyes.  The thinly veiled attack is meant to question your role, experience, and the need to have anyone coordinating the overall technical design of that 28 million line software program that generates 78% of the company’s revenues.

Certainly it is often put forth by ignorance.  Maybe the 2000 line web app that the developer built recently required very few artifacts beyond the source code, but scale matters.   You know that, your management knows that, but this bad agile attitude chalks up your role to not staying current on development techniques like Scrum.  Major software systems require that a few minds are overseeing the direction and coordination of the technical vision and the many hundreds of hands creating it.

In my own experience this attitude came from a developer who, ironically, wanted to be one of the architecture staff.  He felt by critiquing and arguing with the technical leaders and introducing his knowledge of agile techniques they would respect him more and give him the coveted position he craved.  Instead, they found him to be annoying and a troublemaker.  Furthermore his lack of tact in introducing agile concepts left the senior technical leaders with a bad taste in their mouths for anything agile.

“Agile rapidly embraces change; all change.”

My experience with this attitude came from a manager instead of a developer.   It turned out he read “rapidly embrace change” to mean all kinds of changes……not just business requirements as was intended by the original agile creators.  So, fundamental architectural changes became commonplace and shifting between different open source technologies was seen as ‘good’  even though this meant taking the team completely away from their skill sets and setting project delivery back by months.  Organizational experimentation and rapidly dropping people in and out of roles also became part of  ’rapidly embracing change’.   The end result was a mess.

Clearly accepting change presented by customers is important, but without a system for managing that change; you’re asking for trouble.  One needs to keep track of all requirements and changes and their impact to project delivery so that this can be communicated to customers.  This is necessary to make effective project decisions.  If you don’t then the customers get the unrealistic notion that anything they ask for will be included……we know where this leads.

So the bad attitude here is accepting change without managing it.  An unmitigated free for all will only lead to dashed hopes and unmet expectations.  Change is good, but violent change is chaos.

“Agile uses generalists; we test our own software.  There’s no need for a QA group.”

Again this view is accurate in philosophical interpretation but my experience with this on, especially large, software development projects is…….you need a 2nd set of eyes looking at what the developers created and how well it works.  Pride of workmanship is great and should be fostered, but sometimes pride can turn into blind acceptance and defensiveness.  It takes a strong and deeply honest person to recognize their limitations and find ways to mitigate them.

Using generalists puts emphasis on making sure you’re staffed with a nimble group of multi-skilled individuals.  In reflection this recognizes software development as mostly craft and less production assembly.  However, as software development leaders we can’t assume perfection in human resources and ignore the facts.  It’s better to see the risks and plan for them and history has proven that developers don’t find all their own mistakes.

In my own experience the individuals holding this view disliked anyone testing their code and were prickly to any constructive criticism.  In one case in particular we found the underlying reason was because the developer really wasn’t that good at coding.  He was given training and mentoring and after many months of struggle it became clear that he was on the wrong career path. 

So using generalists is fine, but the attitude becomes stale if the hard truths of decades past are ignored in favor of philosophical purity.

Summary

In conclusion these problems could be found in the pre-agile world as well.  But in my experience these bad attitudes are finding refuge and justification in a new technique that in isolation probably never intended to present such a soapbox.  As software development leaders it’s critical that we address these viewpoints before they take hold of the agile methodology and potentially darken a good movement.   Agile has a great message; simplify, engage the customer during the product development, take ownership, and stay connected.  It would be a great disappointment to see this message lost.  So what do you think?  Are these attitudes in your shop?  How do you address them?  I’d like to hear from you.

About the Author

Christopher R. Goldsbury is a software development professional who has played the roles of developer, architect, scrum master, development manager, project manager and quality assurance manager  throughout his career.  Chris writes on his experiences and ideas at his blog: http://www.anagilestory.com.