Predictive *AND* Adaptive Planning

Introduction

If you’ve managed projects for any length of time you understand the truth.  It’s almost never the case that a project is completely predictive ( waterfall ) or completely adaptive ( agile ).  It’s a mix.  There are needs for both in any project.  Those project managers, companies, and consultancies that integrate both approaches into their project management efforts are hybridizing the agile movement and shaping the reality of the future.  Let’s look at some examples that are well known, visible, historical and outside the IT market.  Why?  I want to defuse the agile vs waterfall debate for a moment and abstract things away from technology to help us see the broader picture.

Example 1: Lewis & Clark Expedition

The expedition by Lewis and Clark to navigate the Missouri River and map the western territories was an immense, RISKY, long term project entertained by a focused project team of professionals.  This is a startup.  Their goal was to map a water born ( river ) path to the Pacific Ocean.  To accomplish this goal some level up front planning was required.  You simply couldn’t plop two guys down in St. Louis and say: “For the next two weeks go up the Missouri River, then have a retrospective on what you learned, share that learning with your sponsor in Washington D.C. and plan for the next two weeks of river navigation.”

Lewis & Clark needed supplies, people who knew the land, logistical expertise, some plan up front to start the mission. But while an up front plan was necessary, much was not known.  This is where adaptive planning comes in.  They would need to adapt as maps, events, and people turned out to be unreliable or poorly understood: winters were more severe, the Missouri didn’t go all the way to the Pacific and great Mountain chain blocked their path to the ocean, and accidentally injuries and sickness delayed progress.

While not having an up front plan could have delayed and increased the cost of their project…not being able to adapt along the way would have killed it.  It was an experiment, a gamble, and to make it work required flexibility and persistence.

Standish group would have called Lewis & Clark a failure because they exceeded their baseline schedule.

Example 2:  Apollo missions to the moon.

The missions to land men on the moon and safely return them during the late 1960s and early 1970s in the United States were massive projects of unknown risk and horrible complexity.  Failure would bring down a nation and a philosophy ( democratic freedom).  They closely mirror big ERP implementations or suitably large custom software development efforts in Fortune 500 businesses today.

Could these have succeeded without predictive planning?  It’s hard to imagine they would have.  There were so many variables and unknowns that were required to be nailed down before implementation to ensure success.  In fact, predictive planning to the Nth degree is what made this possible and successful.  Accounting for every risk, having a mitigation plan for every risk, and carefully coordinating all the sub-projects to the common goal would have been hard to accomplish via strict adaptive planning.

This isn’t to say that adaptive planning didn’t play a role in adjusting and dealing with risks/issues along the way ( think Apollo 13 ), but this project was almost completely predictive by necessity at the top-level.  There was too much at stake ( human life and a nation’s perceived status in the world ).

Even though the Apollo moon missions proved to the world that the U.S. was the preeminent technology leader vs the U.S.S.R  and would be revered for decades to come….this project, using Standish’s metrics, would have been deemed ——-> FAIL.

Example 3: Building a Table in Garage with Your Woodworking Equipment.

A more personal, and human project…building a table is something that’s been done many times in the past.  This is like building a new e-commerce site for a company.  It’s been done before, they could have just bought a vended product, but for some business reason they want to build their own.

Ok, since it’s been done before there’s a pattern, and we borrow from it.  We download the diagram, purchase our materials, and define our plan.  This is predictive.  As we begin to execute we discover flaws in the plan or the procurement process.  Wrong nails were purchased or the leg supports were cut too small.   These events require us to adapt.

After building the table we’d dust off our hands and say “Wow, that’s a little over budget and it took some extra time, but I got what I wanted.”   We’d call it a success.  Standish would call it a failure.

More?

I could go on.  What about Michelangelo’s works?  How about building the Dubai Tower?  Great human efforts ( projects ) are vision building.  Successful completion and delivery requires a range of approaches and a commitment to the end goal.

Summary

Hopefully you can visualize the adaptive *AND* predictive elements in these projects.  It wasn’t one or the other.  It was a gradient.  That’s where the present is and the future will continue to be.  Hybridization is good.  It isn’t about agile vs waterfall. It’s about achievement, belief, and success.

Stop Agilizing Everything

Introduction

Agile universities, certifications, agile consulting, traveling coaches, planning poker card sets, agile software products, agile modeling, agile arm bands, countless agile books and the crazed cycle of agile conferences.

WTF????

The buzz cycle is in overdrive and it’s electrocuted the business world with the promise of faster, better and cheaper.  This article is a plea to stop.  Stop all the hype, the opportunistic profiting, and the marketing.

Good Intentions Turned Ugly

What started out as a challenge to the software development community to think outside the box ( invent, create ), abandon a one size fits all model to approaching software development and execute your projects in a pragmatic fashion that takes account of the context you’re working in….has turned into a marketing machine of horrible dimensions.

There was a time when people talked agile and you knew they were on the vanguard; trying to solve the real problems.  They cared.  They were passionate, deliberate, and informed.  Now, when you hear a colleague professing agile…they’re most likely drinking the kool-aid poured by the snake-oil agile coach from Denver or San Fran.  The formulaic response to the core problems is all too familiar and draining:

  • Poor Requirements – You need user stories and iterations.
  • Defects in Software – Continuous integration and TDD will solve that.
  • Bad estimation – Use planning poker.  It always works.
  • Change Management – Break it up into iterations and embrace the changes given in iteration reviews.

I’m not knocking these techniques.  Many are novel inventions that do have their place in SD/AD.  But instead of being offered as potential options, patterns, techniques to solving a problem among many other potential solutions; they have become a sales pitch by the opportunist preying on desperate CIOs.  Buyer beware.  Bubbles pop and my gut says the needle to prick this balloon is getting very sharp and close.

Let’s stop agilizing everything. Good ideas, tools, and techniques don’t need the word ‘agile’ pre or post fixed to be worthwhile.

Come Back Home

So turn off the scrum-o-matic. Wipe the agile makeup from your face, and put the kanban sequin dress away. There are still problems to solve.  We haven’t unraveled this thing called software development.  It’s devilishly vexing and we need good minds focused on it.  Become neo-software-amish, come back home to the forest of software trolls and invent/create again.

NEWS ITEMS: Knowing the Solution & Risk Management in Software Development

Some of my recent news items on InfoQ……..check them out

Software Feature Dilution

Introduction

Business analysts, requirements managers, and project managers will find the greatest interest in this article.

Software Product Planning

It occurred to me last week during one of our weekly iteration planning sessions that one of the most esoteric methods around product planning is deciding which requirements to turn into software features.  The more rigorous approaches look at the cost of the feature, the potential impact to ROI ( assuming there is one ), and the demand.

What’s wrong with this approach is that it considers the features and resulting requirements in isolation from one another.  By not considering how each new feature affects the existing product as a whole teams can and do end up with products in which the original feature set, that made the software successful, become diluted.  Those of you who’ve worked with me know my favorite example is CA’s Remedy product, but I think one could find other examples: the Microsoft Office suite of products may be in this camp.

Feature Dilution:  A Formula

So how would one go about constructing a measurement for feature dilution?  First – some assumptions:

  • You know or can retrieve the cost associated with the original marketable software release.
  • You know or can calculate the benefit ( ROI ) for the original marketable software release.
  • You have an estimated cost and benefit associated with any potential new software features.

Ok, so knowing these let’s construct a model for software feature dilution.  We’ll adapt a formula from the world of finance.

V – Value of sofware after Feature dilution =

((O x OP) +(N x (∑ IP1, IP2….IPn))) / (O + N)

where…….

O = original number of features

OP = Current NPV of product ( could use ROI too )

N = number of new features to be added

IP1, IP2, IPn = NPV of each new feature.

If you run this formula through some examples in time what you’ll find is that as a product matures new features need to continually generate greater returns to justify value to the original product and ultimately diluting the existing feature set.

This is exactly what should happen if we want to avoid the fate of an overly complex and unmanageable software product.  Just like stock market share dilution the product management team needs to justify that further feature dilution will grow the value of the product in terms of existing functionality…..not just that it will add to revenue.

Summary

Simplicity in software design has always been something great software architects knew yielded great products.  With this formula I hope I have provided at least a start to measuring simplicity in software.

I’m Skeptical on Agile – Sell Me

Introduction

This article will address a common reaction to those presented with the possibility of adopting agile in their enterprise: skepticism.  CIOs, application development managers, directors, and senior architects will glean the greatest insight from this but development professionals and project managers will find interest too.

On Being Skeptical

As a software professional your skepticism is not necessarily misplaced. There are plenty of agile coaches in the market today professing to deliver faster, better, and cheaper on a regular basis.  Their message is honey in the ears of the right executive.  It becomes even sweeter when you consider the economic climate that many businesses are facing today.  There is opportunism here and it would be well advised to vet any agile coach.

How do I know an agile coach is worth the money?

I’ve devised a simple matrix ( below ) to help guide one in validating an agile coach.

Weight Coach 1 Coach 2 Coach 3 Coach 1 Score Coach 2 Score Coach 3 Score
Number of Projects Managed 3 2 17 3 6 51 9
Years of experience in SD/AD 2 5 24 7 10 48 14
Highest Budget Management Experience ( 1 = true, 0 = false ) 1 1 1 0 1 1 0
Number of References Validated 3 3 8 1 9 24 3
CSM Certification ( 1 = true , 0 = false ) 1 1 1 1 1 1 1
PMP Certification ( 1 = true , 0 = false ) 1 0 1 1 0 1 1
FINAL SCORE



27 126 28

So let’s talk through this matrix a bit.  First, I give pretty heavy weighting to experience here.  In truth we’re not just looking for an agile coach we’re looking for someone with the battle scars of being in the AD/SD world and knowing when and where agile works vs more predictive methods.

We also want to know that they’ve actually implemented agile methods in other places hence the need for validating references.  The key word here is “implemented”.  There are plenty of folks who can regurgitate the agile manifesto and paraphrase the thinking of leading agile theorists, but agile coaches should show a track record of making it happen.

Budgetary management experience, in my opinion, is essential.  If they haven’t managed the dollars/euros/yen around a capital project ( or operating costs ) then they may have a very misguided notion of why projects succeed or fail.   The CSM or PMP who was merely accountable for a timeline with only a misty concept of how it related to money is ill-equipped to profess a transformation of your SDLC process.  Why?  Agile techniques profess delivering software in iterative cycles ( every 2 weeks ).  If some level of requirements aren’t complete by the end of each iteration you have two options on a fixed bid capital effort:

  • Don’t do that functionality.
  • Postpone it until you do have the capital available.

This sounds fine in theory, but the truth is that every system has some minimal set of requirements that must be completed for the software to be functionally usable.  If the money runs about before the agile projects succeeds in delivering this minimal functionality then your project will be seen as a failure.

Certifications show learning of theory.  I weight them low, but still think its practical to have these ( Certified Scrum Master and Project Management Professional ) if an agile coach is selling himself as a professional in software development delivery.  They should understand and know agile as well as more traditional management concepts and techniques.

Lastly, you should realize that agile adoption is not just a function of the agile coach.  The organization needs to be willing and able to accept the changes agility will introduce.

But my current process works, so why should I switch to agile?

If you have a working process and there is no immediate need to push you to agile then you should take the time to map out a strategy for your development shop.  Agile can be beneficial and Ryan Martens at RallySoft does a decent job of articulating when agile methods can benefit a development project.  His rendition of the Uncertainty vs Complexity diagram proposed in Stand Back and Deliver gives an AD manager a basic tool for plotting his/her projects along these two broad metrics.

project-types

What are the benefits if agile is applied to the right type of projects?

Better Risk Mitigation – Agile methods emphasize iterative delivery of software.  A standard cadence and check point to the project sponsors allows for defects, requirements misunderstanding, and general issues surrounding the effort to be mitigated on a timely basis.  Couple this with a daily stand up meeting where team members determine how to resolve issues and coordinate work and risks to the development effort are generally better managed.

Testing starts earlier – Agile development emphasizes vertically slicing your application and developing functionality incrementally.  This is a technical challenge, but assuming the development team can tier the system architecture this way, then your testers can usually start functional testing much earlier in the development cycle.

Increased Sponsor Satisfaction – Project sponsors are involved routinely through agile.  The developers have a direct line to the customers.  This continuous feedback loop usually leads to better communication and understanding between the team and customer.

Stronger Team Accountability – It takes time, but as the team culture shifts from command and control to a collaborative effort where developers take responsibility, collectively, for their work; the team begins to see how their efforts help/hinder the project.   An adjunct to this is an increased sense of pride in their work and kinship with each other.

What do I need to watch out for when adopting agile?

Cultural Shift – This can’t be under weighted.  Agile places greater emphasis on the team managing itself and its day to day activities. Subtly, agile preaches two things:

1. Development team and customer working together.  Meaning other managers and IT leadership have a de-emphasized role.  Your risking attrition by some of your better players if you ignore this.  Proper coaching and preparation for this change and its effect on roles and responsibilities is essential.

2. Team stepping up and coordinating activities among it’s members.  This is normally done by a PM or Dev manager or even a senior technical leader.  Some methodologies, like Scrum, emphasize a new role ( scrum master ) to take on the facilitation aspects.  For developers unaccustomed or uncomfortable with organizing and planning this may be difficult.

“Documentation is not needed”  – You may hear this from some agile coaches and theorists.  The original agile manifesto emphasizes working code over documentation, but as a development professional you’ll need to decide if this really makes sense  for your project.   Some of us have regulatory and legal reasons for documentation.

Dogmatic Views – I wrote about some of this in Bad Attitudes of Agile, but some team members will see agile as a very strict set of practices and may twist the theories and methodologies to suit their own ends.  By its very description agile is meant to be a flexible approach to software and application development not a rigid set of rules that cannot be altered.  There are the pragmatic agilists and then there are the agile zealots.  Watch out for the latter.

Summary

Benefits can accrue from agile methods.  These benefits, for the right projects, should result in better quality, reduced cost and schedule variance associated with requirements misunderstanding and defect management, and a more complimentary relationship with your customers.  As mentioned earlier skepticism is not misplaced, but by looking for an opportune experimental project to introduce agile a development manager can assess its applicability for his/her shop.

Agile vs Waterfall vs Lean Startup : Who Cares About the Labels?

Labels

After reading Shane Hastie’s latest InfoQ article:  Lean Startup or Agile or Lean Startup and Agile?  it became clear how much we’re all getting caught up in labels and methodologies.  My thought:  let’s drop the flag waving, become pragmatic in our approach and recognize that context plays a crucial role here.   What works for Eric Ries and his start-up may not work for the company you work for and the project you’re currently executing.

The Real Value – Techniques & Tools

The value in methodologies like Agile, Scrum,  Lean Startup,  Waterfall and others is in the techniques and tools they bring to the table.  While they are each packaged with a neat title and some intelligent proponents; we don’t need to use these in a take it or leave it fashion.  Be pragmatic and assemble your own toolkit.  I’ll reference Scott Ambler and say “context matters“.  We all need to move towards a collection, library of project and product management patterns that can be adopted within certain contexts.

Innovate 

Lastly, what’s invigorating about these debates is that we’re growing as a field.  We’re rethinking our ideas, and not settling.  This can only be good for software development and those affected, infected by it.  Yes, I meant to say “infected”.  🙂

Decline of Written Requirements

Introduction
Requirements managers, project managers, and business analysts will find the most interest in this article. Developers will find a nice challenge at the end.  🙂   In this article, I attempt to show that written requirements are no longer necessary….and a new tool, using new media….is necessary.
Why did written requirements fail us?

Requirements gathering, analysis and management has never been easy.  It’s hard work filled with nuance, half-truths, mis-interpretations, ulterior motives, impressions, imagination, emotion and misunderstanding.  It’s vision crafting.
Getting everyone on the same page to a clear and sufficiently detailed model of the proposed system requires a conductor with the right skills.  Too critical to be ignored, a lack of depth in this area can doom a project.
To help codify that vision the software industry quickly turned to documentation and documentation standards.  But, written requirements have proven to  be an inadequate fit with the abstract nature of software and these flaws became apparent:
  • “That’s not how I read it”
  • Versioning Changes
  • Big complex software = big complex requirements = never read/hardly understood
It was realized by a myriad of software professionals that relying exclusively on the documents and omitting en
gaged conversations to clarify the written word was a mistake. But….spoken requirements and conversations have these flaws:
  • “That’s not what I heard or understood.  That’s not what I remember.
  • Hallway conversations, informal clarifications that aren’t heard by all parties.
  • Complex requirements can’t easily be remembered, conveyed through conversation alone.
  • Change management is non-existent.
User stories attempt to blend the spoken and written worlds and emphasize the continuous interaction between customer and software development team.  They rightfully restored the need for conversations and engaged business domain involvement and got us away from the never ending refinement or versioned requirements books.  They get us closer to where we need to be.  But, they still miss the mark.  Some flaws of user stories and their crafting are becoming apparent:
  • Software and business teams change…..leading to a loss of highly valued historical domain knowledge that may or may not be documented in the code or the story.
  • An awful lot of time is wasted writing these down, editing them, and then clarifying the exact meaning.
  • It’s a fine line.  How much to write?  How much to discuss?  Too little and the story is worthless.  Too much and we’re back to writing all our requirements.
  • Complex business domains and logic need to be documented in detail.  A simple story ( or many simple stories ) with acceptance criteria may not be enough.  Think about requirements for the Boeing 787 flight control system.
  • User stories assume the business owner will be available ( static ) for clarification during much of the project: not a safe bet.
Where are we headed?

With the proliferation of digital video, cell phone cameras, webinars, social media, and whiteboards we have a new set of building  blocks to create the 21st century requirements management tool.  And that is my challenge to the community.  Build it.  Use these pieces to create a new tool for managing requirements that further reduces the risk of interpretation.  Here are some advantages I see:
  • The historical integrity of that requirement conversation ( and its vision ) would be preserved for any future teams to review and understand it….nuance and all.
  • Verbal as well as non-verbal communication would be preserved and could be analyzed later for better realization of the requirement.
  • Whiteboarded artifacts could easily be recorded and saved along with the video.  Adding to a thorough documentation of the conversation.
  • While this wouldn’t completely eliminate the need for clarification of requirements; it could sharply reduce it.
  • Productivity in requirements gathering  and recording could potentially jump helping out the entire software development process.
If you feel you’re up to the challenge…:)….then contact me ( chris@effectivelogic.biz ) and I’ll give you further guidance requirements.

About the Author

Christopher R. Goldsbury is a software development professional who has played the roles of developer, architect, scrum master, development manager, project manager and quality assurance manager  throughout his career.  Chris writes on his experiences and ideas at his blog: http://www.anagilestory.com.