The Coming Breakup of Scrum

Introduction

Scrum, probably the most widely adopted agile software development practice, is cracking up.  The signs are evident and this article discusses the trend and what it means for agilists today.  If you practice scrum, are thinking of using it, or have used it in the past you’ll find interest in this article.

Oh yeah?  Prove it!!!

It’s a truism that rapid growth and popularity put a strain on organizations, movements, ideas…….or ,more fundamentally, the people comprising these very human vehicles.  As an idea ferments and then exponentially amplifies to encompass a wider politic of devotees and admirers there is the impending fog of dissent with the origin, the core.  From this comes splintering, debate and eventually more novelty and innovation.

Fine, but what about direct evidence for the rift in scrum?

03/05/2009 – Scrum But…Test.  Jeff Sutherland.

09/15/2009 – Ken Schwaber resigns from Scrum Alliance

06/15/2010 – Ken Schwaber establishes Scrum.org

04/20/2011 – The CSM certification wars

07/26/2011 – Forrester calls Water-Scrum-Fall the norm

08/02/2011 – Bloggers Suggesting Scrum Variations

10/06/2011 – Scrum Extensions announced – allowing modification to basic scrum.

As the agile movement hybridizes and evolves scrum is feeling the brunt of this pain. The forces of hybridization are ripping scrum into distinct camps whose views are aligned with their respective organizational needs.  Who are these camps and what will ultimately happen to scrum as this spider web is tugged from it’s anchors?

The Camps

The Enterprise – Most enterprises don’t build software as a core part of their operations.  Projects here are usually centered on delivering some base set of functionality for a fairly fixed price.  Stability and low cost of operation are prized.  The organization doesn’t usually see this as an investment…but as an expense to be managed and tracked.  The pressures enterprise software development groups face at the capital planning phase and the release phase have sculpted the Water-Scrum-Fall that’s prevalent.

The Software Companies –  In contrast to the enterprise, software companies build software as their core product.  It *IS* their operation.  Software companies see their products as an investment and it’s life cycle is fairly unlimited, or at least tied to the life-cycle of the company.  Pure scrum works in  these companies fairly well.  There’s usually still pressure to reduce the number of production releases but on the planning side managing features in a backlog fashion is workable, even preferable when you have a product that has a potentially unlimited lifespan.

The Startups – Startups face a different set of pressures from their venture capital investors.  The need to rapidly introduce new features and create value to customers and investors by gaining a competitive advantage quickly is paramount.  Startups are tearing scrum towards the Lean Startup methodology professed by Eric Ries.   The focus here is continuous delivery of new software: stability be damned.  Usually these startups don’t build software that’s mission critical or life threatening.

The Purist Methodologists – Among these are the various scrum trainers, agile coaches and general agile philosophers who profess a puritanical approach to scrum.  Their insistence on a dogmatic approach to scrum are rooted less in practice and more in theory.

What are they doing to scrum…where are they taking it?

These groups are doing what’s natural.  They’re catering scrum to their practice and profession of application life-cycle management ( ALM ).  But this tug of war for scrum’s future is changing it.  Instead of a simple, one-size fits all methodology, it’s becoming a framework of patterns & practices.

You can see this through the introduction of scrum extensions, which is a natural way of recognizing the various groups that have adopted scrum in some fashion.  With scrum becoming an umbrella concept, it’s importance is melting away.  It will devolve and eventually disappear. Instead the extensions, much like design patterns in software development, become the value added pieces that development teams will use and rally around.  Each camp will likely pool certain extensions, patterns as their ‘methodology’.   In time, these flavors of scrum will tear it completely apart.

Summary

Scrum’s popularity will likely be its undoing.  But is this something to be avoided or stopped?  Hardly.  Hybridization and evolution of scrum is a natural process that is both pragmatic and necessary.  Jeff and Ken’s baby is growing up and leaving the agile house.  The original agile manifesto signatories accomplished their mission; they broke the one size fits all ALM world down.  The new heterogeneous landscape of practices and patterns, while less clean than the bi-polar world of agile vs waterfall, is primed with greater opportunity for innovation.

Advertisements

I’m Skeptical on Agile – Sell Me

Introduction

This article will address a common reaction to those presented with the possibility of adopting agile in their enterprise: skepticism.  CIOs, application development managers, directors, and senior architects will glean the greatest insight from this but development professionals and project managers will find interest too.

On Being Skeptical

As a software professional your skepticism is not necessarily misplaced. There are plenty of agile coaches in the market today professing to deliver faster, better, and cheaper on a regular basis.  Their message is honey in the ears of the right executive.  It becomes even sweeter when you consider the economic climate that many businesses are facing today.  There is opportunism here and it would be well advised to vet any agile coach.

How do I know an agile coach is worth the money?

I’ve devised a simple matrix ( below ) to help guide one in validating an agile coach.

Weight Coach 1 Coach 2 Coach 3 Coach 1 Score Coach 2 Score Coach 3 Score
Number of Projects Managed 3 2 17 3 6 51 9
Years of experience in SD/AD 2 5 24 7 10 48 14
Highest Budget Management Experience ( 1 = true, 0 = false ) 1 1 1 0 1 1 0
Number of References Validated 3 3 8 1 9 24 3
CSM Certification ( 1 = true , 0 = false ) 1 1 1 1 1 1 1
PMP Certification ( 1 = true , 0 = false ) 1 0 1 1 0 1 1
FINAL SCORE



27 126 28

So let’s talk through this matrix a bit.  First, I give pretty heavy weighting to experience here.  In truth we’re not just looking for an agile coach we’re looking for someone with the battle scars of being in the AD/SD world and knowing when and where agile works vs more predictive methods.

We also want to know that they’ve actually implemented agile methods in other places hence the need for validating references.  The key word here is “implemented”.  There are plenty of folks who can regurgitate the agile manifesto and paraphrase the thinking of leading agile theorists, but agile coaches should show a track record of making it happen.

Budgetary management experience, in my opinion, is essential.  If they haven’t managed the dollars/euros/yen around a capital project ( or operating costs ) then they may have a very misguided notion of why projects succeed or fail.   The CSM or PMP who was merely accountable for a timeline with only a misty concept of how it related to money is ill-equipped to profess a transformation of your SDLC process.  Why?  Agile techniques profess delivering software in iterative cycles ( every 2 weeks ).  If some level of requirements aren’t complete by the end of each iteration you have two options on a fixed bid capital effort:

  • Don’t do that functionality.
  • Postpone it until you do have the capital available.

This sounds fine in theory, but the truth is that every system has some minimal set of requirements that must be completed for the software to be functionally usable.  If the money runs about before the agile projects succeeds in delivering this minimal functionality then your project will be seen as a failure.

Certifications show learning of theory.  I weight them low, but still think its practical to have these ( Certified Scrum Master and Project Management Professional ) if an agile coach is selling himself as a professional in software development delivery.  They should understand and know agile as well as more traditional management concepts and techniques.

Lastly, you should realize that agile adoption is not just a function of the agile coach.  The organization needs to be willing and able to accept the changes agility will introduce.

But my current process works, so why should I switch to agile?

If you have a working process and there is no immediate need to push you to agile then you should take the time to map out a strategy for your development shop.  Agile can be beneficial and Ryan Martens at RallySoft does a decent job of articulating when agile methods can benefit a development project.  His rendition of the Uncertainty vs Complexity diagram proposed in Stand Back and Deliver gives an AD manager a basic tool for plotting his/her projects along these two broad metrics.

project-types

What are the benefits if agile is applied to the right type of projects?

Better Risk Mitigation – Agile methods emphasize iterative delivery of software.  A standard cadence and check point to the project sponsors allows for defects, requirements misunderstanding, and general issues surrounding the effort to be mitigated on a timely basis.  Couple this with a daily stand up meeting where team members determine how to resolve issues and coordinate work and risks to the development effort are generally better managed.

Testing starts earlier – Agile development emphasizes vertically slicing your application and developing functionality incrementally.  This is a technical challenge, but assuming the development team can tier the system architecture this way, then your testers can usually start functional testing much earlier in the development cycle.

Increased Sponsor Satisfaction – Project sponsors are involved routinely through agile.  The developers have a direct line to the customers.  This continuous feedback loop usually leads to better communication and understanding between the team and customer.

Stronger Team Accountability – It takes time, but as the team culture shifts from command and control to a collaborative effort where developers take responsibility, collectively, for their work; the team begins to see how their efforts help/hinder the project.   An adjunct to this is an increased sense of pride in their work and kinship with each other.

What do I need to watch out for when adopting agile?

Cultural Shift – This can’t be under weighted.  Agile places greater emphasis on the team managing itself and its day to day activities. Subtly, agile preaches two things:

1. Development team and customer working together.  Meaning other managers and IT leadership have a de-emphasized role.  Your risking attrition by some of your better players if you ignore this.  Proper coaching and preparation for this change and its effect on roles and responsibilities is essential.

2. Team stepping up and coordinating activities among it’s members.  This is normally done by a PM or Dev manager or even a senior technical leader.  Some methodologies, like Scrum, emphasize a new role ( scrum master ) to take on the facilitation aspects.  For developers unaccustomed or uncomfortable with organizing and planning this may be difficult.

“Documentation is not needed”  – You may hear this from some agile coaches and theorists.  The original agile manifesto emphasizes working code over documentation, but as a development professional you’ll need to decide if this really makes sense  for your project.   Some of us have regulatory and legal reasons for documentation.

Dogmatic Views – I wrote about some of this in Bad Attitudes of Agile, but some team members will see agile as a very strict set of practices and may twist the theories and methodologies to suit their own ends.  By its very description agile is meant to be a flexible approach to software and application development not a rigid set of rules that cannot be altered.  There are the pragmatic agilists and then there are the agile zealots.  Watch out for the latter.

Summary

Benefits can accrue from agile methods.  These benefits, for the right projects, should result in better quality, reduced cost and schedule variance associated with requirements misunderstanding and defect management, and a more complimentary relationship with your customers.  As mentioned earlier skepticism is not misplaced, but by looking for an opportune experimental project to introduce agile a development manager can assess its applicability for his/her shop.

Why Does Agile Adoption Fail In Some Organizations?

Introduction

How often do you hear that a company attempting to adopt agile practices fails?  This article attempts to examine and explain the often overlooked key organizational reasons that agile fails, why it isn’t obvious to most of us and some potential strategies for coping with organizational impediments.  The article’s target audience is managers with budgetary responsibility although technical groups might also find interest.

Historical Perspective on Agile

Where did agile practices originate and why? The Agile Manifesto was originated by a group of software developers.  Their main pupose in creating a new software development methodology was to address some of the core problems with traditional waterfall techniques, specifically: risk around changing requirements, late feedback from quality assurance, and accountability of the development staff.  Their focus was not on how this methodology worked with the budgeting and financial aspects of a funded development effort.  In the information technology world there are two types of funding models.  One is a large  company that sees the software product as its bread and butter or at least a key differentiator for its business model ( think Oracle, or Scotttrade ). We’ll call this company X.  The other model is a company whose development effort it not critical for their overall business model and  the resulting funding is a fixed bid.  This is company Y.

Why is this important?

The answer lies in how the development effort is viewed by company X and Y financially.  In company X the software development effort is viewed as an investment, indeed the primary investment, in the company’s future.  In company Y the development effort is a small application and is viewed as an expense to be time bound and tracked.  In company X the team is funded.  In company Y the project is funded.  Read those last two sentences again.

In company X agile will succeed.  In company Y agile will fail.

In a fixed bid development effort the software development is intended to end at some point. Securing funding for the project requires that you define it up front, estimate it, resource it, develop it, test it, implement it, and then turn it over to support.  This is company Y.

In a time and materials funding scenario the company determines it has need for a software development team as there are many projects that require development well into the future.  An estimate of how big a development team they can afford is created for the budgetary time frame (1 year, 2 years), it is resourced, and then project scoping and scheduling are done.  This is company X.

See the difference?  In company X there will always be software development.  There is no end and the team is funded with that intention.  So putting that work in a backlog, prioritizing it, and estimating and reviewing it in time boxed iterations makes sense.

In company Y the effort can only afford to be funded for some subset of the budgetary period (say 3 months).  After that there is no more money or the company is unwilling to allocate additional money.  They don’t want a long term development team because they can’t afford it and besides there wouldn’t be enough development work to keep them busy anyway.  So deep controls and strong project management is required to ensure that something is delivered in that 3 month period.

Viewed this way…..financially, agile is a luxury.  It assumes that you’ll always have a software team and there will always be development work.  It assumes you’re team is funded year after year and you, as a manager, don’t have to worry about funding individual projects.  As an agile manager you’re primarily concerned with schedule, scope and capacity.  Budgets are an annual or bi-annual thing.  You flex up or down depending on the economic realities facing the company.  The criteria for success are largely centered on functionality delivered over time.

In company Y you might have $50,000.00 that is set aside to complete you’re project in 3 months.  Budgeting and expense tracking are critical and will determine whether the project is a success or not.  A manager here gets funding on a project level at various times throughout the capital cycle.  You may deliver all functionality on time and over budget, but that won’t be seen as a successful project.  As a manager in this company you are concerned with all 3 legs of the iron triangle.  Your team is likely temporary and staffed by contractor labor. Adoption of agile in this situation is a mistake…..even superficially. Why?

Estimation

The key lies in estimation.

In an agile software team you don’t estimate your work till right before you begin.  And you only estimate, in the case of scrum, the next iterations work. So how do you know how long it will take?  You don’t.  Furthermore, you really don’t care.  You’ll continue to deliver functionality every iteration.  As soon as product management and QA say you have a good enough product; it’ll be released as a production version.   You might have a guess, but until the team estimates it….you really don’t know long it will take.

In a fixed bid situation….your estimation needs to take place up front.  The company is asking you how much it will cost to build the application because they are unwilling to fund it forever.  They want it to end…….preferably sooner rather than later.  Funds are limited and your project, although perhaps necessary, is not viewed as critical to the company’s future.  Its ROI may even be negative.  Returning to the leadership of this company with the answer; “I’m not sure how long it will take….just fund the team for a year and we’ll see how it goes every iteration”  would be a mistake that would likely result in your dismissal.

In the 2nd scenario, if I told the team, after securing funding and hiring them, “we’re using scrum”: they’d estimate the work the next iteration. They would assume their estimates would be taken seriously and you’d give them time to complete the work as it unfolds regardless of whether their estimates fit your original project funding or not.  That’s only fair.  Unfortunately, that puts you as the manager in the uncomfortable position of submitting budget/schedule variances and/or cutting scope when you’re original estimate is proven to be inaccurate.  Hence, you’ve failed at managing the project and therefore…….”this agile thing doesn’t work”.

The mistake was to assume the company’s leadership understood and was organizationally committed to scrum and agile principles.  The mere fact that they are asking you to estimate the funding you’ll need to complete the project tells us otherwise.  If they had asked us, “How much does it cost to fund a software development team for the next 3 years?”  Then we’d be wise to approach it from an agile perspective.

The Real Problem

So, what is the real problem with agile adoption in organizations?  It can be boiled down to these points:

  • Agile assumes that the company wants a long term software development effort not a short term project.
  • Agile is often assumed by company leadership to be a development process with no impact on budgeting.  This is not the case.
  • Development teams assume leadership understands the implications of adopting agile at the budgetary level.

The complexity of these points can’t be underweighted.  Developers and development teams often have zero visibility into budgeting so they are unaware of how their agile efforts are being accounted for in monetary terms.  This is evident in so many agile articles on the web.  Likewise, management is often ignorant of development and specifically agile development practices.  Agile adoption requires education to ease the clash and misunderstanding of these two worlds.

So what are some of the consequences of attempting to adopt agile practices on a fixed bid project…essentially laying an agile façade over the waterfall project?

Story Points

Story points are often used on agile teams to determine the complexity of the work being done.  The number of points completed each iteration determines their velocity ( points per iteration ) and gives management an approximation on how much work can be completed in a given iteration.

If you come from a fixed bid shop like company Y your immediate question is, “How does this relate to hours?  How do I project costs and ROI?”  Truthfully, it doesn’t.  It isn’t intended to.  Again, the agilist is assuming you’re funding a software development team not a software development project.

In company X you could estimate things by hamburgers or cigarettes in each iteration.  It doesn’t matter.  You’re going to get the product done at some point ( +/-  functionality requested ).  The only real question is at what point to do we call it complete and release it to production.  Funding for the team is not contingent on estimation of effort.

In company Y project funding is directly related to estimation of effort.  It is critical that we tie this to time because our cost driver is often an hourly rate. Story points have no meaning here.

Scrum master vs PM

“Agile does away with the need for a project manager.”  Ever heard this before?  It’s scary for traditional PMs and unintentionally threatening.  However, it is correct.  If you assume that a team is funded year after year regardless of projected effort then the needs for organizing and managing the development effort are more centered on technical leadership, task and risk management.  Timelines and budgets go out the door. A scrum master is sufficient, preferable for getting the job done.

However, if you’re in a non-agile situation, like company Y….then traditional PM practices are not only valid, but essential to making sure the effort is kept within budgetary and schedule tolerances.  In this situation the leader of the development effort is being entrusted with precious company resources that can’t be wasted and needs to have the skills of a CEO-Lite.

A funded development team does change the project manager role.  A fixed bid project does not.

Daily Stand Up Meetings

Agile uses daily stand up meetings for a variety of reasons: motivation, risk mitigation, status, accountability, team building, etc.  It’s a good idea that is equally at home on a fixed bid waterfall project.  There is no reason this practice can’t continue to be used, but the team on that fixed bid effort has to realize that you’re not really doing agile and there is a deadline looming.  You also might want to weigh the time needs.  The daily stand ups should be short, but 15 minutes a day adds up when you only have 3 months of funding.

Iteration Reviews

With fully funded software teams that use agile the question of when something is done is answered incrementally.  Functionality is reviewed at the end of each iteration ( say 30 days ) and evaluated for readiness for production release.  Again, this is a good idea that could still be used in a fixed bid situation, but the business owners need to be coached to understand the iteration review as a risk mitigation and accountability technique and not a demonstration of the completed product.

Iteration Planning

Iteration planning really does assume you’re using agile in a team funded scenario.  It necessitates your costs are known and fixed for the budgetary cycle and that any estimates the team comes up with won’t play havoc with your budget.  Doing iteration planning in a fixed bid situation will almost certainly result in confusion, budget variances, and/or loss of functionality.

Burn Down Charts

Burn down charts show the progress of completing functionality in a given iteration.  They are a measure of team performance over time.  They do not illustrate how close the project is to completion.  If we were to sum all of them up…..they might show this, but given that they are usually used strictly for the development effort of a project; this won’t always be the case.

In fixed bid scenarios the question is not usually one of how much functionality the team is doing per time period.  It really doesn’t matter.  They need to get all the functionality done within the time frame that the money allows.

So using burn down charts and iterations in a fixed bid project sends the wrong signal to the team and your customers.

Summary

Firstly I would suggest that trying to adopt agile in a fixed bid scenario/project funded situation is not recommended.  Instead, as a manager, you should make the assessment of whether the company can support an agile practice/fully staffed development team through time.  If the company can; then you should use agile…it works well.  If the company can’t….then you’d be wise to stick with traditional project management practices.

If you determine that your company has the resources and the workload to support a software development effort but isn’t using agile then it comes down to convincing the leadership that agile makes sense for this effort.  Put on your change management and sales hats….you’ll need them.

Secondly, project manager and scrum master are not static roles and titles.  In one company a PM/SM may have budgetary responsibilities. In another they may not.  In some they may have direct reports in a traditional organizational structure.  In another the structure may be more matrixed and the PM/SM might have no direct reports.  I mention these differences because agile articles, like all writing, is written from the viewpoint of the author.  Too often what worked in one situation is attributed to being a superior process or technique…….when in actuality the organization and roles fit the process and technique.  Change the roles and organization and it might not work as well.  Context is often missing in the agile vs waterfall blogosphere.

Lastly, the agile debate is often likened to a philosophical war.  But from my experience and vantage the confusion is largely an outgrowth of misunderstanding.  Too often a technical manager hasn’t thought through the business implications of adopting agile.  Likewise business folks frequently misunderstand agile as being ‘some development thing’ with no relevance to how they just funded the effort.  I’ve been fortunate in my career to walk across the bridge of misunderstanding and see both sides.  Doing this has given me insight into the background budgetary assumptions that so frequently go unrecognized as the cause for agile adoption failure.