Second thoughts on Agile…

Documentation aversion

Working software over documentation is fine but ‘only’ working software and no documentation is not a good idea, especially in case of a business process driven applications. It may be acceptable to go easy on technical documentation but there is no justification for not creating and maintaining business requirement documents. Absence of documented business rules and processes creates an enormous amount of confusion, application is hard to maintain, it’s difficult to communicate changes and reverse engineering code to drive business rules is highly unproductive and wasteful process.

Simulated uncertainty

I believe the two primary notions that give agile edge over waterfall methodology are

  • Built in process that encourages frequent and continuous review by end users
  • Ability to keep everybody on toes with daily stand-ups and rapid deliveries

Waterfall assumed that it’s possible to plan everything upfront, agile goes the opposite way and assumes that things are always changing so it’s better not to plan. While frequent changes may be possible in some projects but it’s not always the case, specifically for the projects that are business process centric.

While agile acknowledges the importance of bigger picture during initial project estimation, preliminary release plan, backlog prioritization, technology selection and architecture, it looks the other way when it comes to application design and identification of reusable components. I agree certain projects could have uncertain requirements and putting too much effort in designing would be a waste of time and energy but there are projects in which requirements tend toward a level of certainty.

I feel with agile methodologies like scrum considerable amount of effort goes in refactoring the design/code each sprint because we tried to do just enough or simplest thing possible in one of the previous sprint refusing to acknowledge the bigger picture. Lot of this effort could be saved, if depending on the context of project more time and effort is devoted towards design when project starts. Dedicated design sprint in the beginning may not be a bad idea!

Fixed amount + Fixed deadline + Fixed scope + Change friendly = Team burnout

In most of the scenarios before committing to a project, customer is primarily interested in getting estimated cost with reasonable accuracy for given requirements. The calculation of cost for executing a project is derived with some calculation based on team size and timelines. The customer is confident that cost more or less covers the defined requirements and since project will be executed with agile so change would not be an issue! It’s quite possible that sales team of an organization may further enforce this perception if not deny it.

So now we have a project with fixed cost which was calculated on basis of a fixed team size and timelines. Customer’s perception is that most of the requirements are in scope because cost was estimated on basis of these requirements, so generally customer would be averse to any change in scope. Customer may be open to possibility of varying team size and timelines but any changes to these factors would have a direct impact on the margins of the project so not a good option for company executing the project.

To top it all customer holds the perception that since it’s an agile project changes would not be an issue, process will accommodate them magically when requested. Now imagine being a member of the team who is caught in this situation with frugal initial estimates! While I acknowledge that with waterfall a project runs bigger risk of failure but team is still protected from frequent changes, also customer understands change averse nature of waterfall, so it may be a bit easier for the team to handle the situation.

I personally do not think fixed bid projects fit well with the philosophy of agile, but since they are an uneasy reality we should go easy on selling the ‘change embracing’ attitude of agile and make sure that customer understands that even with Agile change has a cost, you put something in something needs to be pushed out!

primarily
Advertisements

Waterfall guy in an Agile World

After working for considerable period of time with waterfall I recently joined Xebia. I believe it is one of the few organizations which live and breathe agile. From what I’ve seen and learnt so far I’ll say Agile Rocks!!! To me agile is simple no nonsense methodology which is all about common sense. I do not claim to be an expert on Agile or even waterfall, I just want to share what I’ve learned and understood so far working as a guy on the ground.

If somebody asked me to describe core idea of agile in fewest words I would say “frequent inspection”. Everything else in agile methodology is laid around this simple yet powerful notion. I’ve worked on lot of waterfall projects and fortunately none of them failed, all of them had a varied amount of success. For some it was smooth ride to production and others barely made it through. From last so many years I have heard and read about abysmal success record of IT projects and have often wondered why do so many of them fail?  I have always held this opinion that building software is just like building anything else… like maybe building a housing project or maybe some piece of furniture. I can already see heads nodding in disagreement! I understand these analogies may not be perfect or a right fit for all kind of software projects but I still feel on a certain level its valid for significant number of them (or at least for the ones that I worked on!). It sounds almost logical that in order to build something, analysis of what needs to be built is required (consider projects in which users have a fairly good idea of what they want). The abstract requirements discovered from analysis are taken to designing table where an effort is made to propose a domain specific solution. Once that’s done and accepted actual work starts to provide something tangible that can be delivered, deliveries go through cycles of construction/inspection/rework and then the final delivery is done.  Of course the events described in sequence above do not have such strict boundaries, but it can generally be acknowledged as high level framework of building and delivering goods.

This high level framework which I believe inspired waterfall methodology has for years worked successfully in various industries for building and delivering goods but has been a disappointment for software industry. Why this tested and tried logical process would not yield the same kind of results when it comes to software?  When I started working with Agile I sort of got answers to these questions and I’ll share those shortly, but before that I’ll like to point out two subtle differences that exist between process of construction in software and other industries for example lets say building construction.

  1. Prospects of inspection by end users
  2. Ability to accommodate change

Prospects of inspection by end users
For a building construction project end users may not fully comprehend or visualize the architectural plans but they can very well understand the progress by inspecting the construction site. They can see the limits of land by looking at outer walls, they may look at bricks laid for foundation to judge the room areas, they can see when walls are erected and plastered, location of stair case etc. With the ability to inspect they can catch anomalies early and request for change. Now compare this with a software project using similar methodology, at each stage of build nothing tangible that can be inspected by end user is available. End user mostly do not understand the design documents, they do not comprehend software design, code or component interactions. The artifacts produced at various stages of software construction do not make any sense to them, since they do not understand they cannot inspect or suggest change! The opportunity of inspection is provided towards the end of the project and often it’s too late by that time to change.

Ability to accommodate change
The notion of being able to change something significantly after it’s fully built is very powerful, software artifacts offer this rare ability to its builders (developers/designers) like no other. Can you imagine plight of construction project where teams unknowingly/knowingly did not lay down water supply pipes in washrooms? The ramification of such a blunder would be huge!  Builders of a construction project cannot afford such mistakes because a mistake like this is not easy to fix, this very fact forces them to be more rigorous.  Software developers on other hand often believe that they do not need to be too rigorous because they can change at any stage.  The ability to change later, time constraints and hassle of clarification with end user often drive developers/designers to take an easy way out. They code with what they know making assumptions, because they can fix it later if assumption was not up to the mark. This behavior can add quite substantially to project debt. Also, it’s certainly not true that consequences of changes to software would be less severe than that of a construction project, but perceived ability and ease with which it can be done can subconsciously drive procrastination in minds of some individuals.

I understand that ability to accommodate change is one of the biggest strengths of software and if done correctly can provide software construction an edge over others. Agile in fact embraces change and uses the ability of software to change as a tool to deliver.  It would be worth noting that point that I made above with regard to software ‘rigor’ should not be considered in violation of agile principal “Do the simplest thing that could possibly work”. By being rigorous I am no way suggesting that one should make baseless assumptions and try to accommodate everything, all I am saying that ability to change should not be misused to defer legitimate stuff.

Enter agile!
I think Agile was born with core idea of enabling frequent inspections of software artifact by end user, everything else in various agile methodologies was layered around this notion. Agile makes sure that artifacts are always in a form that can be inspected by end user, this enables early feedback and failure is detected early.  Waterfall is considered to be change averse because by the time project is ready for feedback from the end user it is so deep into the build that accommodating change is not easy and in certain cases not even possible without considerable time and effort.

Agile also provides a solution to second problem, though the lack of rigor may still exist but most of the issues introduced are detected early and fixed, unlike waterfall where they keep accumulating making a huge unmanageable lump of project debt.  Since Agile is able to recognize and accommodate for these two subtle differences the chances of success with Agile are much better than waterfall.

After a long stint of working with waterfall I recently joined my current organization Xebia Architects which I believe is one of the few organizations which live and breathe agile. From what I’ve seen and learnt so far I’ll say I love agile! To me agile is simple no nonsense methodology which is all about common sense. I do not claim to be an expert on Agile or even waterfall, I just want to blog what I’ve learned and understood so far working as a guy on the ground.

If somebody asked me to describe core idea of agile in fewest words I would say “frequent inspection”. Everything else in agile methodology is laid around this simple yet powerful notion. I’ve worked on lot of waterfall project and fortunately none of them failed, all of them had a varied amount of success. For some it was smooth ride to production and others barely made it through. From last so many years I have heard and read about abysmal success record of IT projects and have often wondered why do so many of them just fail? We in software industry must be doing something wrong.

I guess I have always held this opinion that building software is just like building anything else like maybe building a housing project or maybe some piece of furniture. I can already see heads nodding in disagreement! I understand these analogies may not be perfect or a right fit for all kind of software projects but I still feel on a certain level its valid for significant number of them (or at least for the ones that I worked on!). It sounds almost logical that in order to build something, analysis of what needs to be built is required, also it would be nice to assume that in kind of project I am referring to users have a fairly good idea of what they want. The abstract requirements discovered from analysis are taken to designing table where an effort is made to propose a domain specific solution. Once that’s done and accepted actual work starts to provide something tangible that can be delivered, deliveries go through cycles of construction/inspection/rework and then the final delivery is done.  Of course the events described in sequence above do not have such strict boundaries, but it can generally be acknowledged as high level framework of building and delivering goods.

This high level framework which I believe inspired waterfall methodology has for years worked successfully in various industries for building and delivering goods but has been a disappointment for software industry. Why this tested and tried logical process would not yield the same kind of results when it comes to software?  When I started working with Agile I sort of got answers to these questions and I’ll share those shortly but before that I’ll like to point out two subtle differences that exist between process of construction in software and other industries for example building construction.

1) Prospects of inspection by end users

2) Ability to accommodate change

Prospects of inspection by end users

For a building construction project end users may not fully comprehend or visualize the architectural plans but they can very well understand the progress by inspecting the construction site. They can see the limits of land by looking at outer walls, they look at bricks laid for foundation to judge the room areas, they can see when walls are erected and plastered, location of stair case etc. With the ability to inspect they can catch anomalies early and request for change. Now compare this with a software project using similar methodology, at each stage of build nothing tangible that can be inspected by end user is available. End user mostly do not understand the design documents, they do not comprehend software design, code or component interactions. The artifacts produced at various stages of software construction do not make any sense to them since they do not understand they cannot inspect or suggest change! The opportunity of inspection is provided towards the end of the project and often it’s too late by that time to change.

Ability to accommodate change

The notion of being able to change something significantly after it’s built is very powerful, software artifacts offer this rare ability to its builders (developers/designer) like no other. Can you imagine plight of construction project where teams unknowingly/knowingly did not lay down water supply pipes in master bedroom? The ramification of such a blunder would be huge!  Builders of a construction project cannot afford such mistakes because a mistake like this is not easy to fix, this very fact forces them to be more rigorous.  Software developers on other hand often believe that they do not need to be too rigorous because they can change at any stage.  The ability to change later, time constraints and hassle of clarification with end user often drive developers/designers to take an easy way out. They code with what they know on assumption, because they can fix it later if assumption was not up to the mark. This behavior can add quite substantially to “construction debt” of the project. Also, it’s certainly not true that consequences of changes to software would be less than that of a construction project, but ability and ease with which it can be done subconsciously can drive procrastination in minds of some individuals.

I understand that ability to accommodate change is one of the biggest strengths of software and if done correctly can provide software construction an edge over others. Agile in fact embraces change and uses the ability of software to change as a tool to deliver.  It would be worth noting that point that I made above with regard to software ‘rigor’ should not be considered in violation of agile principal “Do the simplest thing that could possibly work”. By being rigorous I am no way suggesting that one should make baseless assumptions and try to accommodate everything, all I am saying that ability to change should not be misused to defer legitimate stuff.

Enter agile!

I think idea of Agile was born with core idea of enabling frequent inspection of software artifact by end user and everything else in various agile methodologies was layered around this notion. Agile makes sure that artifacts are always in a form that can be inspected by end user, this enables early feedback and failure is detected early.  Waterfall is considered to be change averse because by the time we get feedback from the end user we are so deep into the build that accommodating change is not easy and in some cases not even possible without considerable time and effort.

Agile also provides a solution to second problem, though the lack of rigor may still exist but most of the issues introduced are detected early and fixed, unlike waterfall where they keep accumulating making a huge unmanageable lump of project debt.