I’m writing this again as a response to Andréas’ post (who wrote this as a response to mine).

As I’ve been looking at the ROI of testing the last few weeks I found out that the most used numbers are still based on the initial study of Boehm from 1979. He calculated the cost of change of a waterfall method and found out that ‘the earlier you fix the problem, the cheaper it is’. That idea is still the most relevant of all. If you look at the implementation of LEAN you get the same. Fix the issues when it occurs, instead of waiting till the end. It improves the lead time and reduces the costs.

But the numbers from Boehm are based on the waterfall method, as said. Now what is the cost of change for Agile? Of course you cannot really compare them both directly, but what I found out surprised me a little bit.

Both of them are based on studies, waterfall by Barry Boehm and STBC, and the Agile by STBC again. Why am I looking at these studies? Well people in QA always use Boehm to elaborate why they want to start testing as soon as possible and both of them are used in the IV&V method for the Public Sector in the US.

When you look at both you see a rising amount of cost after the requirements phase of the Waterfall method and after the test phase these costs just go sky high. When you look at agile there is a slight rise in costs after the coding phase in the Agile cycle, but when going into Production the costs will go sky high.

Why are the costs of Agile lower after the business case? Well, one reason is that with Agile documents are created as needed for the development process. So during the full Agile cycle there is not so much need in changing documentation, only the code when an error is found. But when the product is shipped for production, that documentation still has tob e delivered for the Management & Control team of the organization.

Note: This something that is often forgotten in Agile development teams not doing a good usage of the method. Speed is generated by producing as less as possible documentation, but it still has to be delivered at the end of the project. And it’s only human to ‘forget’ this!

But statements from Agile devotees like ‘collaboration between the various partners in the development process is what makes Agile better compared with Waterfall’ are bullshit. In Waterfall collaboration is also possible! That doesn’t have to be a difference. It’s only true that quite a lot of times this is not done in Waterfall. But maybe you have some ideas around this

But one thing is the same in Agile and Waterfall: issues found in Production are always very expensive to fix!

1. Armando Dörsek Says:
   January 11th, 2012 at 08:33

   Hi Ewald,

   First of all: thanks for the post, it makes one think.

   I am wondering if the graphs by STBC are showing the costs by produced *value*. I don’t think so. I think they’re showing the total project costs? By adhering to the initial project goals the Waterfall method will probably create functionality too that, in the end, holds only little value to the client. Still this (designed, coded and tested) part will cost money too.

   Therefore, I like Agile methods so much, not only because of the focus on collaboration but also because of the ‘LEAN’ part of it: being keen on delivering the most valued functionality only and early. Note: It is important to have maintenance teams on board as well, not only the “end user”, this should take care most of the documentation issues.

   Drawbacks of Agile IMHO are that 1) cost estimation upfront and 2) getting end users to properly collaborate with the agile team prove difficult and 3) getting acceptance (tested) is therefore sometimes difficult too.

   Looking forward to your next posts.

   – Armando

2. streser Says:
   January 11th, 2012 at 13:33

   Hi,

   I have one concern when I’m looking at this charts. When we are talking about agile methods and processes using continuous integration which is used in almost every Agile projects and on higher level it become even continuous delivery we can not specify phases like requirements, testing, coding etc. Every change is done/should be done in circles of analyze-project-code-test-deliver-… there are no separated phases so you can not specify costs on each phase.

   I would like to see details of this reaserch

3. Ewald Roodenrijs Says:
   January 11th, 2012 at 14:34

   Streser,

   Thanks for your reply. Yes I agree that when you completely use an iteration for all phase the numbers don’t add up. But, and that’s why I wrote this post, most ‘Agile’ are not Agile, they are an integration of Requirements and Design and part of the coding. Sometimes they take test into the interations, but then Acceptance is done at the end of an interation or at the end of all interations. Most of the time it just isn’t possible to get users fully into the project.

   The STBC research is interpreted by me in this way, just Google.

   -Ewald

4. Ewald Roodenrijs Says:
   January 11th, 2012 at 14:36

   Armdando,

   Thanks for your reply and the content in it.

   And yes, this is only about costs.

   -Ewald

5. Curtis Yanko Says:
   January 13th, 2012 at 00:12

   How is Agile not the answer? If anything it IS the answer. Agile requires you do the testing in iteration which forces you to do automated testing which in turn drives down it’s cost and increases it’s efficacy.

6. Ewald Roodenrijs Says:
   January 19th, 2012 at 05:49

   Curtis,

   Yes, Agile advices that. But I don’t see it enough. It’s still difficult to implement. But it’s not about the title only, it’s about cost saving with agile compared to waterfall. I don’t think agile is different from waterfall on automated testing. Why can’t I do that in waterfall? The thing you can say, as you are, it forces you more to automate as you do short cycle testing where automation supports you in most ways, but if I’m disciplined enough I can do the same at waterfall.

   Thanks for your comment.

   -Ewald

7. Bob Hartman Says:
   January 22nd, 2012 at 01:01

   I’m confused why you think agile isn’t dramatically better given the two graphs you show. Assume just a single bug shows up in each phase (I won’t get into why agile won’t have phases, let’s just take your data at face value). In the waterfall approach it appears you will have a bug during design, code, test, acceptance and production. Roughly looking at your graph it will cost 5 to fix the bug in design, 10 to fix in the one in code, 20 to fix the one in test, 50 to fix the one in acceptance and 100 to fix the one in production for a total of 185 cost units.

   In agile the phases are similar and the cost to fix a bug in each phase is basically zero until we get to test and production. At this points it costs 10 to fix something in test and 100 to fix something in production. That is a total of 110 cost units to fix a bug occurring at each level. Is not agile 40% lower in cost? And that percentage rises as more things are found earlier than production in the process.

   If we instead assume a decrease of errors along the way, perhaps 10 in the earliest phase, then 7, then 5, 3 and 1 the numbers get crazy. In waterfall you would spend 7×5+5×10+3×20+2×45+1×100=335 cost units. In agile you would spend 7×1+5×1+3×1+2×10+1×100=120 cost units. Far, far less expensive.

   To me these graphs show very clearly that agile IS the answer. What am I missing?