Estimating software development is an art (part 1)

      Estimating work on software development projects… It seems crystal clear. In fact, what can be so complicated to provide a date and a cost for your project? Every industry requires estimation.

      But who estimates? The entire team? Well… yes. But who makes the final decisions? Who provides the required plan and how? How much time can you afford for an estimation? What do you need in order to start estimating? Can you afford not to estimate if you don’t have enough info?

      Initially I was puzzled why there was only ONE single person within a 50 members team that was able and entitled and allowed to estimate. Of course everyone within a project team contributed somehow, but the final decision was in on guy’s hand.

      We were skilled software developers with some experience, but still, only one of them was estimating and providing a COST and a delivery date for all our products.

      After at least 8 years within that company, I become that guy. Without any intention do to that.

      I learned by myself, in time, that ESTIMATING IS AN ART. You can’t just do it from the beginning, unless you are incredibly lucky. If you do it, you need to follow the path of an artist. Be original, find your niche, be open minded, etc…

      There it is no perfect set of rules to estimate. There are “rules of thumb”, there are “best practices” and there it is “lucky guess”, but there it is no clear process you can follow and provide an accurate estimate. You might find hundreds of estimation methodologies and looking through them you can even find great set of rules and advices. But if you are a new commer, all methodologies are useless since estimation is less than 50% about following a methodology. The rest is inspiration. Inspiration can be a muse or can be an incredible amount of knowledge, twisted and interpreted by your brain.

      Within regular processes, it is always someone to suffer: either the provider that estimated waaay to aggressive, or the client that pays waaay too much for the delivered work.

      “Time and materials” type of work might work well, but that is a fake, too, since both parties THINK they win, but they might both lose. Usually the client loses a lot of money, paying for his lack of knowledge and incapability to accurately define the problem from the beginning.

      “Fixed price” should be the “winning” concept if that fixed price is done right. The “win-win” situation in software is something very hard to achieve.

      A combination of fixed price and time and materials might reduce costs, but it is still a compromise accepted because of lack of estimation skills and capacity on any side.

      I am starting a set of various articles on this subject: how can you reach to a perfect estimation of time and cost, something that will bring profit to the provider but will not cost enormous the client, something that will allow providers to deliver in time.

      My target is not to achieve the perfect win-win scenario but to prepare both software providers and clients on the challenges of this process and also provide some quick tools on how to get closer to the truth, how to check the other party, how to approach the process.

      For a start, here are 20 of the most characteristics you have to have if you are a provider estimating a software development project:

      1. You have to learn a lot from experience, not from books. Books are good, hopefully this article will contribute, but they mean nothing if you don’t go through real experience since my experience is only good for me no matter how much of it I share: it won’t fit.
      2. You have to be impassionate about your work, dedicated. You can’t be accurate if you don’t really put your heart on it. There are so many things you have to count on that it will just feel like giving up and providing some random estimation.
      3. You have to be experienced cross-technology. How can you estimate if you didn’t “hit the wall” on s many things before? The estimation final touch is to be made by an expert that can point out risks.
      4. You have to have a past position on various layers, from developer to sys architect to DBA to hardware to project management to quality assurance. Business Analysis knowledge is useful, too. You can’t do all. At least, you can’t do all good, just touch some of them. Eventually you can be documented o the rest. You need to know very well the activities you are estimating or count on the experts on each category.
      5. You have to really really KNOW your people. Know their skills, know their personal life, know their behavior. It is all about people. Estimation is mainly about people. On guy can do it all in one week, the others can do it in at least a month. I’ve seen it happening.
      6. You have to know your business, your managers, your environment. There might be constraints and issues that will increase or reduce the time. A manager that “knows it all” and gets in the way of clean development, might certainly impact the timings.
      7. You have to know and follow closely the economical environment your business is running into. This is so obvious, you need to know your rates but also the industry rate and so on. Easy to say but did you ever tried to figure out how much the competition rates are?
      8. You have to know your client better then he knows himself. Yeah right. It certainly depends on the client, but it is also achievable since most of the clients might not know their capacity of estimating and evaluation how much are they supposed to pay.
      9. You have to know your client’s economical climate and challenges but also success stories. Can your client actually pay? How can you sell your estimate so it gets accepted? What are the milestones dates best fitting your client?
      10. You have to be fast, prompt, quick-decision maker, great negotiator. You cannot afford pending too much on items since they will lead the client to doubt your knowledge. You cannot actually “take time for full analysis” there it is no such thing: some things will be figured out as you develop the project, you just need to know where those are.
      11. You have to be able to focus and work non-stop for a period of time. Estimation cannot really give you a break. It is an activity with spikes, but it certainly keeps you long nights.
      12. You have to be able to parse in a short period of time a very big amount of information, therefore you need to have a great analytic mind. Any estimation starts from an analysis and analysis is usually based on large quantity of information.
      13. You have to be able to split this information in modules and tasks. You have to do that fast, while you are accumulating it. The split is not as simple as it sounds since the accuracy of tasks definition and dependency and assignments might influence seriously final estimation.
      14. You have to be able to split work in very small elements so that the estimated number of days / hours placed on each will have a maximum of 10% error.
      15. You have to be able to guess the unknown. Well, this is impossible right? No. How much do you need to build something that you don’t have ANY idea about. Well… maybe 3 days, maybe a week. If all items above are checked, then you will know that it is 3 days. You will know your buffer. You will ponder your error margin in time.
      16. You have to be able to overpass your ego. You have to be flexible and emulate various personalities, ways of thinking, etc.
      17. You have to be able to start from 0 and accept that it is all wrong when you are almost done.
      18. You have o be able to split information not only by functionality but also by various layers. It is like being able to label information with various labels, then filter it by label. Significant labels.
      19. You have to be able to learn from your mistakes. No estimation is perfect but the next one has to be better.
      20. You have to be both selective and organized. Selective to identify quick what is important, identify fast patterns, and organized to be able to memorize all the estimates you d and all the mistakes you make… etc…

      Is this a super-hero that I am talking about? No. It is just what I describe and every company has one. It is that guy that talks a lot, that guy that knows your birthday, that guy that all clients are talking with, still, it is the developer that thought you the first lessons as part of your induction plan, etc… If a provider does not have someone close to this, then he should look over statistics on estimates and profitability.

      Now here’s the funny part: if you don’t have skills like that within your team and you are still a profitable software provider, it means you are darn lucky and you are building a niche product that allows you to just charge a lot. Well, all fine, but I have a question: if you don’t have the estimation guru guy I am talking about,… how do you know you can’t just make even more profit than you already do? :) It might be the case you can charge more but you don’t know. If you would grow this kind of capability within the team, you would know. You would know with a 10% error margin.

      Come back soon on the blog for more on this subject. I am working on centralizing rules and best practices for estimation. This is only a light preview on the “Estimation as an art” series of articles. Let’s call it an introduction.

(to be continued)

.