The IT industry is plagued by outdated-waterfall methodologies rooted in manufacturing paradigms. If I had a nickel for every home construction or moon landing metaphor referenced in my professional life, the weight of my pockets would rip my pants off. The problem is not that these types of comparisons are completely irrelevant, it’s that they lack imagination and keep us from adopting flexible and people-oriented methodologies. Software development is NOT like deploying the MARS land rover. It’s much simpler than that. By remaining stalwart supporters of waterfall methodologies we overcomplicate what can be a very organic process. Nowhere is this more evident than in the Business Intelligence space. Iterative development can be a magical alternative to antiquated-waterfall methodologies, particularly for BI work, but your potion will require a few key ingredients to work: an awesome team, a transparent budget, structure, and trust.

There certainly is a time and place for “ye-old waterfall” project. Mission critical applications where mistakes can result in loss of revenue, customer dissatisfaction, jail time, or death are well suited for ridged, measurable, and risk-averse methodologies. However, some organizations are so entrenched in this line of thinking that every project is subjected to the tortures of the waterfall. In nature, a rushing waterfall is fluid and fast. In the business world waterfalls are clunky and slow.

In the world of BI, speed is of the utmost importance. Quantifying the moves of your competition or ever-shifting business climate months after a trend has occurred does nobody any good. Modifying a report or data structure should take minutes or hours, not months or years. The lack of speed in traditional IT structures using waterfall methodologies has resulted in the popularity of autonomous reporting groups being created within business units. While this particular workaround results in faster turnaround for key reporting functions, business units often struggle with data governance and the result is multiple “sources of truth.” Whose numbers are right? Whose numbers are wrong? How can we get out of this mess?

Iterative methodologies are the key to cranking out BI deliverables with the speed that your clients desire.  However, it’s not as simple as just picking up an Agile book and having at it. First you need to ensure that your project has the right composition for success in an iterative environment.

Iterative methodologies are all about people. One commonly held lie in IT circles is that if you have a watertight process you can staff the project with chimps and everything will go just swimmingly. Proponents of iterative methodologies firmly reject this notion. It is imperative that you find client partners with the right stuff before engaging on an iterative BI adventure.

Your clients need to know their data inside and out. If they can’t crush you in the speed and ease with which they find errors and inconsistencies in test data sets and reports, try to replace them. If you can’t test quickly and informally you won’t be able to meet the tight timelines required for iterative development. Clients who are used to sitting back and letting IT do all of the work are to be avoided at all costs. Similarly, your development resources need to understand the business and processes they’re being asked to develop data and reports to support. In layman’s terms, avoid business partners who can barely figure out how to turn on their PC and developers with no interest in understanding the business. This is probably the hardest requirement to fulfill for iterative success. If you can’t find people of such a high caliber, don’t go with an iterative project approach as you’ll need the many layers of waterfall bureaucracy to help cover your ass when the project fails.

Budget transparency is key when developing BI solutions iteratively. When you sign on for an iterative project you have to be comfortable with change and an ever shifting scope. This lack of control and predictability is one of the primary reasons that IT organizations are so freaked out by taking an iterative approach with their clients. Your clients are going to want to make massive course corrections that might have dire consequences to your budget. It can’t be your budget, it has to be the team’s budget. This might sound nuts, but I actually like to give my clients a weekly status on how much money is left in the till and what our burn rate is. This allows the team to calculate how many more weeks/months of development are left. This puts the power squarely in your clients’ hands. Would you like to continue to languish in trivial details, burning money all the while, or move on to your next report, Mr. Client? You’d be surprised at how reasonable and nimble your clients can be if financial matters aren’t obscured behind a curtain like the proverbial wizard.

You’ve got your dream team assembled so now you can just roam freely towards your project objectives, right? Not quite. Structure and regimentation are still vital to your team’s success. Iterative development is not the legitimization of chaos. Regular meetings with a defined and repeatable goal must be scheduled for each development iteration. I personally prefer two week iterations with a data structure or group of reports being released at the end of each cycle. One month iterations start to feel too much like waterfall projects and lose peoples’ interest.

Finally, the most important element to any iterative BI project is trust. If you and your clients can’t walk away from the traditional blame game, you won’t be successful with iterative BI development. Like a good marriage, you must possess a fundamental understanding that perfection is a myth and that there will be hard decisions down the road. If you follow some of the suggestions listed above, you not only have a fighting chance, but can deliver real business value without throwing yourself into the metaphorical waterfall.