The Journey

When I started the journey of working with Agile for our Business Intelligence projects I thought I would find a wealth of material available that I could leverage to help us deliver successfully.  I was was expecting to find books and blog that outlined the ‘What it is’, the ‘Why do it’ and most importantly the ‘How to do it’.  I expected to find content that described the Concepts I needed to understand, the processes I needed to implement and the templates I needed to use.

I was wrong.

While there was a raft of content on Agile available the majority is oriented towards using Agile to deliver user applications not Business Intelligence projects.  Business Intelligence projects [[dif between app and data projects ]]And while there was Agile content that clearly articulated the Why and the What, there was limited content that explain the How.  By this I mean content that enabled us to pick up Concepts. Processes and Templates that we could use to accelerate our implementation of data oriented projects.

So I started looking and reading further until I found some pieces of the puzzle.

There have been a couple of seminal authors that have helped me start and then accelerate my journey to date.

Lawrence Corr got me started with his approach for agile data requirements gathering called “Business Event Analysis and Modelling” (BEAM) [[BEAM link]].  We started using using BEAM on internal projects, then expanded its use out to embed it into the way we capture data requirements for customer projects.  It was so successful we built a one day training workshop where we taught customers how to use the approach themselves.

The next person I stumbled upon was Hans Hultgren with his Data Vault focussed approach to Agile Data Modeling.  Again we used data vault on an internal project and found it revolutionised the way we modelled data in the integration layer.  We then embarked on developing an open source tool “Optimal Data Engine” (ODE) to enable us to deliver this data layer quickly and in an automated manner.   As with BEAM we started using data vault on customer projects and moved into leveraging Han’s course material to allow us to train our customers.

Both Lawrence and Hans provided an approach that gave us agility in two of the capabilites we used to deliver business Intellgence projects with our customers.

I was lucky enough to be able to use these capabilities with customers on a number of small (typically 3 months) delivery projects.  As part of these deliveries we also started expanding our knowledge and capabilities on how to utilise the Agile approach “properly”.    From the importance of creating an Agile team with good T Skills, the need for a good Scrum Master, through to the criticality of defining Done Done and delivering Demo days.

This success left me wanting more.

I started looking at the areas that Lawrence and Han’s didn’t cover that were crucial to delivering business Inteligence in an Agile way.

These were areas such as business rules, testing, content development, how to manage the pre-delivery phase.  How we could harden up our understanding of the Agile Approach and move from “doing” Agile to “being” Agile.  And last but by no means least how to start and end a project.

At this stage I was luckily enough to be engaged to help a customer define a new way of working for their BI and DW teams. The reporting lines for the teams had just been merged under a single manager.  She was uncomfortable with the approach the teams were using to deliver, the lack of transparency on the progress they were making and and the amount of rework they seemed to constantly need to do.  The two teams also worked in isolation and needed to start working together to be successfull.

Enter the need for an end to end AgileBI approach!

The difference in being part of a team doing the work compared to a Coach and Mentor who helped the team do things in a new way is massive.  It meant I had to be able to articulate the Concepts the team needed to understand, provide some straw man Processes they could follow and adapt and lastly provide some Templates they could use.

I had already read the Agile books by Ken Collier and Ralph Hughes.  Both had published books that provided an outline of the entire puzzle, from beginning to end.  Each delved into different parts of this puzzle in detail, and were in the majority complimentary which was great.  In addition I had constantly browsed Scott Amblers raft of DAD articles to find answers and approaches to individual problems I had encountered.

The majority of what these gurus wrote made sense.  However some of it didn’t fit with the way we or our customers wanted to deliver, and some of it was not in enough detail to allow us to understand how to leverage it and use it during implementation.  No doubt if we had the ability to attend their courses things would have been different but being based in New Zealand, which really is at the end of the world, popping off to a course across the world is a little bit challenging.

[[what I did next]][[customer projects to refine the process]][[agilebi course]]