Gathering Data Requirements

by | 6.1.3 - Data Requirements, Concepts, Underway

As a scrum master
I want I want to understand how to make retrospectives fun
So that I can keep the sprint team engaged

The Data Requirements Gap

There are two typical challenges when we gather data requirements.  First, we capture and document the requirements in a way the person capturing them understands.  Second, we explore and model the required data structures solely based on the system of record data structures or the users reporting requirements.

Document Hand Over

Let’s look at the way we capture data requirements. We will typically capture our data requirements in a traditional application-centric requirements document which is used to document the requirements for a report or a dashboard.  This is will typically have a list of fields required, plus a specification of layout and a bunch of functional and non-functional requirements.

Once completed the Business Analyst (BA) will hand this document over to a data designer or modeller who then has to understand what the fields mean and where to find them.

Of course, the document typically holds minimal data context and the data modeller will either have a guess or engage with the key users to ask them more questions to discover the context.

This results in a gap between what the key users asked for, what the BA documented and finally what the data modeller designed for.

Data Driven or Report Driven Modelling Approach

When we first started delivering data using a Data Warehouse approach, we often took a data-driven method.  We would spend a long time looking at the data in the Systems of Record and modelling how we could land and store this data to make it easier to use in the future.   The problem with this approach was it took a long time as we often spent time understanding and modelling data that a user may never consume, just in case it was needed in the future.

Then we moved to a report driven method.  We would document the content required by the users and then acquire and model the data to meet only these reporting needs.  This was often faster to deliver, but the problem was we often ended up with a large number of disparate reporting data sets that overlapped and were hard to manage.

Both of these methods lead to a gap in expectation of the time it took to deliver data and content and the ability to quickly add or change this data in the future.

Business Event Driven Approach

Business Events are events that take place in the course of the normal operation of a business and that reoccur as business processes are executed.

What is BEAM✲?

BEAM✲ stands for Business Event Analysis & Modelling, and it’s a methodology for gathering business requirements for Agile Data Warehouses and building those warehouses. It was developed by Lawrence Corr(@LawrenceCorr) and Jim Stagnitto (@JimStag), and published in their book Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema (Amazon, eBook).

Why BEAM✲?

There are three parts to the BEAM method that provides the Corr of its value:

  1. Business Event Driven
    You identify data requirements based on Business Events.
  2. Bridging the language gap
    The data requirements are documented in a langauge and a template that can be understood by business users and developers alike.
  3. Modelstorming
    You gather these requirements using a modelstorming approach that is interactive and fun.

Business Events

One of the benefits of documenting data requirements as Business Events is that an organisations Business Events will not change often.  In fact, they tend to markedly change only when the organisation’s core business model changes.

For example, a company that makes Ice Cream will still be making Ice Cream even when they introduce new Ice Cream products, open’s in new markets or targets new types of customers.  However the Systems of Record that they use to capture the data and the types of analytics and reporting content they need will change over time.  If the organisation does indeed change the core of what it does, for example, starts delivering health care, then it becomes obvious that a major change has happened then the impact to the data requirements will be major as well.

Bridging the Language Gap

We will typically ask the key business users BI specific questions that we understand.  For example, what are the key dimensions of the business data or what is the “grain” of the data?  We understand these because we have worked with them for years, our business users have not and this leads to either the business users or the developers having to translate.

Good requirements need both developers and business users to speak a common language and part of BEAM’s magic is that it gives business users and developers this common language. For example, the business event has a Customer who Purchases Product. Business users understand this concept, there are two business aspects; Customer and Product and the interaction between them. Developers can also understand this.

This enables us to ask the question “what events happen in your business?” which is a much easier way to start gathering requirements than “what entities do you want to report on?” Following up with “what other events happen around the event we just discussed?” is better than “does this FACT table share any dimensions with other FACT tables?”  Rather than relying on the BA or Developer to translate the data requirements, the BA can facilitate the agreement about the answers to the questions using by BEAM method.

BEAM✲ relies on 7 Ws (or really 5 W’s and a couple of H’s); who, what, when, where, why, how and how many.

These simple questions enable you to gather all the data details you need to model and build. You take a business event like Customer Purchases Product and ask these questions to understand the aspects which are important to that process.

  • Who? – In this case, we have Customer and probably an Employee who are involved in this event;
  • What? – The Product is one of our whats, it’s the item that’s involved;
  • When? – The date/time the purchase happened;
  • Where? – The store the purchase occurred in.
  • Why? – This can be interpreted in different way’s, for example, it was a hot day;
  • How? – The key piece of data that tells me this actually happened, for example, OrderID or Purchase Receipt;
  • How Many? – These are the values that you need to know. e.g. How many products were purchased and what were their costs.

Using this methodology, you can understand the data behind every business event. It gives you a structured set of questions to document the requirements you need to model your data warehouse.  Even better Lawerence Corr has developed a set of Open Source Excel templates that allow you to document these data requirements in an easy and consistent way.  You can download the BEAM templates from here.


Modelstorming = Data Modelling + Brain Storming.   Modelstorming is how you gather business requirements using the BEAM✲ method.

??? whiteboard

?? collaborate by visualising

??? populate by examples

Other requirement articles.

Information Products

Information Products

As a Developer
I want to understand how to capture content reqiurements quickly
So that I can estimate their size and obtain a priority for the order to deliver

Business Rules

As a Developer I want to understand how to document business rules So that we can increase the speed of delivery Another of the more difficult capabilities in data warehousing is the definition and...

Champion vs Challenger

One of the more challenging areas in Business Intellegence and Data Warehousing is understanding how to apply tests to data that has been transformed, to prove we have delivered a correct result. It can be as simple as proving that we are presenting the right revenue,...

Mega Visioning

With the death of the 100 page Business Intellgience strategy how do you get your stakeholders to come on the journey and agree to invest in the time for your AgileBI to create new data and content for them. With a waterfall project the process is well know, you do...

Data Discovery

As a Developer I want to understand when and how to understand the source data So that I can be sure I can build the things in the users stories and as an input to the estimate of the effort required   Often the majority of the AgileBI team are...

Vision and Scope

  As a BI Practitioner I want to understand how to set expectations and limit the scope of a set of iterations So that I have some boundaries which means we will not have endless iterations or additions to requirements, without agreeing these changes upfront [[...

Identifying the Stakeholder

[[why you need to do it]] [[stakeholder onion]] [[surprise stakeholders and their new requirements]] [[identifying who should be keep informed]]