Agile BI: The Process and the Platforms

When you hear the words Agile and Business Intelligence (BI) in the same sentence, what comes to mind?  Is it simply applying agile methodologies and concepts to your BI projects or is it something more?  Agile is defined as: “able to move quickly and easily”. It is characterized by being “quick, resourceful and adaptable”.  So what does this mean with respect to BI?  It means quickly responding to constantly changing information and analytic requirements.  It means visualizing data to quickly discern insights and increasing the quality of decisions.  It means having the tools and platforms in addition to appropriate practices in place to enable agility.

In order to understand the impact of agile BI on an organization, it is necessary to establish a foundation on how current BI solutions are developed and deployed and how traditional BI toolsets interact with these solutions. The following diagram depicts the classic approach to developing and deploying BI solutions from a data perspective:

The primary issue with this approach is that there is typically a long cycle time between when the requirements are defined, data sources are identified and a usable deliverable is produced.  This data process is often overplayed with classic waterfall approach.

The waterfall approach is a sequential process that is used in software development but can also be applied to BI solution, data warehouse and data mart development. A top down waterfall approach starts with comprehensive requirements gathering and data source definition before proceeding with design and development.  Conversely, a bottom up waterfall approach focuses on individual data marts and deriving the data source requirements.

A downside to the waterfall approach is that it contains a large amount of inertia, which precludes rapid changes in direction or design approaches.  That is not to say that it should be completely abandoned for enterprise wide BI solutions, but that it should be augmented to increase efficiency.

Another concern is that in a lot of instances, BI data may not be housed in enterprise systems.  In fact, 76% of all data needed to create BI reports already exists in standalone databases or spreadsheets. This means that content producers and information consumers may be able to gain valuable insights without extracting information from enterprise systems.

Also important to note is that users demand more from IT departments with shrinking resources. They ask for self-service platforms and reduced cycle times for BI solution deployments. They want advanced data visualizations that enable faster tactical and strategic decision-making.  In order to address these challenges, the traditional BI approach needs to adopt a more agile framework and platform.

Agile BI Methodology

The agile BI process can be depicted as:

This process, unlike waterfall, allows users to:

• Identify smaller units of work and volumes of data, adding value throughout the process instead of waiting to develop an entire data mart.
• Develop and deploy BI artifacts (reporting and analytics) alone or in workgroups.
• Achieve insights earlier via data analysis and review instead of spending time defining a complex data integration process.
• Adapt the process to specific needs and benefit from the flexibility of the Definition, Discovery, Evaluation, Deployment and Verification steps.

The Agile BI Platform

It is critical to acknowledge that agile BI is not just the implementation of an agile approach.  The platform is also an important component that can lead to the success or failure of the implementation of an agile BI environment. Evaluate solutions based on ease of implementation versus overall value.

The selected platform should address the following platform features and functions at each stage of the agile BI methodology:

Definition

With the need for rapid results and adaptability, agile BI platforms should deliver incremental value within a short duration. Ensure that the agile BI platform allows for social collaboration across multiple audiences and business functions of different sizes and domain knowledge. This will reduce risk and cost because issues will be surfaced earlier due to the smaller units of work.

Discovery

Considering that the content creator/information consumer has direct access to a large quantity of data, the BI platform should have the ability to easily connect to multiple heterogeneous data sources simultaneously. Consumers and content creators should be able to access and explore data with minimal IT intervention.

Evaluation

Data visualizations are key to agile BI platforms.  The platform must produce easy-to-understand charts and graphs in order to provide additional insights and facilitate data exploration. Questions asked of the data can serve to define an overall enterprise architecture at a later date.  This also means that there will be a much deeper collaboration between the content creator, information consumer and IT (co-location).

Deployment

The agile BI platform should facilitate fine grain deployment to audiences of any size and support the social interaction of the audience as the deployed BI artifacts (reports, visualizations, etc.) are reviewed and discussed as the verification process commences.  This further enhances and focuses collaboration, which can lead to better decisions being made at a much faster pace.

Verification

The most important step in the above process is closing the loop between “Verification and Definition” (Re-Definition).  It is imperative that this is performed in order to adapt to constantly changing information and analytic requirements.

The deployment of an agile BI environment is a journey comprised of agile methodology and a supporting BI platform.  It is not necessarily a monolithic environment, however.  There may be various best of breed components of a well-functioning agile BI environment, providing specialized capabilities to fulfill specific business needs. The agile BI environment should continually evolve based on changes in information and analytic requirements, market conditions, organizational DNA and unforeseen future needs.