Shubho Ghosal's Blog on BI Strategy

Just another weblog


leave a comment »

I attended a round table on the Impact of Big Data on Business Intelligence. The event was organized by NewVantage , a Boston based management consultancy, and included Wayne Eckerson, the previous well known face of TDWI.

Invitees included leaders of Boston financial and other firms, and I went along upon invitation by Wayne, who I knew from contributing to the Linked In TDWI group when he was moderating it. I finally met him, as well as Jens Meyer of First Marblehead, who I had known through LinkedIn years ago.

Participants included people like myself who aren’t really experienced in Big Data technologies and practices, and people who had that experience. There were really interesting people who participated, I must say, and many good points were made and insights provided. It got me thinking and prompted me to write a blog after 2 years. How time flies.

Big data has been disruptive to EDW and business intelligence, in my experience as part of an IT organization. This has happened because of business going out to Big Data solution vendors directly, bypassing IT. IT simply did not have the answer to some of business’ big data needs and honestly does not seem to have the vision to understand how it fits. As a result, business groups who are becoming more aware of such externally provided solutions and do not like partnering with IT in the first place, are developing partnerships with Big Data Vendors directly, involving IT only to provide EDW data to such solutions. This seemed like a IT failure at first, but in the end, this felt like the right thing to do, and these cloud solutions became part of the overall solution in the end, an extension of IT-provided EDW and BI. I took this insight with me to the discussion and my focus was to get answers to a couple of questions –

  • I was aware of a skills gap within IT that kept it from meeting the Big Data need. What were these skill sets? I knew about the lack of the predictive skill set within a typical IT organization, but were there others?
  • While I recognized that technologies such as Hadoop allowed querying against vast amounts of data in file based systems in their place (without replication), I saw a gap or barrier between that data and data that traditionally lay in the EDW. How could that barrier be overcome, and data or analysis from the Big Data side be seamlessly merged with data from the EDW?

After a two hour discussion and, as I mentioned, some good insights, I arrived at the following answers to my questions.

  1. There is a skill gap within IT and within BI teams, and it wasn’t just predictive as I was thinking. Some of the newer technologies don’t come with a lot of user-friendly interfaces as the traditional DBMS and BI tools. You need programmers. This explains why the newer BI positions require Java, C++ or Python skills.
  2. IT is truly falling behind because it cannot think beyond its traditional view, and is still caught up in its arcane processes and need to resist and avoid risk. A top-down leadership transformation is needed to think differently. While on this topic, an interesting gentleman named Tarek Abu-Jaber from Harvard Pilgrim Healthcare made some seemingly innocuous but, to me, visionary statements which can be encapsulated as, “Why Not?” Why can’t IT solve the problem of creating architectures that would allow unstructured and streamed data to be analyzed as easily, and in conjunction with, structured data from the DW? IT needs to start recognizing that unless it can provide solutions to the interesting problems faced by analysts today, it will become part of a stoid old bloc that everyone sees as the mainframe of analytics – dependable work-horses but essentially uncool and has-been.
  3. There is a need for architecture that overcomes the barrier between file-based systems and the EDW, between structured and unstructured, etc. Individual solutions exists in each of these silos, but I am not seeing anything that merges the two seamlessly. Wayne briefly mentioned of a Data Lake approach and technology that creates virtual tables on the file systems that might address this, but I think it’s still early stages and probably not as seamless as it seems in theory. Also, there’s bound to be performance hurdles that need to be overcome. IT is usually good at architecture and should lead the fray on this, but… (see 2)

There was another topic of discussion that I pursued as a sidebar that has been of interest to me for years but I have not seen any advances in these fronts, and this had to do with the ease of building a DW.

The speed with which IT produces analytic solutions was a hot topic. There are two aspects to this. One has to do with process. IT loves process and has a dire need to ensure every step is always followed ad nauseum. The other has to do with the technology. There are too many disparate layers in the stack and it takes too much time to build logic into the solution. The result? Weeks to months to produce a solution rather than days. The process aspect can be addressed with Agile methodologies, but this can be a right angle shift for companies not used to that approach. The technology aspect I alluded to in a post written years ago where I talked about vendors that build car parts instead of providing us cars that can be started right away. Changing a single field name in an EDW database can take days, because the change can impact many ETL processes and reports. When will we see a vendor solution that is whole, all the way from source system to BI, with no need for staging, fact or dimension, metadata layer, and finally visualization. All simplified such that the data is taken from source directly to visualization, with no layers between, with ability to scale.

There were other topics of discussion as well, such as Data Governance and the creation of a new role of Chief Data Officer.

All in all, a good day for pontificating and pondering. I feel like a newBi(e) in BI yet again, like I did 16 years ago, and it is an exciting feeling.


Written by Shubho Ghosal

April 30, 2014 at 3:51 am

Posted in Big Data, Strategy, Thought

Tagged with

Formulating a BI strategy

leave a comment »

Formulating a BI strategy for your organization can be a daunting task. Often you have folks on Linked In groups asking the question, “How do I create a BI strategy for my organization?” This can be a tricky thing to undertake. Consulting organizations have processes that evaluate the current state and desired future state or BI and propose a roadmap to get them to the future state. This doesn’t provide a complete strategy, however, though it does provide elements of it.

In order to get a strategy going, I believe you have to set the right context for it. What are you creating the strategy for? What are your goals? You might say, very generally, that your goal is to make BI successful in the organization. Or to make BI pervasive. Or something very non-specific like that. While it is possible to create a strategy to “make BI successful” in general, having more specific goals can mean a more meaningful strategy whose success is measurable. For example, you might come up with a few specific goals, such as below –

– My organization is looking to streamline cost, and so one of my goals is to minimize the cost burden of BI.

– We are a lean IT organization that cannot afford to service every BI request, so I would like to figure out how to create a strategy around self service.

– Our senior managers need quick access to critical information, and they are frustrated by the lack of it. I need a means to provide them ready access to KPIs.

– BI hasn’t really taken off in our organization. No one is excited about it. It is not considered a strategic asset. I would like to change the perception of BI and generate more excitement about it.

(Okay, this was not so specific but this is a very common problem in organizations).

So, formulate your strategy around specific goals. Goals are likely to change as an organization evolves, and therefore your BI strategy will be an evolving thing. Below are some commonly used strategies to deal with situations such as above. These are also tips to “make BI successful” in your organization, if you will.

Aligning your BI deliverables with organizational pain points and needs

This is something that my former employer, a consulting organization, taught me and this makes sense. In order to do this, you need to interview your users. While this can provide very valuable information, where it falls apart is in organizations that don’t know what they don’t know and are not very articulate about their needs. For that, a separate or complementary strategy should be adopted, such as –

Filling in gaps with industry best practices

This was an idea my current manager gave me and I loved it. If your organization cannot articulate its needs, you can propose solutions that are considered industry standard. Most organizations should have a large portion of their needs in common. While functions in the organization differ in their needs, a particular function such as Finance should have broadly similar needs across organizations. You can start with that, put it in front of your users in a “How about this” mode and see if they bite.

Making performance metrics or KPIs your cornerstone

One of the key reasons for BI’s existence is to measure the performance of organizations, often referred to as Corporate Performance Management (CPM) or Business Performance Management (BPM). Often, this gets overlooked and focus is put on very specific and mundane reports for discrete tactical needs, making BI essentially tactical in nature. Returning the focus to performance metrics will elevate the status of BI and make it more strategic. Find out if your organization uses a formal performance management approach such as the Balanced Scorecard and what metrics it is using. Find out how these metrics are delivered currently and how to deliver them better, using your data warehouse foundation. If there are gaps, propose industry standard metrics to fill them in as a starting point, and refine those.

Using Dashboards as the primary means of disseminating the KPIs

Dashboards are essentially compact visuals providing users ready access to KPIs. End users, especially senior managers, take very well to the concept of dashboards, as they tend to be very visual, making the consumption of the metrics easy. Thus, building effective dashboards should be part of every organization’s BI strategy.  Focusing on both strategic and operational needs is critical, otherwise too much focus is often paid to one side, usually the operational one, resulting in great support for the running of day-to-day operations but not enough for the making of strategic decisions. Ensuring dashboards are well designed and provide compact, visually rich views of the most critical metrics catered to each individual’s needs is critical. Enterprise wide coverage is also critical. One exercise that can help determine the set of dashboards needed for your organization is to create a matrix of needs, with the groups or functions in your organization as columns and Need Type (Operational or Strategic) as rows. Well orchestrated interviews that dig into your users’ analytic needs on both sides is required to fill in this matrix.

Creating an effective analytics portal to attract and retain users

This aspect is often overlooked. How is BI dished out? Is it a set of folders with links to reports? That is not very appealing (though it does get the job done). In a very recent project, I was exploring the idea of an analytics portal with a group who was very intent at exploring new ideas. We settled on a portal that a user would go to and directly see their dashboards, standard reports, ad-hoc analyses, BI related documents, notifications, etc. It was a customized view of each person’s analytics offered through a standard look-and-feel interface. End users loved it. You don’t want the experience to be boring and functional. A BI portal should be like a well designed web page, making it easier to get your job done.

Choosing the right technologies

If you handle large amounts of data, choosing a weak database solution will kill your BI. You need to look at Big Data platforms. Enterprise reporting platforms provided by the big vendors usually have enough functionality to meet everyone’s needs but if your organization is cost focused that may not be a very effective strategy to pick one of those. Do you need tools for self service? For OLAP style exploration? All these are considerations for coming up with a technology strategy. The biggest thing about this is that it must be aligned with the business aspect of the BI strategy. Both must occur or one or the other may fail.

Identifying the product / application roadmap

This is the other good thing that my previous consulting employer taught me. This involves coming up with a product or application roadmap to satisfy needs. After gathering needs, you need to prioritize them in importance and scope, and separate the low hanging fruit (quick wins) from the rest. I once came up with a Gartner style bubble chart that had Ease of Access to Data as one axis, and Enterprise Coverage (or something like that) as the other, with bubble size being determined on scope. I will have to dig this up and show this here. This can be a great way of representing the applications and showing which ones the quick wins are. I’ll do that shortly.

And the list goes on.

Written by Shubho Ghosal

August 30, 2012 at 3:02 am

Big Data Analytics shakes up EDW

leave a comment »

Having been on a break from consulting since the year started, I decided to catch up on my reading and started with TDWI’s very well summarized article on Big Data Analytics, written by Phillip Russom (see link elsewhere on this blog). As the article explains, Big Data Analytics is the coming together of Big Data and Advanced Analytics. Big Data is not just about large volumes, but about diversity in data types (structured, semi-structured and unstructured), and variety in data refresh speeds as well – from real-time to delayed, including traditional transactional data which presumably occurs in discrete, varied time intervals, to sensor-type data which is a continuous stream. Advanced Analytics is a revival of everything that was considered too esoteric for most EDW groups to aspire to, from recognizable techniques such as Predictive Analysis, Data Mining, Complex SQL and Statistical Analysis to some less recognizable ones such as Natural Language Processing (used to understand written text) and artificial intelligence (not sure how this is used yet). There is also content analysis, or the analysis of video and audio, that seems even more advanced.

All this is to be supported on a diverse set of fairly new platform choices ranging from hadoop-based implementations to DW appliances to columnar, analytic, or in-memory databases to in-database analytic functions.

It feels as if under the single banner of Big Data Analytics, all that was exciting albeit challenging to consider and certainly not your everyday topic in DW circles is being revived and made mainstream. Now, a distinction can be made between “old-school” EDW-BI solutions which offer a very structured and fairly predictable model of storage and consumption through well-defined data warehouses, dashboards and cubes, and this new “exploratory” or “discovery-oriented” way of looking at data. So, does the thinking around EDW need to change? And how?

Do we need to throw away our EDWs and move all our data to a brand new platform such as above? One point that was made in the article is that the data that is fed into such technologies needs to be as raw as possible, and that traditional ETL processing should not b applied. So, the data can clearly bypass the EDW entirely and be fed directly into one of the above-mentioned technologies to achieve results. Is the role of the EDW diminished by this and will it become simply a historical source to this new, all-encompassing world?

Not so soon. First of all, a little data cleansing and staging doesn’t hurt. Picture raw data with mis-spelled customer names. A little massaging can only add to the quality of the analysis. Thus, the EDW can be used to stage an optimally-cleansed data set that is used as input towards the analytics. If your EDW architecture includes an Operational (ODS) layer to it that already houses cleansed data from the transactional systems, that can be used as a source for the analytics as well.

Beyond that, the traditional database platform that houses the EDW seems a poor choice for being the main platform for supporting Big Data Analytics. For one, it does not support well the storage of the diverse data types that seem to be desired. Nor will it respond well to the exploratory nature of the analysis that seems to be heart and soul of Big Data, with its fixed indexing and partitioning schemes.

It looks like EDW implementations would have to coexist with Big Data Analytics systems, acting as a source of structured data towards it. Either Big Data Analytics would have to be supported on a separate platform, or both it and the EDW would have to be moved to the new platform (This works if the implementation is a hybrid platform supporting the range of structured to unstructured data). This makes sense as traditional EDW platforms have been notoriously ill-suited to the exploratory analysis that some users desire. This coexistence of technologies that support what is well-defined and what is fuzzy seems apt. Also, the traditional EDW approach with its proven way of handling the clearly understood analytics through dashboards and cubes does not have to be thrown away.

Another aspect to think about is that Big Data Analytics is not just for Big Organizations, or even just for implementations literally with immense volumes of data. The Advanced Analytics aspect of it can be applied to any situation. For example, text analytics can be used to extract nuggets of information from the wealth of textual data collected in any organization. I was at a recent engagement with a telecom infrastructure services provider where it occurred to me that the descriptions being collected for tower development projects could be mined for reasons why projects were being killed. Unfortunately, such topics are still considered too far-fetched. One good thing that might happen with the recent upsurge of Big Data is to bring Advanced Analytics more to the forefront.

In conclusion, Big Data Analytics seems disruptive to traditional EDW approaches at first, but in the end it appears a symbiotic union.

Written by Shubho Ghosal

January 5, 2012 at 3:17 am

TDWI Big Data Analytics User Perspectives

leave a comment »

TDWI Big Data Analytics User Perspectives

A TDWI report on Big Data Analytics

Written by Shubho Ghosal

January 5, 2012 at 2:05 am

Posted in Uncategorized

The Interplay of BI and Social Media

leave a comment »

In order to address the role that Social Media plays in BI and vice versa, first let’s look at the identity that social media will take within the organization. I don’t believe it will look anything like Twitter or Facebook, but it will take ideas from it.

Social media within the corporate firewall will be secured, not only from the public, but within groups within the organization. The key focus will be sharing of information across the enterprise, between groups. Currently, the ways information is shared electronically in organizations is via email, chat, and collaborative portals such as Sharepoint. However, few would argue that any of these avenues currently has the appeal of a Facebook, Twitter, or Linked In. So, these avenues have to grow a little and borrow some of the ideas from popular social media channels, or alternatively, we should ditch those altogether and start afresh.

Social Media within an enterprise has to have a focus. For example, I raised the notion of a Decision Tracking System in an earlier post (distinct from the old notion of a Decision Support System, by the way, and simpler). This can be an application of social media within the enterprise. Decisions will be exposed within select groups and secured from others, but visible to all within that group and open to feedback from members within the group. Analyses put forward in support of the decision could also be added to discussion threads (see my earlier post on telling stories about the data). This could be one of the critical applications of social media within the enterprise.

Sharing knowledge across analysts could be another important application. I got this idea from talking to the head of BI of a well known movie rental and streaming service. This ties in with the story-telling notion as well. The idea is to allow analysts to share their analyses with each other and present their own thoughts along with it and get feedback. This would let folks learn from each other and not repeat each other’s mistakes. 

The more global an organization is, the more social media within the enterprise could have an impact. Think about how Facebook or Linked In have brought together people from across the globe. I have always felt a thrill at being able to connect with a like-minded thinker in Spain, Poland, Russia or elsewhere. It demonstrated to me how ideas can come from anywhere and that some people essentially think similar regardless of culture or geography. Now apply this notion to sharing ideas within the global organization’s many locations. My experience is that more people are likely to open up to a social media outlet than in conference calls or face-to-face meetings. And ideas often come in the middle of the night to creative thinkers.

If the social media application had a data or information focus, that is where BI can play a role, as a provider of the information and a basis for decisions made or conclusions derived. Snippets of BI would go into the analysis that supports the discussion thread. Or, the discussion thread itself may drive the need for new BI assets as well, driving the direction that BI should take.

If we had to take BI to the next level and really connect it to decision making, social media within the enterprise could play a big role in that. The need of the hour is to develop these thoughts into product ideas that would merge these concepts and free up the BI assets within the enterprise.

Written by Shubho Ghosal

July 16, 2010 at 8:07 pm

Connecting BI to Decision Making

with one comment

Being able to connect BI to the critical, strategic decisions being made within the organization is BI’s holy grail. How often does this really happen?

BI is well established in the world of course monitoring and correction – the operational dashboard and scorecard takes good care of that. While the need and ability to put together a BI system, along with a DW, a MDM system, and other information assets is well understood, and the world is full of people who can do this and are excited and passionate about this, there are precious few people who can testify to how often these large and expensive systems were actually used in strategic and/or critical decision making in organizations.

Let us briefly speculate on the nature of decision making in organizations. While this may differ from place to place, one can safely assume that a germ of an idea (or thought or opinion) occurs in the decision maker’s mind. This may be triggered by a market event, a desire to get to the next level, or to survive in the current economy. This germ of an idea grows in the decision maker’s mind, until it is well formulated, at which point it can be shared with the group of influencers. The influencers bring their own opinions to the table, the idea gets threshed out and bandied about, and with time a decision is made.

Information can play a significant role in various points in this process. Graphs, charts, and numbers can get thrown about. One must have confidence in the numbers (data quality). Everyone must see the same results (integration). A widget should be a widget (MDM). The right visualization should be used (Visualization tools). Powerful stories must be told (see my earlier post about telling stories with your data).

The unfortunate truth is, even if this happens, there is no way to track it. Imagine if we had systems where we were able to track our decisions. Wherever we used a BI artifact, that link is preserved. If a story influenced a decision, that analysis is included. The ability to tell a story is provided – this itself is lacking in BI technology right now.

If we were able to do this, imagine the sense of importance that BI would gain within the organization. The ROI would be clearly recognizable. At the very least, people would recognize it as being a strategic asset.

There are some gaps we would have to fill technology-wise, and process-wise as well. This snippet which I wrote as part of a TDWI discussion I started captures the thought –

1. The ability to track decisions and capture the basis of them – call this a decision tracking system, if you will. By the way, what happened to decision support systems at least in the world of business? By “basis”, I mean the link to the piece of BI that supported it, if it did.

2. The ability to tell stories about your data. These are the powerful analyses that sway decisions. BI is not very good at doing this – it seems to be missing from vendor’s foci.

3. The ability to present scenarios (again, this correlates with one of Ron’s thoughts). We had seen some examples of tools that enable this, but they went away. Scenario building is critical as it can simulate the potential outcome of making a decision.

Having these capabilities would bring BI closer to decision making and make it more strategic. Vendors need to recognize this though, and provide the capability for it. Organizations would also have to adopt this within their processes.

More on this later.

Written by Shubho Ghosal

July 9, 2010 at 2:55 am

Does your data tell a story?

with 8 comments

I attended a very thought provoking presentation recently by Jonathan Koomey on turning numbers into knowledge. One aspect of the many things he was proposing really caught my attention – that having a good story is as important as having good data. This opened up a floodgate of ideas in my mind. It reminded me of two examples of very compelling analyses I have seen – both told good stories about their data.

One was by a researcher analyzing the progress of nations across time and making the point that, over the years, some nations have progressed faster in terms of learning. This was attributed to certain factors, and a simple bubble chart that was animated to show movement of the bubbles across the chart with time showed how these nations had moved to towards a better position. All along, the researcher made a compelling case around the data, bringing in the factors that were tied to the movement of data across the chart with time. He achieved this by having a narrative around the presentation of the data.

Another was a New York Times analysis on census data presented on the web. The presentation was unique in that the reader could interact with the graph and see how the trends were for a specific gender, age group or ethnicity. However, the reason why it stood out was that a story was drawn alongside the data and the graphical presentation of it – a story that clicked with the data and helped make sense of it, making us draw certain conclusions that the facts supported.

(When discussing this with my colleague at work, I was pointed to yet another example – this one was sketched in 1869! The link is here: This describes Napoleon’s army’s progress to Moscow and back, tracking the way the army dwindled due to casualties – and relating this to climatic conditions. A visionary analysis indeed and far ahead of its time).

In all of the above cases, what made the data meaningful was the story. This wasn’t a case of putting a clickable analysis interface in front of us and leaving us wondering what the heck we are supposed to do with it. The same is true for most BI solutions. We are so caught up with the technology that is being offered to us – in-memory analytics, columnar databases, cloud – that we are missing out on this extremely critical aspect of better BI – the need for technologies or simply the discipline to create a meaningful analysis around the data that we are presenting such that it leads to effective decisionmaking. This is completely opposed to the traditional notion in BI that you need to present clickable interfaces that let you analyze (the vendor classic) “which of my regions are responsible for my drop in sales in August”. It is not that simple.

How about presenting the data as a storyline such that the decisionmaker does not have to click anything? How about presenting each RELEVANT click as another piece of the analysis in a storyline, and providing enough context around the graphics such that the point of each piece of data presented is clear? Providing a headline and then leading us to some conclusions about the data?

I’m sure there are great analysts in organizations that are already doing this – but how? Is the current spate of BI technologies really helping them or hindering the process? Is this why excel is predominant in organizations- because it helps great analysts tell a story? How effective is it in doing that? I once saw a weak attempt at telling such a story in excel and it involved putting hints along with the data – small comments here and there that help you draw certain conclusions from the mess of numbers thrown at you – a first attempt at least but nonetheless a weak one.

Is “Office Integration” all we could come up with? What about putting serious thought into putting the ability to tell the story into the technology? Giving the great analyst the ability to tell their story right alongside the fancy charts and graphs? And NOT asking the users to click?

My guess is that this would transform BI – and allow decisionmakers to actually benefit from having those expensive DWBI solutions around.

I have decided to dedicate my free time to pursuing research on how the information is best presented to make decisionmaking easier – and how technology can aid it as opposed to hindering it with sheer complexity. If you are reading this and are aware of any such research being conducted now or having been in the past, please contact me at

And the next time you present data – don’t forget to tell a story  about it.

Written by Shubho Ghosal

June 26, 2010 at 2:55 am