Big Data Challenges – Picking the Roses Among the Thorns

In the US alone, the total dollars spent on medications reached $329.2 billion in 2013, demonstrating a growth increase of 3.2% compared to a downward rate of -1.0% back in 2012 [1]. Much of this rising success can be attributed to business intelligence (BI) solutions that effectively analyze “Big Data.”

Repositories of “Big Data” are the 21st century’s hottest tickets to help businesses transform the way they operate. Big data is acquired from in-house operations of life science organizations. In addition, thousands of hospitals, clinics, and even drug brands, have Facebook pages, Twitter accounts, newsletters, and Blogs that can generate large amounts of information – in the form of opinions, rumors, and customer feedback.

This explosion of data isn’t something new. The trend started in the 1970s and gave birth to early BI systems. What has changed is the velocity of growth, the complexity of the data streaming into the organization, and the imperative to make better use of information to transform the business.

Enterprises strive to exploit their data – collected, purchased or created.  Collecting and storing Big Data is a challenge, but sifting through those terabytes of data to harvest and harness every piece of relevant data and channel it to pertinent decision makers is a bigger challenge and of greater importance.  Large amounts of data with valuable information are created daily. However, collecting data from social networks, blogs, and public sources can result in a significant amount of “noise” – data with no real business value.  This greatly intensifies the difficulty of extracting meaningful insights from data. Fortunately, today’s most advanced BI technologies, not only support the ability to collect large amounts of data, but also the ability to understand what data is important and to whom it is relevant.

The holy grail of Big Data BI solutions is the ability to efficiently mine and analyze vast amounts of data – take advantage of the immensity of available data without drowning.  The goal is to identify unforeseen relationships between different drivers that might unveil threats and opportunities to grow sales and increase market share.  With the advent of Big Data, data sources are more disparate, making it harder to analyze and find correlations between seemingly unrelated phenomena.  It is humanly impossible to manually perform this task in a timely manner.  Since timing is the key to maximizing the business value of data analysis, only robust, automated BI solutions are able to generate valuable business results from Big Data analysis.

An effective analytical solution for Big Data must perform three essential steps:

  • ETL: Normalize, integrate, bring ALL disparate data sources together to create one large, yet coherent picture that takes into account interdependencies between the different data sources.
  • Filter the noise: Automatically scan the data to find correlations and identify valuable insights to focus on – diamonds in the rough.
  • Present findings through packaged applications that incorporate business logic in an easy to act upon fashion.

These three steps make Big Data analysis truly useful by identifying the right data and delivering it to the right people, in a timely and actionable manner.


1.      IMS Institute for Healthcare Informatics. Medicine Use and Shifting Costs of Healthcare [Internet]. 2014 April [cited 2014 Oct 22]. Available from