“Those who cannot remember the past are condemned to repeat it.”
George Santayana, 1905

Various forms of this quote have been repeated by many famous people since George Santayana wrote in The Life of Reason, 1905.

The quote applies to many aspects of life, for example social interactions, conflict, and business. In business, entire industries have been created to capture history with data and produce historical trends that theoretically can be extrapolated to predict a future outcome.

Predictable Improvement

I don’t believe Mr. Santayana to have said, “Those who remember the past … can predict the future.”  For lack of a better solution, business leaders rely on past trends from data samples to predict the future. Unfortunately the outcome rarely reflects the prediction.

It is common to view a simple trend analysis from a sample of historical data as a window into the future. Anyone in sales or managing a P&L will tell you this is true. It is called forecasting. Many professional sports teams, pioneered by the Oakland A’s and Sabermetrics, have been predicting future performance of players based on past tendencies for over a decade. But how accurate are the predictions? I believe the answer is, “it depend.”

Absolute Certainty

Had Mr. Santayana penned, “Those who remember the past … can, with absolute certainty, predict the future”, we might certainly agree that Mr. Santayana was a fool. However, P&L managers and sales managers are often held accountable to their forecasts as if absolute certainty were possible.

How does the average business executive, who is accountable for future performance, keep his or her job?  They get very good at “sand-bagging” and at explaining variances between results and their original prediction. Of course I am oversimplifying since experience plays a significant role in forecast accuracy. But what is experience but years of data (experiences)? What if we could make years of experience available to all, in the form of data? What if we could find patterns in data that would initiate a corrective action before the outcome instead of months later?

Big Data Analytics

While there may never be an answer for “absolute certainty” of future results, Big Data and Big Data Analytics may provide a promising solution to get far closer than where most businesses are today.

Take for example the weather forecast. While our weather apps and TV forecasters are not able to predict the weather with absolute certainty, they have certainly improved over the last 10 years.  Much of this improvement comes from being able to analyze more and more historical data to find patterns and trends – referred to as weather patterns – which lead to a predictable outcome (history repeating itself).  This combination of more available historical data, Big Data storage solutions, distributed computing power, and Big Data analytics has allowed weather patterns to be identified and matched to a likely outcome within an acceptable duration of time. After all, what good is a prediction if it is not made ahead of the outcome?

Achieving Predictable Business Outcomes

Applying Big Data solutions to achieve predictable business outcomes starts with becoming aware of the possibilities. A simple approach to look at is Hortonworks’ Big Data Maturity Model shown below coupled with their self-assessment that explores 5 critical areas to advancing through the progression successfully.

Become Aware of Big Data

If you are here, Big Data is being discussed inside your company.  Perhaps the conversation involves someone saying, “We don’t have big data” or “What is Big Data”. If you collect and make business decisions on data, you have Big Data. For example, historical reports providing actual results and analyzing trends are available to executives in most businesses. How current are the data in your trend reports?  Could they be better or more current?  What questions require better answers?  If you had a crystal ball, what would you ask?  Once you realize that there is room for, and value in improvement, then you are ready for the next step ….

Explore Big Data

If you are here, IT is most likely experimenting with Big Data solutions such as Hadoop and Spark and perhaps leveraging them as economical solutions to managing and exploiting the explosion of data (quantity and type). Loading structured and unstructured data into Hadoop from more expensive legacy systems has become an early source of value for many businesses. IT may also be exploring the possibility of eliminating data silos to make all corporate data accessible to decision makers while maintaining role based access privileges to protect sensitive data. Generally, in this stage, use cases remain limited. If you are here, you are creating a foundation for optimization …

Optimize

If you are here, the business is optimizing operational performance with data. Through the elimination of data silos, they now have real time access to all data and better visibility to data flow and inefficiencies in customer, product, and facility life cycles. But they have yet to transform to using data to make better decisions and predict outcomes …

Transform

Real transformation and economic value comes with advanced analytics coupled with access to more data bringing prediction closer to certainty. The result is more effective decision making. Of equal or perhaps greater importance, new value streams are created as businesses recognize that Data is like Gold (or Bitcoins).

The Time is Now

If you are still asking what Big Data is or feel that your business does not have Big Data, you are at risk of falling further behind the large enterprises that have begun to transform their decision making processes with Big Data and predictive analytics. The more likely scenario is that you are passed by a smaller, nimbler startup whose business model has Big Data at the center.

Where are you with Big Data?  Can you predict where you will be next year?  Where do you think your competition will be?  I would love to hear your opinion.