The Evolution of Analytics with Data

We have made a tremendous progress in the field of Information & Technology in recent times. Some of the revolutionary feats achieved in the tech-ecosystem are really worth commendable. Data and Analytics have been the most commonly-used words in the last decade or two. As such, it’s important to know why they are inter-related, what roles in the market are currently evolving and how they are reshaping businesses.

Technology ,often regarded as a boon to those already aware of its potential, can also be a curse to audiences who can’t keep up with it’s rapid growth. Each era has had it’s moments of breakthrough and an equal share of victims (or as i’d like to call them collateral damage). As of today, every monetary-driven industry completely relies on Data and Analytics for their survival.

This blog is an attempt to look over these different stages ; simplifying the various buzzwords, narrating the scenarios which were never explained and keeping an eye on the road that lies ahead. So, without further ado , Grab your “cheat-day” meal & lets take a walk down the memory lane.

Analytics 1.0   →  Need for Business Intelligence : This was the uprising of Data warehouse where customer (Business) and production processes (Transactions) were centralised into one huge repository like eCDW (Enterprise Consolidated Data Warehouse) . A real progress was established in gaining an objective, deep understanding of important business phenomena – thereby giving managers the fact-based comprehension to go beyond intuition when making decisions.

The data surrounding eCDW was captured , transformed , queried using ETL & BI tools. The type of analytics exploited during this phase were mainly classified as Descriptive (what happened) and Diagnostic (why something happened).

However , The main limitations observed during this era was that the potential capabilities of data were only utilised within organisations , i.e. , the business intelligence activities addressed only what had happened in the past and offered no predictions about it’s trends in the future.

Analytics 2.0   →  Big Data :  The certain drawbacks of the previous era became more prominent by the day as companies stepped out of their comfort-zone and began their pursuit for a wider (if not better) approach towards attaining a sophisticated form of analytics. Customers surprisingly reacted well to this new strategy and demanded information from external sources (clickstreams , social media , internet , public initiatives etc) . The need for powerful new tools and the opportunity to profit by providing them – quickly became apparent. Inevitably , the term ‘Big data’ was coined to distinguish from small data as it was not generated purely by a firm’s internal transaction systems.

What companies expected from their employees was to help engineer platforms to handle large volumes of data with a fast-processing engine . What they didn’t expect – was a huge response from an emerging group of individuals or what is today better known as the “Open Source Community”. This was the hallmark of Analytics 2.0.

With the unprecedented backing of the community , Roles like Big-Data Engineers , Hadoop Administrators grew upon the job-sector and were now critical to every IT organisation. Tech-firms rushed to build new frameworks that were not only capable of ingesting , transforming and processing big-data around eCDW/Data Lakes but also integrating Predictive (what is likely to happen) analytics above it. This uses the findings of descriptive and diagnostic analytics to detect tendencies, clusters and exceptions, and to predict future trends, which makes it a valuable tool for forecasting.

In today’s tech-ecosystem , I personally think the term big-data has been used, misused & abused on many occasions. So technically, ‘big data’ now really means ‘all data’ — or just Data.

Analytics 3.0  →  Data Enriched Offerings :  The pioneering big data firms began investing in analytics to support customer-facing products, services, and features. They attracted viewers to their websites through better search algorithms, recommendations , suggestions for products to buy, and highly targeted ads, all driven by analytics rooted in enormous amounts of data. The outbreak of the Big-Data phenomena spread like a virus, so now it’s not just tech-firms and online companies that can create products and services from analysis of data. It’s practically every firm in every industry.

On the other hand, the wide-acceptance for big-data technologies had a mixed impact . While the tech-savvy giants forged ahead by making more money, a majority of other enterprises & non-tech firms suffered miserably at the expense of not-knowing about the data. As a result, a field of study Data Science was introduced which used scientific methods, exploratory processes, algorithms and systems to extract knowledge and insights from data in various forms.

Indeed, an interdisciplinary field defined as a “concept to unify statistics, data analysis, machine learning and their related methods” in order to “understand and analyse actual phenomena” with data. In other words , a well-refined data complemented with good training models would yield in better prediction results. The next-generation of quantitative analysts were called data scientists, who possessed both computational and analytical skills.

The tech-industry exploded with the benefits of implementing Data Science techniques and leveraged the full power of predictive & prescriptive (what action to take) analytics ,i.e, eliminate a future problem or take full advantage of a promising trend. Companies began competing on analytics not only in the traditional sense – by improving internal business decisions – but also by creating more valuable products and services. This is the essence of Analytics 3.0.

There has been a paradigm shift in how analytics are used today. Companies are scaling at a speed beyond imagination, identifying disruptive services, encouraging more R&D divisions – many of which are strategic in nature. This requires new organisational structure : positions, priorities and capabilities. A closely-knit team of data-driven roles ( Data Scientists , Data Engineers , Solution Architects , Chief Analysts ) when brought under the same roof, is a guaranteed-recipe for achieving success.

Analytics 4.0  →   Automated Capabilities : 

There have always been four types of analytics: descriptive, which reports on the past; diagnostic, which uses the data of the past to study the present; predictive, which uses insights based on past data to predict the future; and prescriptive, which uses models to specify optimal behaviours and actions. Although , Analytics 3.0 includes all of the above types in a broad sense, it however emphasises on the last . And it introduces — typically on a small scale — the idea of automated analytics.

Analytics 3.0 provides an opportunity to scale decision-making processes to industrial strength. Creating many more models through machine learning can let an organisation become much more granular and precise in its predictions. Having said that ,the cost & time for deploying such customised models wasn’t entirely affordable and summoned for a cheaper or faster approach.  The need for automation through intelligent-systems finally arrived and this idea (deemed as beyond-reach) that loomed on the horizon is where Analytics 4.0 came into existence .

There is no doubt that the use of artificial intelligence, machine learning, deep learning is going to profoundly change knowledge work. We have already seen their innovative capabilities in the form of Neural Machine Translation , Smart Reply , Chat-bots , Meeting Assistants etc ,which will be extensively used for the next couple of years. The data involved here originated from vast heterogenous sources consisting of indigenous types — one that requires complex training methods — and especially one that can sustain (make recommendations, improve decision-making, take appropriate actions) in itself.  Employing data-mining techniques and machine learning algorithms along with the existing descriptive-predictive-prescriptive analytics — comes to full fruition in this era. One reason why Automated Analytics is seen as the next stage in analytic maturity.

Analytics 5.0  →   Future of Analytics and Whats Next ???  : 

Analytics 4.0 is filled with the promise of a utopian society run by machines and managed by peace-loving managers and technologists. We could reframe the threat of automation as an opportunity for augmentation — combining smart humans and smart machines to achieve an overall better result.

Now, instead of pondering “What tasks currently employed by humans will soon be replaced by machines?” I’d rather optimistically question “What newly feats can companies achieve if they had better-thinking machines to assist them? or How can we prevent death tolls in a calamity-prone area with improved evacuation AI routine or Why can’t AI-driven e-schools be implemented in poverty-ridden zones ?”

Most organisations that are exploring “cognitive” technologies—smart machines that automate aspects of decision-making processes—are just putting a toe in the water. They’re doing a pilot to explore the technology. While others are working on the concept of building a Consumer-AI-Controlled platform. Personal AI agents that can communicate with other AI services or so called bots to get the job done. No more manual interventions with an AI-powered framework to steer your personal day-to-day activities.

I wouldn’t be surprised to see either of these technologies making giant leaps in the future. Surely, there’s an element of uncertainty tied to them but unlike many, I’m rather very optimistic about the intent. There’s always something waiting at the end of the road. If you’re not willing to see what it is, you probably shouldn’t be out there in the first place.

“Everything should be made as simple as possible , but not simpler”

                                                                                          Albert Einstein