Home > blog > Evolution Of The Data Analytics Industry

As early as the year 1848 scientists discovered that the “Prefrontal Cortex” of the brain is a small area that defines the personality of an Individual. Later studies have shown that this area is responsible for higher order data processing, decision making and executive functions of human beings.

Drawing a loose analogy one could argue that the Data, Analytics function performs that role within an Organization. As the modern corporation has evolved, so has the “Prefrontal Cortex” over the last 30 years. And one would argue that this evolution and development will gather pace as we move ahead.

This piece examines this evolution over the last thirty years.

Analytics 1.0: The era of ‘Business Intelligence’

Organizations have always recorded, aggregated and analysed data about production processes, sales, and customer interactions. Data sets were small and stable in velocity to allow for segregation in data warehouse for analysis. However, more time was spent in preparing data for analysis and relatively little time on the analysis itself – which was painstakingly slow often taking weeks to perform.

At the start of the new millennium modelling of data for analytics received a boost thanks to Ralph Kimball and Bill Inmon who did some pioneering work. Analytics stepped into mainstream when the relational database came of age. Technology vendors came out with products like IBM DB2, Oracle V3, Sybase (SAP) and the first standardized SQL based decision support systems went live. Still most analytics efforts were focussed on Descriptive and Diagnostic outcomes. This era lasted till the early to middle of the millennium.

Internet goes Global: Enter Analytics 2.0

Amazon (1995), Hotmail (1996), PayPal (1998), Google (1998)

Early and Mid 2000s businesses recognised the need for powerful new tools to get ahead in the market. Many technology Innovator Companies sought ‘first mover’ advantage with accelerated new products – OLAP , Reporting, Data Mining and ETL. This led to the emergence of specialist tool vendors like Informatica ,Business Objects and SAS.

In mid to late 2000’s organizations shifted away from pure RDBMS to MPP (massively parallel processing), specialized toolsets, and advanced analytics – all in recognition of ‘DATA as a critical asset’. And data volumes grew dramatically as did the cost of storage and processing.

Analytics 3.0 starts as the World goes Social

LinkedIn (2003), Skype (2003), Facebook (2004), Twitter (2006)

In this age Web apps went into a hyper growth mode. More events, more users, more transactions and the start of the smart phone and connected era.

Technology players responded with massive multi-rack systems, 100’s of computing cores, and Terabytes of Storage. Distributed computing, advanced query plans, columnar data models and Re-programmable hardware. Major players created a new wave of MPP OLAP’s (Online Analytical Processing) platforms – Vertica (HP), Greenplum (Pivotal), Netezza (IBM), and Exadata (Oracle).

But soon organizations realise that Big Data could not fit or be analysed fast enough on a single server this led to the move to Hadoop and distributed parallel processing. To deal with relatively unstructured data, companies turned to a new class of databases known as NoSQL. New technologies – ‘In memory’ and ‘In database’ analytics were introduced for faster processing. Machine learning models started to be used for advanced analytics. The world of bland boring reports gave way to compelling and intuitive visualizations.

This era continues into the late 2000’s and early ‘10s, coinciding with the rise of the ‘Data Scientists’, the Open source revolution, Fast Data, API’s and IoT’s.

In 2013, it is recorded that WhatsApp in a day sends 31 billion messages and 700 million photos sent. These are unimaginably large data volumes and growing!

Analytics 4.0 : “Fast-Pervasive Data” is replacing “Big Data”

The next generation data scientists used both computational and analytical skills and business context to solve various problems. Analytics got embedded into decision and operational processes. As technology continues to push further – Streaming and Real time analytics is made possible by Apache Spark, Kafka among Open Source platforms. Today their new avatars are available on the leading cloud platforms enabling massive Streaming and complex analytics.

In summary much like the way Human Intelligence (Prefrontal Cortex) drove rapid progress, the Data, Analytics function is driving competitive edge in the world of business. The onus is on CxOs to drive the rapid evolution of data and analytics and integrate it into key business processes.

But one factor decision makers need to understand is the pace of change in Data Analytics technology. And how the right choices could be a major competitive differentiator for their business. As Theodore Roosevelt said “The more you know about the past, the better prepared you are for the future.

This was originally published on Business World website and is being reproduced here.

Article by

Swami Srinivasan

Senior Vice President, DataTech