Home > blog > DataOps : The new DevOps for Analytics

Understanding the similarity and differences between DataOps and DevOps; and the relevance of DataOps in present day Banking

DataOps, while often spoken as the ‘new DevOps for Analytics’ is a collaborative data management practice focused on communication, integration and automation of data flows across an organization.

DevOps and DataOps leverage different organizational people and their expectations. DevOps serves software developers who embrace complex details of code creation, integration and deployment. DataOps users on the other hand, are usually data scientists and analysts focussed on building and deploying models and visualizations. The DataOps mindset focuses on domain expertise, and is interested in getting models to be more predictive or deciding how best to visually render data.

Where by using DevOps (Continuous Integration and Continuous Delivery) leading companies (Amazon, Google, Netflix, Facebook, Apple) have accelerated their software build lifecycle (earlier called ‘release engineering’) to reduce deployment time, decrease time to market, minimize defects, and shorten time required to resolve issues; DataOps seeks to reduce the end to end cycle time of Data Analytics – from idea origination to value creation through charts, graphs and models.

The Similarities between DataOps and DevOps

DevOps and DataOps both rely on similar architectural principles (cloud delivery) for Continuous Integration and Delivery, as also they harvest cross-team (development, operations, analysts, architects, scientists, quality monitoring and customers) collaborative energies that drive value creation and innovation.

In fact as modern organizational cultures promote ‘Data Literacy’, newer approaches (like self-service data preparation tools) come equipped with their own in-built data operations. As a result, today data practitioners not only collaborate and co-develop insights in a zero-code environment, but also streamline work delivery across the organization.

Another significant end purpose that unites the two is large scale global consumption and provisioning. As DevOps and DataOps function in high-speeds, multi-geography scenarios whilst accommodating lots of users, both need a unified management environment, where monitoring, and cataloguing can happen concurrently.

The ‘Factory-model’ in DataOps

Whereas Agile and DevOps relate to analytics development and deployment, data analytics additionally manages and orchestrates a data pipeline. Data continuously enters on one side of the pipeline, progresses through a series of steps and exits in the form of reports, models and views. The data pipeline is the ‘operations’ side of data analytics and is called ‘data factory’, just like in a manufacturing line where quality, efficiency, constraints and uptime need to be managed.

The Intellectual heritage of Data Ops

DataOps combines agile philosophies, DevOps principles, and Statistical process control (a lean engineering tool) in balancing four critical Data aspects, namely, engineering, integration, security and quality – to uniquely manage an enterprise-critical data operations pipeline.

Andy Palmer

Banking and DataOps

Broadly speaking, DataOps targets improving data and analytics quality, reducing cycle times for creating new analytics, and increasing the productivity of the data organization exponentially.

More so, Banks have an additional criticality in adopting DataOps powered infrastructure: that of preventing (and bouncing back) from system breaches, platform outages, and process malfunctions that inevitably erode consumer experience and trust.

As DataOps powered platforms start to play a more pivotal role, banking organizations will realise that data needed to fuel innovation cannot be left siloed in legacy applications. Accurate data for precise decisions at the exact speeds will not only combat bottlenecks and outages but DataOps platforms that combine on-demand data delivery with data compliance will boost Customer experience transformation and innovation – especially as  more customers shift to online and mobile banking channels in the current pandemic crisis.

Article by

Muraleedhar Pai

Executive Director, Maveric Systems Ltd