Chapter I: Stitching data together

Stitching data together

Since the beginning of the pandemic, an increasing number of people have been spending more time on the internet for work, personal requirements, and pleasure. A natural consequence of this is that an enormous amount of data is being generated constantly. In India alone, according to the Economic Survey of 2021-22, there are more than 830 million internet users. That is more than twice the entire population of the US. India generates more data than any other country in the world, save China.

This data, however, is not being captured in any central application. As customers started accessing more services online, corporations too added a plethora of new applications for easy access to their services and microservices, many of which are built on the cloud. The number of applications being built for all industries, but particularly the financial services industry, has grown exponentially. This has led to a leap in the quantity of consumer data being generated. 

Increased generation and availability of data will influence the ways in which data is used. Traditionally, when organizations built their technology systems, there would be one core and, at best, a few peripheral applications. Today, the core itself is being distributed into 20 different applications. That is creating a challenge in terms of stitching together the data across all those applications.

The advent of cloud has added to the complexity. A number of the new capabilities required to run a business or offer specific services to the customer, are now available as bespoke solutions on the cloud. This means organizations need to get together the dozens of different applications – some of which are in the core systems residing on-premise, some are in edge computing while others are on the cloud residing in a data center. Interconnectivity is essential not only for analytical purposes but even basic operations.

Data fabric or mesh architecture is a way to build links between all the services and microservices of an enterprise. A trusted data fabric is a distributed environment that connects all data together within a single, consistent data management framework. Instead of siloed, restricted sets of facts, a data fabric is integrated across the enterprise and supports multi-cloud environments. It creates an intelligent layer of virtual but centralized data ecosystem that is scalable as well as flexible and enables secure data access and processing across end-users and applications in an otherwise siloed data environment. It stitches together the data from different applications and makes it consumption ready as far as the organization is concerned. As a result, data expertise expands beyond the IT department to business groups, internal operations, and customer relationships. Employees across the entire enterprise will manage, routinely access and use the data necessary to improve their decisions and processes. 

At a higher level, when there are multiple data fabrics, there is a need for a trusted data mesh to put all information together in a form that the organization can use. Data mesh is similar to data fabric in terms of its purpose, but the approach is different. While data fabric creates a single layer of virtual management on top of the data storage that houses distributed data, the data mesh approach is more about a distributed group of teams that will manage the data as per the requirement. It promotes the ‘decentralized data ownership’ approach.

As applications grow and so does the amount of data generated daily, there is a need to replace legacy data management frameworks built during the days of core, on-premise IT systems with data fabrics. Many organizations are also choosing to create a data fabric that connects the legacy core data systems with the new applications. Over the next five years, data fabrics and data meshes will become the standard of data management for all big enterprises.