Data fabric and its tech details
Welcome to the final post in the data fabric series. So far we introduced data fabric and talked about the business benefits it brings. Now it is time to look at technical questions that may come to mind as you explore the concept of the data fabric. Integration With Existing Data Systems Unless you are starting a greenfield project, you already have some data systems in place. These can include warehouses, lakes, lakehouses, or some other types of storage that were built to solve the data needs of your organization. Implementing data fabric does not mean building from scratch and throwing existing components away. Instead, it treats these as data sources, the same way as any other source from the hybrid-cloud layer. Once data stewards catalog them, they are either served directly to the consumers or sent for additional integration with other data assets. This means that embracing data fabric will not interfere with the data system investments you made so far. It will build on top of them and provide extra features like governance, lineage, and data democratization. What About Data Mesh? Both data fabric and data mesh are oftentimes mentioned when discussing data democratization. If you are not familiar with the data mesh concept, check out a comprehensive explanation provided in the data mesh architecture post. Although both of them provide a solution for enabling data democratization, they take different approaches. The main difference is that data mesh focuses on organizational pieces, emphasizing the value of data ownership and domain-centric organization, while data fabric is a technology-oriented and domain-agnostic approach for achieving unified and integrated data architecture. At first, these two can be looked at as mutually exclusive, but it does not need to be the case. A company could embrace both approaches, combining organizational and technical aspects. This produces a solution composed of two layers, where the data fabric is encapsulated within the data mesh. In that case, data mesh divides organization into domain teams defining their ownerships, while data fabric is used as a technical solution for all tasks within and across domain teams. These include data virtualization, cataloging, integration, and provisioning access, which enable seamless data exchange and governance. Both standalone and hybrid approaches make sense for the right setup. So as always, the answer to the question of what to choose is “It depends”. In general, you would add data mesh to the story only if you have data complexity that requires domain orientation, a large number of domains, or some other organizational issue that requires decentralization of your data teams. Does Data Virtualization Kill Performance? The core idea of data virtualization is to provide a local virtual view on top of the multiple remote physical data sources, without the need to move or replicate them to a central storage. Moving and replicating data in traditional ETL workloads is a way for data engineers to ensure performance. So a valid question to ask now is how virtualization affects performance, either when accessing virtualized assets directly, or combining them with other data assets inside the data integration stage. The same issue could be explained with SQL. If you have multiple underlying data sources that you need to join, you can either go with a physical or virtual approach. In a physical approach, you would create a new table, and run a classic ETL process that will join underlying tables, do necessary transformations, and load the data to the new table. On the other hand, in a virtual approach, you would only create a view on top of underlying tables. The first one offers instant access to transformed data but requires a synchronization process to keep the final table up-to-date. Also, it brings additional storage requirements, because you have the same data stored twice. The virtual approach always serves the most recent data and does not require extra storage, but can be slow in case you have big tables and complex join conditions. So how does data virtualization solve performance requirements? Depending on the tool, there might be different techniques. Let’s start from the simplest. To speed things up, data virtualization could use parallelization to access all physical sources at once. Caching frequently accessed data is also an option to reduce network and storage latency, which improves performance. More complex optimization would include query optimizations and filter pushdowns, which apply filters on the source system itself, before moving data to the virtualization layer. This optimization minimizes data movement between the source systems and the data virtualization layer. Also, it enables faster joining operations, as data from individual physical sources does not contain unnecessary records. The combination of these techniques improves the performance of data integration and access, while still providing a unified virtual view of data across multiple systems. Unless your latency requirements are particularly stringent, a proper data virtualization tool may serve you very well. Augmented Data Catalog Along with data virtualization, data catalog makes the core of data fabric architecture. A catalog is managed by data stewards and used to enrich data assets with all required metadata. These include data classifications, tags, business terms, data protection rules, and data quality rules. Assigned metadata enables users to get quick access to high-quality data assets, where only authorized users can access sensitive and confidential information. As organizations have many data assets, and each one of them has its own set of columns that needs to be profiled and classified, manual metadata assignments can be time-consuming. That is where data catalog augmentation can help. By leveraging AI, some tools offer automated data profiling capabilities. These include automated data classifications and assignment of business terms, along with the detection of columns containing personal or other sensitive information that should be protected. Because of business complexity and specific requirements, mentioned AI capabilities are not designed for autonomous work. Data stewards still need to monitor and interfere when they notice incorrect assignments. Despite not being perfect, data catalog augmentation still brings major value, especially if we consider the fact