Promoting data democratization and governance with data fabric
Lately, our colleague Valentina Dugan created a great post explaining How to tackle data security and concerns. Some key terms she covers there are unauthorized access, compliance, data masking, and data democratization. I couldn’t have imagined a more perfect backdrop for data fabric, the topic I’ve been wanting to write about for the past few months. I’ll divide the topic in two parts. The first part will provide an explanation of data fabric architecture and tell why it is crucial for companies that aim to be data-driven. In addition, I’ll discuss technical details and provide a list of potential vendor solutions. So in the end, the series might be a valuable resource for anyone involved in data-related work, from governance leaders and business users to IT leaders and developers. Drawbacks of traditional data management Let’s kick off the topic with a few questions. Are you experiencing any discomfort while reading these questions? If not, congratulations, you may close the tab. However, I believe that most of us would struggle to provide satisfactory answers to all of them. If you can relate to this, I suggest you continue reading. Paradigm shift What steps can we take to address the data management issues that were mentioned? You could address each issue individually, or come up with your own framework to tackle them collectively. However, it would require a significant investment of time and money, and if your IT team lacks sufficient expertise, the desired outcomes may not be achieved. The good news is that there already exists a design concept called data fabric, that might be just what you are looking for. Although there are a few varying definitions depending on who you ask, the general idea and architecture remain the same. A data fabric is a data architecture that integrates a set of technologies and services designed to achieve the ultimate objective of data democratization and self-service across the enterprise. If you are not familiar with the term of data democratization, be sure to Deep dive into data democratization with our Valentina. Achieving data democratization is not a trivial task. It has prerequisites that must be met, and all of them are defined in a way to serve the democratization goal. A data fabric incorporates practices to ensure secure access to data that meets regulatory requirements and compliance standards throughout its entire data lifecycle, including: Figure: Data Fabric Architecture The first challenge data fabric addresses is data collection. Oftentimes, enterprises collect the data from various sources that can reside on-premises and in multiple public clouds. Traditionally, data collection processes involved ETL/ELT data integration processes that consolidated the data into a single location. Once centralized, a governance layer is applied to the data, followed by accessing the data by consumers. However, this approach has proven to be unscalable. It requires significant effort from likely out-staffed IT departments. Additionally, data replication brings challenges of data latency, data security, and overall complexity caused by changes in business strategies and acquisitions that add new data for integration. A data fabric takes a different approach. Instead of centralizing all data, enterprises can keep data wherever it makes the most sense and centralize only the governance piece. To accomplish this, data fabric incorporates two core components: data virtualization and data catalog. The data virtualization layer abstracts the data integration complexity from data consumers, enabling real-time access and data integration, regardless of where it resides. In this way, data virtualization reduces time spent on data preparation, which leaves more time for analytics. It cuts down IT costs and efficiently provides the right data to the right people, ultimately leading to increased pace of innovation and competitiveness. Having virtualized data sources alone is not enough to achieve data democratization. Particularly for business users who may not know which data source to consult when searching for specific information. That is where the data catalog comes in. By storing metadata, the catalog helps to organize virtualized data assets and assists users in quickly finding the most appropriate data. It also provides a centralized governance layer where data stewards define data protection rules and access policies to achieve a consistent and secure data management process. I’m sure we can all agree that data quality issues are inevitable, as there will always be data sources that are unreliable by design. However, the way we handle these issues can have a significant impact on the end result, whether it’s customer satisfaction or the accuracy of decisions made based on the data. This is again the task for the data catalog. A good catalog automates data quality tasks to monitor, identify, and prevent data quality issues on both column and data asset level. It also tracks data lineage through the entire data lifecycle, which helps in finding the root cause of the issue. This is especially important in the complex systems where data records go through numerous steps between data collection and presentation layer. Data virtualization and catalog give us virtualized governed data assets. However, to create a comprehensive source of data for our analytics team or business users it’s often necessary to filter, join, and merge multiple assets. Data fabric does not leave integration behind. It places it after the governance phase and treats data assets from the catalog as data sources. The advantage of this approach, instead of directly integrating the raw data source or its replica, is that we ensure data security throughout the entire process. For instance, if you are processing sensitive data such as credit card information, you likely do not want to grant access to the entire IT department while doing integration. By applying masking and data protection rules, you could easily control access level on a user or group level, ensuring only authorized users would have access to the sensitive data. Thus, we have covered the overview and architecture of the data fabric. I hope you found it interesting and useful. The next post in the series will bring additional technical details including performance questions and discussion